string-ops-tokenizer / token_mapping.txt
ardauzunoglu's picture
Upload string operations tokenizer
f9fb8ba verified
Raw
History Blame Contribute Delete
365 Bytes
Token ID
<pad> 0
<s> 1
</s> 2
<unk> 3
a 4
b 5
c 6
d 7
e 8
f 9
g 10
h 11
i 12
j 13
k 14
l 15
OP_IDENTITY 16
OP_REVERSE 17
OP_ROTATE_LEFT_1 18
OP_ROTATE_RIGHT_1 19
OP_ROTATE_LEFT_2 20
OP_ROTATE_RIGHT_2 21
OP_SWAP_HALVES 22
OP_REVERSE_FIRST_HALF 23
OP_REVERSE_SECOND_HALF 24
OP_REVERSE_EACH_2BLOCK 25
OP_REVERSE_EACH_4BLOCK 26
OP_INTERLEAVE_HALVES 27
EQUALS 28
END 29