Skip to content

Commit c7582f0

Browse files
authored
v2.5
1 parent 0110a02 commit c7582f0

File tree

4 files changed

+91170
-0
lines changed

4 files changed

+91170
-0
lines changed

src/Tokenizer/special_tokens_map.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"bos_token": "<s>",
3+
"eos_token": "</s>",
4+
"unk_token": "<unk>"
5+
}

0 commit comments

Comments
 (0)