Kotensor
Toggle table of contents
1.2.0
jvm
Target filter
jvm
Switch theme
Search in API
Skip to content
Kotensor
Kotensor
/
io.kotensor.core
/
BPETokenizer
/
Companion
Companion
object
Companion
(
source
)
Members
Properties
END_
OF_
TEXT
Link copied to clipboard
const
val
END_OF_TEXT
:
String
GPT2_
PRE_
TOKENIZER_
PATTERN
Link copied to clipboard
const
val
GPT2_PRE_TOKENIZER_PATTERN
:
String
IM_
END
Link copied to clipboard
const
val
IM_END
:
String
IM_
START
Link copied to clipboard
const
val
IM_START
:
String
Functions
compile
Unicode
Pattern
Link copied to clipboard
fun
compileUnicodePattern
(
regex
:
String
)
:
Pattern
from
File
Link copied to clipboard
fun
fromFile
(
file
:
File
)
:
BPETokenizer
from
Json
Link copied to clipboard
fun
fromJson
(
json
:
String
)
:
BPETokenizer