Kotensor
Toggle table of contents
1.2.0
jvm
Target filter
jvm
Switch theme
Search in API
Skip to content
Kotensor
Kotensor
/
io.kotensor.core
/
Tokenizer
Tokenizer
interface
Tokenizer
(
source
)
Inheritors
BPETokenizer
SimpleTokenizer
Members
Properties
vocabulary
Size
Link copied to clipboard
abstract
val
vocabularySize
:
Int
Functions
decode
Link copied to clipboard
abstract
fun
decode
(
tokens
:
IntArray
)
:
String
encode
Link copied to clipboard
abstract
fun
encode
(
text
:
String
)
:
IntArray