Kotensor
Toggle table of contents
1.2.0
jvm
Target filter
jvm
Switch theme
Search in API
Skip to content
Kotensor
Kotensor
/
io.kotensor.core
/
SimpleTokenizer
Simple
Tokenizer
class
SimpleTokenizer
(
val
textBody
:
String
)
:
Tokenizer
(
source
)
Members
Constructors
Simple
Tokenizer
Link copied to clipboard
constructor
(
textBody
:
String
)
Properties
text
Body
Link copied to clipboard
val
textBody
:
String
vocabulary
Size
Link copied to clipboard
open
override
val
vocabularySize
:
Int
Functions
decode
Link copied to clipboard
open
override
fun
decode
(
tokens
:
IntArray
)
:
String
encode
Link copied to clipboard
open
override
fun
encode
(
text
:
String
)
:
IntArray