Tokenizer

Properties

Link copied to clipboard
abstract val vocabularySize: Int

Functions

Link copied to clipboard
abstract fun decode(tokens: IntArray): String
Link copied to clipboard
abstract fun encode(text: String): IntArray