SimpleTokenizer

class SimpleTokenizer(val textBody: String) : Tokenizer(source)

Constructors

Link copied to clipboard
constructor(textBody: String)

Properties

Link copied to clipboard
Link copied to clipboard
open override val vocabularySize: Int

Functions

Link copied to clipboard
open override fun decode(tokens: IntArray): String
Link copied to clipboard
open override fun encode(text: String): IntArray