Associations to the word «Tokenize» - Word Associations Network

Noun

Basic
Byte
Identifier
Token
Code
Output
File
Text
User
Memory
Program
Data
Source
Word
Example
Language
Form
Line
Use

Adjective

Lexical

Verb

Store

Wiktionary

TOKENIZE, verb. (transitive) (computing) To reduce to a set of tokens by parsing.

TOKENIZE, verb. (transitive) To treat as a token minority.

Wise words

It is better wither to be silent, or to say things of more value than silence. Sooner throw a pearl at hazard than an idle or useless word; and do not say a little in many words, but a great deal in a few.

Pythagoras