Tokenization
7 items across the graph — tagged with Tokenization.
From the graph · 7
repo
toon-format/toon
→repo🎒 Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware JSON for LLM prompts. Spec, benchmarks, TypeScript SDK.
marcelroed/gigatoken
→repoLanguage model tokenization at GB/s
Apil-Shrestha/token-efficiency-lex
→repoToken Cost Parity: Multilingual LLM Efficiency Analysis 2026
lirantal/tokenu
→repoa unix-like du command line tool to count token usage per files and directories
georg-jung/FastBertTokenizer
→repoFast and memory-efficient library for WordPiece tokenization as it is used by BERT.
cimeister/tokenizer-intrinsic-evals
→repoTokEval: intrinsic quality metrics for tokenizers across natural language, code, and math
Masoudjafaripour/nanochat-VLM
→A minimal, hackable Vision-Language Model built on Karpathy’s nanochat — add image understanding and multimodal chat for under $200 in compute.