repoGitHubTrust 82 · PrimaryPublished 10d agoLive · 10d ago
cimeister/tokenizer-intrinsic-evals
TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Implements
paperChallenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource LanguagespaperMinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological AlignmentpaperAsk, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-ImprovementpaperHow Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple MitigationpaperClinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking
Implements (incoming)
Related across the graph
paperAsk, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-ImprovementpaperBlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching SpeechpaperThe Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMspaperChallenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource LanguagespaperClinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI BenchmarkingpaperHow Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple MitigationpaperMinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment