From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector
Public institutions face a persistent challenge in selecting LLMs suited to their specific context. Existing benchmarks, however, are of limited use as they primarily reflect English-language and US-centric settings, and often only evaluate task performance. In this paper, we present first results of MÖVE, a holistic evaluation framework for the German public sector, examining three rarely considered governance dimensions: energy consumption, provider transparency, and knowledge of German-party positions. Our results reveal significant trade-offs, with no single model excelling across all dime
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 47%Evaluate a model properly →
- FuzzySimilar title/name (fuzzy) · 84%mudler/LocalAI →
“Fuzzy title match (0.92): “From Global Benchmarks to Local Evaluations: Benchmarking LL” ≈ “mudler/LocalAI””
- LinkedLinked via arxiv author · 85%Camilla Dalerci →
“From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector”
- LinkedLinked via arxiv author · 85%Thilo Michael →
“From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector”
- LinkedLinked via arxiv author · 85%Robin Schaefer →
“From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector”
- LinkedLinked via arxiv author · 85%Daniel Weinland →
“From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector”