Alman Institut für Sprachvereinfachung
Prüfungsergebnisse · Formblatt AL-2 AlmanBench v0.1 · Stand 2026-08-05

AlmanBench

AlmanBench measures language and grammatical reasoning capabilities, specifically in German. Its 1,029 public items test whether a language model can apply the complete Alman specification to real German prose, from canonical literature to contemporary text.

AlmanBench also enjoys a rare form of benchmark integrity. No lab is going to burn a training run on gaming a German simplification dataset, so the scores measure what the models can actually do. Obscurity is our contamination policy.

Results

Nr. Model Score ↑
1 GPT-5.5 xhigh
95.4%
2 GPT-5.6 Sol max
95.1%
3 Claude Opus 5 max
94.9%
4 Claude Fable 5 high*
94.0%
5 Claude Fable 5 max*
93.9%
6 GPT-5.6 Sol xhigh
93.5%
7 DeepSeek V4 Flash 0731 max
93.0%
8 Kimi K3 max
91.0%
9 GLM 5.2
90.8%
10 DeepSeek V4 Flash 0422
89.8%
11 Inkling max
89.4%
12 GPT-5.6 Luna xhigh
87.8%
13 GPT-5.6 Terra xhigh
87.7%
14 Kimi K2.7 Code
87.0%
15 DeepSeek V4 Pro
85.3%
16 Claude Sonnet 5 xhigh
83.4%
17 Qwen3.6 27B
81.8%
18 MiniMax M3
79.7%
19 Gemma 4 26B A4B IT
76.7%
20 Claude Opus 4.8 max
74.8%
21 Nemotron 3 Ultra
74.7%
22 Qwen3.6 35B A3B
74.4%
23 Step 3.5 Flash
69.6%
24 GPT-OSS 120B high
68.9%
25 Gemma 4 31B IT
64.8%
26 MiMo V2.5 Pro
64.2%
27 Ternary Bonsai 27B
48.2%
28 Laguna S 2.1
40.4%
29 LongCat Flash Lite Sparse
24.5%
20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Score by tier

Model naturalistictargetedguardscurated
GPT-5.5 xhigh 95.2% 100.0% 87.5% 96.8%
GPT-5.6 Sol max 94.7% 100.0% 87.5% 96.8%
Claude Opus 5 max 95.3% 95.8% 88.3% 98.9%
Claude Fable 5 high* 93.5% 98.1% 87.5% 95.7%
Claude Fable 5 max* 93.5% 96.3% 89.2% 96.8%
GPT-5.6 Sol xhigh 92.2% 99.5% 87.5% 95.7%
DeepSeek V4 Flash 0731 max 92.0% 96.8% 88.3% 96.8%
Kimi K3 max 90.0% 93.1% 88.3% 95.7%
GLM 5.2 89.0% 96.3% 86.7% 94.6%
DeepSeek V4 Flash 0422 86.7% 97.2% 88.3% 94.6%
Inkling max 85.7% 100.0% 85.8% 93.5%
GPT-5.6 Luna xhigh 83.5% 98.1% 85.8% 93.5%
GPT-5.6 Terra xhigh 83.3% 96.3% 88.3% 94.6%
Kimi K2.7 Code 83.0% 94.9% 87.5% 93.5%
DeepSeek V4 Pro 79.7% 97.7% 87.5% 90.3%
Claude Sonnet 5 xhigh 77.2% 95.4% 89.2% 88.2%
Qwen3.6 27B 74.5% 97.2% 86.7% 87.1%
MiniMax M3 71.8% 95.4% 85.0% 87.1%
Gemma 4 26B A4B IT 70.7% 93.5% 77.5% 75.3%
Claude Opus 4.8 max 65.8% 89.8% 84.2% 86.0%
Nemotron 3 Ultra 66.2% 92.1% 77.5% 86.0%
Qwen3.6 35B A3B 64.3% 92.6% 84.2% 84.9%
Step 3.5 Flash 58.3% 88.4% 79.2% 86.0%
GPT-OSS 120B high 60.5% 85.2% 75.8% 76.3%
Gemma 4 31B IT 50.3% 92.6% 81.7% 72.0%
MiMo V2.5 Pro 50.5% 88.0% 80.8% 76.3%
Ternary Bonsai 27B 34.8% 75.5% 65.8% 48.4%
Laguna S 2.1 26.3% 61.6% 65.8% 49.5%
LongCat Flash Lite Sparse 11.8% 44.0% 46.7% 32.3%

Composition

The public set contains 1,029 items in four tiers. A private held-out set of about 200 further items, reviewed to the same standard, is never published and prices training contamination over time. Every published row carries a canary GUID so the data can be filtered from training corpora.

Tier Items Purpose
naturalistic 600 Real prose with interacting rules. Half canonical literature (1500 to 1955), half contemporary German from Wikipedia, Tatoeba, and hand-authored sentences.
targeted 216 Hand-authored items that lift rare rules to at least 25 observations each.
guards 120 Overcorrection traps in eight families. Forms that Alman keeps, which a surface-form stripper would wrongly change.
curated 93 Hand-translated demonstrative core. Every specification rule is the designated target of at least one item.

Example item

A naturalistic item from German Wikipedia. The genitive after Familie may keep der or take the von periphrasis, so the reference lists both valid renderings.

almanbench/modern-wikipedia/pub-023
Standard German
Der Braunbär gehört zu den Säugetieren aus der Familie der Bären.
Valid renderings
Die Braunbär gehört zu die Säugetiere aus die Familie der Bären. (canonical)
Die Braunbär gehört zu die Säugetiere aus die Familie von die Bären.

Running the benchmark

The dataset is published on Hugging Face. The harness ships with the alman repository and runs against any model supported by Inspect AI.

from datasets import load_dataset

ds = load_dataset("osolmaz/almanbench", split="test")
uv run inspect eval alman/bench/task.py --model openai/gpt-5-codex

Citation

@misc{almanbench2026,
  title        = {AlmanBench: A Standard German to Alman Translation Benchmark},
  author       = {Solmaz, Onur},
  year         = {2026},
  howpublished = {\url{https://alman.ai/almanbench/}}
}