Prüfungsergebnisse · Formblatt AL-2 Almanbench v0.1 · Stand 2026-07-17

Almanbench

Almanbench measures language and grammatical reasoning capabilities, specifically in German. Its 1,028 public items test whether a language model can apply the complete Alman specification to real German prose, from canonical literature to contemporary text.

Results

Nr.	Model	Score ↑
1	GPT-5.6 Sol	90.2%
2	Claude Fable 5	90.0%
3	DeepSeek V4 Flash	86.5%
4	GPT-5.6 Terra	84.2%
5	GPT-5.6 Luna	83.8%
6	Kimi K2.7 Code	83.2%
7	DeepSeek V4 Pro	82.7%
8	Claude Sonnet 5	76.3%
9	MiniMax M3	76.1%
10	Nemotron 3 Ultra	71.4%
11	Qwen3.6 35B A3B	71.1%
12	Gemma 4 31B IT	63.5%
13	MiMo V2.5 Pro	61.4%
14	Ternary Bonsai 27B	47.3%
		40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%

Score by tier

Model	naturalistic	targeted	guards	curated
GPT-5.6 Sol	87.3%	98.1%	87.5%	94.4%
Claude Fable 5	87.5%	96.8%	87.5%	94.4%
DeepSeek V4 Flash	81.7%	96.3%	88.3%	92.4%
GPT-5.6 Terra	78.2%	94.9%	88.3%	93.3%
GPT-5.6 Luna	77.5%	96.8%	85.8%	92.1%
Kimi K2.7 Code	77.0%	93.5%	87.5%	93.5%
DeepSeek V4 Pro	75.5%	96.8%	87.5%	90.2%
Claude Sonnet 5	65.5%	94.4%	89.2%	87.6%
MiniMax M3	66.2%	94.9%	85.0%	84.8%
Nemotron 3 Ultra	61.2%	90.3%	77.5%	85.9%
Qwen3.6 35B A3B	59.2%	92.1%	84.2%	82.6%
Gemma 4 31B IT	48.7%	92.1%	81.7%	69.6%
MiMo V2.5 Pro	46.3%	86.6%	80.8%	75.0%
Ternary Bonsai 27B	33.7%	74.5%	65.8%	47.8%

Composition

The public set contains 1,028 items in four tiers. A private held-out set of about 200 further items, reviewed to the same standard, is never published and prices training contamination over time. Every published row carries a canary GUID so the data can be filtered from training corpora.

Tier	Items	Purpose
naturalistic	600	Real prose with interacting rules. Half canonical literature (1500 to 1955), half contemporary German from Wikipedia, Tatoeba, and hand-authored sentences.
targeted	216	Hand-authored items that lift rare rules to at least 25 observations each.
guards	120	Overcorrection traps in eight families. Forms that Alman keeps, which a surface-form stripper would wrongly change.
curated	92	Hand-translated demonstrative core. Every specification rule is the designated target of at least one item.

Example item

A naturalistic item from German Wikipedia. The genitive after Familie may keep der or take the von periphrasis, so the reference lists both valid renderings.

almanbench/modern-wikipedia/pub-023

Standard German: Der Braunbär gehört zu den Säugetieren aus der Familie der Bären.
Valid renderings: Die Braunbär gehört zu die Säugetiere aus die Familie der Bären. (canonical); Die Braunbär gehört zu die Säugetiere aus die Familie von die Bären.

Running the benchmark

The dataset is published on Hugging Face. The harness ships with the alman repository and runs against any model supported by Inspect AI.

from datasets import load_dataset

ds = load_dataset("osolmaz/almanbench", split="test")

uv run inspect eval alman/bench/task.py --model openai/gpt-5-codex

Citation

@misc{almanbench2026,
  title        = {Almanbench: A Standard German to Alman Translation Benchmark},
  author       = {Solmaz, Onur},
  year         = {2026},
  howpublished = {\url{https://alman.ai/almanbench/}}
}