Benchmarking Dutch ASR you can actually reproduce.
Most speech-recognition numbers aren't comparable — a 1-point gap often says more about the scorer than the model. So we fixed a pipeline, ran everyone through it, and released the receipts.
The problem with published WER
Word error rates get quoted side by side as if they mean the same thing. They rarely do: each paper picks its own text normaliser and its own slice of a dataset. The same model can read 3.5%, 4.3% or 5.5% on "Common Voice Dutch" depending only on the scorer and the release. Any claim that "model A beats model B" is only meaningful if A and B went through the same pipeline on the same audio.
What we did instead
We didn't try to produce one authoritative number. We fixed a scoring pipeline, ran every system through it, and used each venue's own official tooling — the community NIST-sclite scorer for Common Voice, the HuggingFace Open ASR harness for the multilingual sets. Then we checked whether the ranking held when we changed the scorer. It did.
The control run
The strongest check is the cheapest: we ran the previous best model class through our own pipeline on the identical clips. It reproduced the board's published 4.3% exactly — so our 3.5% is a real accuracy gain over the same class of model, not an artefact of decoding settings.
| System | WER | How |
|---|---|---|
| Our engine | 3.5% | full CV17 NL test, official sclite |
| Previous best on the board | 4.3% | same test set |
| Same model class, our pipeline (control) | 4.3% | reproduces the board number |
The contamination check
Common Voice re-splits on every release, so a clip that release 17 calls "test" can sit in release 25's "train." Because filenames are stable across releases, we checked this exactly rather than statistically: 9 of 11,266 test clips (0.08%) appeared in our training data, and the leak-free subset moves the result by only 0.1pp. The nine clip IDs and the check script are public.
Reproducibility as a feature
Every figure ships with its per-utterance scorer output, the control run, and the contamination check. Anyone can re-derive the numbers in an afternoon — no trust required.