Benchmark HTR Engines

Choose a handwritten-text-recognition (HTR) or OCR engine with evidence instead of vendor claims: run every candidate over the same labelled samples and compare raw CER, lenient CER, WER, and — the decisive signal for records work — a critical name/date token diff. The harness is deliberately self-contained: scoring implements edit distance directly (no third-party dependency), and a fully mocked self-test validates the math with no API key and no network.

When to Use

• Selecting between vision-LLMs, dedicated HTR services (Transkribus-style REST APIs), and cloud OCR for a handwriting or historical-document task
• Verifying published or vendor accuracy claims on your own material — script, language, ink, and layout rarely match the benchmark corpus behind the claim
• Working with genealogy, archival, or register material where a fluent transcription containing one wrong name or year is the real failure mode
• Re-ranking engines after new models ship, against a frozen sample set
• Any comparison where transcription conventions (line-break hyphenation, case, ß/ss) would otherwise pollute the accuracy signal

Inputs

• Required: 10–20 labelled samples — image + ground-truth text pairs spanning easy to hard material (or an open dataset to pull them from, Step 2)
• Required: Two or more candidate engines (model ids or service endpoints)
• Optional: API keys per engine family (e.g. LLM_API_KEY, HTR service credentials) — the scoring self-test needs none
• Optional: Critical-token rules (default: any token containing a digit, month names including archaic forms, Capitalised words)
• Optional: Lenient-CER folds (default: line-break hyphenation, whitespace, case, ß→ss; adjust per language and script)

Procedure

Step 1: Lay Out the Sample Corpus

Use a flat directory of image + ground-truth pairs sharing a stem. This layout is the whole data contract — every engine and the scorer read only this.

samples/
  register_p034.jpg      # the page or line image (.jpg .jpeg .png .tif .tiff .webp)
  register_p034.gt.txt   # its ground-truth transcription (UTF-8)

Guidelines:

• 10–20 line crops or a few full pages are enough to separate engines; spread them easy → hard (clean vs. faded ink, marginalia, mixed hands).
• Line-level crops score more cleanly on a first pass; full pages also work — vision-LLMs read them whole, HTR services run their own line detection.
• If the material is private (church, court, medical records), gitignore samples/ immediately, before the first commit.

Expected: Every image in samples/ has a same-stem .gt.txt; private material is covered by a .gitignore rule from the start.

On failure: Orphan images (no .gt.txt) are skipped by the runner — either transcribe them or remove them. If private samples were already committed, rewriting history is required; prevention (gitignore first) is far cheaper.

Step 2: Source Ground Truth

Two routes; both end in the Step 1 layout.

Route A — open dataset. Pull a handful of labelled rows from a public corpus matching your script, e.g. Hugging Face dh-unibe/image-text_kurrent-xix (19th-century German Kurrent, MIT licence, DOI 10.57967/hf/8590). Each row is an image plus PAGE XML; parse the plain text out of the XML. Stream — never download the full corpus:

from pathlib import Path
from datasets import load_dataset  # pip install datasets

dataset = load_dataset("dh-unibe/image-text_kurrent-xix",
                       split="train", streaming=True)
written = 0
for row in dataset:
    if written >= 12:
        break
    ground_truth = plain_text_from_page_xml(row["xml_content"]).strip()
    if not (10 <= len(ground_truth) <= 600):
        continue  # skip near-empty rows and multi-column monsters
    Path(f"samples/{written:02d}_kurrent.png").write_bytes(row["image"]["bytes"])
    Path(f"samples/{written:02d}_kurrent.gt.txt").write_text(
        ground_truth, encoding="utf-8")
    written += 1

The PAGE-XML parser must prefer each TextLine's line-level TextEquiv/Unicode and only fall back to joining <Word> children — see references/EXAMPLES.md for a correct implementation.

Route B — hand-transcribe. Transcribe a few target pages yourself. Transcribe the original (old) spelling faithfully — do not modernise — so CER measures the engine, not your edits. This is a labelling task; see label-training-data for conventions.

Expected: Sample pairs on disk; ground truth preserves original spelling and line breaks. Dataset attribution recorded if results will be published.

On failure: If streaming fails, check the datasets version and network; reduce --scan-limit-style row scanning rather than downloading the corpus. If parsed ground truth contains only the first word of each line, your PAGE parser hit the word-segmentation trap (see Common Pitfalls). A near-genre dataset (line crops, related script) is a valid accuracy signal but not a genre-exact one — note the caveat in the report.

Step 3: Define the Engine Adapter Interface

Every engine hides behind one method, so the runner and scorer never know which family they are talking to:

from abc import ABC, abstractmethod

class TranscriptionEngine(ABC):
    """Common interface: given an image file, return the transcribed text."""

    name: str = "engine"  # e.g. "llm:vendor/model-pro", "htr:51170"

    @abstractmethod
    def transcribe(self, image_path: str) -> str:
        """Return the plain-text transcription of the image at image_path."""
        raise NotImplementedError

Three adapter families cover the practical field:

Family	Transport	Notes
Vision-LLM	Any OpenAI-compatible chat endpoint (OpenRouter, vendor APIs)	One code path covers many models; swap the model id
Dedicated HTR	REST: authenticate → submit → poll → fetch PAGE XML	Asynchronous jobs; result needs XML → text extraction
Cloud OCR	Vendor SDK or REST call	Usually synchronous; concatenate returned text blocks

Expected: A base.py with the ABC; each candidate engine is a subclass with a distinct name used in reports and prediction filenames.

On failure: If an engine cannot fit transcribe(image_path) -> str (e.g. it returns only word boxes), do the flattening inside the adapter — never let format differences leak into the scorer.

Step 4: Implement Adapters with Retry-on-Empty

For the vision-LLM family, send the image as a base64 data URI with a hallucination-suppressing prompt (transcribe only what is legible, mark unreadable spans [...], never invent names or dates), temperature=0, and a generous max_tokens. Then guard the known transient failure:

# Reasoning models intermittently return EMPTY content with
# finish_reason="stop" — reasoning tokens consumed, no text emitted.
# Without a retry, one transient empty scores as 100% CER and poisons
# that engine's mean.
last_finish_reason = None
for _attempt in range(self.max_retries):  # e.g. 3
    response = self.client.chat.completions.create(
        model=self.model,
        temperature=0,                       # minimise fabrication
        max_tokens=self.max_output_tokens,   # headroom: reasoning + full page
        messages=messages,
    )
    choice = response.choices[0]
    text = (choice.message.content or "").strip()
    if text:
        return text
    last_finish_reason = choice.finish_reason
raise RuntimeError(
    f"empty content after {self.max_retries} attempts "
    f"(last finish_reason={last_finish_reason})")

For dedicated HTR REST services, the adapter wraps an asynchronous job: authenticate, POST the image (raw base64), poll the process id until FINISHED/FAILED/timeout, fetch the PAGE XML result, and extract plain text. Full templates for both families are in references/EXAMPLES.md.

Expected: Each adapter returns non-empty text for a legible test image, or raises with a diagnostic (not an empty string). Config errors (missing keys) raise at construction time with the variable names spelled out.

On failure: Empty after all retries — raise max_tokens (reasoning tokens compete with output tokens) before blaming the model. HTTP 401 mid-poll — re-authenticate once and retry rather than abandoning a submitted (paid) job. 429/5xx — retry with capped exponential backoff.

Step 5: Implement Zero-Dependency Scoring

Implement Levenshtein distance directly (two-row dynamic programming, ~15 lines) so scoring runs anywhere with no install. Derive four metrics:

• Raw CER — character edit distance / reference length. Honest but inflated by convention differences.
• Lenient CER — CER after folding transcription conventions, so it reflects recognition, not orthography:

def normalize_for_lenient_cer(text: str) -> str:
    normalized = unicodedata.normalize("NFC", text)
    normalized = LINEBREAK_HYPHEN.sub("", normalized)  # join hyphenated breaks
    normalized = WHITESPACE.sub(" ", normalized)       # layout != recognition
    normalized = normalized.lower()                    # case != recognition
    normalized = normalized.replace("ß", "ss")         # script-specific folds
    return normalized.strip()

Keep it deliberately conservative — over-normalising hides real errors. Report lenient CER as a secondary metric next to raw CER, never instead.

• WER — the same edit distance over whitespace-split words.
• Critical name/date token diff — a word-level diff (difflib.SequenceMatcher) where each edit is flagged if it touches a critical token:

def is_critical_token(token: str) -> bool:
    stripped = token.strip(".,;:()[]{}\"'-")
    if not stripped:
        return False
    if any(ch.isdigit() for ch in stripped):   # dates, years, ages
        return True
    if stripped.lower() in MONTH_NAMES:        # incl. archaic forms like "7ber"
        return True
    # Capitalised word -> likely proper noun (over-flags German common
    # nouns by design: safer to surface too much for human review)
    return stripped[0].isalpha() and stripped[0].isupper() \
        and stripped[1:] == stripped[1:].lower()

The critical-token count is the records-relevant signal: a confident-wrong surname or year is invisible inside a good-looking aggregate CER.

Expected: A scoring.py importing only the standard library, exposing character_error_rate, lenient_character_error_rate, word_error_rate, and critical_token_diff.

On failure: If lenient CER ever exceeds raw CER on the same pair, the normalizer is corrupting text (check regex ordering: fold hyphenated line breaks before collapsing whitespace). Empty-reference edge cases must return 0.0 for empty-vs-empty and 1.0 for empty-vs-nonempty, not divide by zero.

Step 6: Validate Scoring with a No-Key Self-Test

Before spending any API budget, assert the math on synthetic tokens (all names/dates below are invented):

assert character_error_rate("Haus", "Hans") == 0.25       # 1 edit / 4 chars
assert word_error_rate("geboren den 12 Merz 1834",
                       "geboren den 12 Merz 1835") == 0.2  # 1 word of 5
# Convention folds to zero...
assert lenient_character_error_rate("Ver¬\nzeichnis", "Ver-\nzeichnis") == 0.0
assert lenient_character_error_rate("Grüßner", "Grüssner") == 0.0
# ...but a genuine name misread survives lenient folding
assert lenient_character_error_rate("Beispielmann", "Beispelmann") > 0.0
edits = critical_token_diff(
    "getauft den 12 Merz 1834 Georg Beispielmann",
    "getauft den 12 Merz 1835 Georg Beispelmann")
assert sum(1 for e in edits if e.touches_critical) == 2   # year AND surname
print("scoring self-test: PASS")

Wire it into the runner as --selftest so it is always one flag away. If an adapter has network seams (the REST family), mock them in the self-test too — the full template in references/EXAMPLES.md runs without requests even installed.

Expected: python3 run_benchmark.py --selftest prints PASS with no API key, no network, and no third-party package.

On failure: A failing assertion is a scoring bug — fix it before running engines, or every downstream comparison is untrustworthy. Never weaken an assertion to make it pass; re-derive the expected value by hand first.

Step 7: Run the Panel and Report

Run all candidate engines over all samples in one invocation. Per engine × sample: score, print the per-sample line plus only the critical edits, and save the raw prediction (re-scoring later must not require re-paying for API calls). Keep one engine's failure from killing the panel — catch per-sample errors, count them, continue.

  07_kurrent.png   CER=6.2%  lenient-CER=3.1%  WER=11.0%  name/date-errors=1
      !! NAME/DATE [replace] gt: '1834' -> pred: '1835'
      (+ 2 minor non-critical edit(s))

===== COMPARISON (sorted by lenient CER, lower is better) =====
  engine                          CER  lenient  name/date     ok
  llm:vendor/model-pro           7.9%     4.1%          3  12/12
  htr:rest-model-51170          11.2%     6.0%          5  12/12
  llm:vendor/model-flash        13.5%     9.8%         14  11/12

Persist everything: the full console report (e.g. reports/last_run.txt) and one reports/<sample>.<engine>.pred.txt per prediction.

Expected: A comparison table sorted by lenient CER; per-sample critical diffs; raw predictions on disk for every successful engine × sample pair.

On failure: An engine erroring on every sample is a config problem (key, model id, endpoint) — its constructor should have said so; check the first error message, not the aggregate. If two engines tie on lenient CER, rank by critical-token count next.

Step 8: Interpret and Decide

Read the table in this priority order:

1. Critical name/date errors — the failure that corrupts records; a fluent engine with more name errors loses to a rougher one with fewer.
2. Lenient CER — recognition quality with convention noise removed.
3. Raw CER / WER — tie-breakers and sanity checks (a large raw-vs-lenient gap means convention differences, not recognition differences).

Then weigh the non-accuracy dimensions — cost per page, privacy (does the image leave your infrastructure?), throughput, and licensing — covered in the companion guide choosing-an-htr-ocr-engine.

Expected: A documented engine choice citing the table, the critical-token counts, and the non-accuracy constraints that applied.

On failure: If no engine is acceptable (critical errors on most samples), re-scope: better images (higher DPI, line crops), a fine-tuned model for the script, or human transcription with engine assist rather than engine-first.

Validation

• Every image in samples/ has a same-stem .gt.txt (no orphans scored)
• run_benchmark.py --selftest passes with no API key, network, or third-party install
• Every LLM adapter retries on empty content before raising
• Report shows raw CER, lenient CER, WER, and critical name/date count per engine
• Comparison table is sorted by lenient CER and states samples-ok per engine
• Raw predictions saved to disk for every successful engine × sample
• Private sample text appears nowhere outside samples/ (gitignored); docs and tests use invented tokens only
• Dataset licence and attribution recorded if results will be published

Common Pitfalls

• Trusting aggregate CER: A 4% CER engine that swaps one surname per page is worse for records work than an 8% engine that gets names right. Always read the critical-token diff before the means.
• Over-normalising lenient CER: Folding too much (punctuation, diacritics, word order) hides real recognition errors. Fold only documented transcription conventions, and always report raw CER alongside.
• Empty LLM responses scored as 100% CER: Reasoning models intermittently return empty content with finish_reason=stop. Without retry-on-empty, one transient blip destroys that engine's mean and your ranking.
• PAGE-XML first-word trap: Word-segmented PAGE XML serializes <Word> children before the line-level <TextEquiv>; a naive pre-order iter() walk returns only the first word of every line — ground truth silently truncates and every engine looks terrible. Prefer the line-level TextEquiv.
• Modernised ground truth: "Correcting" old spelling in the ground truth charges every engine for errors it did not make. Transcribe faithfully; handle convention differences in the lenient normalizer instead.
• Committing private samples: Church, court, or family records must be gitignored before the first commit, and never pasted into prompts, tests, or documentation — use invented tokens (e.g. Beispielmann, Georg *1834).
• One-sample verdicts: A single easy sample cannot separate engines. Use 10–20 samples spanning difficulty, and report per-sample rows so one outlier is visible instead of buried in a mean.

Related Skills

• label-training-data - hand-transcribing ground truth is a labelling task; reuse its conventions and QA loop
• run-ab-test-models - statistical comparison when two shortlisted engines are close and the sample budget can grow
• manage-token-budget - controlling vision-LLM spend when the panel or sample set grows
• choosing-an-htr-ocr-engine - companion guide: the engine landscape, cost, privacy, and decision criteria this benchmark feeds

See references/EXAMPLES.md for the complete generalized harness template (scoring, adapters, runner, self-test, dataset fetcher).

<!-- Keep under 500 lines. Extract large examples to references/EXAMPLES.md if needed. -->