Handwritten maintenance entries are the hardest input any recognition engine will face, and a single global cutoff wastes their signal. This page shows how to calibrate a distinct confidence threshold per field against a labeled sample of scanned logbook pages, so that clean printed serials auto-accept while shaky handwritten defect narratives route to review. It extends the routing logic introduced in OCR Confidence Scoring & Fallbacks, turning an arbitrary 0.90 guess into a defensible, data-derived operating point.
Prerequisites & inputs
Calibration only works when you have ground truth. Before tuning anything, assemble a labeled sample of at least 300–500 real entries drawn from the registrations and OEM formats your line actually handles — for a mixed narrowbody fleet that might mean pages from N512DL, N88AW, and a handful of leased tails. Each sample row needs three things: the raw OCR token, the per-token confidence the engine emitted, and a human-verified correct value. The quality of that raw confidence depends entirely on the engine underneath; if your recognizer collapses every glyph to 0.99, no threshold will separate good from bad, which is why picking Best OCR Engines for Faded Maintenance Logs is a precondition, not an afterthought.
Group your labels by field, because the recognition difficulty is wildly uneven. A stamped part number like AS3701-14 and an ATA reference such as 24-31 (DC electrical generation) are printed or gridded and recognize cleanly; a technician’s cursive description of a bonding-strap replacement does not. Treat part_number, serial_number, ata_chapter, flight_hours, and description as separate calibration targets with separate thresholds.
Calibration loop
Step-by-step implementation
1. Frame the two error types. At any threshold t, a field value auto-accepts when its confidence c >= t. Two things can go wrong. An escaped error is an accepted value that disagrees with ground truth — the expensive failure, because a wrong serial_number propagates silently into the parts ledger. A needless review is a correct value pushed below t and queued for a human — cheap, but it erodes the throughput that justified automation. Tuning is the deliberate trade between these.
2. Sweep the threshold across each field. For every candidate t in a fine grid (e.g. 0.50 to 0.999), compute the auto-accept rate, the escaped-error rate, and the review load. This produces a per-field cost curve rather than a single number.
3. Attach a cost model. Not all escaped errors cost the same. A mis-read flight_hours on a life-limited component can invalidate a compliance calculation, whereas a garbled comment word rarely does. Encode a per-field penalty so the optimizer respects the regulatory weight of each field.
4. Pick the operating point. Choose the threshold that minimizes total expected cost subject to a hard ceiling on escaped-error rate for safety-critical fields. The following module runs the whole loop.
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Sequence
logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("mro.threshold_calibration")
@dataclass(frozen=True)
class LabeledToken:
"""One OCR token from the labeled sample, with ground truth attached."""
field_name: str
ocr_value: str
truth_value: str
confidence: float # engine-reported, range 0.0–1.0
@property
def is_correct(self) -> bool:
return self.ocr_value.strip() == self.truth_value.strip()
@dataclass
class ThresholdResult:
field_name: str
threshold: float
auto_accept_rate: float
escaped_error_rate: float
review_rate: float
expected_cost: float
@dataclass
class FieldCostModel:
# Cost of one escaped (accepted-but-wrong) value, weighted by regulatory impact.
escaped_error_cost: float
# Cost of one needless human review (labor + latency).
review_cost: float = 1.0
# Hard ceiling: safety-critical fields must not exceed this escaped-error rate.
max_escaped_error_rate: float = 0.005
def sweep_field(
tokens: Sequence[LabeledToken],
cost: FieldCostModel,
grid: Sequence[float] = tuple(round(0.50 + 0.005 * i, 3) for i in range(101)),
) -> ThresholdResult:
"""Sweep thresholds for one field and return the minimum-cost operating point.
Auto-accepted values feed FAA-recordable maintenance entries; under
14 CFR 43.9(a)(1) each record must carry "A description (or reference to
data acceptable to the Administrator) of the work performed", so an
escaped error in a description or serial is a recordkeeping defect, not
a cosmetic one.
"""
if not tokens:
raise ValueError("cannot calibrate an empty sample")
n = len(tokens)
best: ThresholdResult | None = None
for t in grid:
accepted = [tok for tok in tokens if tok.confidence >= t]
escaped = [tok for tok in accepted if not tok.is_correct]
reviewed = n - len(accepted)
escaped_rate = len(escaped) / n
expected_cost = (
len(escaped) * cost.escaped_error_cost + reviewed * cost.review_cost
)
result = ThresholdResult(
field_name=tokens[0].field_name,
threshold=t,
auto_accept_rate=len(accepted) / n,
escaped_error_rate=escaped_rate,
review_rate=reviewed / n,
expected_cost=expected_cost,
)
# Enforce the hard safety ceiling before considering cost.
if escaped_rate > cost.max_escaped_error_rate:
continue
if best is None or result.expected_cost < best.expected_cost:
best = result
if best is None:
# No threshold met the ceiling: fall back to the strictest grid point.
logger.warning(
"field %s cannot meet escaped-error ceiling %.4f; forcing t=%.3f",
tokens[0].field_name, cost.max_escaped_error_rate, grid[-1],
)
best = sweep_field(tokens, FieldCostModel(cost.escaped_error_cost, cost.review_cost, 1.0), grid)
logger.info(
"field=%s chosen_t=%.3f accept=%.1f%% escaped=%.3f%% review=%.1f%%",
best.field_name, best.threshold, best.auto_accept_rate * 100,
best.escaped_error_rate * 100, best.review_rate * 100,
)
return bestWorked example
Suppose the labeled sample for tail N512DL yields 400 serial_number tokens. Printed serials from the OEM plate recognize at 0.97–0.99; handwritten transcriptions of the same serials sit around 0.70–0.88 and carry most of the errors. Running sweep_field with a FieldCostModel(escaped_error_cost=250.0, review_cost=1.0, max_escaped_error_rate=0.005) walks the grid and reports:
field=serial_number chosen_t=0.940 accept=71.5% escaped=0.250% review=28.5%
The optimizer settled on 0.940, not the 0.92 a global default would have imposed. At 0.92 the escaped-error rate for this field was 0.75% — above the 0.5% ceiling — because a band of handwritten 8/6 confusions sat between 0.92 and 0.94. Lifting the field threshold by two hundredths dropped three escaped serials into the review queue at the cost of about forty extra human touches per thousand pages, a trade the cost model prefers because a wrong serial in the ledger is a traceability break. Re-running the same sweep for ata_chapter — where OCR reads the gridded 24-31 box almost perfectly — returned chosen_t=0.780, so that field auto-accepts far more aggressively. Every accepted value still flows through structural checks; confidence tuning decides what reaches Validating Parsed Data Against AMM Standards, it never replaces that gate.
Edge cases & compliance gotchas
Mixed OEM handwriting styles. A threshold calibrated on European 1 (with a serif) mis-scores a US flat-top 1. If your line mixes formats, either stratify the sample and calibrate per source, or ensure the sample proportion matches production; a threshold fit on 90% printed pages will under-review the 10% handwritten backlog it never saw.
Confidence drift after a scanner swap. Recognition confidence is not portable across hardware. When a facility replaces a sheet-fed scanner or bumps DPI, re-run calibration — a threshold tuned on 300 DPI grayscale can silently raise the escaped-error rate on a new 200 DPI bitonal feed. Log the scanner model and firmware alongside each calibration so drift is attributable.
Unicode and diacritics in names. Certifying-staff fields carrying names like Muñoz or Zoë produce low confidence on the accented glyph even when the rest is perfect. Do not let one diacritic force an otherwise-correct signature block into review; score name fields on normalized forms and treat the accent as a low-weight token.
Over-fitting to a stale sample. A cutoff frozen two years ago describes a fleet that no longer exists. Because FAA and EASA recordkeeping obligations attach to every current entry, treat calibration as a recurring control: re-label a fresh sample quarterly, re-sweep, and version the resulting thresholds next to the records they gate. A confidence threshold is only as trustworthy as the last sample that proved it, and on handwritten logs that proof expires faster than anywhere else in the pipeline.