Every field that lands in an airworthiness ledger must be traceable back to the exact bytes it came from: which scanned page, which bounding box, which OCR pass, which transform version. This stage of Automated Log Ingestion & Parsing Workflows records that origin metadata alongside the value itself, so that months later an auditor can point at a single normalized entry and reconstruct precisely how it was produced. Provenance tracking is what turns an opaque parsed record into a defensible one.
Where this stage sits
Provenance capture is not a discrete step so much as a cross-cutting obligation threaded through the entire pipeline. Upstream, the OCR and layout stages emit character boxes, page indices, and confidence scores; the field-extraction stage in Regex & NLP Field Extraction then resolves those raw tokens into typed maintenance fields. Lineage tracking sits astride both, attaching a provenance envelope at the moment each field is born and preserving that envelope untouched as the value flows downstream into normalization, schema validation, and the parts-traceability ledger.
Downstream, the lineage store becomes the substrate for audit replay: given any ledger record, the store answers “where did this come from” without re-running the pipeline. That answer is what regulators expect when they inspect an operator’s electronic recordkeeping system.
Provenance capture across the pipeline
Problem & regulatory driver
Consider a normalized record asserting that component P/N HTP-4471-02 was removed from D-AIMA under ATA 24-31 (AC electrical generation) at 18,204.7 flight hours. Absent provenance, that record is an unsupported assertion; a maintenance data-entry error and a faithful transcription look identical. As the FAA Part-145 Recordkeeping Standards detail, FAA 14 CFR § 145.219 requires a repair station to retain records of the work performed, and to make those records available to the Administrator, in a form that establishes what was actually recorded. The advisory framework in AC 120-78B reinforces this for electronic recordkeeping: an electronic system must preserve data integrity and support reconstruction of how a record was created.
Provenance is the mechanism that discharges that expectation at field granularity. It converts “the pipeline says so” into “here is page 3 of the source scan, the bounding box at pixels (612, 344, 118, 22), an OCR confidence of 0.97, and extractor hgen-v3.2.1 that produced this value” — a chain a human reviewer can independently walk.
Interface contract
The provenance stage does not transform field values. Its contract is purely additive: for each extracted field, it accepts the value plus its raw origin metadata, and it emits an immutable provenance record keyed by a stable field identity.
Inputs: a field_path (dotted, e.g. component.part_number), the extracted value, the SHA-256 hash of the source document, a zero-based page_index, a pixel bbox tuple, an OCR confidence in [0.0, 1.0], and the extractor_version string. Timestamps are ISO 8601 UTC.
Outputs: a ProvenanceRecord whose own identity hash is derived from its contents, so the record is self-verifying. Downstream consumers treat these records as write-once. Nothing in the contract permits mutating a value after its provenance is sealed; a correction is a new field with new provenance, never an in-place edit.
Design decisions
Three decisions matter most. First, provenance is stored out of band from the value, referenced by hash, so that the ledger stays lean while the audit trail stays complete. Second, the source document is fingerprinted once at ingestion with SHA-256, and every field points at that single fingerprint rather than duplicating page bytes. Third, the provenance record’s identity is content-derived, which means any tampering — a changed bbox, a doctored confidence — produces a different hash and is trivially detectable during audit reconciliation.
Production Python implementation
The following module builds and seals a provenance record for a single extracted field. It uses Pydantic v2 with a model_validator to compute the self-verifying identity hash, and structured logging for the audit stream.
from __future__ import annotations
import hashlib
import json
import logging
from datetime import datetime, timezone
from typing import Tuple
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("mro.lineage.provenance")
class ProvenanceRecord(BaseModel):
"""Self-verifying origin metadata for one extracted field.
Retained per FAA 14 CFR § 145.219, which requires a repair station to
retain records of the maintenance it performs and make them available
to the Administrator.
"""
model_config = ConfigDict(frozen=True)
field_path: str = Field(min_length=1)
value: str
source_sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
page_index: int = Field(ge=0)
bbox: Tuple[int, int, int, int]
ocr_confidence: float = Field(ge=0.0, le=1.0)
extractor_version: str = Field(min_length=1)
captured_at: datetime
provenance_id: str = Field(default="")
@field_validator("captured_at", mode="before")
@classmethod
def enforce_utc(cls, v: object) -> datetime:
if isinstance(v, str):
return datetime.fromisoformat(v.replace("Z", "+00:00")).astimezone(timezone.utc)
if isinstance(v, datetime):
return v.astimezone(timezone.utc)
raise TypeError("captured_at must be datetime or ISO 8601 string")
@model_validator(mode="after")
def seal_identity(self) -> "ProvenanceRecord":
payload = json.dumps(
{
"field_path": self.field_path,
"value": self.value,
"source_sha256": self.source_sha256,
"page_index": self.page_index,
"bbox": list(self.bbox),
"ocr_confidence": round(self.ocr_confidence, 4),
"extractor_version": self.extractor_version,
"captured_at": self.captured_at.isoformat(),
},
sort_keys=True,
separators=(",", ":"),
)
digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
if self.provenance_id and self.provenance_id != digest:
raise ValueError("provenance_id does not match sealed content")
object.__setattr__(self, "provenance_id", digest)
return self
def capture(field_path: str, value: str, source_sha256: str, page_index: int,
bbox: Tuple[int, int, int, int], ocr_confidence: float,
extractor_version: str) -> ProvenanceRecord:
record = ProvenanceRecord(
field_path=field_path,
value=value,
source_sha256=source_sha256,
page_index=page_index,
bbox=bbox,
ocr_confidence=ocr_confidence,
extractor_version=extractor_version,
captured_at=datetime.now(timezone.utc),
)
logger.info(
"sealed provenance field=%s prov_id=%s conf=%.2f",
field_path, record.provenance_id[:12], ocr_confidence,
)
return record
if __name__ == "__main__":
prov = capture(
field_path="component.part_number",
value="HTP-4471-02",
source_sha256="a" * 64,
page_index=2,
bbox=(612, 344, 118, 22),
ocr_confidence=0.97,
extractor_version="hgen-v3.2.1",
)
print(prov.model_dump_json(indent=2))Validation & testing
Two invariants are worth asserting in CI. First, determinism: constructing two ProvenanceRecord instances from identical inputs (with a fixed captured_at) must yield identical provenance_id values — the seal is a pure function of content. Second, tamper detection: mutate any sealed field and re-run seal_identity; the mismatch guard must raise. Property-based tests over randomized bounding boxes and confidence values catch off-by-one and rounding regressions before they reach production ledgers.
Failure modes & troubleshooting
The common failure is provenance drift, where a value is edited but its provenance is left pointing at the original bytes. The frozen=True config forecloses accidental in-place mutation, but code that constructs a new value without new provenance defeats it; enforce at review that value and provenance are always minted together. A second failure is fingerprint reuse across re-scans: if a document is re-scanned at a different resolution, its SHA-256 changes, and existing provenance must not be silently re-pointed. Treat re-scans as new source documents. When you next map these obligations onto concrete pipeline duties, the provenance store is the evidence layer every other stage leans on to prove that what the ledger asserts is exactly what the source recorded.