Architecture and flow

The path data takes

intake ──► custody ──► runner: pull ──► runner: scrub ──► gate ──► release ──► runner: deliver ──► buyer
 owner      control     raw/ in the      raw/ → clean/     inline    control      watermark, encrypt
 connects   plane,      deal's vault     one source per    per file  plane:       to the buyer's key,
 or         deal key    prefix           job               + re-     output       Merkle root
 uploads    wrapped                                        check     sample
            under KEK                                              review

1. Intake. The owner connects a source (database, bucket, SaaS account, mailbox, repository) with a credential they create in their own account, or uploads files. The control plane seals the credential and creates the deal's data key, wrapped under that product's key-encryption key (KEK). Neither the package nor the runner ever sees the KEK.
2. Custody. One control plane per product owns the deal, its vault prefix, its keys and every job. It hands a runner one job at a time through a versioned claim and receives counts, hashes and reports back in the finish call. The claim and finish JSON is the only interface between runner and control plane.
3. Runner. One machine runs one job and is destroyed. Job types in the reference runner: quantify (sizes and counts, no content leaves), pull (source to raw/), scrub (raw/ to clean/), manifest, deliver (and a buyer-pull variant), reconcile and trace. Boot order: lock outbound traffic to the control plane only, claim, re-lock to the hosts this job type needs, pin them, drop DNS and IPv6, drop to an unprivileged user with no-new-privileges, then work. Details in threat-model.md.
4. Scrub. The engine (this package) does detection and rewriting; the runner does IO, parsing of hostile formats in a sandbox, media, models and the report. One scrub job covers one source.
5. Gate. Every byte the scrub writes passes the inline residual gate before it is uploaded, then the independent re-check, then (when the deal names owner terms) the owner-identity check. A hit holds that file; nothing is ever "fixed and shipped" without passing again. Held files are listed by path hash and reason.
6. Release. The control plane's release gate: the desk reviews a random sample of the delivered output and the version releases only when the profile's thresholds hold (measurement.md).
7. Delivery. A deliver job watermarks per buyer and version, encrypts to the buyer's recipient key, and records a Merkle root over every delivered file. The signed deid_report travels with it.

Engine and runner

A runner that vendors the package gets the engine column on every sync. The runner column is each runner's own code: which of those features another runner wires, and with which switches, is that product's documentation. Everything these pages call "the reference runner" is the reference implementation at the same commit as this package.

Scrub order (reference runner)

jobs/scrub.py run():

1. Verify every baked model against its pinned hash (jobs/models.py); under a profile with no fallback, a mismatch or a missing baked model fails the job.
2. Activate the profile, load the ensemble members, run the canary self-test on every scrub path (measurement.md). Under random stand-ins the canaries run on a throwaway graph.
3. Open the deal's identity map when the claim carries one, then set the k-anonymity tracker, the stand-in salt, the date rule and the business-id map.
4. Per file, in order: the owner's never-share path rules (before download) → line formats stream from the vault through the scrub and the gate writer with no disk; everything else downloads to the work directory under a hashed name → second credential scanner (counts providers so the owner knows what to rotate; inside the sandbox it has no network, so nothing is verified against a provider) → malware scan → macro check → kind → healthcare format check → the kind's path (below) → upload through the gate.
5. After the last file: normalized tables (when the claim asks), the workflow score (when the claim asks), the linkage table and sealed map, the scrub report, the deid_report, and the manifests (scrub-report.json, forensics.json, held.jsonl.gz).

Modalities

Paths that ship (folders, file stems, archive members) are scrubbed like text under every profile (paths.py), extensions kept. File times follow the profile's date rule.

Unpacking and normalization

• Decoded view (textnorm.view). Every detector also reads a normalized view of each string: NFKC, zero-width and bidi controls dropped, combining marks dropped, homoglyphs in mixed-script words mapped, percent-encoding, HTML entities, quoted-printable and bracketed at/dot undone, base64 runs that decode to text decoded. Each replacement maps back to the original span, and the stand-in is keyed on the decoded value, so an encoded email and its plain form get one stand-in. The evasion count says how many values only the view found.
• Containers. Archives, mailboxes, git bundles, Office packages and PDFs are opened and every inner part goes through its own kind's path. Nesting is bounded (archives 2 levels, Office packages and decoded views by their own depth limits below).
• Normalized tables (claim switch, off by default). At the end of a scrub, CSV and TSV become typed JSONL tables and JSON arrays of objects become JSONL tables, each with a stats-only schema (column, inferred type, null rate, stand-in rate, distinct-count estimate; never values). They go through the same gate; a table that trips it is held (normalize_residual).

Limits

• Images: at most 512 MiB and 150,000,000 pixels, 30 frames; scanned PDFs up to 80 pages; 3 OCR redaction rounds with 3 px padding.
• Video: frames sampled at 1.0 fps plus scene changes over 0.3, at most 3600 frames at 1920 px wide; face boxes padded 0.15; bleep padding steps (0.15, 0.4, 0.8) s; at most 2000 bleeped ranges and 1500 boxes per file.
• Decoded view: up to 3 nested decodings and 256 decoder calls per string.
• Re-check: 1 MiB chunks with 512 characters of overlap; a verifier hit counts for review at score 0.9 or more.
• Healthcare PHI signal: threshold 25; weights (points per hit, cap) files 25/4, mrn 3/10, npi 3/10, icd 2/15, cpt 2/15, terms 1/20; Part 2 terms counted apart, part2Over at 3 or more.

The reference runner's sandbox adds per-tool CPU, file-size and output limits and a wall clock; the healthcare worker runs each file under 600 CPU seconds, 3 GB of memory, 2 GB of output and 900 seconds of wall time. A file over any limit is held with a limit_* reason; the job carries on.

Threat model

Five adversaries, what each can do, and the defense mapped to each. "Reference runner" means the reference implementation at this commit; control-plane defenses are named as such and documented by each product.

1. The buyer

Has: every delivered file, the signed report, their evaluation sample, as much time and outside data as they want (public records, data brokers, social media, other purchases). Wants: to re-identify a person, the seller's business, or the seller's customers; or to pass the data on.

What each defense's status is at this commit (live, built behind a switch, or partial) is in the auditor edition.

2. The desk

Has: the control plane's console. Wants (or could be coerced) to: read raw values, copy them, or release a version that fails.

3. The runner host

Has (if compromised): one machine, one job's claim, one deal's data for the job's lifetime.

4. The supply chain

5. Refresh differences

A buyer of two versions of the same deal (a refresh, or a patched version after a recall) can diff them. Without care, a value redacted in version 1 and missed in version 2 is exposed by the diff, and a value whose stand-in changes reveals that it is the same person.

Keyed stand-ins are a pure function of the deal salt and the value, and dates shift by one keyed offset per deal, the same every run. With the deal's identity map, values redacted in an earlier run are redacted again in later runs and person ids carry across runs. Each defense's status is in the auditor edition.

Out of scope

• The control plane's own security (authentication, key-encryption key handling, database, vault), each product's console, and the marketplace. Each product documents these.
• Legal re-identification risk certification. The engine measures and reports; Expert Determination is a human expert's judgment (compliance.md).
• Adversaries with the deal data key or the product's key-encryption key.

Keys

Mechanisms only. No key material, secret values or configuration values appear in these pages, and none is needed to follow them.

The keys and what each opens

The salt is the only key the package ever sees. It takes every key from its caller, refuses one shorter than 16 bytes (keys.MIN_KEY), and has no default, no module-level key and no literal key. Turning stand-ins off (set_anon(None)) produces untokenized tags ([EMAIL]), never output keyed by something guessable.

What the salt keys

Each product label is fixed per product and each runner's tests/test_salt_source.py checks that every salt it hands the package comes from HMAC(deal data key, its own label), and forbids the other product's label. The package's test_salt_labels_stay_put pins both labels: renaming one changes every stand-in that product ever delivered (or orphans every sealed map); sharing one would link the products.

The person id in the linkage table (person_<12 hex>) is CSPRNG output in both modes.

Sealed identity map

The package hands the map over as bytes (Graph.state()) and never stores it. The reference runner seals it:

blob   {"v": 1, "alg": "A256GCM", "map": b64(nonce || ciphertext), "lookup": b64(nonce || ciphertext)}
map    Graph.state()                       under K_map     (AES-256-GCM, 12-byte random nonce)
lookup zlib JSON {index: [random stand-in, ...]}  under K_lookup
AAD    "<label>|<deal id>|<section>"       so a section can't be replayed into another deal or slot

The control plane stores the blob in parts of at most 2.5 MB. A run reads the head version, stages the new version's parts, and commits by moving the head from the version it read; a commit answers 409 when another run committed first, and the runner reads the newer map, absorbs it (nothing dropped, older person ids kept) and retries, up to 6 tries. A map in one mode refuses the other (ModeChanged). A claim whose map won't open under its key fails the job rather than starting an empty map.

Crypto-shredding

Destroying a deal's data key (the control plane deletes the wrapped key):

• the deal's vault objects can no longer be decrypted;
• the identity map no longer opens, so no new run can link to old ones;
• the salt can't be re-derived from the data key, so no new keyed stand-in can be made;
• watermark attribution for every version of the deal ends.

What survives, by design, until its own end: the narrow lookup key, if anything from the deal reached a buyer and the contract term hasn't ended. While it lives, the desk can still answer "which stand-ins could this person have become?" for a deletion request; it can't decrypt anything. Destroying it ends that too. A control plane's database backups can hold wrapped keys until their retention window ends; each product states that window in its deletion certificate.

Isolation contract

The package is shared code only. It never reaches infrastructure, keys or data of any product, and tests/test_isolation.py fails the build of every runner that vendors it if that changes:

• No outside IO. No network, process or host modules; no environment reads; no open() outside a short per-file allowlist, each entry with its reason; every git command local.
• No product names. No product name, domain, host, bucket, app, org, key label or claim field anywhere in the package, these pages included.
• Caller-held keys. Every keyed API takes its key from the caller and refuses a missing, empty or short one.
• Unlinkable output. The same input under two keys gives disjoint stand-ins, person ids, git identities, repository-file identities, owner-identity tokens and fingerprints; the same key always gives the same output.

Denied modules unless granted: aiobotocore, aiohttp, asyncio, azure, boto3, botocore, concurrent, ctypes, dbm, dotenv, ftplib, gcsfs, google, grpc, http, httpx, imaplib, mmap, multiprocessing, nntplib, paramiko, poplib, pty, requests, s3fs, smtplib, socket, socketserver, sqlite3, ssl, subprocess, telnetlib, urllib, urllib3, webbrowser, websocket, websockets, xmlrpc.

Third-party imports allowed: 7 (listed in the auditor edition).

Detectors

Detection has four layers, applied to every string as written and to its decoded view:

1. Rules (detector A). patterns.json: credential formats, labeled secrets, personal-data patterns with validators, header-driven column rules, name rules, plus the company rule (capitalized words before an unambiguous legal suffix). Always on; no model.
2. Ensemble (detectors B and C, language model A). Pinned models the reference runner loads and hands to ensemble.py as plain callables. Off when no runner loads them; then name model A is the name detector.
3. Owner identity (blind.py): the deal's own terms (company, brands, domains, people) in every variant, when the control plane sends them.
4. After the scrub: the inline residual gate (patterns.residual, the same rules over the bytes about to ship), the independent re-check (separate code, below), and the release gate's human review (measurement.md).

Rules

patterns.json 1.4.0.

Validators are checksums and structural tests (checks.py: Luhn for cards, IMEI and Canadian SIN; ISO 7064 mod 97 for IBANs; ABA routing; Verhoeff for Aadhaar; TFN weights; German and Dutch VAT; NPI and VIN check digits; IPv6, MAC, UUID, coordinate and plate shape) plus a few in patterns.py (personal email, URL parameters that name a person, international phone, has-a-digit for labeled ids). A pattern hit whose validator fails isn't redacted.

Ensemble logic

ensemble.merge, for every text a detector covers (the language model routes each block; a detector with a language list runs only on those languages; name model A reads English only):

• Union. A hit at or over its kind's threshold for that detector is redacted, whichever detector found it. Kinds a detector reports without a threshold in thresholds.json are ignored.
• Merge. Overlapping kept hits become one span covering all of them. Its kind is the one first in precedence (a checksummed card beats a name), its score the best score for that kind, by the detectors in it.
• Disputed. A kept span that another detector which ran and covers that kind didn't find. Its stand-in goes into the watch set, and the output sampler puts windows showing it first in the desk's review queue.
• Near. A hit at or over its detector's review floor but under its threshold, overlapping no kept span: not redacted, sent to desk review with its context.
• Only adds. Rule hits are redacted on their own path; the ensemble never removes one.
• Blocks. Short texts are joined into blocks and long ones cut into overlapping windows, so one model call covers many lines. Results are cached per string.
• Columns. For a column whose header says nothing, a sample of its values goes through the detectors; a kind that wins enough of the sample claims the column.
• Failure. A detector exception is counted under its alias (never with its message). Under a profile with no fallback (strict) it raises, and the runner holds the file (detector_failed). Otherwise the other detectors' results stand and the report says which detector failed.
• Third-party companies. Kind org. The profile decides: tokenize (strict, standard) or keep (b2b). The seller's own names are never third-party; on owner-identity deals they are the owner pass's job.

thresholds.json 1.0.0.

• Precedence when overlapping hits merge (first wins the kind): secret, card, iban, ssn, uk_nino, ca_sin, au_tfn, in_aadhaar, in_pan, gov_id, mrn, routing, account, email, phone, ip, dob, address, username, name, org
• Kinds the package's own rules (detector A) report: secret, card, iban, ssn, uk_nino, ca_sin, au_tfn, in_aadhaar, in_pan, gov_id, mrn, routing, account, email, phone, ip, dob, address, username
• Default countries for country-filtered recognizers: us, uk, ca, au, in
• Languages routed to the multilingual detectors: en, de, fr, es, it, pt, nl
• Block size: 1500 characters per model call, 120 characters of overlap
• Column inference: 12 sampled values per column, at least 3, a kind wins the column at a 0.6 share; kinds name, account, org, address, phone, email, gov_id, iban, card
• Watch set cap 20,000 stand-ins; near-miss review items kept per run: 40; context shown: 60 characters

Models

Reports and these pages name models by alias. The tree hash is the one every report carries per alias in engine.models and models[]: the sha256 the reference runner computes over the installed files (for packages) or the weights directory (for weights), checked again before any data moves.

Lock-file commitments (sha256 of the file bytes at this version; the auditor edition prints the files):

What each one is, in plain terms:

• Detector A: the package's own rules and company rule. No model.
• Detector B: a recognizer engine of pattern and checksum recognizers with context words, filtered by the deal's countries, plus phone-number parsing. It runs on a slim tokenizer pipeline with its own entity recognition off: no remote, cloud or large-language-model recognizers.
• Detector C: a multilingual span model (about 205M parameters, CPU, full precision; an int8 quantization was measured to change 46% of outputs and rejected). Names, organizations, addresses, account and government ids.
• Name model A: a small English named-entity model, the name detector when no ensemble runs.
• Language model A: language identification in a low-accuracy trigram mode, to route blocks to detectors.
• Verifier A (optional): a token classifier from a different model family than detector C, used only by the re-check for review counts; never baked into the image, never a hold, loaded only when every pinned file matches its hash.
• Verifier B (reserved): a candidate backup verifier from the same family as detector C (so weaker as an independent check), under a custom license that needs review before adoption. Not used.
• Speech model (audio and video): multilingual speech recognition, int8, CPU. Its transcript feeds the same detectors; detected spans are toned out of the audio.
• Face detector (images and video): a small face detection model, CPU. Faces are blurred under every profile whose faces is blur.

A baked model that doesn't match its hash fails the job for name model A; for an ensemble member it fails a strict job and drops the member otherwise (reported). A missing model fails a strict job; otherwise the fallback path runs and the report says so.

Full manifest

The full manifest (model names, versions, sources, per-file hashes, runtime packages) is in the auditor edition only.

Benchmarks

python -m jobs.bench_adapter --out DIR --check (CI's bench job, with the pinned weights fetched) runs the synthetic benchmark corpus through the reference runner's real scrub path in every modality and compares it with bench/baseline.json. A gated metric below the baseline minus the tolerance, or a file leak rate above it plus the tolerance, fails the job. Text, tables and JSON are always gated; mail, office files, born-digital PDFs and healthcare files are gated when their tools are present; image and scan numbers depend on the OCR build, so they are reported and not gated.

Metrics, per entity kind:

• Recall (overlap): share of planted spans (text, tables, JSON) that any detection touches. Recall (exact): that a detection matches at both boundaries. Keyed by the planted kind.
• Precision (overlap): share of detections that touch a planted span, keyed by the detector's kind. Over-redaction lowers it.
• Removed: share of planted values that don't survive in anything a buyer can pull out of the output (OOXML parts and attribute values, PDF strings, decoded bytes, text the runner reads out, and for text drawn in an image, whether its box was blacked out). A held file leaks nothing.
• Leaked / Held: values that survived, and values in files that were held.

bench/baseline.json v1: corpus ca9a55297689575d, seed 1, regression tolerance 0.02. Recorded at package 0.6.0, patterns 1.3.0. The baseline predates the current rules (patterns 1.4.0, package 0.6.0). CI's bench job fails on any gated metric below these numbers minus the tolerance, so they are floors, not this version's measurements.

Weighted by planted values across kinds. Per-kind results are in the auditor edition.

Per modality and file type (a file leaks when any planted value survives in it):

Modalities in each mode's baseline:

• ensemble: audio, doc, face, image, json, mail, pdf, qr, scan, table, text
• ner: audio, doc, face, health, image, json, mail, pdf, qr, scan, table, text
• regex: json, table, text

Per-modality and per-file-type results are in the auditor edition.

Known caveats: the corpus is synthetic and generated by the package (bench/gen.py); real data is messier. Notes on individual baseline rows are in the auditor edition.

Sensitive attributes, per category, on labelled cases plus a held-out set written after the last tuning pass and never tuned on (read the held-out recall as the expected rate on new phrasing):

ner (cases 00d9dbd1af29a75b)

regex (cases 00d9dbd1af29a75b)

Per-category results are in the auditor edition.

Independent re-check

recheck/ is a second detection pass over every byte about to ship, written apart from patterns.py so a bug in the scrub's path isn't repeated: its own formats, its own checksums (not checks.py), its own decoder (not textnorm.py) and its own stand-in recognizer, which blanks what the scrub left before any rule runs. It rewrites nothing. A hit holds the file (recheck_hit); an unreadable shipped file is held (recheck_unreadable). With verifier A loaded, the re-check also hands it a bounded slice of text it already passed; verifier hits at or over the floor count as review items, never holds.

Re-check version 1; kinds it reports: secret, email, phone, ssn, health_id, card, iban, ip, dob, date, url_param.

Profiles and coverage

Profiles

A profile is the policy for one scrub. The control plane names it in the claim (absent: strict); the runner resolves it with profiles.get, fails the job on an unknown or unavailable name, activates it, and reports profiles.summary(p) (id, version, enforced fields, pending fields). HIPAA profiles are always strict underneath (extends: strict) and add their own date, linkage and stand-in rules.

profiles.json 1.3.0; default strict.

• Enforced fields: dates, zip3, age_cap, stand_ins, fallback_allowed, third_party_companies, linkage_table, special_categories, minors, faces, voices, skus, surrogates
• Pending capabilities (profiles.MISSING): none
• Allowed values: dates year, deal_offset, keep; zip3 yes, no; age_cap integer from 18 to 120, or null for no cap; faces blur, keep; voices hold, keep; special_categories hold, flag; minors hold, flag; third_party_companies tokenize, keep; fallback_allowed yes, no; linkage_table risk_checked, off; stand_ins keyed, random; skus keep, tokenize; surrogates tags, realistic
• Expert Determination certificate fields: id, expert, scope, expiresAt

What each field does:

Expert Determination (hipaa_expert) also needs a certificate reference in the claim (id, expert, scope, expiry). profiles.check_certificate checks its shape and that it hasn't expired; it can't check that the determination is real or covers this data. That is the expert's and the control plane's job.

Sensitive attributes

GDPR Art. 9 special categories, Art. 10 criminal data and minors, by sensitive.json. Text is classified per row, record or message where the format has records, and per file otherwise; code, config and git history are not read for them. A held record is dropped and counted (sensitive_held:<reason>); a held file is listed with the reason. The desk can release the releasable reasons for a source; the never-released ones can't be.

sensitive.json 1.0.0. Person cue window: 80 characters; 149 person cues; 62 global cancel contexts.

• Tiers: strong: flags on its own. medium: a term specific to the category; flags beside a person (within window chars). weak: a common word; needs a person and a second distinct hit. not: contexts that cancel any hit they overlap.
• Releasable by the desk: health, racial_ethnic, political, religion, trade_union, genetic, biometric, sex_life, criminal, minor, image_unclassified, pending_detector
• Never released: sexual_minor, image_sexual, escalated, voice

Media: the image classifier and voice detector hooks exist (policy.image, policy.audio) but no classifier is registered in the reference runner yet. Under a profile with no fallback, an image that would need one is held (image_unclassified, pending_detector). Not live: image content classification.

Coverage matrix

coverage.json maps every identifier class the package claims to handle to the rules that detect it and the corpus cases that prove it. tests/test_coverage.py fails when a row cites a rule or case that doesn't exist, cites a known-gap case as proof, or cites no case at all. Status words:

• covered: rules and passing cases for the class as defined;
• partial: rules and cases exist, but part of the class is missed (the note says which);
• planned: not handled yet; the phase names the plan step that builds it.

coverage.json 1.0.0.

HIPAA Safe Harbor, 45 CFR 164.514(b)(2)(i)

GDPR

PCI DSS cardholder data

Totals: hipaa 7 covered, 9 partial, 2 planned; gdpr 2 covered, 7 partial, 1 planned; pci 1 covered, 4 partial Rules, corpus cases and each row's note on what it misses are in the auditor edition.

Two notes in the matrix are older than the code. hipaa.P (voice prints) says the voices field is pending and hipaa.Q (full-face photographs) says the faces field is pending and images are redacted for text only; at this version profiles.json enforces both fields and the reference runner boxes faces with its pinned face detector. The status words (planned) and notes will change in the next coverage.json version, which also changes every report's engine.rules hash; until then the matrix is the conservative reading.

The report's coverageOpen metric counts partial and planned rows, and the Strong privacy band needs it at 0. With the rows above open, no report can reach Strong privacy at this version; Good is the ceiling.

HIPAA and related law

This page maps legal requirements to mechanisms and says, for each, whether the mechanism is live. It is not legal advice and not a compliance claim. Whether a deal is under HIPAA, whether an agreement is signed, and whether counsel has approved a procedure are each product's facts, not the engine's.

Status at this version

Read the rest of this page as "what would carry the requirement once the switch is on and the agreements exist".

When HIPAA applies

Only when the seller is a covered entity (provider, health plan, clearinghouse), its business associate, or the sponsor of a self-funded employee health plan selling that plan's records. Health details in ordinary business records (injury logs, drug-test results, HR files) are not PHI; the sensitive-attribute rules handle them. The engine's role is the signal: health.phi.Signal scores what the scrub saw and reports over when there is any healthcare-format file, or the score reaches the threshold with at least two kinds of signal. The decision is the control plane's and the desk's.

Safe Harbor, 45 CFR 164.514(b)(2)

Two conditions: the 18 identifiers of the individual and of relatives, employers and household members are removed, and the covered entity has no actual knowledge that what remains could identify the individual.

The engine carries the first condition, partly:

• hipaa_safe_harbor is strict plus dates: year (every date the engine recognizes cut to the year; ages at 90 and over grouped), stand_ins: random, linkage_table: off, ZIP3 with restricted areas 000.
• The coverage matrix maps each of the 18 to rules and cases (profiles-and-coverage.md). At this version most rows are partial or planned; each row's note on what it misses is in the auditor edition.
• What a HIPAA profile can't transform it holds: as-is documents with dates inside, git history (random stand-ins aren't built into the history rewrite), audio with speech, images that would need a content classifier, and every healthcare file a parser refuses or can't finish within its limits.
• Free-text names depend on detectors (rules, the ensemble, name model A) with the measured recall in detectors.md, not on a guarantee.

So the engine alone does not establish Safe Harbor for arbitrary data. The release gate's HIPAA rows (zero misses of any kind in the reviewed output sample) and the desk's decision sit on top. The second condition (no actual knowledge) is the seller's and the desk's, not software's.

Expert Determination, 45 CFR 164.514(b)(1)

A qualified expert determines, with documented methods and results, that the risk of re-identification by an anticipated recipient is very small.

• hipaa_expert is strict plus dates: deal_offset, stand_ins: random, and the person-linkage table under risk_checked. The claim must carry the certificate reference (expertCert: id, expert, scope, expiry); a missing or expired one fails the job (profiles.check_certificate). The engine checks shape and dates only.
• What an expert can use from the report: removals by category, the coverage matrix, the benchmark, k-anonymity on the linked view (kMin, shareUnderK), the miss bound from the desk's output sample, the canary result, the re-check counts, sensitive-category counts, and the engine and model hashes that pin what ran.
• What isn't measured: redaction precision from the desk sample, and anything about outside data an intruder could join. The motivated-intruder test is reported once the desk runs it (measurement.md); until then the report says "not yet measured".
• No expert has been engaged through this code; the certificate is a reference the control plane records.

Re-identification codes, 45 CFR 164.514(c)

A covered entity may assign a code to allow re-identification, if the code is not derived from or related to information about the individual and can't be translated to identify them, and the mechanism isn't disclosed.

• Keyed stand-ins (HMAC(salt, value)) are derived from the value, so they don't qualify. That is why both HIPAA profiles use stand_ins: random.
• Random stand-ins: each token is CSPRNG output, drawn once per value per deal and stored in the sealed identity map; stable across refreshes; never computed from the value. Person ids are CSPRNG output in every mode.
• The way back (the map's lookup section) is indexed by HMAC(salt, "rnd:<kind>:<value>"), sealed, held by the control plane, opened only with the narrow lookup key, and never delivered (keys.md).
• Business-id tokens and mail sender pseudonyms come from the map too under random stand-ins.
• Still keyed under random stand-ins: mail message and thread ids (they identify messages, not people) and path collision tags. Path hashes: see threat-model.md. Git history is held instead.

Business associate agreement and the Security Rule

A seller that is a covered entity needs a BAA before any PHI moves; a business-associate seller needs a subcontractor BAA. None is in force. Each product keeps a draft template for counsel.

How the Security Rule's technical safeguards (45 CFR 164.312) map to what the reference runner does:

Administrative and physical safeguards (risk analysis, workforce training, contingency plans) are each product's, and exist only as drafts.

Breach notification, 45 CFR 164.400-414

What a breach procedure would rely on from the engine and runner: the incident's locations by path hash and the manifest that lists every delivered file with its Merkle leaf (to find affected deliveries), the stand-in list for a person (lookup key), and the watermark for attribution. Each product's draft breach runbook covers the four-factor risk assessment, the 60-calendar-day outer limit, notice to the covered entity as business associate, and treating any privacy recall as a trigger. Drafts, not in force. Recall handling: operations.md.

Washington My Health My Data Act and California CMIA

Consumer health data outside HIPAA. Both turn on consent or authorization for collection and sharing.

• Engine: health is a special category in sensitive.json; under strict it is held per record or file, under standard and b2b flagged and counted. The report's sensitive section carries counts by category.
• Control plane (development branches): consumer health data in the scrub holds delivery until the desk records the seller's authorization.
• Not built: per-consumer consent records, residency tests (whose data falls under which state), MHMDA geofencing rules, and consumer deletion portals. The engine can't tell from the data whose consent exists.

42 CFR Part 2

Substance-use disorder treatment records, with their own consent rules.

• Engine: health.phi counts Part 2 terms (treatment program wording, the regulation's name) separately from the PHI score; at the minimum count part2Over is set in the report.
• Control plane (development branches): part2Over holds every delivery until a Part 2 consent is recorded.
• Not built: Part 2's redisclosure notice on delivered records, and segmentation of Part 2 data from other PHI.

GDPR

Coverage rows gdpr.* map personal data categories, special categories (Art. 9), criminal data (Art. 10) and children's data to rules and cases. Most are partial; what each row misses is in the auditor edition. The engine's output is pseudonymized data (stand-ins are reversible by the lookup key holder), still personal data under the GDPR, unless every key is destroyed. Lawful basis, DPAs, records of processing and transfer mechanisms are each product's. The breach runbook drafts note the GDPR's 72-hour clock.

PCI DSS

Coverage rows pci.*: full track data is covered; primary account numbers (Luhn-valid, 13 to 19 digits; a PAN run straight into more digits is missed), expiry and CVV (labeled only) and cardholder names (named columns and track data; free text depends on the name detectors) are partial. PCI scope of the control planes is each product's.

Measurement and the release gate

Five independent measurements, from cheapest to strongest:

1. Benchmark on the synthetic corpus, per engine version (detectors.md). Says how the detectors do on data like the corpus, not on a given deal.
2. Canaries in every scrub job, before any data moves.
3. Inline residual gate and re-check on every file about to ship.
4. Red teams: an owner-identity set (reference runner) and an evasion set (package).
5. The release gate: a desk review of a uniform sample of the delivered output, with an exact miss bound.

Corpus

Owner-identity red team (reference runner, tests/redteam/): 28 planted identifiers across 6 source types, 6 deal terms; every one must be caught.

gap cases record known misses as such (the case is asserted to still miss, so fixing one fails the test until the case moves to the passing set). miss cases in the red-team set say what an evasion does that no detector handles. CI keeps every corpus case passing except the gaps.

Canaries

jobs/canaries.py, before the job reads any real data. Fresh fake values (new each run) are planted in a shadow file per modality (text, CSV, NDJSON, mail, docx, xlsx, PDF, an image with text, a video with on-screen text and subtitles, a spoken call, git history) and run through the real scrub path: dispatch, the document rewrite, OCR boxing, the mail and git rewrites, the A/V pass, the gate and the re-check. Uploads go to memory only.

• Fails the job: any canary value in what the shadow would upload, a shadow file the re-check holds, or a re-check that can't see the raw canaries in a modality it reads.
• Held: a model or tool that path needs is missing, so nothing of it would ship.
• Skipped: the modality's tool isn't installed, or the media switch never runs that path.

Names are reported (ok or missed), not failed on. The report carries the exact 95% upper bound on the canary miss rate over every value checked. It is a bound on synthetic values, not on the deal's data.

Residual gate and re-check

The residual gate scans the bytes the scrub wrote with the rules and holds the file on a hit (residual; the report's rescanClean is true only when no file hit). The re-check (jobs/recheck.py, recheck.py in the package) reads every file again with its own readers per modality, so a reader bug in the scrub isn't repeated: bytes as written for text and records; its own unzip for Office (text per paragraph, cell and row, plus every attribute value); text extraction in the sandbox for PDF; OCR again with a different segmentation for images; the rewritten log and patches for git. A hit holds the file (recheck_hit); a document it can't read back holds it (recheck_unreadable).

Re-check version 1; kinds it reports: secret, email, phone, ssn, health_id, card, iban, ip, dob, date, url_param.

Ensemble members and verifier A also read a bounded slice of each text file the re-check passed. What they flag is a review count by kind, never a hold. Verifier A is optional and absent from the published runner image.

Red teams

• Owner identity (reference runner, tests/test_redteam.py): for confidential-seller deals, the seller's identity planted the ways real data carries it (company names and variants, domains, signatures, text in images, identifying paths, database, repository and channel names, typos, initials). Every planted identifier must be removed from everything that leaves the scrub or raised to the desk as an identity-review item, which blocks release. CI fails below 100%.
• Evasion (package, corpus/redteam/cases.json): spaced-out letters, nested encodings, homoglyphs and similar. Runs on demand; CI only checks the cases are well-formed and distinct from the tuned corpus. Most cases are expected misses with the reason recorded.

Both sets were written by the same author as the rules. An outside red team has not run. Not live.

Release gate

The runner takes a uniform random sample of the delivered clean text (review.OutputSampler): every delivered file is cut into fixed windows and one reservoir runs over all of them, so a file is sampled in proportion to its size. A held or dropped file's windows are taken back out. A second reservoir collects windows showing a stand-in the ensemble's detectors disagreed on, for the desk to look at first; it doesn't touch the uniform sample. Snippets travel only in the job result and are sealed with the deal key by the control plane. The desk marks each uniform-sample snippet clean or a miss, with a severity.

Output sample: 240-character windows, 300 per scrub job by default (the claim can ask for up to 2000).

One-sided exact (Clopper-Pearson) upper bounds at 95% confidence, from review.miss_bound.

miss_bound(k, n) is the exact one-sided Clopper-Pearson upper bound; for k = 0 it is (1 - \alpha^{1/n}), which the rule of three ((3/n)) approximates. When the desk reviewed every window (a census, flagged census in the report's missBound), the control plane reports the exact share instead of a bound. The control planes carry a TypeScript copy checked against tests/vectors/miss_bound.json.

The thresholds the gate applies are control-plane configuration, not engine data, so they aren't rendered here. On one control plane's development branch (gate config version 1, 95% confidence) they are: under strict, at least 300 reviewed windows with no high-severity miss and an upper bound of at most 1%; under standard and b2b, at most 2%; under HIPAA profiles, no miss of any kind. The gate is behind a control-plane switch. Not live in production at the time of writing; check the control plane's own docs for the current state.

What the bound means: with n windows reviewed and k misses, the true share of 240-character windows of delivered text containing a miss is below the bound with 95% confidence, provided the sample is uniform and the desk's marks are right. It doesn't bound misses in what the sample can't show (held files, images without text, re-identification by linkage).

Independent re-check by a different model

Verifier A is a token classifier from a different model family from the ensemble's span detector, pinned by hash like the rest. It is optional: the reference runner's image doesn't bake it, the job never requires it, and it never holds a file. Its results appear under recheck.verifier when it ran. Not live in the published image.

Report quality bands

report/bands.json 1.0.0. Metrics from the desk's output sample: missUpper95, highMisses, precision.

Privacy

Quality

Provenance

Each area's bands are tried top to bottom; a band applies when every condition it lists holds. An unmeasured metric (null) fails its condition, so the band shown is the best one the measurements already prove. When an unmeasured metric is all that stands between a report and a higher band, the band is marked provisional and names that metric. A metric listed as not applicable passes.

Not measured at this version: precision (desk-marked over-redaction in the sample; always null). The miss bound is null until the release gate runs.

Motivated-intruder test

Reported in privacy.intruder (schema 1.1.0), not banded.

• Desk part: the control plane passes the desk's result in the claim (n, identified, unsure, upper, confidence; the bound counts unsure as identified). The desk tries to identify a person or the business in a sample of the delivered text. Until a desk result exists the basis is "not yet measured" and the report's limitations say so. Whether a control plane runs it today isn't something this code shows.
• Surrogate distinguishability (intruder.py), only when the deal uses realistic stand-ins: can a real name be told from the surrogate that replaced it? A character 1- to 3-gram naive Bayes trained in-process on the run's own (real name, surrogate) pairs and scored on held-out folds (auc), and a lookup against the published surrogate name lists (listAuc). 0.5 is chance. Needs at least 30 pairs, else "not yet measured"; "not applicable" with tag stand-ins. Names stay in memory; only the scalars leave.
• Realistic stand-in emails and phone numbers use reserved domains and number ranges, so they are recognizable as stand-ins by design; the report says so when they're used.

Linkage

Linkage is what lets a buyer follow one person, order or ticket across files, systems and refreshes without learning who or what it is. It is also the main re-identification risk once direct identifiers are gone, so each piece below has a limit.

• Map format 1; modes keyed, random; identifier types paired: emails, phones, names, mail_senders, contributors, chat_users.
• Values longer than 8 words aren't kept for re-redaction.
• Quasi-identifiers tracked for k-anonymity (k = 5): zip, age, dob, gender, title, department, city, region, country, ethnicity, marital, language; at most 1,000,000 distinct tuples per table, 20 tables listed.
• Date offset: 1 to 52 whole weeks either way, never zero.

Stable stand-ins

Within a deal, the same value always becomes the same stand-in: keyed stand-ins are an HMAC of the normalized value under the deal salt; random stand-ins are drawn once and read back from the map. Normalization makes formatting variants one value (digit kinds: NFKC digits only; compact kinds drop spaces, dots and dashes; the rest casefolded with whitespace collapsed). Across deals nothing is shared: each deal's salt is different, and every key and digest in the map is keyed by it.

Identity graph

identity.Graph runs when the claim carries the deal's identity map (the control plane's linkage switch), and is required under random stand-ins. Two passes:

1. Pair. Identifiers seen together are filed as one person: a table row's email, phone and name columns, a mail header's display name and address, a chat profile, a git author's name and email. Strong identifiers (email in canonical form, phone in E.164, platform user id) cluster by union-find on keyed link keys.
2. Scrub as usual. Every stand-in the scrub makes is filed under its identifier's link key.

Names are weak and link only when unambiguous: a full name one cluster carries; a first name alone when one known person has it; an initial plus last name with one match; a greeting in a mail thread when one participant has that first name (scoped to the thread). A second person with the same name withdraws the link.

Each cluster gets a person_<12 hex> id (CSPRNG). The person-linkage table (__linkage__/people.jsonl) maps it to the stand-ins its identifiers became; mentions.jsonl carries the thread-scoped greeting links. Under risk_checked the table ships only rows for people whose linked quasi-identifiers fall in a class of at least k; when more than half would be cut it is held whole. Under off (Safe Harbor) no table is made. The table goes through the residual gate, the owner-identity check and the re-check like any file.

The map only gains members: a merge keeps the older person id, so earlier deliveries keep their meaning. Two concurrent runs: the second to commit gets a conflict, reads the newer map, absorbs it (nothing dropped; for random stand-ins both tokens are kept as alternates and both delivered) and commits again.

Redaction only grows

Every value redacted in a deal is remembered as a keyed 64-bit digest (redacted), with digests of first words as anchors. On every later run, Graph.spans finds those values again in any text the scrub reads (digit runs of 7 or more by their digits; up to 8-word values by their anchors) and redacts them whatever the detectors find this time. A refresh can't un-redact what an earlier refresh caught, even after a detector or rule changes. Values longer than 8 words and digit runs under 7 aren't kept.

Without a map (linkage off for the deal), this doesn't run, and a refresh depends only on that run's detectors.

Date offset

Under deal_offset, every date the engine recognizes moves by the same whole number of weeks for the whole deal: HMAC-SHA256(salt, "scrub date offset v1") gives 1 to 52 weeks with the sign from one bit, never zero. Intervals, cross-person ordering and weekdays survive; the offset is never reported (one known pair would undo every shift). File times that ship follow the rule. Birth dates go to the year under every mode except keep.

The offset is a weak defense on its own: a buyer who knows one true date of one event in the deal recovers it. It hides calendar position from casual reading and from joins on exact dates with outside data; it doesn't withstand a buyer with one ground-truth event.

Business-object ids

With the deal's workflow switch on, order, invoice, ticket and other internal business ids become [LABEL_<10 hex>] over a canonical core (a short letter prefix, #, separators and leading zeros dropped), so INV-00042 in billing and invoice #42 in support join. Keyed under the deal salt, or from the map under random stand-ins. With the switch off, internal ids are left as they are. Public ids (parcel tracking numbers) take the ordinary stand-in on every scrub, so redaction-only-grows covers them; customer ids with a type prefix take the person-id stand-in while the switch is on; SKUs follow the profile's skus field. Classes: workflow-score.md.

k-anonymity

kanon.py groups each delivered table's rows by the quasi-identifiers that survive the scrub (ZIP3, birth year, age or 90+, gender, title, place and the rest listed above). The linked view does the same with one tuple per person across every row linked to them. The report gives the smallest class (kMin) and the share under k. This is a measurement, not a mitigation: beyond the profile's fixed rules (ZIP3, age cap, birth year), nothing is generalized or suppressed to reach k, except the linkage table's own risk check above. Coverage row gdpr.identity_factors is planned for that reason.

What linkage gives away

• The linkage table tells a buyer which stand-ins are one person. That is its purpose and its risk; the k check bounds it on the quasi-identifiers the engine knows, not on behavior (the sequence of a person's actions).
• Stable stand-ins across refreshes let a buyer accumulate a profile per stand-in over time.
• Business-id joins expose the shape of the seller's processes (that's what the workflow score measures).

Operations: holds, failures, recall and deletion

Product-specific procedures (who on the desk does what, notice templates, clocks) live in each product's own runbook. This page covers what the engine and the reference runner do and report.

Holds

A hold keeps one file (or one record, or one part of a file) out of the delivery and lists it in the job result as [path hash, reason, size, item hash]. Holds never carry the value or the real path. The desk sees reasons and counts; it can release only the releasable sensitive reasons (below) for a source. A held file leaks nothing, so holds are the engine's answer to "can't prove this is clean".

tests/test_techdocs.py checks that every code in the tables below exists as a literal in the runner or package.

Intake and unpacking

Detection and gates

Documents

Images, audio and video

Sensitive content

Record-level holds inside a table, mail box or chat export drop the record and count it as sensitive_held: + reason; the file still ships.

Healthcare, git, sandbox

Failure modes

• Permanent (jobs/errors.py): retrying won't help. Unknown or unavailable profile; random stand-ins without the identity map; Expert Determination without a valid certificate reference; a baked model that doesn't match its hash (any profile); a baked model missing, or an ensemble member failing to load or failing the self-test, under a no-fallback profile; a canary surviving the scrub or a re-check blind to a canary. Nothing has been written when these fire. The job fails with retry: false.
• Transient: anything else fails the job with retry: true; the control plane decides whether to requeue.
• Per file: everything in the hold tables. The job goes on.
• Fallback: under a profile with fallback_allowed, a missing ensemble falls back to the rules plus name model A and the report says so; strict profiles don't fall back.
• Errors leave the machine only through safe_error and safe_trace: exception type, a masked message and frame locations, never the text a parser or model was reading.
• Live credentials: a scanner run over the source counts credentials and names the provider (so the owner knows what to rotate); never the value or location. Inside the parser sandbox the scanner has no network, so the count is unverified (liveVerified: false).

Recall and deletion requests

What the engine and runner provide; who decides and the clocks are each product's.

• Deletion request for a person. The control plane holds the narrow lookup key, which unwraps to the deal salt. Keyed stand-ins: it computes HMAC(salt, "<kind>:<normalized value>") and gets the stand-ins directly. Random stand-ins: it computes the index and opens the map's lookup section. Either way it gets the stand-ins the person became, without the data key or any delivered data. It can then find which deliveries carry them and tell those buyers. Later scrubs keep redacting the values (redaction only grows). Removing the person from later deliveries is a source-side or control-plane step; the engine has no per-person exclusion list for people (the owner exclusions cover the owner's own terms and addresses).
• Recall of a delivery. Each delivery has a manifest with a Merkle leaf per record and file, so the affected records can be named exactly. A per-buyer watermark lets a leaked copy be attributed (method in the auditor edition). Recall itself (notifying buyers, revoking pulls) is the control plane's.
• Crypto-shredding. Destroying the deal data key ends new linkage and makes the sealed map unreadable; a claim whose map won't open under its key fails rather than starting an empty one. Destroying the package key also ends watermark attribution for that version (keys.md).

The deid_report

One report per scrub of one source, built by report.build from the finished scrub's counts, signed by the runner, verified and sealed by the control plane, which adds what only it measures (the desk's output sample, the Merkle root once the delivery exists) and re-runs the bands. Counts by plain category, never values; no vendor or library name appears in a report.

• Schema deid_report 1.1.0; envelope v1, signing domain deid_report.v1\n.
• Small-cell rule: counts under 11 read <11; k for k-anonymity: 5.
• Rule-file hashes in engine.rules: patterns (patterns.json), profiles (profiles.json), coverage (coverage.json), thresholds (thresholds.json), sensitive (sensitive.json), bizids (bizids.json), workflowBands (workflow/bands.json), reportBands (report/bands.json)
• Model aliases as reports show them: detector_a → detector A, detector_b → detector B, detector_c → detector C, ner-a → name model A, langid-a → language model A

The runner writes the signed envelope and the buyer's HTML report and data card (report.render) to __report__/deid_report.json and .html, through the same gate as every delivered file, and, when the claim carries them, the product's signed-off plain-language one-pagers next to them as markdown and a one-page PDF (jobs/onepager.py lays the text out and adds no words). With no signed-off text, none are written. The delivery manifest and Merkle root cover them.

Envelope

{"v": 1, "alg": "Ed25519", "kid": "<16 hex>", "payload": "<the report as compact JSON text>", "sig": "<base64>"}

• The payload is the report serialized once (sorted keys, compact separators, UTF-8). The signature covers the signing domain followed by the payload text's bytes, so a verifier checks the exact text and never re-serializes; there is no canonical-JSON drift between implementations.
• kid is the first 16 hex of SHA-256 of the raw 32-byte public key.
• The runner holds the 32-byte seed; the package never reads it from anywhere. Without a seed the envelope says alg: "none" with no signature, and the control planes show it as unsigned, never as signed to a buyer.
• report.verify(envelope, public_keys) returns the payload only when the version, algorithm, kid and signature check against one of the given keys. Vectors for other implementations: tests/vectors/deid_report.json.

Versioning

The schema, the band thresholds and the envelope each carry their own version. A minor schema version may add a field; a field's type or meaning only changes with a major version. Band thresholds change only with a new bands.json version, and every report names the version it was judged by. The engine version, the rule-file hashes and the model hashes in engine pin exactly what ran.

Schema

deid_report/1.1.0 (2020-12 draft), title deid_report.

One de-identification report per scrub of one source, built by the engine and signed by the runner (see envelope). Counts drawn from the data read "<11" under 11. No paths (path hashes only), no names, no values, no vendor names: models and detectors by alias. Versioning: a field may be added in a minor version; a field's type or meaning never changes without a major version (CHANGELOG.md).

subject

profile

engine

privacy

quality

dataCard

provenance

bands

Shared definitions:

Changelog

The schema (schema.json), the band thresholds (bands.json) and the envelope each carry their own version. A minor version may add a field; a field's type or meaning only changes with a major version. Band thresholds change only with a new bands.json version, and every report names the version it was judged by.

Schema 1.1.0, bands 1.0.0, envelope v1 (package 0.6.0)

• privacy.intruder gains the motivated-intruder test. The desk part (basis "review sample", added by the control plane like the miss bound): n, identified, unsure, upper (exact one-sided bound counting unsure as identified) and confidence, from up to 50 snippets per source drawn from the desk's output sample. Until the desk runs it, basis stays "not yet measured" as in 1.0.0.
• privacy.intruder.surrogates, when the deal uses realistic stand-ins: how well real names can be told from their stand-ins (auc for a character n-gram classifier trained in-process on the run's own pairs and scored on held-out folds, listAuc for a lookup against the published stand-in name lists, n pairs, folds); "not applicable" with tag stand-ins, "not yet measured" under 30 pairs. Scalars only.
• Known limitations name the missing intruder test, and that realistic stand-in emails and phone numbers are recognisable as stand-ins by design. The page adds a residual-risk sentence naming the controls the results rest on. Bands are unchanged: the intruder results are reported, not banded.

Schema 1.0.0, bands 1.0.0, envelope v1 (package 0.6.0)

• First version: one report per scrub of one source, built by report.build, signed by the runner (Ed25519 over deid_report.v1\n + the payload text; kid = first 16 hex of sha256 of the raw public key).
• Privacy: removals by plain category, the coverage matrix (coverage.json 1.0.0) with removals per row, the canary self-test, the miss bound (from the desk's output sample, added by the control plane), the engine benchmark, k-anonymity on the linked view (per table when linkage is off), linkage counts, sensitive categories, holds by reason (file-intrinsic reasons only).
• Not yet measured, reported as such: the motivated-intruder test and redaction precision from the desk sample. The independent re-check reads "measured" with its counts when the runner ran one, else "not yet measured".
• Quality and data card: near- and exact-duplicate rates, machine-generated share, padding, languages, estimated tokens, redaction density, span, tables by path hash with their quasi-identifiers, known limitations.
• Provenance: engine version and hash, rule-file hashes, model aliases with their pinned hashes, runner image, Merkle root and manifest hash once the delivery has them.
• Bands: Privacy, Quality, Provenance (bands.json 1.0.0) plus the workflow band when the workflow score ran.

The workflow score

How deep and connected the seller's business processes are in the delivered data: can a buyer follow an order from quote to payment across systems? Built in the package (workflow/), run by the reference runner (jobs/workflow.py) at the end of a scrub. Off by default: it runs only when the claim asks for it (workflow: true or a systems map), which also turns on business-id tokenization for the deal.

Business-object graph

A record is one table row, JSON object, or mail or chat message, tagged with the system it came from. Systems: the claim's map (longest raw path prefix wins), else one per database or platform source, one for mail and chat, and one per top-level item for uploads and buckets. Labels stay on the machine; reports number systems s1, s2, ... by record count.

Business objects are the business-id tokens in a record (internal and public classes from bizids.json), plus conversations (a mail or chat thread). Parties (person stand-ins), SKUs and actors ride along for the event log but never join objects. Objects in one record are related.

Case: a connected group of related business objects (union-find), containing at least one business object (a thread alone isn't a case), followed across every system its records came from. Hubs: an object related to more than 200 others links nothing, and a record carrying more than 200 links is skipped, so one catch-all project or digest mail can't fold a deal into one case.

Trace: a case's timed events sorted by time then activity:

• one per time column a row has whose name says what happened (paid_at → paid; bookkeeping and planned dates such as updated_at or due_date make none; a bare date is recorded; an unrecognized name is other);
• one per terminal status value (status: shipped), timed at the row's latest time, unless a time column already gave that activity;
• one per message (email or chat), plus outcomes a message states in words ("payment received"), which are marked not machine-checkable.

Times parse from ISO dates and datetimes, US m/d/Y, and epoch seconds or milliseconds between 2000 and 2100.

• Hub cut-off: an object related to more than 200 others links nothing. At most 3,000,000 linked records per run; traces cut at 64 steps.
• Activities from time columns: approved, assigned, cancelled, churned, closed, completed, created, delivered, disputed, escalated, failed, invoiced, lost, opened, paid, received, refunded, rejected, reopened, resolved, returned, sent, shipped, signed, started, submitted, won, plus recorded (a bare date) and other.
• Time columns that never make an event (any token): accessed, birth, born, cached, changed, deadline, deleted, dob, due, effective, estimated, eta, etl, expected, expiration, expires, expiry, exported, imported, indexed, last, loaded, login, modified, next, period, planned, removed, renewal, renews, scheduled, seen, sync, synced, trial, until, updated, valid, viewed.
• Terminal statuses: approved, cancelled, churned, closed, completed, delivered, disputed, escalated, failed, lost, paid, refunded, rejected, resolved, returned, shipped, won.
• Outcomes (activity → outcome type): paid → paid, delivered → delivered, resolved → resolved, closed → resolved, completed → resolved, refunded → refunded, cancelled → cancelled, returned → returned, won → won, lost → lost, churned → churned.
• Exceptions: cancelled, disputed, escalated, failed, refunded, rejected, reopened, returned.
• Outcomes read from message text (not machine-checkable): refunded, delivered, paid, resolved, cancelled.
• OCEL relations: up to 20 object-to-object relations per object.
• Breadth nudge order: finance, sales, support, supply_chain, operations, hr.

Business-id canonicalization

bizids.py finds ids in named columns and free text (a class word or a known prefix next to an id-shaped value) and gives each object one token, [<LABEL>_<10 hex>], over its canonical core: a short letter prefix, #, separators and leading zeros removed, so SO-001234, #1234 and Order 1234 are one order. Classes sharing a number share the hex; the label keeps them apart and the graph joins on both. The hex is an HMAC of the core under the deal salt, or a random token from the identity map under random stand-ins.

The format map decides only what free text is recognized, never what a token is: the defaults below, the deal's own formats from the claim, and prefixes learned from the deal's id columns after a few sightings. Learning mid-job can't split an object; ids missed before a prefix was learned show up in linkage kept. Learned prefixes never leave the machine.

bizids.json 1.0.0. Functions: sales, finance, operations, support, supply_chain, hr. A prefix is learned from a deal's id columns after 2 sightings.

Metrics

All from workflow.score.metrics. (n) is the number of cases. Ranges are the nearest-rank median and 90th percentile: for sorted values (v_1 \le \dots \le v_m), (p_q = v_{\lceil q m \rceil}), reported with (m). Shares are reported with their denominator.

Linkage kept

Joins before the scrub against joins after it, on the same files. The raw pass reads the raw files (links only, in memory) using raw (class, core) pairs as objects; the clean pass reads the delivered files using tokens. Basis: cross-system joins (multiSystem) when the raw data has any, else any join (joined).

[ \text{kept} = \min\left(1, \frac{\text{clean}[\text{basis}]}{\text{raw}[\text{basis}]}\right) ]

met is kept at or above the target in bands.json. The raw pass has a 4 GiB budget; files beyond it are skipped and the result says sampled: true. No joins before the scrub: kept is null with the reason.

Sampling, ranges and confidence

The clean pass reads every delivered table, JSON, NDJSON, mail and chat file, up to 3,000,000 linked records (records.truncated past that); single JSON documents over 256 MiB are skipped. Ranges and shares carry their sample sizes; no confidence intervals are computed. A share over a small denominator is withheld (below), not qualified.

Bands

workflow/bands.json 1.0.0: at least 11 cases to score; counts under 11 withheld; linkage-kept target 0.95; an episode needs 3 steps, 2 systems and an outcome.

Every listed threshold is metric >= value; – means the band doesn't test that metric. Confidential-deal buckets:

band is the first band, top to bottom, whose every listed threshold holds (metric >= value, null fails). Under the minimum case count the band is "Not measured" with reason "too few cases"; when none applies, "Not measured" with "no band applies".

OCEL 2.0 export

clean/__workflow__/ocel.json.gz, written as chunks through the residual gate like every delivered byte (a hit holds it), next to metrics.json (the privacy pass's copy below). JSON following the OCEL 2.0 JSON format:

Object ids are the stand-ins the delivered files already use (ORDER_<hex>, [ID_<hex>], user_<hex>@...invalid, or the actor's stand-in); conversations are conversation_<n>. Activities come from the fixed vocabulary only, never from headers or values. Events are emitted in record order, only those with a time. Times are what the scrub left, so under deal_offset they're shifted like the delivered files.

Estimate model

Estimate v2 is not built. Nothing in this package or the reference runner computes a pre-scrub estimate of the workflow score. One control plane has a v1 estimate from what the seller declares in a form, not from data; it has no calibration against measured scores. Calibration status: none. Not live.

Privacy controls

workflow.score.private makes the only copy that leaves the runner (scrub report, buyer summary, bundle):

• counts and samples under 11 are withheld (null); system pairs, outcome types and functions under 11 are dropped;
• systems are numbered with their function, never named; activities and outcomes come from a fixed vocabulary;
• confidential deals get buckets only (edges above): no pairs, no per-system rows, no outcome types, no function names;
• linkage kept over fewer than 11 joins is withheld with "too few joins to report".

The raw pass keeps only object keys in memory for the run; nothing raw is written.

Known limitations

• Some id formats in free text are only partly recognized (coverage row hipaa.R).

• Hubs are dropped wholesale, so a genuinely central object (a big account's master project) links nothing.
• Activity names come from column names; a column named in the seller's own terms (zeta_at) becomes other.
• A status event takes the row's latest time, which may be an update time rather than when the status happened.
• Under dates: year (Safe Harbor) dates become bare years, which don't parse as times: traces lose their timed events and depth metrics are near zero. Inferred from the code, not measured.
• No confidence intervals; no check that the same buyer can reproduce a figure from the OCEL file beyond the shared code path.
• Machine-checkable means the outcome came from a structured field, not that it was verified against anything.