Enterprise-readiness audit — Witness-DAO

Auditor role: Principal SRE / platform engineering Date: 25 July 2026 Scope: witness-dao @ 0.1.0 — durability, observability, resilience, scale, key management, deployability, testing Method: read-only code audit + executable PoCs and measurements. No source file modified. PoCs in /tmp, no third-party APIs contacted, nothing sealed. Question asked: if an enterprise deployed this to produce evidence they will later rely on in court or before a regulator, would it hold up — and would they know if it silently stopped working?

Verdict

No — not production-ready, and the reasons are structural rather than cosmetic. The library's internals are careful and its self-assessment is unusually honest, but three defects break the product's core promise in normal operation, with no error raised: failed seals are recorded to a retry queue that nothing ever retries; evidence reports re-fetch their substantive content live from Snapshot/Safe/RPC at report time, so the same report is not reproducible and silently degrades when a third party is slow or unavailable; and the headline PROVEN grade is unreachable through the shipped CLI and MCP surfaces, because neither passes anchor attestations into report generation. On top of that there is no logging, no metrics, no alerting, no retry scheduler, no key-renewal implementation, and no customer-held-key signing (an advertised tier that does not exist in code). The good news is that the trail itself lives upstream in Witness and is re-fetchable, so local state is largely disposable and none of this is unrecoverable data loss — it is unnoticed incompleteness, which for an evidence product is the worse failure. Budget several weeks, not days.

Readiness scorecard

Findings

Per-finding detail

E-01 — Failed seals are never retried (Critical)

Mechanism. Sealer.sealEvent catches a seal failure and calls journal.recordFailure(eventId, request, error), which durably stores the complete SealRequest. The CLI then reads journal.pending() at run.ts:463 only to print a warning:

N event(s) are queued for retry in <state-dir>/seal-journal.json. They are NOT in the trail until they seal successfully.

An exhaustive search for any consumer that re-submits those stored requests returns nothing. anchor upgrade retries anchors, not seals. There is no seal --retry command and no scheduler.

Worse, FileSealJournal.has() (state.ts:116) checks only succeeded, so a previously-failed event is re-attempted only if the operator happens to re-run a window that still covers it. In the intended deployment — a cron job advancing --since — that window moves on, and the event is never attempted again. The stored SealRequest sits in the file forever.

Production scenario. Witness has a 15-minute outage at 02:00. The nightly job seals 40 of 60 events, journals 20 failures, prints a yellow warning to stderr that nobody is watching, and exits 4. Tomorrow's run covers a new window. Those 20 governance actions are permanently absent from the trail. Six months later a liability-defence pack is generated over that period and reports inferred_absence for a signer who did vote — an actively misleading artefact.

Why this is the headline finding. Fail-open is only defensible if something closes the loop. As shipped, fail-open is silent data loss with extra bookkeeping, and the word "retry" in the code and the warning message is an over-claim.

Fix. Add witness-dao seal --retry that drains journal.pending() by replaying stored requests (the decisionId makes replay idempotent upstream), wire it into the documented cadence, and add an age-of-oldest-pending metric with an alert threshold. Until then, change the warning text — it promises a retry that does not exist.

E-02 — Reports re-ingest content live from third parties (Critical)

Mechanism. report fetches sealed records from upstream Witness (listAllRecords, run.ts:744) — good — but then, at run.ts:768-776, re-runs the whole ingest pipeline against Snapshot / Safe / RPC to pair each record with its event, because the report layer takes actors, kind and binding from the GovernanceEvent, not from the signed record. The code's own comment concedes the consequence: "Without the events, every line is UNSUPPORTED — technically honest, but far less useful — so an ingest failure here is a gap."

Three consequences, in ascending severity:

1. Availability coupling. Generating an evidence report requires Snapshot, Safe and your RPC to all be up. A provider outage silently downgrades every affected line's strength. The problem is appended to problems[] as prose; the rendered report understates the evidence without a reader necessarily understanding why.
2. Non-reproducibility. The same command, same period, run twice, can produce different artefacts — proposals get deleted, spaces reconfigured, APIs change shape, RPC providers prune history. For a document intended for a court, non-reproducibility is close to disqualifying: opposing counsel regenerates it and gets something else.
3. Integrity. The signatures protect the records; the report's substantive claims come from an unsigned live re-fetch. This independently corroborates the cryptographic audit's F-01 from the operational side — the trust anchor and the report content are different objects.

Fix. Reconstruct events from the signed decision.inputs written by Sealer.buildRequest (which already contains kind, actors, binding, detail verbatim) instead of re-ingesting. Re-ingest, if retained at all, should be an explicit --corroborate flag that cross-checks and reports divergence, never the source of truth. This change also removes the availability coupling and makes reports reproducible offline.

E-03 — `PROVEN` is unreachable through the shipped surfaces (Critical)

Mechanism. offlineVerify can only return ANCHORED_VALID when it is given (a) anchor attestations covering the record and (b) an anchorVerifier. WitnessDaoPipeline.buildGovernanceReport / buildLiabilityReport accept both via anchors and verifyOptions (pipeline.ts:241,263,293). Neither the CLI (run.ts:783-800) nor the MCP server (server.ts:720) passes either. Only the standalone verify command supports --anchors.

Therefore every line of every report produced by the product caps at SIGNED_PENDING, and by the documented strength matrix at STRONG. PROVEN — the single cell the README leads with, and the grade the liability-defence pack's value rests on — cannot occur.

There is also an internal inconsistency: assertCompliancePosture gates report generation on the record's upstream-asserted anchorState field, while the grading logic ignores anchors entirely. The gate and the grade disagree about what "anchored" means.

Fix. Load attestations from the FileAnchorRunStore (they are already persisted per run, keyed by Merkle root) and pass them plus a real anchorVerifier into report building. Add a test asserting that an anchored, on-chain-bound record reaches PROVEN end-to-end through the CLI — its absence is why this shipped.

E-04 / E-05 — Durability and concurrency (High)

E-04, no fsync. writeJsonFile (state.ts:82) does writeFile(tmp) → rename(tmp, path) with no fsync on the temp file and no fsync on the containing directory. On ext4 data=ordered, XFS and most modern filesystems, a crash shortly after the rename can leave the rename un-durable or the file zero-length. The header comment states "a crash mid-write cannot corrupt the store. That is the only guarantee offered" — the guarantee is narrower than stated. Fix: fh.sync() before close, then open and fsync the directory after rename.

E-05, lost updates. Both stores load the whole file into memory at open() and write the whole in-memory object on every mutation — last writer wins for the entire document. Demonstrated:

$ node /tmp/wda/lost.mjs
events on disk: [ 'event-B' ]
LOST: [ 'event-A' ]

Two overlapping runs (a slow run plus the next cron tick — routine, not exotic) silently discard one another's entries.

Blast radius, stated precisely. Because reports read records from upstream rather than the journal (run.ts:744), a lost succeeded entry does not corrupt reports — it causes a harmless re-seal, deduplicated upstream by decisionId. The real damage is a lost failed entry (the only record that an event needs re-sealing — see E-01) and a lost anchor run in FileAnchorRunStore, which holds the only copy of the OTS/RFC-3161 attestation material. Losing an anchor run means the pending Bitcoin anchor can never be upgraded and the proof material is gone.

Fix. Advisory lock on the state directory (open with O_EXCL on a lockfile containing the pid, stale-lock detection), or move to SQLite/WAL. Refuse to start if the lock is held, rather than proceeding concurrently.

E-06 / E-07 — Key management is unbuilt (High)

E-06. The product materials position a "higher-assurance, customer-held-key" tier as the answer to DAO scepticism about a custodial vendor. There is no local signing anywhere in src/ — the only privateKey reference is a forbidden-key string in config.ts. The client exclusively POSTs to the Witness API, which signs. The tier is a slide, not a feature. If it is being sold, that is a claim gap; if it is roadmap, README.md should say so rather than presenting it as available.

E-07. Upstream Witness keys are short-lived by design (~24 h for durable accounts) with an Ed25519 controller-key renewal protocol (renew/challenge → sign vda.witness.renew/1|<accountId>|<nonce> → renew). This codebase implements none of it — mintTestKey merely forwards an optional controllerPublicKeyJwk. A production deployment's key therefore expires and sealing stops, surfacing as generic seal failures which, per E-01, are never retried. Two defects compound into permanent silent gaps.

Fix. Implement the renewal flow, store the controller private key via a KMS/HSM interface (not a file), and add a pre-flight key-expiry check with an alarm at 25% of remaining lifetime.

E-08 / E-10 — Observability (High)

There is no logging framework, no metric emission, no trace context, and no health or self-test command anywhere in src/. Everything an operator learns comes from CLI stdout/stderr at invocation time. For a scheduled unattended job producing legal evidence, this is the difference between "working" and "believed to be working".

run.problems is string[] of English prose. It cannot be classified, counted, or alerted on without regex-scraping human sentences — an operability defect in its own right, since it is the only channel for every failure the pipeline tolerates.

client.usage() is implemented and tested but never called by any product code path (only by its own unit tests). The upstream free allowance is 5,000 seals/month and upstream warns but never blocks — so an enterprise can silently sail past its allowance with no signal. Likewise the tier: nothing checks whether the account is actually Anchored rather than Sealed until assertCompliancePosture refuses a report, potentially months of sealing later.

Minimum viable telemetry: seal success/fail counts, journal backlog depth and age-of-oldest, anchor pending age, quorum-met rate, per-provider error rate and latency, usage.remaining, tier, and key time-to-expiry.

E-09 — O(n²) verification (High, measured)

offlineVerifyChain calls offlineVerify per record with chain: ordered.slice(0, idx + 1), and each call re-canonicalises and re-hashes the entire prefix. Measured on this machine (Node 22, compiled dist/):

A clean quadratic (t ≈ 7.7×10⁻³·n² ms). Extrapolating: 10,000 records ≈ 13 minutes, 20,000 ≈ 52 minutes, 50,000 ≈ 5.4 hours, 100,000 ≈ 21 hours. Memory is not the constraint — CPU is.

This matters because a chain is append-only and grows for the life of the DAO. A mid-sized DAO sealing 100 events/week crosses 10,000 records in two years, at which point full-trail verification stops being something you can do inside a CI job or a court deadline.

Fix. Single forward pass: verify each record's own body hash and signature once, and check prevHash against the immediately preceding record — that is O(n) and proves exactly the same continuity property. The current shape re-proves every prefix redundantly.

E-11 to E-17 (Medium / Low)

• E-11. anchor upgrade is documented as "run on a cadence" but nothing schedules it, and no alarm fires if a Bitcoin anchor stays pending far beyond the ~1 h expectation. An un-upgraded anchor silently leaves records at SIGNED_PENDING forever — and per E-03 they were never going to be graded on anchors anyway.
• E-12. src/cli/state.ts — every durability guarantee in the product — has zero test coverage; E-04 and E-05 would both have been caught by one. test/integration/ does not exist despite the test:integration script, so no wire format has ever been checked against a real endpoint (consistent with LIMITATIONS.md's admission about RFC-3161).
• E-13. No Dockerfile, no .github/, no healthcheck, no runbook for: initial setup, scheduling cadence jobs, responding to a failed quorum, responding to a BROKEN verdict, or handing a proof bundle to a third party. DR is the notable bright spot — because records and the DID document are re-fetchable from upstream Witness, local state is nearly disposable; the exception is anchor-runs.json, which holds the only copy of anchor proof material and must be backed up. That distinction is nowhere documented.
• E-14. --state-dir is global, not per-DAO. Multiple DAOs sharing a state dir interleave journals and anchor runs in one file, worsening E-05 and mixing tenants' data in a single 0600 file.
• E-15. Key-validity comparisons use exact timestamps with no skew tolerance. A host clock a few seconds fast relative to the DID document's created yields BROKEN / key_created_at_… after sealedAt — reporting tampering for what is an NTP problem. Given that BROKEN is the CLI's exit code 3 and its loudest possible signal, a false positive here is costly. Add a configurable tolerance (±5 min is conventional) and document the NTP requirement.
• E-16. httpRequest has no response-size cap (res.text() / arrayBuffer() buffer whatever arrives), and buildMerkleTree bounds neither leaf count nor total memory. Both are reachable from hostile or merely malfunctioning upstreams. (The application-security audit covers the adversarial framing; noted here as a capacity-planning gap.)
• E-17. pnpm cli / pnpm mcp run from source via --experimental-strip-types, an unstable Node flag. dist/ is the only production-safe entrypoint; README.md mentions this only in passing in LIMITATIONS.md. State it plainly in the deployment section.

The silent-gap analysis

Every path by which the trail becomes incomplete without anyone noticing, ranked by likelihood in a normal deployment.

The unifying defect: every tolerated failure terminates in a human-readable string, and no deployment described in the README has a human reading it.

Minimum viable production checklist

Ordered. Items 1–5 are blocking; nothing should reach an enterprise pilot without them.

1. Close the seal retry loop (E-01) — seal --retry draining the journal, scheduled, plus backlog-age alerting. Until it exists, correct the warning text that promises retry.
2. Make reports reproducible (E-02) — build report content from the signed decision.inputs, not a live re-ingest. Demote re-ingest to an optional cross-check.
3. Wire anchors into reports (E-03) — load attestations from the anchor store, pass an anchorVerifier, and add an end-to-end test proving PROVEN is reachable.
4. Fix durability (E-04, E-05) — fsync file and directory; advisory lock on the state directory; refuse to run concurrently. Add tests for state.ts.
5. Key lifecycle (E-07) — implement controller-key renewal, KMS-backed key storage, and pre-flight expiry checks with alarms. Correct or remove the customer-held-key claim (E-06).
6. Observability (E-08, E-10) — structured JSON logs, a metrics surface, machine-classifiable problem codes replacing prose, and surfacing of usage/tier/anchor-age.
7. Cadence enforcement (E-11) — ship the scheduler (systemd timer / k8s CronJob) alongside, and alert on pending-anchor age.
8. Linearise verification (E-09) — single forward pass; add a performance regression test.
9. Operational packaging (E-13) — Dockerfile pinned to a Node LTS, CI running typecheck + tests, a selftest command that exercises wiring without sealing, and a runbook covering quorum failure, BROKEN verdicts, backup scope (anchor-runs.json especially) and third-party proof handover.
10. Hardening (E-14, E-15, E-16) — per-tenant state namespacing, clock-skew tolerance with a documented NTP requirement, response-size and leaf-count caps.
11. Integration tests (E-12) — real DigiCert/Sectigo, a real OTS calendar, a real Snapshot query, behind the existing test:integration script.

What is genuinely well done

Calibration matters, and several things here are better than typical:

• Records are re-fetchable from upstream. Reports source records via listAllRecords and the DID via getDidDocument, so a lost state directory does not destroy the evidence. That is a real disaster-recovery property, achieved almost by accident but valuable — with the one exception of anchor-runs.json, which should be called out as the thing that must be backed up.
• Corrupt state files refuse to start (state.ts:69-79), explicitly rather than resetting to empty. Most implementations silently reinitialise and destroy the queue. The reasoning in the comment is exactly right.
• Deterministic decisionId gives genuine upstream idempotency, which makes replay-based retry safe to build and makes the E-05 re-seal case harmless.
• Exit code 4 for an incomplete run, with problems always written to stderr, is the right CI contract — the plumbing for "never treat a gap as success" is in place, it is just not connected to anything that watches.
• Per-chain sequential sealing with parallelism only across chains correctly respects the upstream monotonic seq/prevHash contract — an easy thing to get wrong.
• LIMITATIONS.md is unusually candid and several findings here (no locking, in-memory stores unsuitable, integration tests missing) are already admitted there. It is, however, incomplete rather than inaccurate: it does not mention the missing seal-retry consumer, the live re-ingest in reports, the unreachable PROVEN grade, the absent key renewal, or the unimplemented customer-held-key tier — the five most consequential gaps found. Closing that delta would make it a genuinely trustworthy document.
• 753 passing unit tests with zero dependencies, and the crypto/grading logic is tested adversarially against over-claiming. The coverage gap is specifically the operational layer, not the core.