Witness-DAO operations runbook

For whoever gets paged. It assumes you did not write the code.

The one thing to internalise: this system fails quietly. Almost nothing here crashes. A governance action that never got sealed, an anchor that never confirmed and a credential that expired all look identical from the outside — nothing happens, and the trail is missing something you will not discover until somebody asks for evidence. Every procedure below exists to turn one of those silences into a signal.

1. Initial setup

1.1 Install

Production runs the compiled output. pnpm cli / pnpm mcp run from source through --experimental-strip-types, an unstable Node flag; they are a development convenience and are not supported for anything real.

pnpm install --ignore-scripts
pnpm build
node dist/cli/index.js --version        # the production entrypoint

Node >= 22.6.0 (engines.node). Zero runtime dependencies.

Container:

docker build -t witness-dao:0.1.0 .
docker run --rm -e WITNESS_API_KEY \
  -v witness-state:/var/lib/witness-dao \
  witness-dao:0.1.0 selftest

1.2 Configure

Secrets come from the environment only. A credential in a config file is a hard error (CONFIG_SECRET_IN_FILE), and the check now also matches by value shape — a wtn.-prefixed string anywhere in the file is refused whatever the field is called.

export WITNESS_API_KEY="wtn.<id>.<secret>"
export WITNESS_DAO_RULES_DIR=/srv/governance-rules
export WITNESS_DAO_STATE_DIR=/var/lib/witness-dao
export WITNESS_DAO_LOG_LEVEL=info        # structured JSON logs on stderr
export WITNESS_DAO_METRICS=1             # one JSON metric per line on stderr

1.3 Bind a controller key — do this on day one

Witness API keys are short-lived by design. Without a bound controller key there is nothing to renew with: the key expires, every seal fails, and the failures pile up in the journal until somebody notices.

openssl genpkey -algorithm ed25519 -out /etc/witness-dao/controller.pem
chmod 600 /etc/witness-dao/controller.pem
export WITNESS_DAO_CONTROLLER_KEY_FILE=/etc/witness-dao/controller.pem

node dist/cli/index.js keys bind-controller

Back the private key up somewhere you can still reach if this host is lost — it is the only recovery path once the API key expires. For production, implement the ControllerSigner interface against your KMS or HSM instead of putting the key on disk (see src/cli/controller-key.ts; it is two methods).

The CLI refuses a key file that is group- or world-readable. That is not pedantry: on a shared build host, that is exactly how the account gets stolen.

1.4 Confirm the wiring

node dist/cli/index.js selftest

Checks config, rules directory, providers, anchoring, state-directory writability and lockability, credential reachability, tier, remaining quota, controller-key presence, and current backlogs. Nothing is sealed and no quota is spent.

2. The cadence — all four jobs are required

Scheduling only the first one is the single most common way this product silently stops producing evidence.

purge needs a retention period. Set WITNESS_DAO_RETENTION_DAYS from your limitation-period analysis rather than passing --max-age-days from a crontab — a retention period you cannot point at in writing is not a retention period. Run purge --dry-run first, every time you change the number.

Jobs against the same state directory must not overlap. They take an advisory lock and the loser exits 1 with CLI_STATE_LOCKED rather than proceeding — which is deliberate: two concurrent processes share one journal file and one anchor-run file, and the loser's updates would be discarded silently.

2.1 systemd

/etc/systemd/system/witness-dao-seal.service:

[Unit]
Description=Witness-DAO seal window
After=network-online.target

[Service]
Type=oneshot
User=witness
Environment=WITNESS_DAO_STATE_DIR=/var/lib/witness-dao
Environment=WITNESS_DAO_RULES_DIR=/srv/governance-rules
Environment=WITNESS_DAO_LOG_LEVEL=info
Environment=WITNESS_DAO_METRICS=1
EnvironmentFile=/etc/witness-dao/secrets.env
ExecStart=/usr/bin/node /opt/witness-dao/dist/cli/index.js seal --dao ens.eth --since -1h
# 4 means "there is a gap". Treat it as a failure so the timer's OnFailure fires.
SuccessExitStatus=0

/etc/systemd/system/witness-dao-seal.timer:

[Unit]
Description=Hourly Witness-DAO seal

[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=120

[Install]
WantedBy=timers.target

Systemd will not start a second instance of a oneshot service while the first is running, which is what you want. Repeat for seal --retry (OnCalendar=*:0/15), anchor upgrade (OnCalendar=hourly) and keys renew (OnCalendar=daily).

Alert on the unit's exit status. Exit 4 is not a warning; it means the evidence trail has a hole in it.

2.2 Kubernetes

apiVersion: batch/v1
kind: CronJob
metadata:
  name: witness-dao-seal
spec:
  schedule: "0 * * * *"
  # REQUIRED. Two overlapping runs share one journal file and lose each
  # other's entries; the CLI refuses to run concurrently, so without this
  # you simply get failed jobs instead of sealed evidence.
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 10
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            fsGroup: 1000
          containers:
            - name: witness-dao
              image: witness-dao:0.1.0
              args: ["seal", "--dao", "ens.eth", "--since", "-1h"]
              env:
                - name: WITNESS_API_KEY
                  valueFrom: { secretKeyRef: { name: witness-dao, key: apiKey } }
                - name: WITNESS_DAO_STATE_DIR
                  value: /var/lib/witness-dao
                - name: WITNESS_DAO_LOG_LEVEL
                  value: info
                - name: WITNESS_DAO_METRICS
                  value: "1"
              volumeMounts:
                - { name: state, mountPath: /var/lib/witness-dao }
                - { name: rules, mountPath: /srv/governance-rules, readOnly: true }
          volumes:
            - name: state
              persistentVolumeClaim: { claimName: witness-dao-state }
            - name: rules
              configMap: { name: governance-rules }
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: witness-dao-seal-retry
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: witness-dao
              image: witness-dao:0.1.0
              args: ["seal", "--dao", "ens.eth", "--retry"]
              # …same env and volumes as above
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: witness-dao-anchor-upgrade
spec:
  schedule: "30 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: witness-dao
              image: witness-dao:0.1.0
              args: ["anchor", "upgrade"]

The state PVC must be ReadWriteOnce and mounted by one pod at a time. It holds anchor-runs.json — see §6.

3. What to watch

WITNESS_DAO_METRICS=1 emits one JSON object per line on stderr: {"metric":true,"type":"gauge","name":"witnessdao.journal.backlog","value":3,...}. Bridge it to whatever you use — no dependency required:

// Prometheus textfile collector, in about ten lines.
import { createMetrics } from 'witness-dao/obs';
const values = new Map();
const metrics = createMetrics({
  sink: (e) => values.set(`${e.name}${JSON.stringify(e.labels ?? {})}`, e.value),
});
// …write `values` to /var/lib/node_exporter/witness-dao.prom on exit.

An OpenTelemetry bridge is the same shape: sink: (e) => meter.createGauge(e.name).record(e.value, e.labels).

Alerts worth having

Problems also carry machine-classifiable codes in --json output (problemCodes: [{ code: "journal.backlog", message: "…" }]), so you can alert on the code and show the sentence.

4. Common incidents

4.1 "N event(s) failed to seal and are stored in …"

The window's events are journalled, not lost — but nothing replays them automatically.

node dist/cli/index.js seal --dao ens.eth --retry --json | jq '{queued,sealed,failed,remaining}'

If they fail again, read the code:

Replay is safe: every stored request carries a deterministic decisionId, so re-sending one the server already accepted is deduplicated upstream rather than double-sealed.

4.2 Quorum was not met

anchor: quorum NOT met for this window — rekor:failed (…), rfc3161:failed (…)

The records are sealed and tamper-evident; they are not externally committed. Steps:

1. node dist/cli/index.js anchor status --json — which targets failed, and how old is the run?
2. node dist/cli/index.js anchor upgrade — retries pending and failed attestations.
3. If RFC-3161 is the failure, check whether your TSA is reachable and not rate-limiting. If OTS is the failure, check the calendars.
4. Still failing after two cadences: this is a real gap. Record it. Do not describe the window as anchored, and do not present those records as compliance-grade.

Note that "quorum met" counts only attestations that are confirmed and verified against pinned material. An anchor a third party merely asserts does not count, and the tool will not print it as met.

4.3 An anchor has been pending for hours

anchor status exits 4 and says so once the oldest pending anchor is past its budget (6h by default, WITNESS_DAO_ANCHOR_PENDING_HOURS).

Bitcoin confirms in about an hour. Beyond that, either anchor upgrade is not running (check the timer) or the OTS calendars are unreachable. Until it is promoted, every record in that window stays SIGNED_PENDING and is not compliance-grade.

4.4 `COMPLIANCE_POSTURE_UNSAFE` when building a report

The pipeline refuses to build an evidence report from records it could not verify as ANCHORED_VALID. The message names every offending record and its verdict. Causes, in order of likelihood:

1. The anchors have not been supplied to the report. They come from anchor-runs.json in the state directory — if you are running the report on a different host or a fresh volume, the attestations are not there. Restore the backup (§6) or run the report where the state lives.
2. The anchor is still pending. Run anchor upgrade and retry.
3. The anchor cannot be independently verified. See §7.
4. The account is on the Sealed tier. That state is terminal: those records never become anchored. Enable the Anchored tier and re-seal the window.

--allow-unanchored produces a diagnostic report in which every affected record is named in a caveat. It is for development. Do not send its output to a regulator, an auditor or a court.

4.5 A record verifies as BROKEN (exit 3)

This is the loudest signal the tool can emit and it means integrity actually failed — not missing data. Do not treat it as a bug in the tool until you have excluded tampering.

1. Do not delete or re-seal anything. Preserve the state directory and the bundle.
2. Re-verify offline from an independently obtained DID document: node dist/cli/index.js verify --bundle ./proof.json --did ./did.json --json
3. Read reason:
   • body_hash_mismatch / signature_invalid — the record's bytes do not match what was signed. Escalate as a security incident.
   • chain_* — the hash chain is discontinuous. Also an incident.
   • key_created_at_… after sealedAt — check the host clock first. There is a five-minute skew tolerance; a larger discrepancy is either a real key-rotation problem or an NTP failure. Run NTP.
4. INSUFFICIENT_PROOF (exit 4) is not this. It means the bundle you supplied was incomplete: fetch the predecessor chain, the DID document or the anchors and verify again.

4.6 `CLI_STATE_LOCKED`

Another witness-dao process holds the state directory. The message names the pid, host and command. Either it is genuinely running (lengthen the interval, or set concurrencyPolicy: Forbid), or it died and left the lock — in which case a lock older than an hour is taken over automatically and the takeover is reported. Delete the lock file by hand only if you are certain the process is dead.

5. Handing a proof bundle to a third party

Opposing counsel, an auditor or a regulator should be able to verify without your systems, your credentials, or any trust in the vendor.

# 1. The report artefact itself, plus its canonical JSON pre-image.
node dist/cli/index.js report governance --dao ens.eth \
  --since 2026-01-01T00:00:00Z --until 2026-03-31T23:59:59Z \
  --format md  --out ./handover/evidence.md
node dist/cli/index.js report governance --dao ens.eth \
  --since 2026-01-01T00:00:00Z --until 2026-03-31T23:59:59Z \
  --format json --out ./handover/evidence.json

# 2. A self-contained bundle per record under dispute.
node dist/cli/index.js verify --record rec_01H… --json > ./handover/rec_01H.verify.json

# 3. The DID document, which they should ALSO fetch themselves over TLS.
curl -s https://witness.getvda.ai/.well-known/did.json > ./handover/did.json

Include in the handover:

• the Markdown report and the canonical JSON (the JSON is the pre-image of the reportDigest; the Markdown is for reading);

• the proof bundles: { record, chain, didDocument, anchors };

• the anchor attestations from anchor-runs.json for the windows involved;

• these instructions:

  Verify with npx witness-dao verify --bundle <file>, or with any RFC 8785 + Ed25519 implementation. Fetch the DID document yourself from https://witness.getvda.ai/.well-known/did.json and pass it with --did: a bundle that supplies its own key proves only internal consistency.

Tell them plainly what the report does not prove — the embedded disclaimer does this and must be reproduced verbatim wherever the report is quoted. A verdict of SIGNED_PENDING means tamper-evident but not externally anchored. api_asserted binding means a centralised API said so and nobody could check it.

6. Backup scope — read this once, properly

Back up anchor-runs.json. Everything else in the state directory is disposable.

So: a lost state directory is not an evidence catastrophe, with the single exception of anchor-runs.json. Back that one file up on the same cadence you anchor, verify the restore, and keep it as long as you keep the evidence.

# Minimal, and enough.
tar czf "witness-anchors-$(date -u +%FT%TZ).tgz" \
  -C /var/lib/witness-dao $(cd /var/lib/witness-dao && find . -name anchor-runs.json)

6b. Data-subject requests

Full treatment in DATA-PROTECTION.md. The operational steps:

Erasure (Art. 17).

# 1. Find the records. The subject address is in decision.inputs.actors.
witness-dao report liability --dao <id> --subject 0x… \
  --purpose "Art.17 request REQ-2026-0142 — locating records" --format json

# 2. Crypto-shred each one. --confirm is mandatory; there is no undo.
witness-dao erase --record rec_… --reason "Art.17 request REQ-2026-0142" --confirm

# 3. Confirm the directives are in force.
witness-dao erase --list

The command exits 4, not 0, and that is correct: the VDA SaaS exposes no delete operation, so step 2 destroys your local plaintext and suppresses the record everywhere this tool handles it, but the SaaS copy survives. Raise the deletion with VDA under your Art. 28 contract and record the outcome. Do not tell the data subject the erasure is complete until the processor confirms it in writing.

Put a reference in --reason — a ticket id, a request number. Not the subject's name and not the content: the reason is stored in the clear beside the tombstone, so putting the subject in it defeats the erasure. The CLI refuses a reason that looks like a wallet address.

After an erasure, reports over the affected window carry a caveat naming each tombstone, and no absence finding in those reports may be treated as exhaustive. Verification of a tombstoned record returns INSUFFICIENT_PROOF, never BROKEN — that is the intended, honest outcome.

Rectification (Art. 16). You cannot rewrite a sealed record and should not want to. Seal a correcting record carrying detail.supersedes = { recordId, reason }; both report builders then mark the original [SUPERSEDED by …] inline and in the caveats. See DATA-PROTECTION.md §9.

Access (Art. 15). report liability --subject <address> --purpose "Art.15 subject access request …" produces the participation history. Remember the resulting pack is itself a profile: transmit it to the subject, not to anyone else.

7. Reaching PROVEN

PROVEN requires both an ANCHORED_VALID record and an onchain_confirmed binding. The verification half needs an anchor that was checked against material you pinned in advance — an anchor the issuer merely asserts is not evidence, and the tool will not pretend otherwise.

Out of the box, nothing is pinned, so every anchor is asserted_unverified, every record caps at SIGNED_PENDING, and the report says exactly which material is missing. To go further, supply, via the library API:

• Bitcoin — a block-header lookup (blockHeaderLookup) from your own node or a trusted explorer, so an OTS proof's asserted Merkle root can be confirmed.
• Rekor — the log's public key, pinned out of band (pinnedPublicKeyPem). Fetching it from the log you are verifying is circular and is refused.
• RFC-3161 — a chainValidator that validates the signing certificate against your trusted TSA roots. Deciding which TSAs you rely on is a policy decision only you can make, which is why there is no default.

8. Disaster recovery drill

Run this quarterly. It takes ten minutes and is the only way to know the backup works.

1. Provision a clean host or container with no state directory.
2. Restore anchor-runs.json from backup into the right namespace path.
3. Export WITNESS_API_KEY (or renew with the controller key).
4. node dist/cli/index.js selftest — expect 0, or 4 with an explained warning.
5. node dist/cli/index.js report governance --dao <id> --since <old window> and confirm the report builds and the verdicts match what you had before.
6. node dist/cli/index.js anchor status and confirm the historical runs are present.

If step 5 produces COMPLIANCE_POSTURE_UNSAFE where it previously did not, your anchor-runs.json backup is incomplete. Fix that before you need it.