Operationalization and Deployment

An evidence system that runs on a laptop is a prototype. An evidence system that runs in production for three years and survives a key rotation, an incident, a deposition, and a custodian departure is an evidence system.

Production operation of an evidence system is a different discipline from building one. The builder's question is: does this work? The operator's question is: does this work on Tuesday morning when the pipeline has been running for six months, the signing key is 170 days old, and the audit log has 4 million rows? The answers are not the same.

At a glance

• Five dashboard metrics with alert thresholds: attestations per hour (alert at 50% of expected rate), chain verification pass rate (any value below 100% is an incident, not a warning), worker error rate (alert at 5%, page at 20%), embedding coverage percentage (alert at 98%), and key age in days (alert at 90 days, require action at 180).
• The 30-day runbook has three concrete shell-command entries — Day 1 baseline attestation audit, Day 7 key rotation test on staging, Day 30 Tier 4 chain integrity check — each executable by an on-call engineer without domain knowledge beyond the runbook.
• Chain verification failure is a legal incident, not an ops incident: the correct response is to snapshot the database immediately, make no repairs, escalate to forensic investigation, and treat the tampered state as evidence — fixing the break destroys the evidence of the tampering.

Learning objectives

By the end of this chapter you should be able to:

1. Identify the five critical dashboard metrics and their alert thresholds, and explain why chain verification pass rate has a different escalation path than the other four metrics.
2. Execute the Day 1 baseline procedure: run canon-verify --all over the matter's attestations, store the output outside the database, and explain why external storage of the baseline is required.
3. Respond correctly to a chain verification failure: snapshot before any other action, make no modifications to the affected rows, escalate to forensic investigation, and preserve the tampered-state snapshot as an exhibit rather than an incident report.
4. Implement key rotation on the 90-day reminder schedule: generate a new Ed25519 key pair, upload the new public key PEM to the versioned bucket, issue a rotation attestation signed by the old key, update the Secret Manager secret, and verify that the next pipeline attestation uses the new key.

Dagster software-defined asset pipeline

For deployments where the full processing pipeline is coordinated across multiple assets — observation, refutation, brief generation, sealing — Dagster's software-defined asset model is the recommended orchestration layer:

pip install meridian-canon[pipeline]

The pipeline exposes four assets in dependency order:

observation_attestation → refute_claims → generate_brief → seal_attestation

Three Dagster resources configure the pipeline without code changes:

Resource	Configuration	Environment variable
DatabaseResource	connection_string	MERIDIAN_DB_URL

sequence of upstream assets that produced it. Dagster's UI shows which attestations are current, which are stale (their upstream sources changed), and which failed. This maps naturally to the Canon evidence model: an attestation is a snapshot of a data lineage, and Dagster's asset graph is that lineage made explicit and monitorable.

Langfuse observability

For production deployments, LM calls are traced via Langfuse:

pip install meridian-canon[langfuse]

Set three environment variables:

LANGFUSE_PUBLIC_KEY=<your-project-public-key>
LANGFUSE_SECRET_KEY=<your-project-secret-key>
LANGFUSE_HOST=<https://your-langfuse-instance>   # or cloud.langfuse.com

All LM calls in the pipeline are traced via @_lf_observe(name="meridian.refute.lm.complete"). Each harness run should pass langfuse_session_id=str(attestation_id) to link Langfuse traces to specific attestations. The Langfuse dashboard then shows, per attestation: model used, prompt content, outcome (survived/failed/revised), latency, and cost. This session-linked trace is the operational counterpart to the audit log: the audit log records what the system decided; Langfuse records what the model was asked and what it said. > ▼ Why It Matters — Traceability under cross-examination. > > When an opposing expert challenges a refutation verdict — "what exactly did > your model say?" — the session-linked Langfuse trace is the answer. It > preserves the prompt, the model response, and the outcome as an immutable > record tied to the attestation's attestation_id. Without this tracing, > the model's reasoning is reconstructed from output alone; with it, the > full input-output context is available for inspection. ## The GCP deployment topology The reference Meridian-Cannon stack targets Google Cloud Platform. The five components: Postgres on Cloud SQL. The database runs on Cloud SQL with the pg_audit extension enabled. pg_audit produces a server-level audit log of every DDL and DML statement, separate from the application-level audit_log table. The two audit layers serve different purposes: the application audit log is part of the Canon evidence chain (hash-chained, attestation-aware); the pg_audit layer is the forensic layer that records activity even if the application is compromised. For evidence-grade deployments, both are required. pgvector on Cloud SQL. The same Cloud SQL instance runs pgvector for dense-vector similarity search. Cloud SQL supports pgvector natively on Postgres 15+. No separate vector database is required; the evidence substrate and the retrieval substrate are the same Postgres instance. Workers as Cloud Run jobs. Each source-type worker (Gmail, audio, PDF, iMessage, custom types) runs as a Cloud Run job, not a long-running service. Jobs are triggered by Pub/Sub messages: when a new source file lands in a designated GCS bucket, the bucket event publishes to a Pub/Sub topic, and Cloud Run picks up the job. This architecture means workers scale to zero when there is no work and scale horizontally under load. A 10,000-document ingestion batch triggers 10,000 Pub/Sub messages and Cloud Run handles the concurrency. Private keys in GCP Secret Manager. The signing key for Canon attestations is stored in GCP Secret Manager, not in macOS Keychain. Keychain is appropriate for local development — Chapter 23 describes the macOS Keychain integration — but Keychain is a local credential store. Production uses Secret Manager, which provides versioned secrets, IAM-controlled access, audit logging of every access, and automatic replication across GCP regions. The worker retrieves the key at job start time, signs its attestations, and discards the in-memory key at job end. The key is never written to disk, never logged, never included in error messages. Content in Cloud Storage with signed URLs. Attestation witness entries with content_ref fields point to GCS objects. The URLs are signed: they expire, they are scoped to specific HTTP methods, and they are audit-logged by GCS. A recipient who verifies a content_ref URL gets the content if the URL is still valid and the recipient's IP is in the allowed range; they get a 403 if the URL has expired or been revoked. The expiry policy is a matter decision — some matters require 30-day expiry windows; others require permanent access for long-archive verifiability. Public key PEMs in a versioned public GCS bucket. The seal.public_key_url in every attestation points to a GCS object in a bucket with public read access. The bucket is versioned: every key rotation adds a new object, and old key objects are preserved. A verifier checking a three-year-old attestation can fetch the key version that was active at signing time. The public key is not a secret. It is meant to be freely available; that is the point. > ◆ Going Deeper — Secret Manager vs Cloud HSM. > > GCP offers two tiers of key storage for signing keys. > > Secret Manager stores the private key as an opaque encrypted blob. GCP > operators can, in principle, access the underlying key material (with > appropriate authorization and audit trails). The key can be exported by > an authorized principal. > > Cloud HSM (Hardware Security Module) stores the private key in a dedicated > hardware module. The key cannot be exported — not by GCP operators, not > by the account owner, not by anyone. The signing operation happens inside > the HSM; only the signature exits. The API is identical to Secret Manager > for Cloud KMS-backed signing operations. > > For most litigation contexts, Secret Manager is sufficient. The access is > audited, the key rotation is documented, and an adversary who wants the > key must compromise GCP's infrastructure rather than just the application. > > For high-stakes matters with well-resourced adversaries — government actors, > large institutions, opposing counsel with forensic resources — Cloud HSM > provides a materially stronger security model. The hardware attestation > that HSM provides means a court-appointed expert can verify that the key > was generated in hardware and never exported, without relying on GCP's word. > > The API change to move from Secret Manager to Cloud HSM is a single > configuration line in the deployment. The decision about which tier to use > is a legal one, not a technical one. ## Feature flag environment summary All runtime configuration is via environment variables. No code changes are required to switch backends or enable optional features. | Variable | Default | Effect | |---|---|---| | MERIDIAN_DB_URL | postgresql://localhost:5433/meridian | Postgres connection string | | MERIDIAN_CUSTODIAN | meridian-pipeline | Signing custodian identifier | | MERIDIAN_PUBLIC_KEY_URL | (NORA Foundation URL) | Public PEM URL in sealed envelopes | | MERIDIAN_USE_PARADEDB | 0 | Set to 1 to enable ParadeDB BM25 search | | MERIDIAN_REKOR_ENABLED | 0 | Set to 1 to enable Rekor transparency log | | PYTHON_KEYRING_BACKEND | (platform default) | Override keyring backend; set to keyrings.alt.file.PlaintextKeyring for CI | | LANGFUSE_PUBLIC_KEY | unset | Langfuse project public key (tracing disabled if unset) | | LANGFUSE_SECRET_KEY | unset | Langfuse project secret key | | LANGFUSE_HOST | unset | Langfuse endpoint URL | In a GCP Cloud Run deployment, these variables are set via the Cloud Run service configuration, not in application code. Secrets (MERIDIAN_DB_URL, LANGFUSE_SECRET_KEY) should be sourced from Secret Manager rather than stored as plain-text environment variables. ## PDF backend for court-quality output For court-filed briefs, use the typst backend:

from meridian.export.pdf import render_brief_pdf
pdf_bytes = render_brief_pdf(brief, backend="typst")

Typst produces near-LaTeX typographic quality. It is a standalone Rust binary:

# macOS
brew install typst

# Other platforms: download from https://github.com/typst/typst/releases

WeasyPrint (pip install weasyprint) is a Python alternative that produces higher quality than ReportLab but requires system libraries (libpango, libcairo) that are unavailable in some CI and container environments. ReportLab remains the default for environments where system library installation is not possible. The backend choice does not affect the attestation's canonical form or signature. PDF rendering is a post-sealing output step; the DSSEEnvelope is sealed before any PDF is produced. ## The five dashboard metrics A production evidence system emits metrics. Not every metric matters equally. The five metrics on the dashboard's top row are the ones that, if they move in the wrong direction, require immediate action. 1. Attestations per hour. This is the pipeline's throughput. A healthy pipeline issues attestations at a rate proportional to the ingestion queue depth. A drop in attestations per hour that is not accompanied by a drop in queue depth indicates a worker failure. An alert at 50% of the expected rate; an incident at 10%. 2. Chain verification pass rate. The hash-chained audit log is verified on a rolling schedule — the pipeline re-walks the chain for the last 1,000 rows on every hour tick. A chain verification pass rate below 100% is not a warning. It is an incident. Any value below 100% means a row in the audit log was modified, deleted, or inserted out of sequence after the chain was computed. The first response action is not to fix the database. It is to snapshot the database and escalate. 3. Worker error rate. The percentage of Cloud Run jobs that exit non-zero. A healthy pipeline runs below 1% error rate; spikes indicate source format changes (the Gmail API changed a field), credential expiry, or upstream rate limiting. Alert at 5%; page at 20%. 4. Embedding coverage percentage. The percentage of chunks with non-null embedding vectors. A fresh ingestion batch will have 100% embedding coverage if the embedding worker is keeping pace. Coverage below 99% indicates the embedding worker is falling behind the ingestion worker — a queue-depth imbalance that, if uncorrected, degrades retrieval quality. Alert at 98%; investigate at 95%. 5. Key age in days. The number of days since the last key rotation. The signing key for Canon attestations should be rotated on a regular schedule — 180 days is the reference cadence. Alert at 90 days as a reminder; require action at 180. An old key is not a security failure by itself. It is a risk accumulation: the longer a key is active, the larger the exposure window if the key is compromised. > ✻ Try This — Design your monitoring dashboard. > > For a production Meridian-Cannon deployment handling your capstone matter: > > Define the alert threshold and the escalation path for each of the five > metrics. For each metric, answer: (a) At what value does the on-call > engineer receive a page? (b) What is the first response action — the > action taken before any diagnosis has been done? (c) Who is notified > besides the on-call engineer? > > The escalation path for Chain verification pass rate is different from > the escalation path for Embedding coverage. One is a potential evidence > integrity incident; the other is a performance degradation. The escalation > paths should reflect that difference. > > Write this down before you need it. An incident is not the time to decide > who gets paged. ## The 30-day operational runbook A runbook is not a procedure manual. It is the thing you read at 2am when the alert fires. It should be short, specific, and executable in the order written. Day 1 — Baseline attestation audit. Run canon-verify --all over every attestation in the database. Log the results. This establishes a clean baseline: every attestation was valid at deployment time. Future audit runs compare against this baseline. An attestation that was valid on Day 1 and invalid on Day 30 was modified between Day 1 and Day 30. That is a finding.

# Run the full-corpus verifier
python -m meridian.canon.walk --all \
  --connection-string postgresql://... \
  --matter-id <uuid> \
  --output audit_baseline_$(date +%Y%m%d).json

Store the output alongside the database backup, not in the database. If the database is compromised, the baseline needs to be independently verifiable.

Day 7 — Key rotation. The signing key is rotated on a 180-day cycle. The first rotation occurs on Day 180; the Day 7 entry in the runbook is a rotation test on a staging environment, confirming the rotation procedure is understood before it is needed in production.

The rotation procedure:

1. Generate a new Ed25519 key pair with meridian-canon keygen --rotation. 2. Upload the new public key PEM to the public key GCS bucket with a version suffix (pubkey_v2.pem). 3. Issue a key-rotation attestation signed by the old key. This attestation declares the new key's fingerprint and records the rotation event in the Canon audit trail. 4. Update the Secret Manager secret to the new private key. 5. Verify that the next attestation issued by the pipeline uses the new key and that the new attestation's seal.public_key_url points to the new PEM. The key-rotation attestation signed by the old key is the chain-of-custody record for the rotation. Without it, there is no cryptographic evidence that the rotation was authorized by the legitimate key holder rather than by an attacker who compromised the Secret Manager secret. Day 30 — Audit log chain verification. Run the Tier 4 conformance check against the full audit_log table for the matter:

canon-verify \
  --connection-string postgresql://... \
  --matter-id <uuid> \
  --tier 4 \
  --output chain_audit_$(date +%Y%m%d).json

A clean result (zero gaps, zero mismatches) is the expected outcome. Store the result. A future discovery request for the audit trail will include both the audit log itself and the chain-verification reports showing it has been continuously verified since deployment.

▼ Why It Matters — The audit log is not a log. It is evidence.

In the 2026 TPR proceeding, the parent's attorney subpoenaed the county's case management system audit logs. The county produced them — but without chain verification. The attorney's expert testified that without a hash chain, there was no mechanism to verify that the logs had not been modified between the events they recorded and the date of production.

The DHS system's audit log was not designed as evidence. It was designed as a compliance artifact. Those are not the same thing.

Meridian-Cannon's hash-chained audit log is designed as evidence from the first row. The chain verification report is the proof. That proof survives any attack on the database that modifies the log, because the modification breaks the chain, and the break is detectable.

On chain verification failures

The hardest discipline in evidence-system operations is what you do when something goes wrong. Chain verification failures require a specific response that is counterintuitive to anyone trained in normal system operations.

In normal system operations, when a database has bad data, you fix it. You run an UPDATE or DELETE, correct the corrupted row, and move on. This is exactly wrong for an evidence-system audit log.

A chain verification failure means one of three things:

1. A row was deleted from the audit log after the chain was computed.
2. A row was modified after the chain was computed.
3. A row was inserted out of sequence after the chain was computed.

All three are potential evidence of tampering. Modifying the database to "fix" the break destroys the evidence of the tampering. The correct response:

1. Snapshot the database immediately. Take a consistent point-in-time snapshot before any other action. This snapshot is the tampered state preserved as evidence.
2. Do not modify, delete, or repair any rows. The break in the chain is evidence. Repairing it is spoliation.
3. Escalate to a forensic investigation. Identify when the break occurred (which sequence numbers are affected), what data is in the affected rows, and whether the break is consistent with a system bug or with deliberate modification.
4. Preserve the tampered state. The snapshot, the chain verification report, and the escalation record are exhibits, not incident reports. They belong in the matter's evidence chain, not in a ticketing system.

☉ In the Wild — The Parler scrape and the missing chain of custody.

In January 2021, after Amazon Web Services terminated Parler's hosting following the January 6 Capitol attack, a researcher scraped approximately 70 terabytes of public posts from the Parler platform before it went offline. The scrape was technically successful: the posts were preserved.

Parler disputed the data's integrity in subsequent legal proceedings, arguing — correctly — that without a chain-of-custody record, there was no mechanism to verify that the scraped data had not been modified between the scrape and the time it was produced. There were no acquisition attestations. There were no hash verifications. There was no audit log.

The data existed. Its provenance was unverifiable.

A properly operated acquisition pipeline would have issued a timestamped, signed attestation for every scrape batch — recording the URL, the response headers, the SHA-256 of the downloaded content, and the time of acquisition. Any subsequent modification of the content would have been detectable. The data's integrity would have been auditable by any recipient.

The lesson is not that the scrape was wrong. The lesson is that 70 terabytes of unattested data is evidence of unknown provenance. Known provenance is the work Canon does.

Key compromise response

A compromised signing key is the most serious incident an evidence system faces. Any attestation issued between key creation and discovery of the compromise could have been forged. Not "was forged" — the attacker had access to the key but may not have used it — but "could have been." That uncertainty is the problem.

The response has four phases:

Phase 1 — Contain. Revoke the compromised key's access to Secret Manager immediately. Generate a new key pair. The new key is the signing key from this point forward.

Phase 2 — Bound the exposure window. Identify the earliest and latest possible compromise time. The earliest is when the key was created. The latest is when the compromise was detected. Identify all attestations issued in that window.

Phase 3 — Re-issue. For every attestation in the exposure window, issue a re-attestation signed by the new key. The re-attestation is a new Canon attestation that includes the original attestation's attestation_id in its Witness block, records the key-compromise event, and signs the chain afresh. Recipients who hold the original attestations are informed. Phase 4 — Disclose. Opposing parties in active matters must be informed of the key compromise and provided with the re-attestations. Failing to disclose is spoliation. Disclosing promptly, with re-attestations, converts a potential chain-of-custody crisis into a documented operational event. > § For the Record — WI § 804.12(2)(a) and disclosure obligations. > > Wisconsin's discovery sanctions rule, § 804.12(2)(a), authorizes sanctions > against a party that "fails to obey an order to provide or permit > discovery." A key compromise that affects the integrity of produced > attestations and is not disclosed is a discovery failure. The disclosure > obligation does not require bad faith; it requires a material effect on > the reliability of produced evidence. ## The custodian departure protocol When the person who holds operational knowledge of the evidence system — the paralegal who manages ingestion, the engineer who wrote the workers, the attorney who configured RBAC — leaves the matter, there is a procedure. On the day of departure: 1. Rotate all credentials the departing custodian had access to: Secret Manager secrets, GCS bucket access, database passwords, Cloud Run service accounts. Write a rotation attestation for each. 2. Review the RBAC table for the matter. Remove the departing custodian's entries. Write an audit row. 3. Transfer knowledge of the operational runbook to the incoming custodian. The runbook is a document, not a person. If it can only be executed by one person, it is not a runbook. The goal is that the evidence system continues to operate with full audit continuity through any personnel change. An audit trail that has a gap at the moment a custodian left is an audit trail that will be challenged. > ◆ Going Deeper — The difference between operational continuity and audit continuity. > > Operational continuity means the pipeline keeps running. Audit continuity > means the audit log has no gaps and every action during the transition is > attributed to a named actor with appropriate authorization. > > A custodian departure that causes a 48-hour pipeline outage is an > operational failure, but it is not an audit failure — the pipeline was > down, and no actions were taken during that window. > > A custodian departure where the departing custodian's access was not > revoked for 72 hours — but the pipeline kept running, and the departing > custodian was technically still authorized — is an audit failure, because > the audit log shows authorized actions by an actor who should no longer > have had access. > > The distinction matters in litigation because the audit log is discoverable. > Opposing counsel will ask who had access during the 72-hour window. The > answer "the former paralegal, whose access was not yet revoked" is not the > answer you want to give. ## Disclosure to opposing parties An evidence system that produces attested records has a disclosure obligation that plain-document productions do not. When an opposing party receives a Canon attestation as a produced document, they have the right to verify it. That verification requires: 1. The attestation JSON itself. 2. Access to the content_ref URLs (or inline content). 3. The public key PEM at seal.public_key_url. 4. The verifier — either the standalone binary or the instructions to run meridian-canon walk on the attestation. Items 3 and 4 are freely available (the public key bucket is public; the verifier is open source). Items 1 and 2 are part of the production. The covering letter for a production that includes Canon attestations should include the instructions for verification: how to install the verifier, how to run it, and what a valid verdict looks like. Providing the verification instructions is not charity. It is the posture that makes the attestation useful. An opposing party who cannot verify an attestation will challenge it on foundational grounds. An opposing party who successfully verifies it cannot challenge the integrity of the document, only its interpretation. Produce in a format the recipient can verify. If the attestation is genuine and the verifier confirms it, the recipient's ability to check it is your advantage, not theirs.

## Exercises ### Warm-up 1. Read the GCP Secret Manager documentation on secret versioning. Identify the API calls required to: (a) create a new secret version, (b) disable an old secret version, (c) list all versions with their status. These three operations are the key rotation procedure at the API level. 2. Read schema/10_core.sql. Identify the audit_log table definition. What column stores the hash of the previous row? What column stores the sequence number? What does the chain verification procedure walk? ### Core 3. Write the chain verification procedure in Python. It should: (a) query the audit_log table in sequence-number order for a given matter_id; (b) recompute each row's chain hash; (c) compare to the stored hash; (d) return a list of mismatched sequence numbers. Test it against the reference database. 4. Design the escalation runbook for a Chain verification pass rate alert at 95%. Write it as a numbered list of actions, each executable by an on-call engineer without domain knowledge beyond what the runbook provides. The runbook should fit on one printed page. 5. Using the five dashboard metrics defined in this chapter, write a Prometheus alert rule in YAML for the metric you consider most critical. The alert should: fire when the metric crosses its threshold for more than 5 minutes, include a summary label that a non-technical on-call engineer can understand, and specify a runbook URL that points to the relevant section of this chapter. ### Stretch 5. A worker exits non-zero 15 minutes after a GCS-triggered ingestion batch starts. The error is a 429 (rate limit) from the Gmail API. Write the retry policy: (a) How many times does the worker retry before giving up? (b) What is the backoff schedule? (c) What audit log entry does the worker write on each retry? (d) What happens to the Pub/Sub message if the worker exhausts retries? 6. An opposing expert receives a Canon attestation in production and runs the reference verifier. The verifier returns step4_witness_content_hashes: {"verified": 2, "failed": 1}. Write the disclosure response: (a) What does a Step 4 failure mean? (b) What does it mean that steps 1, 2, 3, 5, and 6 passed? (c) What investigation is required? (d) What does the producing party disclose to opposing counsel, and when? ## Build-your-own prompt For your capstone matter: write the 30-day operational runbook. The runbook has three entries — Day 1, Day 7, Day 30 — as described in this chapter. Add a fourth entry: the incident response procedure for a chain verification failure specific to your corpus. Who do you call? What do you snapshot? What does the chain break mean in the context of your specific matter? The runbook is a deliverable in Week 8 of the capstone plan, alongside the hash-chain audit log. ## Further reading - GCP Secret Manager documentation, https://cloud.google.com/secret-manager/docs. - GCP Cloud HSM documentation, https://cloud.google.com/kms/docs/hsm. - pg_audit extension documentation, https://github.com/pgaudit/pgaudit — the server-level audit layer. - Russ Cox, Transparent Logs for Skeptical Clients, https://research.swtch.com/tlog — the architecture underlying hash-chained audit logs. - The Parler scrape and its evidentiary aftermath — media coverage from January 2021, multiple outlets; the chain-of-custody gap is discussed in legal filings from subsequent proceedings. - RFC 6962 (Certificate Transparency) — the at-scale precedent for hash-chained public logs. - WI § 804.12 — Wisconsin discovery sanctions; the disclosure obligation framework for produced evidence. - meridian/canon/keys.py in this repository — the Keychain integration for local development, and the interface the Secret Manager adapter must match for production. - The Canon spec v0.2.0, §11 (Long-Archive Verifiability) — the requirement that public key PEMs be durably preserved at stable URLs. - Dagster software-defined assets documentation, https://docs.dagster.io/concepts/assets/software-defined-assets. - Langfuse observability documentation, https://langfuse.com/docs. - keyrings.alt documentation — file-based keyring for headless environments, https://pypi.org/project/keyrings.alt/.

---

This is the last chapter of the book. The system you have built, if you have built it, is one that someone else can verify. That is the discipline. The rest is the work.