docs: add durable runtime unblock runbook

This commit is contained in:
Code Queue Review
2026-05-23 02:58:44 +00:00
parent 84211e072f
commit 9f1a904454
2 changed files with 106 additions and 2 deletions
+73 -2
View File
@@ -174,8 +174,7 @@ Do not declare DEV-LIVE complete while `runtime.adapter="memory"` or
`runtime.durable=false`, even when DB live readiness is connected. A selected
Postgres adapter may report `runtime.durableRequested=true`, but it still stays
non-durable until the schema, migration ledger, and read readiness gates are
proven through the configured adapter. Documenting or testing this contract
must not mutate runtime state or PROD.
proven through the configured adapter.
`/health` and `/health/live` expose this split in `readiness.durability`.
`dbLiveEvidenceObserved=true` with
@@ -185,6 +184,13 @@ separate from runtime persistence. If runtime durability is blocked,
layer, and `requiredEvidence` remains the durable adapter schema, migration,
and read query contract.
When the active blocker is `runtime_durable_adapter_query_blocked`, DB
connectivity is live but the durable runtime adapter has not completed the
required read queries through the runtime store. Future reports must call this a
runtime durability blocker, not a DB connectivity blocker, and must keep M3,
M4, and M5 full acceptance blocked until the durable runtime postconditions
below are true.
## Runtime Migration Automation
The repo-owned runtime migration entrypoint is:
@@ -229,6 +235,71 @@ Runtime migration reports separate the blockers as:
| Migration | The required source migration is recorded in the ledger. |
| Readiness | Durable runtime read readiness passed after schema and ledger checks. |
## Durable Runtime Unblock Runbook
This runbook is a reusable checklist for a future authorized live
mutation/migration. It does not itself authorize a live write. Runners working
from source must stop at the read-only rows unless the commander explicitly
authorizes the DEV apply command and the preconditions below are already true.
| Check | Safe for normal runners | Evidence allowed | Not allowed |
| --- | --- | --- | --- |
| Source migration contract | Yes | `node scripts/dev-runtime-migration.mjs --check` and `node scripts/dev-runtime-migration.mjs --dry-run --write-report` | DB connection, Secret read, DEV/PROD write |
| Static runtime boundary contract | Yes | `node scripts/validate-runtime-boundary.mjs`, source docs, manifests, and redacted Secret refs | Live mutation, service restart, Secret value output |
| Public health observation | Yes, when command scope is read-only | `/health/live` fields such as `db.liveDbEvidence`, `runtime.adapter`, `runtime.durable`, `runtime.blocker`, `readiness.durability.blockedLayer`, `requiredEvidence`, and redaction/safety flags | Printing DB URLs, passwords, tokens, kubeconfig material, or Secret data |
| Live DB read verification | Only with explicit DEV read authorization | `--dry-run --allow-live-db-read --confirm-dev --write-report` output with redacted endpoint and `secretValuesPrinted=false` | Kubernetes Secret reads, writes, migration apply, PROD target |
| Live migration/repair apply | No, unless explicitly authorized by commander/operator | `--apply --confirm-dev --confirmed-non-production --write-report` plus post-apply health evidence | PROD apply, Secret value output, service restart, Code Agent POST, hardware write, or acceptance promotion |
Secret handling rules are the same for every row: verify env names,
`secretKeyRef` names/keys, configured provider/model/base URL, redacted endpoint
source, readiness booleans, blocker codes, and `secretValuesPrinted=false`.
Do not run `kubectl get secret`, do not decode Secret data, and do not paste
`HWLAB_CLOUD_DB_URL`, `OPENAI_API_KEY`, bearer tokens, kubeconfig content, or
passwords into reports, PRs, issues, logs, screenshots, or chat.
Before any authorized live migration or repair, all of these preconditions must
be true:
- The commander has explicitly authorized a DEV-only migration/repair for this
blocker; no PROD target, deploy, restart, or unrelated smoke is bundled into
the same action.
- Source validation passes for `internal/db/migrations/0001_cloud_core_skeleton.sql`,
the durable runtime tables/columns, and the `hwlab_schema_migrations` ledger
row.
- The runtime is requesting the Postgres durable adapter, not silently using
memory: `runtime.adapter="postgres"` or equivalent selected-adapter evidence,
with `runtime.durableRequested=true`.
- DB live evidence is already green through the redacted Secret URL host:
`db.liveDbEvidence=true` and `db.endpointSource="secret-url-host"`.
- The current blocker is recorded exactly, for example
`runtime_durable_adapter_query_blocked`, with `blockedLayer` and
`requiredEvidence` preserved in the report.
- The runner/operator can produce a redacted report that states
`secretValuesPrinted=false` and does not require reading Kubernetes Secret
resources.
After an authorized migration/repair, the blocker has moved or cleared only
when a new read-only health/report observation proves one of these exact
postconditions:
- Cleared: `runtime.adapter="postgres"`, `runtime.durable=true`,
`runtime.ready=true`, `runtime.liveRuntimeEvidence=true`,
`readiness.durability.ready=true`, no
`runtime_durable_adapter_query_blocked`, and durable auth/schema/migration/read
gates are all ready.
- Moved: the old `runtime_durable_adapter_query_blocked` code is absent and a
different explicit blocker is present, such as auth, schema, migration, or
driver readiness; the report must name the new blocker and keep acceptance
blocked.
- Still blocked: `runtime_durable_adapter_query_blocked` remains present,
`runtime.durable=false`, or `runtime.liveRuntimeEvidence=false`; no full
M3/M4/M5 acceptance may be claimed.
No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked.
DB live readiness, Code Agent readiness, source migration readiness, and
read-only health observations are supporting evidence only; they do not prove
durable runtime evidence or the M3 trusted hardware loop.
## Runtime Substitution Ban
UniDesk services, provider-gateway, backend-core, microservice proxies, local