diff --git a/docs/reference/dev-runtime-boundary.md b/docs/reference/dev-runtime-boundary.md index c667444d..194b8e39 100644 --- a/docs/reference/dev-runtime-boundary.md +++ b/docs/reference/dev-runtime-boundary.md @@ -174,8 +174,7 @@ Do not declare DEV-LIVE complete while `runtime.adapter="memory"` or `runtime.durable=false`, even when DB live readiness is connected. A selected Postgres adapter may report `runtime.durableRequested=true`, but it still stays non-durable until the schema, migration ledger, and read readiness gates are -proven through the configured adapter. Documenting or testing this contract -must not mutate runtime state or PROD. +proven through the configured adapter. `/health` and `/health/live` expose this split in `readiness.durability`. `dbLiveEvidenceObserved=true` with @@ -185,6 +184,13 @@ separate from runtime persistence. If runtime durability is blocked, layer, and `requiredEvidence` remains the durable adapter schema, migration, and read query contract. +When the active blocker is `runtime_durable_adapter_query_blocked`, DB +connectivity is live but the durable runtime adapter has not completed the +required read queries through the runtime store. Future reports must call this a +runtime durability blocker, not a DB connectivity blocker, and must keep M3, +M4, and M5 full acceptance blocked until the durable runtime postconditions +below are true. + ## Runtime Migration Automation The repo-owned runtime migration entrypoint is: @@ -229,6 +235,71 @@ Runtime migration reports separate the blockers as: | Migration | The required source migration is recorded in the ledger. | | Readiness | Durable runtime read readiness passed after schema and ledger checks. | +## Durable Runtime Unblock Runbook + +This runbook is a reusable checklist for a future authorized live +mutation/migration. It does not itself authorize a live write. Runners working +from source must stop at the read-only rows unless the commander explicitly +authorizes the DEV apply command and the preconditions below are already true. + +| Check | Safe for normal runners | Evidence allowed | Not allowed | +| --- | --- | --- | --- | +| Source migration contract | Yes | `node scripts/dev-runtime-migration.mjs --check` and `node scripts/dev-runtime-migration.mjs --dry-run --write-report` | DB connection, Secret read, DEV/PROD write | +| Static runtime boundary contract | Yes | `node scripts/validate-runtime-boundary.mjs`, source docs, manifests, and redacted Secret refs | Live mutation, service restart, Secret value output | +| Public health observation | Yes, when command scope is read-only | `/health/live` fields such as `db.liveDbEvidence`, `runtime.adapter`, `runtime.durable`, `runtime.blocker`, `readiness.durability.blockedLayer`, `requiredEvidence`, and redaction/safety flags | Printing DB URLs, passwords, tokens, kubeconfig material, or Secret data | +| Live DB read verification | Only with explicit DEV read authorization | `--dry-run --allow-live-db-read --confirm-dev --write-report` output with redacted endpoint and `secretValuesPrinted=false` | Kubernetes Secret reads, writes, migration apply, PROD target | +| Live migration/repair apply | No, unless explicitly authorized by commander/operator | `--apply --confirm-dev --confirmed-non-production --write-report` plus post-apply health evidence | PROD apply, Secret value output, service restart, Code Agent POST, hardware write, or acceptance promotion | + +Secret handling rules are the same for every row: verify env names, +`secretKeyRef` names/keys, configured provider/model/base URL, redacted endpoint +source, readiness booleans, blocker codes, and `secretValuesPrinted=false`. +Do not run `kubectl get secret`, do not decode Secret data, and do not paste +`HWLAB_CLOUD_DB_URL`, `OPENAI_API_KEY`, bearer tokens, kubeconfig content, or +passwords into reports, PRs, issues, logs, screenshots, or chat. + +Before any authorized live migration or repair, all of these preconditions must +be true: + +- The commander has explicitly authorized a DEV-only migration/repair for this + blocker; no PROD target, deploy, restart, or unrelated smoke is bundled into + the same action. +- Source validation passes for `internal/db/migrations/0001_cloud_core_skeleton.sql`, + the durable runtime tables/columns, and the `hwlab_schema_migrations` ledger + row. +- The runtime is requesting the Postgres durable adapter, not silently using + memory: `runtime.adapter="postgres"` or equivalent selected-adapter evidence, + with `runtime.durableRequested=true`. +- DB live evidence is already green through the redacted Secret URL host: + `db.liveDbEvidence=true` and `db.endpointSource="secret-url-host"`. +- The current blocker is recorded exactly, for example + `runtime_durable_adapter_query_blocked`, with `blockedLayer` and + `requiredEvidence` preserved in the report. +- The runner/operator can produce a redacted report that states + `secretValuesPrinted=false` and does not require reading Kubernetes Secret + resources. + +After an authorized migration/repair, the blocker has moved or cleared only +when a new read-only health/report observation proves one of these exact +postconditions: + +- Cleared: `runtime.adapter="postgres"`, `runtime.durable=true`, + `runtime.ready=true`, `runtime.liveRuntimeEvidence=true`, + `readiness.durability.ready=true`, no + `runtime_durable_adapter_query_blocked`, and durable auth/schema/migration/read + gates are all ready. +- Moved: the old `runtime_durable_adapter_query_blocked` code is absent and a + different explicit blocker is present, such as auth, schema, migration, or + driver readiness; the report must name the new blocker and keep acceptance + blocked. +- Still blocked: `runtime_durable_adapter_query_blocked` remains present, + `runtime.durable=false`, or `runtime.liveRuntimeEvidence=false`; no full + M3/M4/M5 acceptance may be claimed. + +No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked. +DB live readiness, Code Agent readiness, source migration readiness, and +read-only health observations are supporting evidence only; they do not prove +durable runtime evidence or the M3 trusted hardware loop. + ## Runtime Substitution Ban UniDesk services, provider-gateway, backend-core, microservice proxies, local diff --git a/scripts/validate-runtime-boundary.mjs b/scripts/validate-runtime-boundary.mjs index 6c9e4c72..cdf36e66 100644 --- a/scripts/validate-runtime-boundary.mjs +++ b/scripts/validate-runtime-boundary.mjs @@ -189,6 +189,38 @@ function assertRuntimePolicyText(value, label) { assert.ok(value.includes("microservice proxy"), `${label} must name microservice proxy`); } +function assertIncludes(value, expected, label) { + assert.equal(typeof value, "string", `${label} must be a string`); + assert.ok(value.includes(expected), `${label} must include ${JSON.stringify(expected)}`); +} + +function assertDurableRuntimeRunbook(value) { + const label = "docs/reference/dev-runtime-boundary.md durable runtime runbook"; + for (const expected of [ + "DB live readiness and durable runtime readiness are separate gates", + "runtime_durable_adapter_query_blocked", + "Source migration contract", + "Static runtime boundary contract", + "Public health observation", + "Live DB read verification", + "Live migration/repair apply", + "Do not run `kubectl get secret`", + "secretValuesPrinted=false", + "Before any authorized live migration or repair", + "After an authorized migration/repair", + "Cleared:", + "Moved:", + "Still blocked:", + "No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked", + "node scripts/dev-runtime-migration.mjs --check", + "node scripts/dev-runtime-migration.mjs --dry-run --write-report", + "node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --write-report", + "node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report" + ]) { + assertIncludes(value, expected, label); + } +} + function assertGuardFixture(guard) { assert.equal(guard.guard, "hwlab-runtime-boundary"); assert.equal(guard.schemaVersion, "v1"); @@ -365,6 +397,7 @@ assertProdDisabled( await readJSON(guard.requiredK3sSkeleton.prodKustomizationPath), await readJSON(guard.requiredK3sSkeleton.prodDisabledPath) ); +assertDurableRuntimeRunbook(await readText("docs/reference/dev-runtime-boundary.md")); console.log( `validated HWLAB runtime boundary guard for ${SERVICE_IDS.length} services; forbidden substitutes: ${forbiddenRuntimeSubstitutes.join(", ")}`