pikasTech-HWLAB/docs/reference/dev-runtime-boundary.md

HWLAB DEV Runtime Boundary

This reference defines the stable DEV environment, port, k3s, and runtime substitution rules.

Public DEV Ports

Surface	URL
Cloud Web browser entry	http://74.48.78.17:16666/
API/edge entry	http://74.48.78.17:16667/
API/live health	http://74.48.78.17:16667/health/live

Public :6666 and :6667 are deprecated as browser/API acceptance targets. Internal k3s services may still listen on 6667; do not rewrite that as a public endpoint.

Route Shape

The public route is:

master hwlab-frps-dev :16666/:16667
-> D601 hwlab-frpc in namespace hwlab-dev
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health

deploy/deploy.json is the source for public endpoint and FRP mapping drift checks. Use:

node scripts/deploy-contract-plan.mjs --check

D601 k3s Source Of Truth

D601 native k3s is the DEV runtime source of truth. Commands that observe or roll a DEV workload must specify:

KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy,svc,pod -o wide

Do not trust the default kubectl context as a DEV runtime signal. On D601 it may point at another cluster such as docker-desktop.

Use the read-only preflight when a runner needs to report k3s visibility:

node scripts/d601-k3s-readonly-observability.mjs

The preflight must not print secret values, kubeconfig material, ConfigMap values, or token content.

Cloud API DB Readiness Authority

hwlab-cloud-api reads the DEV database connection string from Secret reference hwlab-cloud-api-dev-db/database-url. The Secret value is not stored in source, not printed by reports, and not required for offline validation.

Runtime DB readiness uses the redacted host parsed from the injected HWLAB_CLOUD_DB_URL Secret value:

Field	Value
Authority source	secret-url-host
Authority env	HWLAB_CLOUD_DB_URL
Required for readiness	true
Probe target	redacted host and port parsed from the Secret URL

Health output may report endpointSource: secret-url-host, DNS/TCP result, liveConnected, and liveDbEvidence, but must keep the DB endpoint and Secret value redacted.

The non-secret cloud-api-db name is an optional desired alias:

Field	Value
Source	optional-public-dns-alias
Service	cloud-api-db
Namespace	hwlab-dev
Host	cloud-api-db.hwlab-dev.svc.cluster.local
Port	5432
Port name	postgres
Required for readiness	false
Used for probe	false

Missing Service/cloud-api-db, Endpoint, EndpointSlice, or alias env vars must not reverse-fail live DB success from the Secret URL host. A future PR may make that alias operational only if it also owns the Service plus Endpoint or EndpointSlice manifests and the rollout/apply contract.

Source validation checks HWLAB_CLOUD_DB_URL, HWLAB_CLOUD_DB_SSL_MODE, endpointSource=secret-url-host, and liveDbEvidence=true with secret values redacted. It must not require HWLAB_CLOUD_DB_SERVICE_NAME, HWLAB_CLOUD_DB_SERVICE_NAMESPACE, HWLAB_CLOUD_DB_HOST, or HWLAB_CLOUD_DB_PORT as current readiness inputs.

The DB readiness contract is layered:

Layer	Required evidence
DNS	The configured target resolves from the hwlab-cloud-api runtime path and is not a .invalid placeholder.
TCP	A redacted TCP probe reaches the configured Postgres port.
Auth	Authenticated database access succeeds without printing the connection string.
Schema	Required HWLAB schema/migration checks pass without using fixture output as live evidence.

*.invalid and hwlab-dev-db.invalid are forbidden DEV runtime targets. They may appear only as negative test fixtures; source and runtime health must not treat them as desired DEV DB endpoints.

Code Agent Provider Contract

hwlab-cloud-api runs the DEV Code Agent provider through OpenAI Responses. Source-controlled manifests must declare only env names, Secret references, and non-secret egress settings:

Field	Value
Provider env	HWLAB_CODE_AGENT_PROVIDER=openai
Model env	HWLAB_CODE_AGENT_MODEL=gpt-5.5
Provider Secret	OPENAI_API_KEY from hwlab-code-agent-provider/openai-api-key
DEV egress/base URL	HWLAB_CODE_AGENT_OPENAI_BASE_URL=http://172.26.26.227:17680/v1/responses

DEV pods must not call https://api.openai.com/v1/responses directly. The base URL must use the approved DEV egress/proxy path so provider reachability is a repeatable deployment contract instead of a one-off runtime patch.

Reports, smokes, and health payloads may show Secret name/key presence, missingEnv, provider status, model, trace IDs, and redacted egress status. They must never print the OpenAI API key, bearer token, database URL password, kubeconfig material, or other secret values.

Hotfix To Source Rule

DEV runtime hotfixes are temporary recovery actions. After a hotfix proves the minimal live path, the durable follow-up is source automation: update manifests, contracts, docs, and tests so future rollouts reproduce the same env and safety boundaries. The runner must not re-execute the live hotfix, restart services, read Secrets, mutate PROD, or claim live evidence from source-only changes.

Durable Runtime Readiness Contract

DB live readiness and durable runtime readiness are separate gates. A /health/live payload with db.connected=true and db.liveDbEvidence=true only proves that the Cloud API reached the DEV DB endpoint without exposing secret values. It does not prove that runtime writes are persisted through a durable store.

When health reports:

• runtime.adapter: "memory"
• runtime.durable: false
• runtime.status: "degraded"

the Cloud API is using the process-local runtime store in internal/db/runtime-store.mjs. Gateway sessions, box resources, operations, audit events, and evidence records accepted through the runtime can be lost on pod restart, redeploy, or scale-out, and cannot be treated as durable M3/M4/M5 evidence. Users may see accepted operations or evidence disappear after a runtime replacement, and agent-loop/MVP acceptance must remain degraded or blocked even if the DB connection layer is green.

Readiness reports must keep these dimensions distinct:

Dimension	Green condition	Degraded/blocking condition
DB live	db.ready=true, db.connected=true, db.liveDbEvidence=true	Missing env, failed connection, disabled probe, or no liveDbEvidence
Runtime durability	runtime.adapter="postgres", runtime.durable=true, runtime.ready=true, runtime.liveRuntimeEvidence=true, and runtime auth/schema/migration/read gates are all ready	runtime.adapter="memory", runtime.durable=false, missing migration ledger, failed schema/auth/readiness, or no liveRuntimeEvidence

Do not declare DEV-LIVE complete while runtime.adapter="memory" or runtime.durable=false, even when DB live readiness is connected. A selected Postgres adapter may report runtime.durableRequested=true, but it still stays non-durable until the schema, migration ledger, and read readiness gates are proven through the configured adapter.

HWLAB_CLOUD_DB_SSL_MODE is the non-secret runtime authority for Postgres TLS mode. When DEV sets it to disable, the runtime strips stale URL sslmode, ssl, and certificate query parameters before constructing the pg pool; a Secret URL query parameter must not silently override the manifest SSL contract.

/health and /health/live expose this split in readiness.durability. dbLiveEvidenceObserved=true with dbLiveEvidenceIsDurabilityEvidence=false means the DB TCP/live layer is separate from runtime persistence. If runtime durability is blocked, blockedLayer names the current adapter/ssl/auth/schema/migration/durability query layer, and requiredEvidence remains the durable adapter schema, migration, and read query contract.

When the active blocker is runtime_durable_adapter_query_blocked, DB connectivity is live but the durable runtime adapter has not completed the required read queries through the runtime store. Future reports must call this a runtime durability blocker, not a DB connectivity blocker, and must keep M3, M4, and M5 full acceptance blocked until the durable runtime postconditions below are true.

DEV DB Provisioning Automation

The repo-owned DEV DB provisioning entrypoint is:

node scripts/dev-runtime-provisioning.mjs --check
node scripts/dev-runtime-provisioning.mjs --dry-run --allow-live-db-read --confirm-dev --write-report

The default --check path validates the redacted target DB URL contract only: the application role name, target database name, password presence, SecretRef names, and endpoint authority are represented as booleans or redacted endpoint class. It never connects to Postgres, never reads Kubernetes Secret data, and never prints the target role, database name, host, password, token, DSN, or kubeconfig material.

An authorized DEV provisioning apply is:

node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --write-report

The apply path uses HWLAB_CLOUD_DB_URL from hwlab-cloud-api-dev-db/database-url as the redacted target contract and HWLAB_CLOUD_DB_ADMIN_URL from hwlab-cloud-api-dev-db-admin/admin-url as the admin SecretRef env. It may create or update only the DEV target role, target database, database CONNECT grant, and public schema USAGE, CREATE grants needed by the cloud-api durable runtime. It must report created or existed booleans and structured blockers such as role missing, database missing, SSL, auth, schema, migration, or durability readiness. It must not read Secret objects, print Secret values, run PROD changes, use manual psql, or perform schema migration.

Runtime Migration Automation

The repo-owned runtime migration entrypoint is:

node scripts/dev-runtime-migration.mjs --check
node scripts/dev-runtime-migration.mjs --dry-run --write-report

Those default paths are source-only. They validate internal/db/migrations/0001_cloud_core_skeleton.sql, the required hwlab_schema_migrations ledger row, and the durable runtime table/column contract without connecting to Postgres, reading Kubernetes Secrets, or writing DEV/PROD state.

An authorized operator may run a read-only live verification only after injecting the DEV DB URL through the existing Secret/env path:

node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --write-report

A real DEV migration apply is intentionally separate and requires all explicit write flags:

node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report

The apply command is scoped to DEV Postgres schema objects declared by internal/db/migrations/0001_cloud_core_skeleton.sql and the hwlab_schema_migrations ledger row. It must not be run for PROD, must not read Kubernetes Secret resources, must not print HWLAB_CLOUD_DB_URL or any password/token value, and must not be described as M3/M4/M5 acceptance.

Runtime migration reports separate the blockers as:

Gate	Meaning
Auth	DB driver/auth/authorization can run the readiness query.
Schema	Required durable runtime tables and columns are present.
Migration	The required source migration is recorded in the ledger.
Readiness	Durable runtime read readiness passed after schema and ledger checks.

Durable Runtime Postflight

The repo-owned postflight entrypoint is:

node scripts/dev-runtime-postflight.mjs --check
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --write-report

Default --check is source-only. Live mode first reads http://74.48.78.17:16667/health/live and http://74.48.78.17:16667/v1. It only performs the M3 DO1=true -> DI1=true -> DO1=false -> DI1=false postflight when both endpoints prove runtime.adapter="postgres", runtime.durable=true, runtime.ready=true, and runtime.liveRuntimeEvidence=true. The M3 portion must then prove operation, trace, audit, and evidence identifiers with durable green evidence; otherwise the report remains blocked.

Durable Runtime Unblock Runbook

This runbook is a reusable checklist for a future authorized live mutation/migration. It does not itself authorize a live write. Runners working from source must stop at the read-only rows unless the commander explicitly authorizes the DEV apply command and the preconditions below are already true.

Check	Safe for normal runners	Evidence allowed	Not allowed
Source migration contract	Yes	node scripts/dev-runtime-migration.mjs --check and node scripts/dev-runtime-migration.mjs --dry-run --write-report	DB connection, Secret read, DEV/PROD write
Static runtime boundary contract	Yes	node scripts/validate-runtime-boundary.mjs, source docs, manifests, and redacted Secret refs	Live mutation, service restart, Secret value output
Public health observation	Yes, when command scope is read-only	/health/live fields such as db.liveDbEvidence, runtime.adapter, runtime.durable, runtime.blocker, readiness.durability.blockedLayer, requiredEvidence, and redaction/safety flags	Printing DB URLs, passwords, tokens, kubeconfig material, or Secret data
Live provisioning apply	No, unless explicitly authorized by commander/operator	node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --write-report with role/database created/existed booleans and no secret output	PROD apply, Secret value output, manual DB shell, schema migration, service restart, M3 acceptance promotion
Live DB read verification	Only with explicit DEV read authorization	--dry-run --allow-live-db-read --confirm-dev --write-report output with redacted endpoint and secretValuesPrinted=false	Kubernetes Secret reads, writes, migration apply, PROD target
Live migration/repair apply	No, unless explicitly authorized by commander/operator	--apply --confirm-dev --confirmed-non-production --write-report plus post-apply health evidence	PROD apply, Secret value output, service restart, Code Agent POST, hardware write, or acceptance promotion
Durable runtime postflight	Only after durable runtime readiness is already green	node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --write-report with /health/live, /v1, and M3 true/false durable green evidence	Running M3 writes while runtime durable readiness is blocked, Secret output, PROD, or manual rollout

Secret handling rules are the same for every row: verify env names, secretKeyRef names/keys, configured provider/model/base URL, redacted endpoint source, readiness booleans, blocker codes, and secretValuesPrinted=false. Do not run kubectl get secret, do not decode Secret data, and do not paste HWLAB_CLOUD_DB_URL, OPENAI_API_KEY, bearer tokens, kubeconfig content, or passwords into reports, PRs, issues, logs, screenshots, or chat.

Before any authorized live migration or repair, all of these preconditions must be true:

• The commander has explicitly authorized a DEV-only migration/repair for this blocker; no PROD target, deploy, restart, or unrelated smoke is bundled into the same action.
• Source validation passes for internal/db/migrations/0001_cloud_core_skeleton.sql, the durable runtime tables/columns, and the hwlab_schema_migrations ledger row.
• DEV DB provisioning is ready: target role and database are either already present or created by scripts/dev-runtime-provisioning.mjs, and the report carries only created/existed booleans plus redacted endpoint class.
• The runtime is requesting the Postgres durable adapter, not silently using memory: runtime.adapter="postgres" or equivalent selected-adapter evidence, with runtime.durableRequested=true.
• DB live evidence is already green through the redacted Secret URL host: db.liveDbEvidence=true and db.endpointSource="secret-url-host".
• The current blocker is recorded exactly, for example runtime_durable_adapter_query_blocked, with blockedLayer and requiredEvidence preserved in the report.
• The runner/operator can produce a redacted report that states secretValuesPrinted=false and does not require reading Kubernetes Secret resources.

After an authorized migration/repair, the blocker has moved or cleared only when a new read-only health/report observation proves one of these exact postconditions:

• Cleared: runtime.adapter="postgres", runtime.durable=true, runtime.ready=true, runtime.liveRuntimeEvidence=true, readiness.durability.ready=true, no runtime_durable_adapter_query_blocked, and durable auth/schema/migration/read gates are all ready.
• Moved: the old runtime_durable_adapter_query_blocked code is absent and a different explicit blocker is present, such as auth, schema, migration, or driver readiness; the report must name the new blocker and keep acceptance blocked.
• Still blocked: runtime_durable_adapter_query_blocked remains present, runtime.durable=false, or runtime.liveRuntimeEvidence=false; no full M3/M4/M5 acceptance may be claimed.

No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked. DB live readiness, Code Agent readiness, source migration readiness, and read-only health observations are supporting evidence only; they do not prove durable runtime evidence or the M3 trusted hardware loop.

Runtime Substitution Ban

UniDesk services, provider-gateway, backend-core, microservice proxies, local fixtures, and runner-local mocks may support scheduling, CI, CD, source checks, or dry-runs. They cannot be described as HWLAB runtime and cannot satisfy M3, M4, or M5 live evidence.

Stable Sources

• deploy/frp/README.md: FRP DEV contract.
• docs/d601-k3s-readonly-observability.md: read-only k3s visibility command.
• docs/dev-acceptance-matrix.md: DEV endpoint and health acceptance.
• pikasTech/HWLAB#164: Code Agent provider and DB live hotfix follow-up contract.
• pikasTech/HWLAB#61: manual rollout review and automation requirements.