# HWLAB DEV Runtime Boundary

This reference defines the stable DEV environment, port, k3s, and runtime
substitution rules.

## Public DEV Ports

| Surface | URL |
| --- | --- |
| Cloud Web browser entry | `http://74.48.78.17:16666/` |
| API/edge entry | `http://74.48.78.17:16667/` |
| API/live health | `http://74.48.78.17:16667/health/live` |

Public `:6666` and `:6667` are deprecated as browser/API acceptance targets.
Internal k3s services may still listen on `6667`; do not rewrite that as a
public endpoint.

## Route Shape

The public route is:

```text
master hwlab-frps-dev :16666/:16667
-> D601 hwlab-frpc in namespace hwlab-dev
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health
```

`deploy/deploy.json` is the source for public endpoint and FRP mapping drift
checks. Use:

```sh
node scripts/deploy-contract-plan.mjs --check
```

## D601 k3s Source Of Truth

D601 native k3s is the DEV runtime source of truth. Commands that observe or
roll a DEV workload must specify:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy,svc,pod -o wide
```

Do not trust the default `kubectl` context as a DEV runtime signal. On D601 it
may point at another cluster such as `docker-desktop`. The 2026-05-23 control
plane incident is tracked in
[pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138), with
hotfix context in
[pikasTech/unidesk#118](https://github.com/pikasTech/unidesk/issues/118):
Docker Desktop Kubernetes has been disabled and cleaned, but the default
`/home/ubuntu/.kube/config` may still point at the dead
`docker-desktop`/`127.0.0.1:11700` entry. Treat that as residual bad config, not
as DEV runtime evidence.

Before any DEV deploy, rollback, smoke, or manual k3s observation, confirm the
native node explicitly:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes -o jsonpath='{.items[*].metadata.name}'
```

The result must contain `d601`. Seeing `desktop-control-plane`,
`docker-desktop`, or `127.0.0.1:11700` means the command is on the wrong
control-plane path and must stop before mutation or acceptance.

Use the read-only preflight when a runner needs to report k3s visibility:

```sh
node scripts/d601-k3s-readonly-observability.mjs
```

The preflight must not print secret values, kubeconfig material, ConfigMap
values, or token content.

## Cloud API DB Readiness Authority

`hwlab-cloud-api` reads the DEV database connection string from Secret
reference `hwlab-cloud-api-dev-db/database-url`. The Secret value is not stored
in source, not printed by reports, and not required for offline validation.

Runtime DB readiness uses the redacted host parsed from the injected
`HWLAB_CLOUD_DB_URL` Secret value:

| Field | Value |
| --- | --- |
| Authority source | `secret-url-host` |
| Authority env | `HWLAB_CLOUD_DB_URL` |
| Required for readiness | `true` |
| Probe target | redacted host and port parsed from the Secret URL |

Health output may report `endpointSource: secret-url-host`, DNS/TCP result,
`liveConnected`, and `liveDbEvidence`, but must keep the DB endpoint and Secret
value redacted.

The non-secret `cloud-api-db` name is an optional desired alias:

| Field | Value |
| --- | --- |
| Source | `optional-public-dns-alias` |
| Service | `cloud-api-db` |
| Namespace | `hwlab-dev` |
| Host | `cloud-api-db.hwlab-dev.svc.cluster.local` |
| Port | `5432` |
| Port name | `postgres` |
| Required for readiness | `false` |
| Used for probe | `false` |

Missing `Service/cloud-api-db`, Endpoint, EndpointSlice, or alias env vars must
not reverse-fail live DB success from the Secret URL host. A future PR may make
that alias operational only if it also owns the Service plus Endpoint or
EndpointSlice manifests and the rollout/apply contract.

Source validation checks `HWLAB_CLOUD_DB_URL`, `HWLAB_CLOUD_DB_SSL_MODE`,
`endpointSource=secret-url-host`, and `liveDbEvidence=true` with secret values
redacted. It must not require `HWLAB_CLOUD_DB_SERVICE_NAME`,
`HWLAB_CLOUD_DB_SERVICE_NAMESPACE`, `HWLAB_CLOUD_DB_HOST`, or
`HWLAB_CLOUD_DB_PORT` as current readiness inputs.

The DB readiness contract is layered:

| Layer | Required evidence |
| --- | --- |
| DNS | The configured target resolves from the `hwlab-cloud-api` runtime path and is not a `.invalid` placeholder. |
| TCP | A redacted TCP probe reaches the configured Postgres port. |
| Auth | Authenticated database access succeeds without printing the connection string. |
| Schema | Required HWLAB schema/migration checks pass without using fixture output as live evidence. |

`*.invalid` and `hwlab-dev-db.invalid` are forbidden DEV runtime targets. They
may appear only as negative test fixtures; source and runtime health must not
treat them as desired DEV DB endpoints.

## Code Agent Provider Contract

`hwlab-cloud-api` runs the DEV Code Agent provider through repo-owned Codex MCP
stdio sessions. Source-controlled manifests must declare only env names, Secret
references, and non-secret egress/runtime settings:

| Field | Value |
| --- | --- |
| Provider env | `HWLAB_CODE_AGENT_PROVIDER=codex-stdio` |
| Model env | `HWLAB_CODE_AGENT_MODEL=gpt-5.5` |
| Provider Secret | `OPENAI_API_KEY` from `hwlab-code-agent-provider/openai-api-key` |
| DEV egress/base URL | `HWLAB_CODE_AGENT_OPENAI_BASE_URL=http://172.26.26.227:17680/v1/responses` |
| Workspace | `HWLAB_CODE_AGENT_CODEX_WORKSPACE=/workspace/hwlab` with `workspace-write` sandbox |
| Codex state | `CODEX_HOME=/codex-home` |
| Codex command | `HWLAB_CODE_AGENT_CODEX_COMMAND=/app/node_modules/.bin/codex` |

DEV pods must not call `https://api.openai.com/v1/responses` directly. The base
URL must use the approved DEV egress/proxy path so provider reachability is a
repeatable deployment contract instead of a one-off runtime patch.

Reports, smokes, and health payloads may show Secret name/key presence,
`missingEnv`, provider status, model, trace IDs, and redacted egress status.
They must never print the OpenAI API key, bearer token, database URL password,
kubeconfig material, or other secret values.

## Hotfix To Source Rule

DEV runtime hotfixes are temporary recovery actions. After a hotfix proves the
minimal live path, the durable follow-up is source automation: update manifests,
contracts, docs, and tests so future rollouts reproduce the same env and safety
boundaries. The runner must not re-execute the live hotfix, restart services,
read Secrets, mutate PROD, or claim live evidence from source-only changes.

## Durable Runtime Readiness Contract

DB live readiness and durable runtime readiness are separate gates. A
`/health/live` payload with `db.connected=true` and `db.liveDbEvidence=true`
only proves that the Cloud API reached the DEV DB endpoint without exposing
secret values. It does not prove that runtime writes are persisted through a
durable store.

When health reports:

- `runtime.adapter: "memory"`
- `runtime.durable: false`
- `runtime.status: "degraded"`

the Cloud API is using the process-local runtime store in
`internal/db/runtime-store.mjs`. Gateway sessions, box resources, operations,
audit events, and evidence records accepted through the runtime can be lost on
pod restart, redeploy, or scale-out, and cannot be treated as durable M3/M4/M5
evidence. Users may see accepted operations or evidence disappear after a
runtime replacement, and agent-loop/MVP acceptance must remain degraded or
blocked even if the DB connection layer is green.

Readiness reports must keep these dimensions distinct:

| Dimension | Green condition | Degraded/blocking condition |
| --- | --- | --- |
| DB live | `db.ready=true`, `db.connected=true`, `db.liveDbEvidence=true` | Missing env, failed connection, disabled probe, or no `liveDbEvidence` |
| Runtime durability | `runtime.adapter="postgres"`, `runtime.durable=true`, `runtime.ready=true`, `runtime.liveRuntimeEvidence=true`, and runtime auth/schema/migration/read gates are all ready | `runtime.adapter="memory"`, `runtime.durable=false`, missing migration ledger, failed schema/auth/readiness, or no `liveRuntimeEvidence` |

Do not declare DEV-LIVE complete while `runtime.adapter="memory"` or
`runtime.durable=false`, even when DB live readiness is connected. A selected
Postgres adapter may report `runtime.durableRequested=true`, but it still stays
non-durable until the schema, migration ledger, and read readiness gates are
proven through the configured adapter.

`HWLAB_CLOUD_DB_SSL_MODE` is the non-secret runtime authority for Postgres TLS
mode. When DEV sets it to `disable`, the runtime strips stale URL `sslmode`,
`ssl`, and certificate query parameters before constructing the `pg` pool; a
Secret URL query parameter must not silently override the manifest SSL contract.

`/health` and `/health/live` expose this split in `readiness.durability`.
`dbLiveEvidenceObserved=true` with
`dbLiveEvidenceIsDurabilityEvidence=false` means the DB TCP/live layer is
separate from runtime persistence. If runtime durability is blocked,
`blockedLayer` names the current adapter/ssl/auth/schema/migration/durability query
layer, and `requiredEvidence` remains the durable adapter schema, migration,
and read query contract.

When the active blocker is `runtime_durable_adapter_query_blocked`, DB
connectivity is live but the durable runtime adapter has not completed the
required read queries through the runtime store. Future reports must call this a
runtime durability blocker, not a DB connectivity blocker, and must keep M3,
M4, and M5 full acceptance blocked until the durable runtime postconditions
below are true.

## DEV DB Provisioning Automation

The repo-owned DEV DB provisioning entrypoint is:

```sh
node scripts/dev-runtime-provisioning.mjs --check
node scripts/dev-runtime-provisioning.mjs --dry-run --allow-live-db-read --confirm-dev --write-report
```

The default `--check` path validates the redacted target DB URL contract only:
the application role name, target database name, password presence, SecretRef
names, and endpoint authority are represented as booleans or redacted endpoint
class. It never connects to Postgres, never reads Kubernetes Secret data, and
never prints the target role, database name, host, password, token, DSN, or
kubeconfig material.

An authorized DEV provisioning apply is:

```sh
node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

The apply path uses `HWLAB_CLOUD_DB_URL` from
`hwlab-cloud-api-dev-db/database-url` as the redacted target contract and
`HWLAB_CLOUD_DB_ADMIN_URL` from
`hwlab-cloud-api-dev-db-admin/admin-url` as the admin SecretRef env. It may
create or update only the DEV target role, target database, database `CONNECT`
grant, and `public` schema `USAGE, CREATE` grants needed by the cloud-api
durable runtime. It must report `created` or `existed` booleans and structured
blockers such as role missing, database missing, SSL, auth, schema, migration,
or durability readiness. It must not read Secret objects, print Secret values,
run PROD changes, use manual `psql`, or perform schema migration.

Provisioning reports intentionally keep target role, database, host, password,
and admin DSN out of the output. `role missing` and `database missing` are
reported as separate blockers so an auth failure cannot be repaired with an
untracked manual DB write.

## Runtime Migration Automation

The repo-owned runtime migration entrypoint is:

```sh
node scripts/dev-runtime-migration.mjs --check
node scripts/dev-runtime-migration.mjs --dry-run --write-report
```

Those default paths are source-only. They validate
`internal/db/migrations/0001_cloud_core_skeleton.sql`, the required
`hwlab_schema_migrations` ledger row, and the durable runtime table/column
contract without connecting to Postgres, reading Kubernetes Secrets, or writing
DEV/PROD state.

An authorized operator may run a read-only live verification only after
injecting the DEV DB URL through the existing Secret/env path:

```sh
node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --write-report
```

A real DEV migration apply is intentionally separate and requires all explicit
write flags:

```sh
node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

The apply command is scoped to DEV Postgres schema objects declared by
`internal/db/migrations/0001_cloud_core_skeleton.sql` and the
`hwlab_schema_migrations` ledger row. It must not be run for PROD, must not
read Kubernetes Secret resources, must not print `HWLAB_CLOUD_DB_URL` or any
password/token value, and must not be described as M3/M4/M5 acceptance.

Runtime migration reports separate the blockers as:

| Gate | Meaning |
| --- | --- |
| Auth | DB driver/auth/authorization can run the readiness query. |
| Schema | Required durable runtime tables and columns are present. |
| Migration | The required source migration is recorded in the ledger. |
| Readiness | Durable runtime read readiness passed after schema and ledger checks. |

## Durable Runtime Postflight

The repo-owned postflight entrypoint is:

```sh
node scripts/dev-runtime-postflight.mjs --check
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --write-report
```

Default `--check` is source-only. Live mode first reads
`http://74.48.78.17:16667/health/live` and
`http://74.48.78.17:16667/v1`. `/v1` must expose only the controlled
same-origin M3 IO route `/v1/m3/io` with contract `m3-io-control-v1`; generic
frontend gateway/box/patch-panel access is not a valid postflight substitute.
The postflight only performs the M3
`DO1=true -> DI1=true -> DO1=false -> DI1=false` write/read sequence when both
endpoints prove `runtime.adapter="postgres"`, `runtime.durable=true`,
`runtime.ready=true`, and `runtime.liveRuntimeEvidence=true`.

The M3 portion must then prove all four expected operations, exact true/false
values, operation/trace/audit/evidence identifiers, `evidenceState.status="green"`,
`evidenceState.sourceKind="DEV-LIVE"`, `evidenceState.durable=true`, and
`evidenceState.writeStatus="persisted"`. DB SecretRef presence, TCP
connectivity, `db.connected=true`, or `db.liveDbEvidence=true` alone must leave
the postflight blocked with the durable runtime blocker preserved.

## Durable Runtime Unblock Runbook

This runbook is a reusable checklist for a future authorized live
mutation/migration. It does not itself authorize a live write. Runners working
from source must stop at the read-only rows unless the commander explicitly
authorizes the DEV apply command and the preconditions below are already true.

| Check | Safe for normal runners | Evidence allowed | Not allowed |
| --- | --- | --- | --- |
| Source migration contract | Yes | `node scripts/dev-runtime-migration.mjs --check` and `node scripts/dev-runtime-migration.mjs --dry-run --write-report` | DB connection, Secret read, DEV/PROD write |
| Static runtime boundary contract | Yes | `node scripts/validate-runtime-boundary.mjs`, source docs, manifests, and redacted Secret refs | Live mutation, service restart, Secret value output |
| Public health observation | Yes, when command scope is read-only | `/health/live` fields such as `db.liveDbEvidence`, `runtime.adapter`, `runtime.durable`, `runtime.blocker`, `readiness.durability.blockedLayer`, `requiredEvidence`, and redaction/safety flags | Printing DB URLs, passwords, tokens, kubeconfig material, or Secret data |
| Live provisioning apply | No, unless explicitly authorized by commander/operator | `node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --write-report` with role/database created/existed booleans and no secret output | PROD apply, Secret value output, manual DB shell, schema migration, service restart, M3 acceptance promotion |
| Live DB read verification | Only with explicit DEV read authorization | `--dry-run --allow-live-db-read --confirm-dev --write-report` output with redacted endpoint and `secretValuesPrinted=false` | Kubernetes Secret reads, writes, migration apply, PROD target |
| Live migration/repair apply | No, unless explicitly authorized by commander/operator | `--apply --confirm-dev --confirmed-non-production --write-report` plus post-apply health evidence | PROD apply, Secret value output, service restart, Code Agent POST, hardware write, or acceptance promotion |
| Durable runtime postflight | Only after durable runtime readiness is already green | `node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --write-report` with `/health/live`, `/v1`, and M3 true/false durable green evidence | Running M3 writes while runtime durable readiness is blocked, Secret output, PROD, or manual rollout |

Secret handling rules are the same for every row: verify env names,
`secretKeyRef` names/keys, configured provider/model/base URL, redacted endpoint
source, readiness booleans, blocker codes, and `secretValuesPrinted=false`.
Do not run `kubectl get secret`, do not decode Secret data, and do not paste
`HWLAB_CLOUD_DB_URL`, `OPENAI_API_KEY`, bearer tokens, kubeconfig content, or
passwords into reports, PRs, issues, logs, screenshots, or chat.

Before any authorized live migration or repair, all of these preconditions must
be true:

- The commander has explicitly authorized a DEV-only migration/repair for this
  blocker; no PROD target, deploy, restart, or unrelated smoke is bundled into
  the same action.
- Source validation passes for `internal/db/migrations/0001_cloud_core_skeleton.sql`,
  the durable runtime tables/columns, and the `hwlab_schema_migrations` ledger
  row.
- DEV DB provisioning is ready: target role and database are either already
  present or created by `scripts/dev-runtime-provisioning.mjs`, and the report
  carries only created/existed booleans plus redacted endpoint class.
- The runtime is requesting the Postgres durable adapter, not silently using
  memory: `runtime.adapter="postgres"` or equivalent selected-adapter evidence,
  with `runtime.durableRequested=true`.
- DB live evidence is already green through the redacted Secret URL host:
  `db.liveDbEvidence=true` and `db.endpointSource="secret-url-host"`.
- The current blocker is recorded exactly, for example
  `runtime_durable_adapter_query_blocked`, with `blockedLayer` and
  `requiredEvidence` preserved in the report.
- The runner/operator can produce a redacted report that states
  `secretValuesPrinted=false` and does not require reading Kubernetes Secret
  resources.

After an authorized migration/repair, the blocker has moved or cleared only
when a new read-only health/report observation proves one of these exact
postconditions:

- Cleared: `runtime.adapter="postgres"`, `runtime.durable=true`,
  `runtime.ready=true`, `runtime.liveRuntimeEvidence=true`,
  `readiness.durability.ready=true`, no
  `runtime_durable_adapter_query_blocked`, and durable auth/schema/migration/read
  gates are all ready.
- Moved: the old `runtime_durable_adapter_query_blocked` code is absent and a
  different explicit blocker is present, such as auth, schema, migration, or
  driver readiness; the report must name the new blocker and keep acceptance
  blocked.
- Still blocked: `runtime_durable_adapter_query_blocked` remains present,
  `runtime.durable=false`, or `runtime.liveRuntimeEvidence=false`; no full
  M3/M4/M5 acceptance may be claimed.

No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked.
DB live readiness, Code Agent readiness, source migration readiness, and
read-only health observations are supporting evidence only; they do not prove
durable runtime evidence or the M3 trusted hardware loop.

## Runtime Substitution Ban

UniDesk services, provider-gateway, backend-core, microservice proxies, local
fixtures, and runner-local mocks may support scheduling, CI, CD, source checks,
or dry-runs. They cannot be described as HWLAB runtime and cannot satisfy M3,
M4, or M5 live evidence.

## Stable Sources

- [deploy/frp/README.md](../../deploy/frp/README.md): FRP DEV contract.
- [docs/d601-k3s-readonly-observability.md](../d601-k3s-readonly-observability.md):
  read-only k3s visibility command.
- [docs/dev-acceptance-matrix.md](../dev-acceptance-matrix.md): DEV endpoint
  and health acceptance.
- [pikasTech/HWLAB#164](https://github.com/pikasTech/HWLAB/issues/164):
  Code Agent provider and DB live hotfix follow-up contract.
- [pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61): manual
  rollout review and automation requirements.