pikasTech-HWLAB/docs/reference/dev-runtime-boundary.md

# HWLAB DEV Runtime Boundary

This reference defines the stable DEV environment, port, k3s, and runtime
substitution rules.

## Public DEV Ports

| Surface | URL |
| --- | --- |
| Cloud Web browser entry | `http://74.48.78.17:16666/` |
| API/edge entry | `http://74.48.78.17:16667/` |
| API/live health | `http://74.48.78.17:16667/health/live` |

Public `:6666` and `:6667` are deprecated as browser/API acceptance targets.
Internal k3s services may still listen on `6667`; do not rewrite that as a
public endpoint.

## Route Shape

The public route is:

```text
master hwlab-frps-dev :16666/:16667
-> D601 hwlab-frpc in namespace hwlab-dev
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health
```

`deploy/deploy.json` is the source for public endpoint and FRP mapping drift
checks. Use:

```sh
node scripts/deploy-contract-plan.mjs --check
```

## D601 k3s Source Of Truth

D601 native k3s is the DEV runtime source of truth. Commands that observe or
roll a DEV workload must specify:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy,svc,pod -o wide
```

Do not trust the default `kubectl` context as a DEV runtime signal. On D601 it
may point at another cluster such as `docker-desktop`.

Use the read-only preflight when a runner needs to report k3s visibility:

```sh
node scripts/d601-k3s-readonly-observability.mjs
```

The preflight must not print secret values, kubeconfig material, ConfigMap
values, or token content.

## Cloud API DB Readiness Authority

`hwlab-cloud-api` reads the DEV database connection string from Secret
reference `hwlab-cloud-api-dev-db/database-url`. The Secret value is not stored
in source, not printed by reports, and not required for offline validation.

Runtime DB readiness uses the redacted host parsed from the injected
`HWLAB_CLOUD_DB_URL` Secret value:

| Field | Value |
| --- | --- |
| Authority source | `secret-url-host` |
| Authority env | `HWLAB_CLOUD_DB_URL` |
| Required for readiness | `true` |
| Probe target | redacted host and port parsed from the Secret URL |

Health output may report `endpointSource: secret-url-host`, DNS/TCP result,
`liveConnected`, and `liveDbEvidence`, but must keep the DB endpoint and Secret
value redacted.

The non-secret `cloud-api-db` name is an optional desired alias:

| Field | Value |
| --- | --- |
| Source | `optional-public-dns-alias` |
| Service | `cloud-api-db` |
| Namespace | `hwlab-dev` |
| Host | `cloud-api-db.hwlab-dev.svc.cluster.local` |
| Port | `5432` |
| Port name | `postgres` |
| Required for readiness | `false` |
| Used for probe | `false` |

Missing `Service/cloud-api-db`, Endpoint, EndpointSlice, or alias env vars must
not reverse-fail live DB success from the Secret URL host. A future PR may make
that alias operational only if it also owns the Service plus Endpoint or
EndpointSlice manifests and the rollout/apply contract.

Source validation checks `HWLAB_CLOUD_DB_URL`, `HWLAB_CLOUD_DB_SSL_MODE`,
`endpointSource=secret-url-host`, and `liveDbEvidence=true` with secret values
redacted. It must not require `HWLAB_CLOUD_DB_SERVICE_NAME`,
`HWLAB_CLOUD_DB_SERVICE_NAMESPACE`, `HWLAB_CLOUD_DB_HOST`, or
`HWLAB_CLOUD_DB_PORT` as current readiness inputs.

The DB readiness contract is layered:

| Layer | Required evidence |
| --- | --- |
| DNS | The configured target resolves from the `hwlab-cloud-api` runtime path and is not a `.invalid` placeholder. |
| TCP | A redacted TCP probe reaches the configured Postgres port. |
| Auth | Authenticated database access succeeds without printing the connection string. |
| Schema | Required HWLAB schema/migration checks pass without using fixture output as live evidence. |

`*.invalid` and `hwlab-dev-db.invalid` are forbidden DEV runtime targets. They
may appear only as negative test fixtures; source and runtime health must not
treat them as desired DEV DB endpoints.

## Code Agent Provider Contract

`hwlab-cloud-api` runs the DEV Code Agent provider through OpenAI Responses.
Source-controlled manifests must declare only env names, Secret references, and
non-secret egress settings:

| Field | Value |
| --- | --- |
| Provider env | `HWLAB_CODE_AGENT_PROVIDER=openai` |
| Model env | `HWLAB_CODE_AGENT_MODEL=gpt-5.5` |
| Provider Secret | `OPENAI_API_KEY` from `hwlab-code-agent-provider/openai-api-key` |
| DEV egress/base URL | `HWLAB_CODE_AGENT_OPENAI_BASE_URL=http://172.26.26.227:17680/v1/responses` |

DEV pods must not call `https://api.openai.com/v1/responses` directly. The base
URL must use the approved DEV egress/proxy path so provider reachability is a
repeatable deployment contract instead of a one-off runtime patch.

Reports, smokes, and health payloads may show Secret name/key presence,
`missingEnv`, provider status, model, trace IDs, and redacted egress status.
They must never print the OpenAI API key, bearer token, database URL password,
kubeconfig material, or other secret values.

## Hotfix To Source Rule

DEV runtime hotfixes are temporary recovery actions. After a hotfix proves the
minimal live path, the durable follow-up is source automation: update manifests,
contracts, docs, and tests so future rollouts reproduce the same env and safety
boundaries. The runner must not re-execute the live hotfix, restart services,
read Secrets, mutate PROD, or claim live evidence from source-only changes.

## Durable Runtime Readiness Contract

DB live readiness and durable runtime readiness are separate gates. A
`/health/live` payload with `db.connected=true` and `db.liveDbEvidence=true`
only proves that the Cloud API reached the DEV DB endpoint without exposing
secret values. It does not prove that runtime writes are persisted through a
durable store.

When health reports:

- `runtime.adapter: "memory"`
- `runtime.durable: false`
- `runtime.status: "degraded"`

the Cloud API is using the process-local runtime store in
`internal/db/runtime-store.mjs`. Gateway sessions, box resources, operations,
audit events, and evidence records accepted through the runtime can be lost on
pod restart, redeploy, or scale-out, and cannot be treated as durable M3/M4/M5
evidence. Users may see accepted operations or evidence disappear after a
runtime replacement, and agent-loop/MVP acceptance must remain degraded or
blocked even if the DB connection layer is green.

Readiness reports must keep these dimensions distinct:

| Dimension | Green condition | Degraded/blocking condition |
| --- | --- | --- |
| DB live | `db.ready=true`, `db.connected=true`, `db.liveDbEvidence=true` | Missing env, failed connection, disabled probe, or no `liveDbEvidence` |
| Runtime durability | `runtime.adapter="postgres"`, `runtime.durable=true`, `runtime.ready=true`, `runtime.liveRuntimeEvidence=true`, and runtime auth/schema/migration/read gates are all ready | `runtime.adapter="memory"`, `runtime.durable=false`, missing migration ledger, failed schema/auth/readiness, or no `liveRuntimeEvidence` |

Do not declare DEV-LIVE complete while `runtime.adapter="memory"` or
`runtime.durable=false`, even when DB live readiness is connected. A selected
Postgres adapter may report `runtime.durableRequested=true`, but it still stays
non-durable until the schema, migration ledger, and read readiness gates are
proven through the configured adapter.

`HWLAB_CLOUD_DB_SSL_MODE` is the non-secret runtime authority for Postgres TLS
mode. When DEV sets it to `disable`, the runtime strips stale URL `sslmode`,
`ssl`, and certificate query parameters before constructing the `pg` pool; a
Secret URL query parameter must not silently override the manifest SSL contract.

`/health` and `/health/live` expose this split in `readiness.durability`.
`dbLiveEvidenceObserved=true` with
`dbLiveEvidenceIsDurabilityEvidence=false` means the DB TCP/live layer is
separate from runtime persistence. If runtime durability is blocked,
`blockedLayer` names the current adapter/ssl/auth/schema/migration/durability query
layer, and `requiredEvidence` remains the durable adapter schema, migration,
and read query contract.

When the active blocker is `runtime_durable_adapter_query_blocked`, DB
connectivity is live but the durable runtime adapter has not completed the
required read queries through the runtime store. Future reports must call this a
runtime durability blocker, not a DB connectivity blocker, and must keep M3,
M4, and M5 full acceptance blocked until the durable runtime postconditions
below are true.

## Runtime Migration Automation

The repo-owned runtime migration entrypoint is:

```sh
node scripts/dev-runtime-migration.mjs --check
node scripts/dev-runtime-migration.mjs --dry-run --write-report
```

Those default paths are source-only. They validate
`internal/db/migrations/0001_cloud_core_skeleton.sql`, the required
`hwlab_schema_migrations` ledger row, and the durable runtime table/column
contract without connecting to Postgres, reading Kubernetes Secrets, or writing
DEV/PROD state.

An authorized operator may run a read-only live verification only after
injecting the DEV DB URL through the existing Secret/env path:

```sh
node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --write-report
```

A real DEV migration apply is intentionally separate and requires all explicit
write flags:

```sh
node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

The apply command is scoped to DEV Postgres schema objects declared by
`internal/db/migrations/0001_cloud_core_skeleton.sql` and the
`hwlab_schema_migrations` ledger row. It must not be run for PROD, must not
read Kubernetes Secret resources, must not print `HWLAB_CLOUD_DB_URL` or any
password/token value, and must not be described as M3/M4/M5 acceptance.

Runtime migration reports separate the blockers as:

| Gate | Meaning |
| --- | --- |
| Auth | DB driver/auth/authorization can run the readiness query. |
| Schema | Required durable runtime tables and columns are present. |
| Migration | The required source migration is recorded in the ledger. |
| Readiness | Durable runtime read readiness passed after schema and ledger checks. |

## Durable Runtime Unblock Runbook

This runbook is a reusable checklist for a future authorized live
mutation/migration. It does not itself authorize a live write. Runners working
from source must stop at the read-only rows unless the commander explicitly
authorizes the DEV apply command and the preconditions below are already true.

| Check | Safe for normal runners | Evidence allowed | Not allowed |
| --- | --- | --- | --- |
| Source migration contract | Yes | `node scripts/dev-runtime-migration.mjs --check` and `node scripts/dev-runtime-migration.mjs --dry-run --write-report` | DB connection, Secret read, DEV/PROD write |
| Static runtime boundary contract | Yes | `node scripts/validate-runtime-boundary.mjs`, source docs, manifests, and redacted Secret refs | Live mutation, service restart, Secret value output |
| Public health observation | Yes, when command scope is read-only | `/health/live` fields such as `db.liveDbEvidence`, `runtime.adapter`, `runtime.durable`, `runtime.blocker`, `readiness.durability.blockedLayer`, `requiredEvidence`, and redaction/safety flags | Printing DB URLs, passwords, tokens, kubeconfig material, or Secret data |
| Live DB read verification | Only with explicit DEV read authorization | `--dry-run --allow-live-db-read --confirm-dev --write-report` output with redacted endpoint and `secretValuesPrinted=false` | Kubernetes Secret reads, writes, migration apply, PROD target |
| Live migration/repair apply | No, unless explicitly authorized by commander/operator | `--apply --confirm-dev --confirmed-non-production --write-report` plus post-apply health evidence | PROD apply, Secret value output, service restart, Code Agent POST, hardware write, or acceptance promotion |

Secret handling rules are the same for every row: verify env names,
`secretKeyRef` names/keys, configured provider/model/base URL, redacted endpoint
source, readiness booleans, blocker codes, and `secretValuesPrinted=false`.
Do not run `kubectl get secret`, do not decode Secret data, and do not paste
`HWLAB_CLOUD_DB_URL`, `OPENAI_API_KEY`, bearer tokens, kubeconfig content, or
passwords into reports, PRs, issues, logs, screenshots, or chat.

Before any authorized live migration or repair, all of these preconditions must
be true:

- The commander has explicitly authorized a DEV-only migration/repair for this
  blocker; no PROD target, deploy, restart, or unrelated smoke is bundled into
  the same action.
- Source validation passes for `internal/db/migrations/0001_cloud_core_skeleton.sql`,
  the durable runtime tables/columns, and the `hwlab_schema_migrations` ledger
  row.
- The runtime is requesting the Postgres durable adapter, not silently using
  memory: `runtime.adapter="postgres"` or equivalent selected-adapter evidence,
  with `runtime.durableRequested=true`.
- DB live evidence is already green through the redacted Secret URL host:
  `db.liveDbEvidence=true` and `db.endpointSource="secret-url-host"`.
- The current blocker is recorded exactly, for example
  `runtime_durable_adapter_query_blocked`, with `blockedLayer` and
  `requiredEvidence` preserved in the report.
- The runner/operator can produce a redacted report that states
  `secretValuesPrinted=false` and does not require reading Kubernetes Secret
  resources.

After an authorized migration/repair, the blocker has moved or cleared only
when a new read-only health/report observation proves one of these exact
postconditions:

- Cleared: `runtime.adapter="postgres"`, `runtime.durable=true`,
  `runtime.ready=true`, `runtime.liveRuntimeEvidence=true`,
  `readiness.durability.ready=true`, no
  `runtime_durable_adapter_query_blocked`, and durable auth/schema/migration/read
  gates are all ready.
- Moved: the old `runtime_durable_adapter_query_blocked` code is absent and a
  different explicit blocker is present, such as auth, schema, migration, or
  driver readiness; the report must name the new blocker and keep acceptance
  blocked.
- Still blocked: `runtime_durable_adapter_query_blocked` remains present,
  `runtime.durable=false`, or `runtime.liveRuntimeEvidence=false`; no full
  M3/M4/M5 acceptance may be claimed.

No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked.
DB live readiness, Code Agent readiness, source migration readiness, and
read-only health observations are supporting evidence only; they do not prove
durable runtime evidence or the M3 trusted hardware loop.

## Runtime Substitution Ban

UniDesk services, provider-gateway, backend-core, microservice proxies, local
fixtures, and runner-local mocks may support scheduling, CI, CD, source checks,
or dry-runs. They cannot be described as HWLAB runtime and cannot satisfy M3,
M4, or M5 live evidence.

## Stable Sources

- [deploy/frp/README.md](../../deploy/frp/README.md): FRP DEV contract.
- [docs/d601-k3s-readonly-observability.md](../d601-k3s-readonly-observability.md):
  read-only k3s visibility command.
- [docs/dev-acceptance-matrix.md](../dev-acceptance-matrix.md): DEV endpoint
  and health acceptance.
- [pikasTech/HWLAB#164](https://github.com/pikasTech/HWLAB/issues/164):
  Code Agent provider and DB live hotfix follow-up contract.
- [pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61): manual
  rollout review and automation requirements.