pikasTech-HWLAB/docs/reference/dev-runtime-boundary.md

# HWLAB DEV Runtime Boundary

This reference defines the stable DEV environment, port, k3s, and runtime
substitution rules.

## Public G14 Runtime Ports

| Surface | DEV URL | PROD URL |
| --- | --- | --- |
| Cloud Web browser entry | `http://74.48.78.17:17666/` | `http://74.48.78.17:18666/` |
| API/edge entry | `http://74.48.78.17:17667/` | `http://74.48.78.17:18667/` |
| API/live health | `http://74.48.78.17:17667/health/live` | `http://74.48.78.17:18667/health/live` |

Public `:16666` and `:16667` are D601 legacy DEV endpoints. They can be used only for migration comparison or incident replay, not as the current HWLAB runtime acceptance target. Internal k3s services may still listen on `6667`; do not rewrite that as a public endpoint.

## Route Shape

The current G14 DEV route is:

```text
master hwlab-frps-dev :17666/:17667
-> G14 hwlab-g14-frpc in namespace hwlab-dev
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health

master hwlab-frps-dev :18666/:18667
-> G14 hwlab-g14-prod-frpc in namespace hwlab-prod
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health
```

G14 PROD is rendered by GitOps as namespace `hwlab-prod` and Argo CD application `hwlab-g14-prod`; it uses the `:18666/:18667` public endpoints when enabled. G14 desired state is generated into `deploy/gitops/g14/runtime-dev` and `deploy/gitops/g14/runtime-prod` and promoted through the `G14-gitops` branch.

## G14 k3s Source Of Truth

G14 native k3s is the HWLAB DEV/PROD runtime source of truth. Commands that observe or mutate current HWLAB runtime must go through UniDesk route `G14:k3s`, for example:

```sh
tran G14:k3s kubectl -n hwlab-dev get deploy,svc,pod -o wide
tran G14:k3s kubectl -n argocd get application hwlab-g14-dev hwlab-g14-prod
```

Do not use D601 kubeconfig, D601 `dev-cd-apply`, old JS `ci-publish`, Docker Desktop Kubernetes, or master-server local checks as G14 runtime acceptance evidence. D601 remains a legacy migration/reference surface until the branch-role migration is complete.

## Branch Role Migration

The target branch model is: G14 becomes the `main` runtime source branch, and the former D601-oriented `main` behavior moves to a D601 legacy/maintenance line. Until that branch governance operation is explicitly completed and verified, `/root/hwlab` on G14 continues to use the `G14` branch and must merge the latest `origin/main` business updates before G14-specific runtime work.

## Distributed Passthrough Hygiene

When running more than one command on G14, keep all commands inside one remote `script` block:

```sh
tran G14:/root/hwlab script <<'SCRIPT'
sed -n '1,80p' docs/reference/dev-runtime-boundary.md
sed -n '1,80p' AGENTS.md
SCRIPT
```

Do not write `tran G14:/root/hwlab sed ... && sed ...` unless the second command is intentionally local. Shell operators such as `&&`, `;`, and pipes outside the `tran` invocation are evaluated by the local shell; this can make later commands run on the master server and look like a missing-file or truncation bug.

## Cloud API DB Readiness Authority

`hwlab-cloud-api` reads the DEV database connection string from Secret
reference `hwlab-cloud-api-dev-db/database-url`. The Secret value is not stored
in source, not printed by reports, and not required for offline validation.

Runtime DB readiness uses the redacted host parsed from the injected
`HWLAB_CLOUD_DB_URL` Secret value:

| Field | Value |
| --- | --- |
| Authority source | `secret-url-host` |
| Authority env | `HWLAB_CLOUD_DB_URL` |
| Required for readiness | `true` |
| Probe target | redacted host and port parsed from the Secret URL |

Health output may report `endpointSource: secret-url-host`, DNS/TCP result,
`liveConnected`, and `liveDbEvidence`, but must keep the DB endpoint and Secret
value redacted.

The non-secret `cloud-api-db` name is an optional desired alias:

| Field | Value |
| --- | --- |
| Source | `optional-public-dns-alias` |
| Service | `cloud-api-db` |
| Namespace | `hwlab-dev` |
| Host | `cloud-api-db.hwlab-dev.svc.cluster.local` |
| Port | `5432` |
| Port name | `postgres` |
| Required for readiness | `false` |
| Used for probe | `false` |

Missing `Service/cloud-api-db`, Endpoint, EndpointSlice, or alias env vars must
not reverse-fail live DB success from the Secret URL host. A future PR may make
that alias operational only if it also owns the Service plus Endpoint or
EndpointSlice manifests and the rollout/apply contract.

Source validation checks `HWLAB_CLOUD_DB_URL`, `HWLAB_CLOUD_DB_SSL_MODE`,
`endpointSource=secret-url-host`, and `liveDbEvidence=true` with secret values
redacted. It must not require `HWLAB_CLOUD_DB_SERVICE_NAME`,
`HWLAB_CLOUD_DB_SERVICE_NAMESPACE`, `HWLAB_CLOUD_DB_HOST`, or
`HWLAB_CLOUD_DB_PORT` as current readiness inputs.

The DB readiness contract is layered:

| Layer | Required evidence |
| --- | --- |
| DNS | The configured target resolves from the `hwlab-cloud-api` runtime path and is not a `.invalid` placeholder. |
| TCP | A redacted TCP probe reaches the configured Postgres port. |
| Auth | Authenticated database access succeeds without printing the connection string. |
| Schema | Required HWLAB schema/migration checks pass without using fixture output as live evidence. |

`*.invalid` and `hwlab-dev-db.invalid` are forbidden DEV runtime targets. They
may appear only as negative test fixtures; source and runtime health must not
treat them as desired DEV DB endpoints.

## Code Agent Provider Contract

`hwlab-cloud-api` runs the DEV Code Agent provider through repo-owned Codex MCP
stdio sessions. Source-controlled manifests must declare only env names, Secret
references, and non-secret egress/runtime settings:

| Field | Value |
| --- | --- |
| Provider env | `HWLAB_CODE_AGENT_PROVIDER=codex-stdio` |
| Model env | `HWLAB_CODE_AGENT_MODEL=gpt-5.5` |
| Provider Secret | `OPENAI_API_KEY` from `hwlab-code-agent-provider/openai-api-key` |
| DEV egress/base URL | `HWLAB_CODE_AGENT_OPENAI_BASE_URL=http://172.26.26.227:17680/v1/responses` |
| Workspace | `HWLAB_CODE_AGENT_CODEX_WORKSPACE=/workspace/hwlab` with `workspace-write` sandbox |
| Codex state | `CODEX_HOME=/codex-home` |
| Codex command | `HWLAB_CODE_AGENT_CODEX_COMMAND=/app/node_modules/.bin/codex` |

DEV pods must not call `https://api.openai.com/v1/responses` directly. The base
URL must use the approved DEV egress/proxy path so provider reachability is a
repeatable deployment contract instead of a one-off runtime patch.

Reports, smokes, and health payloads may show Secret name/key presence,
`missingEnv`, provider status, model, trace IDs, and redacted egress status.
They must never print the OpenAI API key, bearer token, database URL password,
kubeconfig material, or other secret values.

## Hotfix To Source Rule

DEV runtime hotfixes are temporary recovery actions. After a hotfix proves the
minimal live path, the durable follow-up is source automation: update manifests,
contracts, docs, and tests so future rollouts reproduce the same env and safety
boundaries. The runner must not re-execute the live hotfix, restart services,
read Secrets, mutate PROD, or claim live evidence from source-only changes.

DEV pod/Deployment file overrides, ConfigMap mounts, and pod-local edits are
governed by [dev-runtime-hotfix-runbook.md](dev-runtime-hotfix-runbook.md). The
stable audit entry is:

```sh
node scripts/dev-runtime-hotfix-audit.mjs --pretty
node scripts/dev-runtime-hotfix-audit.mjs --collect-readonly --pretty
```

The default mode only prints a read-only plan. `--collect-readonly` may run
`kubectl get` and `kubectl exec` checks with
`KUBECONFIG=/etc/rancher/k3s/k3s.yaml`; it must not apply, patch, rollout,
restart, delete, create, read Secret resources, or print secret values.

## Durable Runtime Readiness Contract

DB live readiness and durable runtime readiness are separate gates. A
`/health/live` payload with `db.connected=true` and `db.liveDbEvidence=true`
only proves that the Cloud API reached the DEV DB endpoint without exposing
secret values. It does not prove that runtime writes are persisted through a
durable store.

When health reports:

- `runtime.adapter: "memory"`
- `runtime.durable: false`
- `runtime.status: "degraded"`

the Cloud API is using the process-local runtime store in
`internal/db/runtime-store.mjs`. Gateway sessions, box resources, operations,
audit events, and evidence records accepted through the runtime can be lost on
pod restart, redeploy, or scale-out, and cannot be treated as durable M3/M4/M5
evidence. Users may see accepted operations or evidence disappear after a
runtime replacement, and agent-loop/MVP acceptance must remain degraded or
blocked even if the DB connection layer is green.

Readiness reports must keep these dimensions distinct:

| Dimension | Green condition | Degraded/blocking condition |
| --- | --- | --- |
| DB live | `db.ready=true`, `db.connected=true`, `db.liveDbEvidence=true` | Missing env, failed connection, disabled probe, or no `liveDbEvidence` |
| Runtime durability | `runtime.adapter="postgres"`, `runtime.durable=true`, `runtime.ready=true`, `runtime.liveRuntimeEvidence=true`, and runtime auth/schema/migration/read gates are all ready | `runtime.adapter="memory"`, `runtime.durable=false`, missing migration ledger, failed schema/auth/readiness, or no `liveRuntimeEvidence` |

Do not declare DEV-LIVE complete while `runtime.adapter="memory"` or
`runtime.durable=false`, even when DB live readiness is connected. A selected
Postgres adapter may report `runtime.durableRequested=true`, but it still stays
non-durable until the schema, migration ledger, and read readiness gates are
proven through the configured adapter.

`HWLAB_CLOUD_DB_SSL_MODE` is the non-secret runtime authority for Postgres TLS
mode. When DEV sets it to `disable`, the runtime strips stale URL `sslmode`,
`ssl`, and certificate query parameters before constructing the `pg` pool; a
Secret URL query parameter must not silently override the manifest SSL contract.

`/health` and `/health/live` expose this split in `readiness.durability`.
`dbLiveEvidenceObserved=true` with
`dbLiveEvidenceIsDurabilityEvidence=false` means the DB TCP/live layer is
separate from runtime persistence. If runtime durability is blocked,
`blockedLayer` names the current adapter/ssl/auth/schema/migration/durability query
layer, and `requiredEvidence` remains the durable adapter schema, migration,
and read query contract.

When the active blocker is `runtime_durable_adapter_query_blocked`, DB
connectivity is live but the durable runtime adapter has not completed the
required read queries through the runtime store. Future reports must call this a
runtime durability blocker, not a DB connectivity blocker, and must keep M3,
M4, and M5 full acceptance blocked until the durable runtime postconditions
below are true.

## DEV DB Provisioning Automation

The cloud-api image owns a stable DEV DB provisioning entrypoint:

```sh
node cmd/hwlab-cloud-api/provision.mjs --check
node cmd/hwlab-cloud-api/provision.mjs --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json
node cmd/hwlab-cloud-api/provision.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json
```

This is the preferred image-internal operator command for `hwlab-cloud-api`.
It delegates to the same repo-owned provisioning implementation as the root
script and receives admin/target DB inputs only through SecretRef-backed env.

The repo-level compatibility entrypoint remains:

```sh
node scripts/dev-runtime-provisioning.mjs --check
node scripts/dev-runtime-provisioning.mjs --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json
```

The default `--check` path validates the redacted target DB URL contract only:
the application role name, target database name, password presence, SecretRef
names, and endpoint authority are represented as booleans or redacted endpoint
class. It never connects to Postgres, never reads Kubernetes Secret data, and
never prints the target role, database name, host, password, token, DSN, or
kubeconfig material.

An authorized DEV provisioning apply is:

```sh
node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json
```

The apply path uses `HWLAB_CLOUD_DB_URL` from
`hwlab-cloud-api-dev-db/database-url` as the redacted target contract and
`HWLAB_CLOUD_DB_ADMIN_URL` from
`hwlab-cloud-api-dev-db-admin/admin-url` as the admin SecretRef env. It may
create or update only the DEV target role, target database, database `CONNECT`
grant, and `public` schema `USAGE, CREATE` grants needed by the cloud-api
durable runtime. It must report `created` or `existed` booleans and structured
blockers such as role missing, database missing, SSL, auth, schema, migration,
or durability readiness. It must not read Secret objects, print Secret values,
run PROD changes, use manual `psql`, or perform schema migration.

Provisioning reports intentionally keep target role, database, host, password,
and admin DSN out of the output. `role missing` and `database missing` are
reported as separate blockers so an auth failure cannot be repaired with an
untracked manual DB write.

## Runtime Migration Automation

The cloud-api image owns a stable runtime migration entrypoint:

```sh
node cmd/hwlab-cloud-api/migrate.mjs --check
node cmd/hwlab-cloud-api/migrate.mjs --dry-run --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json
node cmd/hwlab-cloud-api/migrate.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json
```

This is the preferred image-internal operator command for `hwlab-cloud-api`.
It delegates to the same repo-owned migration implementation as the root
script, so there is only one migration contract and one output format. The
cloud-api artifact build copies `cmd/`, `scripts/`, `internal/`, `package.json`,
and runtime dependencies into the image; the DEV CD runtime migration Job uses
`node cmd/hwlab-cloud-api/migrate.mjs` from the current cloud-api image and
injects DB inputs only through SecretRef-backed env.

The repo-level compatibility entrypoint remains:

```sh
node scripts/dev-runtime-migration.mjs --check
node scripts/dev-runtime-migration.mjs --dry-run --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json
```

Those default paths are source-only. They validate
`internal/db/migrations/0001_cloud_core_skeleton.sql`, the required
`hwlab_schema_migrations` ledger row, and the durable runtime table/column
contract without connecting to Postgres, reading Kubernetes Secrets, or writing
DEV/PROD state.

An authorized operator may run a read-only live verification only after
injecting the DEV DB URL through the existing Secret/env path:

```sh
node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json
```

A real DEV migration apply is intentionally separate and requires all explicit
write flags:

```sh
node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json
```

The apply command is scoped to DEV Postgres schema objects declared by
`internal/db/migrations/0001_cloud_core_skeleton.sql` and the
`hwlab_schema_migrations` ledger row. It must not be run for PROD, must not
read Kubernetes Secret resources, must not print `HWLAB_CLOUD_DB_URL` or any
password/token value, and must not be described as M3/M4/M5 acceptance.

Runtime migration reports separate the blockers as:

| Gate | Meaning |
| --- | --- |
| Auth | DB driver/auth/authorization can run the readiness query. |
| Schema | Required durable runtime tables and columns are present. |
| Migration | The required source migration is recorded in the ledger. |
| Readiness | Durable runtime read readiness passed after schema and ledger checks. |

## Durable Runtime Postflight

The repo-owned postflight entrypoint is:

```sh
node scripts/dev-runtime-postflight.mjs --check
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --report /tmp/hwlab-dev-gate/dev-runtime-postflight-report.json
```

Default `--check` is source-only. Live mode first reads
`http://74.48.78.17:16667/health/live` and
`http://74.48.78.17:16667/v1`. `/v1` must expose only the controlled
same-origin M3 IO route `/v1/m3/io` with contract `m3-io-control-v1`; generic
frontend gateway/box/patch-panel access is not a valid postflight substitute.
The postflight only performs the M3
`DO1=true -> DI1=true -> DO1=false -> DI1=false` write/read sequence when both
endpoints prove `runtime.adapter="postgres"`, `runtime.durable=true`,
`runtime.ready=true`, and `runtime.liveRuntimeEvidence=true`.

The M3 portion must then prove all four expected operations, exact true/false
values, operation/trace/audit/evidence identifiers, `evidenceState.status="green"`,
`evidenceState.sourceKind="DEV-LIVE"`, `evidenceState.durable=true`, and
`evidenceState.writeStatus="persisted"`. DB SecretRef presence, TCP
connectivity, `db.connected=true`, or `db.liveDbEvidence=true` alone must leave
the postflight blocked with the durable runtime blocker preserved.

## Durable Runtime Unblock Runbook

This runbook is a reusable checklist for a future authorized live
mutation/migration. It does not itself authorize a live write. Runners working
from source must stop at the read-only rows unless the commander explicitly
authorizes the DEV apply command and the preconditions below are already true.

| Check | Safe for normal runners | Evidence allowed | Not allowed |
| --- | --- | --- | --- |
| Source migration contract | Yes | `node scripts/dev-runtime-migration.mjs --check` and `node scripts/dev-runtime-migration.mjs --dry-run --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json` | DB connection, Secret read, DEV/PROD write |
| Static runtime boundary contract | Yes | `node scripts/validate-runtime-boundary.mjs`, source docs, manifests, and redacted Secret refs | Live mutation, service restart, Secret value output |
| Public health observation | Yes, when command scope is read-only | `/health/live` fields such as `db.liveDbEvidence`, `runtime.adapter`, `runtime.durable`, `runtime.blocker`, `readiness.durability.blockedLayer`, `requiredEvidence`, and redaction/safety flags | Printing DB URLs, passwords, tokens, kubeconfig material, or Secret data |
| Live provisioning apply | No, unless explicitly authorized by commander/operator | `node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json` with role/database created/existed booleans and no secret output | PROD apply, Secret value output, manual DB shell, schema migration, service restart, M3 acceptance promotion |
| Live DB read verification | Only with explicit DEV read authorization | `--dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json` output with redacted endpoint and `secretValuesPrinted=false` | Kubernetes Secret reads, writes, migration apply, PROD target |
| Live migration/repair apply | No, unless explicitly authorized by commander/operator | `--apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json` plus post-apply health evidence | PROD apply, Secret value output, service restart, Code Agent POST, hardware write, or acceptance promotion |
| Durable runtime postflight | Only after durable runtime readiness is already green | `node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --report /tmp/hwlab-dev-gate/dev-runtime-postflight-report.json` with `/health/live`, `/v1`, and M3 true/false durable green evidence | Running M3 writes while runtime durable readiness is blocked, Secret output, PROD, or manual rollout |

Secret handling rules are the same for every row: verify env names,
`secretKeyRef` names/keys, configured provider/model/base URL, redacted endpoint
source, readiness booleans, blocker codes, and `secretValuesPrinted=false`.
Do not run `kubectl get secret`, do not decode Secret data, and do not paste
`HWLAB_CLOUD_DB_URL`, `OPENAI_API_KEY`, bearer tokens, kubeconfig content, or
passwords into reports, PRs, issues, logs, screenshots, or chat.

Before any authorized live migration or repair, all of these preconditions must
be true:

- The commander has explicitly authorized a DEV-only migration/repair for this
  blocker; no PROD target, deploy, restart, or unrelated smoke is bundled into
  the same action.
- Source validation passes for `internal/db/migrations/0001_cloud_core_skeleton.sql`,
  the durable runtime tables/columns, and the `hwlab_schema_migrations` ledger
  row.
- DEV DB provisioning is ready: target role and database are either already
  present or created by `scripts/dev-runtime-provisioning.mjs`, and the report
  carries only created/existed booleans plus redacted endpoint class.
- The runtime is requesting the Postgres durable adapter, not silently using
  memory: `runtime.adapter="postgres"` or equivalent selected-adapter evidence,
  with `runtime.durableRequested=true`.
- DB live evidence is already green through the redacted Secret URL host:
  `db.liveDbEvidence=true` and `db.endpointSource="secret-url-host"`.
- The current blocker is recorded exactly, for example
  `runtime_durable_adapter_query_blocked`, with `blockedLayer` and
  `requiredEvidence` preserved in the report.
- The runner/operator can produce a redacted report that states
  `secretValuesPrinted=false` and does not require reading Kubernetes Secret
  resources.

After an authorized migration/repair, the blocker has moved or cleared only
when a new read-only health/report observation proves one of these exact
postconditions:

- Cleared: `runtime.adapter="postgres"`, `runtime.durable=true`,
  `runtime.ready=true`, `runtime.liveRuntimeEvidence=true`,
  `readiness.durability.ready=true`, no
  `runtime_durable_adapter_query_blocked`, and durable auth/schema/migration/read
  gates are all ready.
- Moved: the old `runtime_durable_adapter_query_blocked` code is absent and a
  different explicit blocker is present, such as auth, schema, migration, or
  driver readiness; the report must name the new blocker and keep acceptance
  blocked.
- Still blocked: `runtime_durable_adapter_query_blocked` remains present,
  `runtime.durable=false`, or `runtime.liveRuntimeEvidence=false`; no full
  M3/M4/M5 acceptance may be claimed.

No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked.
DB live readiness, Code Agent readiness, source migration readiness, and
read-only health observations are supporting evidence only; they do not prove
durable runtime evidence or the M3 trusted hardware loop.

## Runtime Substitution Ban

UniDesk services, provider-gateway, backend-core, microservice proxies, local
fixtures, and runner-local mocks may support scheduling, CI, CD, source checks,
or dry-runs. They cannot be described as HWLAB runtime and cannot satisfy M3,
M4, or M5 live evidence.

## Stable Sources

- [deploy/frp/README.md](../../deploy/frp/README.md): FRP DEV contract.
- [docs/d601-k3s-readonly-observability.md](../d601-k3s-readonly-observability.md):
  read-only k3s visibility command.
- [docs/dev-acceptance-matrix.md](../dev-acceptance-matrix.md): DEV endpoint
  and health acceptance.
- [pikasTech/HWLAB#164](https://github.com/pikasTech/HWLAB/issues/164):
  Code Agent provider and DB live hotfix follow-up contract.
- [pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61): manual
  rollout review and automation requirements.