mirrors/pikasTech-HWLAB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c8a8aa505e8cb9414069eefba4fc0c5c33abcd37
pikasTech-HWLAB/docs/reference/dev-runtime-boundary.md
T
Codex 61f58de2d9 docs: sync v02 cicd source isolation
2026-06-01 07:39:05 +08:00

25 KiB
Raw Blame History

HWLAB DEV Runtime Boundary

This reference defines the stable DEV environment, port, k3s, and runtime substitution rules.

Public G14 Runtime Ports

Surface DEV URL PROD URL v0.2 URL
Cloud Web browser entry http://74.48.78.17:17666/ http://74.48.78.17:18666/ http://74.48.78.17:19666/
API/edge entry http://74.48.78.17:17667/ http://74.48.78.17:18667/ http://74.48.78.17:19667/
API/live health http://74.48.78.17:17667/health/live http://74.48.78.17:18667/health/live http://74.48.78.17:19667/health/live

Public :16666 and :16667 are D601 legacy DEV endpoints. They can be used only for migration comparison or incident replay, not as the current HWLAB runtime acceptance target. Internal k3s services may still listen on 6667; do not rewrite that as a public endpoint.

Route Shape

The current G14 DEV route is:

master hwlab-frps-dev :17666/:17667
-> G14 hwlab-g14-frpc in namespace hwlab-dev
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health

master hwlab-frps-dev :18666/:18667
-> G14 hwlab-g14-prod-frpc in namespace hwlab-prod
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health

master hwlab-frps-dev :19666/:19667
-> G14 v0.2 frpc in namespace hwlab-v02
-> hwlab-cloud-web on internal :8080 for browser traffic
-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health

G14 PROD is rendered by GitOps as namespace hwlab-prod and Argo CD application hwlab-g14-prod; it uses the :18666/:18667 public endpoints when enabled. G14 desired state is generated into deploy/gitops/g14/runtime-dev and deploy/gitops/g14/runtime-prod and promoted through the G14-gitops branch.

G14 v0.2 is an additive lane, not a replacement for DEV or PROD. It uses source branch v0.2, fixed development workspace G14:/root/hwlab-v02, dedicated CI/CD source repo G14:/root/hwlab-v02-cicd.git, namespace hwlab-v02, and public ports 19666/19667. The v0.2 GitOps branch, runtime path and Argo Application must be distinct from DEV/PROD before any automated rollout is enabled.

G14 k3s Source Of Truth

G14 native k3s is the HWLAB DEV/PROD runtime source of truth. Commands that observe or mutate current HWLAB runtime must go through UniDesk route G14:k3s, for example:

tran G14:k3s kubectl -n hwlab-dev get deploy,svc,pod -o wide
tran G14:k3s kubectl -n argocd get application hwlab-g14-dev hwlab-g14-prod
tran G14:k3s kubectl -n hwlab-v02 get deploy,svc,pod -o wide

Do not use D601 kubeconfig, D601 dev-cd-apply, old JS ci-publish, Docker Desktop Kubernetes, or master-server local checks as G14 runtime acceptance evidence. D601 remains a legacy migration/reference surface until the branch-role migration is complete.

Branch Role Boundary

Current runtime work uses explicit branch lines: G14 DEV/PROD uses origin/G14, and the additive v0.2 lane uses origin/v0.2, /root/hwlab-v02-cicd.git, devops-infra git mirror/relay and hwlab-v02. Do not merge origin/main as a routine precondition for G14 or v0.2 runtime work. Treat main only as a historical branch-governance or D601 legacy reference unless a current issue explicitly assigns a main migration task. The v0.2 lane remains separate from G14 DEV/PROD; its CI/CD branch, runtime path and acceptance rules are authoritative in spec-v02-cicd.md.

Distributed Passthrough Hygiene

When running more than one command on G14, keep all commands inside one remote script block:

tran G14:/root/hwlab script <<'SCRIPT'
sed -n '1,80p' docs/reference/dev-runtime-boundary.md
sed -n '1,80p' AGENTS.md
SCRIPT

Do not write tran G14:/root/hwlab sed ... && sed ... unless the second command is intentionally local. Shell operators such as &&, ;, and pipes outside the tran invocation are evaluated by the local shell; this can make later commands run on the master server and look like a missing-file or truncation bug.

Cloud API DB Readiness Authority

hwlab-cloud-api reads the DEV database connection string from Secret reference hwlab-cloud-api-dev-db/database-url. The Secret value is not stored in source, not printed by reports, and not required for offline validation.

Runtime DB readiness uses the redacted host parsed from the injected HWLAB_CLOUD_DB_URL Secret value:

Field Value
Authority source secret-url-host
Authority env HWLAB_CLOUD_DB_URL
Required for readiness true
Probe target redacted host and port parsed from the Secret URL

Health output may report endpointSource: secret-url-host, DNS/TCP result, liveConnected, and liveDbEvidence, but must keep the DB endpoint and Secret value redacted.

The non-secret cloud-api-db name is an optional desired alias:

Field Value
Source optional-public-dns-alias
Service cloud-api-db
Namespace hwlab-dev
Host cloud-api-db.hwlab-dev.svc.cluster.local
Port 5432
Port name postgres
Required for readiness false
Used for probe false

Missing Service/cloud-api-db, Endpoint, EndpointSlice, or alias env vars must not reverse-fail live DB success from the Secret URL host. A future PR may make that alias operational only if it also owns the Service plus Endpoint or EndpointSlice manifests and the rollout/apply contract.

Source validation checks HWLAB_CLOUD_DB_URL, HWLAB_CLOUD_DB_SSL_MODE, endpointSource=secret-url-host, and liveDbEvidence=true with secret values redacted. It must not require HWLAB_CLOUD_DB_SERVICE_NAME, HWLAB_CLOUD_DB_SERVICE_NAMESPACE, HWLAB_CLOUD_DB_HOST, or HWLAB_CLOUD_DB_PORT as current readiness inputs.

The DB readiness contract is layered:

Layer Required evidence
DNS The configured target resolves from the hwlab-cloud-api runtime path and is not a .invalid placeholder.
TCP A redacted TCP probe reaches the configured Postgres port.
Auth Authenticated database access succeeds without printing the connection string.
Schema Required HWLAB schema/migration checks pass without using fixture output as live evidence.

*.invalid and hwlab-dev-db.invalid are forbidden DEV runtime targets. They may appear only as negative test fixtures; source and runtime health must not treat them as desired DEV DB endpoints.

Code Agent Provider Contract

hwlab-cloud-api runs the DEV Code Agent provider through repo-owned Codex MCP stdio sessions. Source-controlled manifests must declare only env names, Secret references, and non-secret egress/runtime settings:

Field Value
Provider env HWLAB_CODE_AGENT_PROVIDER=codex-stdio
Model env HWLAB_CODE_AGENT_MODEL=gpt-5.5
Provider Secret OPENAI_API_KEY from hwlab-code-agent-provider/openai-api-key
DEV egress/base URL 按 profile 独立配置；deepseek 走集群 bridge，codex-api 走同 Pod loopback forwarder，forwarder 直连 hyueapi 且必须保留 NO_PROXY
Workspace HWLAB_CODE_AGENT_CODEX_WORKSPACE=/workspace/hwlab with workspace-write sandbox
Codex state CODEX_HOME=/codex-home
Codex command HWLAB_CODE_AGENT_CODEX_COMMAND=/app/node_modules/.bin/codex

DEV pods must not call https://api.openai.com/v1/responses directly. G14 codex-api must send Codex CLI/app-server traffic to a same-Pod 127.0.0.1 loopback forwarder; that forwarder direct-connects to hyueapi.com / .hyueapi.com and keeps both domains in NO_PROXY and no_proxy. DeepSeek uses the cluster bridge/Moon Bridge path. http://172.26.26.227:17680/v1/responses is only a D601 legacy egress or Code Queue runner comparison clue after the migration to G14, not a G14 default runtime contract. Provider reachability must be proven by the target pod minimal closure described in Code Agent Chat Readiness Runbook, not by a one-off runtime patch or by pushing full CI/CD while the closure is still failing.

Reports, smokes, and health payloads may show Secret name/key presence, missingEnv, provider status, model, trace IDs, and redacted egress status. They must never print the OpenAI API key, bearer token, database URL password, kubeconfig material, or other secret values.

Hotfix To Source Rule

DEV runtime hotfixes are temporary recovery actions. After a hotfix proves the minimal live path, the durable follow-up is source automation: update manifests, contracts, docs, and tests so future rollouts reproduce the same env and safety boundaries. The runner must not re-execute the live hotfix, restart services, read Secrets, mutate PROD, or claim live evidence from source-only changes.

DEV pod/Deployment file overrides, ConfigMap mounts, and pod-local edits are governed by dev-runtime-hotfix-runbook.md. The stable audit entry is:

node scripts/dev-runtime-hotfix-audit.mjs --pretty
node scripts/dev-runtime-hotfix-audit.mjs --collect-readonly --pretty

The default mode only prints a read-only plan. --collect-readonly may run kubectl get and kubectl exec checks with KUBECONFIG=/etc/rancher/k3s/k3s.yaml; it must not apply, patch, rollout, restart, delete, create, read Secret resources, or print secret values.

Durable Runtime Readiness Contract

DB live readiness and durable runtime readiness are separate gates. A /health/live payload with db.connected=true and db.liveDbEvidence=true only proves that the Cloud API reached the DEV DB endpoint without exposing secret values. It does not prove that runtime writes are persisted through a durable store.

When health reports:

  • runtime.adapter: "memory"
  • runtime.durable: false
  • runtime.status: "degraded"

the Cloud API is using the process-local runtime store in internal/db/runtime-store.ts. Gateway sessions, box resources, operations, audit events, and evidence records accepted through the runtime can be lost on pod restart, redeploy, or scale-out, and cannot be treated as durable M3/M4/M5 evidence. Users may see accepted operations or evidence disappear after a runtime replacement, and agent-loop/MVP acceptance must remain degraded or blocked even if the DB connection layer is green.

Readiness reports must keep these dimensions distinct:

Dimension Green condition Degraded/blocking condition
DB live db.ready=true, db.connected=true, db.liveDbEvidence=true Missing env, failed connection, disabled probe, or no liveDbEvidence
Runtime durability runtime.adapter="postgres", runtime.durable=true, runtime.ready=true, runtime.liveRuntimeEvidence=true, and runtime auth/schema/migration/read gates are all ready runtime.adapter="memory", runtime.durable=false, missing migration ledger, failed schema/auth/readiness, or no liveRuntimeEvidence

Do not declare DEV-LIVE complete while runtime.adapter="memory" or runtime.durable=false, even when DB live readiness is connected. A selected Postgres adapter may report runtime.durableRequested=true, but it still stays non-durable until the schema, migration ledger, and read readiness gates are proven through the configured adapter.

HWLAB_CLOUD_DB_SSL_MODE is the non-secret runtime authority for Postgres TLS mode. When DEV sets it to disable, the runtime strips stale URL sslmode, ssl, and certificate query parameters before constructing the pg pool; a Secret URL query parameter must not silently override the manifest SSL contract.

/health and /health/live expose this split in readiness.durability. dbLiveEvidenceObserved=true with dbLiveEvidenceIsDurabilityEvidence=false means the DB TCP/live layer is separate from runtime persistence. If runtime durability is blocked, blockedLayer names the current adapter/ssl/auth/schema/migration/durability query layer, and requiredEvidence remains the durable adapter schema, migration, and read query contract.

When the active blocker is runtime_durable_adapter_query_blocked, DB connectivity is live but the durable runtime adapter has not completed the required read queries through the runtime store. Future reports must call this a runtime durability blocker, not a DB connectivity blocker, and must keep M3, M4, and M5 full acceptance blocked until the durable runtime postconditions below are true.

DEV DB Provisioning Automation

The cloud-api image owns a stable DEV DB provisioning entrypoint:

bun cmd/hwlab-cloud-api/provision.ts --check
bun cmd/hwlab-cloud-api/provision.ts --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json
bun cmd/hwlab-cloud-api/provision.ts --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json

This is the preferred image-internal operator command for hwlab-cloud-api. It delegates to the same repo-owned provisioning implementation as the root script and receives admin/target DB inputs only through SecretRef-backed env.

The repo-level compatibility entrypoint remains:

node scripts/dev-runtime-provisioning.mjs --check
node scripts/dev-runtime-provisioning.mjs --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json

The default --check path validates the redacted target DB URL contract only: the application role name, target database name, password presence, SecretRef names, and endpoint authority are represented as booleans or redacted endpoint class. It never connects to Postgres, never reads Kubernetes Secret data, and never prints the target role, database name, host, password, token, DSN, or kubeconfig material.

An authorized DEV provisioning apply is:

node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json

The apply path uses HWLAB_CLOUD_DB_URL from hwlab-cloud-api-dev-db/database-url as the redacted target contract and HWLAB_CLOUD_DB_ADMIN_URL from hwlab-cloud-api-dev-db-admin/admin-url as the admin SecretRef env. It may create or update only the DEV target role, target database, database CONNECT grant, and public schema USAGE, CREATE grants needed by the cloud-api durable runtime. It must report created or existed booleans and structured blockers such as role missing, database missing, SSL, auth, schema, migration, or durability readiness. It must not read Secret objects, print Secret values, run PROD changes, use manual psql, or perform schema migration.

Provisioning reports intentionally keep target role, database, host, password, and admin DSN out of the output. role missing and database missing are reported as separate blockers so an auth failure cannot be repaired with an untracked manual DB write.

Runtime Migration Automation

The cloud-api image owns a stable runtime migration entrypoint:

bun cmd/hwlab-cloud-api/migrate.ts --check
bun cmd/hwlab-cloud-api/migrate.ts --dry-run --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json
bun cmd/hwlab-cloud-api/migrate.ts --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json

This is the preferred image-internal operator command for hwlab-cloud-api. It delegates to the same repo-owned migration implementation as the root script, so there is only one migration contract and one output format. The cloud-api artifact build copies cmd/, scripts/, internal/, package.json, and runtime dependencies into the image; the DEV CD runtime migration Job uses bun cmd/hwlab-cloud-api/migrate.ts from the current cloud-api image and injects DB inputs only through SecretRef-backed env.

The repo-level compatibility entrypoint remains:

node scripts/dev-runtime-migration.mjs --check
node scripts/dev-runtime-migration.mjs --dry-run --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json

Those default paths are source-only. They validate internal/db/migrations/0001_cloud_core_skeleton.sql, the required hwlab_schema_migrations ledger row, and the durable runtime table/column contract without connecting to Postgres, reading Kubernetes Secrets, or writing DEV/PROD state.

An authorized operator may run a read-only live verification only after injecting the DEV DB URL through the existing Secret/env path:

node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json

A real DEV migration apply is intentionally separate and requires all explicit write flags:

node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json

The apply command is scoped to DEV Postgres schema objects declared by internal/db/migrations/0001_cloud_core_skeleton.sql and the hwlab_schema_migrations ledger row. It must not be run for PROD, must not read Kubernetes Secret resources, must not print HWLAB_CLOUD_DB_URL or any password/token value, and must not be described as M3/M4/M5 acceptance.

Runtime migration reports separate the blockers as:

Gate Meaning
Auth DB driver/auth/authorization can run the readiness query.
Schema Required durable runtime tables and columns are present.
Migration The required source migration is recorded in the ledger.
Readiness Durable runtime read readiness passed after schema and ledger checks.

Durable Runtime Postflight

The repo-owned postflight entrypoint is:

node scripts/dev-runtime-postflight.mjs --check
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --report /tmp/hwlab-dev-gate/dev-runtime-postflight-report.json

Default --check is source-only. Live mode first reads http://74.48.78.17:17667/health/live and http://74.48.78.17:17667/v1. /v1 must expose only the controlled same-origin M3 IO route /v1/m3/io with contract m3-io-control-v1; generic frontend gateway/box/patch-panel access is not a valid postflight substitute. The postflight only performs the M3 DO1=true -> DI1=true -> DO1=false -> DI1=false write/read sequence when both endpoints prove runtime.adapter="postgres", runtime.durable=true, runtime.ready=true, and runtime.liveRuntimeEvidence=true.

The M3 portion must then prove all four expected operations, exact true/false values, operation/trace/audit/evidence identifiers, evidenceState.status="green", evidenceState.sourceKind="DEV-LIVE", evidenceState.durable=true, and evidenceState.writeStatus="persisted". DB SecretRef presence, TCP connectivity, db.connected=true, or db.liveDbEvidence=true alone must leave the postflight blocked with the durable runtime blocker preserved.

Durable Runtime Unblock Runbook

This runbook is a reusable checklist for a future authorized live mutation/migration. It does not itself authorize a live write. Runners working from source must stop at the read-only rows unless the commander explicitly authorizes the DEV apply command and the preconditions below are already true.

Check Safe for normal runners Evidence allowed Not allowed
Source migration contract Yes node scripts/dev-runtime-migration.mjs --check and node scripts/dev-runtime-migration.mjs --dry-run --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json DB connection, Secret read, DEV/PROD write
Static runtime boundary contract Yes node scripts/validate-runtime-boundary.mjs, source docs, manifests, and redacted Secret refs Live mutation, service restart, Secret value output
Public health observation Yes, when command scope is read-only /health/live fields such as db.liveDbEvidence, runtime.adapter, runtime.durable, runtime.blocker, readiness.durability.blockedLayer, requiredEvidence, and redaction/safety flags Printing DB URLs, passwords, tokens, kubeconfig material, or Secret data
Live provisioning apply No, unless explicitly authorized by commander/operator node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-provisioning-report.json with role/database created/existed booleans and no secret output PROD apply, Secret value output, manual DB shell, schema migration, service restart, M3 acceptance promotion
Live DB read verification Only with explicit DEV read authorization --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json output with redacted endpoint and secretValuesPrinted=false Kubernetes Secret reads, writes, migration apply, PROD target
Live migration/repair apply No, unless explicitly authorized by commander/operator --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/dev-runtime-migration-report.json plus post-apply health evidence PROD apply, Secret value output, service restart, Code Agent POST, hardware write, or acceptance promotion
Durable runtime postflight Only after durable runtime readiness is already green node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --report /tmp/hwlab-dev-gate/dev-runtime-postflight-report.json with /health/live, /v1, and M3 true/false durable green evidence Running M3 writes while runtime durable readiness is blocked, Secret output, PROD, or manual rollout

Secret handling rules are the same for every row: verify env names, secretKeyRef names/keys, configured provider/model/base URL, redacted endpoint source, readiness booleans, blocker codes, and secretValuesPrinted=false. Do not run kubectl get secret, do not decode Secret data, and do not paste HWLAB_CLOUD_DB_URL, OPENAI_API_KEY, bearer tokens, kubeconfig content, or passwords into reports, PRs, issues, logs, screenshots, or chat.

Before any authorized live migration or repair, all of these preconditions must be true:

  • The commander has explicitly authorized a DEV-only migration/repair for this blocker; no PROD target, deploy, restart, or unrelated smoke is bundled into the same action.
  • Source validation passes for internal/db/migrations/0001_cloud_core_skeleton.sql, the durable runtime tables/columns, and the hwlab_schema_migrations ledger row.
  • DEV DB provisioning is ready: target role and database are either already present or created by scripts/dev-runtime-provisioning.mjs, and the report carries only created/existed booleans plus redacted endpoint class.
  • The runtime is requesting the Postgres durable adapter, not silently using memory: runtime.adapter="postgres" or equivalent selected-adapter evidence, with runtime.durableRequested=true.
  • DB live evidence is already green through the redacted Secret URL host: db.liveDbEvidence=true and db.endpointSource="secret-url-host".
  • The current blocker is recorded exactly, for example runtime_durable_adapter_query_blocked, with blockedLayer and requiredEvidence preserved in the report.
  • The runner/operator can produce a redacted report that states secretValuesPrinted=false and does not require reading Kubernetes Secret resources.

After an authorized migration/repair, the blocker has moved or cleared only when a new read-only health/report observation proves one of these exact postconditions:

  • Cleared: runtime.adapter="postgres", runtime.durable=true, runtime.ready=true, runtime.liveRuntimeEvidence=true, readiness.durability.ready=true, no runtime_durable_adapter_query_blocked, and durable auth/schema/migration/read gates are all ready.
  • Moved: the old runtime_durable_adapter_query_blocked code is absent and a different explicit blocker is present, such as auth, schema, migration, or driver readiness; the report must name the new blocker and keep acceptance blocked.
  • Still blocked: runtime_durable_adapter_query_blocked remains present, runtime.durable=false, or runtime.liveRuntimeEvidence=false; no full M3/M4/M5 acceptance may be claimed.

No full M3, M4, or M5 acceptance is allowed while runtime durability is blocked. DB live readiness, Code Agent readiness, source migration readiness, and read-only health observations are supporting evidence only; they do not prove durable runtime evidence or the M3 trusted hardware loop.

Runtime Substitution Ban

UniDesk services, provider-gateway, backend-core, microservice proxies, local fixtures, and runner-local mocks may support scheduling, CI, CD, source checks, or dry-runs. They cannot be described as HWLAB runtime and cannot satisfy M3, M4, or M5 live evidence.

Stable Sources

Reference in New Issue View Git Blame Copy Permalink