pikasTech-HWLAB/docs/dev-deploy-apply.md

# HWLAB DEV Deploy Apply

This document records the DEV-only apply/preflight backend for
`pikasTech/HWLAB#33` and the apply-plan/report path for
`pikasTech/HWLAB#50`. Real apply side effects are normally entered through
`scripts/dev-cd-apply.mjs`, which sets the transaction environment and
serializes publish/apply/verify with the DEV CD Lease lock. The apply backend
targets only the D601 `hwlab-dev` namespace and must not be used for PROD.

## Scope

- Work from latest `origin/main`.
- Read only HWLAB repository deploy inputs: `deploy/deploy.json`,
  `deploy/artifact-catalog.dev.json`, and `deploy/k8s/dev`.
- Use `scripts/deploy-desired-state-plan.mjs` as source/dry-run support to
  review commit, image, tag, workload image, and env mirror convergence before
  any apply review.
- Validate artifact identity, image commit tags, Kubernetes workload/service
  shape, DEV health, and cloud-api DB connectivity.
- Validate that `hwlab-cloud-api` desired state declares the Code Agent
  provider env contract: `OPENAI_API_KEY` from
  `hwlab-code-agent-provider/openai-api-key` and
  `HWLAB_CODE_AGENT_OPENAI_BASE_URL` through the DEV egress/proxy path.
- Emit an operator-readable plan in `devDeployApply`: plan/apply boundary,
  expected artifact commit, target namespace, workloads, services, remaining
  blockers, minimal manual commands, and rollback hints.
- Plan and, in apply mode, execute a scoped replacement for DEV suspended
  template Jobs whose image changed but whose Kubernetes `spec.template` is
  immutable.
- Plan and, in apply mode, execute a scoped cleanup for stale DEV simulator
  Deployments `hwlab-gateway-simu` and `hwlab-box-simu` only when the source
  manifest contains their indexed StatefulSet replacements. This keeps the M3
  DEV runtime at exactly two gateway-simu sessions and two box-simu resources.
- Stop before mutation when artifact publish evidence, registry digests,
  `kubectl`, DEV health, cloud-api DB connectivity, or Code Agent provider
  env preservation are missing.

## Commands

Run a preflight that is expected to report blockers on the current skeleton:

```sh
node --check scripts/dev-deploy-apply.mjs
node --check scripts/src/dev-deploy-apply.mjs
node --check scripts/dev-runtime-migration.mjs
node --check scripts/src/dev-runtime-migration.mjs
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
node scripts/dev-runtime-migration.mjs --check
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked
node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report /tmp/hwlab-dev-gate/report.json
node scripts/validate-dev-gate-report.mjs
```

Run the existing CLI dry-run surface:

```sh
node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run
```

Only after CI publish evidence and executor prerequisites exist, a real DEV
CD transaction is:

```sh
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
```

The underlying apply backend still requires explicit non-production
confirmation and a transaction environment:

```sh
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
```

Direct `--apply` without `HWLAB_CD_TRANSACTION_ID` is rejected with
`cd-transaction-required`; this is a bypass guard, not a second lock. The
script refuses PROD flags, secret-read flags, heavyweight e2e flags, and
force-push flags. It also refuses to apply if the DEV artifact catalog still
contains skeleton-only unpublished images.

`deploy-desired-state-plan.mjs` is not an apply preflight substitute. Pair it
with `artifact-runtime-readiness-guard.mjs` before apply review so stale
artifact catalogs, stale desired-state, stale API runtime identity, or a stale
Cloud Web `:16666` build stay explicit blockers.

Kubeconfig selection is explicit and safe. The script uses this precedence:

1. `--kubeconfig PATH`
2. `HWLAB_DEV_KUBECONFIG`
3. `KUBECONFIG`
4. `/etc/rancher/k3s/k3s.yaml`

The selected path is used for read-only preflight, dry-run planning, and the
DEV-only apply path. The report records the kubeconfig source and path, but it
does not print kubeconfig contents or Secret values.

Emergency D601 guard: for normal D601 execution, prefer
`/etc/rancher/k3s/k3s.yaml`. Docker Desktop Kubernetes has been disabled and
cleaned, but a residual default kubeconfig may still point at
`docker-desktop`/`127.0.0.1:11700`; do not pass that path through `KUBECONFIG`
or `--kubeconfig`. Any selected kubeconfig whose context/node resolves to
`docker-desktop` or `desktop-control-plane` is a blocker, not DEV-LIVE
evidence. See [pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138).

Runtime DB migration is a separate DEV-only apply boundary. The source-only
check is safe for normal PR validation:

```sh
node scripts/dev-runtime-migration.mjs --check
node scripts/dev-runtime-migration.mjs --dry-run --report /tmp/hwlab-dev-gate/report.json
```

Read-only live DB verification requires explicit DEV confirmation and an
already injected `HWLAB_CLOUD_DB_URL`; the report redacts the value and keeps
auth, schema, migration ledger, and readiness blockers separate:

```sh
node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/report.json
```

A real migration write is not part of default deploy validation. It requires a
separately authorized DEV operation:

```sh
node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
```

That command may only apply
`internal/db/migrations/0001_cloud_core_skeleton.sql` to the DEV Postgres target
provided by `HWLAB_CLOUD_DB_URL`. It must not read Kubernetes Secret resources,
print the DB URL/password/token, mutate PROD, restart services, or claim
M3/M4/M5 acceptance.

`deploy-desired-state-plan.mjs` is not an apply preflight substitute. It is a
read-only source planner and does not prove registry image existence, DEV
cluster pullability, DEV apply, service restart, or M3 DEV-LIVE evidence.

## Report Contract

The machine-readable result is written to
`/tmp/hwlab-dev-gate/dev-deploy-report.json` when `--report /tmp/hwlab-dev-gate/report.json` is used. The
report remains compatible with the #33 DEV gate validator and adds these
operator fields:

- `devDeployApply.conclusion`: `blocked` while blockers are open, otherwise
  `ready` for human approval.
- `devDeployApply.applyBoundary`: default no-write behavior, required apply
  flags, DEV-only write scope, no-write scope, and forbidden actions.
- `devDeployApply.artifactPlan`: expected artifact commit, source commit,
  deploy/catalog commit IDs, publish/registry status, and unpublished services.
- `devDeployApply.target`: DEV endpoint and `hwlab-dev` namespace.
- `devDeployApply.workloadPlan` and `servicePlan`: planned Kubernetes
  workloads, containers, images, service IDs, ports, and replica intent.
- `devDeployApply.templateJobReplacementPolicy`: the DEV-only allowlist for
  suspended template Job replacement. The only allowlisted Jobs are
  `hwlab-agent-worker-template` and `hwlab-cli-template` in `hwlab-dev`.
- `devDeployApply.templateJobReplacements`: per-Job replacement decisions with
  namespace, Job name, old image, new image, and result (`planned`, `replaced`,
  `not_needed`, `not_found`, or blocked/failure states).
- `devDeployApply.legacySimulatorDeploymentCleanupPolicy`: the DEV-only
  allowlist for stale simulator Deployment cleanup after the M3 simulator
  runtime moved to indexed StatefulSets.
- `devDeployApply.legacySimulatorDeploymentCleanups`: per-Deployment cleanup
  decisions for `hwlab-gateway-simu` and `hwlab-box-simu`, including old/new
  images and result (`planned`, `deleted`, `not_found`, or blocked/failure
  states).
- `devDeployApply.cloudWebRollout`: Cloud Web source commit, image tag, image
  reference, digest, live image, and Kubernetes rollout revision when read-only
  Deployment access is available. If access is missing, this field records the
  read/rollout commands and the blocker instead of claiming a live rollout.
- `devDeployApply.codeAgentProvider`: no-secret provider validation gate. It
  records desired-state env/ref readiness plus live `hwlab-cloud-api`
  Deployment env preservation before apply and, in apply mode, after apply.
  This gate reads Deployment env metadata only; it must not read Kubernetes
  Secret data or attempt a provider call.
- `devDeployApply.manualCommands`: dry-run/report commands before human
  approval; the apply command is present only when the plan is ready.
- `devDeployApply.rollbackHint`: pre-apply capture commands, deployment
  rollbacks, suspended job cleanup, and post-rollback health checks.
- `devDeployApply.remainingBlockers`: open blockers with concrete unblock
  hints. These do not claim live apply happened.

When `conclusion.status` is `blocked`, operators should clear the listed
`remainingBlockers` and rerun:

```sh
node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report /tmp/hwlab-dev-gate/report.json
node scripts/validate-dev-gate-report.mjs
```

When `conclusion.status` is `ready`, run the dry-run plan and validator
immediately before human approval:

```sh
node scripts/dev-deploy-apply.mjs --dry-run --report /tmp/hwlab-dev-gate/report.json
node scripts/validate-dev-gate-report.mjs
```

Only after that approval may the DEV CD transaction command be used.

## Cloud Web Manual DEV Rollout

This is the currently verified manual CD path for `hwlab-cloud-web` on D601.
It is the operational baseline that future CLI automation should reproduce from
`deploy/deploy.json`; do not rediscover ports or kube contexts during an
incident.

Start from the D601 deployment copy, not a runner worktree:

```sh
cd /home/ubuntu/workspace/hwlab
git pull --ff-only origin main
```

Validate source in a clean checkout, then refresh the static bundle before
artifact publish. Cloud Web images serve `web/hwlab-cloud-web/dist` first, so
stale ignored build output can keep an old English or old layout bundle alive
even after the source commit is correct. `check.mjs` does not require ignored
`dist/`; `build.mjs` and `dev-artifact-publish.mjs` are the build-first
freshness gates.

```sh
node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
```

Probe the image locally before touching k3s:

```sh
docker run --rm -d --name hwlab-cloud-web-probe -p 127.0.0.1:18088:8080 127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
sleep 1
curl -fsS http://127.0.0.1:18088/health/live
curl -fsS http://127.0.0.1:18088/ | sed -n '1,40p'
docker rm -f hwlab-cloud-web-probe
```

Use the D601 native k3s kubeconfig explicitly:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
```

Do not use the default kube context as the source of truth. On D601 it may be
`docker-desktop`, which is not the public DEV runtime behind `16666/16667`.

Public acceptance must verify all of these:

```sh
curl -fsS http://74.48.78.17:16666/health/live
curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p'
curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh'
curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p'
```

For the Cloud Workbench Chinese/no-outer-scroll rollout, the accepted target
state was:

- `GET /health/live` reports revision
  `1e8805664970839b72be40c34636b08f6d18b131`;
- `GET /` returns `html lang="zh-CN"` and title `HWLAB 云工作台`;
- CSS locks the outer shell with `html, body { overflow: hidden; }` and
  `.workbench-shell { height: 100dvh; overflow: hidden; }`;
- `GET /help.md` returns the Chinese Cloud Workbench help document.

The public path is:

1. master `hwlab-frps-dev` listens on `16666/16667`;
2. D601 `hwlab-frpc` runs in `hwlab-dev` from the `hwlab-frpc-config`
   ConfigMap;
3. `hwlab-dev-cloud-web` maps to
   `hwlab-cloud-web.hwlab-dev.svc.cluster.local:8080`;
4. `hwlab-dev-edge-proxy` maps to
   `hwlab-edge-proxy.hwlab-dev.svc.cluster.local:6667`.

Master `hwlab-browser-proxy-dev` is only an auxiliary local proxy on `18087`;
it is not the public browser endpoint.

## Suspended Template Job Replacement

Kubernetes Jobs cannot update `spec.template` in place. For DEV template Jobs
that are intentionally suspended and used as templates, an image tag change
would otherwise make a normal `kubectl apply -k deploy/k8s/dev` non-idempotent.

The apply script handles only the explicit DEV allowlist:

- `hwlab-agent-worker-template`
- `hwlab-cli-template`

For each allowlisted Job, the script reads the live Job in `hwlab-dev` and
compares its container image with the desired manifest image. If the live Job
exists, is suspended, and the image differs, apply mode deletes that Job before
running the normal kustomize apply so Kubernetes recreates it from the desired
manifest. Dry-run mode records the same decision as `planned` and runs
server-side kustomize dry-run without mutating the cluster. If that dry-run
reports only the expected immutable `spec.template` errors for the planned
allowlisted Jobs, the report records those errors as expected replacement
evidence instead of treating them as blockers.

The policy refuses any other Job name, any non-`hwlab-dev` namespace, and any
live Job that is not suspended. It does not change Deployment, Service,
ConfigMap, Secret, or PROD behavior.

## Current Result

The current DEV apply contract is published for artifact commit `cb35ada`.
`deploy/artifact-catalog.dev.json`, `deploy/deploy.json`, and the DEV
workloads point at `127.0.0.1:5000/hwlab/*:cb35ada` images with sha256 digest
evidence.

On Code Queue runners, the approved mounted kubeconfig path is typically
`/var/lib/unidesk/code-queue/kubeconfig`. Export `HWLAB_DEV_KUBECONFIG` or
pass `--kubeconfig` explicitly when the default D601 path is not available:

```sh
HWLAB_DEV_KUBECONFIG=/var/lib/unidesk/code-queue/kubeconfig node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report /tmp/hwlab-dev-gate/report.json
```

The D601 runner can read and apply the `hwlab-dev` namespace. A real DEV apply
may replace only the allowlisted suspended template Jobs above; the normal
Deployment, Service, ConfigMap, and health-contract apply path remains
unchanged.

`hwlab-cloud-api` has a DEV DB env contract placeholder:
`HWLAB_CLOUD_DB_URL` must come from Secret reference
`hwlab-cloud-api-dev-db/database-url`, and
`HWLAB_CLOUD_DB_SSL_MODE=disable`. Runtime readiness dials the redacted host
parsed from that Secret URL. `cloud-api-db` remains an optional desired alias,
not a readiness gate or TCP probe target, until this repo owns its
Service/Endpoint rollout contract. The local health gate reports DB runtime
config without printing the connection string.

Durable runtime migration now has a repo-owned path:
`scripts/dev-runtime-migration.mjs --check` proves the source migration and
ledger contract without DB access; `--dry-run --allow-live-db-read
--confirm-dev` verifies live auth/schema/migration/readiness without writes;
`--apply --confirm-dev --confirmed-non-production` is the only DEV DB write
mode and remains outside default PR validation.

No PROD resource, secret value, UniDesk runtime substitute, browser e2e,
heavyweight e2e, or force push is part of this flow.

Source plans, dry-run reports, and desired-state convergence checks are M3
support only. They must not be labelled DEV-LIVE without a real DEV observation
of the `DO1 -> patch-panel -> DI1` loop plus operation, trace, audit, and
evidence records.