# HWLAB DEV Deploy Apply

This document records the DEV-only apply/preflight path for
`pikasTech/HWLAB#33` and the apply-plan/report path for
`pikasTech/HWLAB#50`. It targets only the D601 `hwlab-dev` namespace and must
not be used for PROD.

## Scope

- Work from latest `origin/main`.
- Read only HWLAB repository deploy inputs: `deploy/deploy.json`,
  `deploy/artifact-catalog.dev.json`, and `deploy/k8s/dev`.
- Use `scripts/deploy-desired-state-plan.mjs` as source/dry-run support to
  review commit, image, tag, workload image, and env mirror convergence before
  any apply review.
- Validate artifact identity, image commit tags, Kubernetes workload/service
  shape, DEV health, and cloud-api DB connectivity.
- Emit an operator-readable plan in `devDeployApply`: plan/apply boundary,
  expected artifact commit, target namespace, workloads, services, remaining
  blockers, minimal manual commands, and rollback hints.
- Plan and, in apply mode, execute a scoped replacement for DEV suspended
  template Jobs whose image changed but whose Kubernetes `spec.template` is
  immutable.
- Stop before mutation when artifact publish evidence, registry digests,
  `kubectl`, DEV health, or cloud-api DB connectivity are missing.

## Commands

Run a preflight that is expected to report blockers on the current skeleton:

```sh
node --check scripts/dev-deploy-apply.mjs
node --check scripts/src/dev-deploy-apply.mjs
node scripts/deploy-desired-state-plan.mjs --check
node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report
node scripts/validate-dev-gate-report.mjs
```

Run the existing CLI dry-run surface:

```sh
node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run
```

Only after CI publish evidence and executor prerequisites exist, a real DEV
apply requires explicit non-production confirmation:

```sh
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

The script refuses PROD flags, secret-read flags, heavyweight e2e flags, and
force-push flags. It also refuses to apply if the DEV artifact catalog still
contains skeleton-only unpublished images.

`deploy-desired-state-plan.mjs` is not an apply preflight substitute. It is a
read-only source planner and does not prove registry image existence, DEV
cluster pullability, DEV apply, service restart, or M3 DEV-LIVE evidence.

## Report Contract

The machine-readable result is written to
`reports/dev-gate/dev-deploy-report.json` when `--write-report` is used. The
report remains compatible with the #33 DEV gate validator and adds these
operator fields:

- `devDeployApply.conclusion`: `blocked` while blockers are open, otherwise
  `ready` for human approval.
- `devDeployApply.applyBoundary`: default no-write behavior, required apply
  flags, DEV-only write scope, no-write scope, and forbidden actions.
- `devDeployApply.artifactPlan`: expected artifact commit, source commit,
  deploy/catalog commit IDs, publish/registry status, and unpublished services.
- `devDeployApply.target`: DEV endpoint and `hwlab-dev` namespace.
- `devDeployApply.workloadPlan` and `servicePlan`: planned Kubernetes
  workloads, containers, images, service IDs, ports, and replica intent.
- `devDeployApply.templateJobReplacementPolicy`: the DEV-only allowlist for
  suspended template Job replacement. The only allowlisted Jobs are
  `hwlab-agent-worker-template` and `hwlab-cli-template` in `hwlab-dev`.
- `devDeployApply.templateJobReplacements`: per-Job replacement decisions with
  namespace, Job name, old image, new image, and result (`planned`, `replaced`,
  `not_needed`, `not_found`, or blocked/failure states).
- `devDeployApply.cloudWebRollout`: Cloud Web source commit, image tag, image
  reference, digest, live image, and Kubernetes rollout revision when read-only
  Deployment access is available. If access is missing, this field records the
  read/rollout commands and the blocker instead of claiming a live rollout.
- `devDeployApply.manualCommands`: dry-run/report commands before human
  approval; the apply command is present only when the plan is ready.
- `devDeployApply.rollbackHint`: pre-apply capture commands, deployment
  rollbacks, suspended job cleanup, and post-rollback health checks.
- `devDeployApply.remainingBlockers`: open blockers with concrete unblock
  hints. These do not claim live apply happened.

When `conclusion.status` is `blocked`, operators should clear the listed
`remainingBlockers` and rerun:

```sh
node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report
node scripts/validate-dev-gate-report.mjs
```

When `conclusion.status` is `ready`, run the dry-run plan and validator
immediately before human approval:

```sh
node scripts/dev-deploy-apply.mjs --dry-run --write-report
node scripts/validate-dev-gate-report.mjs
```

Only after that approval may the DEV-only apply command be used.

## Cloud Web Manual DEV Rollout

This is the currently verified manual CD path for `hwlab-cloud-web` on D601.
It is the operational baseline that future CLI automation should reproduce from
`deploy/deploy.json`; do not rediscover ports or kube contexts during an
incident.

Start from the D601 deployment copy, not a runner worktree:

```sh
cd /home/ubuntu/workspace/hwlab
git pull --ff-only origin main
```

Validate source in a clean checkout, then refresh the static bundle before
artifact publish. Cloud Web images serve `web/hwlab-cloud-web/dist` first, so
stale ignored build output can keep an old English or old layout bundle alive
even after the source commit is correct. `check.mjs` does not require ignored
`dist/`; `build.mjs` and `dev-artifact-publish.mjs` are the build-first
freshness gates.

```sh
node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web --report reports/dev-gate/dev-artifacts-hwlab-cloud-web-<tag>.json
```

Probe the image locally before touching k3s:

```sh
docker run --rm -d --name hwlab-cloud-web-probe -p 127.0.0.1:18088:8080 127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
sleep 1
curl -fsS http://127.0.0.1:18088/health/live
curl -fsS http://127.0.0.1:18088/ | sed -n '1,40p'
docker rm -f hwlab-cloud-web-probe
```

Use the D601 native k3s kubeconfig explicitly:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
```

Do not use the default kube context as the source of truth. On D601 it may be
`docker-desktop`, which is not the public DEV runtime behind `16666/16667`.

Public acceptance must verify all of these:

```sh
curl -fsS http://74.48.78.17:16666/health/live
curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p'
curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh'
curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p'
```

For the Cloud Workbench Chinese/no-outer-scroll rollout, the accepted target
state was:

- `GET /health/live` reports revision
  `1e8805664970839b72be40c34636b08f6d18b131`;
- `GET /` returns `html lang="zh-CN"` and title `HWLAB 云工作台`;
- CSS locks the outer shell with `html, body { overflow: hidden; }` and
  `.workbench-shell { height: 100dvh; overflow: hidden; }`;
- `GET /help.md` returns the Chinese Cloud Workbench help document.

The public path is:

1. master `hwlab-frps-dev` listens on `16666/16667`;
2. D601 `hwlab-frpc` runs in `hwlab-dev` from the `hwlab-frpc-config`
   ConfigMap;
3. `hwlab-dev-cloud-web` maps to
   `hwlab-cloud-web.hwlab-dev.svc.cluster.local:8080`;
4. `hwlab-dev-edge-proxy` maps to
   `hwlab-edge-proxy.hwlab-dev.svc.cluster.local:6667`.

Master `hwlab-browser-proxy-dev` is only an auxiliary local proxy on `18087`;
it is not the public browser endpoint.

## Suspended Template Job Replacement

Kubernetes Jobs cannot update `spec.template` in place. For DEV template Jobs
that are intentionally suspended and used as templates, an image tag change
would otherwise make a normal `kubectl apply -k deploy/k8s/dev` non-idempotent.

The apply script handles only the explicit DEV allowlist:

- `hwlab-agent-worker-template`
- `hwlab-cli-template`

For each allowlisted Job, the script reads the live Job in `hwlab-dev` and
compares its container image with the desired manifest image. If the live Job
exists, is suspended, and the image differs, apply mode deletes that Job before
running the normal kustomize apply so Kubernetes recreates it from the desired
manifest. Dry-run mode records the same decision as `planned` and runs
server-side kustomize dry-run without mutating the cluster. If that dry-run
reports only the expected immutable `spec.template` errors for the planned
allowlisted Jobs, the report records those errors as expected replacement
evidence instead of treating them as blockers.

The policy refuses any other Job name, any non-`hwlab-dev` namespace, and any
live Job that is not suspended. It does not change Deployment, Service,
ConfigMap, Secret, or PROD behavior.

## Current Result

The current DEV apply contract is published for artifact commit `cb35ada`.
`deploy/artifact-catalog.dev.json`, `deploy/deploy.json`, and the DEV
workloads point at `127.0.0.1:5000/hwlab/*:cb35ada` images with sha256 digest
evidence.

The D601 runner can read and apply the `hwlab-dev` namespace. A real DEV apply
may replace only the allowlisted suspended template Jobs above; the normal
Deployment, Service, ConfigMap, and health-contract apply path remains
unchanged.

`hwlab-cloud-api` has a DEV DB env contract placeholder:
`HWLAB_CLOUD_DB_URL` must come from Secret reference
`hwlab-cloud-api-dev-db/database-url`, and
`HWLAB_CLOUD_DB_SSL_MODE=require`. The non-secret DB DNS contract is
`cloud-api-db.hwlab-dev.svc.cluster.local:5432` via selectorless
`Service/cloud-api-db` in `hwlab-dev`. The local health gate reports DB
runtime config without printing the connection string.

No PROD resource, secret value, UniDesk runtime substitute, browser e2e,
heavyweight e2e, or force push is part of this flow.

Source plans, dry-run reports, and desired-state convergence checks are M3
support only. They must not be labelled DEV-LIVE without a real DEV observation
of the `DO1 -> patch-panel -> DI1` loop plus operation, trace, audit, and
evidence records.