pikasTech-HWLAB/docs/reference/deployment-publish.md

# HWLAB Deployment And Publish Reference

This reference records the stable DEV publish and rollout rules. It does not
authorize PROD, service restarts outside the documented DEV path, secret reads,
or direct pushes to `main`.

It is also the long-term reference for
[pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116): service
deployment regularization must progress from stable docs, to controlled
CLI/script entrypoints, to UniDesk CI/CD plus imageized delivery.

## Workspaces

| Purpose | Path |
| --- | --- |
| Runner and commander worktree | `/workspace/hwlab` |
| D601 publish/build worktree | `/home/ubuntu/workspace/hwlab` |

Use `/home/ubuntu/workspace/hwlab` for D601 build and rollout operations.
Do not treat `/home/ubuntu/hwlab` or other runner worktrees as the publishing
truth.

## Regularization Stages

| Stage | Current HWLAB state | Required next behavior |
| --- | --- | --- |
| 1. Long-term reference | This file records DEV deployment, Cloud Web publish, API/edge/health, frp, D601 registry, k3s rollout, rollback, and verification rules. `AGENTS.md` indexes it. | Keep process issues as sources only; update this reference when a deployment rule changes. |
| 2. CLI or controlled scripts | `scripts/dev-cd-apply.mjs` is the only formal DEV CD side-effect entrypoint for publish, desired-state refresh, apply, and live verify. Other controlled scripts still cover contract planning, dry-run planning, preflight, single-step internals, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. | Converge check, plan, rollback, status, and verify into HWLAB CLI while preserving the single transaction lock for side effects backed by `deploy/deploy.json`. |
| 3. UniDesk CI/CD plus images | DEV services are imageized and published to the D601 registry. D601 rollout still includes manual bridge steps for Cloud Web and frp/k3s execution. | Move build, publish, desired-state refresh, rollout, post-deploy validation, and report upload into standard UniDesk CI/CD jobs without replacing HWLAB runtime. |

## DEV CD Transaction

The stable DEV CD side-effect command is:

```sh
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

This command is the only formal path that may publish DEV artifacts, refresh
desired-state from the publish report, apply to `hwlab-dev`, and verify public
`16666/16667` in one run. It reads `deploy/deploy.json` at transaction start,
records the manifest hash and target commit, acquires
`Lease/hwlab-dev/hwlab-dev-cd-lock`, runs the internal steps serially, writes
`reports/dev-gate/dev-cd-apply.json`, and releases the Lease.

The Lease is a deployment-plane mutex, not a desired-state source. Its
annotations record at least `promotionCommit`, `deployJsonHash`,
`ownerTaskId`, `transactionId`, `phase`, `startedAt`, `updatedAt`,
`ttlSeconds`, `liveBefore`, and `targetNamespace`. `deploy/deploy.json`,
`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml`
remain the only authoritative deploy inputs.

If another transaction holds the Lease, the command must stop before publish or
apply with structured `deploy-lock-held` output that includes holder,
promotion commit, phase, target namespace, and retry seconds. Expired locks are
not silently overwritten; taking over a stale lock requires both
`--break-stale-lock` and `--confirm-dev`, and the previous holder is retained
in audit annotations/report fields.

`scripts/dev-cd-apply.mjs` also owns the read-only status surface. Running it
without arguments, with `--status`, or with `--dry-run` must return compact JSON
for the target ref, `deploy/deploy.json`, current Lease, and public
`16666/16667` live health. These modes must not acquire the Lease, publish
artifacts, apply Kubernetes workloads, or write reports. Normal `--apply`
stdout is also compact by default; the complete transaction record is preserved
through `--write-report`, or printed only when `--full-output` is explicit.

When the current DEV apply must be stopped, the operator may use
`--force-interrupt-lock` together with `--apply --confirm-dev
--confirmed-non-production`. This is a DEV-only takeover path: it first tries
to send `TERM` to local `scripts/dev-cd-apply.mjs --apply` processes, then
replaces the Lease holder and records the interrupted holder and interrupt
attempt in Lease annotations and the transaction report. Do not use this flag
for ordinary contention; prefer waiting for the holder or breaking only a
confirmed stale lock.

The old side-effect commands are now internal transaction steps. Direct manual
use of `scripts/dev-artifact-publish.mjs --publish` or
`scripts/dev-deploy-apply.mjs --apply` without `HWLAB_CD_TRANSACTION_ID` is
rejected with `cd-transaction-required`; preflight, build, dry-run, check, and
read-only modes remain available.

## Artifact Publish

DEV artifacts publish to the D601 local/internal registry:

```text
127.0.0.1:5000/hwlab/*
```

The single-step publish command is a transaction-internal side-effect step:

```sh
node scripts/dev-artifact-publish.mjs --publish
```

It only runs in side-effect mode when `HWLAB_CD_TRANSACTION_ID` is set by
`scripts/dev-cd-apply.mjs`. Use `node scripts/dev-cd-apply.mjs --apply
--confirm-dev --confirmed-non-production --write-report` for normal DEV CD.

For Cloud Web source freshness before the transaction:

```sh
node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

Cloud Web must build `web/hwlab-cloud-web/dist` before publish because the
runtime wrapper serves `dist` before source files. A correct source commit with
stale `dist` can still publish an old UI. The build/check pair is part of the
publish contract, but with separate responsibilities: `check.mjs` validates
source and does not require ignored `dist/` in a clean checkout. `build.mjs`
regenerates `dist` and verifies every runtime file and route alias matches
source. `scripts/dev-artifact-publish.mjs` runs that build/freshness gate and
records the `distFreshness` result before it builds the image.

After a successful multi-service publish, refresh the artifact catalog from the
publish report:

```sh
node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
```

If publish is blocked, keep the report blocked rather than inventing digests:

```sh
node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
```

The desired-state files are a single deployment truth: `deploy/deploy.json`,
`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml` must
carry the same promotion commit and immutable image tag before they are used as
apply input. A `--target-ref` desired-state plan may report a uniform older
state as `promotion_pending`; PR review and CI/CD promotion gates must use
`--target-ref origin/main --check` or `--promotion-commit <sha> --check` so
stale or partially refreshed commit, image, workload, and mirror-env fields are
blocked before any DEV apply.
Reports under `reports/` are evidence snapshots only and cannot override this
desired-state contract.

## Latest-Main Artifact/Runtime Guard

Before any controlled DEV apply for latest `origin/main`, run the read-only
artifact/runtime guard:

```sh
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report
```

When current readiness is expected to be blocked, use:

```sh
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked
```

The guard compares target main, `reports/dev-gate/dev-artifacts.json`,
`deploy/artifact-catalog.dev.json`, the desired-state planner, Cloud API
`/health/live`, and Cloud Web `:16666/health/live`. It is report-only and must
not be used as an apply command. A local source check, static workbench smoke,
or stale report snapshot cannot claim latest-main deployment unless the guard's
runtime and artifact identity checks also pass.

## API, Edge, Health, And FRP

The public route is fixed:

| Surface | Current entry |
| --- | --- |
| Cloud Web | `http://74.48.78.17:16666/` |
| Cloud Web live health | `http://74.48.78.17:16666/health/live` |
| API/edge health | `http://74.48.78.17:16667/health` |
| API/live health | `http://74.48.78.17:16667/health/live` |

FRP maps master public `16666/16667` to D601 `hwlab-dev` services. Use
`deploy/deploy.json` as the source for endpoint and FRP drift checks:

```sh
node scripts/deploy-contract-plan.mjs --check
```

Do not reintroduce public `6666/6667` as active DEV endpoints. Internal k3s
services may still use internal `6667`.

## Cloud Web Manual DEV Rollout

This section is retained as historical operational baseline for what the
transaction reproduces. New DEV side effects should use
`scripts/dev-cd-apply.mjs`; do not bypass the Lease with ad hoc publish,
`kubectl set image`, or restart commands.

The DEV CD transaction owns the durable runtime support sequence before final
live postflight:

```sh
node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --write-report
node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --write-report
```

These commands are called by `scripts/dev-cd-apply.mjs` under the transaction
Lease. Runners without explicit rollout authorization may run the source-only
`--check` variants and prepare PRs/artifacts, but must leave DEV apply,
rollout, live verification, and lock ownership to the host commander.

The previous verified manual path for `hwlab-cloud-web` was:

```sh
cd /home/ubuntu/workspace/hwlab
git pull --ff-only origin main
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
```

That historical sequence is no longer the normal side-effect path; direct
`--publish` is rejected unless it is called by the transaction.

The rollout must use D601 native k3s:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
```

The rollout report must tie the live deployment back to artifact identity:
source commit, image tag, image reference, registry digest or `not_published`,
and Kubernetes rollout revision. `scripts/dev-deploy-apply.mjs` reports this
under `devDeployApply.cloudWebRollout` when read-only Deployment access exists;
without that access, it reports the blocker and the exact read/rollout commands
instead of claiming a live rollout.

Public verification must check the browser entry on `16666`, not legacy
ports:

```sh
curl -fsS http://74.48.78.17:16666/health/live
curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p'
curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh'
curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p'
curl -fsS http://74.48.78.17:16667/health/live
```

Cloud Workbench acceptance also expects Chinese default UI, no outer page
scroll, and an internal Markdown help surface. The frontend contract is
[cloud-workbench.md](cloud-workbench.md).

## Apply Boundary

`scripts/dev-deploy-apply.mjs` is the DEV-only apply/report backend. A real
apply requires the transaction environment created by `scripts/dev-cd-apply.mjs`
plus explicit non-production flags:

```sh
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

Direct manual `--apply` calls without `HWLAB_CD_TRANSACTION_ID` return
`cd-transaction-required`. The apply path must stop before mutation when
artifact evidence, registry digests, kube access, DEV health, or required
runtime readiness are missing. It must not read secret values or target PROD.

Suspended template Jobs are immutable in `spec.template`. Only the explicit
DEV allowlist may be replaced by the apply script:

- `hwlab-agent-worker-template`
- `hwlab-cli-template`

## Rollback Boundary

Rollback remains DEV-only and must be explicit. Capture current image state
before changing a workload:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout history deployment/hwlab-cloud-web
```

For a Deployment rolled by `set image`, use Kubernetes rollout undo only for
the same DEV Deployment:

```sh
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/hwlab-cloud-web
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
```

After rollback, repeat public verification on `16666` and `16667`. Do not use
rollback to change PROD, read secrets, restart unrelated services, or conceal
an artifact/report mismatch.

## Controlled Entrypoint Design

The #116 target is a single controlled deployment surface. Until that lands,
use the repo-owned scripts below instead of ad hoc shell fragments.

| Action | Current controlled entrypoint | Gap to close |
| --- | --- | --- |
| Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. |
| Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. |
| Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. |
| Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. |
| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. |
| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Internal transaction step after publish; read/write desired-state only from repo files. |
| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` inside `HWLAB_CD_TRANSACTION_ID` | Internal transaction step; direct side-effect calls are rejected. |
| Roll back DEV Deployment | `kubectl -n hwlab-dev rollout undo deployment/<name>` with explicit D601 kubeconfig | Add `hwlab deploy rollback --env dev --service <id> --to-revision <n>`. |
| Verify status | `curl` checks plus `node scripts/d601-k3s-readonly-observability.mjs` | Add `hwlab deploy verify --env dev --write-report`. |

## Automation Direction

The stable direction from
[pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61) and
[pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116) is to
converge manual operations into CLI plus `deploy/deploy.json`, then run that
surface from UniDesk CI/CD:

- render and verify frps/frpc from the deploy manifest;
- publish images and write digest reports;
- refresh catalog/workload desired state from the publish report;
- apply DEV with immutable Job replacement policy;
- verify `16666` Cloud Web and `16667/health/live` API;
- invalidate or regenerate active reports after endpoint or artifact changes.

UniDesk CI/CD is infrastructure only. It may schedule build/publish/deploy
jobs and store artifacts, but it must not replace `hwlab-cloud-api`,
`hwlab-cloud-web`, `hwlab-router`, `hwlab-patch-panel`, simulators, gateways,
or agent services as the runtime being validated.

## Stable Sources

- [docs/dev-artifact-publish.md](../dev-artifact-publish.md): artifact publish
  command and report contract.
- [docs/dev-deploy-apply.md](../dev-deploy-apply.md): apply/report and Cloud
  Web manual rollout details.
- [web/hwlab-cloud-web/scripts/dist-contract.mjs](../../web/hwlab-cloud-web/scripts/dist-contract.mjs): Cloud Web `dist` freshness contract used by build/check/publish.
- [pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61): manual
  rollout review and automation target.
- [pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116):
  deployment regularization stages and acceptance.