Code Queue Review
339 lines
17 KiB
Markdown
339 lines
17 KiB
Markdown
# HWLAB Deployment And Publish Reference
|
||
|
||
This reference records the stable DEV publish and rollout rules. It does not
|
||
authorize PROD, service restarts outside the documented DEV path, secret reads,
|
||
or direct pushes to `main`.
|
||
|
||
It is also the long-term reference for
|
||
[pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116): service
|
||
deployment regularization must progress from stable docs, to controlled
|
||
CLI/script entrypoints, to UniDesk CI/CD plus imageized delivery.
|
||
|
||
## Workspaces
|
||
|
||
| Purpose | Path |
|
||
| --- | --- |
|
||
| Runner and commander worktree | `/workspace/hwlab` |
|
||
| D601 publish/build worktree | `/home/ubuntu/workspace/hwlab` |
|
||
|
||
Use `/home/ubuntu/workspace/hwlab` for D601 build and rollout operations.
|
||
Do not treat `/home/ubuntu/hwlab` or other runner worktrees as the publishing
|
||
truth.
|
||
|
||
## Regularization Stages
|
||
|
||
| Stage | Current HWLAB state | Required next behavior |
|
||
| --- | --- | --- |
|
||
| 1. Long-term reference | This file records DEV deployment, Cloud Web publish, API/edge/health, frp, D601 registry, k3s rollout, rollback, and verification rules. `AGENTS.md` indexes it. | Keep process issues as sources only; update this reference when a deployment rule changes. |
|
||
| 2. CLI or controlled scripts | `scripts/dev-cd-apply.mjs` is the only formal DEV CD side-effect entrypoint for publish, desired-state refresh, apply, and live verify. Other controlled scripts still cover contract planning, dry-run planning, preflight, single-step internals, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. | Converge check, plan, rollback, status, and verify into HWLAB CLI while preserving the single transaction lock for side effects backed by `deploy/deploy.json`. |
|
||
| 3. UniDesk CI/CD plus images | DEV services are imageized and published to the D601 registry. D601 rollout still includes manual bridge steps for Cloud Web and frp/k3s execution. | Move build, publish, desired-state refresh, rollout, post-deploy validation, and report upload into standard UniDesk CI/CD jobs without replacing HWLAB runtime. |
|
||
|
||
## DEV CD Transaction
|
||
|
||
正式 DEV CD 写路径只有一个入口：
|
||
|
||
```sh
|
||
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
|
||
```
|
||
|
||
该命令是唯一允许一次性发布 DEV artifact、从发布报告刷新 desired
|
||
state、apply 到 `hwlab-dev`、并复验公开 `16666/16667` 的正式写路径。
|
||
事务开始时读取 `deploy/deploy.json`，记录 manifest hash 和 target commit，
|
||
获取 `Lease/hwlab-dev/hwlab-dev-cd-lock`，串行执行内部步骤，按需写入
|
||
`reports/dev-gate/dev-cd-apply.json`，最后释放 Lease。
|
||
|
||
Lease 只是发布平面的互斥锁，不是 desired-state 来源。Lease annotations
|
||
至少记录 `promotionCommit`、`deployJsonHash`、`ownerTaskId`、
|
||
`transactionId`、`phase`、`startedAt`、`updatedAt`、`ttlSeconds`、
|
||
`liveBefore` 和 `targetNamespace`。`deploy/deploy.json`、
|
||
`deploy/artifact-catalog.dev.json`、`deploy/k8s/base/workloads.yaml` 仍是唯
|
||
一权威部署输入。
|
||
|
||
如果另一个事务持有 Lease，`--apply` 必须在 publish/apply 之前停止，并
|
||
输出结构化 `deploy-lock-held`，包含 holder、promotion commit、phase、
|
||
target namespace 和 retry seconds。过期锁不能静默覆盖；stale lock 只能
|
||
在确认 holder 已失活后显式使用 `--break-stale-lock --confirm-dev`，并把
|
||
previous holder 保留在审计 annotations/report 字段中。未过期 Lease 没有
|
||
抢占入口，普通 status 输出也不得建议强制抢锁。
|
||
|
||
`scripts/dev-cd-apply.mjs` 同时提供只读状态面。无参数、`--status`、或
|
||
`--dry-run` 都返回紧凑 JSON，覆盖 target ref/commit、
|
||
`deploy/deploy.json` commit/hash、当前 Lease 状态摘要、以及公开
|
||
`16666/16667` live health identity 摘要。这些模式 host commander 和
|
||
runner 都可以执行，但必须保持只读：不获取或替换 Lease，不 publish
|
||
artifact，不执行 `kubectl apply/rollout`，不写 report，不修改
|
||
`deploy.json`/catalog，也不读取或打印 Secret value。kubectl 或 kubeconfig
|
||
不可用时，状态面应返回 `lock.status=unavailable` 和脱敏原因，而不是把
|
||
只读观测失败升级成写路径 blocker。
|
||
|
||
正常 `--apply` stdout 默认也是紧凑摘要；完整事务记录继续通过
|
||
`--write-report` 写入既有 report 路径，只有显式 `--full-output` 时才打印
|
||
完整 JSON。DEV apply、rollout 和 live verification 仍由 host commander
|
||
统一执行；runner 默认只使用 status/dry-run 观测面，不竞争 DEV CD lock。
|
||
|
||
旧的 side-effect 命令已经降为事务内部步骤。直接手动执行
|
||
`scripts/dev-artifact-publish.mjs --publish` 或
|
||
`scripts/dev-deploy-apply.mjs --apply` 且没有 `HWLAB_CD_TRANSACTION_ID` 时，
|
||
会以 `cd-transaction-required` 拒绝；preflight、build、dry-run、check 和
|
||
read-only 模式仍可用。
|
||
|
||
## Artifact Publish
|
||
|
||
DEV artifacts publish to the D601 local/internal registry:
|
||
|
||
```text
|
||
127.0.0.1:5000/hwlab/*
|
||
```
|
||
|
||
The single-step publish command is a transaction-internal side-effect step:
|
||
|
||
```sh
|
||
node scripts/dev-artifact-publish.mjs --publish
|
||
```
|
||
|
||
It only runs in side-effect mode when `HWLAB_CD_TRANSACTION_ID` is set by
|
||
`scripts/dev-cd-apply.mjs`. Use `node scripts/dev-cd-apply.mjs --apply
|
||
--confirm-dev --confirmed-non-production --write-report` for normal DEV CD.
|
||
|
||
For Cloud Web source freshness before the transaction:
|
||
|
||
```sh
|
||
node web/hwlab-cloud-web/scripts/check.mjs
|
||
node web/hwlab-cloud-web/scripts/build.mjs
|
||
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
|
||
```
|
||
|
||
Cloud Web must build `web/hwlab-cloud-web/dist` before publish because the
|
||
runtime wrapper serves `dist` before source files. A correct source commit with
|
||
stale `dist` can still publish an old UI. The build/check pair is part of the
|
||
publish contract, but with separate responsibilities: `check.mjs` validates
|
||
source and does not require ignored `dist/` in a clean checkout. `build.mjs`
|
||
regenerates `dist` and verifies every runtime file and route alias matches
|
||
source. `scripts/dev-artifact-publish.mjs` runs that build/freshness gate and
|
||
records the `distFreshness` result before it builds the image.
|
||
|
||
After a successful multi-service publish, refresh the artifact catalog from the
|
||
publish report:
|
||
|
||
```sh
|
||
node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json
|
||
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
|
||
```
|
||
|
||
If publish is blocked, keep the report blocked rather than inventing digests:
|
||
|
||
```sh
|
||
node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked
|
||
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
|
||
```
|
||
|
||
The desired-state files are a single deployment truth: `deploy/deploy.json`,
|
||
`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml` must
|
||
carry the same promotion commit and immutable image tag before they are used as
|
||
apply input. A `--target-ref` desired-state plan may report a uniform older
|
||
state as `promotion_pending`; PR review and CI/CD promotion gates must use
|
||
`--target-ref origin/main --check` or `--promotion-commit <sha> --check` so
|
||
stale or partially refreshed commit, image, workload, and mirror-env fields are
|
||
blocked before any DEV apply.
|
||
Reports under `reports/` are evidence snapshots only and cannot override this
|
||
desired-state contract.
|
||
|
||
## Latest-Main Artifact/Runtime Guard
|
||
|
||
Before any controlled DEV apply for latest `origin/main`, run the read-only
|
||
artifact/runtime guard:
|
||
|
||
```sh
|
||
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report
|
||
```
|
||
|
||
When current readiness is expected to be blocked, use:
|
||
|
||
```sh
|
||
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked
|
||
```
|
||
|
||
The guard compares target main, `reports/dev-gate/dev-artifacts.json`,
|
||
`deploy/artifact-catalog.dev.json`, the desired-state planner, Cloud API
|
||
`/health/live`, and Cloud Web `:16666/health/live`. It is report-only and must
|
||
not be used as an apply command. A local source check, static workbench smoke,
|
||
or stale report snapshot cannot claim latest-main deployment unless the guard's
|
||
runtime and artifact identity checks also pass.
|
||
|
||
## API, Edge, Health, And FRP
|
||
|
||
The public route is fixed:
|
||
|
||
| Surface | Current entry |
|
||
| --- | --- |
|
||
| Cloud Web | `http://74.48.78.17:16666/` |
|
||
| Cloud Web live health | `http://74.48.78.17:16666/health/live` |
|
||
| API/edge health | `http://74.48.78.17:16667/health` |
|
||
| API/live health | `http://74.48.78.17:16667/health/live` |
|
||
|
||
FRP maps master public `16666/16667` to D601 `hwlab-dev` services. Use
|
||
`deploy/deploy.json` as the source for endpoint and FRP drift checks:
|
||
|
||
```sh
|
||
node scripts/deploy-contract-plan.mjs --check
|
||
```
|
||
|
||
Do not reintroduce public `6666/6667` as active DEV endpoints. Internal k3s
|
||
services may still use internal `6667`.
|
||
|
||
## Cloud Web Manual DEV Rollout
|
||
|
||
This section is retained as historical operational baseline for what the
|
||
transaction reproduces. New DEV side effects should use
|
||
`scripts/dev-cd-apply.mjs`; do not bypass the Lease with ad hoc publish,
|
||
`kubectl set image`, or restart commands.
|
||
|
||
The DEV CD transaction owns the durable runtime support sequence before final
|
||
live postflight:
|
||
|
||
```sh
|
||
node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --write-report
|
||
node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report
|
||
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --write-report
|
||
```
|
||
|
||
These commands are called by `scripts/dev-cd-apply.mjs` under the transaction
|
||
Lease. Runners without explicit rollout authorization may run the source-only
|
||
`--check` variants and prepare PRs/artifacts, but must leave DEV apply,
|
||
rollout, live verification, and lock ownership to the host commander.
|
||
|
||
The previous verified manual path for `hwlab-cloud-web` was:
|
||
|
||
```sh
|
||
cd /home/ubuntu/workspace/hwlab
|
||
git pull --ff-only origin main
|
||
node web/hwlab-cloud-web/scripts/build.mjs
|
||
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web
|
||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
|
||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
|
||
```
|
||
|
||
That historical sequence is no longer the normal side-effect path; direct
|
||
`--publish` is rejected unless it is called by the transaction.
|
||
|
||
The rollout must use D601 native k3s:
|
||
|
||
```sh
|
||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
|
||
```
|
||
|
||
The rollout report must tie the live deployment back to artifact identity:
|
||
source commit, image tag, image reference, registry digest or `not_published`,
|
||
and Kubernetes rollout revision. `scripts/dev-deploy-apply.mjs` reports this
|
||
under `devDeployApply.cloudWebRollout` when read-only Deployment access exists;
|
||
without that access, it reports the blocker and the exact read/rollout commands
|
||
instead of claiming a live rollout.
|
||
|
||
Public verification must check the browser entry on `16666`, not legacy
|
||
ports:
|
||
|
||
```sh
|
||
curl -fsS http://74.48.78.17:16666/health/live
|
||
curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p'
|
||
curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh'
|
||
curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p'
|
||
curl -fsS http://74.48.78.17:16667/health/live
|
||
```
|
||
|
||
Cloud Workbench acceptance also expects Chinese default UI, no outer page
|
||
scroll, and an internal Markdown help surface. The frontend contract is
|
||
[cloud-workbench.md](cloud-workbench.md).
|
||
|
||
## Apply Boundary
|
||
|
||
`scripts/dev-deploy-apply.mjs` is the DEV-only apply/report backend. A real
|
||
apply requires the transaction environment created by `scripts/dev-cd-apply.mjs`
|
||
plus explicit non-production flags:
|
||
|
||
```sh
|
||
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
|
||
```
|
||
|
||
Direct manual `--apply` calls without `HWLAB_CD_TRANSACTION_ID` return
|
||
`cd-transaction-required`. The apply path must stop before mutation when
|
||
artifact evidence, registry digests, kube access, DEV health, or required
|
||
runtime readiness are missing. It must not read secret values or target PROD.
|
||
|
||
Suspended template Jobs are immutable in `spec.template`. Only the explicit
|
||
DEV allowlist may be replaced by the apply script:
|
||
|
||
- `hwlab-agent-worker-template`
|
||
- `hwlab-cli-template`
|
||
|
||
## Rollback Boundary
|
||
|
||
Rollback remains DEV-only and must be explicit. Capture current image state
|
||
before changing a workload:
|
||
|
||
```sh
|
||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
|
||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout history deployment/hwlab-cloud-web
|
||
```
|
||
|
||
For a Deployment rolled by `set image`, use Kubernetes rollout undo only for
|
||
the same DEV Deployment:
|
||
|
||
```sh
|
||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/hwlab-cloud-web
|
||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
|
||
```
|
||
|
||
After rollback, repeat public verification on `16666` and `16667`. Do not use
|
||
rollback to change PROD, read secrets, restart unrelated services, or conceal
|
||
an artifact/report mismatch.
|
||
|
||
## Controlled Entrypoint Design
|
||
|
||
The #116 target is a single controlled deployment surface. Until that lands,
|
||
use the repo-owned scripts below instead of ad hoc shell fragments.
|
||
|
||
| Action | Current controlled entrypoint | Gap to close |
|
||
| --- | --- | --- |
|
||
| Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. |
|
||
| Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. |
|
||
| Observe DEV CD status | `node scripts/dev-cd-apply.mjs`, `node scripts/dev-cd-apply.mjs --status`, or `node scripts/dev-cd-apply.mjs --dry-run` | Keep read-only for host and runners; no Lease mutation, artifact publish, apply, rollout, report write, or Secret read. |
|
||
| Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. |
|
||
| Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. |
|
||
| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. |
|
||
| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Internal transaction step after publish; read/write desired-state only from repo files. |
|
||
| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` inside `HWLAB_CD_TRANSACTION_ID` | Internal transaction step; direct side-effect calls are rejected. |
|
||
| Roll back DEV Deployment | `kubectl -n hwlab-dev rollout undo deployment/<name>` with explicit D601 kubeconfig | Add `hwlab deploy rollback --env dev --service <id> --to-revision <n>`. |
|
||
| Verify status | `curl` checks plus `node scripts/d601-k3s-readonly-observability.mjs` | Add `hwlab deploy verify --env dev --write-report`. |
|
||
|
||
## Automation Direction
|
||
|
||
The stable direction from
|
||
[pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61) and
|
||
[pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116) is to
|
||
converge manual operations into CLI plus `deploy/deploy.json`, then run that
|
||
surface from UniDesk CI/CD:
|
||
|
||
- render and verify frps/frpc from the deploy manifest;
|
||
- publish images and write digest reports;
|
||
- refresh catalog/workload desired state from the publish report;
|
||
- apply DEV with immutable Job replacement policy;
|
||
- verify `16666` Cloud Web and `16667/health/live` API;
|
||
- invalidate or regenerate active reports after endpoint or artifact changes.
|
||
|
||
UniDesk CI/CD is infrastructure only. It may schedule build/publish/deploy
|
||
jobs and store artifacts, but it must not replace `hwlab-cloud-api`,
|
||
`hwlab-cloud-web`, `hwlab-router`, `hwlab-patch-panel`, simulators, gateways,
|
||
or agent services as the runtime being validated.
|
||
|
||
## Stable Sources
|
||
|
||
- [docs/dev-artifact-publish.md](../dev-artifact-publish.md): artifact publish
|
||
command and report contract.
|
||
- [docs/dev-deploy-apply.md](../dev-deploy-apply.md): apply/report and Cloud
|
||
Web manual rollout details.
|
||
- [web/hwlab-cloud-web/scripts/dist-contract.mjs](../../web/hwlab-cloud-web/scripts/dist-contract.mjs): Cloud Web `dist` freshness contract used by build/check/publish.
|
||
- [pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61): manual
|
||
rollout review and automation target.
|
||
- [pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116):
|
||
deployment regularization stages and acceptance.
|