Code Queue Review
17 KiB
HWLAB Deployment And Publish Reference
This reference records the stable DEV publish and rollout rules. It does not
authorize PROD, service restarts outside the documented DEV path, secret reads,
or direct pushes to main.
It is also the long-term reference for pikasTech/HWLAB#116: service deployment regularization must progress from stable docs, to controlled CLI/script entrypoints, to UniDesk CI/CD plus imageized delivery.
Workspaces
| Purpose | Path |
|---|---|
| Runner and commander worktree | /workspace/hwlab |
| D601 publish/build worktree | /home/ubuntu/workspace/hwlab |
Use /home/ubuntu/workspace/hwlab for D601 build and rollout operations.
Do not treat /home/ubuntu/hwlab or other runner worktrees as the publishing
truth.
Regularization Stages
| Stage | Current HWLAB state | Required next behavior |
|---|---|---|
| 1. Long-term reference | This file records DEV deployment, Cloud Web publish, API/edge/health, frp, D601 registry, k3s rollout, rollback, and verification rules. AGENTS.md indexes it. |
Keep process issues as sources only; update this reference when a deployment rule changes. |
| 2. CLI or controlled scripts | scripts/dev-cd-apply.mjs is the only formal DEV CD side-effect entrypoint for publish, desired-state refresh, apply, and live verify. Other controlled scripts still cover contract planning, dry-run planning, preflight, single-step internals, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. |
Converge check, plan, rollback, status, and verify into HWLAB CLI while preserving the single transaction lock for side effects backed by deploy/deploy.json. |
| 3. UniDesk CI/CD plus images | DEV services are imageized and published to the D601 registry. D601 rollout still includes manual bridge steps for Cloud Web and frp/k3s execution. | Move build, publish, desired-state refresh, rollout, post-deploy validation, and report upload into standard UniDesk CI/CD jobs without replacing HWLAB runtime. |
DEV CD Transaction
正式 DEV CD 写路径只有一个入口：
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
该命令是唯一允许一次性发布 DEV artifact、从发布报告刷新 desired
state、apply 到 hwlab-dev、并复验公开 16666/16667 的正式写路径。
事务开始时读取 deploy/deploy.json，记录 manifest hash 和 target commit，
获取 Lease/hwlab-dev/hwlab-dev-cd-lock，串行执行内部步骤，按需写入
reports/dev-gate/dev-cd-apply.json，最后释放 Lease。
Lease 只是发布平面的互斥锁，不是 desired-state 来源。Lease annotations
至少记录 promotionCommit、deployJsonHash、ownerTaskId、
transactionId、phase、startedAt、updatedAt、ttlSeconds、
liveBefore 和 targetNamespace。deploy/deploy.json、
deploy/artifact-catalog.dev.json、deploy/k8s/base/workloads.yaml 仍是唯
一权威部署输入。
如果另一个事务持有 Lease，--apply 必须在 publish/apply 之前停止，并
输出结构化 deploy-lock-held，包含 holder、promotion commit、phase、
target namespace 和 retry seconds。过期锁不能静默覆盖；stale lock 只能
在确认 holder 已失活后显式使用 --break-stale-lock --confirm-dev，并把
previous holder 保留在审计 annotations/report 字段中。未过期 Lease 没有
抢占入口，普通 status 输出也不得建议强制抢锁。
scripts/dev-cd-apply.mjs 同时提供只读状态面。无参数、--status、或
--dry-run 都返回紧凑 JSON，覆盖 target ref/commit、
deploy/deploy.json commit/hash、当前 Lease 状态摘要、以及公开
16666/16667 live health identity 摘要。这些模式 host commander 和
runner 都可以执行，但必须保持只读：不获取或替换 Lease，不 publish
artifact，不执行 kubectl apply/rollout，不写 report，不修改
deploy.json/catalog，也不读取或打印 Secret value。kubectl 或 kubeconfig
不可用时，状态面应返回 lock.status=unavailable 和脱敏原因，而不是把
只读观测失败升级成写路径 blocker。
正常 --apply stdout 默认也是紧凑摘要；完整事务记录继续通过
--write-report 写入既有 report 路径，只有显式 --full-output 时才打印
完整 JSON。DEV apply、rollout 和 live verification 仍由 host commander
统一执行；runner 默认只使用 status/dry-run 观测面，不竞争 DEV CD lock。
旧的 side-effect 命令已经降为事务内部步骤。直接手动执行
scripts/dev-artifact-publish.mjs --publish 或
scripts/dev-deploy-apply.mjs --apply 且没有 HWLAB_CD_TRANSACTION_ID 时，
会以 cd-transaction-required 拒绝；preflight、build、dry-run、check 和
read-only 模式仍可用。
Artifact Publish
DEV artifacts publish to the D601 local/internal registry:
127.0.0.1:5000/hwlab/*
The single-step publish command is a transaction-internal side-effect step:
node scripts/dev-artifact-publish.mjs --publish
It only runs in side-effect mode when HWLAB_CD_TRANSACTION_ID is set by
scripts/dev-cd-apply.mjs. Use node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report for normal DEV CD.
For Cloud Web source freshness before the transaction:
node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
Cloud Web must build web/hwlab-cloud-web/dist before publish because the
runtime wrapper serves dist before source files. A correct source commit with
stale dist can still publish an old UI. The build/check pair is part of the
publish contract, but with separate responsibilities: check.mjs validates
source and does not require ignored dist/ in a clean checkout. build.mjs
regenerates dist and verifies every runtime file and route alias matches
source. scripts/dev-artifact-publish.mjs runs that build/freshness gate and
records the distFreshness result before it builds the image.
After a successful multi-service publish, refresh the artifact catalog from the publish report:
node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
If publish is blocked, keep the report blocked rather than inventing digests:
node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
The desired-state files are a single deployment truth: deploy/deploy.json,
deploy/artifact-catalog.dev.json, and deploy/k8s/base/workloads.yaml must
carry the same promotion commit and immutable image tag before they are used as
apply input. A --target-ref desired-state plan may report a uniform older
state as promotion_pending; PR review and CI/CD promotion gates must use
--target-ref origin/main --check or --promotion-commit <sha> --check so
stale or partially refreshed commit, image, workload, and mirror-env fields are
blocked before any DEV apply.
Reports under reports/ are evidence snapshots only and cannot override this
desired-state contract.
Latest-Main Artifact/Runtime Guard
Before any controlled DEV apply for latest origin/main, run the read-only
artifact/runtime guard:
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report
When current readiness is expected to be blocked, use:
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked
The guard compares target main, reports/dev-gate/dev-artifacts.json,
deploy/artifact-catalog.dev.json, the desired-state planner, Cloud API
/health/live, and Cloud Web :16666/health/live. It is report-only and must
not be used as an apply command. A local source check, static workbench smoke,
or stale report snapshot cannot claim latest-main deployment unless the guard's
runtime and artifact identity checks also pass.
API, Edge, Health, And FRP
The public route is fixed:
| Surface | Current entry |
|---|---|
| Cloud Web | http://74.48.78.17:16666/ |
| Cloud Web live health | http://74.48.78.17:16666/health/live |
| API/edge health | http://74.48.78.17:16667/health |
| API/live health | http://74.48.78.17:16667/health/live |
FRP maps master public 16666/16667 to D601 hwlab-dev services. Use
deploy/deploy.json as the source for endpoint and FRP drift checks:
node scripts/deploy-contract-plan.mjs --check
Do not reintroduce public 6666/6667 as active DEV endpoints. Internal k3s
services may still use internal 6667.
Cloud Web Manual DEV Rollout
This section is retained as historical operational baseline for what the
transaction reproduces. New DEV side effects should use
scripts/dev-cd-apply.mjs; do not bypass the Lease with ad hoc publish,
kubectl set image, or restart commands.
The DEV CD transaction owns the durable runtime support sequence before final live postflight:
node scripts/dev-runtime-provisioning.mjs --apply --confirm-dev --confirmed-non-production --write-report
node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --write-report
These commands are called by scripts/dev-cd-apply.mjs under the transaction
Lease. Runners without explicit rollout authorization may run the source-only
--check variants and prepare PRs/artifacts, but must leave DEV apply,
rollout, live verification, and lock ownership to the host commander.
The previous verified manual path for hwlab-cloud-web was:
cd /home/ubuntu/workspace/hwlab
git pull --ff-only origin main
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
That historical sequence is no longer the normal side-effect path; direct
--publish is rejected unless it is called by the transaction.
The rollout must use D601 native k3s:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
The rollout report must tie the live deployment back to artifact identity:
source commit, image tag, image reference, registry digest or not_published,
and Kubernetes rollout revision. scripts/dev-deploy-apply.mjs reports this
under devDeployApply.cloudWebRollout when read-only Deployment access exists;
without that access, it reports the blocker and the exact read/rollout commands
instead of claiming a live rollout.
Public verification must check the browser entry on 16666, not legacy
ports:
curl -fsS http://74.48.78.17:16666/health/live
curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p'
curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh'
curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p'
curl -fsS http://74.48.78.17:16667/health/live
Cloud Workbench acceptance also expects Chinese default UI, no outer page scroll, and an internal Markdown help surface. The frontend contract is cloud-workbench.md.
Apply Boundary
scripts/dev-deploy-apply.mjs is the DEV-only apply/report backend. A real
apply requires the transaction environment created by scripts/dev-cd-apply.mjs
plus explicit non-production flags:
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
Direct manual --apply calls without HWLAB_CD_TRANSACTION_ID return
cd-transaction-required. The apply path must stop before mutation when
artifact evidence, registry digests, kube access, DEV health, or required
runtime readiness are missing. It must not read secret values or target PROD.
Suspended template Jobs are immutable in spec.template. Only the explicit
DEV allowlist may be replaced by the apply script:
hwlab-agent-worker-templatehwlab-cli-template
Rollback Boundary
Rollback remains DEV-only and must be explicit. Capture current image state before changing a workload:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout history deployment/hwlab-cloud-web
For a Deployment rolled by set image, use Kubernetes rollout undo only for
the same DEV Deployment:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/hwlab-cloud-web
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
After rollback, repeat public verification on 16666 and 16667. Do not use
rollback to change PROD, read secrets, restart unrelated services, or conceal
an artifact/report mismatch.
Controlled Entrypoint Design
The #116 target is a single controlled deployment surface. Until that lands, use the repo-owned scripts below instead of ad hoc shell fragments.
| Action | Current controlled entrypoint | Gap to close |
|---|---|---|
| Check deployment contract and FRP/endpoint drift | node scripts/deploy-contract-plan.mjs --check |
Fold into hwlab deploy check --env dev. |
| Plan desired image/workload state | node scripts/deploy-desired-state-plan.mjs --plan --pretty and node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check |
Fold into hwlab deploy plan --env dev --file deploy/deploy.json with a promotion commit gate. |
| Observe DEV CD status | node scripts/dev-cd-apply.mjs, node scripts/dev-cd-apply.mjs --status, or node scripts/dev-cd-apply.mjs --dry-run |
Keep read-only for host and runners; no Lease mutation, artifact publish, apply, rollout, report write, or Secret read. |
| Preflight artifact publish | node scripts/dev-artifact-publish.mjs --preflight |
Keep as backend for hwlab artifact publish --preflight. |
| Publish/apply/verify DEV | node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report |
Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. |
| Publish DEV images | node scripts/dev-artifact-publish.mjs --publish inside HWLAB_CD_TRANSACTION_ID |
Internal step only; direct side-effect calls are rejected. |
| Refresh catalog/desired state | node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json |
Internal transaction step after publish; read/write desired-state only from repo files. |
| Apply DEV rollout | node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report inside HWLAB_CD_TRANSACTION_ID |
Internal transaction step; direct side-effect calls are rejected. |
| Roll back DEV Deployment | kubectl -n hwlab-dev rollout undo deployment/<name> with explicit D601 kubeconfig |
Add hwlab deploy rollback --env dev --service <id> --to-revision <n>. |
| Verify status | curl checks plus node scripts/d601-k3s-readonly-observability.mjs |
Add hwlab deploy verify --env dev --write-report. |
Automation Direction
The stable direction from
pikasTech/HWLAB#61 and
pikasTech/HWLAB#116 is to
converge manual operations into CLI plus deploy/deploy.json, then run that
surface from UniDesk CI/CD:
- render and verify frps/frpc from the deploy manifest;
- publish images and write digest reports;
- refresh catalog/workload desired state from the publish report;
- apply DEV with immutable Job replacement policy;
- verify
16666Cloud Web and16667/health/liveAPI; - invalidate or regenerate active reports after endpoint or artifact changes.
UniDesk CI/CD is infrastructure only. It may schedule build/publish/deploy
jobs and store artifacts, but it must not replace hwlab-cloud-api,
hwlab-cloud-web, hwlab-router, hwlab-patch-panel, simulators, gateways,
or agent services as the runtime being validated.
Stable Sources
- docs/dev-artifact-publish.md: artifact publish command and report contract.
- docs/dev-deploy-apply.md: apply/report and Cloud Web manual rollout details.
- web/hwlab-cloud-web/scripts/dist-contract.mjs: Cloud Web
distfreshness contract used by build/check/publish. - pikasTech/HWLAB#61: manual rollout review and automation target.
- pikasTech/HWLAB#116: deployment regularization stages and acceptance.