Codex
50 KiB
HWLAB Deployment And Publish Reference
This reference records the stable DEV publish and rollout rules. It does not
authorize PROD, service restarts outside the documented DEV path, secret reads,
or direct pushes to main.
It is also the long-term reference for pikasTech/HWLAB#116: service deployment regularization must progress from stable docs, to controlled CLI/script entrypoints, to UniDesk CI/CD plus imageized delivery.
D601 Native k3s Emergency Guard
HWLAB DEV deploy and publish paths must target D601 native k3s only. Docker Desktop Kubernetes on D601 has been disabled and cleaned; it must not be re-enabled or used as a second orchestrator. The cross-repo incident, evidence, and governance plan are tracked in pikasTech/unidesk#138, and the 2026-05-23 commander recovery context is in pikasTech/unidesk#118.
Every DEV CD status, apply, rollback, smoke, and manual Kubernetes command must
use KUBECONFIG=/etc/rancher/k3s/k3s.yaml and verify node d601 before any
mutation or acceptance. Bare kubectl, docker-desktop context,
desktop-control-plane, or 127.0.0.1:11700 are wrong-control-plane signals,
not HWLAB DEV-LIVE evidence. A second hwlab-dev control plane, including a
Docker Desktop cluster that happens to contain similarly named resources, is a
blocker and cannot be used as deploy, rollout, smoke, or acceptance proof.
Workspaces
| Purpose | Path |
|---|---|
| Runner and commander worktree | /workspace/hwlab |
| D601 shared source cache | /home/ubuntu/workspace/hwlab |
| D601 clean CI/CD worktree | /tmp/hwlab-cicd-<target-short-sha> |
| D601 runtime experiment worktree | /tmp/hwlab-<purpose>-<target-short-sha> |
Git remote is the source of truth. D601 may contain several HWLAB clones or
worktrees from earlier reviews, Code Queue runs, runtime experiments, or CI/CD
jobs; none of those local directories is authoritative by itself. Before using
any D601 repo path for checks, publish, or CD, verify its role, origin, HEAD,
and cleanliness.
/home/ubuntu/workspace/hwlab is the shared source cache and convenient seed
clone. It is allowed to be stale, behind origin/main, or dirty with generated
reports/dev-gate/** files from previous runs. Do not reset --hard, delete
reports, or overwrite that directory just to make a release command work. If it
is not clean and already at the target commit, use it only to git fetch and
create a detached clean worktree.
Formal CI publish and DEV CD apply must run from a clean, target-pinned
worktree such as /tmp/hwlab-cicd-<target-short-sha>. The worktree may be
created from the shared cache, but it must prove all of the following before
side effects:
git remote get-url originpoints togit@github.com:pikasTech/HWLAB.gitor the approved GitHub source of truth.git fetch origin mainsucceeds, andgit rev-parse HEADequals the target commit being published or deployed.git status --shortis empty.- The
hwlab-cli cicd submit ...jobcwdinstate.jsonpoints to the clean worktree, not to a stale runner directory or a dirty shared cache.
Runtime hotfix and gateway probing may use purpose-specific D601 worktrees under
/tmp/hwlab-<purpose>-<sha>. Those worktrees are for copying scripts into pods,
running probes, and staging temporary patches; they are not release truth and
must not be used to infer desired state unless they are also clean and pinned to
the target commit.
Do not treat /home/ubuntu/hwlab, historical runner directories, previous PR
review worktrees, or pod-local copied trees as publish/build truth. If a path's
role is unclear, stop and create a new detached worktree from origin/main
instead of repairing the unknown directory in place.
Master Server Check Boundary
Master server is only a control, coordination, Git, issue/PR, and short polling
entrypoint for HWLAB. It must not run HWLAB validation workloads locally:
npm run check, npm run web:check, Playwright/browser smokes, full test
suites, build/preflight jobs, image publishing, or DEV CD checks belong on
D601, Code Queue runners, or the formal CI/CD job plane.
The master-side HWLAB clone may be used for lightweight source reading,
git status, git diff, patch preparation, commits, PR creation, and polling
D601/CI/CD status. If a validation command would consume significant CPU,
memory, browser resources, Docker, k3s, registry, or long-lived process time,
skip it on master and run the equivalent command from a clean D601 worktree or
submit it through hwlab-cli cicd submit.
Master wrappers may start short-lived D601 commands and poll their status, but the validation work itself must execute on D601 or CI/CD. A master-local timeout, browser hang, or resource shortage is not DEV-LIVE evidence and must not be turned into a product blocker; stop the local check and move the verification to D601/CI/CD.
Regularization Stages
| Stage | Current HWLAB state | Required next behavior |
|---|---|---|
| 1. Long-term reference | This file records DEV deployment, Cloud Web publish, API/edge/health, frp, D601 registry, k3s rollout, rollback, and verification rules. AGENTS.md indexes it. |
Keep process issues as sources only; update this reference when a deployment rule changes. |
| 2. CLI or controlled scripts | hwlab-cli cicd submit/status/logs/report 是正式 CI/CD 外层控制入口；scripts/dev-artifact-publish.mjs 和 scripts/dev-cd-apply.mjs 只作为后台 worker 或只读状态脚本使用。Other controlled scripts still cover contract planning, dry-run planning, preflight, single-step internals, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. |
Continue converging check, plan, rollback, status, and verify into HWLAB CLI while preserving CI/CD separation and the single CD transaction lock backed by deploy/deploy.json. |
| 3. UniDesk CI/CD plus images | DEV services are imageized and published to the D601 registry. D601 rollout still includes manual bridge steps for Cloud Web and frp/k3s execution. | Move build, publish, desired-state refresh, rollout, post-deploy validation, and report upload into standard UniDesk CI/CD jobs without replacing HWLAB runtime. |
CI/CD CLI 非阻塞合同
HWLAB CI/CD 正式 CLI 必须遵循 Skill(cli-spec)：可能超过短时间的 build、
push、apply、rollout、runtime Job、smoke 和 postflight 都不得以前台阻塞命令
暴露给指挥官、runner 或 master-side wrapper。正式入口必须使用
Fire-and-Forget 模式：
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind ci-publish --concurrency 4
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status <jobId>
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd logs <jobId> --tail-bytes 8000
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd report <jobId>
submit 只创建 .state/hwlab-cicd/jobs/<jobId>/，写入 spec.json、
state.json、stdout.log、stderr.log 和 report.json 路径，然后启动后台
worker 并立即返回有界 JSON。status 仅读 state.json，logs 默认只返回
尾部摘要，report 默认只返回报告摘要；完整日志和完整报告通过返回的文件路径
渐进查看。任何正式 CLI 调用都不得因为 Docker build、registry push、
kubectl wait --timeout 或 rollout status 在前台长时间占用上下文。
runner pod 内执行 CD 时，127.0.0.1:5000 指向 pod 自身，不等于 D601 host 的
loopback registry；但 image reference 必须继续保持 127.0.0.1:5000/...，因为
k3s/containerd 拉镜像和 CI push 使用的是 host 视角。CD registry manifest 校验
默认先走直接 HTTP；如果 pod loopback 被拒绝，则用 Docker host-network helper
（docker run --rm --pull=never --network host）读取同一个 registry manifest。
正式命令仍必须通过非阻塞入口提交；如需显式 registry HTTP 入口，可在提交时加
--registry-manifest-base-url <url>，该参数只改变 manifest 校验 URL，不改变
deploy.json 或 workload 里的镜像引用。
scripts/dev-artifact-publish.mjs、scripts/dev-cd-apply.mjs 和其他内部脚本仍
保留为 worker/backend：它们可以在后台 job 内执行长流程，但不能作为指挥面常规
入口直接运行。临时手动排障可以直接调用底层脚本，但排障结论必须回填到正式
CLI、长期参考文档或 issue，避免把一次性命令固化为新流程。
D601 Code Queue runner 实测调试法
HWLAB CI/CD 的关键验收必须从真正使用者 runner 的容器或 pod 内复测。master server、指挥官本地 clone、D601 host 普通目录和历史 runner 目录只能作为控制面 或辅助观察面；它们不能替代当前 Code Queue 执行面看到的 worktree、网络、 kubeconfig、registry、Docker socket、Secret/ConfigMap mount 和 CLI 行为。
实地调试顺序固定为：
- 先从调度器、任务、pod 和进程信息确认当前承担生产调度的 Code Queue runner 或 scheduler 容器，避免误入已经废弃、空闲或历史测试容器。
- 通过受控维护入口进入该容器，在容器内建立干净 HWLAB worktree，并快进到
origin/main；不得直接使用/home/ubuntu/hwlab等历史任务集合目录作为 发布真相。 如果 worktree 是从另一个本地目录 clone 出来的，必须先确认origin指向git@github.com:pikasTech/HWLAB.git或等价 GitHub source of truth，再git fetch origin main。DEV CD 的 target-ref guard 会比较 HEAD 与origin/main；本地 clone 的 staleorigin/main会触发target-ref-headblocker，不能通过跳过 guard 处理。 - 在同一个容器内执行
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit ...， 用status、logs、report查询后台 job，而不是前台直接跑长脚本。 - 至少完成一次 CI artifact publish 和一次 DEV CD apply，再用公开
16666/16667health 核对 live commit、service readiness 和 blocker 字段。 - 调试结论必须写回 issue、正式 CLI、preflight 或长期参考文档；不能把一次性 容器命令当作新的常规发布流程。
runner 内最小复测命令形态如下：
cd <clean-hwlab-worktree>
git fetch origin main
git checkout main
git pull --ff-only origin main
git status --short
KUBECONFIG=/etc/rancher/k3s/k3s.yaml \
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind ci-publish --concurrency 4
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status <ci-job-id>
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd report <ci-job-id>
KUBECONFIG=/etc/rancher/k3s/k3s.yaml \
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status <cd-job-id>
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd report <cd-job-id>
一次 runner 实测只有同时满足以下条件，才可判断 CI/CD 链路通畅：
ci-publish和dev-cd-applyjob 都进入终态succeeded；- job report 中
blockers为空，耗时、服务数、镜像 tag/digest 和 target commit 有界展示； Lease/hwlab-dev/hwlab-dev-cd-lock在事务结束后释放或显示为无活动 holder；- 公开
16666和16667health 的 commit 与deploy/deploy.jsonpromotion commit 一致； - runner 容器内没有残留
hwlab-cli、dev-cd-apply或旧发布 worker 进程继续 持有写路径。
本次 CI/CD 疏通暴露出的稳定故障类别也必须长期前置：
127.0.0.1:5000在 runner pod 内、Docker host-network helper 内和 k3s containerd 内含义不同。镜像引用仍使用127.0.0.1:5000/...作为 D601 host registry identity；registry manifest 校验按运行视角选择直接 HTTP 或 Docker host-network helper，不得为了 pod 视角改写 release identity。- Codex stdio 所需
auth.jsonSecret 和config.tomlConfigMap 是发布前置 条件。preflight 只检查对象和 key 名，不读取 secret value；ConfigMap 属于 repo desired-state，Secret 属于外部环境。 - 已经恢复的 live 服务不能替代正式 CD 报告。若第一次 CD 因缺少 mount、 registry 视角或权限失败，即使手动恢复了 live，也必须再次从 runner 内走 非阻塞 DEV CD job，产出通过报告。
- CI 产物事实以 registry manifest 和 artifact catalog 为准；临时
/tmp/hwlab-dev-gate/*.json只用于审计和耗时排障，不得成为 CD 放行硬依赖。 kubectl apply -k不会自动移除由运行面kubectl patch、临时 ConfigMap 热修或其他 field manager 写入、且不属于 desired manifest / last-applied 的 Deploymentvolumes、volumeMounts或 template annotations。正式 CD 不能把 “镜像 tag 已更新”误判为“热修覆盖已撤销”。DEV CD apply 必须在 apply 前比对 desired Deployment 与 live Deployment：若 live 存在 desired 不包含、且明确带hotfix/override语义的额外覆盖字段，应在hwlab-dev内删除该 desired Deployment，再由同一次kubectl apply -k deploy/k8s/dev从源码 desired-state 重新创建；报告中记录 cleanup 计划和实际动作。不要把这种 cleanup 做成某个服务 或某个 skill 的特例。- Kubernetes service links 会为同 namespace Service 自动注入形如
HWLAB_CLOUD_API_PORT=tcp://...的环境变量。应用入口不得直接把这类 env 当作 数字端口解析；desired-state 应显式声明服务自身监听端口HWLAB_CLOUD_API_PORT=6667，入口代码也必须只接受合法数值端口，否则回落到 内置默认值。
DEV CD Transaction
正式 DEV CD 写路径只有一个外层入口：
node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4
该命令后台调用 scripts/dev-cd-apply.mjs，是唯一允许按 deploy.json 对 hwlab-dev 执行 desired-state apply、并复验
公开 16666/16667 的正式 CD 写路径。镜像发布属于 CI artifact 职责；过渡期
latest-main 兼容路径仍可由该命令调用 artifact publish，但正式 rollout 默认应消费
已经发布并经过 registry manifest 验证的 artifact。
事务开始时读取 deploy/deploy.json，记录 manifest hash 和 target commit。
在获取 Lease/hwlab-dev/hwlab-dev-cd-lock 或运行任何 publish、runtime Job、
apply、rollout 类副作用前，脚本必须先完成 apply-only preflight：强制
KUBECONFIG=/etc/rancher/k3s/k3s.yaml，验证节点包含 d601，拒绝
docker-desktop、desktop-control-plane、127.0.0.1:11700、裸
kubectl 指向第二套 hwlab-dev 控制面或其他不可信控制面信号，并且只验证
必要 SecretRef 的存在性和 key 名。通过后才获取 Lease、串行执行内部步骤、
按需写入 /tmp/hwlab-dev-gate/dev-cd-apply.json，最后释放 Lease。
apply preflight 检查的 SecretRef 固定为：
hwlab-cloud-api-dev-db/database-urlhwlab-cloud-api-dev-db-admin/admin-urlhwlab-code-agent-provider/openai-api-keyhwlab-code-agent-codex-auth/auth.json
Code Agent Codex stdio 还必须在 preflight 阶段验证非 secret ConfigMap
hwlab-code-agent-codex-config/config.toml 的存在性和 key 名；该 ConfigMap 由
deploy/k8s/base/code-agent-codex-config.yaml 纳入 DEV desired-state。Codex
auth 仍然只能作为外部 Secret 管理，禁止把 auth.json 内容写进仓库或报告。
preflight 和报告只能记录 Secret 名、key 名、存在性、key-name 观察结果和脱敏
命令摘要；不得读取或打印 Secret value。阻塞时输出结构化 blocker，包含
scope、reason、impact、safeNextAction 和 retryable，默认 stdout 只
给有界摘要，完整 preflight 证据写入事务报告。
deploy/deploy.json 的 commitId 是运行时 promotion commit。正常
latest-main 发布时它可以等于当前 target ref；artifact merge commit 或后续
控制面提交存在时，它也可以指向一个已经发布并在 catalog/workloads 中完全收敛
的早期 promotion commit。scripts/dev-cd-apply.mjs 必须先用
deploy-desired-state-plan --promotion-commit <commit> 验证该 desired-state
已经发布且内部一致；验证通过后，--apply 消费这个 promotion commit 的镜像
执行 apply/verify，不重新 publish 或刷新当前 HEAD 的 artifact。这样 host
commander 可以始终从最新 origin/main 控制入口执行正式 DEV CD，而不需要
checkout promotion parent 或手工拼装发布环境。
状态面和事务报告会显式展示这条边界：target.controlRef 是最新
origin/main 控制入口，target.promotionCommit 是从 deploy/deploy.json
解析出的真实 promotion commit，target.artifactBoundary 记录最近一次触碰
deploy/deploy.json、deploy/artifact-catalog.dev.json、
deploy/k8s/base/workloads.yaml 的 first-parent desired-state commit，以及该
commit 的 promotion parent。artifactCatalog、artifactReport、deployJson
分别记录 hash、commit、digest 计数和 publish source，但只有
deploy/deploy.json、artifact catalog、workload manifest 和 registry manifest
共同决定 CD 是否可执行；/tmp/hwlab-dev-gate/dev-artifacts.json 只是 CI 审计快照。
若 deploy/catalog/workloads 或 desired-state parent 不能同时指向同一个
promotion commit，状态降级为 degraded，真实 --apply 在获取 Lease 前以
artifact-boundary blocker 停止。缺失或过期的 CI 产物报告只能产生诊断提示，不得
阻塞已经由 desired-state 和 registry 证明一致的 CD rollout。这保证
deploy/deploy.json 仍是唯一发布控制源；Lease 只串行化事务，不保存第二套
desired-state。
Lease 只是发布平面的互斥锁，不是 desired-state 来源。Lease annotations
至少记录 promotionCommit、deployJsonHash、ownerTaskId、
transactionId、phase、startedAt、updatedAt、ttlSeconds、
liveBefore 和 targetNamespace。deploy/deploy.json、
deploy/artifact-catalog.dev.json、deploy/k8s/base/workloads.yaml 仍是唯
一权威部署输入。
如果另一个事务持有 Lease，--apply 必须在 publish/apply 之前停止，并
输出结构化 deploy-lock-held，包含 holder、promotion commit、phase、
target namespace 和 retry seconds。过期锁不能静默覆盖；stale lock 只能
在确认 holder 已失活后显式使用 --break-stale-lock --confirm-dev，并把
previous holder 保留在审计 annotations/report 字段中。未过期 Lease 没有
抢占入口，普通 status 输出也不得建议强制抢锁。
scripts/dev-cd-apply.mjs 同时提供只读状态面。无参数、--status、或
--dry-run 都返回紧凑 JSON，覆盖 target ref/commit、
deploy/deploy.json commit/hash、当前 Lease 状态摘要、以及公开
16666/16667 live health identity 摘要。状态面还包含 liveDelta，用于对比
当前公开 live commit 与 target.promotionCommit；这只是 rollout 口径，不会在
status/dry-run 中执行 rollout。这些模式 host commander 和 runner 都可以执行，
但必须保持只读：不获取或替换 Lease，不 publish artifact，不执行
kubectl apply/rollout，不写 report，不修改 deploy.json/catalog，也不读取或
打印 Secret value。kubectl 或 kubeconfig
不可用时，状态面应返回 lock.status=unavailable 和脱敏原因，而不是把
只读观测失败升级成写路径 blocker。
deploy.json DEV CD 路径
当前已经验证可用的过渡路径是：master server 只作为受控指挥入口，通过
UniDesk CLI 进入 D601；D601 上使用干净 HWLAB 工作区运行本仓库
scripts/dev-cd-apply.mjs；发布目标由 deploy/deploy.json 和同 commit
的 artifact catalog / k8s workloads 决定；真实写入只落到 D601 原生 k3s
hwlab-dev namespace。复盘来源见
pikasTech/HWLAB#61，
长期化跟踪见 pikasTech/HWLAB#340，
单事务入口和发布锁见 pikasTech/HWLAB#274，
D601 控制面归一见 pikasTech/unidesk#138。
稳定调用合同如下；外层 master CLI 或 UniDesk wrapper 只能包裹这些 repo-owned 入口，不能绕过它们直接拼接 publish、apply、rollout 或 smoke：
| 目的 | 稳定入口 | 副作用边界 |
|---|---|---|
| 读取 DEV CD 状态 | node scripts/dev-cd-apply.mjs --status |
不写入、不获取 Lease、不读取 Secret value。 |
| 事务 dry-run preflight | node scripts/dev-cd-apply.mjs --dry-run |
不 publish、不 apply、不 rollout。 |
| 检查 desired-state 收敛 | node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check |
只读 source/report。 |
| 提交 CI artifact publish | node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind ci-publish --concurrency 4 |
立刻返回 jobId；后台 build/push，报告写入 .state/hwlab-cicd/jobs/<jobId>/report.json。 |
| 提交 DEV CD 事务 | node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4 |
立刻返回 jobId；后台在 Lease/hwlab-dev/hwlab-dev-cd-lock 下 apply/verify；legacy latest-main publish 仅作过渡兼容。 |
| 查询 CI/CD job | `node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status | logs |
| 读取 D601 k3s 证据 | node scripts/d601-k3s-readonly-observability.mjs |
只读可见性报告；不 rollout、不输出 Secret value。 |
使用 master-side wrapper 时，稳定形态是：
cd /root/unidesk
bun scripts/cli.ts ssh D601 argv bash -lc 'cd <clean-hwlab-cd-worktree> && KUBECONFIG=/etc/rancher/k3s/k3s.yaml node scripts/dev-cd-apply.mjs --status'
bun scripts/cli.ts ssh D601 argv bash -lc 'cd <clean-hwlab-cd-worktree> && KUBECONFIG=/etc/rancher/k3s/k3s.yaml node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4'
<clean-hwlab-cd-worktree> 必须指向干净工作区。该工作区可以是固定 clean
mirror，也可以是一次性 git clone --shared --no-checkout 生成的 ephemeral
worktree；它不能是 /home/ubuntu/hwlab 这类 runner/历史任务集合目录。进入
apply 前必须证明：
git status --short为空；.git/FETCH_HEAD和.git/worktrees没有阻止普通部署用户读取或创建 干净工作区的权限问题；node scripts/dev-cd-apply.mjs --status显示promotionSource=deploy-json；deploy/deploy.json的 commit、deploy/artifact-catalog.dev.json和deploy/k8s/base/workloads.yaml收敛到同一个 promotion commit / image tag；KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes能看到节点d601；kubectl config current-context、cluster server 或节点列表没有docker-desktop、desktop-control-plane、127.0.0.1:11700或第二套hwlab-dev控制面证据；Lease/hwlab-dev-cd-lock未被未过期事务持有，或者 stale lock 已由受控--break-stale-lock --confirm-dev审计处理。
desired-state 来源关系固定如下：
| 来源 | 权威范围 |
|---|---|
deploy/deploy.json |
DEV environment、namespace、endpoint、service set、promotion commit 和事务消费的 image tag。 |
deploy/artifact-catalog.dev.json |
同一 promotion commit 的不可变 artifact digest/tag 期望；CD 必须直接向 registry 校验 tag/digest 是否真实存在。 |
deploy/k8s/base/workloads.yaml |
Kubernetes workload desired state，必须镜像 deploy.json 和 artifact catalog 后才能 apply。 |
| Registry manifest | 镜像事实源；回答目标 tag/digest 是否存在，不能决定哪一组服务应该上线。 |
/tmp/hwlab-dev-gate/*.json |
只作为 CI/CD 审计、排障和耗时统计快照；不能覆盖 repo desired state，也不能成为 CD 放行硬依赖。 |
Lease/hwlab-dev/hwlab-dev-cd-lock |
只作为事务互斥锁和审计状态；永远不是 desired-state 来源。 |
只改文档的 commit 如果没有修改 deploy/deploy.json、
deploy/artifact-catalog.dev.json、deploy/k8s/base/workloads.yaml、service
source 或 runtime manifest，就不会产生新的 DEV image 或 promotion commit。
这类 commit 的 artifact build/publish 结论是 not applicable，除非另一个
artifact 任务明确更新 desired state。
Runner 和 Host 边界
Deployment/CD 工作按以下边界分工：
| 角色 | 可以执行 | 没有明确 rollout 授权时不得执行 |
|---|---|---|
| Code Queue runner | 准备分支和 PR；在任务明确授权且检查通过后更新或自合并 docs/code/artifact PR；通过 repo-owned 路径 build/publish artifact；在被分配时刷新 artifact report 或 desired-state 文件；运行只读 status/dry-run/preflight 检查。 |
未授权自合并、竞争 DEV CD Lease、执行 DEV apply、rollout、声称 16666/16667 live verification、收口 #7 或 #242、修改 PROD、打印 Secret value、手工修 live resource。 |
| Host commander | 在需要指挥官 review 时审阅并合并最终 PR；执行单一 DEV deploy apply 事务；执行 rollout 和 16666/16667 live verification；收口 #7/#242 状态。 |
绕过 repo-owned CD 入口、使用第二控制面、用 UniDesk runtime 替代 HWLAB runtime、只凭报告而不做一手复验就当作 live evidence。 |
Runner 自合并权限只在单个任务和单个 PR 的显式授权边界内生效。它不授予 DEV apply、rollout、live health verification 或看板/status 收口权限。Runner 发布 artifact 时，输出必须包含 tag、digest 和 report 证据，并把 desired-state convergence 与 live rollout evidence 分开。
CI/CD 职责拆分
长期稳定规则是 CI 负责构建和发布镜像，CD 只消费已经构建好的镜像和 repo-owned desired state。CD 入口不得临场重新解释 source、手工拼装 build env、重新构建镜像或把构建失败伪装成 rollout blocker。
CI/CD 三源分工固定如下：
| 来源 | 职责 | 不能承担 |
|---|---|---|
deploy/deploy.json |
唯一发布控制源，声明 promotion commit、service image tag 和 DEV 目标。 | 证明镜像真实存在。 |
| Registry manifest | 镜像事实源，证明目标 tag/digest 可被 registry 读取，并与 catalog 记录一致。 | 决定哪一批服务应该作为一个发布集合上线。 |
| CI 产物报告 | 自动生成的一次性审计记录，包含构建、推送、digest、耗时和日志摘要。 | 作为手工维护状态表，或成为 CD rollout 的必要本地文件。 |
deploy/artifact-catalog.dev.json 是 deploy.json 与 registry 之间的 repo-owned
digest 期望表；deploy/k8s/base/workloads.yaml 是最终 Kubernetes desired state。
CD 在 apply 前必须根据 catalog 的 image tag/digest 直接读取 registry manifest，
确认镜像事实存在且 digest 匹配。过期、缺失或路径漂移的
/tmp/hwlab-dev-gate/dev-artifacts.json 只能帮助定位 CI 问题，不得推翻
deploy.json + catalog + registry 已经证明一致的发布结论。
过渡期如果 scripts/dev-cd-apply.mjs 发现 deploy/deploy.json 已经指向一个
通过 desired-state check 的 promotion commit，应走 promotionSource=deploy-json
路径，只执行 desired-state 校验、runtime 前置、apply、postflight 和 live verify，
不得重新 publish 当前 HEAD。未来 CI 常态化后，latest-main target-ref publish
路径应降级为受限兼容路径，正式 DEV rollout 默认走已发布 artifact 的
deploy-json promotion。
CI/artifact 层可以使用以下加速面：
scripts/dev-artifact-publish.mjs --build --services <id,...>：只构建受影响服务，避免 web-only 变更重建所有服务；scripts/dev-artifact-publish.mjs --publish --services <id,...>：只在 artifact 任务边界内发布选定服务并记录 digest；scripts/dev-artifact-publish.mjs --publish --concurrency <n>：并行构建/推送服务镜像，默认 4，最高 8；scripts/dev-runtime-base-image.mjs：按package.json/package-lock.jsonhash 构建hwlab-node-runtime-base:deps-<hash>，让后续服务镜像复用依赖层。
CD 层不得把上述 CI/artifact 加速面当作 rollout 授权。只有当 desired-state
文件已经收敛、registry manifest 直接验证通过，CD 才能拿 Lease 执行 DEV apply。
CD apply 内部可以用 --rollout-concurrency <n> 或 HWLAB_DEV_CD_CONCURRENCY
并行执行 registry manifest 验证、DEV allowlist 清理和 post-apply rollout status；
并发只加速同一事务内的独立检查，不创建第二个 CD 控制源。
Git Repo Owner 权限治理
GitHub repo owner/admin 权限是治理权限，不是日常构建、合并、发布和清理权限。 长期目标是把 owner 从常规交付链路中解耦，避免 owner 账号或 owner-owned 工作区成为单点阻塞，也避免 root/owner 文件污染让 runner、CI 或 CD 无法继续。
稳定分工如下：
| 权限域 | 可以做什么 | 不应承担什么 |
|---|---|---|
| Repo owner/admin | 维护 branch protection、required checks、runner/App/token 授权、紧急权限恢复和审计规则。 | 日常手工 build、长期持有脏 worktree、替 runner 修权限污染、绕过 PR 合并主线。 |
| CI build identity | 从受保护主线或 PR commit 构建镜像，写 CI artifact、digest report 和受控 PR/commit。 | DEV apply、live rollout、看板收口、读取 Secret value。 |
| CD deploy identity | 读取已合并 desired-state 和 registry artifact，获取 DEV Lease，执行 apply/verify。 | 构建镜像、修改 source、改写 deploy.json promotion commit。 |
| Code Queue runner | 在任务分支完成代码/文档/desired-state PR，运行只读 status/preflight 和授权的 artifact 任务。 | 占用 owner 权限、写 main、污染共享发布 worktree、承担长期 CD lock。 |
所有自动化应使用可审计、可轮换的 GitHub App 或 fine-grained token；token 只授予 所需 repo 和最小 scope。日常主线合并通过 branch protection 和 required CI 结果完成，owner/admin 只处理策略变更、紧急解锁和权限恢复。至少保留两个维护者 具备 owner/admin 恢复能力，避免单人离线造成 repo 或部署链路停摆。
共享发布目录只允许作为缓存或 clean mirror，不允许成为 source of truth。任何
CI/CD 写操作都应在 ephemeral worktree 或明确 owner 的 clean worktree 中执行；
进入写路径前必须检查 git status --short、.git 可读写权限、工作区 owner/group
和 expected remote commit。发现 root-owned 文件、不可快进、未知本地修改或权限
不一致时，应结构化报 environment_blocker，由权限治理流程修复；不得直接在污染
工作区继续 build、apply 或 push。
SecretRef Preflight
apply 前还必须把必要 SecretRef 作为 preflight 条件处理，而不是等待
runtime Job 进入 CreateContainerConfigError：
hwlab-cloud-api-dev-db/database-url；hwlab-cloud-api-dev-db-admin/admin-url；hwlab-code-agent-provider/openai-api-key。
这些检查只能验证 Secret 名称、key 和 env 注入边界，不得打印 Secret value。
如果 hwlab-cloud-api-dev-db-admin/admin-url 缺失，短期 runbook 是由受控
DEV-only secret reconcile 从 unidesk-dev/postgres-dev runtime source 生成
并写入 hwlab-dev；长期应迁移到 declarative secret、sealed/external secret
或统一 secret controller。
SecretRef preflight report 必须使用布尔值和脱敏 metadata：
| 检查项 | 必要证据 | 缺失时 blocker |
|---|---|---|
| Source desired-state reference | deploy/deploy.json 和 deploy/k8s/base/workloads.yaml 声明预期 secretRef / secretKeyRef name 和 key。 |
contract_blocker，scope 指向所属 manifest 或 service。 |
| Live D601 Secret presence | 在 hwlab-dev namespace 中，repo-owned preflight 或 controller 证明必要 Secret resource 和 key 存在。 |
runtime_blocker，scope 指向缺失的 SecretRef。 |
| k3s access for the check | 检查使用 /etc/rancher/k3s/k3s.yaml，解析到节点 d601，没有命中 Docker Desktop 或第二控制面。 |
environment_blocker 或 observability_blocker，scope 指向 D601 k3s visibility。 |
| Redaction | Report 字段声明 secretValuesPrinted=false；日志不包含 decoded data、DSN、token、password、kubeconfig material 或 base64 Secret payload。 |
safety_blocker；在 publish/apply 前停止。 |
preflight 可以报告 Secret name、key name、namespace、presence boolean、
owning workload、missing scope 和 repair runbook。它不得粘贴
HWLAB_CLOUD_DB_URL, HWLAB_CLOUD_DB_ADMIN_URL, OPENAI_API_KEY, bearer
tokens, passwords, kubeconfig contents, or Secret .data values into reports,
issues, PRs, logs, screenshots, or chat.
Blocker 分类
DEV CD report 必须保持 blocker 分类稳定，使 runner 和 host 不读完整事务日志 也能路由工作：
| Type | 含义 |
|---|---|
safety_blocker |
请求动作会违反 DEV-only、non-secret、non-PROD、lock 或控制面安全规则。 |
environment_blocker |
必要 local/D601 tool、kubeconfig、k3s node、registry、workspace 或 permission 不可用。 |
contract_blocker |
Source manifest、desired-state 文件、artifact catalog、report schema 或 promotion provenance 不一致。 |
runtime_blocker |
DEV runtime 前置条件或后置条件缺失，包括 SecretRef presence、DB readiness、durable runtime、rollout 或 workload image convergence。 |
agent_blocker |
Code Agent provider/model/egress/SecretRef readiness 不完整。 |
network_blocker |
public 或 internal endpoint 不可达，或返回无效 health response。 |
observability_blocker |
runner 无法观测足够只读证据来证明状态，但底层 runtime 可能仍然健康。 |
每个 blocker 必须包含 type、scope、status 和单行 summary。可行时还
应包含 unblockHint 或 nextTask。Secret 相关 blocker 必须声明没有打印
value。
成功收口必须同时具备以下证据：
scripts/dev-cd-apply.mjs --status返回status=pass；Lease/hwlab-dev-cd-lock为phase=released、releaseStatus=pass；16666/health/live的 Cloud Web revision 等于deploy.jsonpromotion commit；16667/health/live的 Cloud API commit、DB readiness、runtime durability 均 ready；KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy,statefulset显示 persistent workload image tag 收敛到同一 promotion commit；- 输出保持 bounded summary，完整 report 写入
/tmp/hwlab-dev-gate/dev-cd-apply.json或 CLI dump 文件，避免把长 JSON 直接塞进指挥上下文。
正常 --apply stdout 默认也是紧凑摘要；完整事务记录继续通过
--report /tmp/hwlab-dev-gate/report.json 写入既有 report 路径，只有显式 --full-output 时才打印
完整 JSON。DEV apply、rollout 和 live verification 仍由 host commander
统一执行；runner 默认只使用 status/dry-run 观测面，不竞争 DEV CD lock。
旧的 deploy side-effect 命令已经降为事务内部步骤。直接手动执行
scripts/dev-deploy-apply.mjs --apply 且没有 HWLAB_CD_TRANSACTION_ID 时，会以
cd-transaction-required 拒绝。artifact publish 是 CI 职责，正式入口需要
HWLAB_CI_ARTIFACT_RUN_ID；过渡期也接受 HWLAB_CD_TRANSACTION_ID。preflight、
build、dry-run、check 和 read-only 模式仍可用。
Artifact Publish
DEV artifacts publish to the D601 local/internal registry:
127.0.0.1:5000/hwlab/*
The single-step publish command is a CI artifact side-effect step:
HWLAB_CI_ARTIFACT_RUN_ID=<run-id> node scripts/dev-artifact-publish.mjs --publish
It only runs in side-effect mode when HWLAB_CI_ARTIFACT_RUN_ID is set by the
CI artifact job, or when the legacy HWLAB_CD_TRANSACTION_ID is set by
scripts/dev-cd-apply.mjs during transition. Normal DEV CD must consume
deploy.json and registry-verified catalog state; it should not rebuild images.
For Cloud Web source freshness before the transaction:
node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
Cloud Web must build web/hwlab-cloud-web/dist before publish because the
runtime wrapper serves dist before source files. A correct source commit with
stale dist can still publish an old UI. The build/check pair is part of the
publish contract, but with separate responsibilities: check.mjs validates
source and does not require ignored dist/ in a clean checkout. build.mjs
regenerates dist and verifies every runtime file and route alias matches
source. scripts/dev-artifact-publish.mjs runs that build/freshness gate and
records the distFreshness result before it builds the image.
After a successful multi-service publish, refresh the artifact catalog from the publish report:
node scripts/refresh-artifact-catalog.mjs --target-ref <published-source-commit> --publish-report /tmp/hwlab-dev-gate/dev-artifacts.json
node scripts/deploy-desired-state-plan.mjs --promotion-commit <published-source-commit> --check
ci-publish only proves that images for its sourceCommitId were built and
pushed; it does not change the rollout target by itself. The refresh target
must be the already published sourceCommitId from the CI publish report. A
later documentation, catalog, or desired-state commit may record the promotion,
but it is not automatically an image source and must not be passed as
--target-ref unless a matching publish report proves images for that exact
commit.
Before any formal dev-cd-apply, operators must refresh
deploy/artifact-catalog.dev.json, deploy/k8s/base/workloads.yaml, and
deploy/deploy.json, commit/push that promotion, and verify the desired-state
plan points at the intended sourceCommitId. If deploy/deploy.json still
names an older commit, CD will correctly keep deploying that older promotion
even when a newer publish report exists. Do not treat the newest CI publish
report, or a later bookkeeping commit, as implicit CD desired state.
If publish is blocked, keep the report blocked rather than inventing digests:
node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked
node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check
The desired-state files are a single deployment truth: deploy/deploy.json,
deploy/artifact-catalog.dev.json, and deploy/k8s/base/workloads.yaml must
carry the same promotion commit and immutable image tag before they are used as
apply input. A --target-ref desired-state plan may report a uniform older
state as promotion_pending; PR review and CI/CD promotion gates must use
--target-ref origin/main --check or --promotion-commit <sha> --check so
stale or partially refreshed commit, image, workload, and mirror-env fields are
blocked before any DEV apply.
Reports under repository report directory are evidence snapshots only and cannot override this
desired-state contract.
Latest-Main Artifact/Runtime Guard
Before any controlled DEV apply for latest origin/main, run the read-only
artifact/runtime guard:
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report
When current readiness is expected to be blocked, use:
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked
The guard compares target main, /tmp/hwlab-dev-gate/dev-artifacts.json,
deploy/artifact-catalog.dev.json, the desired-state planner, Cloud API
/health/live, and Cloud Web :16666/health/live. It is report-only and must
not be used as an apply command. A local source check, static workbench smoke,
or stale report snapshot cannot claim latest-main deployment unless the guard's
runtime and artifact identity checks also pass.
API, Edge, Health, And FRP
The public route is fixed:
| Surface | Current entry |
|---|---|
| Cloud Web | http://74.48.78.17:16666/ |
| Cloud Web live health | http://74.48.78.17:16666/health/live |
| API/edge health | http://74.48.78.17:16667/health |
| API/live health | http://74.48.78.17:16667/health/live |
FRP maps master public 16666/16667 to D601 hwlab-dev services. Use
deploy/deploy.json as the source for endpoint and FRP drift checks:
node scripts/deploy-contract-plan.mjs --check
Do not reintroduce public 6666/6667 as active DEV endpoints. Internal k3s
services may still use internal 6667.
Cloud Web Manual DEV Rollout
This section is retained as historical operational baseline for what the
transaction reproduces. New DEV side effects should use
scripts/dev-cd-apply.mjs; do not bypass the Lease with ad hoc publish,
kubectl set image, or restart commands.
The DEV CD transaction owns the durable runtime support sequence before final live postflight:
node cmd/hwlab-cloud-api/provision.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
node cmd/hwlab-cloud-api/migrate.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --report /tmp/hwlab-dev-gate/report.json
These commands are called by scripts/dev-cd-apply.mjs under the transaction
Lease. Provisioning and migration run from the current hwlab-cloud-api image
as stable image-internal operator entrypoints; they receive DB input only
through SecretRef-backed env and print only redacted structured status. Runners
without explicit rollout authorization may run the source-only --check
variants and prepare PRs/artifacts, but must leave DEV apply, rollout, live
verification, and lock ownership to the host commander.
The previous verified manual path for hwlab-cloud-web was:
cd /home/ubuntu/workspace/hwlab
git pull --ff-only origin main
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
That historical sequence is no longer the normal side-effect path; direct
--publish is rejected unless it is called by the transaction.
The rollout must use D601 native k3s:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
The rollout report must tie the live deployment back to artifact identity:
source commit, image tag, image reference, registry digest or not_published,
and Kubernetes rollout revision. scripts/dev-deploy-apply.mjs reports this
under devDeployApply.cloudWebRollout when read-only Deployment access exists;
without that access, it reports the blocker and the exact read/rollout commands
instead of claiming a live rollout.
Public verification must check the browser entry on 16666, not legacy
ports:
curl -fsS http://74.48.78.17:16666/health/live
curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p'
curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh'
curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p'
curl -fsS http://74.48.78.17:16667/health/live
Cloud Workbench acceptance also expects Chinese default UI, no outer page scroll, and an internal Markdown help surface. The frontend contract is cloud-workbench.md.
Apply Boundary
scripts/dev-deploy-apply.mjs is the DEV-only apply/report backend. A real
apply requires the transaction environment created by scripts/dev-cd-apply.mjs
plus explicit non-production flags:
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json
Direct manual --apply calls without HWLAB_CD_TRANSACTION_ID return
cd-transaction-required. The apply path must stop before mutation when
artifact evidence, registry digests, kube access, DEV health, or required
runtime readiness are missing. It must not read secret values or target PROD.
Suspended template Jobs are immutable in spec.template. Only the explicit
DEV allowlist may be replaced by the apply script:
hwlab-agent-worker-templatehwlab-cli-template
Rollback Boundary
Rollback remains DEV-only and must be explicit. Capture current image state before changing a workload:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout history deployment/hwlab-cloud-web
For a Deployment rolled by set image, use Kubernetes rollout undo only for
the same DEV Deployment:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/hwlab-cloud-web
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
After rollback, repeat public verification on 16666 and 16667. Do not use
rollback to change PROD, read secrets, restart unrelated services, or conceal
an artifact/report mismatch.
Controlled Entrypoint Design
The #116 target is a single controlled deployment surface. Until that lands, use the repo-owned scripts below instead of ad hoc shell fragments.
| Action | Current controlled entrypoint | Gap to close |
|---|---|---|
| Check deployment contract and FRP/endpoint drift | node scripts/deploy-contract-plan.mjs --check |
Fold into hwlab deploy check --env dev. |
| Plan desired image/workload state | node scripts/deploy-desired-state-plan.mjs --plan --pretty and node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check |
Fold into hwlab deploy plan --env dev --file deploy/deploy.json with a promotion commit gate. |
| Observe DEV CD status | node scripts/dev-cd-apply.mjs, node scripts/dev-cd-apply.mjs --status, or node scripts/dev-cd-apply.mjs --dry-run |
Keep read-only for host and runners; no Lease mutation, artifact publish, apply, rollout, report write, or Secret read. |
| Preflight artifact publish | node scripts/dev-artifact-publish.mjs --preflight |
Keep as backend for hwlab artifact publish --preflight. |
| Prepare dependency runtime base | node scripts/dev-runtime-base-image.mjs --dry-run and node scripts/dev-runtime-base-image.mjs |
Move into a CI base-image job keyed by dependency hash; CD only consumes the resulting image tag. |
| Publish/apply/verify DEV | node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json |
Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. |
| Publish DEV images | HWLAB_CI_ARTIFACT_RUN_ID=<run-id> node scripts/dev-artifact-publish.mjs --publish |
CI artifact step only; legacy CD transaction env remains transitional. |
| Refresh catalog/desired state | node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report /tmp/hwlab-dev-gate/dev-artifacts.json |
CI promotion step after publish; report is input evidence for catalog refresh, not CD rollout state. |
| Apply DEV rollout | node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json inside HWLAB_CD_TRANSACTION_ID |
Internal transaction step; direct side-effect calls are rejected. |
| Roll back DEV Deployment | KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/<name> |
Add hwlab deploy rollback --env dev --service <id> --to-revision <n>. |
| Verify status | curl checks plus node scripts/d601-k3s-readonly-observability.mjs |
Add hwlab deploy verify --env dev --report /tmp/hwlab-dev-gate/report.json. |
Automation Direction
The stable direction from
pikasTech/HWLAB#61 and
pikasTech/HWLAB#116 is to
converge manual operations into CLI plus deploy/deploy.json, then run that
surface from UniDesk CI/CD:
- render and verify frps/frpc from the deploy manifest;
- publish images and write digest reports as CI audit evidence;
- refresh catalog/workload desired state from a verified CI publish report;
- apply DEV with immutable Job replacement policy;
- verify catalog tag/digest directly against registry manifests before apply;
- verify
16666Cloud Web and16667/health/liveAPI; - invalidate or regenerate active reports after endpoint or artifact changes.
UniDesk CI/CD is infrastructure only. It may schedule build/publish/deploy
jobs and store artifacts, but it must not replace hwlab-cloud-api,
hwlab-cloud-web, hwlab-router, hwlab-patch-panel, simulators, gateways,
or agent services as the runtime being validated.
Stable Sources
- docs/dev-artifact-publish.md: artifact publish command and report contract.
- docs/dev-deploy-apply.md: apply/report and Cloud Web manual rollout details.
- web/hwlab-cloud-web/scripts/dist-contract.mjs: Cloud Web
distfreshness contract used by build/check/publish. - pikasTech/HWLAB#61: manual rollout review and automation target.
- pikasTech/HWLAB#116: deployment regularization stages and acceptance.
- pikasTech/HWLAB#340:
deploy.jsonDEV CD path long-termization and master CLI automation.