diff --git a/AGENTS.md b/AGENTS.md index 98bbfb31..9279939a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,11 +14,11 @@ HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥 - HWLAB #7、用户反馈、长期看板和指挥简报的 GitHub issue 正文写入必须走 UniDesk CLI：`cd /root/unidesk && bun scripts/cli.ts gh ...`；禁止直接用原生 `gh issue edit/create/comment` 写这些 issue。事故和工具补强需求见 [pikasTech/unidesk#142](https://github.com/pikasTech/unidesk/issues/142)。 - 在 UniDesk CLI 局部替换、写前备份和写后 hash 验证能力完成前，不要对 #7 做无 guard 的整篇 body replace；必须先保留 before body、确认维护纪律 heading 仍存在，再写入。 -## P0 DEV CD Promotion 顺序 +## P0 Legacy CD 删除纪律 -- `ci-publish` 只证明镜像已经为 publish report 的 `artifactPublish.sourceCommitId` 构建并推送；它不会改变 rollout 目标。禁止在 `ci-publish` 成功后直接把 `dev-cd-apply` 的旧 desired-state 输出当作新版本上线成功。 -- 正式 DEV CD 前必须用刚成功的 publish report 刷新三份 repo desired-state：`node scripts/refresh-artifact-catalog.mjs --target-ref --publish-report `，再执行 `node scripts/deploy-desired-state-plan.mjs --promotion-commit --check`，确认 `deploy/deploy.json`、`deploy/artifact-catalog.dev.json` 和 `deploy/k8s/base/workloads.yaml` 全部指向同一个已发布 commit。 -- 只有刷新后的 desired-state 已提交/推送，并且 CD 报告与 `16666/16667` live health 都观测到该短 commit，才能声明 DEV 上线完成；如果 CD report 里 `commitId` 仍是旧值，必须先修 desired-state，不要重复跑 CD。长期细节见 [docs/reference/deployment-publish.md](docs/reference/deployment-publish.md)。 +- 旧 D601 `ci-publish`、`dev-cd-apply`、`dev-deploy-apply` 和 `dev-artifact-publish` 脚本入口已经从当前发布面删除；不要恢复这些文件、CLI 子命令或文档入口。 +- 当前发布只走 G14 k3s Tekton + `G14-gitops` + Argo CD；镜像构建 helper 是 `scripts/g14-artifact-publish.mjs`，desired state 由 `scripts/g14-gitops-render.mjs` 生成，细则见 [docs/reference/g14-gitops-cicd.md](docs/reference/g14-gitops-cicd.md)。 +- 发现旧脚本、旧文档或旧 `hwlab-cli cicd` 再次进入 G14 Pipeline、AGENTS 或长期参考时，优先删除旧入口并把调用方改到 G14 GitOps 路径，不再做兼容保留。 ## 工作区 @@ -46,7 +46,7 @@ HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥 - 文档治理与 docs-spec 本地权威：[docs/reference/documentation-governance.md](docs/reference/documentation-governance.md) - 架构和 M3 主线：[docs/reference/architecture.md](docs/reference/architecture.md) - DEV 运行态、端口、k3s 和 DB DNS 边界：[docs/reference/dev-runtime-boundary.md](docs/reference/dev-runtime-boundary.md) -- 部署正规化、`deploy.json` DEV CD 路径、SecretRef preflight、runner/host 边界和镜像发布：[docs/reference/deployment-publish.md](docs/reference/deployment-publish.md) +- G14 GitOps 发布、SecretRef preflight、runner/host 边界和镜像发布：[docs/reference/g14-gitops-cicd.md](docs/reference/g14-gitops-cicd.md) - G14 GitOps CI/CD、Tekton/Argo CD、集群内 registry 和无锁镜像化发布：[docs/reference/g14-gitops-cicd.md](docs/reference/g14-gitops-cicd.md) - Code Agent 对话就绪与真实回复判定：[docs/reference/code-agent-chat-readiness.md](docs/reference/code-agent-chat-readiness.md) - DEV runtime hotfix runbook 与只读审计：[docs/reference/dev-runtime-hotfix-runbook.md](docs/reference/dev-runtime-hotfix-runbook.md) @@ -70,10 +70,10 @@ HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥 - Cloud Web 构建：`npm run web:build` - Cloud Web M3 只读护栏：`npm run web:m3-readonly` - Cloud Workbench 布局/遮挡 smoke：`npm run web:layout`；local-build 用 `npm run web:layout:build`；DEV deploy 后用 `npm run web:layout:live`。 -- G14 artifact build helper：`node scripts/g14-artifact-publish.mjs --publish ...`，只能由 G14 Tekton task 携带 CI artifact identity 调用；人工发布走 G14 poller/GitOps，不走 legacy `ci-publish`。 +- G14 artifact build helper：`node scripts/g14-artifact-publish.mjs --publish ...`，只能由 G14 Tekton task 携带 CI artifact identity 调用；人工发布走 G14 poller/GitOps，不走 legacy CLI CD。 - G14 GitOps 渲染：`npm run g14:gitops:render`；检查已生成 Tekton/Argo CD manifests：`npm run g14:gitops:check` - DEV 依赖 runtime base 构建：`npm run dev-runtime-base:build` -- Legacy D601 DEV CD：`dev-cd-apply`/`ci-publish` 仅供迁移事故回放，不作为 G14 DEV/PROD 发布入口。 +- Legacy D601 DEV CD：旧脚本入口已删除；事故回放只读历史 issue/commit，不恢复旧命令。 - CI/CD job 查询：`node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status|logs|report ` - runner GitHub 可见性预检：`npm run runner:issue-visibility:preflight` - D601 k3s 只读观测：legacy 回溯入口，仅在确认需要 D601 事故复盘时使用；当前 G14 运行面观察使用 UniDesk route `G14:k3s`。 diff --git a/docs/dev-artifact-publish.md b/docs/dev-artifact-publish.md deleted file mode 100644 index 68227e68..00000000 --- a/docs/dev-artifact-publish.md +++ /dev/null @@ -1,264 +0,0 @@ -# HWLAB DEV Artifact Publish - -`scripts/dev-artifact-publish.mjs` is the DEV-only artifact preflight/build -backend for D601 local/internal registry artifacts. Publish side effects are -normally entered by a CI artifact job identity, or by the legacy -`scripts/dev-cd-apply.mjs` transaction path during transition. The artifact -script does not deploy workloads, read secrets, enable PROD, or push to -GHCR/Docker Hub/other third-party registries. - -长期口径：CI 产物报告不是人工维护的镜像库状态，也不是 CD 放行的本地状态源。 -它是 CI 自动生成的一次性审计记录。CD 必须从 `deploy/deploy.json` 读取发布控制 -意图，从 `deploy/artifact-catalog.dev.json` 读取 digest 期望，并直接向 registry -manifest 验证目标 tag/digest 是否存在。`/tmp/hwlab-dev-gate/dev-artifacts.json` -只能用于排障、刷新 catalog 和审计追溯。 - -## Scope - -- Environment: `dev`. -- Namespace: `hwlab-dev`. -- Default registry prefix: `127.0.0.1:5000/hwlab`. -- Base image: resolved by `scripts/preflight-dev-base-image.mjs` from - `HWLAB_DEV_BASE_IMAGE`, a local `node:20-*` Docker image, or a local HWLAB - base tag such as - `127.0.0.1:5000/hwlab/hwlab-dev-base:node20-bookworm-slim`. There is no - base-image pull or third-party fallback during publish; the build runs with - `--pull=false`. -- Node runtime dependencies are installed from the source `package.json` and - `package-lock.json` inside the DEV image build with `npm ci --omit=dev - --ignore-scripts` when the lockfile is present, unless the selected base - image already provides `/opt/hwlab-node-runtime-base/node_modules`. This - includes the Postgres `pg` driver required by the durable cloud-api runtime - adapter. -- Image tag: first seven characters of the Git commit used as build source. -- Required image labels: - - `hwlab.pikastech.local/repo` - - `hwlab.pikastech.local/commit` - - `hwlab.pikastech.local/service-id` - - `hwlab.pikastech.local/environment=dev` - -## Commands - -Static check: - -```sh -node --check scripts/preflight-dev-base-image.mjs -node --check scripts/src/dev-base-image-preflight.mjs -node --check scripts/src/dev-artifact-services.mjs -node --check scripts/src/registry-capabilities.mjs -node --check scripts/dev-artifact-publish.mjs -node --check scripts/dev-runtime-base-image.mjs -node --check scripts/refresh-artifact-catalog.mjs -node --test scripts/refresh-artifact-catalog.test.mjs -``` - -Base image preflight: - -```sh -node scripts/preflight-dev-base-image.mjs -``` - -Preflight and report generation: - -```sh -node scripts/dev-artifact-publish.mjs --preflight -``` - -Build only: - -```sh -node scripts/dev-artifact-publish.mjs --build -``` - -Build only one or more selected services for a CI artifact task: - -```sh -node scripts/dev-artifact-publish.mjs --build --services hwlab-cloud-web -``` - -`--services` narrows the build/publish report to selected service IDs. It is a -CI/artifact acceleration surface; the normal DEV rollout still consumes -repo-owned desired state through `scripts/dev-cd-apply.mjs`. - -Build and publish to the D601 local/internal registry inside a CI artifact job: - -```sh -HWLAB_CI_ARTIFACT_RUN_ID= node scripts/dev-artifact-publish.mjs --publish -``` - -The legacy transaction entry remains available during migration: - -```sh -node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -Direct `--publish` without `HWLAB_CI_ARTIFACT_RUN_ID` or -`HWLAB_CD_TRANSACTION_ID` is rejected with `cd-transaction-required`; this is a -bypass guard, not a second lock. - -Daily Cloud Web source validation does not require ignored `dist/` to exist: - -```sh -node web/hwlab-cloud-web/scripts/check.mjs -``` - -For `hwlab-cloud-web`, the build path refreshes and verifies the static bundle -before image build; publish still goes through the transaction command above: - -```sh -node web/hwlab-cloud-web/scripts/build.mjs -node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -The runtime wrapper serves `/app/web/hwlab-cloud-web/dist` before the source -directory. If `dist` is stale, a newly tagged image can still serve old HTML, -CSS, and JavaScript. A publish for Cloud Web is not valid until the image has -fresh `dist` evidence. `build.mjs` regenerates `dist` and verifies every -runtime file and route alias matches source. `dev-artifact-publish.mjs` -re-runs that build/freshness gate and records -`artifactPublish.services[].distFreshness` for `hwlab-cloud-web` before the -Docker build. `check.mjs` remains a clean-checkout source contract and must not -fail only because ignored `dist/` is absent. - -Use `--registry-prefix` only for another localhost/private/internal D601 -registry prefix. The script rejects third-party registry hosts and any prefix -that names PROD. Use `--base-image` only to override `HWLAB_DEV_BASE_IMAGE` with -an approved local image for that run. - -Docker build output is visible by default so a slow layer is observable from the -runner log. Use `--quiet-build` only when a caller already captures the full -Docker log elsewhere. The artifact report records per-service -`dockerBuildDurationMs`, `cloudWebBuildDurationMs`, and `publishDurationMs`. - -An optional dependency base image can be prepared with: - -```sh -node scripts/dev-runtime-base-image.mjs --dry-run -node scripts/dev-runtime-base-image.mjs -``` - -The default tag is -`127.0.0.1:5000/hwlab/hwlab-node-runtime-base:deps-`. -Selecting that image through `HWLAB_DEV_BASE_IMAGE` lets service builds reuse a -preinstalled `node_modules` layer instead of repeating `npm ci` for every -service. - -## Report - -The script writes `/tmp/hwlab-dev-gate/dev-artifacts.json`. The file is kept in -the existing DEV gate report envelope for compatibility with -`scripts/validate-dev-gate-report.mjs`, while the `artifactPublish` object is -the HWLAB#35-specific publish record. This report is evidence, not desired -state. A later CD rollout must verify catalog images directly against the -registry instead of trusting this file. - -Each service record contains: - -- `serviceId` -- `image` -- `imageTag` -- `digest` -- `distFreshness` for `hwlab-cloud-web` build/publish runs -- `publishEnabled` -- `artifactRequired` -- `artifactScope` -- `notPublishedReason` -- `runtimeKind` -- `implementationState` -- `sourceState` -- `entrypoint` - -The report also carries `artifactPublish.serviceInventory` and -`artifactPublish.publishPlan`. `serviceInventory` is the resolved frozen service -list with enabled/disabled state. `publishPlan` is the machine-readable v2 plan: -source commit, image tag, per-service registry target, image reference, digest -placeholder or real digest, registry capability evidence, and the reason a -service was not published. - -The report also includes `artifactPublish.registryCapabilities`, split into -three dimensions: - -- `process-http-access`: runner-process HTTP access to the registry `/v2/` - endpoint. This is diagnostic only. A failed - `http://127.0.0.1:5000/v2/` fetch from the runner process is `degraded`, not - proof that Docker publish failed. -- `docker-daemon-push-access`: Docker daemon view of the local/internal - registry target. This is the publish-path capability and is the only registry - capability that blocks `--publish` before a push attempt. -- `k3s-pull-access`: read-only k3s view of `hwlab-dev` image pull state. This - is the deploy-path capability and does not prove or disprove Docker daemon - push access. - -`digest` is only set to a registry digest after `docker push` succeeds and the -push output contains a `sha256:<64 hex>` digest. If the push succeeds but no -digest is observable, the service remains `published_unverified_digest`, -`digest` stays `not_published`, and the report carries a blocker. The script -records blockers instead of claiming a publish when build, push, digest -observation, base image, registry, contract, or safety checks fail. - -After a fully successful publish, update the catalog from the report: - -```sh -node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report /tmp/hwlab-dev-gate/dev-artifacts.json -``` - -If publish remains blocked, refresh only commit/tag identity and keep digests -blocked: - -```sh -node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked -``` - -Before any later DEV apply review, run the read-only artifact/runtime guard so -artifact evidence is not mistaken for deployed runtime evidence: - -```sh -node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked -``` - -The guard must remain blocked until the target commit, publish report, artifact -catalog, desired-state files, Cloud API `/health/live`, and Cloud Web -`:16666/health/live` all report the same identity. It does not publish, deploy, -restart services, mutate Kubernetes, touch PROD, or read secrets. - -The report also includes `artifactPublish.baseImagePreflight` with the image -source, local tag, local image ID, `publishUsable` gate, blockers, and next -steps. It also carries `recommendation`, `provision`, and `blockedReport` -objects so the blocked report includes the exact blocker, expected image tag, -recommended `HWLAB_DEV_BASE_IMAGE` value, and preload commands. When that -preflight is blocked, artifact publish is blocked and no build or push is -attempted. - -If the D601 builder has no approved Node 20 base image, or the selected DEV -registry is not reachable from the Docker daemon publish path, the report must -remain `status: "blocked"`. Do not turn a base-image or registry blocker into a -fake digest or published state. Do not treat runner-process loopback HTTP -failure as a publish failure by itself. - -Committed DEV artifact reports are evidence snapshots and may lag the deploy -desired-state files. Use `scripts/deploy-desired-state-plan.mjs` to see whether -the report source matches the desired-state commit, then refresh the catalog -from a publish report only when that report proves the same target source -commit. If a later read-only runner cannot fetch catalog manifests from -`127.0.0.1:5000`, treat that as a `#66` registry reachability dimension, not as -a missing base-image or missing-publish blocker. - -## Known Implementation States - -- `repo-entrypoint`: the repository has `cmd//main.mjs`. -- `static-web-wrapper`: the cloud web static files are packaged behind a small - health/static-file wrapper. -- `repo-bundle`: the image packages repository-owned skill artifacts. -- `library-only`: the repository has library code but no executable package bin. -- `missing-runtime-entrypoint`: the image is only a health placeholder and is - not a real service implementation. - -## Known Source States - -- `source-present`: the service remains in the MVP artifact catalog and has a - repository-owned source path or entrypoint conclusion. -- `intentionally-disabled`: the service is explicitly out of the MVP artifact - publish set until it is given source. It must not remain a vague blocker. - -Runtime implementation blockers do not fake a service implementation. They are -recorded as the next minimum work needed after artifact publication. diff --git a/docs/dev-deploy-apply.md b/docs/dev-deploy-apply.md deleted file mode 100644 index 43f9cd7c..00000000 --- a/docs/dev-deploy-apply.md +++ /dev/null @@ -1,347 +0,0 @@ -# HWLAB DEV Deploy Apply - -This document records the DEV-only apply/preflight backend for -`pikasTech/HWLAB#33` and the apply-plan/report path for -`pikasTech/HWLAB#50`. Real apply side effects are normally entered through -`scripts/dev-cd-apply.mjs`, which sets the transaction environment and -serializes publish/apply/verify with the DEV CD Lease lock. The apply backend -targets only the D601 `hwlab-dev` namespace and must not be used for PROD. - -## Scope - -- Work from latest `origin/main`. -- Read only HWLAB repository deploy inputs: `deploy/deploy.json`, - `deploy/artifact-catalog.dev.json`, and `deploy/k8s/dev`. -- Use `scripts/deploy-desired-state-plan.mjs` as source/dry-run support to - review commit, image, tag, workload image, and env mirror convergence before - any apply review. -- Validate artifact identity, image commit tags, Kubernetes workload/service - shape, DEV health, and cloud-api DB connectivity. -- Validate that `hwlab-cloud-api` desired state declares the Code Agent - provider env contract: `OPENAI_API_KEY` from - `hwlab-code-agent-provider/openai-api-key` and - `HWLAB_CODE_AGENT_OPENAI_BASE_URL` through the DEV egress/proxy path. -- Emit an operator-readable plan in `devDeployApply`: plan/apply boundary, - expected artifact commit, target namespace, workloads, services, remaining - blockers, minimal manual commands, and rollback hints. -- Plan and, in apply mode, execute a scoped replacement for DEV suspended - template Jobs whose image changed but whose Kubernetes `spec.template` is - immutable. -- Plan and, in apply mode, execute a scoped cleanup for stale DEV simulator - Deployments `hwlab-gateway-simu` and `hwlab-box-simu` only when the source - manifest contains their indexed StatefulSet replacements. This keeps the M3 - DEV runtime at exactly two gateway-simu sessions and two box-simu resources. -- Stop before mutation when artifact publish evidence, registry digests, - `kubectl`, DEV health, cloud-api DB connectivity, or Code Agent provider - env preservation are missing. - -## Commands - -Run a preflight that is expected to report blockers on the current skeleton: - -```sh -node --check scripts/dev-deploy-apply.mjs -node --check scripts/src/dev-deploy-apply.mjs -node --check scripts/dev-runtime-migration.mjs -node --check scripts/src/dev-runtime-migration.mjs -node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check -node scripts/dev-runtime-migration.mjs --check -node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked -node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report /tmp/hwlab-dev-gate/report.json -node scripts/validate-dev-gate-report.mjs -``` - -Run the existing CLI dry-run surface: - -```sh -node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run -``` - -Only after CI publish evidence and executor prerequisites exist, a real DEV -CD transaction is: - -```sh -node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -The underlying apply backend still requires explicit non-production -confirmation and a transaction environment: - -```sh -node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -Direct `--apply` without `HWLAB_CD_TRANSACTION_ID` is rejected with -`cd-transaction-required`; this is a bypass guard, not a second lock. The -script refuses PROD flags, secret-read flags, heavyweight e2e flags, and -force-push flags. It also refuses to apply if the DEV artifact catalog still -contains skeleton-only unpublished images. - -`deploy-desired-state-plan.mjs` is not an apply preflight substitute. Pair it -with `artifact-runtime-readiness-guard.mjs` before apply review so stale -artifact catalogs, stale desired-state, stale API runtime identity, or a stale -Cloud Web `:16666` build stay explicit blockers. - -Kubeconfig selection is explicit and safe. The script uses this precedence: - -1. `--kubeconfig PATH` -2. `HWLAB_DEV_KUBECONFIG` -3. `KUBECONFIG` -4. `/etc/rancher/k3s/k3s.yaml` - -The selected path is used for read-only preflight, dry-run planning, and the -DEV-only apply path. The report records the kubeconfig source and path, but it -does not print kubeconfig contents or Secret values. - -Emergency D601 guard: for normal D601 execution, prefer -`/etc/rancher/k3s/k3s.yaml`. Docker Desktop Kubernetes has been disabled and -cleaned, but a residual default kubeconfig may still point at -`docker-desktop`/`127.0.0.1:11700`; do not pass that path through `KUBECONFIG` -or `--kubeconfig`. Any selected kubeconfig whose context/node resolves to -`docker-desktop` or `desktop-control-plane` is a blocker, not DEV-LIVE -evidence. See [pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138). - -Runtime DB migration is a separate DEV-only apply boundary. The source-only -check is safe for normal PR validation: - -```sh -node scripts/dev-runtime-migration.mjs --check -node scripts/dev-runtime-migration.mjs --dry-run --report /tmp/hwlab-dev-gate/report.json -``` - -Read-only live DB verification requires explicit DEV confirmation and an -already injected `HWLAB_CLOUD_DB_URL`; the report redacts the value and keeps -auth, schema, migration ledger, and readiness blockers separate: - -```sh -node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --report /tmp/hwlab-dev-gate/report.json -``` - -A real migration write is not part of default deploy validation. It requires a -separately authorized DEV operation: - -```sh -node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -That command may only apply -`internal/db/migrations/0001_cloud_core_skeleton.sql` to the DEV Postgres target -provided by `HWLAB_CLOUD_DB_URL`. It must not read Kubernetes Secret resources, -print the DB URL/password/token, mutate PROD, restart services, or claim -M3/M4/M5 acceptance. - -`deploy-desired-state-plan.mjs` is not an apply preflight substitute. It is a -read-only source planner and does not prove registry image existence, DEV -cluster pullability, DEV apply, service restart, or M3 DEV-LIVE evidence. - -## Report Contract - -The machine-readable result is written to -`/tmp/hwlab-dev-gate/dev-deploy-report.json` when `--report /tmp/hwlab-dev-gate/report.json` is used. The -report remains compatible with the #33 DEV gate validator and adds these -operator fields: - -- `devDeployApply.conclusion`: `blocked` while blockers are open, otherwise - `ready` for human approval. -- `devDeployApply.applyBoundary`: default no-write behavior, required apply - flags, DEV-only write scope, no-write scope, and forbidden actions. -- `devDeployApply.artifactPlan`: expected artifact commit, source commit, - deploy/catalog commit IDs, publish/registry status, and unpublished services. -- `devDeployApply.target`: DEV endpoint and `hwlab-dev` namespace. -- `devDeployApply.workloadPlan` and `servicePlan`: planned Kubernetes - workloads, containers, images, service IDs, ports, and replica intent. -- `devDeployApply.templateJobReplacementPolicy`: the DEV-only allowlist for - suspended template Job replacement. The only allowlisted Jobs are - `hwlab-agent-worker-template` and `hwlab-cli-template` in `hwlab-dev`. -- `devDeployApply.templateJobReplacements`: per-Job replacement decisions with - namespace, Job name, old image, new image, and result (`planned`, `replaced`, - `not_needed`, `not_found`, or blocked/failure states). -- `devDeployApply.legacySimulatorDeploymentCleanupPolicy`: the DEV-only - allowlist for stale simulator Deployment cleanup after the M3 simulator - runtime moved to indexed StatefulSets. -- `devDeployApply.legacySimulatorDeploymentCleanups`: per-Deployment cleanup - decisions for `hwlab-gateway-simu` and `hwlab-box-simu`, including old/new - images and result (`planned`, `deleted`, `not_found`, or blocked/failure - states). -- `devDeployApply.cloudWebRollout`: Cloud Web source commit, image tag, image - reference, digest, live image, and Kubernetes rollout revision when read-only - Deployment access is available. If access is missing, this field records the - read/rollout commands and the blocker instead of claiming a live rollout. -- `devDeployApply.codeAgentProvider`: no-secret provider validation gate. It - records desired-state env/ref readiness plus live `hwlab-cloud-api` - Deployment env preservation before apply and, in apply mode, after apply. - This gate reads Deployment env metadata only; it must not read Kubernetes - Secret data or attempt a provider call. -- `devDeployApply.manualCommands`: dry-run/report commands before human - approval; the apply command is present only when the plan is ready. -- `devDeployApply.rollbackHint`: pre-apply capture commands, deployment - rollbacks, suspended job cleanup, and post-rollback health checks. -- `devDeployApply.remainingBlockers`: open blockers with concrete unblock - hints. These do not claim live apply happened. - -When `conclusion.status` is `blocked`, operators should clear the listed -`remainingBlockers` and rerun: - -```sh -node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report /tmp/hwlab-dev-gate/report.json -node scripts/validate-dev-gate-report.mjs -``` - -When `conclusion.status` is `ready`, run the dry-run plan and validator -immediately before human approval: - -```sh -node scripts/dev-deploy-apply.mjs --dry-run --report /tmp/hwlab-dev-gate/report.json -node scripts/validate-dev-gate-report.mjs -``` - -Only after that approval may the DEV CD transaction command be used. - -## Cloud Web Manual DEV Rollout - -This is the currently verified manual CD path for `hwlab-cloud-web` on D601. -It is the operational baseline that future CLI automation should reproduce from -`deploy/deploy.json`; do not rediscover ports or kube contexts during an -incident. - -Start from the D601 deployment copy, not a runner worktree: - -```sh -cd /home/ubuntu/workspace/hwlab -git pull --ff-only origin main -``` - -Validate source in a clean checkout, then refresh the static bundle before -artifact publish. Cloud Web images serve `web/hwlab-cloud-web/dist` first, so -stale ignored build output can keep an old English or old layout bundle alive -even after the source commit is correct. `check.mjs` does not require ignored -`dist/`; `build.mjs` and `dev-artifact-publish.mjs` are the build-first -freshness gates. - -```sh -node web/hwlab-cloud-web/scripts/check.mjs -node web/hwlab-cloud-web/scripts/build.mjs -node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -Probe the image locally before touching k3s: - -```sh -docker run --rm -d --name hwlab-cloud-web-probe -p 127.0.0.1:18088:8080 127.0.0.1:5000/hwlab/hwlab-cloud-web: -sleep 1 -curl -fsS http://127.0.0.1:18088/health/live -curl -fsS http://127.0.0.1:18088/ | sed -n '1,40p' -docker rm -f hwlab-cloud-web-probe -``` - -Use the D601 native k3s kubeconfig explicitly: - -```sh -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web: -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide -``` - -Do not use the default kube context as the source of truth. On D601 it may be -`docker-desktop`, which is not the public DEV runtime behind `16666/16667`. - -Public acceptance must verify all of these: - -```sh -curl -fsS http://74.48.78.17:16666/health/live -curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p' -curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh' -curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p' -``` - -For the Cloud Workbench Chinese/no-outer-scroll rollout, the accepted target -state was: - -- `GET /health/live` reports revision - `1e8805664970839b72be40c34636b08f6d18b131`; -- `GET /` returns `html lang="zh-CN"` and title `HWLAB 云工作台`; -- CSS locks the outer shell with `html, body { overflow: hidden; }` and - `.workbench-shell { height: 100dvh; overflow: hidden; }`; -- `GET /help.md` returns the Chinese Cloud Workbench help document. - -The public path is: - -1. master `hwlab-frps-dev` listens on `16666/16667`; -2. D601 `hwlab-frpc` runs in `hwlab-dev` from the `hwlab-frpc-config` - ConfigMap; -3. `hwlab-dev-cloud-web` maps to - `hwlab-cloud-web.hwlab-dev.svc.cluster.local:8080`; -4. `hwlab-dev-edge-proxy` maps to - `hwlab-edge-proxy.hwlab-dev.svc.cluster.local:6667`. - -Master `hwlab-browser-proxy-dev` is only an auxiliary local proxy on `18087`; -it is not the public browser endpoint. - -## Suspended Template Job Replacement - -Kubernetes Jobs cannot update `spec.template` in place. For DEV template Jobs -that are intentionally suspended and used as templates, an image tag change -would otherwise make a normal `kubectl apply -k deploy/k8s/dev` non-idempotent. - -The apply script handles only the explicit DEV allowlist: - -- `hwlab-agent-worker-template` -- `hwlab-cli-template` - -For each allowlisted Job, the script reads the live Job in `hwlab-dev` and -compares its container image with the desired manifest image. If the live Job -exists, is suspended, and the image differs, apply mode deletes that Job before -running the normal kustomize apply so Kubernetes recreates it from the desired -manifest. Dry-run mode records the same decision as `planned` and runs -server-side kustomize dry-run without mutating the cluster. If that dry-run -reports only the expected immutable `spec.template` errors for the planned -allowlisted Jobs, the report records those errors as expected replacement -evidence instead of treating them as blockers. - -The policy refuses any other Job name, any non-`hwlab-dev` namespace, and any -live Job that is not suspended. It does not change Deployment, Service, -ConfigMap, Secret, or PROD behavior. - -## Current Result - -The current DEV apply contract is published for artifact commit `cb35ada`. -`deploy/artifact-catalog.dev.json`, `deploy/deploy.json`, and the DEV -workloads point at `127.0.0.1:5000/hwlab/*:cb35ada` images with sha256 digest -evidence. - -On Code Queue runners, the approved mounted kubeconfig path is typically -`/var/lib/unidesk/code-queue/kubeconfig`. Export `HWLAB_DEV_KUBECONFIG` or -pass `--kubeconfig` explicitly when the default D601 path is not available: - -```sh -HWLAB_DEV_KUBECONFIG=/var/lib/unidesk/code-queue/kubeconfig node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report /tmp/hwlab-dev-gate/report.json -``` - -The D601 runner can read and apply the `hwlab-dev` namespace. A real DEV apply -may replace only the allowlisted suspended template Jobs above; the normal -Deployment, Service, ConfigMap, and health-contract apply path remains -unchanged. - -`hwlab-cloud-api` has a DEV DB env contract placeholder: -`HWLAB_CLOUD_DB_URL` must come from Secret reference -`hwlab-cloud-api-dev-db/database-url`, and -`HWLAB_CLOUD_DB_SSL_MODE=disable`. Runtime readiness dials the redacted host -parsed from that Secret URL. `cloud-api-db` remains an optional desired alias, -not a readiness gate or TCP probe target, until this repo owns its -Service/Endpoint rollout contract. The local health gate reports DB runtime -config without printing the connection string. - -Durable runtime migration now has a repo-owned path: -`scripts/dev-runtime-migration.mjs --check` proves the source migration and -ledger contract without DB access; `--dry-run --allow-live-db-read ---confirm-dev` verifies live auth/schema/migration/readiness without writes; -`--apply --confirm-dev --confirmed-non-production` is the only DEV DB write -mode and remains outside default PR validation. - -No PROD resource, secret value, UniDesk runtime substitute, browser e2e, -heavyweight e2e, or force push is part of this flow. - -Source plans, dry-run reports, and desired-state convergence checks are M3 -support only. They must not be labelled DEV-LIVE without a real DEV observation -of the `DO1 -> patch-panel -> DI1` loop plus operation, trace, audit, and -evidence records. diff --git a/docs/m2-dev-deploy-smoke.md b/docs/m2-dev-deploy-smoke.md deleted file mode 100644 index 88caf788..00000000 --- a/docs/m2-dev-deploy-smoke.md +++ /dev/null @@ -1,90 +0,0 @@ -# M2 DEV Deployment Smoke - -`m2-dev-deploy-smoke` is a local dry-run smoke harness for the HWLAB DEV -deployment shape. It validates the frozen route phases, service identities, -artifact metadata, and safety gates without contacting the real DEV ingress by -default. - -Run the dry-run from the repository root: - -```sh -node scripts/m2-dev-deploy-smoke.mjs --dry-run -``` - -## What It Checks - -- Parses `fixtures/dev-deploy-smoke/dev-deploy-smoke.json`. -- Verifies the route phases in order: - `master-edge-proxy`, `frp-tunnel`, `d601-router`, `cloud-surface`, - `agent-runtime`, and `sim-and-patch-panel`. -- Validates the frozen HWLAB service IDs for master edge proxy, frp tunnel, - D601 `hwlab-dev/hwlab-router`, cloud-api/web, agent, sim, and patch-panel. -- Checks `commitId`, `image`, `digest` or `not_applicable`, `tag`, - `buildSource`, `deployEnv`, `healthTimestamp`, and failure classification - for every recorded service contract. -- Confirms the cloud surface bundles `hwlab-cloud-api` and `hwlab-cloud-web` - artifacts together under the same route phase. -- Marks the fixture as dry-run evidence only. Any DB fields in fixture health - records are contract placeholders, not live DB evidence. -- Keeps the default mode in dry-run and refuses real requests unless an - explicit live gate is supplied. - -## Live Gate - -The live gate is intentionally strict. It is only for a future real DEV smoke -observation, not for deployment, and this harness stays fixture-only unless the -operator supplies the live flags. The public DEV endpoint split is: - -- browser/frontend: `http://74.48.78.17:16666/` -- API/edge health: `http://74.48.78.17:16667/health` and - `http://74.48.78.17:16667/health/live` -- legacy public `:6667`: deprecated and never accepted as active green evidence - -Required flags: - -```sh -node scripts/m2-dev-deploy-smoke.mjs --live --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -Before using `--live`, confirm all of the following: - -- the API/edge target remains `http://74.48.78.17:16667`; -- the frontend target remains `http://74.48.78.17:16666`; -- the workspace is not a PROD or shared deployment context; -- no prohibited action is being attempted; -- the request is observational only and does not mutate runtime state. - -The script rejects live execution if any of these are missing: - -- `--confirm-dev` -- `--confirmed-non-production` -- `--prod` -- `--heavyweight-e2e` -- `--read-secret` -- `--force-push` - -When `--report /tmp/hwlab-dev-gate/report.json` is provided, the script writes -`/tmp/hwlab-dev-gate/dev-m2-deploy-smoke-active.json`. The report contains only -read-only GET evidence and marks legacy `:6667` observations as ineligible for -active green. - -Validate the written report with: - -```sh -node scripts/validate-m2-deploy-smoke-active.mjs -``` - -## Boundary - -This harness does not: - -- deploy to DEV or PROD; -- call the real DEV endpoint in dry-run mode; -- run heavy e2e coverage; -- read secrets or tokens; -- replace HWLAB runtime services with UniDesk substitutes; -- touch L5 deploy manifests. - -The fixture is a dry-run record only. It exists to prove that the M2 smoke -entry point understands the frozen route and artifact contract before any live -DEV observation is attempted. diff --git a/docs/reference/cloud-workbench.md b/docs/reference/cloud-workbench.md index 41e455d9..05c060be 100644 --- a/docs/reference/cloud-workbench.md +++ b/docs/reference/cloud-workbench.md @@ -262,5 +262,5 @@ blockers/failures：`viewport`、`selector`、`failureType`、summary 和 artifa chat 同源 readiness 和 provider credential blocker 边界。 - [docs/cloud-web-workbench.md](../cloud-web-workbench.md): detailed frontend contract. -- [docs/dev-deploy-apply.md](../dev-deploy-apply.md): Cloud Web manual rollout +- [g14-gitops-cicd.md](g14-gitops-cicd.md): Cloud Web rollout through G14 Tekton, GitOps and Argo CD path. diff --git a/docs/reference/deployment-publish.md b/docs/reference/deployment-publish.md deleted file mode 100644 index d678cf0f..00000000 --- a/docs/reference/deployment-publish.md +++ /dev/null @@ -1,821 +0,0 @@ -# HWLAB Deployment And Publish Reference - -This reference records the stable DEV publish and rollout rules. It does not -authorize PROD, service restarts outside the documented DEV path, secret reads, -or direct pushes to `main`. - -It is also the long-term reference for -[pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116): service -deployment regularization must progress from stable docs, to controlled -CLI/script entrypoints, to UniDesk CI/CD plus imageized delivery. - -## D601 Native k3s Emergency Guard - -HWLAB DEV deploy and publish paths must target D601 native k3s only. Docker -Desktop Kubernetes on D601 has been disabled and cleaned; it must not be -re-enabled or used as a second orchestrator. The cross-repo incident, evidence, -and governance plan are tracked in -[pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138), and -the 2026-05-23 commander recovery context is in -[pikasTech/unidesk#118](https://github.com/pikasTech/unidesk/issues/118). - -Every DEV CD status, apply, rollback, smoke, and manual Kubernetes command must -use `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` and verify node `d601` before any -mutation or acceptance. Bare `kubectl`, `docker-desktop` context, -`desktop-control-plane`, or `127.0.0.1:11700` are wrong-control-plane signals, -not HWLAB DEV-LIVE evidence. A second `hwlab-dev` control plane, including a -Docker Desktop cluster that happens to contain similarly named resources, is a -blocker and cannot be used as deploy, rollout, smoke, or acceptance proof. - -## Workspaces - -| Purpose | Path | -| --- | --- | -| Runner and commander worktree | `/workspace/hwlab` | -| D601 shared source cache | `/home/ubuntu/workspace/hwlab` | -| D601 clean CI/CD worktree | `/tmp/hwlab-cicd-` | -| D601 runtime experiment worktree | `/tmp/hwlab--` | - -Git remote is the source of truth. D601 may contain several HWLAB clones or -worktrees from earlier reviews, Code Queue runs, runtime experiments, or CI/CD -jobs; none of those local directories is authoritative by itself. Before using -any D601 repo path for checks, publish, or CD, verify its role, `origin`, HEAD, -and cleanliness. - -`/home/ubuntu/workspace/hwlab` is the shared source cache and convenient seed -clone. It is allowed to be stale, behind `origin/main`, or dirty with generated -`reports/dev-gate/**` files from previous runs. Do not `reset --hard`, delete -reports, or overwrite that directory just to make a release command work. If it -is not clean and already at the target commit, use it only to `git fetch` and -create a detached clean worktree. - -Formal CI publish and DEV CD apply must run from a clean, target-pinned -worktree such as `/tmp/hwlab-cicd-`. The worktree may be -created from the shared cache, but it must prove all of the following before -side effects: - -- `git remote get-url origin` points to `git@github.com:pikasTech/HWLAB.git` - or the approved GitHub source of truth. -- `git fetch origin main` succeeds, and `git rev-parse HEAD` equals the target - commit being published or deployed. -- `git status --short` is empty. -- The `hwlab-cli cicd submit ...` job `cwd` in `state.json` points to the clean - worktree, not to a stale runner directory or a dirty shared cache. - -Runtime hotfix and gateway probing may use purpose-specific D601 worktrees under -`/tmp/hwlab--`. Those worktrees are for copying scripts into pods, -running probes, and staging temporary patches; they are not release truth and -must not be used to infer desired state unless they are also clean and pinned to -the target commit. - -Do not treat `/home/ubuntu/hwlab`, historical runner directories, previous PR -review worktrees, or pod-local copied trees as publish/build truth. If a path's -role is unclear, stop and create a new detached worktree from `origin/main` -instead of repairing the unknown directory in place. - -## Master Server Check Boundary - -Master server is only a control, coordination, Git, issue/PR, and short polling -entrypoint for HWLAB. It must not run HWLAB validation workloads locally: -`npm run check`, `npm run web:check`, Playwright/browser smokes, full test -suites, build/preflight jobs, image publishing, or DEV CD checks belong on -D601, Code Queue runners, or the formal CI/CD job plane. - -The master-side HWLAB clone may be used for lightweight source reading, -`git status`, `git diff`, patch preparation, commits, PR creation, and polling -D601/CI/CD status. If a validation command would consume significant CPU, -memory, browser resources, Docker, k3s, registry, or long-lived process time, -skip it on master and run the equivalent command from a clean D601 worktree or -submit it through `hwlab-cli cicd submit`. - -Master wrappers may start short-lived D601 commands and poll their status, but -the validation work itself must execute on D601 or CI/CD. A master-local -timeout, browser hang, or resource shortage is not DEV-LIVE evidence and must -not be turned into a product blocker; stop the local check and move the -verification to D601/CI/CD. - -## Regularization Stages - -| Stage | Current HWLAB state | Required next behavior | -| --- | --- | --- | -| 1. Long-term reference | This file records DEV deployment, Cloud Web publish, API/edge/health, frp, D601 registry, k3s rollout, rollback, and verification rules. `AGENTS.md` indexes it. | Keep process issues as sources only; update this reference when a deployment rule changes. | -| 2. CLI or controlled scripts | `hwlab-cli cicd submit/status/logs/report` 是正式 CI/CD 外层控制入口；`scripts/dev-artifact-publish.mjs` 和 `scripts/dev-cd-apply.mjs` 只作为后台 worker 或只读状态脚本使用。Other controlled scripts still cover contract planning, dry-run planning, preflight, single-step internals, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. | Continue converging check, plan, rollback, status, and verify into HWLAB CLI while preserving CI/CD separation and the single CD transaction lock backed by `deploy/deploy.json`. | -| 3. UniDesk CI/CD plus images | DEV services are imageized and published to the D601 registry. D601 rollout still includes manual bridge steps for Cloud Web and frp/k3s execution. | Move build, publish, desired-state refresh, rollout, post-deploy validation, and report upload into standard UniDesk CI/CD jobs without replacing HWLAB runtime. | - -## CI/CD CLI 非阻塞合同 - -HWLAB CI/CD 正式 CLI 必须遵循 `Skill(cli-spec)`：可能超过短时间的 build、 -push、apply、rollout、runtime Job、smoke 和 postflight 都不得以前台阻塞命令 -暴露给指挥官、runner 或 master-side wrapper。正式入口必须使用 -Fire-and-Forget 模式： - -```sh -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind ci-publish --concurrency 4 -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4 -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd logs --tail-bytes 8000 -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd report -``` - -`submit` 只创建 `.state/hwlab-cicd/jobs//`，写入 `spec.json`、 -`state.json`、`stdout.log`、`stderr.log` 和 `report.json` 路径，然后启动后台 -worker 并立即返回有界 JSON。`status` 仅读 `state.json`，`logs` 默认只返回 -尾部摘要，`report` 默认只返回报告摘要；完整日志和完整报告通过返回的文件路径 -渐进查看。任何正式 CLI 调用都不得因为 Docker build、registry push、 -`kubectl wait --timeout` 或 rollout status 在前台长时间占用上下文。 - -runner pod 内执行 CD 时，`127.0.0.1:5000` 指向 pod 自身，不等于 D601 host 的 -loopback registry；但 image reference 必须继续保持 `127.0.0.1:5000/...`，因为 -k3s/containerd 拉镜像和 CI push 使用的是 host 视角。CD registry manifest 校验 -默认先走直接 HTTP；如果 pod loopback 被拒绝，则用 Docker host-network helper -（`docker run --rm --pull=never --network host`）读取同一个 registry manifest。 -正式命令仍必须通过非阻塞入口提交；如需显式 registry HTTP 入口，可在提交时加 -`--registry-manifest-base-url `，该参数只改变 manifest 校验 URL，不改变 -`deploy.json` 或 workload 里的镜像引用。 - -`scripts/dev-artifact-publish.mjs`、`scripts/dev-cd-apply.mjs` 和其他内部脚本仍 -保留为 worker/backend：它们可以在后台 job 内执行长流程，但不能作为指挥面常规 -入口直接运行。临时手动排障可以直接调用底层脚本，但排障结论必须回填到正式 -CLI、长期参考文档或 issue，避免把一次性命令固化为新流程。 - -## D601 Code Queue runner 实测调试法 - -HWLAB CI/CD 的关键验收必须从真正使用者 runner 的容器或 pod 内复测。master -server、指挥官本地 clone、D601 host 普通目录和历史 runner 目录只能作为控制面 -或辅助观察面；它们不能替代当前 Code Queue 执行面看到的 worktree、网络、 -kubeconfig、registry、Docker socket、Secret/ConfigMap mount 和 CLI 行为。 - -实地调试顺序固定为： - -1. 先从调度器、任务、pod 和进程信息确认当前承担生产调度的 Code Queue runner - 或 scheduler 容器，避免误入已经废弃、空闲或历史测试容器。 -2. 通过受控维护入口进入该容器，在容器内建立干净 HWLAB worktree，并快进到 - `origin/main`；不得直接使用 `/home/ubuntu/hwlab` 等历史任务集合目录作为 - 发布真相。 - 如果 worktree 是从另一个本地目录 clone 出来的，必须先确认 `origin` 指向 - `git@github.com:pikasTech/HWLAB.git` 或等价 GitHub source of truth，再 - `git fetch origin main`。DEV CD 的 target-ref guard 会比较 HEAD 与 - `origin/main`；本地 clone 的 stale `origin/main` 会触发 - `target-ref-head` blocker，不能通过跳过 guard 处理。 -3. 在同一个容器内执行 `node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit ...`， - 用 `status`、`logs`、`report` 查询后台 job，而不是前台直接跑长脚本。 -4. 至少完成一次 CI artifact publish 和一次 DEV CD apply，再用公开 - `16666/16667` health 核对 live commit、service readiness 和 blocker 字段。 -5. 调试结论必须写回 issue、正式 CLI、preflight 或长期参考文档；不能把一次性 - 容器命令当作新的常规发布流程。 - -runner 内最小复测命令形态如下： - -```sh -cd -git fetch origin main -git checkout main -git pull --ff-only origin main -git status --short - -KUBECONFIG=/etc/rancher/k3s/k3s.yaml \ - node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind ci-publish --concurrency 4 -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd report - -KUBECONFIG=/etc/rancher/k3s/k3s.yaml \ - node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4 -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd report -``` - -一次 runner 实测只有同时满足以下条件，才可判断 CI/CD 链路通畅： - -- `ci-publish` 和 `dev-cd-apply` job 都进入终态 `succeeded`； -- job report 中 `blockers` 为空，耗时、服务数、镜像 tag/digest 和 target commit - 有界展示； -- `Lease/hwlab-dev/hwlab-dev-cd-lock` 在事务结束后释放或显示为无活动 holder； -- 公开 `16666` 和 `16667` health 的 commit 与 `deploy/deploy.json` promotion - commit 一致； -- runner 容器内没有残留 `hwlab-cli`、`dev-cd-apply` 或旧发布 worker 进程继续 - 持有写路径。 - -本次 CI/CD 疏通暴露出的稳定故障类别也必须长期前置： - -- `127.0.0.1:5000` 在 runner pod 内、Docker host-network helper 内和 k3s - containerd 内含义不同。镜像引用仍使用 `127.0.0.1:5000/...` 作为 D601 host - registry identity；registry manifest 校验按运行视角选择直接 HTTP 或 Docker - host-network helper，不得为了 pod 视角改写 release identity。 -- Codex stdio 所需 `auth.json` Secret 和 `config.toml` ConfigMap 是发布前置 - 条件。preflight 只检查对象和 key 名，不读取 secret value；ConfigMap 属于 - repo desired-state，Secret 属于外部环境。 -- 已经恢复的 live 服务不能替代正式 CD 报告。若第一次 CD 因缺少 mount、 - registry 视角或权限失败，即使手动恢复了 live，也必须再次从 runner 内走 - 非阻塞 DEV CD job，产出通过报告。 -- CI 产物事实以 registry manifest 和 artifact catalog 为准；临时 - `/tmp/hwlab-dev-gate/*.json` 只用于审计和耗时排障，不得成为 CD 放行硬依赖。 -- `kubectl apply -k` 不会自动移除由运行面 `kubectl patch`、临时 ConfigMap - 热修或其他 field manager 写入、且不属于 desired manifest / last-applied 的 - Deployment `volumes`、`volumeMounts` 或 template annotations。正式 CD 不能把 - “镜像 tag 已更新”误判为“热修覆盖已撤销”。DEV CD apply 必须在 apply 前比对 - desired Deployment 与 live Deployment：若 live 存在 desired 不包含、且明确带 - `hotfix`/`override` 语义的额外覆盖字段，应在 `hwlab-dev` 内删除该 desired - Deployment，再由同一次 `kubectl apply -k deploy/k8s/dev` 从源码 desired-state - 重新创建；报告中记录 cleanup 计划和实际动作。不要把这种 cleanup 做成某个服务 - 或某个 skill 的特例。 -- Kubernetes service links 会为同 namespace Service 自动注入形如 - `HWLAB_CLOUD_API_PORT=tcp://...` 的环境变量。应用入口不得直接把这类 env 当作 - 数字端口解析；desired-state 应显式声明服务自身监听端口 - `HWLAB_CLOUD_API_PORT=6667`，入口代码也必须只接受合法数值端口，否则回落到 - 内置默认值。 - -## DEV CD Transaction - -正式 DEV CD 写路径只有一个外层入口： - -```sh -node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4 -``` - -该命令后台调用 `scripts/dev-cd-apply.mjs`，是唯一允许按 `deploy.json` 对 `hwlab-dev` 执行 desired-state apply、并复验 -公开 `16666/16667` 的正式 CD 写路径。镜像发布属于 CI artifact 职责；过渡期 -latest-main 兼容路径仍可由该命令调用 artifact publish，但正式 rollout 默认应消费 -已经发布并经过 registry manifest 验证的 artifact。 -事务开始时读取 `deploy/deploy.json`，记录 manifest hash 和 target commit。 -在获取 `Lease/hwlab-dev/hwlab-dev-cd-lock` 或运行任何 publish、runtime Job、 -apply、rollout 类副作用前，脚本必须先完成 apply-only preflight：强制 -`KUBECONFIG=/etc/rancher/k3s/k3s.yaml`，验证节点包含 `d601`，拒绝 -`docker-desktop`、`desktop-control-plane`、`127.0.0.1:11700`、裸 -`kubectl` 指向第二套 `hwlab-dev` 控制面或其他不可信控制面信号，并且只验证 -必要 SecretRef 的存在性和 key 名。通过后才获取 Lease、串行执行内部步骤、 -按需写入 `/tmp/hwlab-dev-gate/dev-cd-apply.json`，最后释放 Lease。 - -apply preflight 检查的 SecretRef 固定为： - -- `hwlab-cloud-api-dev-db/database-url` -- `hwlab-cloud-api-dev-db-admin/admin-url` -- `hwlab-code-agent-provider/openai-api-key` -- `hwlab-code-agent-codex-auth/auth.json` - -Code Agent Codex stdio 还必须在 preflight 阶段验证非 secret ConfigMap -`hwlab-code-agent-codex-config/config.toml` 的存在性和 key 名；该 ConfigMap 由 -`deploy/k8s/base/code-agent-codex-config.yaml` 纳入 DEV desired-state。Codex -auth 仍然只能作为外部 Secret 管理，禁止把 `auth.json` 内容写进仓库或报告。 - -preflight 和报告只能记录 Secret 名、key 名、存在性、key-name 观察结果和脱敏 -命令摘要；不得读取或打印 Secret value。阻塞时输出结构化 blocker，包含 -`scope`、`reason`、`impact`、`safeNextAction` 和 `retryable`，默认 stdout 只 -给有界摘要，完整 preflight 证据写入事务报告。 - -`deploy/deploy.json` 的 `commitId` 是运行时 promotion commit。正常 -latest-main 发布时它可以等于当前 target ref；artifact merge commit 或后续 -控制面提交存在时，它也可以指向一个已经发布并在 catalog/workloads 中完全收敛 -的早期 promotion commit。`scripts/dev-cd-apply.mjs` 必须先用 -`deploy-desired-state-plan --promotion-commit ` 验证该 desired-state -已经发布且内部一致；验证通过后，`--apply` 消费这个 promotion commit 的镜像 -执行 apply/verify，不重新 publish 或刷新当前 HEAD 的 artifact。这样 host -commander 可以始终从最新 `origin/main` 控制入口执行正式 DEV CD，而不需要 -checkout promotion parent 或手工拼装发布环境。 - -状态面和事务报告会显式展示这条边界：`target.controlRef` 是最新 -`origin/main` 控制入口，`target.promotionCommit` 是从 `deploy/deploy.json` -解析出的真实 promotion commit，`target.artifactBoundary` 记录最近一次触碰 -`deploy/deploy.json`、`deploy/artifact-catalog.dev.json`、 -`deploy/k8s/base/workloads.yaml` 的 first-parent desired-state commit，以及该 -commit 的 promotion parent。`artifactCatalog`、`artifactReport`、`deployJson` -分别记录 hash、commit、digest 计数和 publish source，但只有 -`deploy/deploy.json`、artifact catalog、workload manifest 和 registry manifest -共同决定 CD 是否可执行；`/tmp/hwlab-dev-gate/dev-artifacts.json` 只是 CI 审计快照。 -若 deploy/catalog/workloads 或 desired-state parent 不能同时指向同一个 -promotion commit，状态降级为 `degraded`，真实 `--apply` 在获取 Lease 前以 -artifact-boundary blocker 停止。缺失或过期的 CI 产物报告只能产生诊断提示，不得 -阻塞已经由 desired-state 和 registry 证明一致的 CD rollout。这保证 -`deploy/deploy.json` 仍是唯一发布控制源；Lease 只串行化事务，不保存第二套 -desired-state。 - -Lease 只是发布平面的互斥锁，不是 desired-state 来源。Lease annotations -至少记录 `promotionCommit`、`deployJsonHash`、`ownerTaskId`、 -`transactionId`、`phase`、`startedAt`、`updatedAt`、`ttlSeconds`、 -`liveBefore` 和 `targetNamespace`。`deploy/deploy.json`、 -`deploy/artifact-catalog.dev.json`、`deploy/k8s/base/workloads.yaml` 仍是唯 -一权威部署输入。 - -如果另一个事务持有 Lease，`--apply` 必须在 publish/apply 之前停止，并 -输出结构化 `deploy-lock-held`，包含 holder、promotion commit、phase、 -target namespace 和 retry seconds。过期锁不能静默覆盖；stale lock 只能 -在确认 holder 已失活后显式使用 `--break-stale-lock --confirm-dev`，并把 -previous holder 保留在审计 annotations/report 字段中。未过期 Lease 没有 -抢占入口，普通 status 输出也不得建议强制抢锁。 - -`scripts/dev-cd-apply.mjs` 同时提供只读状态面。无参数、`--status`、或 -`--dry-run` 都返回紧凑 JSON，覆盖 target ref/commit、 -`deploy/deploy.json` commit/hash、当前 Lease 状态摘要、以及公开 -`16666/16667` live health identity 摘要。状态面还包含 `liveDelta`，用于对比 -当前公开 live commit 与 `target.promotionCommit`；这只是 rollout 口径，不会在 -status/dry-run 中执行 rollout。这些模式 host commander 和 runner 都可以执行， -但必须保持只读：不获取或替换 Lease，不 publish artifact，不执行 -`kubectl apply/rollout`，不写 report，不修改 `deploy.json`/catalog，也不读取或 -打印 Secret value。kubectl 或 kubeconfig -不可用时，状态面应返回 `lock.status=unavailable` 和脱敏原因，而不是把 -只读观测失败升级成写路径 blocker。 - -## `deploy.json` DEV CD 路径 - -当前已经验证可用的过渡路径是：master server 只作为受控指挥入口，通过 -UniDesk CLI 进入 D601；D601 上使用干净 HWLAB 工作区运行本仓库 -`scripts/dev-cd-apply.mjs`；发布目标由 `deploy/deploy.json` 和同 commit -的 artifact catalog / k8s workloads 决定；真实写入只落到 D601 原生 k3s -`hwlab-dev` namespace。复盘来源见 -[pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61#issuecomment-4525401375)， -长期化跟踪见 [pikasTech/HWLAB#340](https://github.com/pikasTech/HWLAB/issues/340)， -单事务入口和发布锁见 [pikasTech/HWLAB#274](https://github.com/pikasTech/HWLAB/issues/274)， -D601 控制面归一见 [pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138)。 - -稳定调用合同如下；外层 master CLI 或 UniDesk wrapper 只能包裹这些 -repo-owned 入口，不能绕过它们直接拼接 publish、apply、rollout 或 smoke： - -| 目的 | 稳定入口 | 副作用边界 | -| --- | --- | --- | -| 读取 DEV CD 状态 | `node scripts/dev-cd-apply.mjs --status` | 不写入、不获取 Lease、不读取 Secret value。 | -| 事务 dry-run preflight | `node scripts/dev-cd-apply.mjs --dry-run` | 不 publish、不 apply、不 rollout。 | -| 检查 desired-state 收敛 | `node scripts/deploy-desired-state-plan.mjs --promotion-commit --check` | 只读 source/report。 | -| 提交 CI artifact publish | `node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind ci-publish --concurrency 4` | 立刻返回 jobId；后台 build/push，报告写入 `.state/hwlab-cicd/jobs//report.json`。 | -| 提交 DEV CD 事务 | `node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4` | 立刻返回 jobId；后台在 `Lease/hwlab-dev/hwlab-dev-cd-lock` 下 apply/verify；legacy latest-main publish 仅作过渡兼容。 | -| 查询 CI/CD job | `node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status|logs|report ` | 只读 `.state/hwlab-cicd/jobs//`；日志和报告默认有界摘要。 | -| 读取 D601 k3s 证据 | `node scripts/d601-k3s-readonly-observability.mjs` | 只读可见性报告；不 rollout、不输出 Secret value。 | - -使用 master-side wrapper 时，稳定形态是： - -```sh -cd /root/unidesk -bun scripts/cli.ts ssh D601 argv bash -lc 'cd && KUBECONFIG=/etc/rancher/k3s/k3s.yaml node scripts/dev-cd-apply.mjs --status' -bun scripts/cli.ts ssh D601 argv bash -lc 'cd && KUBECONFIG=/etc/rancher/k3s/k3s.yaml node tools/hwlab-cli/bin/hwlab-cli.mjs cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4' -``` - -`` 必须指向干净工作区。该工作区可以是固定 clean -mirror，也可以是一次性 `git clone --shared --no-checkout` 生成的 ephemeral -worktree；它不能是 `/home/ubuntu/hwlab` 这类 runner/历史任务集合目录。进入 -apply 前必须证明： - -- `git status --short` 为空； -- `.git/FETCH_HEAD` 和 `.git/worktrees` 没有阻止普通部署用户读取或创建 - 干净工作区的权限问题； -- `node scripts/dev-cd-apply.mjs --status` 显示 `promotionSource=deploy-json`； -- `deploy/deploy.json` 的 commit、`deploy/artifact-catalog.dev.json` 和 - `deploy/k8s/base/workloads.yaml` 收敛到同一个 promotion commit / image tag； -- `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes` 能看到节点 `d601`； -- `kubectl config current-context`、cluster server 或节点列表没有 - `docker-desktop`、`desktop-control-plane`、`127.0.0.1:11700` 或第二套 - `hwlab-dev` 控制面证据； -- `Lease/hwlab-dev-cd-lock` 未被未过期事务持有，或者 stale lock 已由受控 - `--break-stale-lock --confirm-dev` 审计处理。 - -desired-state 来源关系固定如下： - -| 来源 | 权威范围 | -| --- | --- | -| `deploy/deploy.json` | DEV environment、namespace、endpoint、service set、promotion commit 和事务消费的 image tag。 | -| `deploy/artifact-catalog.dev.json` | 同一 promotion commit 的不可变 artifact digest/tag 期望；CD 必须直接向 registry 校验 tag/digest 是否真实存在。 | -| `deploy/k8s/base/workloads.yaml` | Kubernetes workload desired state，必须镜像 `deploy.json` 和 artifact catalog 后才能 apply。 | -| Registry manifest | 镜像事实源；回答目标 tag/digest 是否存在，不能决定哪一组服务应该上线。 | -| `/tmp/hwlab-dev-gate/*.json` | 只作为 CI/CD 审计、排障和耗时统计快照；不能覆盖 repo desired state，也不能成为 CD 放行硬依赖。 | -| `Lease/hwlab-dev/hwlab-dev-cd-lock` | 只作为事务互斥锁和审计状态；永远不是 desired-state 来源。 | - -只改文档的 commit 如果没有修改 `deploy/deploy.json`、 -`deploy/artifact-catalog.dev.json`、`deploy/k8s/base/workloads.yaml`、service -source 或 runtime manifest，就不会产生新的 DEV image 或 promotion commit。 -这类 commit 的 artifact build/publish 结论是 not applicable，除非另一个 -artifact 任务明确更新 desired state。 - -## Runner 和 Host 边界 - -Deployment/CD 工作按以下边界分工： - -| 角色 | 可以执行 | 没有明确 rollout 授权时不得执行 | -| --- | --- | --- | -| Code Queue runner | 准备分支和 PR；在任务明确授权且检查通过后更新或自合并 docs/code/artifact PR；通过 repo-owned 路径 build/publish artifact；在被分配时刷新 artifact report 或 desired-state 文件；运行只读 `status`/`dry-run`/preflight 检查。 | 未授权自合并、竞争 DEV CD Lease、执行 DEV apply、rollout、声称 `16666/16667` live verification、收口 `#7` 或 `#242`、修改 PROD、打印 Secret value、手工修 live resource。 | -| Host commander | 在需要指挥官 review 时审阅并合并最终 PR；执行单一 DEV deploy apply 事务；执行 rollout 和 `16666/16667` live verification；收口 `#7`/`#242` 状态。 | 绕过 repo-owned CD 入口、使用第二控制面、用 UniDesk runtime 替代 HWLAB runtime、只凭报告而不做一手复验就当作 live evidence。 | - -Runner 自合并权限只在单个任务和单个 PR 的显式授权边界内生效。它不授予 -DEV apply、rollout、live health verification 或看板/status 收口权限。Runner -发布 artifact 时，输出必须包含 tag、digest 和 report 证据，并把 -desired-state convergence 与 live rollout evidence 分开。 - -## CI/CD 职责拆分 - -长期稳定规则是 CI 负责构建和发布镜像，CD 只消费已经构建好的镜像和 -repo-owned desired state。CD 入口不得临场重新解释 source、手工拼装 build -env、重新构建镜像或把构建失败伪装成 rollout blocker。 - -CI/CD 三源分工固定如下： - -| 来源 | 职责 | 不能承担 | -| --- | --- | --- | -| `deploy/deploy.json` | 唯一发布控制源，声明 promotion commit、service image tag 和 DEV 目标。 | 证明镜像真实存在。 | -| Registry manifest | 镜像事实源，证明目标 tag/digest 可被 registry 读取，并与 catalog 记录一致。 | 决定哪一批服务应该作为一个发布集合上线。 | -| CI 产物报告 | 自动生成的一次性审计记录，包含构建、推送、digest、耗时和日志摘要。 | 作为手工维护状态表，或成为 CD rollout 的必要本地文件。 | - -`deploy/artifact-catalog.dev.json` 是 `deploy.json` 与 registry 之间的 repo-owned -digest 期望表；`deploy/k8s/base/workloads.yaml` 是最终 Kubernetes desired state。 -CD 在 apply 前必须根据 catalog 的 image tag/digest 直接读取 registry manifest， -确认镜像事实存在且 digest 匹配。过期、缺失或路径漂移的 -`/tmp/hwlab-dev-gate/dev-artifacts.json` 只能帮助定位 CI 问题，不得推翻 -`deploy.json + catalog + registry` 已经证明一致的发布结论。 - -过渡期如果 `scripts/dev-cd-apply.mjs` 发现 `deploy/deploy.json` 已经指向一个 -通过 desired-state check 的 promotion commit，应走 `promotionSource=deploy-json` -路径，只执行 desired-state 校验、runtime 前置、apply、postflight 和 live verify， -不得重新 publish 当前 HEAD。未来 CI 常态化后，latest-main target-ref publish -路径应降级为受限兼容路径，正式 DEV rollout 默认走已发布 artifact 的 -deploy-json promotion。 - -CI/artifact 层可以使用以下加速面： - -- `scripts/dev-artifact-publish.mjs --build --services `：只构建受影响服务，避免 web-only 变更重建所有服务； -- `scripts/dev-artifact-publish.mjs --publish --services `：只在 artifact 任务边界内发布选定服务并记录 digest； -- `scripts/dev-artifact-publish.mjs --publish --concurrency `：并行构建/推送服务镜像，默认 4，最高 8； -- `scripts/dev-runtime-base-image.mjs`：按 `package.json` / `package-lock.json` - hash 构建 `hwlab-node-runtime-base:deps-`，让后续服务镜像复用依赖层。 - -CD 层不得把上述 CI/artifact 加速面当作 rollout 授权。只有当 desired-state -文件已经收敛、registry manifest 直接验证通过，CD 才能拿 Lease 执行 DEV apply。 -CD apply 内部可以用 `--rollout-concurrency ` 或 `HWLAB_DEV_CD_CONCURRENCY` -并行执行 registry manifest 验证、DEV allowlist 清理和 post-apply rollout status； -并发只加速同一事务内的独立检查，不创建第二个 CD 控制源。 - -## Git Repo Owner 权限治理 - -GitHub repo owner/admin 权限是治理权限，不是日常构建、合并、发布和清理权限。 -长期目标是把 owner 从常规交付链路中解耦，避免 owner 账号或 owner-owned -工作区成为单点阻塞，也避免 root/owner 文件污染让 runner、CI 或 CD 无法继续。 - -稳定分工如下： - -| 权限域 | 可以做什么 | 不应承担什么 | -| --- | --- | --- | -| Repo owner/admin | 维护 branch protection、required checks、runner/App/token 授权、紧急权限恢复和审计规则。 | 日常手工 build、长期持有脏 worktree、替 runner 修权限污染、绕过 PR 合并主线。 | -| CI build identity | 从受保护主线或 PR commit 构建镜像，写 CI artifact、digest report 和受控 PR/commit。 | DEV apply、live rollout、看板收口、读取 Secret value。 | -| CD deploy identity | 读取已合并 desired-state 和 registry artifact，获取 DEV Lease，执行 apply/verify。 | 构建镜像、修改 source、改写 `deploy.json` promotion commit。 | -| Code Queue runner | 在任务分支完成代码/文档/desired-state PR，运行只读 status/preflight 和授权的 artifact 任务。 | 占用 owner 权限、写 main、污染共享发布 worktree、承担长期 CD lock。 | - -所有自动化应使用可审计、可轮换的 GitHub App 或 fine-grained token；token 只授予 -所需 repo 和最小 scope。日常主线合并通过 branch protection 和 required CI -结果完成，owner/admin 只处理策略变更、紧急解锁和权限恢复。至少保留两个维护者 -具备 owner/admin 恢复能力，避免单人离线造成 repo 或部署链路停摆。 - -共享发布目录只允许作为缓存或 clean mirror，不允许成为 source of truth。任何 -CI/CD 写操作都应在 ephemeral worktree 或明确 owner 的 clean worktree 中执行； -进入写路径前必须检查 `git status --short`、`.git` 可读写权限、工作区 owner/group -和 expected remote commit。发现 root-owned 文件、不可快进、未知本地修改或权限 -不一致时，应结构化报 `environment_blocker`，由权限治理流程修复；不得直接在污染 -工作区继续 build、apply 或 push。 - -## SecretRef Preflight - -apply 前还必须把必要 SecretRef 作为 preflight 条件处理，而不是等待 -runtime Job 进入 `CreateContainerConfigError`： - -- `hwlab-cloud-api-dev-db/database-url`； -- `hwlab-cloud-api-dev-db-admin/admin-url`； -- `hwlab-code-agent-provider/openai-api-key`。 - -这些检查只能验证 Secret 名称、key 和 env 注入边界，不得打印 Secret value。 -如果 `hwlab-cloud-api-dev-db-admin/admin-url` 缺失，短期 runbook 是由受控 -DEV-only secret reconcile 从 `unidesk-dev/postgres-dev` runtime source 生成 -并写入 `hwlab-dev`；长期应迁移到 declarative secret、sealed/external secret -或统一 secret controller。 - -SecretRef preflight report 必须使用布尔值和脱敏 metadata： - -| 检查项 | 必要证据 | 缺失时 blocker | -| --- | --- | --- | -| Source desired-state reference | `deploy/deploy.json` 和 `deploy/k8s/base/workloads.yaml` 声明预期 `secretRef` / `secretKeyRef` name 和 key。 | `contract_blocker`，scope 指向所属 manifest 或 service。 | -| Live D601 Secret presence | 在 `hwlab-dev` namespace 中，repo-owned preflight 或 controller 证明必要 Secret resource 和 key 存在。 | `runtime_blocker`，scope 指向缺失的 SecretRef。 | -| k3s access for the check | 检查使用 `/etc/rancher/k3s/k3s.yaml`，解析到节点 `d601`，没有命中 Docker Desktop 或第二控制面。 | `environment_blocker` 或 `observability_blocker`，scope 指向 D601 k3s visibility。 | -| Redaction | Report 字段声明 `secretValuesPrinted=false`；日志不包含 decoded data、DSN、token、password、kubeconfig material 或 base64 Secret payload。 | `safety_blocker`；在 publish/apply 前停止。 | - -preflight 可以报告 Secret name、key name、namespace、presence boolean、 -owning workload、missing scope 和 repair runbook。它不得粘贴 -`HWLAB_CLOUD_DB_URL`, `HWLAB_CLOUD_DB_ADMIN_URL`, `OPENAI_API_KEY`, bearer -tokens, passwords, kubeconfig contents, or Secret `.data` values into reports, -issues, PRs, logs, screenshots, or chat. - -## Blocker 分类 - -DEV CD report 必须保持 blocker 分类稳定，使 runner 和 host 不读完整事务日志 -也能路由工作： - -| Type | 含义 | -| --- | --- | -| `safety_blocker` | 请求动作会违反 DEV-only、non-secret、non-PROD、lock 或控制面安全规则。 | -| `environment_blocker` | 必要 local/D601 tool、kubeconfig、k3s node、registry、workspace 或 permission 不可用。 | -| `contract_blocker` | Source manifest、desired-state 文件、artifact catalog、report schema 或 promotion provenance 不一致。 | -| `runtime_blocker` | DEV runtime 前置条件或后置条件缺失，包括 SecretRef presence、DB readiness、durable runtime、rollout 或 workload image convergence。 | -| `agent_blocker` | Code Agent provider/model/egress/SecretRef readiness 不完整。 | -| `network_blocker` | public 或 internal endpoint 不可达，或返回无效 health response。 | -| `observability_blocker` | runner 无法观测足够只读证据来证明状态，但底层 runtime 可能仍然健康。 | - -每个 blocker 必须包含 `type`、`scope`、`status` 和单行 `summary`。可行时还 -应包含 `unblockHint` 或 `nextTask`。Secret 相关 blocker 必须声明没有打印 -value。 - -成功收口必须同时具备以下证据： - -- `scripts/dev-cd-apply.mjs --status` 返回 `status=pass`； -- `Lease/hwlab-dev-cd-lock` 为 `phase=released`、`releaseStatus=pass`； -- `16666/health/live` 的 Cloud Web revision 等于 `deploy.json` promotion commit； -- `16667/health/live` 的 Cloud API commit、DB readiness、runtime durability 均 ready； -- `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy,statefulset` - 显示 persistent workload image tag 收敛到同一 promotion commit； -- 输出保持 bounded summary，完整 report 写入 `/tmp/hwlab-dev-gate/dev-cd-apply.json` - 或 CLI dump 文件，避免把长 JSON 直接塞进指挥上下文。 - -正常 `--apply` stdout 默认也是紧凑摘要；完整事务记录继续通过 -`--report /tmp/hwlab-dev-gate/report.json` 写入既有 report 路径，只有显式 `--full-output` 时才打印 -完整 JSON。DEV apply、rollout 和 live verification 仍由 host commander -统一执行；runner 默认只使用 status/dry-run 观测面，不竞争 DEV CD lock。 - -旧的 deploy side-effect 命令已经降为事务内部步骤。直接手动执行 -`scripts/dev-deploy-apply.mjs --apply` 且没有 `HWLAB_CD_TRANSACTION_ID` 时，会以 -`cd-transaction-required` 拒绝。artifact publish 是 CI 职责，正式入口需要 -`HWLAB_CI_ARTIFACT_RUN_ID`；过渡期也接受 `HWLAB_CD_TRANSACTION_ID`。preflight、 -build、dry-run、check 和 read-only 模式仍可用。 - -## Artifact Publish - -DEV artifacts publish to the D601 local/internal registry: - -```text -127.0.0.1:5000/hwlab/* -``` - -The single-step publish command is a CI artifact side-effect step: - -```sh -HWLAB_CI_ARTIFACT_RUN_ID= node scripts/dev-artifact-publish.mjs --publish -``` - -It only runs in side-effect mode when `HWLAB_CI_ARTIFACT_RUN_ID` is set by the -CI artifact job, or when the legacy `HWLAB_CD_TRANSACTION_ID` is set by -`scripts/dev-cd-apply.mjs` during transition. Normal DEV CD must consume -`deploy.json` and registry-verified catalog state; it should not rebuild images. - -For Cloud Web source freshness before the transaction: - -```sh -node web/hwlab-cloud-web/scripts/check.mjs -node web/hwlab-cloud-web/scripts/build.mjs -node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -Cloud Web must build `web/hwlab-cloud-web/dist` before publish because the -runtime wrapper serves `dist` before source files. A correct source commit with -stale `dist` can still publish an old UI. The build/check pair is part of the -publish contract, but with separate responsibilities: `check.mjs` validates -source and does not require ignored `dist/` in a clean checkout. `build.mjs` -regenerates `dist` and verifies every runtime file and route alias matches -source. `scripts/dev-artifact-publish.mjs` runs that build/freshness gate and -records the `distFreshness` result before it builds the image. - -After a successful multi-service publish, refresh the artifact catalog from the -publish report: - -```sh -node scripts/refresh-artifact-catalog.mjs --target-ref --publish-report /tmp/hwlab-dev-gate/dev-artifacts.json -node scripts/deploy-desired-state-plan.mjs --promotion-commit --check -``` - -`ci-publish` only proves that images for its `sourceCommitId` were built and -pushed; it does not change the rollout target by itself. The refresh target -must be the already published `sourceCommitId` from the CI publish report. A -later documentation, catalog, or desired-state commit may record the promotion, -but it is not automatically an image source and must not be passed as -`--target-ref` unless a matching publish report proves images for that exact -commit. - -Before any formal `dev-cd-apply`, operators must refresh -`deploy/artifact-catalog.dev.json`, `deploy/k8s/base/workloads.yaml`, and -`deploy/deploy.json`, commit/push that promotion, and verify the desired-state -plan points at the intended `sourceCommitId`. If `deploy/deploy.json` still -names an older commit, CD will correctly keep deploying that older promotion -even when a newer publish report exists. Do not treat the newest CI publish -report, or a later bookkeeping commit, as implicit CD desired state. - -If publish is blocked, keep the report blocked rather than inventing digests: - -```sh -node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked -node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --check -``` - -The desired-state files are a single deployment truth: `deploy/deploy.json`, -`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml` must -carry the same promotion commit and immutable image tag before they are used as -apply input. A `--target-ref` desired-state plan may report a uniform older -state as `promotion_pending`; PR review and CI/CD promotion gates must use -`--target-ref origin/main --check` or `--promotion-commit --check` so -stale or partially refreshed commit, image, workload, and mirror-env fields are -blocked before any DEV apply. -Reports under `repository report directory ` are evidence snapshots only and cannot override this -desired-state contract. - -## Latest-Main Artifact/Runtime Guard - -Before any controlled DEV apply for latest `origin/main`, run the read-only -artifact/runtime guard: - -```sh -node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report -``` - -When current readiness is expected to be blocked, use: - -```sh -node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked -``` - -The guard compares target main, `/tmp/hwlab-dev-gate/dev-artifacts.json`, -`deploy/artifact-catalog.dev.json`, the desired-state planner, Cloud API -`/health/live`, and Cloud Web `:16666/health/live`. It is report-only and must -not be used as an apply command. A local source check, static workbench smoke, -or stale report snapshot cannot claim latest-main deployment unless the guard's -runtime and artifact identity checks also pass. - -## API, Edge, Health, And FRP - -The public route is fixed: - -| Surface | Current entry | -| --- | --- | -| Cloud Web | `http://74.48.78.17:16666/` | -| Cloud Web live health | `http://74.48.78.17:16666/health/live` | -| API/edge health | `http://74.48.78.17:16667/health` | -| API/live health | `http://74.48.78.17:16667/health/live` | - -FRP maps master public `16666/16667` to D601 `hwlab-dev` services. Use -`deploy/deploy.json` as the source for endpoint and FRP drift checks: - -```sh -node scripts/deploy-contract-plan.mjs --check -``` - -Do not reintroduce public `6666/6667` as active DEV endpoints. Internal k3s -services may still use internal `6667`. - -## Cloud Web Manual DEV Rollout - -This section is retained as historical operational baseline for what the -transaction reproduces. New DEV side effects should use -`scripts/dev-cd-apply.mjs`; do not bypass the Lease with ad hoc publish, -`kubectl set image`, or restart commands. - -The DEV CD transaction owns the durable runtime support sequence before final -live postflight: - -```sh -node cmd/hwlab-cloud-api/provision.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -node cmd/hwlab-cloud-api/migrate.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -node scripts/dev-runtime-postflight.mjs --live --confirm-dev --confirmed-non-production --target api --report /tmp/hwlab-dev-gate/report.json -``` - -These commands are called by `scripts/dev-cd-apply.mjs` under the transaction -Lease. Provisioning and migration run from the current `hwlab-cloud-api` image -as stable image-internal operator entrypoints; they receive DB input only -through SecretRef-backed env and print only redacted structured status. Runners -without explicit rollout authorization may run the source-only `--check` -variants and prepare PRs/artifacts, but must leave DEV apply, rollout, live -verification, and lock ownership to the host commander. - -The previous verified manual path for `hwlab-cloud-web` was: - -```sh -cd /home/ubuntu/workspace/hwlab -git pull --ff-only origin main -node web/hwlab-cloud-web/scripts/build.mjs -node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web: -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s -``` - -That historical sequence is no longer the normal side-effect path; direct -`--publish` is rejected unless it is called by the transaction. - -The rollout must use D601 native k3s: - -```sh -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide -``` - -The rollout report must tie the live deployment back to artifact identity: -source commit, image tag, image reference, registry digest or `not_published`, -and Kubernetes rollout revision. `scripts/dev-deploy-apply.mjs` reports this -under `devDeployApply.cloudWebRollout` when read-only Deployment access exists; -without that access, it reports the blocker and the exact read/rollout commands -instead of claiming a live rollout. - -Public verification must check the browser entry on `16666`, not legacy -ports: - -```sh -curl -fsS http://74.48.78.17:16666/health/live -curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p' -curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh' -curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p' -curl -fsS http://74.48.78.17:16667/health/live -``` - -Cloud Workbench acceptance also expects Chinese default UI, no outer page -scroll, and an internal Markdown help surface. The frontend contract is -[cloud-workbench.md](cloud-workbench.md). - -## Apply Boundary - -`scripts/dev-deploy-apply.mjs` is the DEV-only apply/report backend. A real -apply requires the transaction environment created by `scripts/dev-cd-apply.mjs` -plus explicit non-production flags: - -```sh -node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json -``` - -Direct manual `--apply` calls without `HWLAB_CD_TRANSACTION_ID` return -`cd-transaction-required`. The apply path must stop before mutation when -artifact evidence, registry digests, kube access, DEV health, or required -runtime readiness are missing. It must not read secret values or target PROD. - -Suspended template Jobs are immutable in `spec.template`. Only the explicit -DEV allowlist may be replaced by the apply script: - -- `hwlab-agent-worker-template` -- `hwlab-cli-template` - -## Rollback Boundary - -Rollback remains DEV-only and must be explicit. Capture current image state -before changing a workload: - -```sh -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout history deployment/hwlab-cloud-web -``` - -For a Deployment rolled by `set image`, use Kubernetes rollout undo only for -the same DEV Deployment: - -```sh -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/hwlab-cloud-web -KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s -``` - -After rollback, repeat public verification on `16666` and `16667`. Do not use -rollback to change PROD, read secrets, restart unrelated services, or conceal -an artifact/report mismatch. - -## Controlled Entrypoint Design - -The #116 target is a single controlled deployment surface. Until that lands, -use the repo-owned scripts below instead of ad hoc shell fragments. - -| Action | Current controlled entrypoint | Gap to close | -| --- | --- | --- | -| Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. | -| Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. | -| Observe DEV CD status | `node scripts/dev-cd-apply.mjs`, `node scripts/dev-cd-apply.mjs --status`, or `node scripts/dev-cd-apply.mjs --dry-run` | Keep read-only for host and runners; no Lease mutation, artifact publish, apply, rollout, report write, or Secret read. | -| Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. | -| Prepare dependency runtime base | `node scripts/dev-runtime-base-image.mjs --dry-run` and `node scripts/dev-runtime-base-image.mjs` | Move into a CI base-image job keyed by dependency hash; CD only consumes the resulting image tag. | -| Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. | -| Publish DEV images | `HWLAB_CI_ARTIFACT_RUN_ID= node scripts/dev-artifact-publish.mjs --publish` | CI artifact step only; legacy CD transaction env remains transitional. | -| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report /tmp/hwlab-dev-gate/dev-artifacts.json` | CI promotion step after publish; report is input evidence for catalog refresh, not CD rollout state. | -| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --report /tmp/hwlab-dev-gate/report.json` inside `HWLAB_CD_TRANSACTION_ID` | Internal transaction step; direct side-effect calls are rejected. | -| Roll back DEV Deployment | `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/` | Add `hwlab deploy rollback --env dev --service --to-revision `. | -| Verify status | `curl` checks plus `node scripts/d601-k3s-readonly-observability.mjs` | Add `hwlab deploy verify --env dev --report /tmp/hwlab-dev-gate/report.json`. | - -## Automation Direction - -The stable direction from -[pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61) and -[pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116) is to -converge manual operations into CLI plus `deploy/deploy.json`, then run that -surface from UniDesk CI/CD: - -- render and verify frps/frpc from the deploy manifest; -- publish images and write digest reports as CI audit evidence; -- refresh catalog/workload desired state from a verified CI publish report; -- apply DEV with immutable Job replacement policy; -- verify catalog tag/digest directly against registry manifests before apply; -- verify `16666` Cloud Web and `16667/health/live` API; -- invalidate or regenerate active reports after endpoint or artifact changes. - -UniDesk CI/CD is infrastructure only. It may schedule build/publish/deploy -jobs and store artifacts, but it must not replace `hwlab-cloud-api`, -`hwlab-cloud-web`, `hwlab-router`, `hwlab-patch-panel`, simulators, gateways, -or agent services as the runtime being validated. - -## Stable Sources - -- [docs/dev-artifact-publish.md](../dev-artifact-publish.md): artifact publish - command and report contract. -- [docs/dev-deploy-apply.md](../dev-deploy-apply.md): apply/report and Cloud - Web manual rollout details. -- [web/hwlab-cloud-web/scripts/dist-contract.mjs](../../web/hwlab-cloud-web/scripts/dist-contract.mjs): Cloud Web `dist` freshness contract used by build/check/publish. -- [pikasTech/HWLAB#61](https://github.com/pikasTech/HWLAB/issues/61): manual - rollout review and automation target. -- [pikasTech/HWLAB#116](https://github.com/pikasTech/HWLAB/issues/116): - deployment regularization stages and acceptance. -- [pikasTech/HWLAB#340](https://github.com/pikasTech/HWLAB/issues/340): - `deploy.json` DEV CD path long-termization and master CLI automation. diff --git a/docs/reference/dev-runtime-hotfix-runbook.md b/docs/reference/dev-runtime-hotfix-runbook.md index a4d53f3d..f6cea908 100644 --- a/docs/reference/dev-runtime-hotfix-runbook.md +++ b/docs/reference/dev-runtime-hotfix-runbook.md @@ -22,7 +22,7 @@ export KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes -o jsonpath='{.items[*].metadata.name}' ``` -结果必须包含 `d601`。出现 `docker-desktop`、`desktop-control-plane`、`127.0.0.1:11700` 或第二套 `hwlab-dev` 控制面时，停止 hotfix、回滚和验收判断。此边界与 [dev-runtime-boundary.md](dev-runtime-boundary.md) 和 [deployment-publish.md](deployment-publish.md) 保持一致。 +结果必须包含 `d601`。出现 `docker-desktop`、`desktop-control-plane`、`127.0.0.1:11700` 或第二套 `hwlab-dev` 控制面时，停止 hotfix、回滚和验收判断。此边界与 [dev-runtime-boundary.md](dev-runtime-boundary.md) 和 [g14-gitops-cicd.md](g14-gitops-cicd.md) 保持一致。 ## kubectl 热修命令形状 diff --git a/fixtures/dev-deploy-smoke/dev-deploy-smoke.json b/fixtures/dev-deploy-smoke/dev-deploy-smoke.json deleted file mode 100644 index 57a54928..00000000 --- a/fixtures/dev-deploy-smoke/dev-deploy-smoke.json +++ /dev/null @@ -1,307 +0,0 @@ -{ - "smokeId": "m2-dev-deploy-smoke", - "issue": "pikasTech/HWLAB#23", - "environment": "dev", - "endpoint": "http://74.48.78.17:16667", - "frontendEndpoint": "http://74.48.78.17:16666", - "legacyPublicEndpoint": { - "endpoint": "http://74.48.78.17:6667", - "status": "deprecated", - "activeGreenEligible": false - }, - "defaultMode": "dry-run", - "safetyGates": { - "dryRunOnly": true, - "realRequestsRequire": [ - "explicit --live flag", - "confirmed DEV endpoint", - "confirmed non-production workspace", - "no forbidden actions" - ], - "forbiddenActions": [ - "real-dev-deploy", - "prod-deploy", - "heavyweight-e2e", - "secret-read", - "force-push", - "unidesk-runtime-substitution" - ] - }, - "serviceContracts": [ - { - "routePhase": 1, - "phase": "master-edge-proxy", - "serviceId": "hwlab-edge-proxy", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-edge-proxy", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:40b2218b7a1ede3d215eaf99c56caf85cd61318da4e4348122e4e2da4c072cd8", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "ok", - "route": "public-dev-ingress", - "forwardTarget": "frp", - "artifactIdentity": "master edge proxy" - }, - "failureClassification": "network_blocker" - }, - { - "routePhase": 2, - "phase": "frp-tunnel", - "serviceId": "hwlab-tunnel-client", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-tunnel-client", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:53bdcb1f16d8a761d836e09e9bb0fe37d3f726b85769be8658d1e2702ff3d448", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "connected", - "route": "master-edge-proxy", - "forwardTarget": "D601 hwlab-dev/hwlab-router", - "artifactIdentity": "frp tunnel" - }, - "failureClassification": "network_blocker" - }, - { - "routePhase": 3, - "phase": "d601-router", - "serviceId": "hwlab-router", - "namespace": "hwlab-dev", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-router", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:be7e87979bbe78fc040d52d64b264bb8494f873c14b455dbf14681e2b4fd3dfb", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "live", - "route": "hwlab-dev/hwlab-router", - "forwardTarget": "cloud-api/web", - "artifactIdentity": "D601 router" - }, - "failureClassification": "runtime_blocker" - }, - { - "routePhase": 4, - "phase": "cloud-surface", - "serviceId": "hwlab-cloud-api", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-cloud-api", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:a147c7dbb6df1d30b0ac5e747f2f1fc42d455a55410b82578573b9b3ae2d6074", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "ok", - "live": true, - "route": "cloud-api/web", - "db": { - "fixtureOnly": true, - "liveDbEvidence": false, - "requiredEnv": [ - { - "name": "HWLAB_CLOUD_DB_URL", - "present": false, - "redacted": true - }, - { - "name": "HWLAB_CLOUD_DB_SSL_MODE", - "present": false, - "redacted": false - } - ], - "missingEnv": [ - "HWLAB_CLOUD_DB_URL", - "HWLAB_CLOUD_DB_SSL_MODE" - ], - "summary": "Fixture records the DB env contract shape only; it is not live DB evidence." - }, - "artifacts": [ - { - "serviceId": "hwlab-cloud-api", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-cloud-api", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:a147c7dbb6df1d30b0ac5e747f2f1fc42d455a55410b82578573b9b3ae2d6074" - , - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z" - }, - { - "serviceId": "hwlab-cloud-web", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-cloud-web", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:4a546582e6318b77530b9141dfd764c36b974442d6c7916a1294c54d0b02a14e", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z" - } - ] - }, - "failureClassification": "runtime_blocker" - }, - { - "routePhase": 5, - "phase": "agent-runtime", - "serviceId": "hwlab-agent-mgr", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-agent-mgr", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:cf2dab7d0aa49ef6e77dd8e1bacebd588bf2dbd65e86a4b8c2dc23cd248c026b", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "live", - "trace": "session-scoped", - "cleanup": "no worker leak" - }, - "failureClassification": "agent_blocker" - }, - { - "routePhase": 6, - "phase": "agent-runtime", - "serviceId": "hwlab-agent-worker", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-agent-worker", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:be2941f18376e8e39e0aa27857bb2db4300e861f2c759a7bc447e96e1f0e8bf1", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "scoped", - "projectId": "proj_mvp-l6", - "trace": "session-scoped", - "cleanup": "worker session closed" - }, - "failureClassification": "agent_blocker" - }, - { - "routePhase": 7, - "phase": "agent-runtime", - "serviceId": "hwlab-agent-skills", - "commitId": "5583e4b", - "image": "skills/hwlab-agent-runtime", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "not_applicable", - "digestReason": "local dry-run fixture for repo-local skills bundle", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "traceable", - "bundle": "repo-local skills artifact", - "cleanup": "explicit commitId required" - }, - "failureClassification": "agent_blocker" - }, - { - "routePhase": 8, - "phase": "sim-and-patch-panel", - "serviceId": "hwlab-gateway", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-gateway", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:dcf57c4a0fab873ab571c1f6314d352d38853ff2cf125356a05a06ae5cfdb444", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "live", - "route": "hardware boundary", - "artifactIdentity": "gateway boundary" - }, - "failureClassification": "runtime_blocker" - }, - { - "routePhase": 9, - "phase": "sim-and-patch-panel", - "serviceId": "hwlab-gateway-simu", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-gateway-simu", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:0bc140f52733255c937d08cea4d51639c83a71df8ac7a78bb107f60401bc392e", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "live", - "route": "simulator", - "artifactIdentity": "gateway simulator", - "observedAt": "2026-05-21T00:00:00.000Z" - }, - "failureClassification": "runtime_blocker" - }, - { - "routePhase": 10, - "phase": "sim-and-patch-panel", - "serviceId": "hwlab-box-simu", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-box-simu", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:0506757987fb7955ec0089fd9c0d895c33eb714a940fc7b8d58153368f96e8c8", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "live", - "route": "patch-panel-owned", - "artifactIdentity": "box simulator" - }, - "failureClassification": "runtime_blocker" - }, - { - "routePhase": 11, - "phase": "sim-and-patch-panel", - "serviceId": "hwlab-patch-panel", - "commitId": "5583e4b", - "image": "ghcr.io/pikastech/hwlab-patch-panel", - "tag": "m2-dev-deploy-smoke-5583e4b", - "digest": "sha256:bd3045c3235353e2437fbb11308ffe6f1ea3e52ca1e481dc80d4317a4b1600d1", - "buildSource": "git+https://github.com/pikasTech/HWLAB.git#origin/main:scripts/m2-dev-deploy-smoke.mjs", - "deployEnv": "dev", - "healthTimestamp": "2026-05-21T00:00:00.000Z", - "health": { - "status": "active", - "route": "patch-panel-owned", - "artifactIdentity": "patch panel", - "observedAt": "2026-05-21T00:00:00.000Z" - }, - "failureClassification": "runtime_blocker" - } - ], - "observations": { - "serviceIds": [ - "hwlab-edge-proxy", - "hwlab-tunnel-client", - "hwlab-router", - "hwlab-cloud-api", - "hwlab-cloud-web", - "hwlab-agent-mgr", - "hwlab-agent-worker", - "hwlab-agent-skills", - "hwlab-gateway", - "hwlab-gateway-simu", - "hwlab-box-simu", - "hwlab-patch-panel" - ], - "routePhases": [ - "master-edge-proxy", - "frp-tunnel", - "d601-router", - "cloud-surface", - "agent-runtime", - "sim-and-patch-panel" - ], - "healthTimestamp": "2026-05-21T00:00:00.000Z" - } -} diff --git a/package.json b/package.json index 1655eced..09517938 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "type": "module", "scripts": { "validate": "node scripts/repo-reports-guard.mjs && node scripts/validate-contract.mjs && node scripts/deploy-contract-plan.mjs --check && node scripts/deploy-desired-state-plan.mjs --check && node scripts/dev-runtime-provisioning.mjs --check && node scripts/dev-runtime-migration.mjs --check && node scripts/dev-runtime-postflight.mjs --check && node scripts/dev-runtime-hotfix-audit.mjs && node scripts/rpt004-mvp-e2e-harness.mjs --check --no-write && node --check scripts/artifact-runtime-readiness-guard.mjs && node --check scripts/src/artifact-runtime-readiness-guard.mjs && node --check scripts/dev-runtime-hotfix-audit.mjs && node --check scripts/src/dev-runtime-hotfix-audit.mjs && node --test scripts/artifact-runtime-readiness-guard.test.mjs scripts/src/dev-runtime-hotfix-audit.test.mjs", - "check": "node scripts/repo-reports-guard.mjs && node --check scripts/repo-reports-guard.mjs && node --check scripts/src/report-paths.mjs && node --check internal/protocol/index.mjs && node --check internal/build-metadata.mjs && node --check internal/agent/index.mjs && node --check internal/agent/runtime.mjs && node --check internal/audit/index.mjs && node --check internal/db/runtime-store.mjs && node --check internal/db/runtime-store.test.mjs && node --check internal/mvp-gate/summary.mjs && node --check internal/cloud/db-contract.mjs && node --check internal/dev-entrypoint/cloud-web-routes.mjs && node --check internal/dev-entrypoint/cloud-web-proxy.mjs && node --check internal/dev-entrypoint/cloud-web-proxy.test.mjs && node --check internal/dev-entrypoint/http.mjs && node --check internal/dev-entrypoint/http.test.mjs && node --check internal/cloud/code-agent-contract.mjs && node --check internal/cloud/code-agent-session-registry.mjs && node --check internal/cloud/code-agent-session-registry.test.mjs && node --check internal/cloud/code-agent-trace-store.mjs && node --check internal/cloud/codex-stdio-session.mjs && node --check internal/cloud/json-rpc.mjs && node --check internal/cloud/code-agent-chat.mjs && node --check internal/cloud/m3-io-control.mjs && node --check internal/cloud/server.mjs && node --check internal/sim/model.mjs && node --check internal/sim/model.test.mjs && node --check internal/sim/http.mjs && node --check internal/sim/l2-runtime.mjs && node --check internal/patchpanel/model.mjs && node --check internal/patchpanel/runtime.mjs && node --check cmd/hwlab-cloud-api/main.mjs && node --check cmd/hwlab-cloud-api/provision.mjs && node --check cmd/hwlab-cloud-api/migrate.mjs && node --check cmd/hwlab-edge-proxy/main.mjs && node --check cmd/hwlab-agent-mgr/main.mjs && node --check cmd/hwlab-agent-worker/main.mjs && node --check cmd/hwlab-box-simu/main.mjs && node --check cmd/hwlab-gateway/main.mjs && node --check cmd/hwlab-gateway-simu/main.mjs && node --check tools/hwlab-gateway-shell.mjs && node --check scripts/cloud-api-runtime-smoke.mjs && node --check scripts/code-agent-chat-smoke.mjs && node --check scripts/dev-edge-health-smoke.mjs && node --check scripts/src/dev-edge-health-smoke-lib.mjs && node --check scripts/validate-contract.mjs && node --check scripts/deploy-contract-plan.test.mjs && node --check scripts/deploy-desired-state-plan.mjs && node --check scripts/src/deploy-desired-state-plan.mjs && node --check scripts/artifact-runtime-readiness-guard.mjs && node --check scripts/src/artifact-runtime-readiness-guard.mjs && node --check scripts/artifact-runtime-readiness-guard.test.mjs && node --check scripts/report-lifecycle.mjs && node --check scripts/report-lifecycle.test.mjs && node --check scripts/validate-dev-gate-report.mjs && node --check scripts/dev-cd-apply.mjs && node --check scripts/src/dev-cd-apply.mjs && node --check scripts/src/dev-cd-apply.test.mjs && node --check scripts/dev-m3-hardware-loop-smoke.test.mjs && node --check scripts/m3-io-control-e2e.mjs && node --check scripts/src/m3-io-control-e2e.mjs && node --check scripts/m3-io-control-e2e.test.mjs && node --check scripts/dev-cloud-workbench-smoke.test.mjs && node --check scripts/dev-cloud-workbench-layout-smoke.mjs && node --check scripts/rpt004-mvp-e2e-harness.mjs && node --check scripts/src/rpt004-mvp-e2e-harness.mjs && node --check scripts/src/rpt004-mvp-e2e-harness.test.mjs && node --check scripts/validate-dev-m3-cardinality.mjs && node --check scripts/validate-dev-m3-cardinality.test.mjs && node --check scripts/validate-artifact-catalog.mjs && node --check scripts/refresh-artifact-catalog.mjs && node --check scripts/refresh-artifact-catalog.test.mjs && node --check scripts/artifact-publish.mjs && node --check scripts/g14-artifact-publish.mjs && node --check scripts/dev-runtime-base-image.mjs && node --check scripts/src/dev-artifact-services.mjs && node --check scripts/src/registry-capabilities.mjs && node --check scripts/preflight-dev-base-image.mjs && node --check scripts/src/dev-base-image-preflight.mjs && node --check scripts/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.test.mjs && node --check scripts/src/dev-m4-agent-loop-smoke-lib.test.mjs && node --check scripts/d601-k3s-readonly-observability.mjs && node --check scripts/src/d601-k3s-readonly-observability.mjs && node --check scripts/dev-runtime-provisioning.mjs && node --check scripts/src/dev-runtime-provisioning.mjs && node --check scripts/src/dev-runtime-provisioning.test.mjs && node --check scripts/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.test.mjs && node --check scripts/dev-runtime-postflight.mjs && node --check scripts/src/dev-runtime-postflight.mjs && node --check scripts/src/dev-runtime-postflight.test.mjs && node --check scripts/dev-runtime-hotfix-audit.mjs && node --check scripts/src/dev-runtime-hotfix-audit.mjs && node --check scripts/src/dev-runtime-hotfix-audit.test.mjs && node --test scripts/src/dev-runtime-hotfix-audit.test.mjs && node --check scripts/l2-runtime-contract-smoke.mjs && node --check scripts/patch-panel-runtime-smoke.mjs && node --check scripts/export-web-gate-summary.mjs && node --check scripts/l6-cli-web-smoke.mjs && node --check skills/hwlab-agent-runtime/scripts/hwlab-agent-runtime-cli.mjs && node --check skills/hwlab-agent-runtime/scripts/src/m3-io-skill-client.mjs && node --check tools/hwlab-cli/bin/hwlab-cli.mjs && node --check tools/hwlab-cli/lib/cli.mjs && node --check tools/hwlab-cli/lib/cicd-jobs.mjs && node --check tools/hwlab-cli/lib/cli.test.mjs && node --check web/hwlab-cloud-web/app.mjs && node --check web/hwlab-cloud-web/code-agent-facts.mjs && node --check web/hwlab-cloud-web/code-agent-facts.test.mjs && node --check web/hwlab-cloud-web/code-agent-status.mjs && node --check web/hwlab-cloud-web/code-agent-status.test.mjs && node --check web/hwlab-cloud-web/code-agent-m3-evidence.mjs && node --check web/hwlab-cloud-web/live-status.mjs && node --check web/hwlab-cloud-web/wiring-status.mjs && node --check web/hwlab-cloud-web/gate-summary.mjs && node --check web/hwlab-cloud-web/scripts/check.mjs && node --check web/hwlab-cloud-web/scripts/live-status-contract.test.mjs && node --check web/hwlab-cloud-web/scripts/build.mjs && node --check web/hwlab-cloud-web/scripts/dist-contract.mjs && node scripts/validate-contract.mjs && node scripts/validate-dev-gate-report.mjs && node scripts/report-lifecycle.test.mjs && node scripts/validate-dev-m3-cardinality.mjs && node scripts/validate-artifact-catalog.mjs && node scripts/deploy-desired-state-plan.mjs --check && node scripts/dev-runtime-provisioning.mjs --check && node scripts/dev-runtime-migration.mjs --check && node cmd/hwlab-cloud-api/provision.mjs --check && node cmd/hwlab-cloud-api/migrate.mjs --check && node scripts/dev-runtime-postflight.mjs --check && node scripts/rpt004-mvp-e2e-harness.mjs --check --no-write && node scripts/dev-evidence-blocker-aggregator.mjs --check && node scripts/l2-runtime-contract-smoke.mjs && node scripts/l6-cli-web-smoke.mjs && node web/hwlab-cloud-web/scripts/check.mjs && node scripts/code-agent-chat-smoke.mjs && node --test web/hwlab-cloud-web/code-agent-facts.test.mjs web/hwlab-cloud-web/code-agent-status.test.mjs web/hwlab-cloud-web/wiring-status.test.mjs scripts/artifact-runtime-readiness-guard.test.mjs scripts/deploy-contract-plan.test.mjs scripts/dev-m3-hardware-loop-smoke.test.mjs scripts/validate-dev-m3-cardinality.test.mjs scripts/dev-cloud-workbench-smoke.test.mjs web/hwlab-cloud-web/scripts/live-status-contract.test.mjs scripts/deploy-desired-state-plan.test.mjs scripts/refresh-artifact-catalog.test.mjs scripts/src/dev-cd-apply.test.mjs scripts/src/dev-deploy-apply.test.mjs scripts/src/dev-runtime-provisioning.test.mjs scripts/src/dev-runtime-migration.test.mjs scripts/src/dev-runtime-postflight.test.mjs scripts/src/dev-m4-agent-loop-smoke-lib.test.mjs scripts/src/dev-evidence-blocker-aggregator.test.mjs tools/hwlab-cli/lib/cli.test.mjs internal/agent/index.test.mjs internal/audit/index.test.mjs internal/db/schema.test.mjs internal/db/runtime-store.test.mjs internal/cloud/json-rpc.test.mjs internal/cloud/m3-io-control.test.mjs internal/cloud/code-agent-session-registry.test.mjs internal/cloud/server.test.mjs internal/dev-entrypoint/cloud-web-proxy.test.mjs internal/dev-entrypoint/http.test.mjs internal/sim/model.test.mjs internal/patchpanel/model.test.mjs internal/patchpanel/runtime.test.mjs && node scripts/cloud-api-runtime-smoke.mjs && sh -n scripts/bootstrap-skills.sh scripts/worker-entrypoint.sh", + "check": "node scripts/repo-reports-guard.mjs && node --check scripts/repo-reports-guard.mjs && node --check scripts/src/report-paths.mjs && node --check internal/protocol/index.mjs && node --check internal/build-metadata.mjs && node --check internal/agent/index.mjs && node --check internal/agent/runtime.mjs && node --check internal/audit/index.mjs && node --check internal/db/runtime-store.mjs && node --check internal/db/runtime-store.test.mjs && node --check internal/mvp-gate/summary.mjs && node --check internal/cloud/db-contract.mjs && node --check internal/dev-entrypoint/cloud-web-routes.mjs && node --check internal/dev-entrypoint/cloud-web-proxy.mjs && node --check internal/dev-entrypoint/cloud-web-proxy.test.mjs && node --check internal/dev-entrypoint/http.mjs && node --check internal/dev-entrypoint/http.test.mjs && node --check internal/cloud/code-agent-contract.mjs && node --check internal/cloud/code-agent-session-registry.mjs && node --check internal/cloud/code-agent-session-registry.test.mjs && node --check internal/cloud/code-agent-trace-store.mjs && node --check internal/cloud/codex-stdio-session.mjs && node --check internal/cloud/json-rpc.mjs && node --check internal/cloud/code-agent-chat.mjs && node --check internal/cloud/m3-io-control.mjs && node --check internal/cloud/server.mjs && node --check internal/sim/model.mjs && node --check internal/sim/model.test.mjs && node --check internal/sim/http.mjs && node --check internal/sim/l2-runtime.mjs && node --check internal/patchpanel/model.mjs && node --check internal/patchpanel/runtime.mjs && node --check cmd/hwlab-cloud-api/main.mjs && node --check cmd/hwlab-cloud-api/provision.mjs && node --check cmd/hwlab-cloud-api/migrate.mjs && node --check cmd/hwlab-edge-proxy/main.mjs && node --check cmd/hwlab-agent-mgr/main.mjs && node --check cmd/hwlab-agent-worker/main.mjs && node --check cmd/hwlab-box-simu/main.mjs && node --check cmd/hwlab-gateway/main.mjs && node --check cmd/hwlab-gateway-simu/main.mjs && node --check tools/hwlab-gateway-shell.mjs && node --check scripts/cloud-api-runtime-smoke.mjs && node --check scripts/code-agent-chat-smoke.mjs && node --check scripts/dev-edge-health-smoke.mjs && node --check scripts/src/dev-edge-health-smoke-lib.mjs && node --check scripts/validate-contract.mjs && node --check scripts/deploy-contract-plan.test.mjs && node --check scripts/deploy-desired-state-plan.mjs && node --check scripts/src/deploy-desired-state-plan.mjs && node --check scripts/artifact-runtime-readiness-guard.mjs && node --check scripts/src/artifact-runtime-readiness-guard.mjs && node --check scripts/artifact-runtime-readiness-guard.test.mjs && node --check scripts/report-lifecycle.mjs && node --check scripts/report-lifecycle.test.mjs && node --check scripts/validate-dev-gate-report.mjs && node --check scripts/dev-m3-hardware-loop-smoke.test.mjs && node --check scripts/m3-io-control-e2e.mjs && node --check scripts/src/m3-io-control-e2e.mjs && node --check scripts/m3-io-control-e2e.test.mjs && node --check scripts/dev-cloud-workbench-smoke.test.mjs && node --check scripts/dev-cloud-workbench-layout-smoke.mjs && node --check scripts/rpt004-mvp-e2e-harness.mjs && node --check scripts/src/rpt004-mvp-e2e-harness.mjs && node --check scripts/src/rpt004-mvp-e2e-harness.test.mjs && node --check scripts/validate-dev-m3-cardinality.mjs && node --check scripts/validate-dev-m3-cardinality.test.mjs && node --check scripts/validate-artifact-catalog.mjs && node --check scripts/refresh-artifact-catalog.mjs && node --check scripts/refresh-artifact-catalog.test.mjs && node --check scripts/artifact-publish.mjs && node --check scripts/g14-artifact-publish.mjs && node --check scripts/dev-runtime-base-image.mjs && node --check scripts/src/dev-artifact-services.mjs && node --check scripts/src/registry-capabilities.mjs && node --check scripts/preflight-dev-base-image.mjs && node --check scripts/src/dev-base-image-preflight.mjs && node --check scripts/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.test.mjs && node --check scripts/src/dev-m4-agent-loop-smoke-lib.test.mjs && node --check scripts/d601-k3s-readonly-observability.mjs && node --check scripts/src/d601-k3s-readonly-observability.mjs && node --check scripts/dev-runtime-provisioning.mjs && node --check scripts/src/dev-runtime-provisioning.mjs && node --check scripts/src/dev-runtime-provisioning.test.mjs && node --check scripts/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.test.mjs && node --check scripts/dev-runtime-postflight.mjs && node --check scripts/src/dev-runtime-postflight.mjs && node --check scripts/src/dev-runtime-postflight.test.mjs && node --check scripts/dev-runtime-hotfix-audit.mjs && node --check scripts/src/dev-runtime-hotfix-audit.mjs && node --check scripts/src/dev-runtime-hotfix-audit.test.mjs && node --test scripts/src/dev-runtime-hotfix-audit.test.mjs && node --check scripts/l2-runtime-contract-smoke.mjs && node --check scripts/patch-panel-runtime-smoke.mjs && node --check scripts/export-web-gate-summary.mjs && node --check scripts/l6-cli-web-smoke.mjs && node --check skills/hwlab-agent-runtime/scripts/hwlab-agent-runtime-cli.mjs && node --check skills/hwlab-agent-runtime/scripts/src/m3-io-skill-client.mjs && node --check tools/hwlab-cli/bin/hwlab-cli.mjs && node --check tools/hwlab-cli/lib/cli.mjs && node --check tools/hwlab-cli/lib/cli.test.mjs && node --check web/hwlab-cloud-web/app.mjs && node --check web/hwlab-cloud-web/code-agent-facts.mjs && node --check web/hwlab-cloud-web/code-agent-facts.test.mjs && node --check web/hwlab-cloud-web/code-agent-status.mjs && node --check web/hwlab-cloud-web/code-agent-status.test.mjs && node --check web/hwlab-cloud-web/code-agent-m3-evidence.mjs && node --check web/hwlab-cloud-web/live-status.mjs && node --check web/hwlab-cloud-web/wiring-status.mjs && node --check web/hwlab-cloud-web/gate-summary.mjs && node --check web/hwlab-cloud-web/scripts/check.mjs && node --check web/hwlab-cloud-web/scripts/live-status-contract.test.mjs && node --check web/hwlab-cloud-web/scripts/build.mjs && node --check web/hwlab-cloud-web/scripts/dist-contract.mjs && node scripts/validate-contract.mjs && node scripts/validate-dev-gate-report.mjs && node scripts/report-lifecycle.test.mjs && node scripts/validate-dev-m3-cardinality.mjs && node scripts/validate-artifact-catalog.mjs && node scripts/deploy-desired-state-plan.mjs --check && node scripts/dev-runtime-provisioning.mjs --check && node scripts/dev-runtime-migration.mjs --check && node cmd/hwlab-cloud-api/provision.mjs --check && node cmd/hwlab-cloud-api/migrate.mjs --check && node scripts/dev-runtime-postflight.mjs --check && node scripts/rpt004-mvp-e2e-harness.mjs --check --no-write && node scripts/dev-evidence-blocker-aggregator.mjs --check && node scripts/l2-runtime-contract-smoke.mjs && node scripts/l6-cli-web-smoke.mjs && node web/hwlab-cloud-web/scripts/check.mjs && node scripts/code-agent-chat-smoke.mjs && node scripts/cloud-api-runtime-smoke.mjs && sh -n scripts/bootstrap-skills.sh scripts/worker-entrypoint.sh", "dev-base-image:preflight": "node scripts/preflight-dev-base-image.mjs", "dev-runtime-base:build": "node scripts/dev-runtime-base-image.mjs", "cloud-api:smoke": "node scripts/cloud-api-runtime-smoke.mjs", diff --git a/scripts/artifact-publish.mjs b/scripts/artifact-publish.mjs index 46d2e27e..cfcc5734 100644 --- a/scripts/artifact-publish.mjs +++ b/scripts/artifact-publish.mjs @@ -793,7 +793,7 @@ function dockerfile(baseImage, port) { "ENV HWLAB_CODE_AGENT_SKILLS_DIRS=/app/skills:/root/.agents/skills:/home/ubuntu/.agents/skills", "COPY package.json ./package.json", "COPY package-lock.json* ./", - "RUN if [ -d /opt/hwlab-node-runtime-base/node_modules ]; then cp -a /opt/hwlab-node-runtime-base/node_modules ./node_modules; elif [ -f package-lock.json ]; then npm ci --omit=dev --ignore-scripts; else npm install --omit=dev --ignore-scripts; fi", + "RUN if [ -d /opt/hwlab-node-runtime-base/node_modules ]; then cp -a /opt/hwlab-node-runtime-base/node_modules ./node_modules; elif [ -f package-lock.json ]; then npm ci --omit=dev --include=optional --ignore-scripts; else npm install --omit=dev --include=optional --ignore-scripts; fi && codex_version=\"$(node -p \"require('./node_modules/@openai/codex/package.json').version\")\" && case \"$(uname -m)\" in x86_64|amd64) test -e node_modules/@openai/codex-linux-x64 || npm install --omit=dev --include=optional --ignore-scripts \"@openai/codex-linux-x64@npm:@openai/codex@${codex_version}-linux-x64\" ;; aarch64|arm64) test -e node_modules/@openai/codex-linux-arm64 || npm install --omit=dev --include=optional --ignore-scripts \"@openai/codex-linux-arm64@npm:@openai/codex@${codex_version}-linux-arm64\" ;; esac", "COPY internal ./internal", "COPY cmd ./cmd", "COPY scripts ./scripts", @@ -1331,7 +1331,7 @@ function createReport({ args, repo, commitId, shortCommit, mode, services, artif "docs/dev-acceptance-matrix.md", "docs/m0-contract-audit.md", "docs/artifact-catalog.md", - "docs/dev-artifact-publish.md", + "docs/reference/g14-gitops-cicd.md", "docs/dev-base-image-preflight.md" ], summary: "DEV artifact publish evidence for pikasTech/HWLAB#35, stored in the DEV gate report envelope." diff --git a/scripts/dev-cd-apply.mjs b/scripts/dev-cd-apply.mjs deleted file mode 100644 index 43882411..00000000 --- a/scripts/dev-cd-apply.mjs +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env node -import { formatDevCdApplyFailure, runDevCdApply } from "./src/dev-cd-apply.mjs"; - -try { - process.exitCode = await runDevCdApply(process.argv.slice(2)); -} catch (error) { - process.stdout.write(`${JSON.stringify(formatDevCdApplyFailure(error), null, 2)}\n`); - process.exitCode = 1; -} diff --git a/scripts/dev-deploy-apply.mjs b/scripts/dev-deploy-apply.mjs deleted file mode 100755 index 7bc2f5d4..00000000 --- a/scripts/dev-deploy-apply.mjs +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env node -import { formatFailure, runDevDeployApply } from "./src/dev-deploy-apply.mjs"; - -try { - process.exitCode = await runDevDeployApply(process.argv.slice(2)); -} catch (error) { - process.stdout.write(`${JSON.stringify(formatFailure(error), null, 2)}\n`); - process.exitCode = 1; -} diff --git a/scripts/m2-dev-deploy-smoke.mjs b/scripts/m2-dev-deploy-smoke.mjs deleted file mode 100644 index 6d056300..00000000 --- a/scripts/m2-dev-deploy-smoke.mjs +++ /dev/null @@ -1,475 +0,0 @@ -#!/usr/bin/env node -import assert from "node:assert/strict"; -import { execFile } from "node:child_process"; -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import path from "node:path"; -import { promisify } from "node:util"; -import { fileURLToPath } from "node:url"; - -import { activeReportLifecycle } from "../internal/dev-report-lifecycle.mjs"; -import { - DEV_ENDPOINT, - DEV_FRONTEND_ENDPOINT, - ENVIRONMENT_DEV, - SERVICE_IDS -} from "../internal/protocol/index.mjs"; -import { ensureNotRepoReportsPath, tempReportPath } from "./src/report-paths.mjs"; - -const execFileAsync = promisify(execFile); -const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); -const fixturePath = path.join(repoRoot, "fixtures/dev-deploy-smoke/dev-deploy-smoke.json"); -const defaultActiveReportPath = tempReportPath("dev-m2-deploy-smoke-active.json"); -const legacyPublicEndpoint = "http://74.48.78.17:6667"; -const timestampPattern = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/; -const digestPattern = /^(sha256:[a-f0-9]{64}|not_applicable)$/; -const tagPattern = /^m2-dev-deploy-smoke-[a-f0-9]{7}$/; -const allowedFailureClassifications = new Set([ - "network_blocker", - "runtime_blocker", - "agent_blocker", - "observability_blocker", - "safety_blocker" -]); - -function parseArgs(argv) { - const flags = new Set(); - const options = { - reportPath: defaultActiveReportPath, - writeReport: false, - timeoutMs: 5000 - }; - const rest = []; - for (let index = 0; index < argv.length; index += 1) { - const arg = argv[index]; - if (arg === "--report") { - flags.add(arg); - index += 1; - assert.ok(argv[index], "--report requires a path"); - options.reportPath = argv[index]; - options.writeReport = true; - } else if (arg === "--timeout-ms") { - flags.add(arg); - index += 1; - options.timeoutMs = Number.parseInt(argv[index], 10); - assert.ok(Number.isInteger(options.timeoutMs) && options.timeoutMs > 0, "--timeout-ms must be positive"); - } else if (arg.startsWith("--")) { - flags.add(arg); - } else { - rest.push(arg); - } - } - return { flags, options, rest }; -} - -function loadFixture() { - return readFile(fixturePath, "utf8").then((raw) => JSON.parse(raw)); -} - -function assertTimestamp(value, context) { - assert.equal(typeof value, "string", `${context} must be a string`); - assert.match(value, timestampPattern, `${context} must be an RFC 3339 UTC timestamp`); -} - -function assertServiceContract(entry) { - assert.ok(SERVICE_IDS.includes(entry.serviceId), `unknown serviceId ${entry.serviceId}`); - assert.equal(entry.deployEnv, ENVIRONMENT_DEV, `${entry.serviceId} deployEnv must be dev`); - assert.ok(!Object.hasOwn(entry, "endpoint"), `${entry.serviceId} service contract must not carry endpoint`); - assert.equal(typeof entry.commitId, "string", `${entry.serviceId} commitId must be a string`); - assert.match(entry.commitId, /^[a-f0-9]{7}$/u, `${entry.serviceId} commitId must be a short sha`); - assert.equal(typeof entry.image, "string", `${entry.serviceId} image must be a string`); - assert.ok(entry.image.length > 0, `${entry.serviceId} image must not be empty`); - assert.equal(typeof entry.tag, "string", `${entry.serviceId} tag must be a string`); - assert.match(entry.tag, tagPattern, `${entry.serviceId} tag must encode the smoke commit`); - assert.equal(typeof entry.digest, "string", `${entry.serviceId} digest must be a string`); - assert.match(entry.digest, digestPattern, `${entry.serviceId} digest must be sha256 or not_applicable`); - if (entry.digest === "not_applicable") { - assert.equal(typeof entry.digestReason, "string", `${entry.serviceId} digestReason required when digest is not_applicable`); - assert.ok(entry.digestReason.length > 0, `${entry.serviceId} digestReason must not be empty`); - } else { - assert.equal(entry.digestReason, undefined, `${entry.serviceId} digestReason must be absent when digest is present`); - } - assert.equal(typeof entry.buildSource, "string", `${entry.serviceId} buildSource must be a string`); - assert.ok(entry.buildSource.includes("origin/main"), `${entry.serviceId} buildSource must point at origin/main`); - assertTimestamp(entry.healthTimestamp, `${entry.serviceId} healthTimestamp`); - assert.equal(typeof entry.routePhase, "number", `${entry.serviceId} routePhase must be numeric`); - assert.ok(entry.routePhase > 0, `${entry.serviceId} routePhase must be positive`); - assert.ok(entry.phase.length > 0, `${entry.serviceId} phase must not be empty`); - assert.equal(typeof entry.health, "object", `${entry.serviceId} health must be an object`); - assert.ok(entry.health && !Array.isArray(entry.health), `${entry.serviceId} health must be a plain object`); - assert.ok(typeof entry.health.status === "string", `${entry.serviceId} health.status must be a string`); - assert.equal(typeof entry.failureClassification, "string", `${entry.serviceId} failureClassification must be a string`); - assert.ok(entry.failureClassification.endsWith("_blocker"), `${entry.serviceId} failureClassification must end in _blocker`); - assert.ok( - allowedFailureClassifications.has(entry.failureClassification), - `${entry.serviceId} failureClassification must be one of the frozen blocker classes` - ); -} - -function assertRoutePhaseOrder(entries) { - const phases = entries.map((entry) => entry.routePhase); - assert.deepEqual(phases, [...phases].sort((a, b) => a - b), "route phases must be sorted"); - assert.equal(new Set(phases).size, phases.length, "route phases must be unique"); -} - -function assertAllServiceIds(entries) { - const serviceIds = entries.map((entry) => entry.serviceId); - assert.equal(new Set(serviceIds).size, serviceIds.length, "service ids must be unique in fixture"); - for (const serviceId of serviceIds) { - assert.ok(SERVICE_IDS.includes(serviceId), `fixture uses unknown serviceId ${serviceId}`); - } -} - -function assertCloudSurface(entry) { - const healthArtifacts = entry.health.artifacts ?? []; - assert.equal(entry.serviceId, "hwlab-cloud-api", "cloud surface entry must start at cloud-api"); - assert.equal(entry.health.db?.fixtureOnly, true, "cloud surface DB note must be fixture-only"); - assert.equal(entry.health.db?.liveDbEvidence, false, "cloud surface fixture must not claim live DB evidence"); - assert.deepEqual( - entry.health.db?.missingEnv, - ["HWLAB_CLOUD_DB_URL", "HWLAB_CLOUD_DB_SSL_MODE"], - "cloud surface fixture must name missing DB env without values" - ); - assert.equal(healthArtifacts.length, 2, "cloud surface must include cloud-api and cloud-web artifacts"); - assert.deepEqual( - healthArtifacts.map((artifact) => artifact.serviceId), - ["hwlab-cloud-api", "hwlab-cloud-web"], - "cloud surface artifacts must list api/web" - ); - for (const artifact of healthArtifacts) { - assert.ok(SERVICE_IDS.includes(artifact.serviceId), `cloud surface artifact ${artifact.serviceId} must be known`); - assert.equal(typeof artifact.image, "string", `cloud surface artifact ${artifact.serviceId} image must exist`); - assert.equal(typeof artifact.tag, "string", `cloud surface artifact ${artifact.serviceId} tag must exist`); - assert.match(artifact.tag, tagPattern, `cloud surface artifact ${artifact.serviceId} tag must encode the smoke commit`); - assert.match(artifact.digest, /^sha256:[a-f0-9]{64}$/u, `cloud surface artifact ${artifact.serviceId} digest must be sha256`); - assert.equal(typeof artifact.commitId, "string", `cloud surface artifact ${artifact.serviceId} commitId must exist`); - assert.equal(typeof artifact.buildSource, "string", `cloud surface artifact ${artifact.serviceId} buildSource must exist`); - assert.equal(artifact.deployEnv, ENVIRONMENT_DEV, `cloud surface artifact ${artifact.serviceId} deployEnv must be dev`); - assertTimestamp(artifact.healthTimestamp ?? entry.healthTimestamp, `cloud surface artifact ${artifact.serviceId} healthTimestamp`); - } -} - -function assertFixture(fixture) { - assert.equal(fixture.smokeId, "m2-dev-deploy-smoke"); - assert.equal(fixture.issue, "pikasTech/HWLAB#23"); - assert.equal(fixture.environment, ENVIRONMENT_DEV); - assert.equal(fixture.endpoint, DEV_ENDPOINT); - assert.equal(fixture.frontendEndpoint, DEV_FRONTEND_ENDPOINT); - assert.equal(fixture.legacyPublicEndpoint?.endpoint, legacyPublicEndpoint); - assert.equal(fixture.legacyPublicEndpoint?.activeGreenEligible, false); - assert.equal(fixture.defaultMode, "dry-run"); - assert.equal(fixture.safetyGates.dryRunOnly, true, "dry-run must be the default"); - assert.ok(Array.isArray(fixture.safetyGates.realRequestsRequire), "real request gates must exist"); - assert.ok(fixture.safetyGates.realRequestsRequire.includes("explicit --live flag")); - - const entries = fixture.serviceContracts; - assert.ok(Array.isArray(entries) && entries.length >= 8, "serviceContracts must contain route observations"); - assertAllServiceIds(entries); - assertRoutePhaseOrder(entries); - - for (const entry of entries) { - assertServiceContract(entry); - } - - const cloudSurface = entries.find((entry) => entry.phase === "cloud-surface"); - assert.ok(cloudSurface, "cloud-surface entry must exist"); - assertCloudSurface(cloudSurface); - - const agentSkills = entries.find((entry) => entry.serviceId === "hwlab-agent-skills"); - assert.equal(agentSkills.digest, "not_applicable"); - assert.equal(agentSkills.digestReason, "local dry-run fixture for repo-local skills bundle"); - - const observations = fixture.observations; - const expectedObservationIds = []; - for (const entry of entries) { - expectedObservationIds.push(entry.serviceId); - if (entry.serviceId === "hwlab-cloud-api") { - expectedObservationIds.push("hwlab-cloud-web"); - } - } - assert.deepEqual(observations.serviceIds, expectedObservationIds, "observation service ids must track validated entries"); - assert.deepEqual( - observations.routePhases, - ["master-edge-proxy", "frp-tunnel", "d601-router", "cloud-surface", "agent-runtime", "sim-and-patch-panel"], - "route phase labels must stay frozen" - ); - assertTimestamp(observations.healthTimestamp, "observations.healthTimestamp"); -} - -function redactedHeaders(headers) { - return Object.fromEntries( - ["content-type", "content-length", "cache-control", "date"].flatMap((name) => { - const value = headers.get(name); - return value ? [[name, value]] : []; - }) - ); -} - -async function probeHttp({ id, url, expect }) { - const startedAt = Date.now(); - const controller = new AbortController(); - const timeout = setTimeout(() => controller.abort(), expect.timeoutMs); - try { - const response = await fetch(url, { signal: controller.signal }); - const body = await response.text(); - const contentType = response.headers.get("content-type") ?? ""; - let json = null; - if (contentType.includes("application/json")) { - try { - json = JSON.parse(body); - } catch { - json = null; - } - } - return { - id, - url, - ok: response.ok, - status: response.status, - statusText: response.statusText, - durationMs: Date.now() - startedAt, - contentType, - headers: redactedHeaders(response.headers), - json, - title: body.match(/([^<]+)<\/title>/iu)?.[1] ?? null, - bodyPreview: body.slice(0, 512) - }; - } catch (error) { - return { - id, - url, - ok: false, - status: null, - statusText: null, - durationMs: Date.now() - startedAt, - contentType: null, - headers: {}, - json: null, - title: null, - bodyPreview: "", - error: error?.name === "AbortError" ? `timeout after ${expect.timeoutMs}ms` : String(error?.message ?? error) - }; - } finally { - clearTimeout(timeout); - } -} - -function acceptedProbe(probe) { - if (!probe.ok) return false; - if (probe.id === "frontend-root") { - return probe.contentType?.includes("text/html") && probe.title === "HWLAB DEV MVP Gate"; - } - return Boolean( - probe.json && - SERVICE_IDS.includes(probe.json.serviceId) && - probe.json.environment === ENVIRONMENT_DEV && - (probe.json.endpoint === undefined || probe.json.endpoint === DEV_ENDPOINT) - ); -} - -function probeEvidenceLine(probe) { - const accepted = acceptedProbe(probe) ? "yes" : "no"; - const identity = probe.json?.serviceId ?? probe.title ?? "unknown"; - return `${probe.id}: HTTP ${probe.status ?? "none"} accepted=${accepted} identity=${identity} url=${probe.url}`; -} - -async function gitCommitId() { - try { - const result = await execFileAsync("git", ["rev-parse", "--short=12", "HEAD"], { - cwd: repoRoot, - timeout: 5000 - }); - return result.stdout.trim(); - } catch { - return "unknown"; - } -} - -function buildActiveReport({ fixture, probes, generatedAt, commitId }) { - const passed = probes.every(acceptedProbe); - const status = passed ? "pass" : "blocked"; - const commands = [ - "curl -fsS --max-time 10 http://74.48.78.17:16667/health", - "curl -fsS --max-time 10 http://74.48.78.17:16667/health/live", - "curl -fsS --max-time 10 http://74.48.78.17:16666/" - ]; - - return { - $schema: "https://hwlab.pikastech.local/schemas/dev-gate-report.schema.json", - $id: "https://hwlab.pikastech.local/dev-gate/dev-m2-deploy-smoke-active.json", - reportVersion: "v1", - issue: "pikasTech/HWLAB#23", - taskId: "m2-dev-deploy-smoke", - commitId, - acceptanceLevel: "dev_m2_deploy_smoke", - devOnly: true, - prodDisabled: true, - reportLifecycle: activeReportLifecycle( - "Active M2 read-only public endpoint smoke; route reachability is not M5 DEV-LIVE acceptance." - ), - status, - generatedAt, - endpoint: DEV_ENDPOINT, - frontendEndpoint: DEV_FRONTEND_ENDPOINT, - sourceContract: { - status: "pass", - documents: [ - "docs/m2-dev-deploy-smoke.md", - "docs/dev-acceptance-matrix.md", - "fixtures/dev-deploy-smoke/dev-deploy-smoke.json" - ], - summary: "M2 DEV deploy smoke is rebaselined to frontend :16666 and API/edge :16667." - }, - validationCommands: [ - "node --check scripts/m2-dev-deploy-smoke.mjs", - "node scripts/m2-dev-deploy-smoke.mjs --dry-run", - `node scripts/m2-dev-deploy-smoke.mjs --live --confirm-dev --confirmed-non-production --report ${defaultActiveReportPath}`, - "node --check scripts/validate-dev-gate-report.mjs", - "node scripts/validate-dev-gate-report.mjs" - ], - localSmoke: { - status: "pass", - commands: ["node scripts/m2-dev-deploy-smoke.mjs --dry-run"], - evidence: [ - `[m2-smoke] validated ${fixture.serviceContracts.length} service contracts`, - `[m2-smoke] mode=dry-run endpoint=${DEV_ENDPOINT}`, - "[m2-smoke] no real DEV/PROD request was made" - ], - summary: "The fixture smoke validates the M2 route and artifact contract without contacting DEV." - }, - dryRun: { - status: "pass", - commands: ["node scripts/m2-dev-deploy-smoke.mjs --dry-run"], - evidence: [ - `fixture endpoint=${DEV_ENDPOINT}`, - `fixture frontendEndpoint=${DEV_FRONTEND_ENDPOINT}`, - "legacy :6667 public evidence is marked ineligible for active green" - ], - summary: "Dry-run evidence is contract-only and pinned to the current DEV public endpoints." - }, - devPreconditions: { - status, - requirements: [ - "GET http://74.48.78.17:16667/health returns HWLAB DEV JSON", - "GET http://74.48.78.17:16667/health/live returns HWLAB DEV JSON", - "GET http://74.48.78.17:16666/ returns the HWLAB DEV frontend", - "No PROD, secret read, runtime restart, deployment, or heavyweight e2e action is performed" - ], - commands, - evidence: probes.map(probeEvidenceLine), - summary: passed - ? "Read-only DEV public endpoint probes passed on :16666/:16667." - : "One or more read-only DEV public endpoint probes failed; no active green is claimed." - }, - blockers: passed - ? [] - : [ - { - type: "network_blocker", - scope: "m2-public-endpoints", - status: "open", - summary: "One or more M2 DEV public endpoint probes failed on :16666/:16667." - } - ], - runtimeSmoke: { - mode: "live-read-only", - status, - generatedAt, - apiEndpoint: DEV_ENDPOINT, - frontendEndpoint: DEV_FRONTEND_ENDPOINT, - safety: { - environment: ENVIRONMENT_DEV, - prodTouched: false, - secretsRead: false, - restarts: false, - deployAttempted: false, - heavyE2E: false, - unideskRuntimeSubstitute: false - }, - probes, - legacyPublicEndpoint: { - endpoint: legacyPublicEndpoint, - status: "deprecated", - activeGreenEligible: false, - summary: "Legacy public :6667 observations are retained only as historical blockers and are not accepted as active green evidence." - } - }, - notes: "M2 active report uses read-only GET probes only; PROD, service restarts, deployments, and script-driven apply flows were not touched." - }; -} - -async function writeActiveReport(report, reportPath) { - const absoluteReportPath = ensureNotRepoReportsPath(repoRoot, reportPath, "--report"); - await mkdir(path.dirname(absoluteReportPath), { recursive: true }); - await writeFile(absoluteReportPath, `${JSON.stringify(report, null, 2)}\n`); -} - -async function runLiveSmoke(fixture, options) { - const probes = await Promise.all([ - probeHttp({ - id: "api-health", - url: new URL("/health", DEV_ENDPOINT).href, - expect: { timeoutMs: options.timeoutMs } - }), - probeHttp({ - id: "api-live", - url: new URL("/health/live", DEV_ENDPOINT).href, - expect: { timeoutMs: options.timeoutMs } - }), - probeHttp({ - id: "frontend-root", - url: new URL("/", DEV_FRONTEND_ENDPOINT).href, - expect: { timeoutMs: options.timeoutMs } - }) - ]); - - const report = buildActiveReport({ - fixture, - probes, - generatedAt: new Date().toISOString(), - commitId: await gitCommitId() - }); - return report; -} - -async function main() { - const { flags, options } = parseArgs(process.argv.slice(2)); - const isLive = flags.has("--live") || flags.has("--allow-live"); - - if (isLive) { - assert.ok(!flags.has("--dry-run"), "do not combine --live with --dry-run"); - assert.ok(flags.has("--confirm-dev"), "live mode requires --confirm-dev"); - assert.ok(flags.has("--confirmed-non-production"), "live mode requires --confirmed-non-production"); - assert.ok(!flags.has("--prod"), "PROD requests are forbidden"); - assert.ok(!flags.has("--heavyweight-e2e"), "heavyweight e2e is forbidden"); - assert.ok(!flags.has("--read-secret"), "secret reads are forbidden"); - assert.ok(!flags.has("--force-push"), "force push is forbidden"); - } - - const fixture = await loadFixture(); - assertFixture(fixture); - assert.equal(fixture.endpoint, DEV_ENDPOINT, "fixture must stay pinned to the frozen DEV endpoint"); - - process.stdout.write(`[m2-smoke] validated ${fixture.serviceContracts.length} service contracts\n`); - process.stdout.write(`[m2-smoke] mode=${isLive ? "live" : "dry-run"} endpoint=${fixture.endpoint}\n`); - if (isLive) { - const report = await runLiveSmoke(fixture, options); - if (options.writeReport) { - await writeActiveReport(report, options.reportPath); - process.stdout.write(`[m2-smoke] wrote ${path.relative(repoRoot, options.reportPath)}\n`); - } - process.stdout.write(`[m2-smoke] active-report=${report.status} probes=${report.runtimeSmoke.probes.length}\n`); - process.stdout.write("[m2-smoke] no DEV/PROD mutation was made\n"); - if (report.status !== "pass") { - process.exitCode = 2; - } - } else { - process.stdout.write("[m2-smoke] no real DEV/PROD request was made\n"); - } -} - -try { - await main(); -} catch (error) { - process.stderr.write(`[m2-smoke] ${error instanceof Error ? error.message : String(error)}\n`); - process.exitCode = 1; -} diff --git a/scripts/refresh-artifact-catalog.test.mjs b/scripts/refresh-artifact-catalog.test.mjs index 0d45bbc8..f4f1cde4 100644 --- a/scripts/refresh-artifact-catalog.test.mjs +++ b/scripts/refresh-artifact-catalog.test.mjs @@ -31,7 +31,7 @@ test("refresh accepts an absolute publish report path", async () => { const reportPath = path.join(tempDir, "dev-artifacts.json"); await writeFile(reportPath, `${JSON.stringify({ reportVersion: "v1", - taskId: "dev-artifact-publish", + taskId: "g14-artifact-publish", commitId: shortCommitId, artifactPublish: { status: "published", diff --git a/scripts/src/artifact-runtime-readiness-guard.mjs b/scripts/src/artifact-runtime-readiness-guard.mjs index 4ba43df7..46d990da 100644 --- a/scripts/src/artifact-runtime-readiness-guard.mjs +++ b/scripts/src/artifact-runtime-readiness-guard.mjs @@ -260,9 +260,7 @@ export function buildArtifactRuntimeReadinessReport({ sourceContract: { status: "pass", documents: [ - "docs/dev-artifact-publish.md", - "docs/dev-deploy-apply.md", - "docs/reference/deployment-publish.md" + "docs/reference/g14-gitops-cicd.md" ], summary: "Target commit, artifact publish/catalog identity, desired-state convergence, API runtime identity, and Cloud Web served runtime identity are separate gates." }, @@ -1172,7 +1170,7 @@ function nextCommands({ canPublish, canApply, blockedReasons }) { return [ "node web/hwlab-cloud-web/scripts/build.mjs", "node web/hwlab-cloud-web/scripts/check.mjs", - "node scripts/dev-artifact-publish.mjs --preflight --no-report", + "node scripts/g14-artifact-publish.mjs --preflight --no-report", "node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked --no-write", guardCommand ]; @@ -1181,7 +1179,7 @@ function nextCommands({ canPublish, canApply, blockedReasons }) { return [ "node scripts/deploy-desired-state-plan.mjs --target-ref origin/main --pretty", "node scripts/deploy-desired-state-plan.mjs --promotion-commit <origin-main-sha> --check", - `node scripts/dev-deploy-apply.mjs --dry-run --report ${tempReportPath("dev-deploy-report.json")}`, + "node scripts/g14-gitops-render.mjs --check", guardCommand ]; } @@ -1194,7 +1192,7 @@ function nextCommands({ canPublish, canApply, blockedReasons }) { ]; } return [ - `node scripts/dev-deploy-apply.mjs --dry-run --report ${tempReportPath("dev-deploy-report.json")}`, + "node scripts/g14-gitops-render.mjs --check", "node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --live --no-report" ]; } diff --git a/scripts/src/dev-base-image-preflight.mjs b/scripts/src/dev-base-image-preflight.mjs index 0d340ef0..72073353 100644 --- a/scripts/src/dev-base-image-preflight.mjs +++ b/scripts/src/dev-base-image-preflight.mjs @@ -67,7 +67,7 @@ function provisionPlan({ exactBlocker = null, envValue = recommendedNode20Image, `${envVarName}=${recommendedNode20Image} node scripts/preflight-dev-base-image.mjs` ], artifactPublishDryRun: [ - `${envVarName}=${envValue} node scripts/dev-artifact-publish.mjs --preflight --no-report` + `${envVarName}=${envValue} node scripts/g14-artifact-publish.mjs --preflight --no-report` ] } }; diff --git a/scripts/src/dev-cd-apply.mjs b/scripts/src/dev-cd-apply.mjs deleted file mode 100644 index 6014ce59..00000000 --- a/scripts/src/dev-cd-apply.mjs +++ /dev/null @@ -1,3024 +0,0 @@ -import { spawn } from "node:child_process"; -import { createHash, randomUUID } from "node:crypto"; -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { request as httpRequest } from "node:http"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; - -import { activeReportLifecycle } from "../../internal/dev-report-lifecycle.mjs"; -import { DEV_ENDPOINT, ENVIRONMENT_DEV } from "../../internal/protocol/index.mjs"; -import { - buildKubectlCommandPrefix, - redactSensitiveText, - resolveDevKubeconfigSelection -} from "./dev-deploy-apply.mjs"; -import { ensureNotRepoReportsPath, tempReportPath } from "./report-paths.mjs"; - -const defaultRepoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); -const defaultNamespace = "hwlab-dev"; -const defaultLockName = "hwlab-dev-cd-lock"; -const defaultTtlSeconds = 3600; -const defaultReportPath = tempReportPath("dev-cd-apply.json"); -const d601NativeKubeconfigPath = "/etc/rancher/k3s/k3s.yaml"; -const artifactCatalogPath = "deploy/artifact-catalog.dev.json"; -const artifactReportPath = tempReportPath("dev-artifacts.json"); -const artifactDesiredStatePaths = [ - "deploy/artifact-catalog.dev.json", - "deploy/deploy.json", - "deploy/k8s/base/workloads.yaml" -]; -const runtimeProvisioningReportPath = tempReportPath("dev-runtime-provisioning-report.json"); -const runtimeMigrationReportPath = tempReportPath("dev-runtime-migration-report.json"); -const runtimePostflightReportPath = tempReportPath("dev-runtime-postflight-report.json"); -const deployApplyReportPath = tempReportPath("dev-deploy-report.json"); -const runtimeJobReportPrefix = "/tmp/hwlab-dev-runtime-report"; -const browserLiveUrl = "http://74.48.78.17:16666/health/live"; -const apiLiveUrl = `${DEV_ENDPOINT}/health/live`; -const digestPattern = /^sha256:[a-f0-9]{64}$/u; -const forbiddenControlPlanePattern = /docker-desktop|desktop-control-plane|127\.0\.0\.1:11700/iu; -const requiredDevSecretRefs = Object.freeze([ - Object.freeze({ secretName: "hwlab-cloud-api-dev-db", secretKey: "database-url" }), - Object.freeze({ secretName: "hwlab-cloud-api-dev-db-admin", secretKey: "admin-url" }), - Object.freeze({ secretName: "hwlab-code-agent-provider", secretKey: "openai-api-key" }), - Object.freeze({ secretName: "hwlab-code-agent-codex-auth", secretKey: "auth.json" }) -]); -const requiredDevConfigMapRefs = Object.freeze([ - Object.freeze({ configMapName: "hwlab-code-agent-codex-config", configMapKey: "config.toml" }) -]); -const lockAnnotationPrefix = "hwlab.pikastech.local"; -const lockAnnotationFields = { - promotionCommit: `${lockAnnotationPrefix}/promotionCommit`, - deployJsonHash: `${lockAnnotationPrefix}/deployJsonHash`, - ownerTaskId: `${lockAnnotationPrefix}/ownerTaskId`, - transactionId: `${lockAnnotationPrefix}/transactionId`, - phase: `${lockAnnotationPrefix}/phase`, - startedAt: `${lockAnnotationPrefix}/startedAt`, - updatedAt: `${lockAnnotationPrefix}/updatedAt`, - ttlSeconds: `${lockAnnotationPrefix}/ttlSeconds`, - liveBefore: `${lockAnnotationPrefix}/liveBefore`, - targetNamespace: `${lockAnnotationPrefix}/targetNamespace`, - targetRef: `${lockAnnotationPrefix}/targetRef`, - lockBackend: `${lockAnnotationPrefix}/lockBackend`, - releasedAt: `${lockAnnotationPrefix}/releasedAt`, - releaseStatus: `${lockAnnotationPrefix}/releaseStatus`, - staleLockBrokenAt: `${lockAnnotationPrefix}/staleLockBrokenAt`, - staleLockPrevious: `${lockAnnotationPrefix}/staleLockPrevious` -}; - -class DevCdApplyError extends Error { - constructor(message, details = {}) { - super(message); - this.name = "DevCdApplyError"; - Object.assign(this, details); - } -} - -export function parseArgs(argv) { - const args = { - apply: false, - confirmDev: false, - confirmedNonProduction: false, - writeReport: false, - reportPath: defaultReportPath, - targetRef: "origin/main", - kubeconfig: null, - kubeconfigSpecified: false, - registryManifestBaseUrl: null, - lockName: defaultLockName, - targetNamespace: defaultNamespace, - ttlSeconds: defaultTtlSeconds, - ownerTaskId: null, - breakStaleLock: false, - skipLiveVerify: false, - skipRuntimePostflight: false, - status: false, - dryRun: false, - fullOutput: false, - help: false, - flags: new Set() - }; - - for (let index = 0; index < argv.length; index += 1) { - const arg = argv[index]; - if (arg === "--apply") { - args.apply = true; - args.flags.add(arg); - } else if (arg === "--confirm-dev") { - args.confirmDev = true; - args.flags.add(arg); - } else if (arg === "--confirmed-non-production") { - args.confirmedNonProduction = true; - args.flags.add(arg); - } else if (arg === "--break-stale-lock") { - args.breakStaleLock = true; - args.flags.add(arg); - } else if (arg === "--dry-run" || arg === "--status") { - args.status = true; - args.dryRun = arg === "--dry-run"; - args.flags.add(arg); - } else if (arg === "--full-output") { - args.fullOutput = true; - args.flags.add(arg); - } else if (arg === "--skip-live-verify") { - args.skipLiveVerify = true; - args.flags.add(arg); - } else if (arg === "--skip-runtime-postflight") { - args.skipRuntimePostflight = true; - args.flags.add(arg); - } else if (arg === "--report") { - args.reportPath = readOption(argv, ++index, arg); - args.writeReport = true; - args.flags.add(arg); - } else if (arg === "--target-ref") { - args.targetRef = readOption(argv, ++index, arg); - args.flags.add(arg); - } else if (arg === "--kubeconfig") { - args.kubeconfig = readOption(argv, ++index, arg); - args.kubeconfigSpecified = true; - args.flags.add(arg); - } else if (arg.startsWith("--kubeconfig=")) { - args.kubeconfig = arg.slice("--kubeconfig=".length); - args.kubeconfigSpecified = true; - args.flags.add("--kubeconfig"); - } else if (arg === "--registry-manifest-base-url") { - args.registryManifestBaseUrl = readOption(argv, ++index, arg); - args.flags.add(arg); - } else if (arg.startsWith("--registry-manifest-base-url=")) { - args.registryManifestBaseUrl = arg.slice("--registry-manifest-base-url=".length); - args.flags.add("--registry-manifest-base-url"); - } else if (arg === "--lock-name") { - args.lockName = readOption(argv, ++index, arg); - args.flags.add(arg); - } else if (arg === "--target-namespace") { - args.targetNamespace = readOption(argv, ++index, arg); - args.flags.add(arg); - } else if (arg === "--ttl-seconds") { - args.ttlSeconds = Number.parseInt(readOption(argv, ++index, arg), 10); - args.flags.add(arg); - } else if (arg === "--owner-task-id") { - args.ownerTaskId = readOption(argv, ++index, arg); - args.flags.add(arg); - } else if (arg === "--help" || arg === "-h") { - args.help = true; - } else { - throw new DevCdApplyError(`unknown argument ${arg}`, { - code: "unknown-argument", - blockers: [{ - type: "safety_blocker", - scope: "unknown-argument", - status: "open", - summary: `${arg} is not a supported DEV CD option. Use --status/--dry-run for read-only observability or --apply with explicit DEV confirmations.` - }] - }); - } - } - - return args; -} - -function readOption(argv, index, name) { - const value = argv[index]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value`); - } - return value; -} - -export function usage() { - return { - command: "node scripts/dev-cd-apply.mjs", - summary: "Single DEV CD transaction for latest-main publish, desired-state refresh, apply, and live verify.", - defaultMode: "status", - examples: [ - "node scripts/dev-cd-apply.mjs --status", - "node scripts/dev-cd-apply.mjs --dry-run", - `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report ${defaultReportPath}` - ], - options: { - "--status": "Read-only status: target ref, deploy.json, current Lease, and public live health.", - "--dry-run": "Read-only alias for --status; does not acquire the Lease or run side effects.", - "--apply": "Run the DEV CD transaction side effects.", - "--confirm-dev": "Required for DEV mutation and explicit stale-lock recovery.", - "--confirmed-non-production": "Required for DEV mutation.", - "--target-ref REF": "git ref to publish/apply; default: origin/main", - "--kubeconfig PATH": "DEV kubeconfig path; also honors HWLAB_DEV_KUBECONFIG/KUBECONFIG", - "--ttl-seconds SECONDS": "lock TTL; default: 3600", - "--owner-task-id ID": "lock holder; default: Code Queue/env/user derived", - "--break-stale-lock": "May take over an expired lock only with --confirm-dev.", - "--report PATH": `write the full transaction JSON outside the repo; packaged scripts use ${defaultReportPath}`, - "--full-output": "Print the full report to stdout instead of the concise summary.", - "--skip-runtime-postflight": "Skip the M3/durable runtime postflight while still applying DEV and verifying 16666/16667 health.", - "--skip-live-verify": "test-only escape hatch; do not use for DEV acceptance" - } - }; -} - -function nowIso() { - return new Date().toISOString(); -} - -function kubernetesMicroTime(value) { - const date = value instanceof Date ? value : new Date(value); - if (!Number.isFinite(date.getTime())) { - throw new DevCdApplyError("invalid Kubernetes Lease timestamp", { - code: "invalid-lease-timestamp" - }); - } - return date.toISOString().replace(/\.(\d{3})Z$/u, ".$1000Z"); -} - -function parseJsonMaybe(value) { - try { - return JSON.parse(value); - } catch { - return null; - } -} - -function oneLine(value) { - return String(value ?? "").replace(/\s+/g, " ").trim(); -} - -function blocker({ type = "environment_blocker", scope, reason, impact, safeNextAction, retryable }) { - return { - type, - scope, - status: "open", - reason: oneLine(reason), - impact: oneLine(impact), - safeNextAction: oneLine(safeNextAction), - retryable: Boolean(retryable), - summary: oneLine(`${reason} Impact: ${impact} Safe next action: ${safeNextAction}`) - }; -} - -function sha256(value) { - return `sha256:${createHash("sha256").update(value).digest("hex")}`; -} - -function shortCommit(value) { - return typeof value === "string" && value.length >= 7 ? value.slice(0, 7) : value; -} - -function commitMatches(actual, expected) { - if (typeof actual !== "string" || typeof expected !== "string") return false; - return actual === expected || actual.startsWith(expected) || expected.startsWith(actual); -} - -function shellCommand(command, args) { - return [command, ...args].map((part) => (/\s/u.test(part) ? JSON.stringify(part) : part)).join(" "); -} - -function defaultOwnerTaskId(env = process.env) { - return ( - env.CODE_QUEUE_TASK_ID || - env.HWLAB_OWNER_TASK_ID || - env.GITHUB_RUN_ID || - env.USER || - "unknown-dev-cd-runner" - ); -} - -function defaultRunCommand(command, args, options = {}) { - return new Promise((resolve) => { - const abortController = options.timeoutMs ? new AbortController() : null; - const child = spawn(command, args, { - cwd: options.cwd ?? defaultRepoRoot, - env: options.env ?? process.env, - stdio: ["pipe", "pipe", "pipe"], - signal: abortController?.signal - }); - let stdout = ""; - let stderr = ""; - const timeout = options.timeoutMs - ? setTimeout(() => abortController.abort(), options.timeoutMs) - : null; - - child.stdin.on("error", () => {}); - child.stdin.end(options.input ?? ""); - child.stdout.on("data", (chunk) => { - stdout += chunk; - }); - child.stderr.on("data", (chunk) => { - stderr += chunk; - }); - child.on("error", (error) => { - if (timeout) clearTimeout(timeout); - resolve({ - code: error.name === "AbortError" ? 124 : 127, - signal: null, - stdout, - stderr: error.message - }); - }); - child.on("close", (code, signal) => { - if (timeout) clearTimeout(timeout); - resolve({ code: code ?? 1, signal, stdout, stderr }); - }); - }); -} - -async function readDeployJson(repoRoot) { - const relativePath = "deploy/deploy.json"; - const raw = await readFile(path.join(repoRoot, relativePath), "utf8"); - const manifest = JSON.parse(raw); - return { - path: relativePath, - hash: sha256(raw), - commitId: manifest.commitId ?? "unknown", - namespace: manifest.namespace ?? defaultNamespace, - environment: manifest.environment ?? "unknown", - endpoint: manifest.endpoint ?? null, - manifest - }; -} - -async function readJsonFileSummary(repoRoot, relativePath, summarize) { - try { - const absolutePath = path.isAbsolute(relativePath) ? relativePath : path.join(repoRoot, relativePath); - const raw = await readFile(absolutePath, "utf8"); - const json = JSON.parse(raw); - return { - path: relativePath, - hash: sha256(raw), - exists: true, - ...summarize(json) - }; - } catch (error) { - if (error?.code === "ENOENT") { - return { - path: relativePath, - hash: null, - exists: false, - error: "missing" - }; - } - throw error; - } -} - -function servicesArray(value) { - if (Array.isArray(value)) return value; - if (value && typeof value === "object") return Object.values(value); - return []; -} - -function summarizeArtifactCatalog(catalog) { - const services = servicesArray(catalog?.services); - const digestCounts = { sha256: 0, notPublished: 0, invalid: 0 }; - for (const service of services) { - if (digestPattern.test(service?.digest ?? "")) digestCounts.sha256 += 1; - else if (service?.digest === "not_published") digestCounts.notPublished += 1; - else digestCounts.invalid += 1; - } - return { - kind: catalog?.kind ?? null, - commitId: catalog?.commitId ?? null, - artifactState: catalog?.artifactState ?? null, - ciPublished: catalog?.publish?.ciPublished ?? null, - registryVerified: catalog?.publish?.registryVerified ?? null, - provenance: catalog?.publish?.provenance ?? null, - serviceCount: services.length, - digestCounts - }; -} - -function summarizeArtifactReport(report) { - const publish = report?.artifactPublish ?? {}; - const services = servicesArray(publish.services); - const requiredServices = services.filter((service) => service?.artifactRequired !== false); - const publishedRequiredServices = requiredServices.filter((service) => service?.status === "published"); - const digestCounts = { sha256: 0, notPublished: 0, invalid: 0 }; - for (const service of services) { - if (digestPattern.test(service?.digest ?? "")) digestCounts.sha256 += 1; - else if (service?.digest === "not_published") digestCounts.notPublished += 1; - else digestCounts.invalid += 1; - } - return { - commitId: report?.commitId ?? null, - status: report?.status ?? null, - sourceCommitId: publish.sourceCommitId ?? null, - registryPrefix: publish.registryPrefix ?? null, - serviceCount: publish.serviceCount ?? services.length, - requiredServiceCount: publish.requiredServiceCount ?? requiredServices.length, - publishedCount: publish.publishedCount ?? publishedRequiredServices.length, - digestCounts, - generatedAt: publish.generatedAt ?? null - }; -} - -async function readArtifactEvidence(repoRoot) { - const [catalog, report] = await Promise.all([ - readJsonFileSummary(repoRoot, artifactCatalogPath, summarizeArtifactCatalog), - readJsonFileSummary(repoRoot, artifactReportPath, summarizeArtifactReport) - ]); - return { catalog, report }; -} - -async function resolveTargetRef(ctx, targetRef) { - const target = await ctx.runCommand("git", ["rev-parse", "--verify", `${targetRef}^{commit}`], { - cwd: ctx.repoRoot, - timeoutMs: 10000 - }); - if (target.code !== 0) { - throw new DevCdApplyError(`target ref ${targetRef} could not be resolved`, { - code: "target-ref-unresolved", - targetRef, - stderr: redactSensitiveText(target.stderr) - }); - } - const head = await ctx.runCommand("git", ["rev-parse", "--verify", "HEAD^{commit}"], { - cwd: ctx.repoRoot, - timeoutMs: 10000 - }); - if (head.code !== 0) { - throw new DevCdApplyError("HEAD could not be resolved", { - code: "head-unresolved", - stderr: redactSensitiveText(head.stderr) - }); - } - const commitId = target.stdout.trim(); - const headCommitId = head.stdout.trim(); - return { - ref: targetRef, - commitId, - shortCommitId: shortCommit(commitId), - headCommitId, - headShortCommitId: shortCommit(headCommitId), - headMatchesTarget: commitId === headCommitId - }; -} - -async function gitOutput(ctx, args, { allowFailure = false, timeoutMs = 10000 } = {}) { - const result = await ctx.runCommand("git", args, { - cwd: ctx.repoRoot, - timeoutMs - }); - if (result.code !== 0) { - if (allowFailure) return null; - throw new DevCdApplyError("git command failed while resolving DEV CD provenance", { - code: "git-provenance-failed", - command: shellCommand("git", args), - stderr: redactSensitiveText(result.stderr || result.stdout) - }); - } - return result.stdout.trim(); -} - -async function commitParents(ctx, commitId) { - const line = await gitOutput(ctx, ["rev-list", "--parents", "-n", "1", commitId], { allowFailure: true }); - if (!line) return []; - return line.split(/\s+/u).slice(1); -} - -async function shortGitCommit(ctx, commitId) { - const value = await gitOutput(ctx, ["rev-parse", "--short=7", commitId], { allowFailure: true }); - return value || shortCommit(commitId); -} - -async function filesChangedBetween(ctx, left, right) { - const output = await gitOutput(ctx, ["diff-tree", "--no-commit-id", "--name-only", "-r", left, right], { - allowFailure: true - }); - if (!output) return []; - return output.split(/\r?\n/u).map((item) => item.trim()).filter(Boolean); -} - -async function latestArtifactMergeCommit(ctx, controlCommit) { - const output = await gitOutput(ctx, [ - "log", - "--first-parent", - "--format=%H", - "-n", - "1", - controlCommit, - "--", - ...artifactDesiredStatePaths - ], { allowFailure: true }); - return output?.split(/\r?\n/u).find(Boolean) ?? null; -} - -async function resolveArtifactBoundary(ctx, control, promotion, deployBefore, artifactEvidence) { - const mergeCommitId = await latestArtifactMergeCommit(ctx, control.commitId); - if (!mergeCommitId) { - return { - status: "unavailable", - reason: "No first-parent commit touching artifact desired-state files was found.", - desiredStatePaths: artifactDesiredStatePaths - }; - } - - const parents = await commitParents(ctx, mergeCommitId); - const mergeShortCommitId = await shortGitCommit(ctx, mergeCommitId); - const firstParent = parents[0] ?? null; - const secondParent = parents[1] ?? null; - const promotionParentMatches = firstParent ? commitMatches(firstParent, promotion.commitId) : false; - const changedPaths = firstParent ? await filesChangedBetween(ctx, firstParent, mergeCommitId) : []; - const touchedDesiredStatePaths = artifactDesiredStatePaths.filter((entry) => changedPaths.includes(entry)); - const artifactCommitId = secondParent || mergeCommitId; - const artifactShortCommitId = await shortGitCommit(ctx, artifactCommitId); - const artifactParents = secondParent ? await commitParents(ctx, secondParent) : []; - const artifactParentMatches = artifactParents.some((parent) => commitMatches(parent, promotion.commitId)); - const catalogCommitMatches = commitMatches(artifactEvidence.catalog.commitId, promotion.commitId); - const reportSource = artifactEvidence.report.sourceCommitId ?? artifactEvidence.report.commitId; - const reportCommitMatches = commitMatches(reportSource, promotion.commitId); - const deployCommitMatches = commitMatches(deployBefore.commitId, promotion.commitId); - const isMergeCommit = parents.length >= 2; - const desiredStateStatus = ( - deployCommitMatches && - catalogCommitMatches && - ( - promotionParentMatches || - artifactParentMatches || - commitMatches(mergeCommitId, promotion.commitId) - ) - ) ? "pass" : "degraded"; - const reportStatus = artifactEvidence.report.exists - ? (reportCommitMatches ? "matches_promotion" : "stale_or_unrelated") - : "missing"; - - return { - status: desiredStateStatus, - reportStatus, - reportIsReleaseGate: false, - reportPolicy: "CI artifact reports are audit snapshots; deploy.json, artifact catalog, workloads, and registry manifests are the CD release gate.", - controlCommit: { - commitId: control.commitId, - shortCommitId: control.shortCommitId - }, - artifactMergeCommit: { - commitId: mergeCommitId, - shortCommitId: mergeShortCommitId, - isMergeCommit, - parents, - promotionParent: firstParent, - artifactParent: secondParent, - promotionParentMatches, - changedPaths: touchedDesiredStatePaths - }, - artifactCommit: { - commitId: artifactCommitId, - shortCommitId: artifactShortCommitId, - parents: artifactParents, - promotionParentMatches: artifactParentMatches - }, - desiredState: { - deployCommitId: deployBefore.commitId, - deployJsonHash: deployBefore.hash, - catalogCommitId: artifactEvidence.catalog.commitId, - catalogHash: artifactEvidence.catalog.hash, - reportCommitId: artifactEvidence.report.commitId, - reportSourceCommitId: artifactEvidence.report.sourceCommitId, - reportHash: artifactEvidence.report.hash, - deployCommitMatches, - catalogCommitMatches, - reportCommitMatches, - reportExists: artifactEvidence.report.exists, - desiredStatePaths: artifactDesiredStatePaths - } - }; -} - -async function resolveCommitRef(ctx, ref) { - const result = await ctx.runCommand("git", ["rev-parse", "--verify", `${ref}^{commit}`], { - cwd: ctx.repoRoot, - timeoutMs: 10000 - }); - if (result.code !== 0) { - throw new DevCdApplyError(`promotion commit ${ref} could not be resolved`, { - code: "promotion-commit-unresolved", - targetRef: ref, - stderr: redactSensitiveText(result.stderr) - }); - } - const commitId = result.stdout.trim(); - return { - commitId, - shortCommitId: shortCommit(commitId) - }; -} - -function compactDesiredStateCheck(result) { - const parsed = parseJsonMaybe(result.stdout ?? ""); - return { - status: result.code === 0 ? "pass" : "blocked", - code: result.code, - summary: parsed?.summary ?? null, - convergence: parsed?.target?.convergence?.state ?? null, - diagnostics: Array.isArray(parsed?.diagnostics) - ? parsed.diagnostics.slice(0, 5).map((item) => ({ - code: item.code ?? null, - path: item.path ?? null, - message: item.message ?? null - })) - : [], - stderrTail: redactSensitiveText(result.stderr ?? "").trim().slice(-1000) - }; -} - -async function checkPromotionDesiredState(ctx, promotionCommit) { - const result = await ctx.runCommand(process.execPath, [ - "scripts/deploy-desired-state-plan.mjs", - "--promotion-commit", - promotionCommit, - "--check", - "--pretty" - ], { - cwd: ctx.repoRoot, - timeoutMs: 120000 - }); - return compactDesiredStateCheck(result); -} - -async function resolveEffectiveTarget(ctx, args, deployBefore, artifactEvidence = null) { - const control = await resolveTargetRef(ctx, args.targetRef); - if (deployJsonMatchesTarget(deployBefore, control)) { - return { - ...control, - controlRef: { - ref: control.ref, - commitId: control.commitId, - shortCommitId: control.shortCommitId - }, - promotionSource: "target-ref", - publishRequired: true, - desiredStateCheck: null, - artifactBoundary: artifactEvidence - ? await resolveArtifactBoundary(ctx, control, control, deployBefore, artifactEvidence) - : null - }; - } - - const promotion = await resolveCommitRef(ctx, deployBefore.commitId); - const desiredStateCheck = await checkPromotionDesiredState(ctx, promotion.commitId); - if (desiredStateCheck.status !== "pass") { - return { - ...control, - controlRef: { - ref: control.ref, - commitId: control.commitId, - shortCommitId: control.shortCommitId - }, - promotionSource: "target-ref", - publishRequired: true, - desiredStateCheck, - artifactBoundary: artifactEvidence - ? await resolveArtifactBoundary(ctx, control, promotion, deployBefore, artifactEvidence) - : null - }; - } - - return { - ref: control.ref, - commitId: promotion.commitId, - shortCommitId: promotion.shortCommitId, - headCommitId: control.headCommitId, - headShortCommitId: control.headShortCommitId, - headMatchesTarget: control.headMatchesTarget, - controlRef: { - ref: control.ref, - commitId: control.commitId, - shortCommitId: control.shortCommitId - }, - promotionSource: "deploy-json", - publishRequired: false, - desiredStateCheck, - artifactBoundary: artifactEvidence - ? await resolveArtifactBoundary(ctx, control, promotion, deployBefore, artifactEvidence) - : null - }; -} - -function validateArgs(args) { - const blockers = []; - if (!args.apply && !args.status) { - blockers.push({ - type: "safety_blocker", - scope: "apply-mode", - status: "open", - summary: "DEV CD side effects require --apply; use --status or --dry-run for read-only observability." - }); - } - if (args.apply && (!args.confirmDev || !args.confirmedNonProduction)) { - blockers.push({ - type: "safety_blocker", - scope: "dev-confirmation", - status: "open", - summary: "DEV CD apply requires --confirm-dev and --confirmed-non-production." - }); - } - if (args.breakStaleLock && !args.confirmDev) { - blockers.push({ - type: "safety_blocker", - scope: "break-stale-lock-confirmation", - status: "open", - summary: "Breaking a stale DEV CD lock requires --break-stale-lock and --confirm-dev." - }); - } - if (!Number.isInteger(args.ttlSeconds) || args.ttlSeconds < 60 || args.ttlSeconds > 24 * 60 * 60) { - blockers.push({ - type: "safety_blocker", - scope: "lock-ttl", - status: "open", - summary: "--ttl-seconds must be an integer between 60 and 86400." - }); - } - for (const forbidden of ["--prod", "--production", "--read-secret", "--force-push"]) { - if (args.flags.has(forbidden)) { - blockers.push({ - type: "safety_blocker", - scope: forbidden.slice(2), - status: "open", - summary: `${forbidden} is forbidden for DEV CD apply.` - }); - } - } - if (blockers.length > 0) { - throw new DevCdApplyError("invalid DEV CD apply arguments", { - code: "invalid-arguments", - blockers - }); - } -} - -function artifactBoundaryRequiresApplyBlock(target) { - if (target.publishRequired !== false) return false; - if (target.artifactBoundary?.status !== "degraded") return false; - - const desiredState = target.artifactBoundary.desiredState ?? {}; - const desiredStateCheckPass = target.desiredStateCheck?.status === "pass"; - const releaseGatePinsMatch = - desiredState.deployCommitMatches === true && - desiredState.catalogCommitMatches === true; - - return !(desiredStateCheckPass && releaseGatePinsMatch); -} - -function lockAnnotationValue(value) { - if (value === undefined || value === null) return ""; - if (typeof value === "string") return value; - return JSON.stringify(value); -} - -export function buildLockAnnotations(lock) { - const annotations = {}; - for (const [field, key] of Object.entries(lockAnnotationFields)) { - if (Object.hasOwn(lock, field)) { - annotations[key] = lockAnnotationValue(lock[field]); - } - } - return annotations; -} - -function readAnnotation(annotations, field) { - return annotations?.[lockAnnotationFields[field]] ?? null; -} - -function readJsonAnnotation(annotations, field) { - const value = readAnnotation(annotations, field); - return value ? parseJsonMaybe(value) : null; -} - -export function parseDeployLock(lease) { - if (!lease) return null; - const annotations = lease.metadata?.annotations ?? {}; - const ttlSeconds = Number.parseInt( - readAnnotation(annotations, "ttlSeconds") || lease.spec?.leaseDurationSeconds || `${defaultTtlSeconds}`, - 10 - ); - const updatedAt = - readAnnotation(annotations, "updatedAt") || - lease.spec?.renewTime || - lease.metadata?.creationTimestamp || - null; - const transactionId = readAnnotation(annotations, "transactionId") || lease.spec?.holderIdentity || null; - const ownerTaskId = readAnnotation(annotations, "ownerTaskId") || lease.spec?.holderIdentity || null; - return { - lockBackend: readAnnotation(annotations, "lockBackend") || "Lease", - lockName: lease.metadata?.name ?? defaultLockName, - resourceVersion: lease.metadata?.resourceVersion ?? null, - holderIdentity: lease.spec?.holderIdentity ?? null, - promotionCommit: readAnnotation(annotations, "promotionCommit"), - deployJsonHash: readAnnotation(annotations, "deployJsonHash"), - ownerTaskId, - transactionId, - phase: readAnnotation(annotations, "phase") || "unknown", - startedAt: readAnnotation(annotations, "startedAt") || lease.spec?.acquireTime || lease.metadata?.creationTimestamp || null, - updatedAt, - ttlSeconds: Number.isInteger(ttlSeconds) ? ttlSeconds : defaultTtlSeconds, - liveBefore: readJsonAnnotation(annotations, "liveBefore"), - targetNamespace: readAnnotation(annotations, "targetNamespace") || lease.metadata?.namespace || defaultNamespace, - targetRef: readAnnotation(annotations, "targetRef"), - releasedAt: readAnnotation(annotations, "releasedAt"), - releaseStatus: readAnnotation(annotations, "releaseStatus"), - staleLockBrokenAt: readAnnotation(annotations, "staleLockBrokenAt"), - staleLockPrevious: readJsonAnnotation(annotations, "staleLockPrevious") - }; -} - -export function classifyDeployLock(lock, now = new Date()) { - if (!lock || lock.phase === "released") { - return { - held: false, - stale: false, - retryAfterSeconds: 0, - expiresAt: null - }; - } - const updatedAtMs = Number.isFinite(Date.parse(lock.updatedAt)) ? Date.parse(lock.updatedAt) : 0; - const expiresAtMs = updatedAtMs + lock.ttlSeconds * 1000; - const retryAfterSeconds = Math.max(0, Math.ceil((expiresAtMs - now.getTime()) / 1000)); - return { - held: retryAfterSeconds > 0, - stale: retryAfterSeconds <= 0, - retryAfterSeconds, - expiresAt: Number.isFinite(expiresAtMs) ? new Date(expiresAtMs).toISOString() : null - }; -} - -export function deployLockHeldFailure(lock, now = new Date()) { - const classification = classifyDeployLock(lock, now); - return { - ok: false, - status: "blocked", - error: "deploy-lock-held", - code: "deploy-lock-held", - holder: lock.ownerTaskId || lock.transactionId || lock.holderIdentity || "unknown", - promotionCommit: lock.promotionCommit ?? null, - deployJsonHash: lock.deployJsonHash ?? null, - transactionId: lock.transactionId ?? null, - phase: lock.phase ?? "unknown", - targetNamespace: lock.targetNamespace ?? defaultNamespace, - retryAfterSeconds: classification.retryAfterSeconds, - stale: classification.stale, - requiresBreakStaleLock: classification.stale, - lockName: lock.lockName ?? defaultLockName, - lockBackend: lock.lockBackend ?? "Lease", - summary: classification.stale - ? "A stale DEV CD transaction lock exists; rerun with --break-stale-lock --confirm-dev only after confirming the holder is inactive." - : "Another DEV CD transaction is already running; retry after the current holder releases the lock.", - mutationAttempted: false, - prodTouched: false - }; -} - -function isNotFound(result) { - return /not\s*found|notfound/i.test(`${result.stderr}\n${result.stdout}`); -} - -function isAlreadyExists(result) { - return /already\s*exists|alreadyexists/i.test(`${result.stderr}\n${result.stdout}`); -} - -function isConflict(result) { - return /conflict|object has been modified|operation cannot be fulfilled/i.test(`${result.stderr}\n${result.stdout}`); -} - -function compactLockSummary(lock) { - if (!lock) return null; - return { - lockName: lock.lockName ?? defaultLockName, - lockBackend: lock.lockBackend ?? "Lease", - ownerTaskId: lock.ownerTaskId ?? null, - transactionId: lock.transactionId ?? null, - holderIdentity: lock.holderIdentity ?? null, - phase: lock.phase ?? null, - promotionCommit: lock.promotionCommit ?? null, - deployJsonHash: lock.deployJsonHash ?? null, - updatedAt: lock.updatedAt ?? null, - ttlSeconds: lock.ttlSeconds ?? null, - targetNamespace: lock.targetNamespace ?? defaultNamespace, - targetRef: lock.targetRef ?? null, - releasedAt: lock.releasedAt ?? null, - releaseStatus: lock.releaseStatus ?? null, - staleLockBrokenAt: lock.staleLockBrokenAt ?? null - }; -} - -function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTaskId, startedAt, phase, liveBefore, staleLockPrevious = null }) { - const lock = { - promotionCommit: target.shortCommitId, - deployJsonHash: deployBefore.hash, - ownerTaskId, - transactionId, - phase, - startedAt, - updatedAt: startedAt, - ttlSeconds: args.ttlSeconds, - liveBefore, - targetNamespace: args.targetNamespace, - targetRef: target.ref, - lockBackend: "Lease" - }; - if (staleLockPrevious) { - lock.staleLockBrokenAt = startedAt; - lock.staleLockPrevious = staleLockPrevious; - } - return { - apiVersion: "coordination.k8s.io/v1", - kind: "Lease", - metadata: { - name: args.lockName, - namespace: args.targetNamespace, - labels: { - "app.kubernetes.io/part-of": "hwlab", - "hwlab.pikastech.local/profile": "dev", - "hwlab.pikastech.local/cd-lock": "true" - }, - annotations: buildLockAnnotations(lock) - }, - spec: { - holderIdentity: `${ownerTaskId}/${transactionId}`, - leaseDurationSeconds: args.ttlSeconds, - acquireTime: startedAt, - renewTime: startedAt - } - }; -} - -function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, phase, startedAt, updatedAt, liveBefore, staleLockPrevious = null }) { - const lock = { - promotionCommit: target.shortCommitId, - deployJsonHash: deployBefore.hash, - ownerTaskId, - transactionId, - phase, - startedAt, - updatedAt, - ttlSeconds: args.ttlSeconds, - liveBefore, - targetNamespace: args.targetNamespace, - targetRef: target.ref, - lockBackend: "Lease" - }; - if (staleLockPrevious) { - lock.staleLockBrokenAt = updatedAt; - lock.staleLockPrevious = staleLockPrevious; - } - return { - metadata: { - labels: { - "app.kubernetes.io/part-of": "hwlab", - "hwlab.pikastech.local/profile": "dev", - "hwlab.pikastech.local/cd-lock": "true" - }, - annotations: buildLockAnnotations(lock) - }, - spec: { - holderIdentity: `${ownerTaskId}/${transactionId}`, - leaseDurationSeconds: args.ttlSeconds, - acquireTime: startedAt, - renewTime: updatedAt - } - }; -} - -async function kubectl(ctx, kubectlContext, args, options = {}) { - const result = await ctx.runCommand(kubectlContext.executor, args, { - cwd: ctx.repoRoot, - env: kubectlContext.env, - input: options.input, - timeoutMs: options.timeoutMs ?? 30000 - }); - return { - ...result, - command: shellCommand("kubectl", args), - stdout: result.stdout ?? "", - stderr: result.stderr ?? "" - }; -} - -async function replaceLease(ctx, kubectlContext, args, lease, timeoutMs = 30000) { - return kubectl( - ctx, - kubectlContext, - ["-n", args.targetNamespace, "replace", "-f", "-", "-o", "json"], - { - input: JSON.stringify(lease), - timeoutMs - } - ); -} - -function mergeLease(lease, patch, { resetLockAnnotations = false } = {}) { - const baseAnnotations = { ...(lease.metadata?.annotations ?? {}) }; - if (resetLockAnnotations) { - for (const key of Object.keys(baseAnnotations).filter((name) => name.startsWith(`${lockAnnotationPrefix}/`))) { - delete baseAnnotations[key]; - } - } - return { - ...lease, - metadata: { - ...lease.metadata, - ...(patch.metadata ?? {}), - labels: { - ...(lease.metadata?.labels ?? {}), - ...(patch.metadata?.labels ?? {}) - }, - annotations: { - ...baseAnnotations, - ...(patch.metadata?.annotations ?? {}) - } - }, - spec: { - ...(lease.spec ?? {}), - ...(patch.spec ?? {}) - } - }; -} - -async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefore, transactionId, ownerTaskId, liveBefore, now }) { - const startedAt = kubernetesMicroTime(now); - const get = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "get", "lease", args.lockName, "-o", "json"]); - if (get.code === 0) { - const lease = JSON.parse(get.stdout); - const existing = parseDeployLock(lease); - const classification = classifyDeployLock(existing, now); - if (classification.held) { - const failure = deployLockHeldFailure(existing, now); - throw new DevCdApplyError(failure.summary, { - code: "deploy-lock-held", - lockFailure: failure - }); - } - if (classification.stale && !args.breakStaleLock) { - const failure = deployLockHeldFailure(existing, now); - throw new DevCdApplyError(failure.summary, { - code: "deploy-lock-held", - lockFailure: failure - }); - } - - const patch = lockPatch({ - args, - target, - deployBefore, - transactionId, - ownerTaskId, - phase: "publishing", - startedAt, - updatedAt: startedAt, - liveBefore, - staleLockPrevious: classification.stale ? existing : null - }); - const replaced = await replaceLease(ctx, kubectlContext, args, mergeLease(lease, patch, { resetLockAnnotations: true })); - if (isConflict(replaced)) { - return acquireDeployLock({ ctx, args, kubectlContext, target, deployBefore, transactionId, ownerTaskId, liveBefore, now }); - } - if (replaced.code !== 0) { - throw new DevCdApplyError("failed to acquire DEV CD Lease lock", { - code: "lock-acquire-failed", - stderr: redactSensitiveText(replaced.stderr || replaced.stdout) - }); - } - return { - acquired: true, - lock: parseDeployLock(JSON.parse(replaced.stdout)), - staleBreak: classification.stale ? existing : null, - command: replaced.command - }; - } - - if (!isNotFound(get)) { - throw new DevCdApplyError("failed to read DEV CD Lease lock", { - code: "lock-read-failed", - stderr: redactSensitiveText(get.stderr || get.stdout) - }); - } - - const manifest = buildLeaseManifest({ - args, - target, - deployBefore, - transactionId, - ownerTaskId, - startedAt, - phase: "publishing", - liveBefore - }); - const created = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "create", "-f", "-", "-o", "json"], { - input: JSON.stringify(manifest), - timeoutMs: 30000 - }); - if (created.code === 0) { - return { - acquired: true, - lock: parseDeployLock(JSON.parse(created.stdout)), - staleBreak: null, - command: created.command - }; - } - if (isAlreadyExists(created)) { - return acquireDeployLock({ ctx, args, kubectlContext, target, deployBefore, transactionId, ownerTaskId, liveBefore, now }); - } - throw new DevCdApplyError("failed to create DEV CD Lease lock", { - code: "lock-create-failed", - stderr: redactSensitiveText(created.stderr || created.stdout) - }); -} - -async function readOwnedLease({ ctx, args, kubectlContext, transactionId, timeoutMs = 15000 }) { - const get = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "get", "lease", args.lockName, "-o", "json"], { - timeoutMs - }); - if (get.code !== 0) { - throw new DevCdApplyError("failed to read DEV CD lock ownership", { - code: "lock-owner-read-failed", - stderr: redactSensitiveText(get.stderr || get.stdout) - }); - } - const lease = JSON.parse(get.stdout); - const lock = parseDeployLock(lease); - if (lock.transactionId !== transactionId) { - throw new DevCdApplyError("DEV CD lock ownership changed before transaction update", { - code: "lock-owner-mismatch", - holder: lock.ownerTaskId || lock.transactionId || "unknown", - phase: lock.phase - }); - } - return { lease, lock }; -} - -async function patchOwnedLease({ ctx, args, kubectlContext, lockState, transactionId, patch, errorCode, errorMessage }) { - const { lease } = await readOwnedLease({ ctx, args, kubectlContext, transactionId }); - const result = await replaceLease(ctx, kubectlContext, args, mergeLease(lease, patch)); - if (result.code !== 0) { - throw new DevCdApplyError(errorMessage, { - code: errorCode, - stderr: redactSensitiveText(result.stderr || result.stdout) - }); - } - lockState.lock = parseDeployLock(JSON.parse(result.stdout)); - return lockState.lock; -} - -async function updateDeployLockPhase({ ctx, args, kubectlContext, transactionId, lockState, phase, status = "running" }) { - const updatedAt = kubernetesMicroTime(ctx.now()); - const patch = { - metadata: { - annotations: { - [lockAnnotationFields.phase]: phase, - [lockAnnotationFields.updatedAt]: updatedAt, - [lockAnnotationFields.releaseStatus]: status - } - }, - spec: { - renewTime: updatedAt - } - }; - return await patchOwnedLease({ - ctx, - args, - kubectlContext, - transactionId, - lockState, - patch, - errorCode: "lock-phase-update-failed", - errorMessage: `failed to update DEV CD lock phase to ${phase}` - }); -} - -async function updateDeployLockLiveBefore({ ctx, args, kubectlContext, transactionId, lockState, liveBefore }) { - const updatedAt = kubernetesMicroTime(ctx.now()); - const patch = { - metadata: { - annotations: { - [lockAnnotationFields.liveBefore]: JSON.stringify(liveBefore), - [lockAnnotationFields.updatedAt]: updatedAt - } - }, - spec: { - renewTime: updatedAt - } - }; - return await patchOwnedLease({ - ctx, - args, - kubectlContext, - transactionId, - lockState, - patch, - errorCode: "lock-live-before-update-failed", - errorMessage: "failed to update DEV CD lock liveBefore evidence" - }); -} - -async function releaseDeployLock({ ctx, args, kubectlContext, transactionId, status }) { - const get = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "get", "lease", args.lockName, "-o", "json"], { - timeoutMs: 15000 - }); - if (get.code !== 0) { - return { - status: "release_read_failed", - command: get.command, - error: redactSensitiveText(get.stderr || get.stdout) - }; - } - const current = parseDeployLock(JSON.parse(get.stdout)); - if (current.transactionId !== transactionId) { - return { - status: "not_owner", - holder: current.ownerTaskId || current.transactionId || "unknown", - phase: current.phase - }; - } - const releasedAt = kubernetesMicroTime(ctx.now()); - const patch = { - metadata: { - annotations: { - [lockAnnotationFields.phase]: "released", - [lockAnnotationFields.updatedAt]: releasedAt, - [lockAnnotationFields.releasedAt]: releasedAt, - [lockAnnotationFields.releaseStatus]: status - } - }, - spec: { - holderIdentity: "", - leaseDurationSeconds: 1, - renewTime: releasedAt - } - }; - const result = await replaceLease(ctx, kubectlContext, args, mergeLease(JSON.parse(get.stdout), patch), 15000); - if (isConflict(result)) { - return { - status: "release_conflict", - command: result.command, - error: "Lease changed before release; lock was not overwritten." - }; - } - if (result.code !== 0) { - return { - status: "release_failed", - command: result.command, - error: redactSensitiveText(result.stderr || result.stdout) - }; - } - return { - status: "released", - command: result.command, - lock: parseDeployLock(JSON.parse(result.stdout)) - }; -} - -function commandSummary(result, maxLength = 500) { - return oneLine(redactSensitiveText(result?.stderr || result?.stdout || "")).slice(0, maxLength); -} - -function envWithoutKubeconfig(env) { - const sanitized = { ...env }; - delete sanitized.KUBECONFIG; - delete sanitized.HWLAB_DEV_KUBECONFIG; - return sanitized; -} - -function parseNodeNames(stdout) { - return String(stdout ?? "").trim().split(/\s+/u).filter(Boolean); -} - -function parseSecretDescribeKeys(stdout) { - const keys = new Set(); - let inData = false; - for (const rawLine of String(stdout ?? "").split(/\r?\n/u)) { - const line = rawLine.trim(); - if (/^Data\b/iu.test(line)) { - inData = true; - continue; - } - if (!inData || line === "" || /^=+$/u.test(line)) continue; - const match = line.match(/^([A-Za-z0-9_.-]+):\s+\d+\s+bytes\b/iu); - if (match) keys.add(match[1]); - } - return [...keys].sort(); -} - -function parseKeyLines(stdout) { - return String(stdout ?? "") - .split(/\r?\n/u) - .map((line) => line.trim()) - .filter(Boolean); -} - -function commandObservation(args, result) { - return { - command: shellCommand("kubectl", args), - code: result.code ?? null, - ok: result.code === 0, - summary: commandSummary(result) - }; -} - -async function preflightKubectl(ctx, kubectlContext, args, options = {}) { - const result = await ctx.runCommand(kubectlContext.executor, args, { - cwd: ctx.repoRoot, - env: options.env ?? kubectlContext.env, - timeoutMs: options.timeoutMs ?? 15000 - }); - return { - ...result, - stdout: result.stdout ?? "", - stderr: result.stderr ?? "", - command: shellCommand("kubectl", args) - }; -} - -function addControlPlaneBlockers(blockers, observation, { scopePrefix, requiredForApply }) { - if (observation.contextName && forbiddenControlPlanePattern.test(observation.contextName)) { - blockers.push(blocker({ - scope: `${scopePrefix}-context`, - reason: `kubectl context resolved to forbidden control plane ${observation.contextName}.`, - impact: "DEV CD cannot prove it is targeting D601 native k3s before mutation.", - safeNextAction: `Use KUBECONFIG=${d601NativeKubeconfigPath} and remove or repair the stale Docker Desktop/default kube context before retrying.`, - retryable: true - })); - } - if (observation.serverUrl && forbiddenControlPlanePattern.test(observation.serverUrl)) { - blockers.push(blocker({ - scope: `${scopePrefix}-server`, - reason: `kubectl server resolved to forbidden endpoint ${observation.serverUrl}.`, - impact: "DEV CD could write to the wrong control plane or treat stale Docker Desktop output as DEV evidence.", - safeNextAction: `Point kubectl at D601 native k3s with KUBECONFIG=${d601NativeKubeconfigPath} before retrying.`, - retryable: true - })); - } - if (observation.nodeNames.includes("desktop-control-plane")) { - blockers.push(blocker({ - scope: `${scopePrefix}-nodes`, - reason: "kubectl observed node desktop-control-plane.", - impact: "DEV CD is observing Docker Desktop Kubernetes instead of D601 native k3s.", - safeNextAction: `Stop and rerun only after kubectl nodes come from KUBECONFIG=${d601NativeKubeconfigPath} and include d601.`, - retryable: true - })); - } else if (requiredForApply && !observation.nodeNames.includes("d601")) { - blockers.push(blocker({ - scope: `${scopePrefix}-nodes`, - reason: `kubectl nodes did not include d601; observed ${observation.nodeNames.join(",") || "none"}.`, - impact: "DEV apply/job/rollout side effects are not allowed without proving the D601 native k3s node.", - safeNextAction: `Restore access to KUBECONFIG=${d601NativeKubeconfigPath} and verify kubectl get nodes includes d601.`, - retryable: true - })); - } else if (!requiredForApply && observation.nodeNames.length > 0 && !observation.nodeNames.includes("d601")) { - blockers.push(blocker({ - scope: `${scopePrefix}-nodes`, - reason: `bare kubectl observed non-D601 nodes ${observation.nodeNames.join(",")}.`, - impact: "A second or stale control plane may still own hwlab-dev evidence, so DEV CD refuses to mutate.", - safeNextAction: "Clean the default kubectl context or ensure bare kubectl cannot resolve stale HWLAB DEV resources, then retry.", - retryable: true - })); - } -} - -async function observeControlPlane(ctx, kubectlContext, { env, label, requiredForApply }) { - const [context, server, nodes] = await Promise.all([ - preflightKubectl(ctx, kubectlContext, ["config", "current-context"], { env, timeoutMs: 10000 }), - preflightKubectl(ctx, kubectlContext, ["config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"], { env, timeoutMs: 10000 }), - preflightKubectl(ctx, kubectlContext, ["get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], { env, timeoutMs: 15000 }) - ]); - return { - label, - contextCommand: commandObservation(["config", "current-context"], context), - serverCommand: commandObservation(["config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"], server), - nodesCommand: commandObservation(["get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], nodes), - contextName: context.code === 0 ? oneLine(context.stdout) : null, - serverUrl: server.code === 0 ? oneLine(server.stdout) : null, - nodeNames: nodes.code === 0 ? parseNodeNames(nodes.stdout) : [], - status: context.code === 0 && server.code === 0 && nodes.code === 0 ? "observed" : requiredForApply ? "blocked" : "unavailable" - }; -} - -async function observeBareKubectl(ctx, kubectlContext, forcedObservation) { - const bareEnv = envWithoutKubeconfig(ctx.env); - const bare = await observeControlPlane(ctx, kubectlContext, { - env: bareEnv, - label: "bare-kubectl", - requiredForApply: false - }); - const namespace = await preflightKubectl(ctx, kubectlContext, ["get", "namespace", defaultNamespace, "-o", "name"], { - env: bareEnv, - timeoutMs: 10000 - }); - const secondControlPlane = - namespace.code === 0 && - ( - (bare.serverUrl && forcedObservation.serverUrl && bare.serverUrl !== forcedObservation.serverUrl) || - (!bare.serverUrl && bare.contextName && bare.contextName !== forcedObservation.contextName) - ); - return { - ...bare, - namespace: commandObservation(["get", "namespace", defaultNamespace, "-o", "name"], namespace), - hwlabDevNamespaceObserved: namespace.code === 0, - secondControlPlaneCandidate: secondControlPlane - }; -} - -async function observeRequiredSecretRef(ctx, kubectlContext, args, ref) { - const existsArgs = ["-n", args.targetNamespace, "get", "secret", ref.secretName, "-o", "name"]; - const describeArgs = ["-n", args.targetNamespace, "describe", "secret", ref.secretName]; - const exists = await preflightKubectl(ctx, kubectlContext, existsArgs, { timeoutMs: 10000 }); - if (exists.code !== 0) { - return { - secretName: ref.secretName, - secretKey: ref.secretKey, - status: "missing-secret", - exists: false, - keyPresent: false, - keysObserved: [], - existenceCommand: shellCommand("kubectl", existsArgs), - keyCommand: null, - error: commandSummary(exists), - secretValueRead: false, - secretValuePrinted: false - }; - } - const describe = await preflightKubectl(ctx, kubectlContext, describeArgs, { timeoutMs: 10000 }); - if (describe.code !== 0) { - return { - secretName: ref.secretName, - secretKey: ref.secretKey, - status: "key-observation-blocked", - exists: true, - keyPresent: false, - keysObserved: [], - existenceCommand: shellCommand("kubectl", existsArgs), - keyCommand: shellCommand("kubectl", describeArgs), - error: commandSummary(describe), - secretValueRead: false, - secretValuePrinted: false - }; - } - const keysObserved = parseSecretDescribeKeys(describe.stdout); - const keyPresent = keysObserved.includes(ref.secretKey); - return { - secretName: ref.secretName, - secretKey: ref.secretKey, - status: keyPresent ? "present" : "missing-key", - exists: true, - keyPresent, - keysObserved, - existenceCommand: shellCommand("kubectl", existsArgs), - keyCommand: shellCommand("kubectl", describeArgs), - secretValueRead: false, - secretValuePrinted: false - }; -} - -function secretRefBlocker(ref, observation) { - const scope = `secretref:${ref.secretName}/${ref.secretKey}`; - if (observation.status === "missing-secret") { - return blocker({ - type: "runtime_blocker", - scope, - reason: `Required DEV Secret ${ref.secretName} was not observed in ${defaultNamespace}.`, - impact: "Runtime provisioning, migration, deploy apply, or Code Agent provider startup would fail after mutation.", - safeNextAction: `Create or restore SecretRef ${ref.secretName}/${ref.secretKey} in ${defaultNamespace} without printing the Secret value, then retry DEV CD.`, - retryable: true - }); - } - if (observation.status === "key-observation-blocked") { - return blocker({ - type: "runtime_blocker", - scope, - reason: `Required DEV Secret ${ref.secretName} exists but key names could not be verified.`, - impact: "DEV CD cannot prove the pod SecretRef contract before creating runtime Jobs or applying workloads.", - safeNextAction: `Restore metadata/key-name visibility for ${ref.secretName} without granting or printing Secret values, then retry.`, - retryable: true - }); - } - return blocker({ - type: "runtime_blocker", - scope, - reason: `Required DEV Secret key ${ref.secretName}/${ref.secretKey} was not observed.`, - impact: "The target workload or runtime Job would reference a missing Secret key after mutation.", - safeNextAction: `Add key ${ref.secretKey} to Secret ${ref.secretName} without printing the value, then retry DEV CD.`, - retryable: true - }); -} - -async function observeRequiredConfigMapRef(ctx, kubectlContext, args, ref) { - const existsArgs = ["-n", args.targetNamespace, "get", "configmap", ref.configMapName, "-o", "name"]; - const keyArgs = [ - "-n", - args.targetNamespace, - "get", - "configmap", - ref.configMapName, - "-o", - "go-template={{range $k,$v := .data}}{{printf \"%s\\n\" $k}}{{end}}" - ]; - const exists = await preflightKubectl(ctx, kubectlContext, existsArgs, { timeoutMs: 10000 }); - if (exists.code !== 0) { - return { - configMapName: ref.configMapName, - configMapKey: ref.configMapKey, - status: "missing-configmap", - exists: false, - keyPresent: false, - keysObserved: [], - existenceCommand: shellCommand("kubectl", existsArgs), - keyCommand: null, - error: commandSummary(exists), - configMapValueRead: false, - configMapValuePrinted: false - }; - } - const keys = await preflightKubectl(ctx, kubectlContext, keyArgs, { timeoutMs: 10000 }); - if (keys.code !== 0) { - return { - configMapName: ref.configMapName, - configMapKey: ref.configMapKey, - status: "key-observation-blocked", - exists: true, - keyPresent: false, - keysObserved: [], - existenceCommand: shellCommand("kubectl", existsArgs), - keyCommand: shellCommand("kubectl", keyArgs), - error: commandSummary(keys), - configMapValueRead: false, - configMapValuePrinted: false - }; - } - const keysObserved = parseKeyLines(keys.stdout); - const keyPresent = keysObserved.includes(ref.configMapKey); - return { - configMapName: ref.configMapName, - configMapKey: ref.configMapKey, - status: keyPresent ? "present" : "missing-key", - exists: true, - keyPresent, - keysObserved, - existenceCommand: shellCommand("kubectl", existsArgs), - keyCommand: shellCommand("kubectl", keyArgs), - configMapValueRead: false, - configMapValuePrinted: false - }; -} - -function configMapRefBlocker(ref, observation) { - const scope = `configmap:${ref.configMapName}/${ref.configMapKey}`; - if (observation.status === "missing-configmap") { - return blocker({ - type: "runtime_blocker", - scope, - reason: `Required DEV ConfigMap ${ref.configMapName} was not observed in ${defaultNamespace}.`, - impact: "The Code Agent Codex stdio provider pod would fail to mount config.toml after mutation.", - safeNextAction: `Create or restore ConfigMap ${ref.configMapName}/${ref.configMapKey} in ${defaultNamespace}, then retry DEV CD.`, - retryable: true - }); - } - if (observation.status === "key-observation-blocked") { - return blocker({ - type: "runtime_blocker", - scope, - reason: `Required DEV ConfigMap ${ref.configMapName} exists but key names could not be verified.`, - impact: "DEV CD cannot prove the Code Agent Codex config mount before applying workloads.", - safeNextAction: `Restore key-name visibility for ${ref.configMapName}, then retry.`, - retryable: true - }); - } - return blocker({ - type: "runtime_blocker", - scope, - reason: `Required DEV ConfigMap key ${ref.configMapName}/${ref.configMapKey} was not observed.`, - impact: "The Code Agent Codex stdio provider pod would reference a missing config.toml key after mutation.", - safeNextAction: `Add key ${ref.configMapKey} to ConfigMap ${ref.configMapName}, then retry DEV CD.`, - retryable: true - }); -} - -async function runDevCdPreflight({ ctx, args, kubectlContext }) { - const blockers = []; - const preflight = { - status: "blocked", - scope: "pre-lock-pre-side-effect", - generatedAt: ctx.now().toISOString(), - namespace: args.targetNamespace, - kubeconfig: { - required: d601NativeKubeconfigPath, - selected: kubectlContext.kubeconfig, - source: kubectlContext.kubeconfigSource, - enforcedEnv: kubectlContext.env.KUBECONFIG === d601NativeKubeconfigPath - }, - kubectl: { - executor: kubectlContext.executor, - commandPrefix: kubectlContext.commandPrefix - }, - controlPlane: null, - secretRefs: [], - configMaps: [], - safety: { - writeSideEffectsAttempted: false, - prodTouched: false, - secretValuesRead: false, - secretValuesPrinted: false, - configMapValuesRead: false, - configMapValuesPrinted: false, - secretKeyNamesOnly: true - }, - blockers - }; - - if (kubectlContext.kubeconfig !== d601NativeKubeconfigPath) { - blockers.push(blocker({ - scope: "d601-kubeconfig-path", - reason: `DEV CD selected kubeconfig ${kubectlContext.kubeconfig || "<unset>"} from ${kubectlContext.kubeconfigSource}.`, - impact: "Any apply/job/rollout could target a non-D601 or stale control plane.", - safeNextAction: `Rerun with KUBECONFIG=${d601NativeKubeconfigPath} or --kubeconfig ${d601NativeKubeconfigPath}.`, - retryable: true - })); - preflight.status = "blocked"; - return preflight; - } - - const forced = await observeControlPlane(ctx, kubectlContext, { - env: kubectlContext.env, - label: "forced-d601-k3s", - requiredForApply: true - }); - const bare = await observeBareKubectl(ctx, kubectlContext, forced); - preflight.controlPlane = { forced, bare }; - - if (forced.status !== "observed") { - blockers.push(blocker({ - scope: "d601-native-k3s-unobservable", - reason: `Forced D601 kubeconfig could not prove context/server/nodes: ${[forced.contextCommand.summary, forced.serverCommand.summary, forced.nodesCommand.summary].filter(Boolean).join("; ") || "no kubectl output"}.`, - impact: "DEV CD refuses to acquire the Lease or create runtime Jobs without D601 native k3s proof.", - safeNextAction: `Restore readable ${d601NativeKubeconfigPath} access and verify kubectl get nodes includes d601.`, - retryable: true - })); - } - addControlPlaneBlockers(blockers, forced, { - scopePrefix: "d601-native-k3s", - requiredForApply: true - }); - addControlPlaneBlockers(blockers, bare, { - scopePrefix: "bare-kubectl", - requiredForApply: false - }); - if (bare.secondControlPlaneCandidate) { - blockers.push(blocker({ - scope: "second-hwlab-dev-control-plane", - reason: `bare kubectl can observe namespace ${defaultNamespace} outside the forced D601 kubeconfig context.`, - impact: "A second hwlab-dev control plane may still carry stale HWLAB resources, so DEV CD cannot safely classify apply evidence.", - safeNextAction: "Remove the stale hwlab-dev control plane/default kube context or make bare kubectl unable to resolve it, then rerun preflight.", - retryable: true - })); - } - - if (blockers.length === 0) { - [preflight.secretRefs, preflight.configMaps] = await Promise.all([ - Promise.all(requiredDevSecretRefs.map((ref) => observeRequiredSecretRef(ctx, kubectlContext, args, ref))), - Promise.all(requiredDevConfigMapRefs.map((ref) => observeRequiredConfigMapRef(ctx, kubectlContext, args, ref))) - ]); - for (const [index, observation] of preflight.secretRefs.entries()) { - if (observation.status !== "present") { - blockers.push(secretRefBlocker(requiredDevSecretRefs[index], observation)); - } - } - for (const [index, observation] of preflight.configMaps.entries()) { - if (observation.status !== "present") { - blockers.push(configMapRefBlocker(requiredDevConfigMapRefs[index], observation)); - } - } - } - - preflight.status = blockers.length === 0 ? "pass" : "blocked"; - return preflight; -} - -async function resolveKubectlContext(args, env, runCommand) { - const selection = resolveDevKubeconfigSelection({ flagValue: args.kubeconfig, env }); - const which = await runCommand("which", ["kubectl"], { timeoutMs: 5000 }); - const executor = which.code === 0 ? which.stdout.trim() : "kubectl"; - const kubectlEnv = { - ...env, - KUBECONFIG: selection.kubeconfig - }; - return { - executor, - kubeconfig: selection.kubeconfig, - kubeconfigSource: selection.source, - commandPrefix: buildKubectlCommandPrefix(selection.kubeconfig), - env: kubectlEnv - }; -} - -function commandEnv(ctx, transaction) { - return { - ...ctx.env, - HWLAB_CD_TRANSACTION_ID: transaction.transactionId, - HWLAB_CD_TRANSACTION_OWNER: transaction.ownerTaskId, - HWLAB_CD_LOCK_NAME: transaction.lockName, - HWLAB_CD_LOCK_NAMESPACE: transaction.targetNamespace - }; -} - -async function runStep(ctx, transaction, step) { - const startedAt = ctx.now().toISOString(); - const command = shellCommand(step.command, step.args); - const result = await ctx.runCommand(step.command, step.args, { - cwd: ctx.repoRoot, - env: commandEnv(ctx, transaction), - timeoutMs: step.timeoutMs ?? 300000 - }); - const finishedAt = ctx.now().toISOString(); - const stdout = redactSensitiveText(result.stdout ?? ""); - const stderr = redactSensitiveText(result.stderr ?? ""); - const parsedStdout = parseJsonMaybe(stdout); - return { - id: step.id, - phase: step.phase, - status: result.code === 0 ? "pass" : "blocked", - command, - code: result.code, - startedAt, - finishedAt, - stdoutJson: parsedStdout, - stdoutTail: parsedStdout ? null : stdout.trim().slice(-2000), - stderrTail: stderr.trim().slice(-2000), - reportPath: step.reportPath ?? null - }; -} - -async function runRuntimeK8sJobStep(ctx, transaction, step) { - const startedAt = ctx.now().toISOString(); - const result = await runRuntimeK8sJob(ctx, transaction, step); - const finishedAt = ctx.now().toISOString(); - return { - id: step.id, - phase: step.phase, - status: result.status, - command: result.command, - code: result.status === "pass" ? 0 : 1, - startedAt, - finishedAt, - stdoutJson: result.report, - stdoutTail: null, - stderrTail: result.error ?? "", - reportPath: step.reportPath ?? null, - k8sJob: result.k8sJob - }; -} - -async function runRuntimeK8sJob(ctx, transaction, step) { - const jobName = `${step.jobNamePrefix}-${transaction.transactionId.slice(0, 8)}`; - const reportFile = `${runtimeJobReportPrefix}-${step.id}.json`; - const image = step.image ?? await resolveRuntimeJobImage(ctx); - const manifest = buildRuntimeK8sJobManifest({ - jobName, - namespace: transaction.targetNamespace, - image, - commandArgs: [...step.commandArgs, "--report", reportFile], - transaction - }); - const kubectl = transaction.kubectlContext; - const apply = await kubectlCommandResult(ctx, kubectl, ["-n", transaction.targetNamespace, "apply", "-f", "-"], { - input: JSON.stringify(manifest), - timeoutMs: 30000 - }); - if (apply.code !== 0) { - return runtimeJobFailure(step, "apply", apply, jobName, image); - } - - const wait = await kubectlCommandResult(ctx, kubectl, [ - "-n", - transaction.targetNamespace, - "wait", - "--for=condition=complete", - `job/${jobName}`, - "--timeout=300s" - ], { timeoutMs: step.timeoutMs ?? 6 * 60 * 1000 }); - if (wait.code !== 0) { - return runtimeJobFailure(step, "wait", wait, jobName, image); - } - - const logs = await kubectlCommandResult(ctx, kubectl, ["-n", transaction.targetNamespace, "logs", `job/${jobName}`], { - timeoutMs: 30000 - }); - const logText = redactSensitiveText(`${logs.stdout}\n${logs.stderr}`); - const report = parseLastJsonObject(logText); - if (logs.code !== 0 || !report) { - return { - status: "blocked", - command: `${shellCommand("kubectl", ["-n", transaction.targetNamespace, "logs", `job/${jobName}`])}`, - error: logText.trim().slice(-2000) || "runtime job did not emit a JSON report", - report: null, - k8sJob: runtimeJobSummary(jobName, step, "logs", image) - }; - } - const reportStatus = report.conclusion?.status ?? report.summary?.status ?? report.status; - return { - status: ["ready", "pass"].includes(reportStatus) ? "pass" : "blocked", - command: shellCommand("kubectl", ["-n", transaction.targetNamespace, "apply/wait/logs", `job/${jobName}`]), - error: null, - report, - k8sJob: runtimeJobSummary(jobName, step, "complete", image) - }; -} - -function runtimeJobFailure(step, phase, result, jobName, image) { - return { - status: "blocked", - command: result.command ?? shellCommand("kubectl", [phase, `job/${jobName}`]), - error: redactSensitiveText(`${result.stderr}\n${result.stdout}`).trim().slice(-2000), - report: null, - k8sJob: runtimeJobSummary(jobName, step, phase, image) - }; -} - -function runtimeJobSummary(jobName, step, phase, image = step.image ?? "deploy-current:hwlab-cloud-api") { - return { - jobName, - phase, - serviceId: "hwlab-cloud-api", - image, - secretRefsOnly: true, - secretValuesRead: false, - secretValuesPrinted: false - }; -} - -async function resolveRuntimeJobImage(ctx) { - const deploy = await readDeployJson(ctx.repoRoot); - const image = deploy.manifest?.services?.find((service) => service.serviceId === "hwlab-cloud-api")?.image; - if (typeof image === "string" && image.trim().length > 0) { - return image; - } - if (typeof deploy.commitId === "string" && deploy.commitId !== "unknown") { - return `127.0.0.1:5000/hwlab/hwlab-cloud-api:${shortCommit(deploy.commitId)}`; - } - throw new DevCdApplyError("could not resolve hwlab-cloud-api image for runtime maintenance Job", { - code: "runtime-job-image-unresolved" - }); -} - -function buildRuntimeK8sJobManifest({ jobName, namespace, image, commandArgs, transaction }) { - return { - apiVersion: "batch/v1", - kind: "Job", - metadata: { - name: jobName, - namespace, - labels: { - "app.kubernetes.io/part-of": "hwlab", - "app.kubernetes.io/name": "hwlab-runtime-maintenance", - "hwlab.pikastech.local/profile": "dev", - "hwlab.pikastech.local/service-id": "hwlab-cloud-api", - "hwlab.pikastech.local/cd-transaction": transaction.transactionId - } - }, - spec: { - backoffLimit: 0, - ttlSecondsAfterFinished: 600, - template: { - metadata: { - labels: { - "app.kubernetes.io/name": "hwlab-runtime-maintenance", - "hwlab.pikastech.local/service-id": "hwlab-cloud-api", - "hwlab.pikastech.local/cd-transaction": transaction.transactionId - } - }, - spec: { - restartPolicy: "Never", - containers: [ - { - name: "runtime-maintenance", - image, - command: ["node"], - args: commandArgs, - env: [ - { name: "HWLAB_ENVIRONMENT", value: ENVIRONMENT_DEV }, - { name: "HWLAB_CLOUD_RUNTIME_ADAPTER", value: "postgres" }, - { name: "HWLAB_CLOUD_RUNTIME_DURABLE", value: "true" }, - { - name: "HWLAB_CLOUD_DB_URL", - valueFrom: { - secretKeyRef: { - name: "hwlab-cloud-api-dev-db", - key: "database-url", - optional: false - } - } - }, - { - name: "HWLAB_CLOUD_DB_ADMIN_URL", - valueFrom: { - secretKeyRef: { - name: "hwlab-cloud-api-dev-db-admin", - key: "admin-url", - optional: false - } - } - }, - { name: "HWLAB_CLOUD_DB_SSL_MODE", value: "disable" }, - { name: "HWLAB_CD_TRANSACTION_ID", value: transaction.transactionId }, - { name: "HWLAB_CD_TRANSACTION_OWNER", value: transaction.ownerTaskId }, - { name: "HWLAB_CD_LOCK_NAME", value: transaction.lockName } - ] - } - ] - } - } - } - }; -} - -async function kubectlCommandResult(ctx, kubectlContext, args, options = {}) { - const result = await ctx.runCommand(kubectlContext.executor, args, { - cwd: ctx.repoRoot, - env: kubectlContext.env, - input: options.input, - timeoutMs: options.timeoutMs ?? 30000 - }); - return { - ...result, - command: shellCommand("kubectl", args), - stdout: result.stdout ?? "", - stderr: result.stderr ?? "" - }; -} - -function parseLastJsonObject(text) { - const normalized = String(text ?? "").trim(); - const lines = normalized.split(/\r?\n/u).reverse(); - for (const line of lines) { - const parsed = parseJsonMaybe(line.trim()); - if (parsed && typeof parsed === "object") return parsed; - } - const starts = []; - for (let index = 0; index < normalized.length; index += 1) { - if (normalized[index] === "{") starts.push(index); - } - for (const start of starts.reverse()) { - const parsed = parseJsonMaybe(normalized.slice(start)); - if (parsed && typeof parsed === "object") return parsed; - } - return parseJsonMaybe(normalized); -} - -function stepBlocker(stepResult) { - const stdoutBlockers = Array.isArray(stepResult.stdoutJson?.blockers) ? stepResult.stdoutJson.blockers : []; - const primary = stdoutBlockers.find((item) => item?.status === "open") ?? stdoutBlockers[0] ?? null; - const reason = primary?.reason ?? primary?.summary ?? `${stepResult.command} exited ${stepResult.code}.`; - const impact = primary?.impact ?? stepBlockerImpact(stepResult); - const safeNextAction = primary?.safeNextAction ?? stepBlockerSafeNextAction(stepResult); - return structuredBlocker({ - type: "runtime_blocker", - scope: stepResult.id, - reason, - impact, - safeNextAction, - retryable: primary?.retryable ?? true, - evidence: { - command: stepResult.command, - code: stepResult.code, - reportPath: stepResult.reportPath ?? null, - blockerScope: primary?.scope ?? null, - k8sJob: stepResult.k8sJob ?? null - } - }); -} - -function stepBlockerImpact(stepResult) { - if (stepResult.id === "runtime-db-provisioning") return "DEV CD cannot prove DB role/database provisioning before workload apply."; - if (stepResult.id === "runtime-db-migration") return "DEV CD cannot prove runtime schema migration before workload apply."; - if (stepResult.id === "runtime-durable-postflight") return "DEV CD cannot prove /health/live, /v1, and M3 durable evidence persistence after apply."; - if (stepResult.id === "dev-deploy-apply") return "DEV desired-state apply did not complete, so live readiness cannot be trusted."; - return "DEV CD transaction cannot safely continue."; -} - -function stepBlockerSafeNextAction(stepResult) { - if (stepResult.id === "runtime-db-provisioning") return "Repair repo-owned runtime provisioning inputs/SecretRefs without printing Secret values, then rerun DEV CD."; - if (stepResult.id === "runtime-db-migration") return "Repair repo-owned runtime migration/schema state, then rerun DEV CD."; - if (stepResult.id === "runtime-durable-postflight") return "Inspect the runtime postflight report, repair durable readiness or M3 evidence persistence, then rerun DEV CD after host-controlled rollout if needed."; - if (stepResult.id === "dev-deploy-apply") return `Inspect ${deployApplyReportPath} and rerun the repo-owned DEV CD path after the apply blocker is fixed.`; - return "Fix the reported step blocker and rerun the repo-owned DEV CD transaction."; -} - -function structuredBlocker({ type = "runtime_blocker", scope, reason, impact, safeNextAction, retryable = true, evidence = null }) { - return { - type, - scope, - status: "open", - reason: oneLine(reason), - impact: oneLine(impact), - safeNextAction: oneLine(safeNextAction), - retryable: Boolean(retryable), - summary: oneLine(`${reason} Impact: ${impact} Safe next action: ${safeNextAction}`), - ...(evidence ? { evidence } : {}) - }; -} - -function healthCommit(json) { - return ( - json?.commit?.id || - json?.commitId || - json?.revision || - json?.image?.tag || - null - ); -} - -function healthImage(json) { - if (typeof json?.image === "string") return json.image; - return json?.image?.reference ?? null; -} - -function summarizeHealthJson(json, expectedServiceId, expectedCommit) { - const observedCommit = healthCommit(json); - return { - serviceId: json?.serviceId ?? json?.service?.id ?? null, - environment: json?.environment ?? null, - applicationStatus: json?.status ?? null, - ready: Object.hasOwn(json ?? {}, "ready") ? json.ready : null, - observedCommit, - image: healthImage(json), - imageTag: json?.image?.tag ?? null, - observedAt: json?.observedAt ?? null, - serviceMatches: (json?.serviceId ?? json?.service?.id) === expectedServiceId, - environmentMatches: json?.environment === ENVIRONMENT_DEV, - commitMatches: expectedCommit ? commitMatches(observedCommit, expectedCommit) : null, - blockerCodes: Array.isArray(json?.blockerCodes) ? json.blockerCodes : [] - }; -} - -async function defaultHttpGetJson(url, timeoutMs = 10000) { - return new Promise((resolve, reject) => { - const request = httpRequest(url, { method: "GET", timeout: timeoutMs }, (response) => { - response.setEncoding("utf8"); - let body = ""; - response.on("data", (chunk) => { - body += chunk; - }); - response.on("end", () => { - let json = null; - try { - json = JSON.parse(body); - } catch {} - resolve({ - statusCode: response.statusCode ?? 0, - body, - json - }); - }); - }); - request.on("timeout", () => { - request.destroy(new Error(`timeout after ${timeoutMs}ms`)); - }); - request.on("error", reject); - request.end(); - }); -} - -async function probeLiveEndpoint(ctx, { id, url, expectedServiceId, expectedCommit }) { - const startedAt = ctx.now().toISOString(); - try { - const response = await ctx.httpGetJson(url, 10000); - const summary = summarizeHealthJson(response.json, expectedServiceId, expectedCommit); - const reachable = response.statusCode >= 200 && response.statusCode < 300 && Boolean(response.json); - const identityMatches = reachable && summary.serviceMatches && summary.environmentMatches && (expectedCommit ? summary.commitMatches : true); - return { - id, - url, - status: identityMatches ? "pass" : "blocked", - httpStatus: response.statusCode, - reachable, - identityMatches, - startedAt, - finishedAt: ctx.now().toISOString(), - expectedServiceId, - expectedCommit: expectedCommit ?? null, - ...summary - }; - } catch (error) { - return { - id, - url, - status: "blocked", - reachable: false, - identityMatches: false, - startedAt, - finishedAt: ctx.now().toISOString(), - expectedServiceId, - expectedCommit: expectedCommit ?? null, - error: oneLine(error.message) - }; - } -} - -export async function verifyDevLive(ctx, { expectedCommit = null } = {}) { - const [cloudWeb, cloudApi] = await Promise.all([ - probeLiveEndpoint(ctx, { - id: "cloud-web-16666", - url: browserLiveUrl, - expectedServiceId: "hwlab-cloud-web", - expectedCommit - }), - probeLiveEndpoint(ctx, { - id: "cloud-api-16667", - url: apiLiveUrl, - expectedServiceId: "hwlab-cloud-api", - expectedCommit - }) - ]); - const endpoints = [cloudWeb, cloudApi]; - return { - status: endpoints.every((endpoint) => endpoint.status === "pass") ? "pass" : "blocked", - expectedCommit: expectedCommit ?? null, - endpoints, - summary: { - checked: endpoints.length, - reachable: endpoints.filter((endpoint) => endpoint.reachable).length, - identityMatches: endpoints.filter((endpoint) => endpoint.identityMatches).length, - commitMatches: expectedCommit - ? endpoints.filter((endpoint) => endpoint.commitMatches === true).length - : null, - ports: [16666, 16667] - } - }; -} - -function deployServiceCount(manifest) { - const services = manifest?.services; - if (Array.isArray(services)) return services.length; - if (services && typeof services === "object") return Object.keys(services).length; - return null; -} - -function summarizeDeployJson(deploy) { - return { - path: deploy.path, - hash: deploy.hash, - commitId: deploy.commitId, - namespace: deploy.namespace, - environment: deploy.environment, - endpoint: deploy.endpoint, - serviceCount: deployServiceCount(deploy.manifest) - }; -} - -function deployJsonMatchesTarget(deploy, target) { - return deploy.commitId === target.shortCommitId || deploy.commitId === target.commitId; -} - -function compactLiveEndpoint(endpoint) { - return { - id: endpoint.id, - url: endpoint.url, - status: endpoint.status, - httpStatus: endpoint.httpStatus ?? null, - reachable: endpoint.reachable, - identityMatches: endpoint.identityMatches, - serviceId: endpoint.serviceId, - environment: endpoint.environment, - observedCommit: endpoint.observedCommit, - commitMatches: endpoint.commitMatches, - blockerCodes: endpoint.blockerCodes ?? [], - error: endpoint.error ?? null - }; -} - -function compactLiveReport(live) { - return { - status: live.status, - expectedCommit: live.expectedCommit ?? null, - summary: live.summary ?? { checked: 0 }, - endpoints: (live.endpoints ?? []).map(compactLiveEndpoint) - }; -} - -function buildLiveDelta(live, target) { - const expectedCommit = target?.shortCommitId ?? target?.commitId ?? null; - const endpoints = (live?.endpoints ?? []).map((endpoint) => ({ - id: endpoint.id, - observedCommit: endpoint.observedCommit ?? null, - expectedCommit, - matchesPromotion: expectedCommit ? commitMatches(endpoint.observedCommit, expectedCommit) : null, - status: endpoint.status - })); - const mismatched = endpoints.filter((endpoint) => endpoint.matchesPromotion === false); - return { - status: expectedCommit && mismatched.length === 0 && endpoints.length > 0 ? "pass" : expectedCommit ? "mismatch" : "unknown", - expectedCommit, - mismatched: mismatched.length, - endpoints - }; -} - -async function readDeployLockStatus({ ctx, args, kubectlContext }) { - const get = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "get", "lease", args.lockName, "-o", "json"], { - timeoutMs: 15000 - }); - if (get.code === 0) { - return { - status: "read", - command: get.command, - lock: parseDeployLock(JSON.parse(get.stdout)) - }; - } - if (isNotFound(get)) { - return { - status: "absent", - command: get.command, - lockName: args.lockName, - lock: null - }; - } - return { - status: "unavailable", - command: get.command, - lockName: args.lockName, - lock: null, - error: redactSensitiveText(get.stderr || get.stdout) - }; -} - -function summarizeLockStatus(lockRead, now) { - if (lockRead.status === "absent") { - return { - status: "absent", - lockName: lockRead.lockName ?? defaultLockName, - current: null, - retryAfterSeconds: 0, - expiresAt: null - }; - } - if (lockRead.status === "unavailable") { - return { - status: "unavailable", - lockName: lockRead.lockName ?? defaultLockName, - current: null, - retryAfterSeconds: null, - expiresAt: null, - reason: "kubectl or DEV kubeconfig unavailable; command output is redacted", - redactedReason: oneLine(lockRead.error).slice(0, 500) - }; - } - const classification = classifyDeployLock(lockRead.lock, now); - const status = lockRead.lock?.phase === "released" - ? "released" - : classification.held - ? "held" - : classification.stale - ? "stale" - : "unknown"; - return { - status, - lockName: lockRead.lock?.lockName ?? lockRead.lockName ?? defaultLockName, - current: compactLockSummary(lockRead.lock), - retryAfterSeconds: classification.retryAfterSeconds, - expiresAt: classification.expiresAt, - allowedLockAction: { - breakStale: classification.stale ? "--apply --confirm-dev --confirmed-non-production --break-stale-lock" : null - } - }; -} - -function buildStatusNextActions({ target, deployBefore, lock, liveVerify }) { - const actions = []; - if (target.desiredStateCheck?.status === "blocked") { - actions.push("deploy/deploy.json does not match the target ref and the deploy-json promotion desired-state check failed; refresh artifacts/catalog or choose an explicit published promotion."); - } - if (!target.headMatchesTarget) { - const controlShortCommit = target.controlRef?.shortCommitId ?? target.shortCommitId; - actions.push(`Checkout or fast-forward to ${target.ref} ${controlShortCommit} before apply.`); - } - if (deployBefore.commitId !== target.shortCommitId && deployBefore.commitId !== target.commitId) { - actions.push(`deploy/deploy.json commitId is ${deployBefore.commitId}; target ref is ${target.shortCommitId}.`); - } - if (lock.status === "held") { - actions.push(`Wait for the current DEV CD holder to release the Lease or for the ${lock.retryAfterSeconds}s TTL to expire before retrying apply.`); - } else if (lock.status === "stale") { - actions.push("Use --break-stale-lock only after confirming the holder is inactive."); - } else if (lock.status === "unavailable") { - actions.push("Fix kubectl/KUBECONFIG access before applying."); - } - if (liveVerify.status !== "pass") { - actions.push("Inspect live endpoint identity/health before treating DEV as converged."); - } - const liveDelta = buildLiveDelta(liveVerify, target); - if (liveDelta.status === "mismatch") { - actions.push(`Live DEV is still serving ${liveDelta.mismatched} endpoint(s) that do not match promotion ${target.shortCommitId}; host commander should run the controlled DEV rollout after the apply report is ready.`); - } - if (actions.length === 0) { - actions.push(target.publishRequired === false - ? `Run --apply --confirm-dev --confirmed-non-production --report ${defaultReportPath} when ready to apply the published deploy.json promotion without republishing HEAD.` - : `Run --apply --confirm-dev --confirmed-non-production --report ${defaultReportPath} when ready to publish/apply DEV.`); - } - return actions; -} - -async function runDevCdStatus(args, ctx, stdout) { - const generatedAt = ctx.now().toISOString(); - const deployBefore = await readDeployJson(ctx.repoRoot); - const artifactEvidence = await readArtifactEvidence(ctx.repoRoot); - const target = await resolveEffectiveTarget(ctx, args, deployBefore, artifactEvidence); - const kubectlContext = await resolveKubectlContext(args, ctx.env, ctx.runCommand); - const lockRead = await readDeployLockStatus({ ctx, args, kubectlContext }); - const lock = summarizeLockStatus(lockRead, ctx.now()); - const liveVerify = args.skipLiveVerify - ? { status: "not_run", reason: "--skip-live-verify", endpoints: [], summary: { checked: 0 } } - : await verifyDevLive(ctx, { expectedCommit: null }); - const deployMatchesTarget = deployJsonMatchesTarget(deployBefore, target); - const liveDelta = buildLiveDelta(liveVerify, target); - const status = ( - lock.status === "unavailable" || - liveVerify.status !== "pass" || - liveDelta.status === "mismatch" || - !target.headMatchesTarget || - !deployMatchesTarget || - target.publishRequired === false && - target.artifactBoundary?.status === "degraded" - ) ? "degraded" : "pass"; - const report = { - ok: true, - status, - mode: args.dryRun ? "dry-run" : "status", - command: "dev-cd-apply", - generatedAt, - mutationAttempted: false, - prodTouched: false, - target: { - ref: target.ref, - promotionCommit: target.commitId, - shortCommitId: target.shortCommitId, - promotionSource: target.promotionSource, - publishRequired: target.publishRequired, - controlRef: target.controlRef, - headCommitId: target.headCommitId, - headMatchesTarget: target.headMatchesTarget, - desiredStateCheck: target.desiredStateCheck, - artifactBoundary: target.artifactBoundary, - namespace: args.targetNamespace - }, - deployJson: { - ...summarizeDeployJson(deployBefore), - matchesTarget: deployMatchesTarget - }, - artifactCatalog: artifactEvidence.catalog, - artifactReport: artifactEvidence.report, - kubectl: { - kubeconfigSource: kubectlContext.kubeconfigSource, - kubeconfig: kubectlContext.kubeconfig ? "<set>" : "<unset>", - lockReadCommand: lockRead.command - }, - lock, - live: compactLiveReport(liveVerify), - liveDelta, - nextActions: buildStatusNextActions({ target, deployBefore, lock, liveVerify }), - fullOutputHint: `Use --apply --confirm-dev --confirmed-non-production --report ${defaultReportPath} for the full transaction JSON; use --full-output only when stdout needs the full JSON.` - }; - stdout.write(`${JSON.stringify(report, null, 2)}\n`); - return 0; -} - -function summarizeDevCdApplyReport(report) { - const apply = report.devCdApply ?? {}; - return { - ok: report.status === "pass", - status: report.status, - mode: apply.mode ?? "apply", - command: "dev-cd-apply", - commitId: report.commitId, - generatedAt: report.generatedAt, - mutationAttempted: apply.safety?.mutationAttempted ?? false, - prodTouched: false, - transaction: { - transactionId: report.transaction?.transactionId ?? null, - ownerTaskId: report.transaction?.ownerTaskId ?? null, - targetNamespace: report.transaction?.targetNamespace ?? defaultNamespace, - lockName: report.transaction?.lockName ?? defaultLockName, - phases: report.transaction?.phases ?? [], - release: report.transaction?.release ?? null - }, - target: apply.target ?? null, - lock: { - acquired: Boolean(apply.lock?.acquired), - current: compactLockSummary(apply.lock?.current), - staleBreak: compactLockSummary(apply.lock?.staleBreak) - }, - steps: (apply.steps ?? []).map((step) => ({ - id: step.id, - phase: step.phase, - status: step.status, - code: step.code, - reportPath: step.reportPath - })), - liveBefore: apply.liveBefore ? compactLiveReport(apply.liveBefore) : null, - liveVerify: apply.liveVerify ? compactLiveReport(apply.liveVerify) : null, - preflight: apply.preflight ? { - status: apply.preflight.status, - scope: apply.preflight.scope, - kubeconfig: apply.preflight.kubeconfig, - blockerCount: apply.preflight.blockers?.length ?? 0, - secretRefs: (apply.preflight.secretRefs ?? []).map((ref) => ({ - secretName: ref.secretName, - secretKey: ref.secretKey, - status: ref.status, - exists: ref.exists, - keyPresent: ref.keyPresent - })) - } : null, - blockers: report.blockers ?? [], - reportPaths: apply.reportPaths ?? {}, - fullOutputHint: `Use --full-output to print the full report to stdout. Use --report ${defaultReportPath} to persist the full report outside the repo.` - }; -} - -function buildReport({ - args, - transaction, - target, - deployBefore, - deployAfter, - lockState, - steps, - blockers, - status, - startedAt, - finishedAt, - preflight, - liveBefore, - liveVerify, - release -}) { - return { - $schema: "https://hwlab.pikastech.local/schemas/dev-cd-apply.schema.json", - $id: "https://hwlab.pikastech.local/dev-cd/dev-cd-apply.json", - reportVersion: "v1", - issue: "pikasTech/HWLAB#274", - taskId: "dev-cd-apply", - commitId: target.shortCommitId, - acceptanceLevel: "dev_cd_apply", - devOnly: true, - prodDisabled: true, - status, - generatedAt: finishedAt, - reportLifecycle: activeReportLifecycle("Current DEV CD transaction report; the only DEV side-effect entrypoint is scripts/dev-cd-apply.mjs."), - supports: [ - "pikasTech/HWLAB#274", - "pikasTech/HWLAB#116", - "pikasTech/HWLAB#235", - "pikasTech/HWLAB#50", - "pikasTech/HWLAB#57", - "pikasTech/HWLAB#67", - "pikasTech/HWLAB#73" - ], - sourceContract: { - status: blockers.length ? "blocked" : "pass", - documents: [ - "docs/reference/deployment-publish.md", - "docs/reference/dev-runtime-boundary.md", - "docs/dev-artifact-publish.md", - "docs/dev-deploy-apply.md", - "docs/artifact-catalog.md" - ], - summary: "DEV CD apply/verify side effects are serialized by one Kubernetes Lease and use deploy/deploy.json as the publish-control desired-state source; CI artifact reports are audit snapshots." - }, - validationCommands: [ - "node --check scripts/dev-cd-apply.mjs", - "node --check scripts/src/dev-cd-apply.mjs", - "node --test scripts/src/dev-cd-apply.test.mjs", - "node --check scripts/dev-artifact-publish.mjs", - "node --check scripts/src/dev-deploy-apply.mjs", - "node --test scripts/src/dev-deploy-apply.test.mjs", - "node scripts/deploy-desired-state-plan.mjs --check", - "node scripts/validate-artifact-catalog.mjs", - "git diff --check" - ], - localSmoke: { - status: "not_run", - commands: ["npm run check"], - evidence: ["Local checks are listed as validation commands and are not implied by a DEV CD transaction."], - summary: "The transaction report records DEV publish/apply/verify evidence; local smoke is a separate gate." - }, - dryRun: { - status: "not_run", - commands: [`node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report ${deployApplyReportPath}`], - evidence: ["The transaction path owns live side effects; dry-run remains a preflight support mode."], - summary: "DEV CD mutation requires --apply and the transaction Lease lock." - }, - devPreconditions: { - status: blockers.length ? "blocked" : "pass", - requirements: [ - "DEV CD apply must force KUBECONFIG=/etc/rancher/k3s/k3s.yaml and verify native node d601 before any Lease, Job, apply, or rollout side effect.", - "DEV CD apply must reject docker-desktop, desktop-control-plane, 127.0.0.1:11700, bare kubectl ambiguity, and second hwlab-dev control-plane signals before mutation.", - "DEV CD apply must verify required SecretRef existence and key names without reading or printing Secret values.", - "HEAD must match the requested target ref before publish.", - "Lease/hwlab-dev-cd-lock must be acquired before DEV apply/verify side effects.", - "Artifact publish must run under CI artifact identity, with HWLAB_CD_TRANSACTION_ID accepted only for the legacy transition path.", - "DEV DB role/database provisioning and runtime migration must run through repo-owned commands before workload apply.", - "deploy/deploy.json, artifact catalog, and workloads must converge before apply.", - "When deploy/deploy.json already points to a published promotion commit, HEAD may be a later control commit and the transaction must consume that promotion without republishing HEAD.", - "Public 16666 and 16667 live health plus M3 durable postflight must be recorded in this report." - ], - summary: blockers.length ? "One or more DEV CD transaction preconditions failed." : "DEV CD transaction preconditions passed." - }, - transaction: { - transactionId: transaction.transactionId, - ownerTaskId: transaction.ownerTaskId, - targetNamespace: transaction.targetNamespace, - lockName: transaction.lockName, - lockBackend: "Lease", - startedAt, - finishedAt, - ttlSeconds: args.ttlSeconds, - phases: transaction.phases, - release - }, - devCdApply: { - mode: args.apply ? "apply" : "dry-run", - target: { - ref: target.ref, - promotionCommit: target.commitId, - shortCommitId: target.shortCommitId, - promotionSource: target.promotionSource, - publishRequired: target.publishRequired, - controlRef: target.controlRef, - headCommitId: target.headCommitId, - headMatchesTarget: target.headMatchesTarget, - desiredStateCheck: target.desiredStateCheck, - artifactBoundary: target.artifactBoundary, - namespace: args.targetNamespace, - apiEndpoint: DEV_ENDPOINT, - browserEndpoint: "http://74.48.78.17:16666/" - }, - deployJson: { - before: deployBefore, - after: deployAfter - }, - lock: { - acquired: Boolean(lockState?.acquired), - current: lockState?.lock ?? null, - staleBreak: lockState?.staleBreak ?? null - }, - steps, - liveBefore, - liveVerify, - preflight, - reportPaths: { - transaction: args.writeReport ? args.reportPath : null, - artifacts: artifactReportPath, - deployApply: deployApplyReportPath, - runtimeProvisioning: runtimeProvisioningReportPath, - runtimeMigration: runtimeMigrationReportPath, - runtimePostflight: runtimePostflightReportPath - }, - safety: { - prodTouched: false, - secretValuesRead: false, - secretValuesPrinted: false, - manualDbWrite: false, - oldPublicPortsUsed: false, - mutationAttempted: steps.some((step) => ["artifact-publish", "dev-deploy-apply"].includes(step.id)), - singleLock: "Lease/hwlab-dev/hwlab-dev-cd-lock" - } - }, - blockers, - notes: "DEV-only CD transaction evidence. This does not claim M3 PASS or runtime durability." - }; -} - -function buildArgumentFailure(error) { - const blockers = error.blockers ?? [{ - type: "safety_blocker", - scope: error.code ?? "dev-cd-apply", - status: "open", - summary: oneLine(error.message) - }]; - return { - ok: false, - status: "blocked", - error: error.code ?? "dev-cd-apply-failed", - blockers, - mutationAttempted: false, - prodTouched: false - }; -} - -export function formatDevCdApplyFailure(error) { - if (error?.lockFailure) return error.lockFailure; - if (error instanceof DevCdApplyError) return buildArgumentFailure(error); - return { - ok: false, - status: "failed", - error: "dev-cd-apply-failed", - summary: error instanceof Error ? error.message : String(error), - mutationAttempted: false, - prodTouched: false - }; -} - -export async function runDevCdApply(argv, io = {}) { - const stdout = io.stdout ?? process.stdout; - const env = io.env ?? process.env; - const ctx = { - repoRoot: io.repoRoot ?? defaultRepoRoot, - env, - runCommand: io.runCommand ?? defaultRunCommand, - httpGetJson: io.httpGetJson ?? defaultHttpGetJson, - now: io.now ?? (() => new Date()) - }; - - let args; - try { - args = parseArgs(argv); - if (args.help) { - stdout.write(`${JSON.stringify({ ok: true, status: "pass", help: usage() }, null, 2)}\n`); - return 0; - } - if (!args.apply && !args.status) { - args.status = true; - } - validateArgs(args); - if (args.status) { - return await runDevCdStatus(args, ctx, stdout); - } - } catch (error) { - const failure = formatDevCdApplyFailure(error); - stdout.write(`${JSON.stringify(failure, null, 2)}\n`); - return 2; - } - - const startedAt = ctx.now().toISOString(); - const transaction = { - transactionId: randomUUID(), - ownerTaskId: args.ownerTaskId ?? defaultOwnerTaskId(env), - targetNamespace: args.targetNamespace, - lockName: args.lockName, - phases: [], - kubectlContext: null - }; - const steps = []; - const blockers = []; - let lockState = null; - let release = null; - let target = null; - let deployBefore = null; - let deployAfter = null; - let liveBefore = null; - let liveVerify = null; - let preflight = null; - let status = "blocked"; - - try { - deployBefore = await readDeployJson(ctx.repoRoot); - const artifactEvidence = await readArtifactEvidence(ctx.repoRoot); - target = await resolveEffectiveTarget(ctx, args, deployBefore, artifactEvidence); - if (target.desiredStateCheck?.status === "blocked") { - blockers.push({ - type: "contract_blocker", - scope: "desired-state-promotion", - status: "open", - summary: "deploy/deploy.json is not aligned with the target ref, and the deploy-json promotion check did not pass." - }); - throw new DevCdApplyError("deploy desired-state does not match the target ref or a published deploy.json promotion", { - code: "desired-state-promotion-check-failed", - suppressCatchBlocker: true - }); - } - if (artifactBoundaryRequiresApplyBlock(target)) { - blockers.push({ - type: "contract_blocker", - scope: "artifact-boundary", - status: "open", - summary: "deploy/deploy.json, deploy/artifact-catalog.dev.json, deploy/k8s/base/workloads.yaml, and registry manifests must prove the same promotion commit before apply." - }); - throw new DevCdApplyError("artifact desired-state provenance does not match the deploy.json promotion", { - code: "artifact-boundary-provenance-mismatch", - suppressCatchBlocker: true - }); - } - if (!target.headMatchesTarget) { - const controlShortCommit = target.controlRef?.shortCommitId ?? target.shortCommitId; - blockers.push({ - type: "contract_blocker", - scope: "target-ref-head", - status: "open", - summary: `HEAD ${target.headShortCommitId} must match target ref ${target.ref} ${controlShortCommit} before DEV CD apply.` - }); - throw new DevCdApplyError("checkout is behind the requested DEV CD control ref", { - code: "target-ref-head-mismatch", - suppressCatchBlocker: true - }); - } - const kubectlContext = await resolveKubectlContext(args, env, ctx.runCommand); - transaction.kubectlContext = kubectlContext; - preflight = await runDevCdPreflight({ ctx, args, kubectlContext }); - if (preflight.status !== "pass") { - blockers.push(...preflight.blockers); - throw new DevCdApplyError("DEV CD preflight blocked before any write side effect", { - code: "dev-cd-preflight-blocked", - suppressCatchBlocker: true - }); - } - liveBefore = { - status: "pending", - reason: "liveBefore is captured immediately after Lease acquisition so publish/apply cannot race before the transaction lock." - }; - lockState = await acquireDeployLock({ - ctx, - args, - kubectlContext, - target, - deployBefore, - transactionId: transaction.transactionId, - ownerTaskId: transaction.ownerTaskId, - liveBefore, - now: ctx.now() - }); - liveBefore = await verifyDevLive(ctx, { expectedCommit: null }); - await updateDeployLockLiveBefore({ - ctx, - args, - kubectlContext, - transactionId: transaction.transactionId, - lockState, - liveBefore - }); - transaction.phases.push({ phase: "publishing", status: "entered", at: ctx.now().toISOString() }); - - const publishSteps = target.publishRequired === false - ? [ - { - id: "desired-state-check-existing-promotion", - phase: "publishing", - command: process.execPath, - args: [ - "scripts/deploy-desired-state-plan.mjs", - "--promotion-commit", - target.commitId, - "--check", - "--pretty" - ], - timeoutMs: 120000 - }, - { - id: "artifact-catalog-validate", - phase: "publishing", - command: process.execPath, - args: ["scripts/validate-artifact-catalog.mjs"], - timeoutMs: 120000 - } - ] - : [ - { - id: "desired-state-check-before-publish", - phase: "publishing", - command: process.execPath, - args: ["scripts/deploy-desired-state-plan.mjs", "--check", "--pretty"], - timeoutMs: 120000 - }, - { - id: "artifact-publish", - phase: "publishing", - command: process.execPath, - args: ["scripts/dev-artifact-publish.mjs", "--publish", "--report", artifactReportPath], - timeoutMs: 60 * 60 * 1000, - reportPath: artifactReportPath - }, - { - id: "artifact-catalog-refresh", - phase: "publishing", - command: process.execPath, - args: [ - "scripts/refresh-artifact-catalog.mjs", - "--target-ref", - target.commitId, - "--publish-report", - artifactReportPath - ], - timeoutMs: 120000 - }, - { - id: "artifact-catalog-validate", - phase: "publishing", - command: process.execPath, - args: ["scripts/validate-artifact-catalog.mjs"], - timeoutMs: 120000 - }, - { - id: "desired-state-check-after-refresh", - phase: "publishing", - command: process.execPath, - args: [ - "scripts/deploy-desired-state-plan.mjs", - "--promotion-commit", - target.commitId, - "--check", - "--pretty" - ], - timeoutMs: 120000 - }, - ]; - - const plannedSteps = [ - ...publishSteps, - { - id: "runtime-db-provisioning", - phase: "applying", - kind: "runtime-k8s-job", - jobNamePrefix: "hwlab-runtime-provision", - commandArgs: [ - "cmd/hwlab-cloud-api/provision.mjs", - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--report", - runtimeProvisioningReportPath, - "--fail-on-blocked" - ], - timeoutMs: 5 * 60 * 1000, - reportPath: runtimeProvisioningReportPath - }, - { - id: "runtime-db-migration", - phase: "applying", - kind: "runtime-k8s-job", - jobNamePrefix: "hwlab-runtime-migrate", - commandArgs: [ - "cmd/hwlab-cloud-api/migrate.mjs", - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--report", - runtimeMigrationReportPath, - "--fail-on-blocked" - ], - timeoutMs: 5 * 60 * 1000, - reportPath: runtimeMigrationReportPath - }, - { - id: "dev-deploy-apply", - phase: "applying", - command: process.execPath, - args: [ - "scripts/dev-deploy-apply.mjs", - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--report", - deployApplyReportPath, - ...(args.kubeconfig ? ["--kubeconfig", args.kubeconfig] : []), - ...(args.registryManifestBaseUrl ? ["--registry-manifest-base-url", args.registryManifestBaseUrl] : []) - ], - timeoutMs: 20 * 60 * 1000, - reportPath: deployApplyReportPath - }, - ...( - args.skipRuntimePostflight - ? [] - : [{ - id: "runtime-durable-postflight", - phase: "verifying", - command: process.execPath, - args: [ - "scripts/dev-runtime-postflight.mjs", - "--live", - "--confirm-dev", - "--confirmed-non-production", - "--target", - "api", - "--report", - runtimePostflightReportPath, - "--fail-on-blocked" - ], - timeoutMs: 2 * 60 * 1000, - reportPath: runtimePostflightReportPath - }] - ) - ]; - - if (blockers.length === 0) { - for (const step of plannedSteps) { - if (step.phase !== lockState.lock?.phase) { - await updateDeployLockPhase({ - ctx, - args, - kubectlContext, - transactionId: transaction.transactionId, - lockState, - phase: step.phase - }); - transaction.phases.push({ phase: step.phase, status: "entered", at: ctx.now().toISOString() }); - } - const result = step.kind === "runtime-k8s-job" - ? await runRuntimeK8sJobStep(ctx, transaction, step) - : await runStep(ctx, transaction, step); - steps.push(result); - if (result.status !== "pass") { - blockers.push(stepBlocker(result)); - break; - } - } - } - - if (blockers.length === 0) { - if (lockState.lock?.phase !== "verifying") { - await updateDeployLockPhase({ - ctx, - args, - kubectlContext, - transactionId: transaction.transactionId, - lockState, - phase: "verifying" - }); - transaction.phases.push({ phase: "verifying", status: "entered", at: ctx.now().toISOString() }); - } - deployAfter = await readDeployJson(ctx.repoRoot); - liveVerify = args.skipLiveVerify - ? { status: "not_run", reason: "--skip-live-verify", endpoints: [], summary: { checked: 0 } } - : await verifyDevLive(ctx, { expectedCommit: target.shortCommitId }); - if (liveVerify.status !== "pass") { - blockers.push({ - type: "observability_blocker", - scope: "live-verify", - status: "open", - summary: "16666/16667 live health did not match the DEV CD promotion commit." - }); - } - } else { - deployAfter = deployBefore; - liveVerify = { status: "not_run", reason: "transaction blocked before live verify", endpoints: [], summary: { checked: 0 } }; - } - - status = blockers.length === 0 ? "pass" : "blocked"; - release = await releaseDeployLock({ - ctx, - args, - kubectlContext, - transactionId: transaction.transactionId, - status - }); - transaction.phases.push({ phase: "released", status: release.status, at: ctx.now().toISOString() }); - } catch (error) { - if (error?.lockFailure) { - stdout.write(`${JSON.stringify(error.lockFailure, null, 2)}\n`); - return 2; - } - if (!error?.suppressCatchBlocker) { - blockers.push({ - type: "runtime_blocker", - scope: error.code ?? "dev-cd-apply", - status: "open", - summary: oneLine(error.message) - }); - } - status = "blocked"; - if (lockState?.acquired) { - const kubectlContext = await resolveKubectlContext(args, env, ctx.runCommand); - release = await releaseDeployLock({ - ctx, - args, - kubectlContext, - transactionId: transaction.transactionId, - status - }); - transaction.phases.push({ phase: "released", status: release.status, at: ctx.now().toISOString() }); - } - } - - const finishedAt = ctx.now().toISOString(); - const report = buildReport({ - args, - transaction, - target: target ?? { - ref: args.targetRef, - commitId: "unknown", - shortCommitId: "unknown", - headCommitId: "unknown", - headMatchesTarget: false - }, - deployBefore: deployBefore ?? { path: "deploy/deploy.json", hash: "unknown", commitId: "unknown" }, - deployAfter: deployAfter ?? deployBefore ?? { path: "deploy/deploy.json", hash: "unknown", commitId: "unknown" }, - lockState, - steps, - blockers, - status, - startedAt, - finishedAt, - preflight: preflight ?? { - status: "not_run", - scope: "pre-lock-pre-side-effect", - blockers: [], - safety: { - writeSideEffectsAttempted: false, - prodTouched: false, - secretValuesRead: false, - secretValuesPrinted: false - } - }, - liveBefore: liveBefore ?? { status: "not_run" }, - liveVerify: liveVerify ?? { status: "not_run", endpoints: [], summary: { checked: 0 } }, - release - }); - - if (args.writeReport) { - const absoluteReportPath = ensureNotRepoReportsPath(ctx.repoRoot, args.reportPath, "--report"); - await mkdir(path.dirname(absoluteReportPath), { recursive: true }); - await writeFile(absoluteReportPath, `${JSON.stringify(report, null, 2)}\n`); - } - - const stdoutPayload = args.fullOutput ? report : summarizeDevCdApplyReport(report); - stdout.write(`${JSON.stringify(stdoutPayload, null, 2)}\n`); - return status === "pass" ? 0 : 2; -} diff --git a/scripts/src/dev-cd-apply.test.mjs b/scripts/src/dev-cd-apply.test.mjs deleted file mode 100644 index 6dd39e0e..00000000 --- a/scripts/src/dev-cd-apply.test.mjs +++ /dev/null @@ -1,1790 +0,0 @@ -import assert from "node:assert/strict"; -import test from "node:test"; -import { execFile } from "node:child_process"; -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import os from "node:os"; -import path from "node:path"; -import { promisify } from "node:util"; - -import { - buildLockAnnotations, - classifyDeployLock, - deployLockHeldFailure, - parseDeployLock, - runDevCdApply, - verifyDevLive -} from "./dev-cd-apply.mjs"; - -const execFileAsync = promisify(execFile); -const repoRoot = path.resolve(path.dirname(new URL(import.meta.url).pathname), "../.."); -const tempReportRoot = "/tmp/hwlab-dev-gate"; -const artifactReportFixturePath = path.join(tempReportRoot, "dev-artifacts.json"); - -function tempReport(name) { - return path.join(tempReportRoot, name); -} - -function stateReport(repoRoot, name) { - return path.join(repoRoot, ".state/test-reports", name); -} - -function iso(offsetMs = 0) { - return new Date(Date.parse("2026-05-23T08:00:00.000Z") + offsetMs).toISOString(); -} - -function leaseFromLock(lock) { - return { - apiVersion: "coordination.k8s.io/v1", - kind: "Lease", - metadata: { - name: "hwlab-dev-cd-lock", - namespace: "hwlab-dev", - resourceVersion: "1", - annotations: buildLockAnnotations(lock) - }, - spec: { - holderIdentity: `${lock.ownerTaskId}/${lock.transactionId}`, - leaseDurationSeconds: lock.ttlSeconds, - acquireTime: lock.startedAt, - renewTime: lock.updatedAt - } - }; -} - -function parsePatch(args) { - const index = args.indexOf("-p"); - assert.notEqual(index, -1, "kubectl patch must include -p"); - return JSON.parse(args[index + 1]); -} - -async function makeRepo(options = {}) { - const commitId = options.commitId ?? "abc1234"; - const repoRoot = await fsTempDir(); - await mkdir(path.join(repoRoot, "deploy/k8s/base"), { recursive: true }); - await mkdir(tempReportRoot, { recursive: true }); - await writeFile( - path.join(repoRoot, "deploy/deploy.json"), - `${JSON.stringify({ - manifestVersion: "v1", - environment: "dev", - namespace: "hwlab-dev", - endpoint: "http://74.48.78.17:16667", - commitId, - services: [ - { - serviceId: "hwlab-cloud-api", - image: `127.0.0.1:5000/hwlab/hwlab-cloud-api:${commitId.slice(0, 7)}`, - namespace: "hwlab-dev", - profile: "dev", - env: { HWLAB_COMMIT_ID: commitId.slice(0, 7) } - }, - { - serviceId: "hwlab-cloud-web", - image: `127.0.0.1:5000/hwlab/hwlab-cloud-web:${commitId.slice(0, 7)}`, - namespace: "hwlab-dev", - profile: "dev", - env: { HWLAB_COMMIT_ID: commitId.slice(0, 7) } - } - ] - }, null, 2)}\n` - ); - await writeFile( - path.join(repoRoot, "deploy/artifact-catalog.dev.json"), - `${JSON.stringify({ - catalogVersion: "v1", - kind: "hwlab-artifact-catalog", - environment: "dev", - profile: "dev", - namespace: "hwlab-dev", - endpoint: "http://74.48.78.17:16667", - commitId, - artifactState: "published", - publish: { - ciPublished: true, - registryVerified: true, - provenance: "ci-artifact:dev-artifacts.json" - }, - services: [ - { - serviceId: "hwlab-cloud-api", - commitId, - image: `127.0.0.1:5000/hwlab/hwlab-cloud-api:${commitId.slice(0, 7)}`, - imageTag: commitId.slice(0, 7), - digest: "sha256:b3713aa07164a9ff79a436bd583552427de42922835a5baff50484f53d98f296", - publishState: "published", - profile: "dev", - namespace: "hwlab-dev", - artifactRequired: true - }, - { - serviceId: "hwlab-cloud-web", - commitId, - image: `127.0.0.1:5000/hwlab/hwlab-cloud-web:${commitId.slice(0, 7)}`, - imageTag: commitId.slice(0, 7), - digest: "sha256:3be80fcfa769d68b0a7ec44e9215b93546d01bf23411533c5cb4888438f2bd84", - publishState: "published", - profile: "dev", - namespace: "hwlab-dev", - artifactRequired: true - } - ] - }, null, 2)}\n` - ); - await writeFile( - artifactReportFixturePath, - `${JSON.stringify({ - reportVersion: "v1", - taskId: "dev-artifact-publish", - status: "published", - commitId, - artifactPublish: { - status: "published", - mode: "publish", - sourceCommitId: commitId, - registryPrefix: "127.0.0.1:5000/hwlab", - serviceCount: 2, - requiredServiceCount: 2, - publishedCount: 2, - generatedAt: iso(), - services: [ - { - serviceId: "hwlab-cloud-api", - status: "published", - artifactRequired: true, - image: `127.0.0.1:5000/hwlab/hwlab-cloud-api:${commitId.slice(0, 7)}`, - digest: "sha256:b3713aa07164a9ff79a436bd583552427de42922835a5baff50484f53d98f296" - }, - { - serviceId: "hwlab-cloud-web", - status: "published", - artifactRequired: true, - image: `127.0.0.1:5000/hwlab/hwlab-cloud-web:${commitId.slice(0, 7)}`, - digest: "sha256:3be80fcfa769d68b0a7ec44e9215b93546d01bf23411533c5cb4888438f2bd84" - } - ] - } - }, null, 2)}\n` - ); - await writeFile(path.join(repoRoot, "deploy/k8s/base/workloads.yaml"), `# fixture ${commitId}\n`); - return repoRoot; -} - -async function fsTempDir() { - return await import("node:fs/promises").then((fs) => fs.mkdtemp(path.join(os.tmpdir(), "hwlab-cd-"))); -} - -function makeHttpGetJson() { - return async (url) => { - if (url.includes("16666")) { - return { - statusCode: 200, - json: { - serviceId: "hwlab-cloud-web", - environment: "dev", - status: "ok", - revision: "abc1234", - image: { reference: "127.0.0.1:5000/hwlab/hwlab-cloud-web:abc1234", tag: "abc1234" }, - observedAt: iso() - } - }; - } - return { - statusCode: 200, - json: { - serviceId: "hwlab-cloud-api", - environment: "dev", - status: "degraded", - ready: false, - commit: { id: "abc1234", source: "runtime-env" }, - image: { reference: "127.0.0.1:5000/hwlab/hwlab-cloud-api:abc1234", tag: "abc1234" }, - blockerCodes: ["runtime_durable_adapter_auth_blocked"], - observedAt: iso() - } - }; - }; -} - -const publishedCommit = "abc1234abc1234abc1234abc1234abc1234abc1"; -const laterControlCommit = "def5678def5678def5678def5678def5678def5"; -const artifactMergeCommit = "fedcba9fedcba9fedcba9fedcba9fedcba9fedc"; -const artifactReportCommit = "a77efa1a77efa1a77efa1a77efa1a77efa1a77e"; - -function makeRunCommand({ - heldLock = null, - commandLog = [], - targetCommitId = publishedCommit, - headCommitId = targetCommitId, - extraGitRefs = {}, - commitParents = {}, - latestArtifactMergeCommit = null, - changedPaths = [ - "deploy/artifact-catalog.dev.json", - "deploy/deploy.json", - "deploy/k8s/base/workloads.yaml", - "ci-artifact:dev-artifacts.json" - ], - desiredStatePromotionStatus = "pass", - runtimeJobLogSuffix = "", - runtimePostflightReport = null, - kubeContext = "d601", - kubeServer = "https://d601:6443", - kubeNodes = ["d601"], - bareKubeContext = null, - bareKubeServer = null, - bareKubeNodes = null, - bareHwlabDevNamespace = false, - secretRefs = { - "hwlab-cloud-api-dev-db": ["database-url"], - "hwlab-cloud-api-dev-db-admin": ["admin-url"], - "hwlab-code-agent-provider": ["openai-api-key"], - "hwlab-code-agent-codex-auth": ["auth.json"] - }, - configMaps = { - "hwlab-code-agent-codex-config": ["config.toml"] - } -} = {}) { - let lease = heldLock ? leaseFromLock(heldLock) : null; - const assertLeaseMicroTime = (value) => { - assert.match(value, /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{6}Z$/u); - }; - return async (command, args, options = {}) => { - commandLog.push({ command, args, env: options.env ?? {}, input: options.input ?? "" }); - if (command === "git" && args[0] === "rev-parse" && args.includes("origin/main^{commit}")) { - return { code: 0, stdout: `${targetCommitId}\n`, stderr: "" }; - } - if (command === "git" && args[0] === "rev-parse" && args.includes("--short=7")) { - const ref = String(args.at(-1) ?? ""); - const commitId = extraGitRefs[ref] ?? ref; - return { code: 0, stdout: `${commitId.slice(0, 7)}\n`, stderr: "" }; - } - if (command === "git" && args[0] === "rev-parse" && args.includes("HEAD^{commit}")) { - return { code: 0, stdout: `${headCommitId}\n`, stderr: "" }; - } - if (command === "git" && args[0] === "rev-parse" && args[1] === "--verify") { - const ref = String(args[2] ?? "").replace(/\^\{commit\}$/u, ""); - const commitId = extraGitRefs[ref]; - if (commitId) return { code: 0, stdout: `${commitId}\n`, stderr: "" }; - return { code: 1, stdout: "", stderr: `fatal: ambiguous argument '${ref}'` }; - } - if (command === "git" && args[0] === "log" && args.includes("--first-parent")) { - return latestArtifactMergeCommit - ? { code: 0, stdout: `${latestArtifactMergeCommit}\n`, stderr: "" } - : { code: 1, stdout: "", stderr: "no matching commits" }; - } - if (command === "git" && args[0] === "rev-list" && args.includes("--parents")) { - const ref = String(args.at(-1) ?? ""); - const parents = commitParents[ref] ?? []; - return { code: 0, stdout: `${[ref, ...parents].join(" ")}\n`, stderr: "" }; - } - if (command === "git" && args[0] === "diff-tree") { - return { code: 0, stdout: `${changedPaths.join("\n")}\n`, stderr: "" }; - } - if (command === "which" && args[0] === "kubectl") { - return { code: 0, stdout: "/usr/local/bin/kubectl\n", stderr: "" }; - } - if (command.includes("kubectl") && args[0] === "config" && args[1] === "current-context") { - const hasKubeconfig = Object.hasOwn(options.env ?? {}, "KUBECONFIG"); - return { code: 0, stdout: `${hasKubeconfig ? kubeContext : (bareKubeContext ?? kubeContext)}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args[0] === "config" && args[1] === "view") { - const hasKubeconfig = Object.hasOwn(options.env ?? {}, "KUBECONFIG"); - return { code: 0, stdout: `${hasKubeconfig ? kubeServer : (bareKubeServer ?? kubeServer)}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args[0] === "get" && args[1] === "nodes") { - const hasKubeconfig = Object.hasOwn(options.env ?? {}, "KUBECONFIG"); - return { code: 0, stdout: `${(hasKubeconfig ? kubeNodes : (bareKubeNodes ?? kubeNodes)).join(" ")}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args[0] === "get" && args[1] === "namespace") { - return bareHwlabDevNamespace - ? { code: 0, stdout: "namespace/hwlab-dev\n", stderr: "" } - : { code: 1, stdout: "", stderr: "Error from server (NotFound): namespaces \"hwlab-dev\" not found" }; - } - if (command.includes("kubectl") && args.includes("get") && args.includes("secret")) { - const name = args[args.indexOf("secret") + 1]; - if (!secretRefs[name]) { - return { code: 1, stdout: "", stderr: `Error from server (NotFound): secrets "${name}" not found` }; - } - return { code: 0, stdout: `secret/${name}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args.includes("describe") && args.includes("secret")) { - const name = args[args.indexOf("secret") + 1]; - const keys = secretRefs[name]; - if (!keys) { - return { code: 1, stdout: "", stderr: `Error from server (NotFound): secrets "${name}" not found` }; - } - return { - code: 0, - stdout: `Name: ${name}\nNamespace: hwlab-dev\nType: Opaque\n\nData\n====\n${keys.map((key) => `${key}: 16 bytes`).join("\n")}\n`, - stderr: "" - }; - } - if (command.includes("kubectl") && args.includes("get") && args.includes("configmap")) { - const name = args[args.indexOf("configmap") + 1]; - const keys = configMaps[name]; - if (!keys) { - return { code: 1, stdout: "", stderr: `Error from server (NotFound): configmaps "${name}" not found` }; - } - if (args.some((arg) => String(arg).startsWith("go-template="))) { - return { code: 0, stdout: `${keys.join("\n")}\n`, stderr: "" }; - } - return { code: 0, stdout: `configmap/${name}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args.includes("get") && args.includes("lease")) { - if (!lease) return { code: 1, stdout: "", stderr: "Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found" }; - return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args.includes("create")) { - lease = JSON.parse(options.input); - assertLeaseMicroTime(lease.spec.acquireTime); - assertLeaseMicroTime(lease.spec.renewTime); - lease.metadata.resourceVersion = "2"; - return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args.includes("replace")) { - const nextLease = JSON.parse(options.input); - assert.equal(nextLease.metadata?.resourceVersion, lease?.metadata?.resourceVersion); - if (nextLease.spec?.acquireTime) assertLeaseMicroTime(nextLease.spec.acquireTime); - if (nextLease.spec?.renewTime) assertLeaseMicroTime(nextLease.spec.renewTime); - lease = nextLease; - lease.metadata.resourceVersion = String(Number(lease.metadata.resourceVersion ?? "2") + 1); - return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; - } - if (command.includes("kubectl") && args.includes("apply") && args.includes("-f") && args.includes("-")) { - const manifest = JSON.parse(options.input); - assert.equal(manifest.kind, "Job"); - assert.equal(manifest.metadata.namespace, "hwlab-dev"); - assert.ok(manifest.metadata.name.startsWith("hwlab-runtime-")); - assert.equal(manifest.spec.template.spec.containers[0].env.some((entry) => entry.name === "HWLAB_CLOUD_DB_URL" && entry.valueFrom?.secretKeyRef?.name === "hwlab-cloud-api-dev-db"), true); - assert.equal(manifest.spec.template.spec.containers[0].env.some((entry) => entry.name === "HWLAB_CLOUD_DB_ADMIN_URL" && entry.valueFrom?.secretKeyRef?.name === "hwlab-cloud-api-dev-db-admin"), true); - assert.equal(JSON.stringify(manifest).includes("database-url"), true); - assert.equal(JSON.stringify(manifest).includes("postgresql://"), false); - return { code: 0, stdout: JSON.stringify(manifest), stderr: "" }; - } - if (command.includes("kubectl") && args.includes("wait") && args.some((arg) => String(arg).startsWith("job/hwlab-runtime-"))) { - return { code: 0, stdout: "job.batch/hwlab-runtime complete\n", stderr: "" }; - } - if (command.includes("kubectl") && args.includes("logs") && args.some((arg) => String(arg).startsWith("job/hwlab-runtime-provision"))) { - return { - code: 0, - stdout: `${runtimeJobLogSuffix}${JSON.stringify({ conclusion: { status: "ready" }, safety: { secretValuesPrinted: false } }, null, 2)}\n`, - stderr: "" - }; - } - if (command.includes("kubectl") && args.includes("logs") && args.some((arg) => String(arg).startsWith("job/hwlab-runtime-migrate"))) { - return { - code: 0, - stdout: `${runtimeJobLogSuffix}${JSON.stringify({ conclusion: { status: "ready" }, safety: { secretValuesPrinted: false } }, null, 2)}\n`, - stderr: "" - }; - } - if (command.includes("kubectl") && args.includes("patch")) { - assert.ok(lease, "lease must exist before patch"); - const patch = parsePatch(args); - lease.metadata.annotations = { - ...lease.metadata.annotations, - ...(patch.metadata?.annotations ?? {}) - }; - lease.spec = { - ...lease.spec, - ...(patch.spec ?? {}) - }; - lease.metadata.resourceVersion = String(Number(lease.metadata.resourceVersion ?? "2") + 1); - return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; - } - if ((command === process.execPath || command.endsWith("/node")) && args.includes("scripts/deploy-desired-state-plan.mjs") && args.includes("--promotion-commit")) { - const promotionCommit = args[args.indexOf("--promotion-commit") + 1]; - const status = desiredStatePromotionStatus; - return { - code: status === "pass" ? 0 : 1, - stdout: `${JSON.stringify({ - status, - summary: { - desiredCommitId: promotionCommit.slice(0, 7), - artifactState: "published", - ciPublished: true, - registryVerified: true, - blockers: status === "pass" ? 0 : 1 - }, - target: { - convergence: { - state: status === "pass" ? "already_promoted" : "promotion_mismatch" - } - }, - diagnostics: status === "pass" ? [] : [{ code: "promotion_commit_mismatch", path: "desired-state", message: "fixture mismatch" }] - })}\n`, - stderr: "" - }; - } - if (command === process.execPath || command.endsWith("/node")) { - if (args.includes("scripts/dev-runtime-postflight.mjs")) { - const report = runtimePostflightReport ?? { - conclusion: { status: "pass" }, - blockers: [], - safety: { secretValuesPrinted: false }, - apiHealth: { runtimeDurableReadiness: { ready: true } }, - apiV1: { runtimeDurableReadiness: { ready: true } }, - m3: { - persistenceContract: { ready: true }, - operationCount: 4 - } - }; - return { - code: report.conclusion?.status === "pass" ? 0 : 1, - stdout: `${JSON.stringify(report, null, 2)}\n`, - stderr: "" - }; - } - return { code: 0, stdout: JSON.stringify({ ok: true, command: args.join(" ") }), stderr: "" }; - } - return { code: 0, stdout: "", stderr: "" }; - }; -} - -function assertNoReadOnlySideEffects(commandLog) { - const mutatingKubectlVerbs = new Set(["apply", "create", "replace", "patch", "delete", "rollout", "wait", "logs"]); - assert.equal(commandLog.some((entry) => entry.command === "ps"), false); - assert.equal(commandLog.some((entry) => entry.command === "kill"), false); - assert.equal(commandLog.some((entry) => - entry.command.includes("kubectl") && entry.args.some((arg) => mutatingKubectlVerbs.has(arg)) - ), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/refresh-artifact-catalog.mjs")), false); -} - -function assertNoWriteSideEffects(commandLog) { - const writeKubectlVerbs = new Set(["apply", "create", "replace", "patch", "delete", "rollout", "wait", "logs"]); - assert.equal(commandLog.some((entry) => - entry.command.includes("kubectl") && entry.args.some((arg) => writeKubectlVerbs.has(arg)) - ), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/refresh-artifact-catalog.mjs")), false); -} - -test("lock classifier reports held and stale states with retryAfterSeconds", () => { - const lock = { - ownerTaskId: "task-a", - transactionId: "tx-a", - promotionCommit: "abc1234", - deployJsonHash: "sha256:a", - phase: "applying", - startedAt: iso(0), - updatedAt: iso(0), - ttlSeconds: 300, - targetNamespace: "hwlab-dev" - }; - const parsed = parseDeployLock(leaseFromLock(lock)); - const held = classifyDeployLock(parsed, new Date(iso(120000))); - assert.equal(held.held, true); - assert.equal(held.stale, false); - assert.equal(held.retryAfterSeconds, 180); - - const stale = classifyDeployLock(parsed, new Date(iso(301000))); - assert.equal(stale.held, false); - assert.equal(stale.stale, true); - assert.equal(stale.retryAfterSeconds, 0); -}); - -test("dev-cd status and dry-run are read-only compact JSON", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--dry-run", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json") - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ commandLog }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - assert.equal(code, 0); - assert.equal(status.mode, "dry-run"); - assert.equal(status.mutationAttempted, false); - assert.equal(status.prodTouched, false); - assert.equal(status.lock.status, "absent"); - assert.equal(status.deployJson.manifest, undefined); - assert.equal(status.deployJson.matchesTarget, true); - assert.equal(status.live.summary.checked, 2); - assertNoReadOnlySideEffects(commandLog); - await assert.rejects( - readFile(stateReport(repoRoot, "dev-cd-apply.json"), "utf8"), - { code: "ENOENT" } - ); -}); - -test("dev-cd default status is read-only compact JSON", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ commandLog }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - assert.equal(code, 0); - assert.equal(status.mode, "status"); - assert.equal(status.mutationAttempted, false); - assert.equal(status.prodTouched, false); - assert.equal(status.target.shortCommitId, "abc1234"); - assert.equal(status.deployJson.hash.startsWith("sha256:"), true); - assert.equal(status.deployJson.matchesTarget, true); - assert.equal(status.lock.status, "absent"); - assertNoReadOnlySideEffects(commandLog); -}); - -test("dev-cd status accepts a published deploy-json promotion under a later control HEAD", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - const commandLog = []; - let output = ""; - const code = await runDevCdApply(["--status", "--kubeconfig", "/etc/rancher/k3s/k3s.yaml"], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: laterControlCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [publishedCommit, artifactReportCommit], - [artifactReportCommit]: [publishedCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - assert.equal(code, 0); - assert.equal(status.status, "pass"); - assert.equal(status.target.promotionCommit, publishedCommit); - assert.equal(status.target.shortCommitId, "abc1234"); - assert.equal(status.target.promotionSource, "deploy-json"); - assert.equal(status.target.publishRequired, false); - assert.equal(status.target.controlRef.shortCommitId, "def5678"); - assert.equal(status.target.artifactBoundary.status, "pass"); - assert.equal(status.target.artifactBoundary.artifactMergeCommit.shortCommitId, "fedcba9"); - assert.equal(status.target.artifactBoundary.artifactMergeCommit.promotionParentMatches, true); - assert.equal(status.target.artifactBoundary.artifactCommit.shortCommitId, "a77efa1"); - assert.equal(status.target.artifactBoundary.artifactCommit.promotionParentMatches, true); - assert.equal(status.artifactCatalog.commitId, "abc1234"); - assert.equal(status.artifactCatalog.digestCounts.sha256, 2); - assert.equal(status.artifactReport.sourceCommitId, "abc1234"); - assert.equal(status.liveDelta.status, "pass"); - assert.equal(status.liveDelta.expectedCommit, "abc1234"); - assert.equal(status.target.headMatchesTarget, true); - assert.equal(status.target.desiredStateCheck.status, "pass"); - assert.equal(status.deployJson.matchesTarget, true); - assert.match(status.nextActions[0], /without republishing HEAD/u); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); - assert.equal(commandLog.some((entry) => - entry.args.includes("scripts/deploy-desired-state-plan.mjs") && - entry.args.includes("--promotion-commit") && - entry.args.includes(publishedCommit) - ), true); - assertNoReadOnlySideEffects(commandLog); -}); - -test("dev-cd status treats stale CI artifact report as audit-only when desired state matches", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - await writeFile( - artifactReportFixturePath, - `${JSON.stringify({ - status: "published", - commitId: "9999999", - artifactPublish: { - status: "published", - sourceCommitId: "9999999", - serviceCount: 0, - requiredServiceCount: 0, - publishedCount: 0, - services: [] - } - }, null, 2)}\n` - ); - const commandLog = []; - let output = ""; - const code = await runDevCdApply(["--status", "--kubeconfig", "/etc/rancher/k3s/k3s.yaml"], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: laterControlCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [publishedCommit, artifactReportCommit], - [artifactReportCommit]: [publishedCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - assert.equal(code, 0); - assert.equal(status.status, "pass"); - assert.equal(status.target.artifactBoundary.status, "pass"); - assert.equal(status.target.artifactBoundary.reportStatus, "stale_or_unrelated"); - assert.equal(status.target.artifactBoundary.reportIsReleaseGate, false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assertNoReadOnlySideEffects(commandLog); -}); - -test("dev-cd status exposes moving origin/main when checkout is behind a deploy-json promotion control ref", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - const commandLog = []; - let output = ""; - const code = await runDevCdApply(["--status", "--kubeconfig", "/etc/rancher/k3s/k3s.yaml"], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: publishedCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [laterControlCommit, artifactReportCommit], - [artifactReportCommit]: [laterControlCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - assert.equal(code, 0); - assert.equal(status.status, "degraded"); - assert.equal(status.target.promotionCommit, publishedCommit); - assert.equal(status.target.controlRef.shortCommitId, "def5678"); - assert.equal(status.target.headCommitId, publishedCommit); - assert.equal(status.target.headMatchesTarget, false); - assert.match(status.nextActions.join("\n"), /Checkout or fast-forward/u); - assertNoReadOnlySideEffects(commandLog); -}); - -test("apply stops before lock acquisition when checkout is behind deploy-json promotion control ref", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-moving-main", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: publishedCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [publishedCommit, artifactReportCommit], - [artifactReportCommit]: [publishedCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.target.controlRef.shortCommitId, "def5678"); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "target-ref-head"), true); - assert.equal(commandLog.some((entry) => entry.command.includes("kubectl")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); -}); - -test("dev-cd status degrades artifact merge provenance when catalog does not match deploy-json promotion", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - await writeFile( - path.join(repoRoot, "deploy/artifact-catalog.dev.json"), - `${JSON.stringify({ - kind: "hwlab-artifact-catalog", - commitId: "9999999", - artifactState: "published", - publish: { ciPublished: true, registryVerified: true, provenance: "ci-artifact:dev-artifacts.json" }, - services: [] - }, null, 2)}\n` - ); - const commandLog = []; - let output = ""; - const code = await runDevCdApply(["--status", "--kubeconfig", "/etc/rancher/k3s/k3s.yaml"], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: laterControlCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [publishedCommit, artifactReportCommit], - [artifactReportCommit]: [publishedCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - assert.equal(code, 0); - assert.equal(status.status, "degraded"); - assert.equal(status.target.promotionCommit, publishedCommit); - assert.equal(status.target.artifactBoundary.status, "degraded"); - assert.equal(status.target.artifactBoundary.desiredState.deployCommitMatches, true); - assert.equal(status.target.artifactBoundary.desiredState.catalogCommitMatches, false); - assert.equal(status.target.artifactBoundary.desiredState.reportCommitMatches, true); - assertNoReadOnlySideEffects(commandLog); -}); - -test("apply treats artifact audit mismatch as non-blocking when deploy-json promotion desired state passes", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - await writeFile( - artifactReportFixturePath, - `${JSON.stringify({ - status: "published", - commitId: "9999999", - artifactPublish: { - status: "published", - sourceCommitId: "9999999", - serviceCount: 0, - requiredServiceCount: 0, - publishedCount: 0, - services: [] - } - }, null, 2)}\n` - ); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-provenance", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--full-output" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: laterControlCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [laterControlCommit, artifactReportCommit], - [artifactReportCommit]: [laterControlCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - assert.equal(code, 0); - assert.equal(summary.status, "pass"); - assert.equal(summary.devCdApply.target.promotionSource, "deploy-json"); - assert.equal(summary.devCdApply.target.publishRequired, false); - assert.equal(summary.devCdApply.target.desiredStateCheck.status, "pass"); - assert.equal(summary.devCdApply.target.artifactBoundary.status, "degraded"); - assert.equal(summary.devCdApply.target.artifactBoundary.reportStatus, "stale_or_unrelated"); - assert.equal(summary.devCdApply.target.artifactBoundary.reportIsReleaseGate, false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/refresh-artifact-catalog.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/deploy-desired-state-plan.mjs")), true); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), true); -}); - -test("apply stops before lock acquisition when deploy-json promotion desired-state check fails", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-promotion-check-failed", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: laterControlCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - desiredStatePromotionStatus: "blocked", - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [laterControlCommit, artifactReportCommit], - [artifactReportCommit]: [laterControlCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "desired-state-promotion"), true); - assert.equal(commandLog.some((entry) => entry.command.includes("kubectl")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); -}); - -test("dev-cd status summarizes held, released, and stale locks without unsafe takeover hints", async () => { - const cases = [ - { - name: "held", - phase: "applying", - now: iso(100000), - expectedStatus: "held", - expectedRetryAfterSeconds: 200 - }, - { - name: "released", - phase: "released", - releasedAt: iso(50000), - releaseStatus: "pass", - now: iso(100000), - expectedStatus: "released", - expectedRetryAfterSeconds: 0 - }, - { - name: "stale", - phase: "verifying", - now: iso(301000), - expectedStatus: "stale", - expectedRetryAfterSeconds: 0 - } - ]; - - for (const item of cases) { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply(["--status", "--kubeconfig", "/etc/rancher/k3s/k3s.yaml"], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - heldLock: { - promotionCommit: "abc1234", - deployJsonHash: "sha256:a", - ownerTaskId: `task-${item.name}`, - transactionId: `tx-${item.name}`, - phase: item.phase, - startedAt: iso(0), - updatedAt: iso(0), - ttlSeconds: 300, - liveBefore: { status: "pass" }, - targetNamespace: "hwlab-dev", - targetRef: "origin/main", - lockBackend: "Lease", - ...(item.releasedAt ? { releasedAt: item.releasedAt } : {}), - ...(item.releaseStatus ? { releaseStatus: item.releaseStatus } : {}) - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(item.now), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - assert.equal(code, 0); - assert.equal(status.lock.status, item.expectedStatus); - assert.equal(status.lock.lockName, "hwlab-dev-cd-lock"); - assert.equal(status.lock.current.ownerTaskId, `task-${item.name}`); - assert.equal(status.lock.retryAfterSeconds, item.expectedRetryAfterSeconds); - assert.equal(JSON.stringify(status).includes("force-interrupt"), false); - assert.equal(JSON.stringify(status.nextActions).includes("force"), false); - if (item.expectedStatus === "stale") { - assert.match(status.lock.allowedLockAction.breakStale, /--break-stale-lock/u); - } - assertNoReadOnlySideEffects(commandLog); - } -}); - -test("dev-cd status reports unavailable kubectl with redacted reason", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const runCommand = async (command, args, options = {}) => { - commandLog.push({ command, args, env: options.env ?? {}, input: options.input ?? "" }); - if (command === "git" && args[0] === "rev-parse" && args.includes("origin/main^{commit}")) { - return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" }; - } - if (command === "git" && args[0] === "rev-parse" && args.includes("--short=7")) { - return { code: 0, stdout: `${String(args.at(-1)).slice(0, 7)}\n`, stderr: "" }; - } - if (command === "git" && args[0] === "rev-parse" && args.includes("HEAD^{commit}")) { - return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" }; - } - if (command === "git" && args[0] === "log" && args.includes("--first-parent")) { - return { code: 1, stdout: "", stderr: "no matching commits" }; - } - if (command === "git" && args[0] === "rev-list" && args.includes("--parents")) { - const ref = String(args.at(-1) ?? ""); - return { code: 0, stdout: `${ref}\n`, stderr: "" }; - } - if (command === "git" && args[0] === "diff-tree") { - return { code: 0, stdout: "", stderr: "" }; - } - if (command === "which" && args[0] === "kubectl") { - return { code: 1, stdout: "", stderr: "kubectl missing" }; - } - if (command === "kubectl") { - return { - code: 1, - stdout: "", - stderr: "connection failed token=SECRET_TOKEN password=SECRET_PASSWORD postgresql://user:SECRET_DB@db" - }; - } - return { code: 0, stdout: "", stderr: "" }; - }; - - const code = await runDevCdApply(["--status"], { - repoRoot, - env: {}, - runCommand, - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const status = JSON.parse(output); - const text = JSON.stringify(status); - assert.equal(code, 0); - assert.equal(status.status, "degraded"); - assert.equal(status.lock.status, "unavailable"); - assert.equal(status.lock.reason, "kubectl or DEV kubeconfig unavailable; command output is redacted"); - assert.equal(text.includes("SECRET_TOKEN"), false); - assert.equal(text.includes("SECRET_PASSWORD"), false); - assert.equal(text.includes("SECRET_DB"), false); - assert.match(status.lock.redactedReason, /<redacted>/u); - assertNoReadOnlySideEffects(commandLog); -}); - -test("force-interrupt option is rejected before process discovery or lease mutation", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply(["--force-interrupt-lock"], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ commandLog }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const failure = JSON.parse(output); - assert.equal(code, 2); - assert.equal(failure.error, "unknown-argument"); - assert.equal(failure.mutationAttempted, false); - assert.equal(failure.prodTouched, false); - assert.deepEqual(commandLog, []); -}); - -test("deploy-lock-held failure is structured before publish/apply", () => { - const failure = deployLockHeldFailure({ - ownerTaskId: "task-a", - transactionId: "tx-a", - promotionCommit: "abc1234", - deployJsonHash: "sha256:a", - phase: "applying", - updatedAt: iso(0), - ttlSeconds: 300, - targetNamespace: "hwlab-dev", - lockName: "hwlab-dev-cd-lock", - lockBackend: "Lease" - }, new Date(iso(100000))); - - assert.equal(failure.ok, false); - assert.equal(failure.error, "deploy-lock-held"); - assert.equal(failure.holder, "task-a"); - assert.equal(failure.promotionCommit, "abc1234"); - assert.equal(failure.phase, "applying"); - assert.equal(failure.retryAfterSeconds, 200); - assert.equal(failure.mutationAttempted, false); -}); - -test("second transaction exits deploy-lock-held without running side effects", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-b", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - heldLock: { - promotionCommit: "abc1234", - deployJsonHash: "sha256:a", - ownerTaskId: "task-a", - transactionId: "tx-a", - phase: "publishing", - startedAt: iso(0), - updatedAt: iso(0), - ttlSeconds: 300, - liveBefore: { status: "pass" }, - targetNamespace: "hwlab-dev", - targetRef: "origin/main", - lockBackend: "Lease" - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const failure = JSON.parse(output); - assert.equal(code, 2); - assert.equal(failure.error, "deploy-lock-held"); - assert.equal(failure.holder, "task-a"); - assert.equal(failure.retryAfterSeconds, 200); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); -}); - -test("stale lock requires explicit break-stale-lock confirmation", async () => { - const repoRoot = await makeRepo(); - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-b", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - heldLock: { - promotionCommit: "abc1234", - deployJsonHash: "sha256:a", - ownerTaskId: "task-a", - transactionId: "tx-a", - phase: "verifying", - startedAt: iso(0), - updatedAt: iso(0), - ttlSeconds: 300, - liveBefore: { status: "pass" }, - targetNamespace: "hwlab-dev", - targetRef: "origin/main", - lockBackend: "Lease" - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(301000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const failure = JSON.parse(output); - assert.equal(code, 2); - assert.equal(failure.error, "deploy-lock-held"); - assert.equal(failure.stale, true); - assert.equal(failure.requiresBreakStaleLock, true); -}); - -test("apply preflight passes with D601 native k3s and required SecretRef keys before mutation", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-preflight-pass", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json"), - "--full-output" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ commandLog }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const report = JSON.parse(output); - const preflight = report.devCdApply.preflight; - assert.equal(code, 0); - assert.equal(preflight.status, "pass"); - assert.equal(preflight.kubeconfig.required, "/etc/rancher/k3s/k3s.yaml"); - assert.equal(preflight.kubeconfig.enforcedEnv, true); - assert.deepEqual(preflight.controlPlane.forced.nodeNames, ["d601"]); - assert.deepEqual( - preflight.secretRefs.map((ref) => `${ref.secretName}/${ref.secretKey}:${ref.status}`), - [ - "hwlab-cloud-api-dev-db/database-url:present", - "hwlab-cloud-api-dev-db-admin/admin-url:present", - "hwlab-code-agent-provider/openai-api-key:present", - "hwlab-code-agent-codex-auth/auth.json:present" - ] - ); - assert.deepEqual( - preflight.configMaps.map((ref) => `${ref.configMapName}/${ref.configMapKey}:${ref.status}`), - ["hwlab-code-agent-codex-config/config.toml:present"] - ); - assert.equal(preflight.safety.secretValuesRead, false); - assert.equal(preflight.safety.secretValuesPrinted, false); - assert.equal(preflight.safety.configMapValuesRead, false); - assert.equal(preflight.safety.configMapValuesPrinted, false); - - const firstWriteIndex = commandLog.findIndex((entry) => - entry.command.includes("kubectl") && ["create", "replace", "apply", "patch", "delete", "rollout", "wait", "logs"].some((verb) => entry.args.includes(verb)) - ); - const secretDescribeIndexes = commandLog - .map((entry, index) => ({ entry, index })) - .filter(({ entry }) => entry.command.includes("kubectl") && entry.args.includes("describe") && entry.args.includes("secret")) - .map(({ index }) => index); - assert.ok(secretDescribeIndexes.length > 0); - assert.ok(secretDescribeIndexes.every((index) => index < firstWriteIndex)); -}); - -test("apply preflight blocks missing SecretRef before Lease or side-effect commands", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-missing-secret", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json") - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - secretRefs: { - "hwlab-cloud-api-dev-db": ["database-url"], - "hwlab-cloud-api-dev-db-admin": ["admin-url"], - "hwlab-code-agent-codex-auth": ["auth.json"] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - const writtenReport = JSON.parse(await readFile(stateReport(repoRoot, "dev-cd-apply.json"), "utf8")); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.preflight.status, "blocked"); - assert.equal(summary.preflight.blockerCount, 1); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "secretref:hwlab-code-agent-provider/openai-api-key"), true); - assert.equal(writtenReport.devCdApply.preflight.safety.writeSideEffectsAttempted, false); - assert.equal(writtenReport.devCdApply.preflight.safety.secretValuesRead, false); - assert.equal(JSON.stringify(writtenReport).includes("SECRET"), false); - assertNoWriteSideEffects(commandLog); -}); - -test("apply preflight blocks missing Codex ConfigMap before Lease or workload apply", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-missing-codex-config", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json") - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - configMaps: {} - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - const writtenReport = JSON.parse(await readFile(stateReport(repoRoot, "dev-cd-apply.json"), "utf8")); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.preflight.status, "blocked"); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "configmap:hwlab-code-agent-codex-config/config.toml"), true); - assert.equal(writtenReport.devCdApply.preflight.safety.writeSideEffectsAttempted, false); - assert.equal(writtenReport.devCdApply.preflight.safety.configMapValuesPrinted, false); - assertNoWriteSideEffects(commandLog); -}); - -test("apply preflight blocks non-D601 kubeconfig path before kubectl mutation", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-wrong-path", - "--kubeconfig", - "/tmp/kubeconfig" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ commandLog }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.preflight.status, "blocked"); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "d601-kubeconfig-path"), true); - assertNoWriteSideEffects(commandLog); -}); - -test("apply preflight blocks docker-desktop and 127.0.0.1 control-plane signals before mutation", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-docker-desktop", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - kubeContext: "docker-desktop", - kubeServer: "https://127.0.0.1:11700", - kubeNodes: ["desktop-control-plane"] - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "d601-native-k3s-context"), true); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "d601-native-k3s-server"), true); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "d601-native-k3s-nodes"), true); - assertNoWriteSideEffects(commandLog); -}); - -test("apply preflight blocks second hwlab-dev control plane visible through bare kubectl", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-second-control-plane", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - bareKubeContext: "docker-desktop", - bareKubeServer: "https://127.0.0.1:11700", - bareKubeNodes: ["desktop-control-plane"], - bareHwlabDevNamespace: true - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "bare-kubectl-context"), true); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "bare-kubectl-server"), true); - assert.equal(summary.blockers.some((blocker) => blocker.scope === "second-hwlab-dev-control-plane"), true); - assertNoWriteSideEffects(commandLog); -}); - -test("transaction runs phases, allows internal side-effect env, releases lock, and reports live verify", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--break-stale-lock", - "--owner-task-id", - "task-b", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json"), - "--full-output" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - heldLock: { - promotionCommit: "old1234", - deployJsonHash: "sha256:old", - ownerTaskId: "task-a", - transactionId: "tx-a", - phase: "verifying", - startedAt: iso(0), - updatedAt: iso(0), - ttlSeconds: 300, - liveBefore: { status: "pass" }, - targetNamespace: "hwlab-dev", - targetRef: "origin/main", - lockBackend: "Lease" - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(301000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const report = JSON.parse(output); - assert.equal(code, 0); - assert.equal(report.status, "pass"); - assert.deepEqual( - report.transaction.phases.map((phase) => phase.phase), - ["publishing", "applying", "verifying", "released"] - ); - assert.equal(report.devCdApply.lock.staleBreak.ownerTaskId, "task-a"); - assert.equal(report.transaction.release.status, "released"); - assert.equal(report.devCdApply.liveVerify.status, "pass"); - assert.equal(report.devCdApply.liveVerify.summary.checked, 2); - assert.equal(report.devCdApply.liveVerify.endpoints.some((endpoint) => endpoint.id === "cloud-web-16666"), true); - assert.equal(report.devCdApply.liveVerify.endpoints.some((endpoint) => endpoint.id === "cloud-api-16667"), true); - assert.equal(report.devCdApply.reportPaths.transaction, stateReport(repoRoot, "dev-cd-apply.json")); - assert.equal(report.devCdApply.reportPaths.runtimeProvisioning, tempReport("dev-runtime-provisioning-report.json")); - assert.equal(report.devCdApply.reportPaths.runtimeMigration, tempReport("dev-runtime-migration-report.json")); - assert.equal(report.devCdApply.reportPaths.runtimePostflight, tempReport("dev-runtime-postflight-report.json")); - - const publishCall = commandLog.find((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")); - const applyCall = commandLog.find((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")); - const provisioningCall = commandLog.find((entry) => entry.args.includes("cmd/hwlab-cloud-api/provision.mjs")); - const postflightCall = commandLog.find((entry) => entry.args.includes("scripts/dev-runtime-postflight.mjs")); - const refreshCall = commandLog.find((entry) => entry.args.includes("scripts/refresh-artifact-catalog.mjs")); - assert.ok(publishCall?.env.HWLAB_CD_TRANSACTION_ID); - assert.equal(applyCall?.env.HWLAB_CD_TRANSACTION_ID, publishCall.env.HWLAB_CD_TRANSACTION_ID); - const provisioningJobApply = commandLog.find((entry) => - entry.command.includes("kubectl") && - entry.args.includes("apply") && - entry.input.includes("hwlab-runtime-provision") - ); - const migrationJobApply = commandLog.find((entry) => - entry.command.includes("kubectl") && - entry.args.includes("apply") && - entry.input.includes("hwlab-runtime-migrate") - ); - assert.ok(provisioningJobApply); - assert.ok(migrationJobApply); - const provisioningJob = JSON.parse(provisioningJobApply.input); - const migrationJob = JSON.parse(migrationJobApply.input); - assert.equal(provisioningJob.spec.template.spec.containers[0].args[0], "cmd/hwlab-cloud-api/provision.mjs"); - assert.equal(migrationJob.spec.template.spec.containers[0].args[0], "cmd/hwlab-cloud-api/migrate.mjs"); - assert.equal( - provisioningJob.spec.template.spec.containers[0].env.some((entry) => entry.name === "HWLAB_CD_TRANSACTION_ID"), - true - ); - assert.equal( - migrationJob.spec.template.spec.containers[0].env.some((entry) => entry.name === "HWLAB_CD_TRANSACTION_ID"), - true - ); - assert.equal(provisioningCall, undefined); - assert.equal(postflightCall?.env.HWLAB_CD_TRANSACTION_ID, publishCall.env.HWLAB_CD_TRANSACTION_ID); - assert.deepEqual(postflightCall.args.slice(0, 4), ["scripts/dev-runtime-postflight.mjs", "--live", "--confirm-dev", "--confirmed-non-production"]); - assert.equal(refreshCall.args[refreshCall.args.indexOf("--target-ref") + 1], "abc1234abc1234abc1234abc1234abc1234abc1"); - assert.ok( - commandLog.findIndex((entry) => entry.input.includes("hwlab-runtime-provision")) < - commandLog.findIndex((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")) - ); - assert.ok( - commandLog.findIndex((entry) => entry.input.includes("hwlab-runtime-migrate")) < - commandLog.findIndex((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")) - ); - - const writtenReport = JSON.parse(await readFile(stateReport(repoRoot, "dev-cd-apply.json"), "utf8")); - assert.equal(writtenReport.devCdApply.liveVerify.summary.checked, 2); -}); - -test("transaction can skip runtime postflight while preserving publish apply and live verify", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-skip-postflight", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json"), - "--full-output", - "--skip-runtime-postflight" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ commandLog }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const report = JSON.parse(output); - assert.equal(code, 0); - assert.equal(report.status, "pass"); - assert.equal(report.devCdApply.steps.some((step) => step.id === "artifact-publish"), true); - assert.equal(report.devCdApply.steps.some((step) => step.id === "dev-deploy-apply"), true); - assert.equal(report.devCdApply.steps.some((step) => step.id === "runtime-durable-postflight"), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-runtime-postflight.mjs")), false); - assert.equal(report.devCdApply.liveVerify.status, "pass"); - assert.equal(report.devCdApply.liveVerify.summary.checked, 2); -}); - -test("transaction blocks with structured runtime postflight blocker when M3 evidence is not persisted", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const runtimePostflightReport = { - conclusion: { - status: "blocked", - blockerCount: 1 - }, - apiHealth: { - runtimeDurableReadiness: { - ready: true, - classification: "runtime_durable_ready" - } - }, - apiV1: { - runtimeDurableReadiness: { - ready: true, - classification: "runtime_durable_ready" - } - }, - m3: { - persistenceContract: { - ready: false, - sequence: [ - { id: "write-do1-true", persisted: true, idsPresent: true }, - { id: "read-di1-true", persisted: false, idsPresent: true } - ] - }, - operationCount: 4 - }, - blockers: [ - { - type: "runtime_blocker", - scope: "m3-durable-postflight", - status: "open", - reason: "M3 DO1 true/false durable postflight did not prove persisted operation/audit/evidence for every step.", - impact: "A later rollout could appear ready while losing evidence persistence.", - safeNextAction: "Repair durable audit/evidence persistence, then rerun DEV runtime postflight.", - retryable: true - } - ], - safety: { - secretValuesPrinted: false, - readsKubernetesSecrets: false - } - }; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-postflight-blocked", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json") - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ commandLog, runtimePostflightReport }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - const report = JSON.parse(await readFile(stateReport(repoRoot, "dev-cd-apply.json"), "utf8")); - const blocker = report.blockers.find((item) => item.scope === "runtime-durable-postflight"); - assert.equal(code, 2); - assert.equal(summary.status, "blocked"); - assert.equal(summary.devCdApply, undefined); - assert.equal(output.length < 12000, true); - assert.equal(report.status, "blocked"); - assert.equal(report.devCdApply.steps.at(-1).id, "runtime-durable-postflight"); - assert.equal(report.devCdApply.steps.at(-1).status, "blocked"); - assert.ok(blocker); - assert.equal(blocker.reason, runtimePostflightReport.blockers[0].reason); - assert.equal(blocker.impact, runtimePostflightReport.blockers[0].impact); - assert.equal(blocker.safeNextAction, runtimePostflightReport.blockers[0].safeNextAction); - assert.equal(blocker.retryable, true); - assert.equal(blocker.evidence.blockerScope, "m3-durable-postflight"); - assert.equal(report.devCdApply.liveVerify.status, "not_run"); - assert.equal(JSON.stringify(report).includes("postgresql://"), false); - assert.equal(JSON.stringify(report).includes("SECRET"), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-runtime-postflight.mjs")), true); -}); - -test("runtime maintenance job reports are parsed after redacting secret-like log output", async () => { - const repoRoot = await makeRepo(); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-redaction", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json"), - "--full-output" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - runtimeJobLogSuffix: "debug postgresql://hwlab:SECRET_DB@db.example:5432/hwlab password=SECRET_PASSWORD token=SECRET_TOKEN\n" - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const report = JSON.parse(output); - const serialized = JSON.stringify(report); - assert.equal(code, 0); - assert.equal(report.status, "pass"); - assert.equal(serialized.includes("SECRET_DB"), false); - assert.equal(serialized.includes("SECRET_PASSWORD"), false); - assert.equal(serialized.includes("SECRET_TOKEN"), false); - assert.equal(report.devCdApply.steps.find((step) => step.id === "runtime-db-provisioning").stdoutJson.safety.secretValuesPrinted, false); - assert.equal(report.devCdApply.steps.find((step) => step.id === "runtime-db-migration").stdoutJson.safety.secretValuesPrinted, false); -}); - -test("transaction can apply an already published deploy-json promotion without republishing control HEAD", async () => { - const repoRoot = await makeRepo({ commitId: "abc1234" }); - const commandLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-promotion", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json"), - "--full-output" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - commandLog, - targetCommitId: laterControlCommit, - headCommitId: laterControlCommit, - extraGitRefs: { - abc1234: publishedCommit - }, - latestArtifactMergeCommit: artifactMergeCommit, - commitParents: { - [artifactMergeCommit]: [publishedCommit, artifactReportCommit], - [artifactReportCommit]: [publishedCommit] - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const report = JSON.parse(output); - assert.equal(code, 0); - assert.equal(report.status, "pass"); - assert.equal(report.devCdApply.target.promotionCommit, publishedCommit); - assert.equal(report.devCdApply.target.promotionSource, "deploy-json"); - assert.equal(report.devCdApply.target.publishRequired, false); - assert.equal(report.devCdApply.target.artifactBoundary.status, "pass"); - assert.equal(report.devCdApply.target.artifactBoundary.artifactMergeCommit.shortCommitId, "fedcba9"); - assert.deepEqual( - report.devCdApply.steps.map((step) => step.id), - [ - "desired-state-check-existing-promotion", - "artifact-catalog-validate", - "runtime-db-provisioning", - "runtime-db-migration", - "dev-deploy-apply", - "runtime-durable-postflight" - ] - ); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/refresh-artifact-catalog.mjs")), false); - assert.equal(commandLog.some((entry) => - entry.args.includes("scripts/deploy-desired-state-plan.mjs") && - entry.args.includes("--promotion-commit") && - entry.args.includes(publishedCommit) - ), true); - const applyCall = commandLog.find((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")); - assert.ok(applyCall?.env.HWLAB_CD_TRANSACTION_ID); - assert.equal(applyCall.env.HWLAB_CD_TRANSACTION_OWNER, "task-promotion"); - assert.equal(report.devCdApply.liveVerify.status, "pass"); - assert.equal(report.devCdApply.liveVerify.summary.commitMatches, 2); -}); - -test("apply stdout is concise by default while write-report keeps the full report", async () => { - const repoRoot = await makeRepo(); - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - "task-c", - "--kubeconfig", - "/etc/rancher/k3s/k3s.yaml", - "--report", stateReport(repoRoot, "dev-cd-apply.json") - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand(), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const summary = JSON.parse(output); - assert.equal(code, 0); - assert.equal(summary.status, "pass"); - assert.equal(summary.devCdApply, undefined); - assert.equal(summary.fullOutputHint.includes("--full-output"), true); - assert.equal(summary.lock.acquired, true); - assert.equal(summary.steps.some((step) => step.id === "dev-deploy-apply"), true); - - const writtenReport = JSON.parse(await readFile(stateReport(repoRoot, "dev-cd-apply.json"), "utf8")); - assert.equal(writtenReport.devCdApply.deployJson.before.manifest.environment, "dev"); -}); - -test("live verify summarizes 16666 and 16667 health identity", async () => { - const result = await verifyDevLive({ - now: () => new Date(iso()), - httpGetJson: makeHttpGetJson() - }, { expectedCommit: "abc1234" }); - - assert.equal(result.status, "pass"); - assert.equal(result.summary.checked, 2); - assert.deepEqual(result.endpoints.map((endpoint) => endpoint.url), [ - "http://74.48.78.17:16666/health/live", - "http://74.48.78.17:16667/health/live" - ]); - assert.equal(result.endpoints.every((endpoint) => endpoint.commitMatches), true); -}); - -test("artifact publish CLI rejects side effects without CI or transaction identity", async () => { - await assert.rejects( - execFileAsync(process.execPath, ["scripts/dev-artifact-publish.mjs", "--publish", "--no-report"], { - cwd: repoRoot, - env: { - HWLAB_CD_TRANSACTION_ID: "", - HWLAB_CI_ARTIFACT_RUN_ID: "" - } - }), - (error) => { - assert.equal(error.code, 2); - const payload = JSON.parse(error.stdout); - assert.equal(payload.error, "artifact-publish-identity-required"); - assert.equal(payload.script, "scripts/dev-artifact-publish.mjs"); - assert.equal(payload.cdLockRequired, false); - assert.equal(payload.rolloutAttempted, false); - assert.equal(payload.mutationAttempted, false); - return true; - } - ); -}); diff --git a/scripts/src/dev-cd-transaction-guard.mjs b/scripts/src/dev-cd-transaction-guard.mjs deleted file mode 100644 index ad3c276a..00000000 --- a/scripts/src/dev-cd-transaction-guard.mjs +++ /dev/null @@ -1,122 +0,0 @@ -import { tempReportPath } from "./report-paths.mjs"; - -const transactionEnvNames = [ - "HWLAB_CD_TRANSACTION_ID", - "HWLAB_CD_TRANSACTION_OWNER", - "HWLAB_CD_LOCK_NAME" -]; - -const ciArtifactIdentityEnvNames = [ - "HWLAB_CI_ARTIFACT_RUN_ID", - "GITHUB_RUN_ID", - "CI_PIPELINE_ID", - "BUILDKITE_BUILD_ID" -]; - -const artifactPublishIdentityEnvNames = [ - ...ciArtifactIdentityEnvNames, - ...transactionEnvNames -]; - -export function hasDevCdTransactionEnv(env = process.env) { - return typeof env.HWLAB_CD_TRANSACTION_ID === "string" && env.HWLAB_CD_TRANSACTION_ID.trim().length > 0; -} - -export function hasCiArtifactPublishEnv(env = process.env) { - return ciArtifactIdentityEnvNames.some((name) => typeof env[name] === "string" && env[name].trim().length > 0); -} - -export function hasDevCiArtifactEnv(env = process.env) { - return hasCiArtifactPublishEnv(env); -} - -export function hasDevArtifactSideEffectEnv(env = process.env) { - return hasDevCdTransactionEnv(env) || hasCiArtifactPublishEnv(env); -} - -export function devArtifactProducer(env = process.env) { - if (hasCiArtifactPublishEnv(env)) { - const idName = ciArtifactIdentityEnvNames.find((name) => typeof env[name] === "string" && env[name].trim().length > 0); - return { - kind: "ci-artifact", - runId: env[idName], - owner: env.HWLAB_CI_ARTIFACT_RUN_OWNER ?? idName ?? null - }; - } - if (hasDevCdTransactionEnv(env)) { - return { - kind: "cd-transaction-legacy", - runId: env.HWLAB_CD_TRANSACTION_ID, - owner: env.HWLAB_CD_TRANSACTION_OWNER ?? null - }; - } - return { - kind: "missing", - runId: null, - owner: null - }; -} - -export function devCdTransactionGuardFailure({ - script, - mode, - requiredEnv = "HWLAB_CD_TRANSACTION_ID" -}) { - return { - ok: false, - status: "blocked", - error: "cd-transaction-required", - code: "cd-transaction-required", - script, - mode, - requiredEnv, - acceptedEnv: transactionEnvNames, - entrypoint: `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report ${tempReportPath("dev-cd-apply.json")}`, - summary: `${script} ${mode} is a DEV CD side-effect step and must run inside scripts/dev-cd-apply.mjs.`, - devOnly: true, - prodTouched: false, - mutationAttempted: false - }; -} - -export function devArtifactPublishIdentityGuardFailure({ - script, - mode, - requiredEnv = "HWLAB_CI_ARTIFACT_RUN_ID or HWLAB_CD_TRANSACTION_ID" -}) { - return { - ok: false, - status: "blocked", - error: "artifact-publish-identity-required", - code: "artifact-publish-identity-required", - script, - mode, - requiredEnv, - acceptedEnv: artifactPublishIdentityEnvNames, - entrypoint: `HWLAB_CI_ARTIFACT_RUN_ID=<run-id> node scripts/artifact-publish.mjs --publish --report ${tempReportPath("dev-artifacts.json")}`, - summary: `${script} ${mode} is an artifact publish side-effect step and must run under CI artifact identity.`, - devOnly: true, - prodTouched: false, - cdLockRequired: false, - rolloutAttempted: false, - mutationAttempted: false - }; -} - -export function devArtifactSideEffectGuardFailure(options) { - return devArtifactPublishIdentityGuardFailure(options); -} - -export function requireDevCdTransactionForSideEffect({ env = process.env, script, mode }) { - if (hasDevCdTransactionEnv(env)) return null; - return devCdTransactionGuardFailure({ script, mode }); -} - -export function requireDevArtifactPublishIdentity({ env = process.env, script, mode }) { - if (hasCiArtifactPublishEnv(env) || hasDevCdTransactionEnv(env)) return null; - return devArtifactPublishIdentityGuardFailure({ script, mode }); -} - -export function requireDevArtifactSideEffectEnv(options) { - return requireDevArtifactPublishIdentity(options); -} diff --git a/scripts/src/dev-cloud-workbench-smoke-lib.mjs b/scripts/src/dev-cloud-workbench-smoke-lib.mjs index d0138fd7..ea63b09a 100644 --- a/scripts/src/dev-cloud-workbench-smoke-lib.mjs +++ b/scripts/src/dev-cloud-workbench-smoke-lib.mjs @@ -488,7 +488,7 @@ function runStaticSmoke() { const blockers = []; const files = readStaticFiles(); const source = Object.values(files).join("\n"); - const artifactPublisher = readText("scripts/dev-artifact-publish.mjs"); + const artifactPublisher = readText("scripts/artifact-publish.mjs"); const buildScript = readText("web/hwlab-cloud-web/scripts/build.mjs"); const distContractScript = readText("web/hwlab-cloud-web/scripts/dist-contract.mjs"); @@ -1745,7 +1745,7 @@ function readStaticFiles() { panel: readText("web/hwlab-cloud-web/workbench-hardware-panel.mjs"), codeAgentM3Evidence: readText("web/hwlab-cloud-web/code-agent-m3-evidence.mjs"), help: readText("web/hwlab-cloud-web/help.md"), - artifactPublisher: readText("scripts/dev-artifact-publish.mjs") + artifactPublisher: readText("scripts/artifact-publish.mjs") }; } diff --git a/scripts/src/dev-deploy-apply.mjs b/scripts/src/dev-deploy-apply.mjs deleted file mode 100755 index d6745a2a..00000000 --- a/scripts/src/dev-deploy-apply.mjs +++ /dev/null @@ -1,2795 +0,0 @@ -#!/usr/bin/env node -import { constants as fsConstants } from "node:fs"; -import { access, mkdir, readFile, writeFile } from "node:fs/promises"; -import { request as httpRequest } from "node:http"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; -import { spawn } from "node:child_process"; - -import { - DEV_DB_ENV_CONTRACT, - summarizeDbContract -} from "../../internal/cloud/db-contract.mjs"; -import { - DEV_CODE_AGENT_PROVIDER_CONTRACT, - inspectCodeAgentProviderManifestRefs, - inspectCodeAgentProviderWorkloadEnv -} from "../../internal/cloud/code-agent-contract.mjs"; -import { activeReportLifecycle } from "../../internal/dev-report-lifecycle.mjs"; -import { DEV_ENDPOINT, ENVIRONMENT_DEV, SERVICE_IDS } from "../../internal/protocol/index.mjs"; -import { requireDevCdTransactionForSideEffect } from "./dev-cd-transaction-guard.mjs"; -import { ensureNotRepoReportsPath, tempReportPath } from "./report-paths.mjs"; - -const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); -const defaultReportPath = tempReportPath("dev-deploy-report.json"); -const namespace = "hwlab-dev"; -const healthPath = "/health/live"; -const defaultD601KubeconfigPath = "/etc/rancher/k3s/k3s.yaml"; -const devKubeconfigEnvName = "HWLAB_DEV_KUBECONFIG"; -const kubeconfigEnvName = "KUBECONFIG"; -const registryManifestBaseUrlEnvName = "HWLAB_DEV_REGISTRY_MANIFEST_BASE_URL"; -const registryVerifyModeEnvName = "HWLAB_DEV_REGISTRY_VERIFY_MODE"; -const registryHostNetworkHelperImageEnvName = "HWLAB_DEV_REGISTRY_HOSTNET_HELPER_IMAGE"; -const defaultRegistryHostNetworkHelperImage = "unidesk-code-queue:d601"; -const devTemplateJobReplacementPolicy = { - status: "active", - namespace, - scope: "DEV-only suspended template Jobs", - allowedJobs: [ - { - name: "hwlab-agent-worker-template", - serviceId: "hwlab-agent-worker", - reason: "suspended session worker template Job has immutable spec.template" - }, - { - name: "hwlab-cli-template", - serviceId: "hwlab-cli", - reason: "suspended CLI template Job has immutable spec.template" - } - ] -}; -const replaceableDevTemplateJobNames = new Set(devTemplateJobReplacementPolicy.allowedJobs.map((job) => job.name)); -const devLegacySimulatorDeploymentCleanupPolicy = { - status: "active", - namespace, - scope: "DEV-only stale simulator Deployments replaced by indexed StatefulSets", - allowedDeployments: [ - { - name: "hwlab-gateway-simu", - serviceId: "hwlab-gateway-simu", - replacementKind: "StatefulSet", - reason: "M3 DEV requires exactly two indexed gateway-simu identities/sessions" - }, - { - name: "hwlab-box-simu", - serviceId: "hwlab-box-simu", - replacementKind: "StatefulSet", - reason: "M3 DEV requires exactly two indexed box-simu resource identities" - } - ] -}; -const cleanableLegacySimulatorDeploymentNames = new Set( - devLegacySimulatorDeploymentCleanupPolicy.allowedDeployments.map((deployment) => deployment.name) -); -const devUnmanagedHotfixDeploymentCleanupPolicy = { - status: "active", - namespace, - scope: "DEV-only unmanaged live Deployment hotfix/override fields", - reason: - "kubectl apply preserves fields owned by a live hotfix field manager; delete desired Deployments with unmanaged hotfix fields so apply recreates them from source truth", - matchPatterns: ["hotfix", "override"] -}; -const requiredValidationCommands = [ - "node --check scripts/validate-dev-gate-report.mjs", - "node scripts/validate-dev-gate-report.mjs", - "node --check scripts/dev-deploy-apply.mjs", - "node --check scripts/src/dev-deploy-apply.mjs", - "node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked" -]; -const applyCommand = `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --report ${tempReportPath("dev-cd-apply.json")}`; -const dryRunPlanCommand = `node scripts/dev-deploy-apply.mjs --dry-run --report ${defaultReportPath}`; -const blockedDryRunCommand = `node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --report ${defaultReportPath}`; -const forbiddenActions = [ - "prod-deploy", - "secret-read", - "runtime-substitute", - "unidesk-restart", - "code-queue-restart", - "backend-core-restart", - "heavy-master-e2e", - "force-push" -]; -const runtimeIdentityEnvNames = ["HWLAB_COMMIT_ID", "HWLAB_IMAGE", "HWLAB_IMAGE_TAG"]; -const digestPattern = /^sha256:[a-f0-9]{64}$/u; -const defaultCdConcurrency = parseBoundedInteger(process.env.HWLAB_DEV_CD_CONCURRENCY, 4, 1, 8); - -export function parseArgs(argv) { - const flags = new Set(); - const errors = []; - let kubeconfig = null; - let kubeconfigSpecified = false; - let registryManifestBaseUrl = null; - let reportPath = defaultReportPath; - let rolloutConcurrency = defaultCdConcurrency; - - for (let index = 0; index < argv.length; index += 1) { - const arg = argv[index]; - if (!arg.startsWith("--")) continue; - if (arg === "--kubeconfig") { - flags.add("--kubeconfig"); - kubeconfigSpecified = true; - const next = argv[index + 1]; - if (typeof next !== "string" || next.startsWith("--")) { - errors.push("--kubeconfig requires a non-empty path value"); - } else { - kubeconfig = next; - index += 1; - } - continue; - } - if (arg.startsWith("--kubeconfig=")) { - flags.add("--kubeconfig"); - kubeconfigSpecified = true; - kubeconfig = arg.slice("--kubeconfig=".length); - if (!kubeconfig.trim()) { - errors.push("--kubeconfig requires a non-empty path value"); - } - continue; - } - if (arg === "--report") { - flags.add("--report"); - const next = argv[index + 1]; - if (typeof next !== "string" || next.startsWith("--")) { - errors.push("--report requires a non-empty path value"); - } else { - reportPath = next; - flags.add("--report-output"); - index += 1; - } - continue; - } - if (arg.startsWith("--report=")) { - flags.add("--report"); - reportPath = arg.slice("--report=".length); - flags.add("--report-output"); - if (!reportPath.trim()) { - errors.push("--report requires a non-empty path value"); - } - continue; - } - if (arg === "--rollout-concurrency") { - flags.add("--rollout-concurrency"); - const next = argv[index + 1]; - if (typeof next !== "string" || next.startsWith("--")) { - errors.push("--rollout-concurrency requires a numeric value"); - } else { - rolloutConcurrency = parseBoundedInteger(next, defaultCdConcurrency, 1, 8); - index += 1; - } - continue; - } - if (arg.startsWith("--rollout-concurrency=")) { - flags.add("--rollout-concurrency"); - rolloutConcurrency = parseBoundedInteger(arg.slice("--rollout-concurrency=".length), defaultCdConcurrency, 1, 8); - continue; - } - if (arg === "--registry-manifest-base-url") { - flags.add("--registry-manifest-base-url"); - const next = argv[index + 1]; - if (typeof next !== "string" || next.startsWith("--")) { - errors.push("--registry-manifest-base-url requires a non-empty URL value"); - } else { - registryManifestBaseUrl = next; - index += 1; - } - continue; - } - if (arg.startsWith("--registry-manifest-base-url=")) { - flags.add("--registry-manifest-base-url"); - registryManifestBaseUrl = arg.slice("--registry-manifest-base-url=".length); - if (!registryManifestBaseUrl.trim()) { - errors.push("--registry-manifest-base-url requires a non-empty URL value"); - } - continue; - } - flags.add(arg); - } - - return { - apply: flags.has("--apply"), - dryRun: !flags.has("--apply"), - expectBlocked: flags.has("--expect-blocked"), - skipLiveProbe: flags.has("--skip-live-probe"), - writeReport: flags.has("--report-output"), - reportPath, - rolloutConcurrency, - registryManifestBaseUrl, - kubeconfig, - kubeconfigSpecified, - errors, - flags - }; -} - -function parseBoundedInteger(value, fallback, min, max) { - const parsed = Number.parseInt(String(value ?? ""), 10); - if (!Number.isInteger(parsed)) return fallback; - return Math.min(Math.max(parsed, min), max); -} - -function addBlocker(blockers, type, scope, summary) { - const key = `${type}::${scope}`; - if (!blockers.some((blocker) => `${blocker.type}::${blocker.scope}` === key)) { - blockers.push({ type, scope, status: "open", summary: oneLine(summary) }); - } -} - -function oneLine(value) { - return String(value).replace(/\s+/g, " ").trim(); -} - -async function mapWithConcurrency(items, concurrency, worker) { - if (items.length === 0) return []; - const results = new Array(items.length); - let nextIndex = 0; - const workerCount = Math.min(Math.max(concurrency, 1), items.length); - await Promise.all(Array.from({ length: workerCount }, async () => { - while (nextIndex < items.length) { - const index = nextIndex; - nextIndex += 1; - results[index] = await worker(items[index], index); - } - })); - return results; -} - -async function readJson(relativePath, blockers) { - try { - return JSON.parse(await readFile(path.join(repoRoot, relativePath), "utf8")); - } catch (error) { - addBlocker(blockers, "contract_blocker", relativePath, `Cannot parse ${relativePath}: ${error.message}`); - return null; - } -} - -async function readText(relativePath) { - return readFile(path.join(repoRoot, relativePath), "utf8"); -} - -async function commandResult(command, args, timeoutMs = 15000, options = {}) { - return new Promise((resolve) => { - const child = spawn(command, args, { - cwd: repoRoot, - env: options.env ?? process.env, - stdio: ["pipe", "pipe", "pipe"] - }); - const timer = setTimeout(() => child.kill("SIGTERM"), timeoutMs); - let stdout = ""; - let stderr = ""; - child.stdin.on("error", () => {}); - child.stdin.end(options.input ?? ""); - child.stdout.on("data", (chunk) => { - stdout += chunk; - }); - child.stderr.on("data", (chunk) => { - stderr += chunk; - }); - child.on("error", (error) => { - clearTimeout(timer); - resolve({ ok: false, code: null, stdout, stderr: error.message }); - }); - child.on("close", (code, signal) => { - clearTimeout(timer); - resolve({ ok: code === 0, code, signal, stdout: stdout.trim(), stderr: stderr.trim() }); - }); - }); -} - -async function findExecutable(name) { - for (const dir of (process.env.PATH || "").split(path.delimiter)) { - if (!dir) continue; - const candidate = path.join(dir, name); - try { - await access(candidate, fsConstants.X_OK); - return candidate; - } catch {} - } - return null; -} - -function isNonEmpty(value) { - return typeof value === "string" && value.trim().length > 0; -} - -export function resolveDevKubeconfigSelection({ flagValue = null, env = process.env } = {}) { - const candidates = [ - { source: "flag:--kubeconfig", value: flagValue }, - { source: `env:${devKubeconfigEnvName}`, value: env?.[devKubeconfigEnvName] }, - { source: `env:${kubeconfigEnvName}`, value: env?.[kubeconfigEnvName] }, - { source: "default:d601-k3s", value: defaultD601KubeconfigPath } - ]; - const selected = candidates.find((candidate) => isNonEmpty(candidate.value)); - const kubeconfig = selected.value.trim(); - return { - kubeconfig, - source: selected.source, - paths: kubeconfig.split(path.delimiter).filter((item) => item.length > 0) - }; -} - -function shellQuote(value) { - if (/^[A-Za-z0-9_@%+=:,./-]+$/u.test(value)) return value; - return `'${String(value).replaceAll("'", "'\\''")}'`; -} - -export function buildKubectlCommandPrefix(kubeconfig) { - return `KUBECONFIG=${shellQuote(kubeconfig)} kubectl`; -} - -export function formatKubeconfigAccessFailure(selection, unreadablePath) { - return `DEV kubeconfig from ${selection.source} is not readable at ${unreadablePath}; set --kubeconfig, ${devKubeconfigEnvName}, or ${kubeconfigEnvName} to a readable DEV kubeconfig path. Kubeconfig contents and Secret values are never printed.`; -} - -async function validateD601NativeKubeconfig(kubectlPath, env) { - const context = await commandResult(kubectlPath, ["config", "current-context"], 10_000, { env }); - if (!context.ok) { - return `DEV kubeconfig could not report a kubectl context: ${redactSensitiveText(context.stderr || context.stdout)}`; - } - const contextName = context.stdout.trim(); - if (/docker-desktop|desktop-control-plane/iu.test(contextName)) { - return `DEV kubeconfig resolved to forbidden context ${contextName}; D601 DEV must use native k3s at ${defaultD601KubeconfigPath}.`; - } - - const server = await commandResult(kubectlPath, ["config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"], 10_000, { env }); - if (server.ok && /127\.0\.0\.1:11700|docker-desktop/iu.test(server.stdout)) { - return "DEV kubeconfig resolved to Docker Desktop Kubernetes server 127.0.0.1:11700; use D601 native k3s kubeconfig instead."; - } - - const nodes = await commandResult(kubectlPath, ["get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], 15_000, { env }); - if (!nodes.ok) { - return `DEV kubeconfig could not read native k3s nodes: ${redactSensitiveText(nodes.stderr || nodes.stdout)}`; - } - const nodeNames = nodes.stdout.trim().split(/\s+/u).filter(Boolean); - if (!nodeNames.includes("d601")) { - return `DEV kubeconfig did not resolve to D601 native k3s; observed nodes=${nodeNames.join(",") || "none"}.`; - } - return null; -} - -export function redactSensitiveText(value) { - return String(value) - .replace(/\b(Bearer\s+)[A-Za-z0-9._~+/=-]+/giu, "$1<redacted>") - .replace(/\b(postgres(?:ql)?:\/\/[^:\s/@]+:)([^@\s]+)(@)/giu, "$1<redacted>$3") - .replace(/\b(token|password|client-key-data|client-certificate-data|certificate-authority-data)\b(\s*[:=]\s*)(["']?)([^"'\s,}]+)/giu, "$1$2$3<redacted>"); -} - -function redactCommandResult(result) { - return { - ...result, - redactedStdout: redactSensitiveText(result.stdout ?? ""), - redactedStderr: redactSensitiveText(result.stderr ?? "") - }; -} - -function commandOutput(result) { - return result.redactedStderr || result.redactedStdout || result.stderr || result.stdout || ""; -} - -async function resolveD601Kubectl(args = {}, env = process.env) { - const kubeconfigSelection = resolveDevKubeconfigSelection({ flagValue: args.kubeconfig, env }); - const kubectlPath = await findExecutable("kubectl"); - const commandPrefix = buildKubectlCommandPrefix(kubeconfigSelection.kubeconfig); - const base = { - executor: kubectlPath, - kubeconfig: kubeconfigSelection.kubeconfig, - kubeconfigSource: kubeconfigSelection.source, - kubeconfigPaths: kubeconfigSelection.paths, - commandPrefix, - env: { - ...env, - KUBECONFIG: kubeconfigSelection.kubeconfig - } - }; - - if (!kubectlPath) { - return { - ...base, - status: "blocked", - reason: "kubectl is not installed in the runner" - }; - } - - try { - let readablePathCount = 0; - let firstUnreadablePath = null; - for (const kubeconfigPath of kubeconfigSelection.paths) { - try { - await access(kubeconfigPath, fsConstants.R_OK); - readablePathCount += 1; - } catch (error) { - firstUnreadablePath ??= error?.path ?? kubeconfigPath; - } - } - if (readablePathCount === 0) { - throw Object.assign(new Error("no readable kubeconfig paths"), { - path: firstUnreadablePath ?? kubeconfigSelection.kubeconfig - }); - } - } catch (error) { - const unreadablePath = error?.path ?? kubeconfigSelection.kubeconfig; - return { - ...base, - status: "blocked", - reason: formatKubeconfigAccessFailure(kubeconfigSelection, unreadablePath) - }; - } - - const nativeK3sFailure = await validateD601NativeKubeconfig(kubectlPath, base.env); - if (nativeK3sFailure !== null) { - return { - ...base, - status: "blocked", - reason: nativeK3sFailure - }; - } - - return { - ...base, - status: "ready", - reason: "D601 k3s kubeconfig is readable" - }; -} - -function kubectlCommand(kubectl, args) { - return `${kubectl.commandPrefix} ${args.join(" ")}`; -} - -async function kubectlResult(kubectl, args, timeoutMs = 15000, options = {}) { - if (kubectl.status !== "ready") { - return { - ok: false, - code: null, - stdout: "", - stderr: kubectl.reason, - blocked: true - }; - } - const result = await commandResult(kubectl.executor, args, timeoutMs, { - ...options, - env: kubectl.env - }); - return redactCommandResult(result); -} - -function listItems(document) { - if (!document) return []; - return document.kind === "List" ? document.items ?? [] : [document]; -} - -function containersFor(item) { - return item?.spec?.template?.spec?.containers ?? []; -} - -function serviceIdFor(item, container) { - return ( - item?.metadata?.labels?.["hwlab.pikastech.local/service-id"] || - item?.spec?.template?.metadata?.labels?.["hwlab.pikastech.local/service-id"] || - container?.name - ); -} - -function itemNamespace(item) { - return item?.metadata?.namespace ?? (item?.kind === "Namespace" ? item?.metadata?.name : namespace); -} - -function uniqueStrings(values) { - return [...new Set(values.filter((value) => typeof value === "string" && value.length > 0))]; -} - -function imageRecordsFor(item) { - return containersFor(item).map((container) => ({ - name: container.name, - serviceId: serviceIdFor(item, container), - image: container.image - })); -} - -function primaryImage(images) { - if (!images.length) return null; - if (images.length === 1) return images[0].image ?? null; - return images.map((image) => `${image.name}=${image.image}`).join(", "); -} - -function imagesDiffer(oldImages, newImages) { - if (oldImages.length !== newImages.length) return true; - const oldByName = new Map(oldImages.map((image) => [image.name, image.image])); - return newImages.some((image) => oldByName.get(image.name) !== image.image); -} - -function isAllowedDesiredTemplateJob(item) { - return ( - item?.kind === "Job" && - itemNamespace(item) === namespace && - item?.spec?.suspend === true && - replaceableDevTemplateJobNames.has(item?.metadata?.name) - ); -} - -function isDesiredLegacySimulatorReplacement(item) { - return ( - item?.kind === "StatefulSet" && - itemNamespace(item) === namespace && - cleanableLegacySimulatorDeploymentNames.has(item?.metadata?.name) - ); -} - -function deploymentKey(item) { - return `${itemNamespace(item)}/${item?.metadata?.name ?? "unknown"}`; -} - -function isDesiredDeployment(item) { - return item?.kind === "Deployment" && itemNamespace(item) === namespace && typeof item?.metadata?.name === "string"; -} - -function objectTextContainsHotfixMarker(value) { - if (value === null || value === undefined) return false; - if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") { - return /\bhotfix\b|override/iu.test(String(value)); - } - if (Array.isArray(value)) return value.some((entry) => objectTextContainsHotfixMarker(entry)); - if (typeof value === "object") { - return Object.entries(value).some(([key, entry]) => - objectTextContainsHotfixMarker(key) || objectTextContainsHotfixMarker(entry) - ); - } - return false; -} - -function volumeIdentity(volume) { - const parts = [volume?.name ?? ""]; - if (volume?.configMap?.name) parts.push(`configMap:${volume.configMap.name}`); - if (volume?.secret?.secretName) parts.push(`secret:${volume.secret.secretName}`); - if (volume?.persistentVolumeClaim?.claimName) parts.push(`pvc:${volume.persistentVolumeClaim.claimName}`); - return parts.join("|"); -} - -function volumeMountIdentity(containerName, mount) { - return [ - containerName ?? "", - mount?.name ?? "", - mount?.mountPath ?? "", - mount?.subPath ?? "", - mount?.readOnly === true ? "ro" : "rw" - ].join("|"); -} - -function templateAnnotationIdentity(key, value) { - return `${key}=${String(value ?? "")}`; -} - -function desiredDeploymentIndex(workloads) { - return new Map( - listItems(workloads) - .filter(isDesiredDeployment) - .map((item) => [deploymentKey(item), item]) - ); -} - -function deploymentVolumes(item) { - return item?.spec?.template?.spec?.volumes ?? []; -} - -function deploymentContainerMounts(item) { - return (item?.spec?.template?.spec?.containers ?? []).flatMap((container) => - (container.volumeMounts ?? []).map((mount) => ({ - containerName: container.name, - mount - })) - ); -} - -function deploymentTemplateAnnotations(item) { - return item?.spec?.template?.metadata?.annotations ?? {}; -} - -export function decideDevTemplateJobReplacement({ - jobName, - jobNamespace = namespace, - desiredSuspended, - liveExists, - liveSuspended, - oldImages = [], - newImages = [] -}) { - const base = { - namespace: jobNamespace, - jobName, - allowed: replaceableDevTemplateJobNames.has(jobName), - desiredSuspended: desiredSuspended === true, - liveExists: liveExists === true, - liveSuspended: liveExists === true ? liveSuspended === true : null, - oldImage: primaryImage(oldImages), - newImage: primaryImage(newImages), - oldImages, - newImages, - replace: false, - result: "not_needed", - reason: "Live suspended template Job already matches the desired image." - }; - - if (!base.allowed) { - return { - ...base, - result: "ignored", - reason: "Job is not in the DEV template Job replacement allowlist." - }; - } - if (jobNamespace !== namespace) { - return { - ...base, - result: "blocked", - reason: `Replacement policy is DEV-only and refuses namespace ${jobNamespace}.` - }; - } - if (desiredSuspended !== true) { - return { - ...base, - result: "blocked", - reason: "Desired Job is not suspended, so it is not a template Job replacement candidate." - }; - } - if (liveExists !== true) { - return { - ...base, - result: "not_found", - reason: "Live Job is absent; kubectl apply can create it without replacement." - }; - } - if (liveSuspended !== true) { - return { - ...base, - result: "blocked", - reason: "Live Job is not suspended; refusing to delete a potentially active Job." - }; - } - if (imagesDiffer(oldImages, newImages)) { - return { - ...base, - replace: true, - result: "planned", - reason: "Live suspended template Job image differs from the desired manifest; delete/recreate is required because spec.template is immutable." - }; - } - return base; -} - -export function decideDevLegacySimulatorDeploymentCleanup({ - deploymentName, - deploymentNamespace = namespace, - desiredReplacementExists, - desiredReplacementKind, - liveExists, - oldImages = [], - newImages = [] -}) { - const policy = devLegacySimulatorDeploymentCleanupPolicy.allowedDeployments.find( - (deployment) => deployment.name === deploymentName - ); - const base = { - namespace: deploymentNamespace, - deploymentName, - serviceId: policy?.serviceId ?? deploymentName, - allowed: Boolean(policy), - desiredReplacementExists: desiredReplacementExists === true, - desiredReplacementKind: desiredReplacementKind ?? null, - liveExists: liveExists === true, - oldImage: primaryImage(oldImages), - newImage: primaryImage(newImages), - oldImages, - newImages, - cleanup: false, - result: "not_needed", - reason: "No stale legacy simulator Deployment exists." - }; - - if (!policy) { - return { - ...base, - result: "ignored", - reason: "Deployment is not in the DEV stale simulator cleanup allowlist." - }; - } - if (deploymentNamespace !== namespace) { - return { - ...base, - result: "blocked", - reason: `Cleanup policy is DEV-only and refuses namespace ${deploymentNamespace}.` - }; - } - if (desiredReplacementExists !== true || desiredReplacementKind !== policy.replacementKind) { - return { - ...base, - result: "blocked", - reason: `Refusing to clean up ${deploymentName} without desired ${policy.replacementKind} replacement in source manifests.` - }; - } - if (liveExists !== true) { - return { - ...base, - result: "not_found", - reason: "Live legacy Deployment is absent; no cleanup is needed." - }; - } - return { - ...base, - cleanup: true, - result: "planned", - reason: `Live legacy Deployment ${deploymentName} duplicates the desired indexed StatefulSet and must be removed for exact M3 cardinality.` - }; -} - -export function decideDevUnmanagedHotfixDeploymentCleanup({ - deploymentName, - deploymentNamespace = namespace, - desiredDeployment, - liveDeployment -}) { - const base = { - namespace: deploymentNamespace, - deploymentName, - desiredExists: Boolean(desiredDeployment), - liveExists: Boolean(liveDeployment), - cleanup: false, - result: "not_needed", - reason: "Live Deployment has no unmanaged hotfix or override fields outside desired source.", - unmanagedVolumes: [], - unmanagedVolumeMounts: [], - unmanagedTemplateAnnotations: [] - }; - - if (deploymentNamespace !== namespace) { - return { - ...base, - result: "blocked", - reason: `Unmanaged hotfix cleanup is DEV-only and refuses namespace ${deploymentNamespace}.` - }; - } - if (!desiredDeployment) { - return { - ...base, - result: "ignored", - reason: "No desired Deployment exists in source; cleanup must not delete unknown live Deployments." - }; - } - if (!liveDeployment) { - return { - ...base, - result: "not_found", - reason: "Live Deployment is absent; kubectl apply can create it from desired source." - }; - } - - const desiredVolumeIdentities = new Set(deploymentVolumes(desiredDeployment).map(volumeIdentity)); - const desiredMountIdentities = new Set( - deploymentContainerMounts(desiredDeployment).map(({ containerName, mount }) => volumeMountIdentity(containerName, mount)) - ); - const desiredAnnotationIdentities = new Set( - Object.entries(deploymentTemplateAnnotations(desiredDeployment)).map(([key, value]) => - templateAnnotationIdentity(key, value) - ) - ); - const unmanagedVolumes = deploymentVolumes(liveDeployment) - .filter((volume) => !desiredVolumeIdentities.has(volumeIdentity(volume)) && objectTextContainsHotfixMarker(volume)) - .map((volume) => ({ - name: volume.name ?? null, - configMap: volume.configMap?.name ?? null, - secret: volume.secret?.secretName ?? null, - identity: volumeIdentity(volume) - })); - const unmanagedVolumeMounts = deploymentContainerMounts(liveDeployment) - .filter(({ containerName, mount }) => - !desiredMountIdentities.has(volumeMountIdentity(containerName, mount)) && - objectTextContainsHotfixMarker({ containerName, mount }) - ) - .map(({ containerName, mount }) => ({ - containerName, - name: mount.name ?? null, - mountPath: mount.mountPath ?? null, - subPath: mount.subPath ?? null, - identity: volumeMountIdentity(containerName, mount) - })); - const unmanagedTemplateAnnotations = Object.entries(deploymentTemplateAnnotations(liveDeployment)) - .filter(([key, value]) => - !desiredAnnotationIdentities.has(templateAnnotationIdentity(key, value)) && - objectTextContainsHotfixMarker({ key, value }) - ) - .map(([key, value]) => ({ key, value: String(value ?? "") })); - - if (unmanagedVolumes.length === 0 && unmanagedVolumeMounts.length === 0 && unmanagedTemplateAnnotations.length === 0) { - return base; - } - - return { - ...base, - cleanup: true, - result: "planned", - reason: - "Live Deployment contains unmanaged hotfix/override fields that kubectl apply may preserve; delete before apply so desired source recreates it exactly.", - unmanagedVolumes, - unmanagedVolumeMounts, - unmanagedTemplateAnnotations - }; -} - -function replicaPlan(item) { - if (item?.kind === "Job") { - return item?.spec?.suspend === true ? 0 : 1; - } - return item?.spec?.replicas ?? null; -} - -function buildWorkloadPlan(workloads) { - return listItems(workloads).map((item) => { - const containers = containersFor(item).map((container) => ({ - name: container.name, - serviceId: serviceIdFor(item, container), - image: container.image - })); - return { - kind: item?.kind ?? "unknown", - name: item?.metadata?.name ?? "unknown", - namespace: item?.metadata?.namespace ?? namespace, - replicas: replicaPlan(item), - serviceIds: uniqueStrings(containers.map((container) => container.serviceId)), - containers - }; - }); -} - -function buildServicePlan(services) { - return listItems(services).map((item) => ({ - kind: item?.kind ?? "unknown", - name: item?.metadata?.name ?? "unknown", - namespace: item?.metadata?.namespace ?? namespace, - serviceId: item?.metadata?.labels?.["hwlab.pikastech.local/service-id"] ?? item?.metadata?.name ?? "unknown", - type: item?.spec?.type ?? "unknown", - ports: (item?.spec?.ports ?? []).map((port) => ({ - name: port.name, - port: port.port, - targetPort: port.targetPort - })) - })); -} - -function buildArtifactPlan(deploy, catalog, sourceCommitId, artifactEvidence) { - const requiredServices = (catalog?.services ?? []).filter((service) => service.artifactRequired !== false); - const serviceCommitIds = uniqueStrings([ - ...(catalog?.services ?? []).map((service) => service.commitId), - ...(artifactEvidence ?? []).map((service) => service.catalogCommitId) - ]); - const expectedArtifactCommit = deploy?.commitId ?? catalog?.commitId ?? "unknown"; - return { - expectedArtifactCommit, - deployCommitId: deploy?.commitId ?? "unknown", - catalogCommitId: catalog?.commitId ?? "unknown", - sourceCommitId, - serviceCommitIds, - matchesSourceCommit: - deploy?.commitId && catalog?.commitId - ? commitMatchesSource(deploy.commitId, sourceCommitId) && commitMatchesSource(catalog.commitId, sourceCommitId) - : false, - published: catalog?.publish?.ciPublished === true, - registryVerified: catalog?.publish?.registryVerified === true, - imageCount: artifactEvidence.length, - requiredServiceCount: requiredServices.length, - disabledServiceCount: (catalog?.services ?? []).filter((service) => service.artifactRequired === false).length, - unpublishedServices: requiredServices - .filter((service) => service.publishState !== "published" || !service.digest?.startsWith("sha256:")) - .map((service) => service.serviceId) - }; -} - -function artifactIdentityForService(deploy, catalog, serviceId) { - const deployService = deploy?.services?.find((service) => service.serviceId === serviceId) ?? null; - const catalogService = catalog?.services?.find((service) => service.serviceId === serviceId) ?? null; - return { - serviceId, - sourceCommitId: catalogService?.commitId ?? deploy?.commitId ?? "unknown", - image: deployService?.image ?? catalogService?.image ?? null, - imageTag: catalogService?.imageTag ?? parseImageTag(deployService?.image), - digest: catalogService?.digest ?? "not_published", - repositoryDigest: catalogService?.repositoryDigest ?? null, - publishState: catalogService?.publishState ?? "unknown" - }; -} - -function parseImageTag(image) { - if (typeof image !== "string") return null; - const slashIndex = image.lastIndexOf("/"); - const colonIndex = image.lastIndexOf(":"); - if (colonIndex <= slashIndex) return null; - return image.slice(colonIndex + 1); -} - -function parseTaggedImageReference(image) { - if (typeof image !== "string" || image.includes("@")) return null; - const firstSlash = image.indexOf("/"); - const lastSlash = image.lastIndexOf("/"); - const colonIndex = image.lastIndexOf(":"); - if (firstSlash <= 0 || colonIndex <= lastSlash) return null; - return { - image, - registry: image.slice(0, firstSlash), - repository: image.slice(firstSlash + 1, colonIndex), - tag: image.slice(colonIndex + 1) - }; -} - -function registryManifestBaseUrlForParsed(parsed, options = {}) { - const env = options.env ?? process.env; - const explicitBaseUrl = options.registryManifestBaseUrl ?? env?.[registryManifestBaseUrlEnvName] ?? null; - if (typeof explicitBaseUrl === "string" && explicitBaseUrl.trim().length > 0) { - return { - baseUrl: explicitBaseUrl.trim().replace(/\/+$/u, ""), - source: options.registryManifestBaseUrl ? "flag:--registry-manifest-base-url" : `env:${registryManifestBaseUrlEnvName}` - }; - } - return { - baseUrl: `http://${parsed.registry}`, - source: "image-registry" - }; -} - -export function registryManifestUrlForImage(image, options = {}) { - const parsed = parseTaggedImageReference(image); - if (!parsed) return null; - const repositoryPath = parsed.repository.split("/").map((part) => encodeURIComponent(part)).join("/"); - const manifestBase = registryManifestBaseUrlForParsed(parsed, options); - return { - ...parsed, - url: `${manifestBase.baseUrl}/v2/${repositoryPath}/manifests/${encodeURIComponent(parsed.tag)}`, - registryAccess: { - baseUrl: manifestBase.baseUrl, - source: manifestBase.source - } - }; -} - -function registryAcceptHeader() { - return [ - "application/vnd.docker.distribution.manifest.v2+json", - "application/vnd.oci.image.manifest.v1+json", - "application/vnd.docker.distribution.manifest.list.v2+json", - "application/vnd.oci.image.index.v1+json", - "application/vnd.docker.distribution.manifest.v1+json" - ].join(", "); -} - -function httpRequestText(url, { method = "GET", headers = {}, timeoutMs = 15000 } = {}) { - return new Promise((resolve, reject) => { - const request = httpRequest(url, { method, headers, timeout: timeoutMs }, (response) => { - response.setEncoding("utf8"); - let body = ""; - response.on("data", (chunk) => { - body += chunk; - if (body.length > 4096) { - request.destroy(new Error("registry manifest response exceeded 4096 bytes during CD verification")); - } - }); - response.on("end", () => resolve({ - statusCode: response.statusCode ?? 0, - headers: response.headers, - body - })); - }); - request.on("timeout", () => { - request.destroy(new Error(`timeout after ${timeoutMs}ms`)); - }); - request.on("error", reject); - request.end(); - }); -} - -function registryVerifyMode(options = {}) { - const env = options.env ?? process.env; - const value = String(options.registryVerifyMode ?? env?.[registryVerifyModeEnvName] ?? "auto").trim(); - if (["auto", "direct", "docker-host-network"].includes(value)) return value; - return "auto"; -} - -function dockerHostNetworkHelperImage(options = {}) { - const env = options.env ?? process.env; - return String( - options.registryHostNetworkHelperImage ?? - env?.[registryHostNetworkHelperImageEnvName] ?? - defaultRegistryHostNetworkHelperImage - ).trim(); -} - -async function readRegistryManifestViaDockerHostNetwork(parsed, options = {}) { - const runCommand = options.runCommand ?? commandResult; - const helperImage = dockerHostNetworkHelperImage(options); - const helperScript = ` -const http = require("node:http"); -const url = process.argv[1]; -const accept = process.argv[2]; -const req = http.request(url, { - method: "HEAD", - headers: { Accept: accept }, - timeout: 15000 -}, (response) => { - const digest = response.headers["docker-content-digest"] || null; - response.resume(); - response.on("end", () => { - console.log(JSON.stringify({ statusCode: response.statusCode || 0, headers: { "docker-content-digest": digest } })); - }); -}); -req.on("timeout", () => req.destroy(new Error("timeout after 15000ms"))); -req.on("error", (error) => { - console.error(error.message); - process.exitCode = 1; -}); -req.end(); -`.trim(); - const args = [ - "run", - "--rm", - "--pull=never", - "--network", - "host", - helperImage, - "node", - "-e", - helperScript, - parsed.url, - registryAcceptHeader() - ]; - const result = await runCommand("docker", args, 60000); - if (!result.ok) { - throw new Error( - `docker host-network registry manifest read failed with ${result.code ?? "unknown"}: ${oneLine(result.stderr || result.stdout || "no output")}` - ); - } - const parsedOutput = JSON.parse(result.stdout || "{}"); - return { - statusCode: Number.parseInt(String(parsedOutput.statusCode ?? 0), 10), - headers: parsedOutput.headers ?? {}, - body: "", - registryAccess: { - mode: "docker-host-network", - helperImage, - directUrl: parsed.url, - directAccessSource: parsed.registryAccess?.source ?? null - } - }; -} - -async function readRegistryManifest(parsed, options = {}) { - const mode = registryVerifyMode(options); - const httpReader = options.httpRequestText ?? httpRequestText; - if (mode === "docker-host-network") { - return readRegistryManifestViaDockerHostNetwork(parsed, options); - } - try { - const response = await httpReader(parsed.url, { - method: "HEAD", - headers: { - Accept: registryAcceptHeader() - }, - timeoutMs: 15000 - }); - return { - ...response, - registryAccess: { - mode: "direct", - url: parsed.url, - source: parsed.registryAccess?.source ?? null - } - }; - } catch (error) { - if (mode === "direct") throw error; - const fallback = await readRegistryManifestViaDockerHostNetwork(parsed, options); - return { - ...fallback, - registryAccess: { - ...fallback.registryAccess, - directError: oneLine(error.message) - } - }; - } -} - -export async function verifyCatalogRegistryManifest(service, blockers, options = {}) { - const image = service.image; - const expectedDigest = service.digest; - const parsed = registryManifestUrlForImage(image, options); - const base = { - serviceId: service.serviceId, - image, - imageTag: service.imageTag ?? parseImageTag(image), - expectedDigest, - url: parsed?.url ?? null, - registryAccess: parsed?.registryAccess ?? null - }; - - if (!parsed) { - const reason = `${service.serviceId} image is not a tagged registry image`; - addBlocker(blockers, "contract_blocker", `registry-manifest-${service.serviceId}`, reason); - return { ...base, status: "blocked", reason }; - } - if (!digestPattern.test(expectedDigest ?? "")) { - const reason = `${service.serviceId} catalog digest is not an immutable sha256 digest`; - addBlocker(blockers, "contract_blocker", `registry-manifest-${service.serviceId}`, reason); - return { ...base, status: "blocked", reason }; - } - - try { - const response = await readRegistryManifest(parsed, options); - const observedDigest = String(response.headers["docker-content-digest"] ?? "").trim(); - const okStatus = response.statusCode >= 200 && response.statusCode < 300; - const digestMatches = observedDigest === expectedDigest; - if (!okStatus || !digestMatches) { - const reason = !okStatus - ? `${service.serviceId} registry manifest returned HTTP ${response.statusCode}` - : `${service.serviceId} registry digest mismatch: expected ${expectedDigest}, observed ${observedDigest || "missing"}`; - addBlocker(blockers, "environment_blocker", `registry-manifest-${service.serviceId}`, reason); - return { - ...base, - status: "blocked", - reason, - httpStatus: response.statusCode, - observedDigest, - digestMatches, - registryAccess: response.registryAccess ?? base.registryAccess - }; - } - return { - ...base, - status: "pass", - httpStatus: response.statusCode, - observedDigest, - digestMatches: true, - registryAccess: response.registryAccess ?? base.registryAccess - }; - } catch (error) { - const reason = `${service.serviceId} registry manifest read failed: ${oneLine(error.message)}`; - addBlocker(blockers, "environment_blocker", `registry-manifest-${service.serviceId}`, reason); - return { - ...base, - status: "blocked", - reason - }; - } -} - -async function verifyCatalogRegistryManifests(catalog, blockers, concurrency = defaultCdConcurrency, options = {}) { - const services = (catalog?.services ?? []).filter((service) => service.artifactRequired !== false); - const results = await mapWithConcurrency(services, concurrency, async (service) => - verifyCatalogRegistryManifest(service, blockers, options) - ); - return { - status: results.every((result) => result.status === "pass") ? "pass" : "blocked", - source: "registry-manifest", - releaseGate: true, - serviceCount: results.length, - concurrency, - verifyMode: registryVerifyMode(options), - results - }; -} - -function expectedRuntimeIdentityEnv(artifact) { - return { - HWLAB_COMMIT_ID: artifact.sourceCommitId, - HWLAB_IMAGE: artifact.image, - HWLAB_IMAGE_TAG: artifact.imageTag - }; -} - -function envObjectFromEnvList(envList) { - const env = {}; - if (!Array.isArray(envList)) return env; - for (const entry of envList) { - if (!entry?.name) continue; - env[entry.name] = Object.hasOwn(entry, "value") ? entry.value : null; - } - return env; -} - -function runtimeIdentityValueMatches(name, actual, expected) { - if (typeof actual !== "string" || typeof expected !== "string") return false; - if (name === "HWLAB_COMMIT_ID") return commitMatchesSource(actual, expected); - return actual === expected; -} - -export function compareRuntimeIdentityEnv(env, expected) { - const fields = runtimeIdentityEnvNames.map((name) => { - const actual = Object.hasOwn(env ?? {}, name) ? env[name] : null; - const expectedValue = expected?.[name] ?? null; - const status = runtimeIdentityValueMatches(name, actual, expectedValue) ? "match" : "drift"; - return { - name, - expected: expectedValue, - actual, - status - }; - }); - return { - status: fields.every((field) => field.status === "match") ? "match" : "drift", - matchesDesired: fields.every((field) => field.status === "match"), - fields, - drift: fields.filter((field) => field.status === "drift") - }; -} - -function formatRuntimeIdentityDrift(drift) { - return drift.map((field) => `${field.name} expected ${field.expected ?? "missing"} actual ${field.actual ?? "missing"}`).join("; "); -} - -function deploymentRevision(deployment) { - return deployment?.metadata?.annotations?.["deployment.kubernetes.io/revision"] ?? null; -} - -function containerImageForDeployment(deployment, containerName) { - const containers = deployment?.spec?.template?.spec?.containers ?? []; - return containers.find((container) => container.name === containerName)?.image ?? null; -} - -function buildRolloutVerificationCommand(kubectl, serviceId) { - return kubectlCommand(kubectl, ["-n", namespace, "rollout", "status", `deployment/${serviceId}`, "--timeout=180s"]); -} - -function buildDeploymentRolloutBase(kubectl, deploy, catalog, serviceId, observationPhase) { - const artifact = artifactIdentityForService(deploy, catalog, serviceId); - const commandArgs = ["-n", namespace, "get", "deployment", serviceId, "-o", "json"]; - return { - serviceId, - namespace, - status: "not_evaluated", - observationPhase, - sourceCommitId: artifact.sourceCommitId, - image: artifact.image ?? "unknown", - imageTag: artifact.imageTag ?? "unknown", - digest: artifact.digest, - repositoryDigest: artifact.repositoryDigest, - publishState: artifact.publishState, - rolloutRevision: null, - liveImage: null, - imageMatchesDesired: false, - expectedRuntimeEnv: expectedRuntimeIdentityEnv(artifact), - liveRuntimeEnv: null, - runtimeEnvMatchesDesired: false, - runtimeEnvDrift: [], - verificationCommand: buildRolloutVerificationCommand(kubectl, serviceId), - readCommand: kubectlCommand(kubectl, commandArgs), - kubeconfig: kubectl.kubeconfig, - kubeconfigSource: kubectl.kubeconfigSource - }; -} - -function skippedDeploymentRolloutObservation(kubectl, deploy, catalog, serviceId, observationPhase, reason) { - return { - ...buildDeploymentRolloutBase(kubectl, deploy, catalog, serviceId, observationPhase), - status: "not_evaluated", - blocker: reason - }; -} - -async function observeDeploymentRollout(kubectl, deploy, catalog, serviceId, blockers, options = {}) { - const observationPhase = options.observationPhase ?? "read"; - const blockRuntimeEnvDrift = options.blockRuntimeEnvDrift === true; - const commandArgs = ["-n", namespace, "get", "deployment", serviceId, "-o", "json"]; - const base = buildDeploymentRolloutBase(kubectl, deploy, catalog, serviceId, observationPhase); - - if (kubectl.status !== "ready") { - addBlocker(blockers, "environment_blocker", `rollout-read-${serviceId}`, kubectl.reason); - return { - ...base, - status: "blocked", - blocker: kubectl.reason - }; - } - - const result = await kubectlResult(kubectl, commandArgs, 15000); - if (!result.ok) { - const summary = oneLine(commandOutput(result)); - addBlocker(blockers, "environment_blocker", `rollout-read-${serviceId}`, `Cannot read DEV Deployment ${serviceId} rollout revision: ${summary}`); - return { - ...base, - status: "blocked", - blocker: summary - }; - } - - try { - const deployment = JSON.parse(result.stdout); - const liveImage = containerImageForDeployment(deployment, serviceId); - const liveRuntimeEnv = compareRuntimeIdentityEnv( - envObjectFromEnvList(deployment?.spec?.template?.spec?.containers?.find((container) => container.name === serviceId)?.env), - base.expectedRuntimeEnv - ); - if (blockRuntimeEnvDrift && !liveRuntimeEnv.matchesDesired) { - addBlocker( - blockers, - "runtime_blocker", - `rollout-runtime-env-${observationPhase}-${serviceId}`, - `DEV Deployment ${serviceId} runtime identity env drift after apply: ${formatRuntimeIdentityDrift(liveRuntimeEnv.drift)}` - ); - } - return { - ...base, - status: "observed", - rolloutRevision: deploymentRevision(deployment), - liveImage, - imageMatchesDesired: Boolean(base.image && liveImage === base.image), - liveRuntimeEnv, - runtimeEnvMatchesDesired: liveRuntimeEnv.matchesDesired, - runtimeEnvDrift: liveRuntimeEnv.drift - }; - } catch (error) { - const summary = oneLine(error.message); - addBlocker(blockers, "contract_blocker", `rollout-read-${serviceId}`, `Cannot parse DEV Deployment ${serviceId} rollout revision: ${summary}`); - return { - ...base, - status: "blocked", - blocker: summary - }; - } -} - -function blockerHint(blocker) { - if (blocker.scope === "artifact-publish") { - return "Publish DEV images from the intended commit and update the catalog with registry digests and verification evidence."; - } - if (blocker.scope === "artifact-source-commit") { - return "Regenerate deploy/deploy.json and deploy/artifact-catalog.dev.json for the source commit that will be promoted."; - } - if (blocker.scope === "kubectl" || blocker.scope === "d601-kubeconfig" || blocker.scope === "kubectl-version") { - return `Provide kubectl with a readable DEV kubeconfig through --kubeconfig, ${devKubeconfigEnvName}, ${kubeconfigEnvName}, or the D601 default path without exposing token or secret values.`; - } - if (blocker.scope === "cluster-read") { - return "Restore read-only visibility for pods, services, configmaps, deployments, and jobs in hwlab-dev."; - } - if (blocker.scope.startsWith("rollout-read-")) { - return "Restore read-only Deployment visibility in hwlab-dev so rollout revision, image tag, digest, and source commit can be reported after apply."; - } - if (blocker.scope.startsWith("code-agent-provider-env-read-")) { - return "Restore read-only Deployment visibility for hwlab-cloud-api so provider env names and secretKeyRef metadata can be verified without reading Secret values."; - } - if (blocker.scope.startsWith("code-agent-provider-env-preservation-") || blocker.scope === "code-agent-provider-desired-state") { - return "Repair deploy/deploy.json and deploy/k8s/base/workloads.yaml so hwlab-cloud-api preserves OPENAI_API_KEY from hwlab-code-agent-provider/openai-api-key and the DEV egress proxy base URL."; - } - if (blocker.scope.startsWith("deploy-runtime-env-") || blocker.scope.startsWith("k8s-runtime-env-")) { - return "Update deploy/deploy.json and deploy/k8s/base/workloads.yaml so the service owns HWLAB_COMMIT_ID, HWLAB_IMAGE, and HWLAB_IMAGE_TAG."; - } - if (blocker.scope.startsWith("rollout-runtime-env-")) { - return "Reconcile the DEV Deployment runtime identity env to the desired source image and rerun post-apply read-only verification."; - } - if (blocker.scope === "dev-health" || blocker.scope === "dev-health-identity") { - return "Restore the public DEV health path and confirm it identifies HWLAB dev, not a substitute runtime."; - } - if (blocker.scope === "dev-health-db" || blocker.scope === "cloud-api-db" || blocker.scope === "cloud-api-db-config") { - return "Configure and verify the DEV cloud-api DB Secret URL host readiness through health output, without reading secret values."; - } - return blocker.summary; -} - -function buildRemainingBlockers(blockers) { - return blockers.map((blocker) => ({ - type: blocker.type, - scope: blocker.scope, - status: blocker.status, - summary: blocker.summary, - unblockHint: oneLine(blockerHint(blocker)) - })); -} - -function buildRollbackHint(kubectl, workloads) { - const deploymentNames = buildWorkloadPlan(workloads) - .filter((workload) => workload.kind === "Deployment") - .map((workload) => workload.name); - const jobNames = buildWorkloadPlan(workloads) - .filter((workload) => workload.kind === "Job") - .map((workload) => workload.name); - - return { - namespace, - strategy: "DEV-only Kubernetes rollback using deployment revision history; do not touch PROD or UniDesk services.", - captureBeforeApply: [ - kubectlCommand(kubectl, ["-n", namespace, "get", "pods,services,configmaps,deployments,jobs", "-o", "wide"]), - kubectlCommand(kubectl, ["-n", namespace, "get", "deployments", "-o", "json"]) - ], - deploymentRollbackCommands: deploymentNames.map((name) => kubectlCommand(kubectl, ["-n", namespace, "rollout", "undo", `deployment/${name}`])), - jobCleanupCommands: jobNames.map((name) => kubectlCommand(kubectl, ["-n", namespace, "delete", "job", name, "--ignore-not-found"])), - postRollbackChecks: [ - kubectlCommand(kubectl, ["-n", namespace, "rollout", "status", "deployment/hwlab-cloud-api", "--timeout=120s"]), - `curl -fsS ${new URL(healthPath, DEV_ENDPOINT).href}` - ] - }; -} - -function workloadEnvByServiceId(workloads, serviceId) { - for (const item of listItems(workloads)) { - for (const container of containersFor(item)) { - if (serviceIdFor(item, container) !== serviceId) continue; - const env = new Map((container.env ?? []).filter((entry) => entry?.name).map((entry) => [entry.name, entry])); - env.__volumeMounts = Array.isArray(container.volumeMounts) ? container.volumeMounts : []; - env.__volumes = Array.isArray(item?.spec?.template?.spec?.volumes) ? item.spec.template.spec.volumes : []; - return env; - } - } - return new Map(); -} - -export function inspectCodeAgentProviderDesiredState(deploy, workloads) { - const deployEnv = deploy?.services?.find((service) => service.serviceId === "hwlab-cloud-api")?.env ?? {}; - return inspectCodeAgentProviderManifestRefs({ - deployEnv, - workloadEnv: workloadEnvByServiceId(workloads, "hwlab-cloud-api") - }); -} - -export function inspectCodeAgentProviderLiveDeployment(deployment) { - const container = containerForDeployment(deployment, "hwlab-cloud-api"); - const workloadEnv = new Map((container?.env ?? []).filter((entry) => entry?.name).map((entry) => [entry.name, entry])); - workloadEnv.__volumeMounts = Array.isArray(container?.volumeMounts) ? container.volumeMounts : []; - workloadEnv.__volumes = Array.isArray(deployment?.spec?.template?.spec?.volumes) ? deployment.spec.template.spec.volumes : []; - const inspection = inspectCodeAgentProviderWorkloadEnv(workloadEnv); - return { - ...inspection, - namespace: deployment?.metadata?.namespace ?? namespace, - deployment: deployment?.metadata?.name ?? "hwlab-cloud-api", - container: container?.name ?? "hwlab-cloud-api", - validationScope: "deployment-env-and-secretKeyRef-metadata-only", - secretValuesRead: false, - kubernetesSecretDataRead: false - }; -} - -function containerForDeployment(deployment, containerName) { - return (deployment?.spec?.template?.spec?.containers ?? []).find((container) => container.name === containerName) ?? null; -} - -function codeAgentProviderBlockerSummary(inspection) { - const missing = [ - ...(inspection.missingDeployEnv ?? []), - ...(inspection.missingWorkloadEnv ?? inspection.missingEnv ?? []), - ...(inspection.deployMismatches ?? []).map((entry) => entry.name), - ...(inspection.workloadMismatches ?? inspection.mismatchedEnv ?? []).map((entry) => entry.name), - ...(inspection.missingSecretRefs ?? []), - ...(inspection.missingEgressContract ?? []) - ]; - const detail = uniqueStrings(missing).join(", "); - return detail - ? `Code Agent provider env preservation is incomplete; missing or mismatched ${detail}` - : "Code Agent provider env preservation is incomplete"; -} - -function addCodeAgentProviderDesiredStateBlocker(blockers, desiredState) { - if (desiredState.ready) return; - addBlocker( - blockers, - "agent_blocker", - "code-agent-provider-desired-state", - codeAgentProviderBlockerSummary(desiredState) - ); -} - -async function observeCodeAgentProviderLiveDeployment(kubectl, blockers, { phase, blockOnMismatch }) { - const commandArgs = ["-n", namespace, "get", "deployment", "hwlab-cloud-api", "-o", "json"]; - const base = { - phase, - namespace, - serviceId: "hwlab-cloud-api", - status: "not_evaluated", - readCommand: kubectlCommand(kubectl, commandArgs), - kubeconfig: kubectl.kubeconfig, - expectedSecretRef: `${DEV_CODE_AGENT_PROVIDER_CONTRACT.secretRefs[0].secretName}/${DEV_CODE_AGENT_PROVIDER_CONTRACT.secretRefs[0].secretKey}`, - expectedEgressTarget: DEV_CODE_AGENT_PROVIDER_CONTRACT.egress.target, - secretValuesRead: false, - kubernetesSecretDataRead: false, - valuesRedacted: true - }; - - if (kubectl.status !== "ready") { - if (blockOnMismatch) { - addBlocker(blockers, "environment_blocker", `code-agent-provider-env-read-${phase}`, kubectl.reason); - } - return { - ...base, - status: "blocked", - blocker: kubectl.reason - }; - } - - const result = await kubectlResult(kubectl, commandArgs, 15000); - if (!result.ok) { - const summary = oneLine(result.stderr || result.stdout); - if (blockOnMismatch) { - addBlocker(blockers, "environment_blocker", `code-agent-provider-env-read-${phase}`, `Cannot read DEV Deployment hwlab-cloud-api provider env: ${summary}`); - } - return { - ...base, - status: "blocked", - blocker: summary - }; - } - - try { - const inspection = inspectCodeAgentProviderLiveDeployment(JSON.parse(result.stdout)); - if (!inspection.ready && blockOnMismatch) { - addBlocker(blockers, "runtime_blocker", `code-agent-provider-env-preservation-${phase}`, codeAgentProviderBlockerSummary(inspection)); - } - return { - ...base, - ...inspection, - phase, - status: inspection.ready ? "pass" : "blocked" - }; - } catch (error) { - const summary = oneLine(error.message); - if (blockOnMismatch) { - addBlocker(blockers, "contract_blocker", `code-agent-provider-env-read-${phase}`, `Cannot parse DEV Deployment hwlab-cloud-api provider env: ${summary}`); - } - return { - ...base, - status: "blocked", - blocker: summary - }; - } -} - -function buildManualCommands(status) { - if (status === "blocked") { - return { - status: "blocked", - beforeHumanApproval: [blockedDryRunCommand, "node scripts/validate-dev-gate-report.mjs"], - afterHumanApproval: [], - summary: "No apply command is offered while blockers remain open." - }; - } - - return { - status: "ready", - beforeHumanApproval: [dryRunPlanCommand, "node scripts/validate-dev-gate-report.mjs"], - afterHumanApproval: [applyCommand], - summary: "Run the dry-run plan and validator immediately before requesting human approval; live apply still requires the explicit DEV confirmation flags." - }; -} - -function buildApplyBoundary(args, status, applyStep, kubectl) { - return { - currentMode: args.apply ? "apply-requested" : "dry-run", - defaultNoWrite: !args.apply, - mutationAttempted: applyStep.mutationAttempted === true && args.apply, - mutationAllowed: status === "pass" && args.apply, - applyRequiresFlags: ["--apply", "--confirm-dev", "--confirmed-non-production"], - kubeconfigSource: kubectl.kubeconfigSource, - writeScope: `${kubectlCommand(kubectl, ["apply", "-k", "deploy/k8s/dev"])}, namespace hwlab-dev only; delete/recreate is allowed only for explicitly allowlisted suspended DEV template Jobs and stale legacy simulator Deployments`, - noWriteScope: `source/catalog/k8s validation, optional DEV health probe, ${kubectl.commandPrefix} read, replacement planning, and server-side dry-run only`, - forbiddenActions - }; -} - -function buildPlanConclusion(status, blockers) { - return { - status: status === "blocked" ? "blocked" : "ready", - reason: - status === "blocked" - ? `${blockers.length} blocker(s) must be cleared before any live DEV apply.` - : "All DEV-only preconditions passed; human approval is still required before apply.", - blockerCount: blockers.length - }; -} - -function validateArgs(args, blockers) { - for (const error of args.errors ?? []) { - const scope = error.startsWith("--report") ? "report-argument" : "kubeconfig-argument"; - addBlocker(blockers, "safety_blocker", scope, error); - } - for (const forbidden of ["--prod", "--production", "--read-secret", "--force-push", "--heavyweight-e2e"]) { - if (args.flags.has(forbidden)) { - addBlocker(blockers, "safety_blocker", forbidden.slice(2), `${forbidden} is forbidden for DEV deploy apply`); - } - } - if (args.apply && (!args.flags.has("--confirm-dev") || !args.flags.has("--confirmed-non-production"))) { - addBlocker( - blockers, - "safety_blocker", - "apply-confirmation", - "Live apply requires --confirm-dev and --confirmed-non-production" - ); - } -} - -function commitMatchesSource(commitId, sourceCommitId) { - return sourceCommitId.startsWith(commitId) || commitId.startsWith(sourceCommitId); -} - -function validateDeployRuntimeIdentityEnv(deploy, catalog, serviceId, blockers) { - const deployService = deploy?.services?.find((service) => service.serviceId === serviceId) ?? null; - if (!deployService) return null; - const artifact = artifactIdentityForService(deploy, catalog, serviceId); - const expected = expectedRuntimeIdentityEnv(artifact); - const comparison = compareRuntimeIdentityEnv(deployService.env ?? {}, expected); - if (!comparison.matchesDesired) { - addBlocker( - blockers, - "contract_blocker", - `deploy-runtime-env-${serviceId}`, - `${serviceId} deploy env does not own desired runtime identity: ${formatRuntimeIdentityDrift(comparison.drift)}` - ); - } - return { - serviceId, - source: "deploy", - expected, - ...comparison - }; -} - -function validateDeployAndCatalog(deploy, catalog, sourceCommitId, blockers) { - const artifactEvidence = []; - if (!deploy || !catalog) return artifactEvidence; - - if (deploy.environment !== ENVIRONMENT_DEV || deploy.namespace !== namespace) { - addBlocker(blockers, "safety_blocker", "deploy-target", "deploy/deploy.json must target only dev hwlab-dev"); - } - if (deploy.profiles?.prod?.enabled !== false) { - addBlocker(blockers, "safety_blocker", "prod-profile", "PROD profile must remain disabled"); - } - if (catalog.environment !== ENVIRONMENT_DEV || catalog.namespace !== namespace) { - addBlocker(blockers, "safety_blocker", "catalog-target", "artifact catalog must target only dev hwlab-dev"); - } - if (!commitMatchesSource(deploy.commitId, sourceCommitId) || !commitMatchesSource(catalog.commitId, sourceCommitId)) { - addBlocker( - blockers, - "observability_blocker", - "artifact-source-commit", - `deploy/catalog artifact commit ${deploy.commitId}/${catalog.commitId} does not match source ${sourceCommitId}` - ); - } - - const deployById = new Map((deploy.services ?? []).map((service) => [service.serviceId, service])); - const catalogById = new Map((catalog.services ?? []).map((service) => [service.serviceId, service])); - for (const serviceId of SERVICE_IDS) { - const deployService = deployById.get(serviceId); - const catalogService = catalogById.get(serviceId); - if (!deployService || !catalogService) { - addBlocker(blockers, "contract_blocker", `service-${serviceId}`, `${serviceId} missing from deploy or catalog`); - continue; - } - if (deployService.namespace !== namespace || deployService.profile !== ENVIRONMENT_DEV) { - addBlocker(blockers, "safety_blocker", `service-target-${serviceId}`, `${serviceId} is not DEV-only`); - } - if (deployService.image !== catalogService.image) { - addBlocker(blockers, "contract_blocker", `service-image-${serviceId}`, `${serviceId} image differs in deploy/catalog`); - } - artifactEvidence.push({ - serviceId, - deployCommitId: deploy.commitId, - catalogCommitId: catalogService.commitId, - image: deployService.image, - digest: catalogService.digest, - publishState: catalogService.publishState - }); - } - validateDeployRuntimeIdentityEnv(deploy, catalog, "hwlab-cloud-web", blockers); - - const requiredServices = (catalog.services ?? []).filter((service) => service.artifactRequired !== false); - const disabledServices = (catalog.services ?? []).filter((service) => service.artifactRequired === false); - const hasPublishedCatalog = - catalog.publish?.ciPublished === true && - catalog.publish?.registryVerified === true && - requiredServices.every((service) => service.digest?.startsWith("sha256:") && service.publishState === "published") && - disabledServices.every((service) => service.digest === "not_published" && service.publishState === "skeleton-only"); - if (!hasPublishedCatalog) { - addBlocker( - blockers, - "observability_blocker", - "artifact-publish", - "DEV artifact catalog has no CI publish, registry verification, or required-service registry digests" - ); - } - return artifactEvidence; -} - -function validateK8s(devKustomization, namespaceDoc, workloads, services, healthContract, deploy, catalog, blockers) { - const items = [...listItems(namespaceDoc), ...listItems(workloads), ...listItems(services), ...listItems(healthContract)]; - if (devKustomization?.namespace !== namespace) { - addBlocker(blockers, "safety_blocker", "dev-kustomization", "DEV kustomization must pin namespace hwlab-dev"); - } - if ((devKustomization?.resources ?? []).some((resource) => resource.includes("prod"))) { - addBlocker(blockers, "safety_blocker", "dev-kustomization-prod", "DEV kustomization must not reference PROD resources"); - } - for (const item of items) { - const itemNamespace = item?.metadata?.namespace ?? (item?.kind === "Namespace" ? item?.metadata?.name : namespace); - if (itemNamespace !== namespace) { - addBlocker(blockers, "safety_blocker", `k8s-${item?.kind}-${item?.metadata?.name}`, "Kubernetes resource is not in hwlab-dev"); - } - } - - const expectedImages = new Map((deploy?.services ?? []).map((service) => [service.serviceId, service.image])); - const cloudWebExpectedRuntimeEnv = expectedRuntimeIdentityEnv(artifactIdentityForService(deploy, catalog, "hwlab-cloud-web")); - const workloadItems = listItems(workloads); - const images = []; - const runtimeIdentityEnv = []; - for (const item of workloadItems) { - for (const container of containersFor(item)) { - const serviceId = serviceIdFor(item, container); - const expectedImage = expectedImages.get(serviceId); - images.push({ serviceId, image: container.image, kind: item.kind, name: item.metadata?.name }); - if (expectedImage && container.image !== expectedImage) { - addBlocker(blockers, "contract_blocker", `k8s-image-${serviceId}`, `${serviceId} image differs in k8s workload`); - } - if (serviceId === "hwlab-cloud-web") { - const comparison = compareRuntimeIdentityEnv(envObjectFromEnvList(container.env), cloudWebExpectedRuntimeEnv); - runtimeIdentityEnv.push({ - serviceId, - kind: item.kind, - name: item.metadata?.name, - container: container.name, - expected: cloudWebExpectedRuntimeEnv, - ...comparison - }); - if (!comparison.matchesDesired) { - addBlocker( - blockers, - "contract_blocker", - `k8s-runtime-env-${serviceId}`, - `${serviceId} workload env does not own desired runtime identity: ${formatRuntimeIdentityDrift(comparison.drift)}` - ); - } - } - } - } - return { - resourceCounts: { - workloads: workloadItems.length, - services: listItems(services).length, - configMaps: listItems(healthContract).filter((item) => item.kind === "ConfigMap").length - }, - images, - runtimeIdentityEnv - }; -} - -async function checkCloudApiDb(deploy, workloads, services, blockers) { - const deployEnv = deploy?.services?.find((service) => service.serviceId === "hwlab-cloud-api")?.env ?? {}; - const envNames = new Set(Object.keys(deployEnv)); - const workloadEnv = new Map(); - for (const item of listItems(workloads)) { - for (const container of containersFor(item)) { - if (serviceIdFor(item, container) === "hwlab-cloud-api") { - for (const env of container.env ?? []) { - envNames.add(env.name); - workloadEnv.set(env.name, env); - } - } - } - } - const requiredEnv = DEV_DB_ENV_CONTRACT.requiredEnv.map((name) => { - const secretRef = DEV_DB_ENV_CONTRACT.secretRefs.find((ref) => ref.env === name) ?? null; - const workloadEntry = workloadEnv.get(name) ?? null; - const secretRefPresent = secretRef - ? deployEnv[name] === `secretRef:${secretRef.secretName}/${secretRef.secretKey}` && - workloadEntry?.valueFrom?.secretKeyRef?.name === secretRef.secretName && - workloadEntry?.valueFrom?.secretKeyRef?.key === secretRef.secretKey - : true; - return { - name, - manifestPresent: Object.hasOwn(deployEnv, name), - k8sPresent: workloadEnv.has(name), - redacted: Boolean(secretRef), - source: secretRef ? "k8s-secret-ref" : "runtime-env", - secretRef: secretRef - ? { - secretName: secretRef.secretName, - secretKey: secretRef.secretKey, - present: secretRefPresent, - redacted: true - } - : null - }; - }); - const missingManifest = requiredEnv.filter((env) => !env.manifestPresent).map((env) => env.name); - const missingK8s = requiredEnv.filter((env) => !env.k8sPresent).map((env) => env.name); - const missingSecretRefs = requiredEnv - .filter((env) => env.secretRef && !env.secretRef.present) - .map((env) => `${env.secretRef.secretName}/${env.secretRef.secretKey}`); - const expectedSslMode = DEV_DB_ENV_CONTRACT.nonSecretDefaults.HWLAB_CLOUD_DB_SSL_MODE; - const sslModeMatches = - deployEnv.HWLAB_CLOUD_DB_SSL_MODE === expectedSslMode && - workloadEnv.get("HWLAB_CLOUD_DB_SSL_MODE")?.value === expectedSslMode; - const mismatchedNonSecretEnv = sslModeMatches - ? [] - : [`HWLAB_CLOUD_DB_SSL_MODE must be ${expectedSslMode}`]; - const dnsContract = inspectCloudApiDbOptionalAliasContract(deployEnv, workloadEnv, services); - const missingDnsContract = dnsContract.missing; - if ( - missingManifest.length > 0 || - missingK8s.length > 0 || - missingSecretRefs.length > 0 || - mismatchedNonSecretEnv.length > 0 || - missingDnsContract.length > 0 - ) { - addBlocker( - blockers, - "runtime_blocker", - "cloud-api-db-config", - `cloud-api DEV DB env/authority contract is incomplete; missing ${[ - ...missingManifest, - ...missingK8s, - ...missingSecretRefs, - ...mismatchedNonSecretEnv, - ...missingDnsContract - ].join(", ")}` - ); - } - - const configReady = - missingManifest.length === 0 && - missingK8s.length === 0 && - missingSecretRefs.length === 0 && - mismatchedNonSecretEnv.length === 0 && - missingDnsContract.length === 0; - const health = { - status: configReady ? "degraded" : "blocked", - ready: false, - configReady, - connected: false, - connectionChecked: false, - fields: requiredEnv.map((env) => ({ - name: env.name, - present: env.manifestPresent && env.k8sPresent && (!env.secretRef || env.secretRef.present), - redacted: env.redacted - })), - missingEnv: uniqueStrings([...missingManifest, ...missingK8s]), - secretRefs: requiredEnv - .filter((env) => env.secretRef) - .map((env) => ({ - env: env.name, - secretName: env.secretRef.secretName, - secretKey: env.secretRef.secretKey, - present: env.secretRef.present, - redacted: true - })), - safety: { - liveDbEvidence: false - } - }; - - return { - status: configReady ? "contract-ready" : "contract-blocked", - requiredEnv, - missingManifest, - missingK8s, - missingSecretRefs, - mismatchedNonSecretEnv, - missingDnsContract, - dns: dnsContract, - runtimeHealth: summarizeDbContract(health), - secretMaterialRead: false, - valuesRedacted: true, - liveDbEvidence: false, - fixtureEvidence: false, - note: "This check records DB env, Secret reference, and Secret URL host authority only. Optional cloud-api-db alias presence is diagnostic and is not live DB evidence." - }; -} - -function inspectCloudApiDbOptionalAliasContract(deployEnv, workloadEnv, services) { - const alias = DEV_DB_ENV_CONTRACT.dns; - const service = listItems(services).find((item) => item?.metadata?.name === alias.serviceName) ?? null; - const actual = { - deployServiceName: deployEnv.HWLAB_CLOUD_DB_SERVICE_NAME, - deployServiceNamespace: deployEnv.HWLAB_CLOUD_DB_SERVICE_NAMESPACE, - deployHost: deployEnv.HWLAB_CLOUD_DB_HOST, - deployPort: deployEnv.HWLAB_CLOUD_DB_PORT, - workloadServiceName: workloadEnv.get("HWLAB_CLOUD_DB_SERVICE_NAME")?.value, - workloadServiceNamespace: workloadEnv.get("HWLAB_CLOUD_DB_SERVICE_NAMESPACE")?.value, - workloadHost: workloadEnv.get("HWLAB_CLOUD_DB_HOST")?.value, - workloadPort: workloadEnv.get("HWLAB_CLOUD_DB_PORT")?.value, - serviceName: service?.metadata?.name - }; - const missing = []; - for (const [name, value] of Object.entries(actual)) { - if (value !== undefined) { - missing.push(`${name} must be absent because ${alias.serviceName} is not the readiness authority`); - } - } - - return { - ready: missing.length === 0, - source: alias.source, - serviceName: alias.serviceName, - namespace: alias.namespace, - port: alias.port, - portName: alias.portName, - requiredForReadiness: alias.requiredForReadiness, - usedForProbe: alias.usedForProbe, - servicePresent: Boolean(service), - secretMaterialRead: false, - liveDbEvidence: false, - missing, - actual - }; -} - -async function probeDevEndpoint(blockers) { - const url = new URL(healthPath, DEV_ENDPOINT).href; - try { - const response = await httpGet(url, 8000); - const text = response.body; - let json = null; - try { - json = JSON.parse(text); - } catch {} - if (response.statusCode < 200 || response.statusCode >= 300) { - addBlocker(blockers, "network_blocker", "dev-health", `DEV health returned HTTP ${response.statusCode}`); - } - if (json?.serviceId !== "hwlab-cloud-api" || json?.environment !== ENVIRONMENT_DEV) { - addBlocker(blockers, "runtime_blocker", "dev-health-identity", "DEV health did not identify hwlab-cloud-api in dev"); - } - if (json?.db?.ready !== true && json?.db?.configReady !== true) { - addBlocker(blockers, "runtime_blocker", "dev-health-db", "DEV health did not confirm cloud-api DB config readiness"); - } - return { - status: response.statusCode >= 200 && response.statusCode < 300 ? "pass" : "blocked", - url, - httpStatus: response.statusCode, - body: text.slice(0, 500) - }; - } catch (error) { - const reason = oneLine(error.cause?.message ?? error.message); - addBlocker(blockers, "network_blocker", "dev-health", `Cannot reach ${url}: ${reason}`); - return { status: "blocked", url, error: reason }; - } -} - -function httpGet(url, timeoutMs) { - return new Promise((resolve, reject) => { - const request = httpRequest(url, { method: "GET", timeout: timeoutMs }, (response) => { - response.setEncoding("utf8"); - let body = ""; - response.on("data", (chunk) => { - body += chunk; - }); - response.on("end", () => resolve({ statusCode: response.statusCode ?? 0, body })); - }); - request.on("timeout", () => { - request.destroy(new Error(`timeout after ${timeoutMs}ms`)); - }); - request.on("error", reject); - request.end(); - }); -} - -async function observeCluster(kubectl, blockers) { - const blockedBase = { - status: "blocked", - executor: kubectl.executor ?? "missing", - kubeconfig: kubectl.kubeconfig, - kubeconfigSource: kubectl.kubeconfigSource, - commandPrefix: kubectl.commandPrefix, - resources: [] - }; - if (!kubectl.executor) { - addBlocker(blockers, "environment_blocker", "kubectl", kubectl.reason); - return blockedBase; - } - if (kubectl.status !== "ready") { - addBlocker(blockers, "environment_blocker", "d601-kubeconfig", kubectl.reason); - return blockedBase; - } - const version = await kubectlResult(kubectl, ["version", "--client=true", "--output=yaml"]); - if (!version.ok) { - addBlocker(blockers, "environment_blocker", "kubectl-version", `kubectl client failed: ${version.stderr || version.stdout}`); - return blockedBase; - } - const getArgs = ["get", "pods,services,configmaps,deployments,jobs", "-n", namespace, "-o", "json"]; - const get = await kubectlResult(kubectl, getArgs, 20000); - if (!get.ok) { - addBlocker(blockers, "environment_blocker", "cluster-read", `Cannot read hwlab-dev resources: ${get.stderr || get.stdout}`); - return { ...blockedBase, version: version.stdout, readCommand: kubectlCommand(kubectl, getArgs) }; - } - const parsed = JSON.parse(get.stdout); - return { - status: "pass", - executor: kubectl.executor, - kubeconfig: kubectl.kubeconfig, - kubeconfigSource: kubectl.kubeconfigSource, - commandPrefix: kubectl.commandPrefix, - readCommand: kubectlCommand(kubectl, getArgs), - version: version.stdout, - resources: (parsed.items ?? []).map((item) => ({ - kind: item.kind, - name: item.metadata?.name, - phase: item.status?.phase, - readyReplicas: item.status?.readyReplicas, - replicas: item.status?.replicas - })) - }; -} - -function isNotFoundOutput(value) { - return /\bnot\s+found\b|notfound/iu.test(value); -} - -async function readLiveTemplateJob(kubectl, jobName, blockers) { - if (kubectl.status !== "ready") { - return { status: "not_evaluated", reason: kubectl.reason }; - } - const get = await kubectlResult(kubectl, ["-n", namespace, "get", "job", jobName, "-o", "json"], 15000); - if (!get.ok) { - const output = commandOutput(get); - if (isNotFoundOutput(output)) { - return { status: "not_found", liveJob: null }; - } - addBlocker(blockers, "environment_blocker", `template-job-read-${jobName}`, `Cannot read live DEV template Job ${jobName}: ${output}`); - return { status: "blocked", reason: oneLine(output), liveJob: null }; - } - try { - return { status: "found", liveJob: JSON.parse(get.stdout) }; - } catch (error) { - addBlocker(blockers, "contract_blocker", `template-job-read-${jobName}`, `Cannot parse live DEV template Job ${jobName}: ${error.message}`); - return { status: "blocked", reason: oneLine(error.message), liveJob: null }; - } -} - -async function readLiveLegacySimulatorDeployment(kubectl, deploymentName, blockers) { - if (kubectl.status !== "ready") { - return { status: "not_evaluated", reason: kubectl.reason }; - } - const get = await kubectlResult(kubectl, ["-n", namespace, "get", "deployment", deploymentName, "-o", "json"], 15000); - if (!get.ok) { - const output = commandOutput(get); - if (isNotFoundOutput(output)) { - return { status: "not_found", liveDeployment: null }; - } - addBlocker(blockers, "environment_blocker", `legacy-simulator-deployment-read-${deploymentName}`, `Cannot read live DEV legacy simulator Deployment ${deploymentName}: ${output}`); - return { status: "blocked", reason: oneLine(output), liveDeployment: null }; - } - try { - return { status: "found", liveDeployment: JSON.parse(get.stdout) }; - } catch (error) { - addBlocker(blockers, "contract_blocker", `legacy-simulator-deployment-read-${deploymentName}`, `Cannot parse live DEV legacy simulator Deployment ${deploymentName}: ${error.message}`); - return { status: "blocked", reason: oneLine(error.message), liveDeployment: null }; - } -} - -async function readLiveDeployment(kubectl, deploymentName, blockers, scopePrefix) { - if (kubectl.status !== "ready") { - return { status: "not_evaluated", reason: kubectl.reason }; - } - const get = await kubectlResult(kubectl, ["-n", namespace, "get", "deployment", deploymentName, "-o", "json"], 15000); - if (!get.ok) { - const output = commandOutput(get); - if (isNotFoundOutput(output)) { - return { status: "not_found", liveDeployment: null }; - } - addBlocker(blockers, "environment_blocker", `${scopePrefix}-${deploymentName}`, `Cannot read live DEV Deployment ${deploymentName}: ${output}`); - return { status: "blocked", reason: oneLine(output), liveDeployment: null }; - } - try { - return { status: "found", liveDeployment: JSON.parse(get.stdout) }; - } catch (error) { - addBlocker(blockers, "contract_blocker", `${scopePrefix}-${deploymentName}`, `Cannot parse live DEV Deployment ${deploymentName}: ${error.message}`); - return { status: "blocked", reason: oneLine(error.message), liveDeployment: null }; - } -} - -async function buildDevTemplateJobReplacementPlan(kubectl, workloads, blockers) { - const desiredJobs = listItems(workloads).filter(isAllowedDesiredTemplateJob); - const replacements = []; - for (const desiredJob of desiredJobs) { - const jobName = desiredJob.metadata.name; - const live = await readLiveTemplateJob(kubectl, jobName, blockers); - if (live.status === "not_evaluated") { - replacements.push({ - namespace, - jobName, - allowed: true, - desiredSuspended: true, - liveExists: null, - liveSuspended: null, - oldImage: null, - newImage: primaryImage(imageRecordsFor(desiredJob)), - oldImages: [], - newImages: imageRecordsFor(desiredJob), - replace: false, - result: "not_evaluated", - reason: live.reason - }); - continue; - } - if (live.status === "blocked") { - replacements.push({ - namespace, - jobName, - allowed: true, - desiredSuspended: true, - liveExists: null, - liveSuspended: null, - oldImage: null, - newImage: primaryImage(imageRecordsFor(desiredJob)), - oldImages: [], - newImages: imageRecordsFor(desiredJob), - replace: false, - result: "blocked", - reason: live.reason - }); - continue; - } - - const decision = decideDevTemplateJobReplacement({ - jobName, - jobNamespace: itemNamespace(desiredJob), - desiredSuspended: desiredJob.spec?.suspend, - liveExists: live.status === "found", - liveSuspended: live.liveJob?.spec?.suspend, - oldImages: live.liveJob ? imageRecordsFor(live.liveJob) : [], - newImages: imageRecordsFor(desiredJob) - }); - if (decision.result === "blocked") { - addBlocker(blockers, "safety_blocker", `template-job-replace-${jobName}`, decision.reason); - } - replacements.push(decision); - } - return replacements; -} - -async function buildDevLegacySimulatorDeploymentCleanupPlan(kubectl, workloads, blockers) { - const desiredReplacements = new Map( - listItems(workloads) - .filter(isDesiredLegacySimulatorReplacement) - .map((item) => [item.metadata.name, item]) - ); - const cleanups = []; - for (const policy of devLegacySimulatorDeploymentCleanupPolicy.allowedDeployments) { - const desiredReplacement = desiredReplacements.get(policy.name) ?? null; - const live = await readLiveLegacySimulatorDeployment(kubectl, policy.name, blockers); - if (live.status === "not_evaluated") { - cleanups.push({ - namespace, - deploymentName: policy.name, - serviceId: policy.serviceId, - allowed: true, - desiredReplacementExists: Boolean(desiredReplacement), - desiredReplacementKind: desiredReplacement?.kind ?? null, - liveExists: null, - oldImage: null, - newImage: primaryImage(desiredReplacement ? imageRecordsFor(desiredReplacement) : []), - oldImages: [], - newImages: desiredReplacement ? imageRecordsFor(desiredReplacement) : [], - cleanup: false, - result: "not_evaluated", - reason: live.reason - }); - continue; - } - if (live.status === "blocked") { - cleanups.push({ - namespace, - deploymentName: policy.name, - serviceId: policy.serviceId, - allowed: true, - desiredReplacementExists: Boolean(desiredReplacement), - desiredReplacementKind: desiredReplacement?.kind ?? null, - liveExists: null, - oldImage: null, - newImage: primaryImage(desiredReplacement ? imageRecordsFor(desiredReplacement) : []), - oldImages: [], - newImages: desiredReplacement ? imageRecordsFor(desiredReplacement) : [], - cleanup: false, - result: "blocked", - reason: live.reason - }); - continue; - } - - const decision = decideDevLegacySimulatorDeploymentCleanup({ - deploymentName: policy.name, - deploymentNamespace: namespace, - desiredReplacementExists: Boolean(desiredReplacement), - desiredReplacementKind: desiredReplacement?.kind, - liveExists: live.status === "found", - oldImages: live.liveDeployment ? imageRecordsFor(live.liveDeployment) : [], - newImages: desiredReplacement ? imageRecordsFor(desiredReplacement) : [] - }); - if (decision.result === "blocked") { - addBlocker(blockers, "safety_blocker", `legacy-simulator-deployment-cleanup-${policy.name}`, decision.reason); - } - cleanups.push(decision); - } - return cleanups; -} - -async function buildDevUnmanagedHotfixDeploymentCleanupPlan(kubectl, workloads, blockers) { - const desiredDeployments = desiredDeploymentIndex(workloads); - const cleanups = []; - for (const [key, desiredDeployment] of desiredDeployments.entries()) { - const deploymentName = desiredDeployment.metadata.name; - const live = await readLiveDeployment(kubectl, deploymentName, blockers, "unmanaged-hotfix-deployment-read"); - if (live.status === "not_evaluated") { - cleanups.push({ - namespace, - deploymentName, - desiredExists: true, - liveExists: null, - cleanup: false, - result: "not_evaluated", - reason: live.reason, - unmanagedVolumes: [], - unmanagedVolumeMounts: [], - unmanagedTemplateAnnotations: [] - }); - continue; - } - if (live.status === "blocked") { - cleanups.push({ - namespace, - deploymentName, - desiredExists: true, - liveExists: null, - cleanup: false, - result: "blocked", - reason: live.reason, - unmanagedVolumes: [], - unmanagedVolumeMounts: [], - unmanagedTemplateAnnotations: [] - }); - continue; - } - const decision = decideDevUnmanagedHotfixDeploymentCleanup({ - deploymentName, - deploymentNamespace: key.split("/")[0], - desiredDeployment, - liveDeployment: live.liveDeployment - }); - if (decision.result === "blocked") { - addBlocker(blockers, "safety_blocker", `unmanaged-hotfix-deployment-cleanup-${deploymentName}`, decision.reason); - } - cleanups.push(decision); - } - return cleanups; -} - -async function executeDevTemplateJobReplacements(kubectl, replacements, blockers, concurrency = defaultCdConcurrency) { - const planned = replacements.filter((replacement) => replacement.replace && replacement.result === "planned"); - await mapWithConcurrency(planned, concurrency, async (replacement) => { - const commandArgs = [ - "-n", - replacement.namespace, - "delete", - "job", - replacement.jobName, - "--ignore-not-found=true" - ]; - const result = await kubectlResult(kubectl, commandArgs, 20000); - replacement.deleteCommand = kubectlCommand(kubectl, commandArgs); - replacement.deleteStdout = result.redactedStdout ?? result.stdout; - replacement.deleteStderr = result.redactedStderr ?? result.stderr; - if (!result.ok) { - replacement.result = "delete_failed"; - replacement.reason = `Failed to delete live suspended template Job before apply: ${oneLine(commandOutput(result))}`; - addBlocker(blockers, "environment_blocker", `template-job-replace-${replacement.jobName}`, replacement.reason); - return; - } - replacement.result = "deleted_pending_recreate"; - replacement.reason = "Live suspended template Job was deleted; kubectl apply must recreate it from the desired manifest."; - }); - return planned.length; -} - -async function executeDevLegacySimulatorDeploymentCleanups(kubectl, cleanups, blockers, concurrency = defaultCdConcurrency) { - const planned = cleanups.filter((cleanup) => cleanup.cleanup && cleanup.result === "planned"); - await mapWithConcurrency(planned, concurrency, async (cleanup) => { - const commandArgs = [ - "-n", - cleanup.namespace, - "delete", - "deployment", - cleanup.deploymentName, - "--ignore-not-found=true" - ]; - const result = await kubectlResult(kubectl, commandArgs, 20000); - cleanup.deleteCommand = kubectlCommand(kubectl, commandArgs); - cleanup.deleteStdout = result.redactedStdout ?? result.stdout; - cleanup.deleteStderr = result.redactedStderr ?? result.stderr; - if (!result.ok) { - cleanup.result = "delete_failed"; - cleanup.reason = `Failed to delete stale legacy simulator Deployment before apply: ${oneLine(commandOutput(result))}`; - addBlocker(blockers, "environment_blocker", `legacy-simulator-deployment-cleanup-${cleanup.deploymentName}`, cleanup.reason); - return; - } - cleanup.result = "deleted_pending_apply"; - cleanup.reason = "Live stale legacy simulator Deployment was deleted; kubectl apply must leave the desired indexed StatefulSet in place."; - }); - return planned.length; -} - -async function executeDevUnmanagedHotfixDeploymentCleanups(kubectl, cleanups, blockers, concurrency = defaultCdConcurrency) { - const planned = cleanups.filter((cleanup) => cleanup.cleanup && cleanup.result === "planned"); - await mapWithConcurrency(planned, concurrency, async (cleanup) => { - const commandArgs = [ - "-n", - cleanup.namespace, - "delete", - "deployment", - cleanup.deploymentName, - "--ignore-not-found=true" - ]; - const result = await kubectlResult(kubectl, commandArgs, 20000); - cleanup.deleteCommand = kubectlCommand(kubectl, commandArgs); - cleanup.deleteStdout = result.redactedStdout ?? result.stdout; - cleanup.deleteStderr = result.redactedStderr ?? result.stderr; - if (!result.ok) { - cleanup.result = "delete_failed"; - cleanup.reason = `Failed to delete live Deployment with unmanaged hotfix fields before apply: ${oneLine(commandOutput(result))}`; - addBlocker(blockers, "environment_blocker", `unmanaged-hotfix-deployment-cleanup-${cleanup.deploymentName}`, cleanup.reason); - return; - } - cleanup.result = "deleted_pending_apply"; - cleanup.reason = "Live Deployment with unmanaged hotfix fields was deleted; kubectl apply must recreate it from desired source."; - }); - return planned.length; -} - -function rolloutApiName(kind) { - if (kind === "Deployment") return "deployment"; - if (kind === "StatefulSet") return "statefulset"; - return null; -} - -function rolloutTargetsFromWorkloads(workloads) { - return listItems(workloads) - .map((item) => { - const apiName = rolloutApiName(item?.kind); - if (!apiName) return null; - return { - kind: item.kind, - apiName, - name: item?.metadata?.name ?? "unknown", - namespace: item?.metadata?.namespace ?? namespace, - replicas: replicaPlan(item), - serviceIds: uniqueStrings(containersFor(item).map((container) => serviceIdFor(item, container))) - }; - }) - .filter(Boolean); -} - -async function verifyRolloutTarget(kubectl, target, blockers) { - const commandArgs = [ - "-n", - target.namespace, - "rollout", - "status", - `${target.apiName}/${target.name}`, - "--timeout=180s" - ]; - const base = { - ...target, - command: kubectlCommand(kubectl, commandArgs) - }; - if (kubectl.status !== "ready") { - addBlocker(blockers, "environment_blocker", `rollout-status-${target.name}`, kubectl.reason); - return { - ...base, - status: "blocked", - reason: kubectl.reason - }; - } - const result = await kubectlResult(kubectl, commandArgs, 190000); - if (!result.ok) { - const reason = oneLine(commandOutput(result)); - addBlocker(blockers, "runtime_blocker", `rollout-status-${target.name}`, reason); - return { - ...base, - status: "blocked", - reason, - stdout: result.redactedStdout ?? result.stdout, - stderr: result.redactedStderr ?? result.stderr - }; - } - return { - ...base, - status: "pass", - stdout: result.redactedStdout ?? result.stdout, - stderr: result.redactedStderr ?? result.stderr - }; -} - -async function verifyRolloutsAfterApply(args, kubectl, workloads, blockers, applyStep) { - const targets = rolloutTargetsFromWorkloads(workloads); - if (!args.apply) { - return { - status: "not_run", - reason: "dry-run mode", - concurrency: args.rolloutConcurrency, - targets - }; - } - if (applyStep.status !== "pass") { - return { - status: "not_run", - reason: "apply did not complete", - concurrency: args.rolloutConcurrency, - targets - }; - } - const results = await mapWithConcurrency(targets, args.rolloutConcurrency, async (target) => - verifyRolloutTarget(kubectl, target, blockers) - ); - return { - status: results.every((result) => result.status === "pass") ? "pass" : "blocked", - concurrency: args.rolloutConcurrency, - targetCount: targets.length, - results - }; -} - -function isExpectedTemplateJobImmutableFailure(result, replacementsNeeded) { - const plannedNames = replacementsNeeded.map((replacement) => replacement.jobName); - if (result.ok || plannedNames.length === 0) return false; - const redactedOutput = `${result.redactedStdout ?? result.stdout}\n${result.redactedStderr ?? result.stderr}`; - const effectiveOutput = redactedOutput.trim().length > 0 ? redactedOutput : `${result.stdout}\n${result.stderr}`; - const errorSections = effectiveOutput.split("Error from server").slice(1); - const immutableSections = errorSections.filter( - (section) => section.includes("spec.template") && section.includes("field is immutable") - ); - if (immutableSections.length === 0 || immutableSections.length !== errorSections.length) return false; - for (const name of plannedNames) { - if (!immutableSections.some((section) => section.includes(name))) return false; - } - return immutableSections.every( - (section) => - section.includes("Resource=jobs") && - section.includes("Kind=Job") && - plannedNames.some((name) => section.includes(name)) - ); -} - -async function runApplyStep( - args, - kubectl, - blockers, - templateJobReplacements, - legacySimulatorDeploymentCleanups, - unmanagedHotfixDeploymentCleanups -) { - if (blockers.length > 0) { - return { status: "not_run", summary: "Skipped because preflight blockers are open" }; - } - let mutationAttempted = false; - const replacementsNeeded = templateJobReplacements.filter((replacement) => replacement.replace); - - if (args.apply) { - const replaceCount = await executeDevTemplateJobReplacements(kubectl, templateJobReplacements, blockers, args.rolloutConcurrency); - const cleanupCount = await executeDevLegacySimulatorDeploymentCleanups(kubectl, legacySimulatorDeploymentCleanups, blockers, args.rolloutConcurrency); - const hotfixCleanupCount = await executeDevUnmanagedHotfixDeploymentCleanups( - kubectl, - unmanagedHotfixDeploymentCleanups, - blockers, - args.rolloutConcurrency - ); - mutationAttempted = replaceCount > 0 || cleanupCount > 0 || hotfixCleanupCount > 0; - if (blockers.length > 0) { - return { - status: "blocked", - command: kubectlCommand(kubectl, ["apply", "-k", "deploy/k8s/dev"]), - mutationAttempted, - summary: "Skipped kubectl apply because DEV pre-apply cleanup failed" - }; - } - } - - const commandArgs = args.apply - ? ["apply", "-k", "deploy/k8s/dev"] - : ["apply", "--dry-run=server", "-k", "deploy/k8s/dev"]; - const result = await kubectlResult(kubectl, commandArgs, 30000); - const expectedImmutableDryRun = !args.apply && isExpectedTemplateJobImmutableFailure(result, replacementsNeeded); - mutationAttempted = mutationAttempted || args.apply; - if (!result.ok && !expectedImmutableDryRun) { - addBlocker(blockers, "environment_blocker", args.apply ? "kubectl-apply" : "kubectl-dry-run", commandOutput(result)); - } - for (const replacement of templateJobReplacements) { - if (replacement.result === "deleted_pending_recreate") { - replacement.result = result.ok ? "replaced" : "delete_succeeded_apply_failed"; - replacement.reason = result.ok - ? "Live suspended template Job was deleted and recreated by kubectl apply." - : "Live suspended template Job was deleted, but kubectl apply failed before confirming recreation."; - } - } - for (const cleanup of legacySimulatorDeploymentCleanups) { - if (cleanup.result === "deleted_pending_apply") { - cleanup.result = result.ok ? "deleted" : "delete_succeeded_apply_failed"; - cleanup.reason = result.ok - ? "Live stale legacy simulator Deployment was deleted and the desired indexed StatefulSet apply completed." - : "Live stale legacy simulator Deployment was deleted, but kubectl apply failed before confirming desired state."; - } - } - for (const cleanup of unmanagedHotfixDeploymentCleanups) { - if (cleanup.result === "deleted_pending_apply") { - cleanup.result = result.ok ? "deleted" : "delete_succeeded_apply_failed"; - cleanup.reason = result.ok - ? "Live Deployment with unmanaged hotfix fields was deleted and recreated by kubectl apply." - : "Live Deployment with unmanaged hotfix fields was deleted, but kubectl apply failed before confirming recreation."; - } - } - return { - status: result.ok || expectedImmutableDryRun ? "pass" : "blocked", - command: kubectlCommand(kubectl, commandArgs), - stdout: result.redactedStdout ?? result.stdout, - stderr: result.redactedStderr ?? result.stderr, - mutationAttempted, - expectedImmutableTemplateJobDryRun: expectedImmutableDryRun, - expectedImmutableTemplateJobs: expectedImmutableDryRun - ? replacementsNeeded.map((replacement) => replacement.jobName) - : [] - }; -} - -async function gitHeadCommit() { - const result = await commandResult("git", ["rev-parse", "--short=12", "HEAD"]); - return result.ok ? result.stdout : "unknown"; -} - -export function resolveApplySourceCommit(deploy, catalog, gitHeadCommitId) { - const deployCommit = deploy?.commitId; - const catalogCommit = catalog?.commitId; - if ( - typeof deployCommit === "string" && - typeof catalogCommit === "string" && - commitMatchesSource(deployCommit, catalogCommit) - ) { - return deployCommit.length >= catalogCommit.length ? deployCommit : catalogCommit; - } - return gitHeadCommitId; -} - -export async function runDevDeployApply(argv, io = {}) { - const stdout = io.stdout ?? process.stdout; - const args = parseArgs(argv); - const blockers = []; - validateArgs(args, blockers); - if (args.apply) { - const transactionGuard = requireDevCdTransactionForSideEffect({ - env: io.env ?? process.env, - script: "scripts/dev-deploy-apply.mjs", - mode: "--apply" - }); - if (transactionGuard) { - stdout.write(`${JSON.stringify(transactionGuard, null, 2)}\n`); - return 2; - } - } - const gitHeadCommitIdPromise = gitHeadCommit(); - - const [deploy, catalog, devKustomization, namespaceDoc, workloads, services, healthContract] = await Promise.all([ - readJson("deploy/deploy.json", blockers), - readJson("deploy/artifact-catalog.dev.json", blockers), - readJson("deploy/k8s/dev/kustomization.yaml", blockers), - readJson("deploy/k8s/base/namespace.yaml", blockers), - readJson("deploy/k8s/base/workloads.yaml", blockers), - readJson("deploy/k8s/base/services.yaml", blockers), - readJson("deploy/k8s/dev/health-contract.yaml", blockers) - ]); - const gitHeadCommitId = await gitHeadCommitIdPromise; - const commitId = resolveApplySourceCommit(deploy, catalog, gitHeadCommitId); - - const artifactEvidence = validateDeployAndCatalog(deploy, catalog, commitId, blockers); - const registryManifests = await verifyCatalogRegistryManifests(catalog, blockers, args.rolloutConcurrency, { - env: io.env ?? process.env, - registryManifestBaseUrl: args.registryManifestBaseUrl - }); - const k8sManifest = validateK8s(devKustomization, namespaceDoc, workloads, services, healthContract, deploy, catalog, blockers); - const cloudApiDb = await checkCloudApiDb(deploy, workloads, services, blockers); - const codeAgentProviderDesiredState = inspectCodeAgentProviderDesiredState(deploy, workloads); - addCodeAgentProviderDesiredStateBlocker(blockers, codeAgentProviderDesiredState); - if (!(await readText("internal/cloud/server.mjs")).includes('url.pathname === "/health/live"')) { - addBlocker(blockers, "contract_blocker", "cloud-api-health-path", "cloud-api does not implement the k8s /health/live path"); - } - - const kubectl = await resolveD601Kubectl(args); - const clusterObservation = await observeCluster(kubectl, blockers); - const cloudWebRolloutBeforeApply = await observeDeploymentRollout(kubectl, deploy, catalog, "hwlab-cloud-web", blockers, { - observationPhase: "before_apply", - blockRuntimeEnvDrift: false - }); - const codeAgentProviderLiveBeforeApply = await observeCodeAgentProviderLiveDeployment(kubectl, blockers, { - phase: "before_apply", - blockOnMismatch: !args.apply - }); - const liveProbe = args.skipLiveProbe ? { status: "not_run", reason: "--skip-live-probe" } : await probeDevEndpoint(blockers); - const templateJobReplacements = await buildDevTemplateJobReplacementPlan(kubectl, workloads, blockers); - const legacySimulatorDeploymentCleanups = await buildDevLegacySimulatorDeploymentCleanupPlan(kubectl, workloads, blockers); - const unmanagedHotfixDeploymentCleanups = await buildDevUnmanagedHotfixDeploymentCleanupPlan(kubectl, workloads, blockers); - const applyStep = await runApplyStep( - args, - kubectl, - blockers, - templateJobReplacements, - legacySimulatorDeploymentCleanups, - unmanagedHotfixDeploymentCleanups - ); - let rolloutStatusAfterApply; - let cloudWebRolloutAfterApply; - let codeAgentProviderLiveAfterApply; - if (args.apply && applyStep.status === "pass") { - [rolloutStatusAfterApply, cloudWebRolloutAfterApply, codeAgentProviderLiveAfterApply] = await Promise.all([ - verifyRolloutsAfterApply(args, kubectl, workloads, blockers, applyStep), - observeDeploymentRollout(kubectl, deploy, catalog, "hwlab-cloud-web", blockers, { - observationPhase: "after_apply", - blockRuntimeEnvDrift: true - }), - observeCodeAgentProviderLiveDeployment(kubectl, blockers, { - phase: "after_apply", - blockOnMismatch: true - }) - ]); - } else { - rolloutStatusAfterApply = await verifyRolloutsAfterApply(args, kubectl, workloads, blockers, applyStep); - cloudWebRolloutAfterApply = skippedDeploymentRolloutObservation( - kubectl, - deploy, - catalog, - "hwlab-cloud-web", - "after_apply", - args.apply ? "apply did not complete successfully; post-apply rollout env verification was not run" : "dry-run mode; no post-apply rollout env verification" - ); - codeAgentProviderLiveAfterApply = { - phase: "after_apply", - status: "not_run", - reason: args.apply ? "apply did not complete" : "dry-run mode", - secretValuesRead: false, - kubernetesSecretDataRead: false, - valuesRedacted: true - }; - } - const cloudWebRollout = args.apply ? cloudWebRolloutAfterApply : cloudWebRolloutBeforeApply; - const status = blockers.length > 0 ? "blocked" : "pass"; - const artifactPlan = buildArtifactPlan(deploy, catalog, commitId, artifactEvidence); - const workloadPlan = buildWorkloadPlan(workloads); - const servicePlan = buildServicePlan(services); - const planConclusion = buildPlanConclusion(status, blockers); - const manualCommands = buildManualCommands(status); - const remainingBlockers = buildRemainingBlockers(blockers); - - const report = { - $schema: "https://hwlab.pikastech.local/schemas/dev-gate-report.schema.json", - $id: "https://hwlab.pikastech.local/dev-cd/dev-deploy-report.json", - reportVersion: "v1", - issue: "pikasTech/HWLAB#33", - supports: ["pikasTech/HWLAB#164", "pikasTech/HWLAB#143", "pikasTech/HWLAB#99", "pikasTech/HWLAB#63", "pikasTech/HWLAB#50", "pikasTech/HWLAB#7", "pikasTech/HWLAB#33", "pikasTech/HWLAB#17", "pikasTech/HWLAB#32", "pikasTech/HWLAB#30"], - taskId: "dev-deploy-apply", - commitId, - acceptanceLevel: "dev_deploy_apply", - devOnly: true, - prodDisabled: true, - reportLifecycle: activeReportLifecycle("Current DEV deploy apply report; active public API/health endpoint is 16667 and browser endpoint is 16666."), - status, - generatedAt: new Date().toISOString(), - namespace, - endpoint: DEV_ENDPOINT, - sourceContract: { - status: "pass", - documents: [ - "docs/dev-acceptance-matrix.md", - "docs/m0-contract-audit.md", - "docs/dev-deploy-apply.md", - "docs/artifact-catalog.md", - "deploy/README.md" - ], - summary: "DEV deploy apply is pinned to hwlab-dev and the frozen HWLAB service IDs" - }, - validationCommands: requiredValidationCommands, - localSmoke: { - status: "not_run", - commands: ["node scripts/m1-contract-smoke.mjs"], - evidence: ["This deploy apply task did not run local smoke because artifact/executor blockers stop promotion first."], - summary: "Local smoke remains available but is not a substitute for DEV artifact publish evidence." - }, - dryRun: { - status, - commands: [ - "node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run", - "node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked" - ], - evidence: [ - `artifact services checked: ${artifactEvidence.length}`, - `expected artifact commit: ${artifactPlan.expectedArtifactCommit}`, - `registry manifests: ${registryManifests.status} (${registryManifests.serviceCount} services)`, - `namespace: ${namespace}`, - `workloads planned: ${workloadPlan.length}`, - `kubectl executor: ${kubectl.executor ?? "missing"}`, - `kubectl kubeconfig source: ${kubectl.kubeconfigSource}`, - `kubectl kubeconfig: ${kubectl.kubeconfig}`, - `live health: ${liveProbe.status}`, - `code agent provider desired-state: ${codeAgentProviderDesiredState.status}`, - `code agent provider live env before apply: ${codeAgentProviderLiveBeforeApply.status}`, - `rollout status concurrency: ${args.rolloutConcurrency}`, - `template job replacements: ${templateJobReplacements.filter((replacement) => replacement.replace).length}`, - `legacy simulator deployment cleanups: ${legacySimulatorDeploymentCleanups.filter((cleanup) => cleanup.cleanup).length}`, - `unmanaged hotfix deployment cleanups: ${unmanagedHotfixDeploymentCleanups.filter((cleanup) => cleanup.cleanup).length}` - ], - summary: status === "blocked" ? "DEV apply is blocked before mutation." : "DEV dry-run preflight passed." - }, - devPreconditions: { - status, - requirements: [ - "DEV artifact catalog must contain CI publish expectations and registry digests; CD must verify those tag/digest pairs directly against registry manifests", - "kubectl must be available for D601 hwlab-dev and must not target PROD", - "hwlab-cloud-api /health/live must report serviceId, dev environment, and DB env readiness without exposing secret values", - "hwlab-cloud-api must declare and preserve OPENAI_API_KEY from hwlab-code-agent-provider/openai-api-key plus the DEV Code Agent egress proxy env without reading Secret values", - "pods, services, configmaps, and image commit tags must be observable in hwlab-dev" - ], - summary: status === "blocked" ? "One or more required DEV deploy preconditions are blocked." : "DEV deploy preconditions are satisfied." - }, - devDeployApply: { - mode: args.apply ? "apply" : "dry-run", - conclusion: planConclusion, - applyBoundary: buildApplyBoundary(args, status, applyStep, kubectl), - mutationAttempted: applyStep.mutationAttempted === true && args.apply, - target: { - environment: ENVIRONMENT_DEV, - namespace, - endpoint: DEV_ENDPOINT, - prodDisabled: true - }, - artifactPlan, - workloadPlan, - servicePlan, - templateJobReplacementPolicy: devTemplateJobReplacementPolicy, - templateJobReplacements, - legacySimulatorDeploymentCleanupPolicy: devLegacySimulatorDeploymentCleanupPolicy, - legacySimulatorDeploymentCleanups, - unmanagedHotfixDeploymentCleanupPolicy: devUnmanagedHotfixDeploymentCleanupPolicy, - unmanagedHotfixDeploymentCleanups, - cloudWebRollout, - cloudWebRolloutBeforeApply, - cloudWebRolloutAfterApply, - rolloutStatusAfterApply, - codeAgentProvider: { - desiredState: codeAgentProviderDesiredState, - liveBeforeApply: codeAgentProviderLiveBeforeApply, - liveAfterApply: codeAgentProviderLiveAfterApply, - validationGate: "no-secret provider env/secretKeyRef preservation", - providerCallAttempted: false, - secretValuesRead: false, - kubernetesSecretDataRead: false, - valuesRedacted: true - }, - applyStep, - manualCommands, - rollbackHint: buildRollbackHint(kubectl, workloads), - remainingBlockers, - artifactEvidence, - registryManifests, - cloudApiDb, - k8sManifest, - clusterObservation, - liveProbe - }, - blockers, - notes: "DEV-only result. No PROD resources, secret values, force push, heavy e2e, browser e2e, or UniDesk runtime substitute were used." - }; - - if (args.writeReport) { - const absoluteReportPath = ensureNotRepoReportsPath(repoRoot, args.reportPath, "--report"); - await mkdir(path.dirname(absoluteReportPath), { recursive: true }); - await writeFile(absoluteReportPath, `${JSON.stringify(report, null, 2)}\n`); - } - - stdout.write(`${JSON.stringify(report, null, 2)}\n`); - return status === "blocked" && !args.expectBlocked ? 2 : 0; -} - -export function formatFailure(error) { - return { - reportVersion: "v1", - issue: "pikasTech/HWLAB#33", - taskId: "dev-deploy-apply", - status: "failed", - blockers: [{ type: "observability_blocker", scope: "script", status: "open", summary: oneLine(error.message) }] - }; -} diff --git a/scripts/src/dev-deploy-apply.test.mjs b/scripts/src/dev-deploy-apply.test.mjs deleted file mode 100644 index af6deddd..00000000 --- a/scripts/src/dev-deploy-apply.test.mjs +++ /dev/null @@ -1,648 +0,0 @@ -import assert from "node:assert/strict"; -import test from "node:test"; - -import { - buildKubectlCommandPrefix, - compareRuntimeIdentityEnv, - decideDevLegacySimulatorDeploymentCleanup, - decideDevTemplateJobReplacement, - decideDevUnmanagedHotfixDeploymentCleanup, - formatKubeconfigAccessFailure, - inspectCodeAgentProviderDesiredState, - inspectCodeAgentProviderLiveDeployment, - parseArgs, - redactSensitiveText, - registryManifestUrlForImage, - resolveApplySourceCommit, - resolveDevKubeconfigSelection, - verifyCatalogRegistryManifest -} from "./dev-deploy-apply.mjs"; -import { - devArtifactProducer, - devArtifactPublishIdentityGuardFailure, - devCdTransactionGuardFailure, - hasCiArtifactPublishEnv, - hasDevCdTransactionEnv, - requireDevArtifactPublishIdentity, - requireDevCdTransactionForSideEffect -} from "./dev-cd-transaction-guard.mjs"; - -const codeAgentBaseUrl = "http://172.26.26.227:17680/v1/responses"; - -function codeAgentDeploy(env = {}) { - return { - services: [ - { - serviceId: "hwlab-cloud-api", - env: { - HWLAB_CODE_AGENT_PROVIDER: "codex-stdio", - HWLAB_CODE_AGENT_MODEL: "gpt-5.5", - HWLAB_CODE_AGENT_OPENAI_BASE_URL: codeAgentBaseUrl, - CODEX_HOME: "/codex-home", - NO_PROXY: "hyueapi.com,.hyueapi.com,hyui.com,.hyui.com,127.0.0.1,localhost,::1,api.minimaxi.com,.minimaxi.com", - no_proxy: "hyueapi.com,.hyueapi.com,hyui.com,.hyui.com,127.0.0.1,localhost,::1,api.minimaxi.com,.minimaxi.com", - OPENAI_API_KEY: "secretRef:hwlab-code-agent-provider/openai-api-key", - ...env - } - } - ] - }; -} - -function codeAgentWorkloads(envOverrides = []) { - return { - kind: "List", - items: [ - { - kind: "Deployment", - metadata: { - name: "hwlab-cloud-api", - namespace: "hwlab-dev", - labels: { - "hwlab.pikastech.local/service-id": "hwlab-cloud-api" - } - }, - spec: { - template: { - spec: { - containers: [ - { - name: "hwlab-cloud-api", - env: [ - { name: "HWLAB_CODE_AGENT_PROVIDER", value: "codex-stdio" }, - { name: "HWLAB_CODE_AGENT_MODEL", value: "gpt-5.5" }, - { name: "HWLAB_CODE_AGENT_OPENAI_BASE_URL", value: codeAgentBaseUrl }, - { name: "CODEX_HOME", value: "/codex-home" }, - { name: "NO_PROXY", value: "hyueapi.com,.hyueapi.com,hyui.com,.hyui.com,127.0.0.1,localhost,::1,api.minimaxi.com,.minimaxi.com" }, - { name: "no_proxy", value: "hyueapi.com,.hyueapi.com,hyui.com,.hyui.com,127.0.0.1,localhost,::1,api.minimaxi.com,.minimaxi.com" }, - { - name: "OPENAI_API_KEY", - valueFrom: { - secretKeyRef: { - name: "hwlab-code-agent-provider", - key: "openai-api-key" - } - } - }, - ...envOverrides - ], - volumeMounts: [ - { name: "hwlab-code-agent-codex-home", mountPath: "/codex-home" }, - { name: "hwlab-code-agent-codex-config", mountPath: "/codex-home/config.toml", subPath: "config.toml", readOnly: true }, - { name: "hwlab-code-agent-codex-auth", mountPath: "/codex-home/auth.json", subPath: "auth.json", readOnly: true } - ] - } - ], - volumes: [ - { name: "hwlab-code-agent-codex-home", emptyDir: {} }, - { name: "hwlab-code-agent-codex-config", configMap: { name: "hwlab-code-agent-codex-config", items: [{ key: "config.toml", path: "config.toml" }] } }, - { name: "hwlab-code-agent-codex-auth", secret: { secretName: "hwlab-code-agent-codex-auth", items: [{ key: "auth.json", path: "auth.json" }] } } - ] - } - } - } - } - ] - }; -} - -test("allowlisted suspended template Job image change plans replacement", () => { - const decision = decideDevTemplateJobReplacement({ - jobName: "hwlab-agent-worker-template", - jobNamespace: "hwlab-dev", - desiredSuspended: true, - liveExists: true, - liveSuspended: true, - oldImages: [ - { - name: "hwlab-agent-worker", - serviceId: "hwlab-agent-worker", - image: "127.0.0.1:5000/hwlab/hwlab-agent-worker:old1234" - } - ], - newImages: [ - { - name: "hwlab-agent-worker", - serviceId: "hwlab-agent-worker", - image: "127.0.0.1:5000/hwlab/hwlab-agent-worker:new5678" - } - ] - }); - - assert.equal(decision.replace, true); - assert.equal(decision.result, "planned"); - assert.equal(decision.namespace, "hwlab-dev"); - assert.equal(decision.jobName, "hwlab-agent-worker-template"); - assert.equal(decision.oldImage, "127.0.0.1:5000/hwlab/hwlab-agent-worker:old1234"); - assert.equal(decision.newImage, "127.0.0.1:5000/hwlab/hwlab-agent-worker:new5678"); -}); - -test("DEV CD transaction guard rejects direct legacy side-effect calls", () => { - assert.equal(hasDevCdTransactionEnv({}), false); - assert.equal(hasDevCdTransactionEnv({ HWLAB_CD_TRANSACTION_ID: "tx-123" }), true); - - const failure = requireDevCdTransactionForSideEffect({ - env: {}, - script: "scripts/dev-deploy-apply.mjs", - mode: "--apply" - }); - assert.deepEqual(failure, devCdTransactionGuardFailure({ - script: "scripts/dev-deploy-apply.mjs", - mode: "--apply" - })); - assert.equal(failure.error, "cd-transaction-required"); - assert.equal(failure.mutationAttempted, false); - assert.match(failure.entrypoint, /scripts\/dev-cd-apply\.mjs/u); - - assert.equal( - requireDevCdTransactionForSideEffect({ - env: { HWLAB_CD_TRANSACTION_ID: "tx-123" }, - script: "scripts/dev-deploy-apply.mjs", - mode: "--apply" - }), - null - ); -}); - -test("DEV artifact publish guard allows CI identity without CD transaction", () => { - const ciEnv = { - HWLAB_CI_ARTIFACT_RUN_ID: "ci-artifact-123", - HWLAB_CI_ARTIFACT_RUN_OWNER: "tekton" - }; - assert.equal(hasCiArtifactPublishEnv({}), false); - assert.equal(hasCiArtifactPublishEnv(ciEnv), true); - assert.equal(requireDevArtifactPublishIdentity({ - env: ciEnv, - script: "scripts/dev-artifact-publish.mjs", - mode: "--publish" - }), null); - assert.deepEqual(devArtifactProducer(ciEnv), { - kind: "ci-artifact", - runId: "ci-artifact-123", - owner: "tekton" - }); - - const failure = requireDevArtifactPublishIdentity({ - env: {}, - script: "scripts/dev-artifact-publish.mjs", - mode: "--publish" - }); - assert.deepEqual(failure, devArtifactPublishIdentityGuardFailure({ - script: "scripts/dev-artifact-publish.mjs", - mode: "--publish" - })); - assert.equal(failure.error, "artifact-publish-identity-required"); - assert.equal(failure.cdLockRequired, false); - assert.equal(failure.rolloutAttempted, false); - assert.equal(failure.mutationAttempted, false); - assert.match(failure.entrypoint, /HWLAB_CI_ARTIFACT_RUN_ID=.*dev-artifact-publish\.mjs/u); - - assert.equal( - requireDevArtifactPublishIdentity({ - env: { HWLAB_CI_ARTIFACT_RUN_ID: "ci-123" }, - script: "scripts/dev-artifact-publish.mjs", - mode: "--publish" - }), - null - ); - assert.equal( - requireDevArtifactPublishIdentity({ - env: { HWLAB_CD_TRANSACTION_ID: "tx-123" }, - script: "scripts/dev-artifact-publish.mjs", - mode: "--publish" - }), - null - ); -}); - -test("matching allowlisted suspended template Job image does not replace", () => { - const image = "127.0.0.1:5000/hwlab/hwlab-cli:73b379f"; - const decision = decideDevTemplateJobReplacement({ - jobName: "hwlab-cli-template", - jobNamespace: "hwlab-dev", - desiredSuspended: true, - liveExists: true, - liveSuspended: true, - oldImages: [{ name: "hwlab-cli", serviceId: "hwlab-cli", image }], - newImages: [{ name: "hwlab-cli", serviceId: "hwlab-cli", image }] - }); - - assert.equal(decision.replace, false); - assert.equal(decision.result, "not_needed"); -}); - -test("replacement policy refuses non-allowlisted or active Jobs", () => { - const ignored = decideDevTemplateJobReplacement({ - jobName: "some-other-job", - jobNamespace: "hwlab-dev", - desiredSuspended: true, - liveExists: true, - liveSuspended: true, - oldImages: [{ name: "worker", serviceId: "worker", image: "image:old" }], - newImages: [{ name: "worker", serviceId: "worker", image: "image:new" }] - }); - assert.equal(ignored.replace, false); - assert.equal(ignored.result, "ignored"); - - const active = decideDevTemplateJobReplacement({ - jobName: "hwlab-cli-template", - jobNamespace: "hwlab-dev", - desiredSuspended: true, - liveExists: true, - liveSuspended: false, - oldImages: [{ name: "hwlab-cli", serviceId: "hwlab-cli", image: "image:old" }], - newImages: [{ name: "hwlab-cli", serviceId: "hwlab-cli", image: "image:new" }] - }); - assert.equal(active.replace, false); - assert.equal(active.result, "blocked"); -}); - -test("allowlisted stale simulator Deployment plans cleanup only with StatefulSet replacement", () => { - const decision = decideDevLegacySimulatorDeploymentCleanup({ - deploymentName: "hwlab-box-simu", - deploymentNamespace: "hwlab-dev", - desiredReplacementExists: true, - desiredReplacementKind: "StatefulSet", - liveExists: true, - oldImages: [ - { - name: "hwlab-box-simu", - serviceId: "hwlab-box-simu", - image: "127.0.0.1:5000/hwlab/hwlab-box-simu:old1234" - } - ], - newImages: [ - { - name: "hwlab-box-simu", - serviceId: "hwlab-box-simu", - image: "127.0.0.1:5000/hwlab/hwlab-box-simu:new5678" - } - ] - }); - - assert.equal(decision.cleanup, true); - assert.equal(decision.result, "planned"); - assert.equal(decision.namespace, "hwlab-dev"); - assert.equal(decision.deploymentName, "hwlab-box-simu"); - assert.equal(decision.serviceId, "hwlab-box-simu"); - assert.equal(decision.desiredReplacementKind, "StatefulSet"); - assert.equal(decision.oldImage, "127.0.0.1:5000/hwlab/hwlab-box-simu:old1234"); - assert.equal(decision.newImage, "127.0.0.1:5000/hwlab/hwlab-box-simu:new5678"); -}); - -test("legacy simulator cleanup refuses non-DEV or missing replacement cleanup", () => { - const wrongNamespace = decideDevLegacySimulatorDeploymentCleanup({ - deploymentName: "hwlab-gateway-simu", - deploymentNamespace: "hwlab-prod", - desiredReplacementExists: true, - desiredReplacementKind: "StatefulSet", - liveExists: true - }); - assert.equal(wrongNamespace.cleanup, false); - assert.equal(wrongNamespace.result, "blocked"); - - const missingReplacement = decideDevLegacySimulatorDeploymentCleanup({ - deploymentName: "hwlab-gateway-simu", - deploymentNamespace: "hwlab-dev", - desiredReplacementExists: false, - desiredReplacementKind: null, - liveExists: true - }); - assert.equal(missingReplacement.cleanup, false); - assert.equal(missingReplacement.result, "blocked"); - - const ignored = decideDevLegacySimulatorDeploymentCleanup({ - deploymentName: "hwlab-cloud-api", - deploymentNamespace: "hwlab-dev", - desiredReplacementExists: true, - desiredReplacementKind: "StatefulSet", - liveExists: true - }); - assert.equal(ignored.cleanup, false); - assert.equal(ignored.result, "ignored"); -}); - -test("unmanaged hotfix Deployment cleanup plans recreation when live has extra hotfix mounts", () => { - const desired = codeAgentWorkloads().items[0]; - const live = structuredClone(desired); - live.spec.template.metadata = { - annotations: { - "hwlab.pikastech.local/runtime-hotfix": "2026-05-24" - } - }; - live.spec.template.spec.volumes.unshift({ - name: "hwlab-code-agent-hotfix-20260524", - configMap: { name: "hwlab-code-agent-hotfix-20260524" } - }); - live.spec.template.spec.containers[0].volumeMounts.unshift({ - name: "hwlab-code-agent-hotfix-20260524", - mountPath: "/app/internal/cloud/server.mjs", - subPath: "server.mjs", - readOnly: true - }); - - const decision = decideDevUnmanagedHotfixDeploymentCleanup({ - deploymentName: "hwlab-cloud-api", - deploymentNamespace: "hwlab-dev", - desiredDeployment: desired, - liveDeployment: live - }); - - assert.equal(decision.cleanup, true); - assert.equal(decision.result, "planned"); - assert.equal(decision.unmanagedVolumes[0].configMap, "hwlab-code-agent-hotfix-20260524"); - assert.equal(decision.unmanagedVolumeMounts[0].mountPath, "/app/internal/cloud/server.mjs"); - assert.equal(decision.unmanagedTemplateAnnotations[0].key, "hwlab.pikastech.local/runtime-hotfix"); -}); - -test("unmanaged hotfix Deployment cleanup ignores fields already present in desired source", () => { - const desired = codeAgentWorkloads().items[0]; - desired.spec.template.spec.volumes.unshift({ - name: "hwlab-code-agent-hotfix-20260524", - configMap: { name: "hwlab-code-agent-hotfix-20260524" } - }); - desired.spec.template.spec.containers[0].volumeMounts.unshift({ - name: "hwlab-code-agent-hotfix-20260524", - mountPath: "/app/internal/cloud/server.mjs", - subPath: "server.mjs", - readOnly: true - }); - const live = structuredClone(desired); - - const decision = decideDevUnmanagedHotfixDeploymentCleanup({ - deploymentName: "hwlab-cloud-api", - deploymentNamespace: "hwlab-dev", - desiredDeployment: desired, - liveDeployment: live - }); - - assert.equal(decision.cleanup, false); - assert.equal(decision.result, "not_needed"); -}); - -test("apply source commit follows deploy/catalog artifact identity", () => { - assert.equal( - resolveApplySourceCommit({ commitId: "73b379f" }, { commitId: "73b379f" }, "cb35ada68606"), - "73b379f" - ); - assert.equal( - resolveApplySourceCommit({ commitId: "73b379f" }, { commitId: "deadbee" }, "cb35ada68606"), - "cb35ada68606" - ); -}); - -test("runtime identity env comparator accepts matching commit image and tag", () => { - const comparison = compareRuntimeIdentityEnv( - { - HWLAB_COMMIT_ID: "7de6edd", - HWLAB_IMAGE: "127.0.0.1:5000/hwlab/hwlab-cloud-web:7de6edd", - HWLAB_IMAGE_TAG: "7de6edd" - }, - { - HWLAB_COMMIT_ID: "7de6edd2c41fb50dcb007353780338be8832b27e", - HWLAB_IMAGE: "127.0.0.1:5000/hwlab/hwlab-cloud-web:7de6edd", - HWLAB_IMAGE_TAG: "7de6edd" - } - ); - - assert.equal(comparison.status, "match"); - assert.equal(comparison.matchesDesired, true); - assert.deepEqual(comparison.drift, []); -}); - -test("runtime identity env comparator detects stale cloud-web env drift", () => { - const comparison = compareRuntimeIdentityEnv( - { - HWLAB_COMMIT_ID: "c7de474", - HWLAB_IMAGE: "127.0.0.1:5000/hwlab/hwlab-cloud-web:c7de474", - HWLAB_IMAGE_TAG: "c7de474" - }, - { - HWLAB_COMMIT_ID: "7de6edd", - HWLAB_IMAGE: "127.0.0.1:5000/hwlab/hwlab-cloud-web:7de6edd", - HWLAB_IMAGE_TAG: "7de6edd" - } - ); - - assert.equal(comparison.status, "drift"); - assert.equal(comparison.matchesDesired, false); - assert.deepEqual( - comparison.drift.map((field) => field.name), - ["HWLAB_COMMIT_ID", "HWLAB_IMAGE", "HWLAB_IMAGE_TAG"] - ); -}); - -test("Code Agent provider desired state preserves Secret ref and egress base URL", () => { - const inspection = inspectCodeAgentProviderDesiredState(codeAgentDeploy(), codeAgentWorkloads()); - - assert.equal(inspection.ready, true); - assert.equal(inspection.status, "pass"); - assert.equal(inspection.provider, "codex-stdio"); - assert.equal(inspection.runtimeProvider, "openai-responses"); - assert.equal(inspection.backend, "hwlab-cloud-api/codex-app-server-stdio"); - assert.equal(inspection.secretRef.present, true); - assert.equal(inspection.secretMaterialRead, false); - assert.equal(inspection.egress.deployMatchesDevProxy, true); - assert.equal(inspection.egress.workloadMatchesDevProxy, true); - assert.equal(inspection.codexHome.present, true); - assert.equal(inspection.codexHome.configMapPresent, true); - assert.equal(inspection.codexHome.authSecretPresent, true); -}); - -test("Code Agent provider desired state blocks direct public OpenAI base URL", () => { - const inspection = inspectCodeAgentProviderDesiredState( - codeAgentDeploy({ HWLAB_CODE_AGENT_OPENAI_BASE_URL: "https://api.openai.com/v1/responses" }), - codeAgentWorkloads([ - { name: "HWLAB_CODE_AGENT_OPENAI_BASE_URL", value: "https://api.openai.com/v1/responses" } - ]) - ); - - assert.equal(inspection.ready, false); - assert.equal(inspection.status, "blocked"); - assert.equal(inspection.egress.directPublicOpenAi, true); - assert.ok(inspection.missingEgressContract.some((entry) => entry.includes("api.openai.com"))); -}); - -test("Code Agent live Deployment inspection reads only env metadata", () => { - const deployment = codeAgentWorkloads().items[0]; - const inspection = inspectCodeAgentProviderLiveDeployment(deployment); - - assert.equal(inspection.ready, true); - assert.equal(inspection.secretRef.present, true); - assert.equal(inspection.secretValuesRead, false); - assert.equal(inspection.kubernetesSecretDataRead, false); - assert.equal(inspection.validationScope, "deployment-env-and-secretKeyRef-metadata-only"); -}); - -test("DEV kubeconfig selection prefers flag then HWLAB_DEV_KUBECONFIG then KUBECONFIG then D601 default", () => { - assert.deepEqual( - resolveDevKubeconfigSelection({ - flagValue: "/tmp/from-flag", - env: { - HWLAB_DEV_KUBECONFIG: "/tmp/from-hwlab", - KUBECONFIG: "/tmp/from-kubeconfig" - } - }), - { - kubeconfig: "/tmp/from-flag", - source: "flag:--kubeconfig", - paths: ["/tmp/from-flag"] - } - ); - - assert.deepEqual( - resolveDevKubeconfigSelection({ - env: { - HWLAB_DEV_KUBECONFIG: "/tmp/from-hwlab", - KUBECONFIG: "/tmp/from-kubeconfig" - } - }), - { - kubeconfig: "/tmp/from-hwlab", - source: "env:HWLAB_DEV_KUBECONFIG", - paths: ["/tmp/from-hwlab"] - } - ); - - assert.deepEqual( - resolveDevKubeconfigSelection({ - env: { - KUBECONFIG: "/tmp/from-kubeconfig" - } - }), - { - kubeconfig: "/tmp/from-kubeconfig", - source: "env:KUBECONFIG", - paths: ["/tmp/from-kubeconfig"] - } - ); - - assert.deepEqual( - resolveDevKubeconfigSelection({ env: {} }), - { - kubeconfig: "/etc/rancher/k3s/k3s.yaml", - source: "default:d601-k3s", - paths: ["/etc/rancher/k3s/k3s.yaml"] - } - ); -}); - -test("DEV kubeconfig selection keeps multi-path KUBECONFIG readable checks explicit", () => { - const delimiter = process.platform === "win32" ? ";" : ":"; - const selection = resolveDevKubeconfigSelection({ - env: { - KUBECONFIG: [`/tmp/one`, `/tmp/two`].join(delimiter) - } - }); - - assert.equal(selection.kubeconfig, [`/tmp/one`, `/tmp/two`].join(delimiter)); - assert.equal(selection.source, "env:KUBECONFIG"); - assert.deepEqual(selection.paths, ["/tmp/one", "/tmp/two"]); -}); - -test("kubectl command prefix quotes kubeconfig paths without exposing contents", () => { - assert.equal( - buildKubectlCommandPrefix("/var/lib/unidesk/code-queue/kubeconfig"), - "KUBECONFIG=/var/lib/unidesk/code-queue/kubeconfig kubectl" - ); - assert.equal( - buildKubectlCommandPrefix("/tmp/path with space/kubeconfig"), - "KUBECONFIG='/tmp/path with space/kubeconfig' kubectl" - ); -}); - -test("kubeconfig access failure names source and path without secret material", () => { - const message = formatKubeconfigAccessFailure( - { - source: "env:HWLAB_DEV_KUBECONFIG" - }, - "/missing/kubeconfig" - ); - - assert.match(message, /env:HWLAB_DEV_KUBECONFIG/u); - assert.match(message, /\/missing\/kubeconfig/u); - assert.match(message, /Secret values are never printed/u); - assert.equal(message.includes("client-key-data:"), false); -}); - -test("parseArgs reports a missing kubeconfig path as a blocking error", () => { - const args = parseArgs(["--apply", "--kubeconfig"]); - assert.equal(args.apply, true); - assert.equal(args.kubeconfigSpecified, true); - assert.deepEqual(args.errors, ["--kubeconfig requires a non-empty path value"]); -}); - -test("registry manifest URL can be redirected without changing deploy image identity", () => { - const parsed = registryManifestUrlForImage("127.0.0.1:5000/hwlab/hwlab-cloud-api:abc1234", { - registryManifestBaseUrl: "http://hwlab-registry-proxy.hwlab-dev.svc.cluster.local:5000/" - }); - - assert.equal(parsed.registry, "127.0.0.1:5000"); - assert.equal(parsed.repository, "hwlab/hwlab-cloud-api"); - assert.equal(parsed.url, "http://hwlab-registry-proxy.hwlab-dev.svc.cluster.local:5000/v2/hwlab/hwlab-cloud-api/manifests/abc1234"); - assert.equal(parsed.registryAccess.source, "flag:--registry-manifest-base-url"); -}); - -test("registry manifest verification falls back to docker host-network helper when pod loopback is blocked", async () => { - const expectedDigest = "sha256:2cc48ca2bd0e260d7ea6df1e1a791df6615a87da3d89f4b11103735a1e2d9b63"; - const blockers = []; - const commands = []; - const result = await verifyCatalogRegistryManifest( - { - serviceId: "hwlab-cloud-api", - image: "127.0.0.1:5000/hwlab/hwlab-cloud-api:fad9fc9", - imageTag: "fad9fc9", - digest: expectedDigest - }, - blockers, - { - env: { - HWLAB_DEV_REGISTRY_HOSTNET_HELPER_IMAGE: "helper:local" - }, - httpRequestText: async () => { - throw new Error("connect ECONNREFUSED 127.0.0.1:5000"); - }, - runCommand: async (command, args) => { - commands.push([command, args]); - return { - ok: true, - code: 0, - stdout: JSON.stringify({ - statusCode: 200, - headers: { "docker-content-digest": expectedDigest } - }), - stderr: "" - }; - } - } - ); - - assert.equal(result.status, "pass"); - assert.equal(result.registryAccess.mode, "docker-host-network"); - assert.equal(result.registryAccess.helperImage, "helper:local"); - assert.match(result.registryAccess.directError, /ECONNREFUSED/u); - assert.deepEqual(blockers, []); - assert.equal(commands[0][0], "docker"); - assert.deepEqual(commands[0][1].slice(0, 6), ["run", "--rm", "--pull=never", "--network", "host", "helper:local"]); -}); - -test("redaction removes common token and password material from kubectl output", () => { - const output = redactSensitiveText( - [ - "Authorization: Bearer abc.def-123", - "token: abcdef123456", - "password=plain-secret", - "client-key-data: LS0tPRIVATE", - "postgresql://hwlab:db-secret@example.invalid:5432/hwlab" - ].join("\n") - ); - - assert.equal(output.includes("abc.def-123"), false); - assert.equal(output.includes("plain-secret"), false); - assert.equal(output.includes("LS0tPRIVATE"), false); - assert.equal(output.includes("db-secret"), false); - assert.match(output, /Bearer <redacted>/u); - assert.match(output, /password=<redacted>/u); - assert.match(output, /client-key-data: <redacted>/u); - assert.match(output, /postgresql:\/\/hwlab:<redacted>@example.invalid/u); -}); diff --git a/scripts/src/dev-gate-preflight.mjs b/scripts/src/dev-gate-preflight.mjs index e23ef5f6..ac53c8c1 100644 --- a/scripts/src/dev-gate-preflight.mjs +++ b/scripts/src/dev-gate-preflight.mjs @@ -58,7 +58,7 @@ const artifactBuildInputFiles = new Set([ "package.json", "package-lock.json", "npm-shrinkwrap.json", - "scripts/dev-artifact-publish.mjs", + "scripts/g14-artifact-publish.mjs", "scripts/src/dev-artifact-services.mjs" ]); @@ -654,10 +654,10 @@ async function validateLocalContracts(reporter, contracts, optionalReports, targ function validateArtifactPublishReport(reporter, artifactReport, artifactIdentity, targetShortCommit, targetCommit, targetRef) { if (!artifactReport) { - reporter.check("dev-artifact-publish-report", "registry", "blocked", "No DEV artifact publish report is present."); + reporter.check("g14-artifact-publish-report", "registry", "blocked", "No G14 artifact publish report is present."); reporter.block({ type: "runtime_blocker", - scope: "dev-artifact-publish", + scope: "g14-artifact-publish", summary: `${defaultArtifactReportPath} is missing, so the preflight has no publish evidence for HWLAB runtime artifacts.`, nextTask: `Run the DEV artifact publish workflow and keep its temporary JSON at ${defaultArtifactReportPath}; do not commit report output.` }); @@ -701,7 +701,7 @@ function validateArtifactPublishReport(reporter, artifactReport, artifactIdentit const summary = sourceMatchesTarget ? "DEV artifact publish report covers all required enabled service IDs with immutable digests for the target commit, and records disabled service reasons." : `DEV artifact publish report covers artifact source ${artifactIdentity.artifactSource.shortCommitId}; target ${targetRef} ${targetShortCommit} has no artifact build input changes since that source.`; - reporter.check("dev-artifact-publish-report", "registry", "pass", summary, [artifactIdentity?.targetCoverage].filter(Boolean)); + reporter.check("g14-artifact-publish-report", "registry", "pass", summary, [artifactIdentity?.targetCoverage].filter(Boolean)); return; } @@ -711,14 +711,14 @@ function validateArtifactPublishReport(reporter, artifactReport, artifactIdentit ? `source states resolved: ${sourcePresentCount} source-present, ${intentionallyDisabledCount} intentionally-disabled` : "source states are not fully resolved"; reporter.check( - "dev-artifact-publish-report", + "g14-artifact-publish-report", "registry", "blocked", `DEV artifact publish report status is ${artifactPublish?.status ?? "missing"} with ${publishedCount}/${requiredServices.length || serviceCount} required services published for source ${artifactPublish?.sourceCommitId ?? artifactReport.commitId ?? "unknown"}; ${sourceSummary}.` ); reporter.block({ type: "runtime_blocker", - scope: "dev-artifact-publish", + scope: "g14-artifact-publish", summary: `${defaultArtifactReportPath} does not prove all required HWLAB service artifacts for ${targetRef} ${targetShortCommit}; current status is ${artifactPublish?.status ?? "missing"} with ${publishedCount}/${requiredServices.length || serviceCount} required services published and ${sourceSummary}.`, nextTask: "Complete DEV artifact publishing for every enabled required HWLAB service at the artifact source commit, or prove the target commit has no artifact build input changes; keep disabled services marked not_published with reasons." }); diff --git a/scripts/src/m3-io-control-e2e.mjs b/scripts/src/m3-io-control-e2e.mjs index a4a03558..35c9a9f2 100644 --- a/scripts/src/m3-io-control-e2e.mjs +++ b/scripts/src/m3-io-control-e2e.mjs @@ -230,7 +230,7 @@ export async function runM3IoControlSourceChecks({ repoRoot: root = repoRoot } = readRepoText(root, "internal/cloud/m3-io-control.mjs"), readRepoText(root, "web/hwlab-cloud-web/app.mjs"), readRepoText(root, "web/hwlab-cloud-web/index.html"), - readRepoText(root, "scripts/dev-artifact-publish.mjs"), + readRepoText(root, "scripts/artifact-publish.mjs"), readRepoText(root, "internal/dev-entrypoint/cloud-web-routes.mjs") ]); diff --git a/scripts/validate-runtime-boundary.mjs b/scripts/validate-runtime-boundary.mjs index f3a2b671..d8d7fec7 100644 --- a/scripts/validate-runtime-boundary.mjs +++ b/scripts/validate-runtime-boundary.mjs @@ -236,7 +236,7 @@ function assertDurableRuntimeRunbook(value) { } } -function assertCloudApiImageRuntimeEntrypoints({ packageJson, ciJson, artifactPublisher, devCdApplySource }) { +function assertCloudApiImageRuntimeEntrypoints({ packageJson, ciJson, artifactPublisher, gitopsRenderSource }) { const label = "cloud-api image runtime maintenance entrypoints"; for (const expected of [ "COPY internal ./internal", @@ -250,14 +250,16 @@ function assertCloudApiImageRuntimeEntrypoints({ packageJson, ciJson, artifactPu "cmd/hwlab-cloud-api/provision.mjs", "cmd/hwlab-cloud-api/migrate.mjs" ]) { - assertIncludes(devCdApplySource, expected, `${label} DEV CD job command`); assertIncludes(packageJson.scripts?.check, `node --check ${expected}`, `${label} package check`); assertIncludes(packageJson.scripts?.check, `node ${expected} --check`, `${label} package runtime check`); } + for (const expected of ["scripts/g14-artifact-publish.mjs", "runtime-dev", "runtime-prod"]) { + assertIncludes(gitopsRenderSource, expected, `${label} G14 GitOps render source`); + } assert.equal( - ciJson.commands?.some((command) => command.run === "npm run check"), + ciJson.commands?.some((command) => command.id === "g14-contract-check"), true, - `${label} CI must run npm run check` + `${label} CI must run the G14 contract check` ); } @@ -441,8 +443,8 @@ assertDurableRuntimeRunbook(await readText("docs/reference/dev-runtime-boundary. assertCloudApiImageRuntimeEntrypoints({ packageJson: await readJSON("package.json"), ciJson: await readJSON("CI.json"), - artifactPublisher: await readText("scripts/dev-artifact-publish.mjs"), - devCdApplySource: await readText("scripts/src/dev-cd-apply.mjs") + artifactPublisher: await readText("scripts/artifact-publish.mjs"), + gitopsRenderSource: await readText("scripts/g14-gitops-render.mjs") }); console.log( diff --git a/tools/hwlab-cli/lib/cicd-job-worker.mjs b/tools/hwlab-cli/lib/cicd-job-worker.mjs deleted file mode 100644 index 87d972fb..00000000 --- a/tools/hwlab-cli/lib/cicd-job-worker.mjs +++ /dev/null @@ -1,9 +0,0 @@ -import { runCicdWorker } from "./cicd-jobs.mjs"; - -const jobId = process.argv[2]; -if (!jobId) { - process.stderr.write(`${JSON.stringify({ ok: false, code: "missing-job-id", message: "usage: node tools/hwlab-cli/lib/cicd-job-worker.mjs <jobId>" }, null, 2)}\n`); - process.exitCode = 1; -} else { - process.exitCode = await runCicdWorker({ repoRoot: process.cwd(), jobId }); -} diff --git a/tools/hwlab-cli/lib/cicd-jobs.mjs b/tools/hwlab-cli/lib/cicd-jobs.mjs deleted file mode 100644 index 79920195..00000000 --- a/tools/hwlab-cli/lib/cicd-jobs.mjs +++ /dev/null @@ -1,398 +0,0 @@ -import { spawn } from "node:child_process"; -import { randomBytes } from "node:crypto"; -import { createWriteStream } from "node:fs"; -import { mkdir, open, readdir, readFile, stat, writeFile } from "node:fs/promises"; -import os from "node:os"; -import path from "node:path"; - -const stateRoot = ".state/hwlab-cicd/jobs"; -const knownKinds = new Set(["ci-publish", "dev-cd-apply"]); - -function isoNow(now) { - return (now ? now() : new Date()).toISOString(); -} - -function compactTimestamp(value) { - return value.replace(/[-:.]/gu, "").slice(0, 15); -} - -function makeJobId(kind, now) { - return `${kind}-${compactTimestamp(isoNow(now))}-${randomBytes(3).toString("hex")}`; -} - -function jobDir(repoRoot, jobId) { - return path.join(repoRoot, stateRoot, jobId); -} - -function jobPaths(repoRoot, jobId) { - const dir = jobDir(repoRoot, jobId); - return { - dir, - specPath: path.join(dir, "spec.json"), - statePath: path.join(dir, "state.json"), - stdoutPath: path.join(dir, "stdout.log"), - stderrPath: path.join(dir, "stderr.log"), - reportPath: path.join(dir, "report.json") - }; -} - -async function readJson(filePath) { - return JSON.parse(await readFile(filePath, "utf8")); -} - -async function writeJsonFile(filePath, value) { - await writeFile(filePath, `${JSON.stringify(value, null, 2)}\n`, "utf8"); -} - -function result(exitCode, payload, stream = "stdout") { - return { exitCode, payload, stream }; -} - -function error(code, message, details = {}) { - return { ok: false, code, message, ...details }; -} - -function followup(jobId) { - return { - status: `hwlab-cli cicd status ${jobId}`, - logs: `hwlab-cli cicd logs ${jobId}`, - report: `hwlab-cli cicd report ${jobId}` - }; -} - -function optionValue(options, name, fallback = null) { - return options.get(name) ?? fallback; -} - -function positiveInteger(value, fallback) { - if (value == null) return fallback; - const parsed = Number.parseInt(value, 10); - return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; -} - -function buildJobSpec({ kind, flags, options, repoRoot, env, now }) { - if (!knownKinds.has(kind)) { - return { error: error("unsupported-cicd-kind", "Use --kind ci-publish or --kind dev-cd-apply.", { kind }) }; - } - - const jobId = makeJobId(kind, now); - const paths = jobPaths(repoRoot, jobId); - const envPatch = {}; - const args = []; - - if (kind === "ci-publish") { - args.push("scripts/dev-artifact-publish.mjs", "--publish", "--report", paths.reportPath, "--quiet-build"); - const services = optionValue(options, "--services"); - if (services) args.push("--services", services); - const concurrency = positiveInteger(optionValue(options, "--concurrency"), 4); - args.push("--concurrency", String(concurrency)); - envPatch.HWLAB_CI_ARTIFACT_RUN_ID = optionValue(options, "--owner-task-id", jobId); - envPatch.HWLAB_CI_ARTIFACT_RUN_OWNER = optionValue(options, "--owner", "hwlab-cli"); - } else { - if (!flags.has("--confirm-dev") || !flags.has("--confirmed-non-production")) { - return { - error: error("confirmation-required", "DEV CD apply requires --confirm-dev --confirmed-non-production.", { - requiredFlags: ["--confirm-dev", "--confirmed-non-production"] - }) - }; - } - args.push( - "scripts/dev-cd-apply.mjs", - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--owner-task-id", - optionValue(options, "--owner-task-id", jobId), - "--report", - paths.reportPath - ); - if (flags.has("--break-stale-lock")) args.push("--break-stale-lock"); - const concurrency = optionValue(options, "--concurrency"); - if (concurrency) envPatch.HWLAB_DEV_CD_CONCURRENCY = String(positiveInteger(concurrency, 4)); - const registryManifestBaseUrl = optionValue(options, "--registry-manifest-base-url"); - if (registryManifestBaseUrl) args.push("--registry-manifest-base-url", registryManifestBaseUrl); - } - - if (optionValue(options, "--target-ref")) args.push("--target-ref", optionValue(options, "--target-ref")); - if (optionValue(options, "--kubeconfig")) args.push("--kubeconfig", optionValue(options, "--kubeconfig")); - - return { - spec: { - jobId, - kind, - cwd: repoRoot, - node: process.execPath, - command: process.execPath, - args, - envPatch, - createdAt: isoNow(now), - host: os.hostname(), - paths - }, - paths, - env - }; -} - -async function defaultSpawnDetached({ repoRoot, jobId, env }) { - const workerPath = path.join(repoRoot, "tools/hwlab-cli/lib/cicd-job-worker.mjs"); - const child = spawn(process.execPath, [workerPath, jobId], { - cwd: repoRoot, - env, - detached: true, - stdio: "ignore" - }); - child.unref(); - return { pid: child.pid }; -} - -async function submitJob(ctx) { - const kind = optionValue(ctx.options, "--kind"); - const built = buildJobSpec({ ...ctx, kind }); - if (built.error) return result(1, built.error, "stderr"); - - await mkdir(built.paths.dir, { recursive: true }); - await writeJsonFile(built.paths.specPath, built.spec); - const initialState = { - ok: true, - jobId: built.spec.jobId, - kind: built.spec.kind, - status: "submitted", - nonBlocking: true, - createdAt: built.spec.createdAt, - updatedAt: built.spec.createdAt, - cwd: built.spec.cwd, - command: [path.basename(built.spec.command), ...built.spec.args].join(" "), - statePath: built.paths.statePath, - stdoutPath: built.paths.stdoutPath, - stderrPath: built.paths.stderrPath, - reportPath: built.paths.reportPath - }; - await writeJsonFile(built.paths.statePath, initialState); - - const launcher = ctx.spawnDetached ?? defaultSpawnDetached; - const launched = await launcher({ repoRoot: ctx.repoRoot, jobId: built.spec.jobId, env: ctx.env, spec: built.spec }); - const submitted = { - ...initialState, - pid: launched?.pid ?? null, - followup: followup(built.spec.jobId) - }; - await writeJsonFile(built.paths.statePath, submitted); - return result(0, submitted); -} - -async function appendLog(filePath, line) { - await writeFile(filePath, `${line}\n`, { flag: "a" }); -} - -function reportSummary(report) { - if (report?.artifactPublish) { - return { - kind: "ci-publish", - status: report.artifactPublish.status ?? report.status ?? null, - sourceCommitId: report.sourceCommitId ?? report.commitId ?? null, - serviceCount: report.artifactPublish.serviceCount ?? null, - requiredServiceCount: report.artifactPublish.requiredServiceCount ?? null, - publishedCount: report.artifactPublish.publishedCount ?? null, - timings: report.artifactPublish.timings ?? null, - blockers: report.blockers ?? [] - }; - } - if (report?.devCdApply) { - return { - kind: "dev-cd-apply", - status: report.status ?? null, - commitId: report.commitId ?? null, - target: report.target ?? null, - transaction: report.transaction - ? { - transactionId: report.transaction.transactionId ?? null, - ownerTaskId: report.transaction.ownerTaskId ?? null, - phases: report.transaction.phases ?? [], - release: report.transaction.release ?? null - } - : null, - steps: report.devCdApply.steps?.map((step) => ({ id: step.id, status: step.status, reportPath: step.reportPath })) ?? [], - blockers: report.blockers ?? [] - }; - } - return { - kind: "generic", - status: report?.status ?? report?.ok ?? null, - keys: report && typeof report === "object" ? Object.keys(report).slice(0, 20) : [] - }; -} - -async function runJob(ctx) { - const jobId = ctx.rest[0]; - if (!jobId) return result(1, error("missing-job-id", "usage: hwlab-cli cicd run-job <jobId>"), "stderr"); - const paths = jobPaths(ctx.repoRoot, jobId); - const spec = await readJson(paths.specPath); - const startedAt = isoNow(ctx.now); - await writeJsonFile(paths.statePath, { - ok: true, - jobId, - kind: spec.kind, - status: "running", - nonBlocking: true, - createdAt: spec.createdAt, - startedAt, - updatedAt: startedAt, - cwd: spec.cwd, - pid: process.pid, - command: [path.basename(spec.command), ...spec.args].join(" "), - statePath: paths.statePath, - stdoutPath: paths.stdoutPath, - stderrPath: paths.stderrPath, - reportPath: paths.reportPath - }); - await appendLog(paths.stdoutPath, `[hwlab-cli] job ${jobId} started at ${startedAt}`); - - const child = spawn(spec.command, spec.args, { - cwd: spec.cwd, - env: { ...process.env, ...spec.envPatch }, - stdio: ["ignore", "pipe", "pipe"] - }); - const stdoutStream = createWriteStream(paths.stdoutPath, { flags: "a" }); - const stderrStream = createWriteStream(paths.stderrPath, { flags: "a" }); - child.stdout.pipe(stdoutStream); - child.stderr.pipe(stderrStream); - const exitCode = await new Promise((resolve) => { - child.on("close", resolve); - }); - stdoutStream.end(); - stderrStream.end(); - - const finishedAt = isoNow(ctx.now); - const state = await readJson(paths.statePath); - let summary = null; - try { - summary = reportSummary(await readJson(paths.reportPath)); - } catch { - summary = null; - } - const finalState = { - ...state, - status: exitCode === 0 ? "succeeded" : "failed", - ok: exitCode === 0, - exitCode, - updatedAt: finishedAt, - finishedAt, - durationMs: Date.parse(finishedAt) - Date.parse(startedAt), - reportSummary: summary, - followup: followup(jobId) - }; - await writeJsonFile(paths.statePath, finalState); - return result(exitCode === 0 ? 0 : 2, finalState, exitCode === 0 ? "stdout" : "stderr"); -} - -async function readState(repoRoot, jobId) { - const paths = jobPaths(repoRoot, jobId); - const state = await readJson(paths.statePath); - return { paths, state }; -} - -async function statusJob(ctx) { - const jobId = ctx.rest[0]; - if (!jobId) return result(1, error("missing-job-id", "usage: hwlab-cli cicd status <jobId>"), "stderr"); - try { - const { state } = await readState(ctx.repoRoot, jobId); - return result(0, { ...state, followup: followup(jobId) }); - } catch (err) { - return result(2, error("job-not-found", `No CI/CD job state found for ${jobId}.`, { detail: err.message }), "stderr"); - } -} - -async function tailFile(filePath, bytes) { - try { - const info = await stat(filePath); - const length = Math.min(info.size, bytes); - const handle = await open(filePath, "r"); - try { - const buffer = Buffer.alloc(length); - await handle.read(buffer, 0, length, info.size - length); - return { path: filePath, totalBytes: info.size, returnedBytes: length, truncated: info.size > length, text: buffer.toString("utf8") }; - } finally { - await handle.close(); - } - } catch { - return { path: filePath, totalBytes: 0, returnedBytes: 0, truncated: false, text: "" }; - } -} - -async function logsJob(ctx) { - const jobId = ctx.rest[0]; - if (!jobId) return result(1, error("missing-job-id", "usage: hwlab-cli cicd logs <jobId>"), "stderr"); - const bytes = Math.min(positiveInteger(optionValue(ctx.options, "--tail-bytes"), 8000), 20000); - try { - const { paths, state } = await readState(ctx.repoRoot, jobId); - return result(0, { - ok: true, - jobId, - status: state.status, - tailBytes: bytes, - stdout: await tailFile(paths.stdoutPath, bytes), - stderr: await tailFile(paths.stderrPath, bytes), - followup: followup(jobId) - }); - } catch (err) { - return result(2, error("job-not-found", `No CI/CD job state found for ${jobId}.`, { detail: err.message }), "stderr"); - } -} - -async function reportJob(ctx) { - const jobId = ctx.rest[0]; - if (!jobId) return result(1, error("missing-job-id", "usage: hwlab-cli cicd report <jobId>"), "stderr"); - try { - const { paths, state } = await readState(ctx.repoRoot, jobId); - const report = await readJson(paths.reportPath); - return result(0, { - ok: true, - jobId, - status: state.status, - reportPath: paths.reportPath, - summary: reportSummary(report), - report: ctx.flags.has("--full") ? report : undefined, - followup: followup(jobId) - }); - } catch (err) { - return result(2, error("report-not-ready", `No report is available for ${jobId}.`, { detail: err.message }), "stderr"); - } -} - -async function listJobs(ctx) { - const root = path.join(ctx.repoRoot, stateRoot); - const limit = Math.min(positiveInteger(optionValue(ctx.options, "--limit"), 20), 100); - try { - const names = (await readdir(root)).sort().reverse().slice(0, limit); - const jobs = []; - for (const name of names) { - try { - jobs.push((await readState(ctx.repoRoot, name)).state); - } catch { - // Ignore incomplete job directories; logs/report still remain on disk. - } - } - return result(0, { ok: true, command: "hwlab-cli cicd jobs", count: jobs.length, jobs }); - } catch { - return result(0, { ok: true, command: "hwlab-cli cicd jobs", count: 0, jobs: [] }); - } -} - -export async function runCicdCommand(ctx) { - if (ctx.subcommand === "submit") return submitJob(ctx); - if (ctx.subcommand === "status") return statusJob(ctx); - if (ctx.subcommand === "logs") return logsJob(ctx); - if (ctx.subcommand === "report") return reportJob(ctx); - if (ctx.subcommand === "jobs") return listJobs(ctx); - return result(1, error("usage", "usage: hwlab-cli cicd submit --kind <ci-publish|dev-cd-apply> | status|logs|report <jobId> | jobs"), "stderr"); -} - -export async function runCicdWorker({ repoRoot = process.cwd(), jobId, now }) { - const outcome = await runJob({ - repoRoot, - rest: [jobId], - now - }); - return outcome.exitCode; -} diff --git a/tools/hwlab-cli/lib/cli.mjs b/tools/hwlab-cli/lib/cli.mjs index 1a077ba0..bb883db7 100644 --- a/tools/hwlab-cli/lib/cli.mjs +++ b/tools/hwlab-cli/lib/cli.mjs @@ -1,6 +1,5 @@ import { DEV_ENDPOINT, loadMvpGateSummary } from "../../../internal/mvp-gate/summary.mjs"; import { runM3IoSkillCommand } from "../../../skills/hwlab-agent-runtime/scripts/src/m3-io-skill-client.mjs"; -import { runCicdCommand } from "./cicd-jobs.mjs"; function formatJson(value) { return JSON.stringify(value, null, 2); @@ -174,12 +173,7 @@ function writeHelp(stdout) { "m3 status --api-base-url URL", "m3 io --action do.write --value true --approved --api-base-url URL", "m3 io --action di.read --api-base-url URL", - "cicd submit --kind ci-publish --concurrency 4", - "cicd submit --kind dev-cd-apply --confirm-dev --confirmed-non-production --concurrency 4", - "cicd status JOB_ID", - "cicd logs JOB_ID --tail-bytes 8000", - "cicd report JOB_ID", - "cicd jobs --limit 20", + "G14 CI/CD is managed by native Tekton + GitOps; hwlab-cli cicd is removed", "test e2e --env dev --mvp", "test e2e --env dev --mvp --dry-run", "test e2e --env dev --mvp --live --confirm-dev --confirmed-non-production" @@ -202,18 +196,15 @@ export async function runCli(argv, io) { const { flags, options } = parseArgs(rest); if (command === "cicd") { - const outcome = await runCicdCommand({ - subcommand, - rest, - flags, - options, - repoRoot, - env, - now: io.now, - spawnDetached: io.spawnDetached - }); - writeJson(outcome.stream === "stderr" ? stderr : stdout, outcome.payload); - return outcome.exitCode; + writeJson(stderr, makeError("legacy-cicd-removed", "hwlab-cli cicd was removed from the active G14 release path. Use G14 k3s Tekton/GitOps via tran G14:k3s and scripts/g14-gitops-render.mjs.", { + subcommand: subcommand ?? null, + replacement: { + observePipelineRuns: "tran G14:k3s kubectl get pipelineruns -n hwlab-ci", + observeArgo: "tran G14:k3s kubectl -n argocd get application hwlab-g14-dev hwlab-g14-prod", + renderSource: "node scripts/g14-gitops-render.mjs --check" + } + })); + return 2; } if (command === "m3" && ["io", "status"].includes(subcommand)) { diff --git a/tools/hwlab-cli/lib/cli.test.mjs b/tools/hwlab-cli/lib/cli.test.mjs index 34e62955..0bed6773 100644 --- a/tools/hwlab-cli/lib/cli.test.mjs +++ b/tools/hwlab-cli/lib/cli.test.mjs @@ -1,7 +1,4 @@ import assert from "node:assert/strict"; -import { mkdtemp, readFile, writeFile } from "node:fs/promises"; -import os from "node:os"; -import path from "node:path"; import test from "node:test"; import { @@ -34,80 +31,14 @@ async function captureCli(args, options = {}) { return { exitCode, stdout, stderr }; } -test("repo hwlab-cli cicd submit is non-blocking and records follow-up commands", async () => { - const cwd = await mkdtemp(path.join(os.tmpdir(), "hwlab-cli-cicd-")); - const launched = []; - const result = await captureCli( - ["cicd", "submit", "--kind", "ci-publish", "--services", "hwlab-cloud-web", "--concurrency", "2"], - { - cwd, - spawnDetached: async (job) => { - launched.push(job); - return { pid: 123456 }; - } - } - ); - - assert.equal(result.exitCode, 0, result.stderr); - const body = JSON.parse(result.stdout); - assert.equal(body.ok, true); - assert.equal(body.nonBlocking, true); - assert.equal(body.kind, "ci-publish"); - assert.match(body.jobId, /^ci-publish-/u); - assert.equal(body.pid, 123456); - assert.equal(body.followup.status, `hwlab-cli cicd status ${body.jobId}`); - assert.equal(launched.length, 1); - - const spec = JSON.parse(await readFile(path.join(cwd, ".state", "hwlab-cicd", "jobs", body.jobId, "spec.json"), "utf8")); - assert.deepEqual(spec.args.slice(0, 4), ["scripts/dev-artifact-publish.mjs", "--publish", "--report", body.reportPath]); - assert.ok(spec.args.includes("--quiet-build")); - assert.equal(spec.envPatch.HWLAB_CI_ARTIFACT_RUN_ID, body.jobId); -}); - -test("repo hwlab-cli cicd status/logs/report disclose bounded job state", async () => { - const cwd = await mkdtemp(path.join(os.tmpdir(), "hwlab-cli-cicd-")); - const submit = await captureCli(["cicd", "submit", "--kind", "ci-publish"], { - cwd, - spawnDetached: async () => ({ pid: 123457 }) - }); - const submitted = JSON.parse(submit.stdout); - const jobDir = path.join(cwd, ".state", "hwlab-cicd", "jobs", submitted.jobId); - await writeFile(path.join(jobDir, "stdout.log"), "build line 1\nbuild line 2\n", "utf8"); - await writeFile(path.join(jobDir, "stderr.log"), "warning line\n", "utf8"); - await writeFile( - path.join(jobDir, "report.json"), - `${JSON.stringify({ - sourceCommitId: "abc123", - artifactPublish: { - status: "published", - serviceCount: 1, - requiredServiceCount: 1, - publishedCount: 1, - timings: { durationMs: 42 } - }, - blockers: [] - })}\n`, - "utf8" - ); - - const status = await captureCli(["cicd", "status", submitted.jobId], { cwd }); - assert.equal(status.exitCode, 0, status.stderr); - assert.equal(JSON.parse(status.stdout).status, "submitted"); - - const logs = await captureCli(["cicd", "logs", submitted.jobId, "--tail-bytes", "12"], { cwd }); - assert.equal(logs.exitCode, 0, logs.stderr); - const logBody = JSON.parse(logs.stdout); - assert.equal(logBody.stdout.returnedBytes, 12); - assert.equal(logBody.stdout.truncated, true); - assert.ok(logBody.stdout.text.includes("line 2")); - - const report = await captureCli(["cicd", "report", submitted.jobId], { cwd }); - assert.equal(report.exitCode, 0, report.stderr); - const reportBody = JSON.parse(report.stdout); - assert.equal(reportBody.summary.kind, "ci-publish"); - assert.equal(reportBody.summary.status, "published"); - assert.equal(reportBody.summary.publishedCount, 1); - assert.equal(Object.hasOwn(reportBody, "report"), false); +test("repo hwlab-cli cicd is removed from the active G14 release path", async () => { + const result = await captureCli(["cicd", "submit", "--kind", "ci-publish"]); + assert.equal(result.exitCode, 2); + const body = JSON.parse(result.stderr); + assert.equal(body.ok, false); + assert.equal(body.code, "legacy-cicd-removed"); + assert.match(body.message, /G14 release path/u); + assert.match(body.replacement.observePipelineRuns, /tran G14:k3s/u); }); test("repo hwlab-cli m3 status uses Skill CLI and calls only HWLAB API /v1/m3/status", async () => { diff --git a/tools/hwlab-cli/package.json b/tools/hwlab-cli/package.json index 483ff8b1..29e2f8f5 100644 --- a/tools/hwlab-cli/package.json +++ b/tools/hwlab-cli/package.json @@ -7,7 +7,7 @@ "hwlab-cli": "./bin/hwlab-cli.mjs" }, "scripts": { - "check": "node --check bin/hwlab-cli.mjs && node --check lib/cli.mjs && node --check lib/cicd-jobs.mjs && node --check lib/cicd-job-worker.mjs && node --test lib/cli.test.mjs && node ../../scripts/l6-cli-web-smoke.mjs", + "check": "node --check bin/hwlab-cli.mjs && node --check lib/cli.mjs && node --test lib/cli.test.mjs && node ../../scripts/l6-cli-web-smoke.mjs", "health": "node bin/hwlab-cli.mjs health" } } diff --git a/web/hwlab-cloud-web/scripts/check.mjs b/web/hwlab-cloud-web/scripts/check.mjs index 17891eb1..cdb62c49 100644 --- a/web/hwlab-cloud-web/scripts/check.mjs +++ b/web/hwlab-cloud-web/scripts/check.mjs @@ -56,7 +56,7 @@ const helpMarkdown = fs.readFileSync(path.resolve(rootDir, "help.md"), "utf8"); const buildScript = fs.readFileSync(path.resolve(rootDir, "scripts/build.mjs"), "utf8"); const distContractScript = fs.readFileSync(path.resolve(rootDir, "scripts/dist-contract.mjs"), "utf8"); const markedLicense = fs.readFileSync(path.resolve(rootDir, "third_party/marked/LICENSE"), "utf8"); -const artifactPublisher = fs.readFileSync(path.resolve(repoRoot, "scripts/dev-artifact-publish.mjs"), "utf8"); +const artifactPublisher = fs.readFileSync(path.resolve(repoRoot, "scripts/artifact-publish.mjs"), "utf8"); const cloudWebRoutes = fs.readFileSync(path.resolve(repoRoot, "internal/dev-entrypoint/cloud-web-routes.mjs"), "utf8"); const cloudWebProxy = fs.readFileSync(path.resolve(repoRoot, "internal/dev-entrypoint/cloud-web-proxy.mjs"), "utf8"); const cloudApiServer = fs.readFileSync(path.resolve(repoRoot, "internal/cloud/server.mjs"), "utf8"); diff --git a/web/hwlab-cloud-web/scripts/m3-control-panel-guard.mjs b/web/hwlab-cloud-web/scripts/m3-control-panel-guard.mjs index dce8a59e..eb9c04ae 100644 --- a/web/hwlab-cloud-web/scripts/m3-control-panel-guard.mjs +++ b/web/hwlab-cloud-web/scripts/m3-control-panel-guard.mjs @@ -28,7 +28,7 @@ export function runM3ControlPanelGuard() { const html = readText("web/hwlab-cloud-web/index.html"); const app = readText("web/hwlab-cloud-web/app.mjs"); const styles = readText("web/hwlab-cloud-web/styles.css"); - const artifactPublisher = readText("scripts/dev-artifact-publish.mjs"); + const artifactPublisher = readText("scripts/artifact-publish.mjs"); const cloudWebRouteSource = readText("internal/dev-entrypoint/cloud-web-routes.mjs"); const browserSource = `${html}\n${app}\n${styles}`;