diff --git a/AGENTS.md b/AGENTS.md index 1bd6cbee..acba6672 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -45,6 +45,7 @@ HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥 - Cloud Web M3 只读护栏：`npm run web:m3-readonly` - Cloud Workbench 布局/遮挡 smoke：`node scripts/dev-cloud-workbench-layout-smoke.mjs --static` - DEV artifact 发布预检：`npm run dev-artifact:preflight` +- DEV CD 单事务发布/应用/验证：`npm run dev-cd:apply` - runner GitHub 可见性预检：`npm run runner:issue-visibility:preflight` - D601 k3s 只读观测：`npm run d601:k3s:readonly` diff --git a/docs/dev-artifact-publish.md b/docs/dev-artifact-publish.md index 347ede4b..86b3891c 100644 --- a/docs/dev-artifact-publish.md +++ b/docs/dev-artifact-publish.md @@ -1,8 +1,11 @@ # HWLAB DEV Artifact Publish -`scripts/dev-artifact-publish.mjs` is the DEV-only build and publish path for -D601 local/internal registry artifacts. It does not deploy workloads, read -secrets, enable PROD, or push to GHCR/Docker Hub/other third-party registries. +`scripts/dev-artifact-publish.mjs` is the DEV-only artifact preflight/build +backend for D601 local/internal registry artifacts. Publish side effects are +normally entered through `scripts/dev-cd-apply.mjs`, which sets the transaction +environment and serializes publish/apply/verify with the DEV CD Lease lock. +The artifact script does not deploy workloads, read secrets, enable PROD, or +push to GHCR/Docker Hub/other third-party registries. ## Scope @@ -58,24 +61,35 @@ Build only: node scripts/dev-artifact-publish.mjs --build ``` -Build and publish to the D601 local/internal registry: +Build and publish to the D601 local/internal registry inside the DEV CD +transaction: + +```sh +node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report +``` + +The underlying single-step command is kept only for transaction internals and +specialized tests that explicitly inject the transaction environment: ```sh node scripts/dev-artifact-publish.mjs --publish ``` +Direct `--publish` without `HWLAB_CD_TRANSACTION_ID` is rejected with +`cd-transaction-required`; this is a bypass guard, not a second lock. + Daily Cloud Web source validation does not require ignored `dist/` to exist: ```sh node web/hwlab-cloud-web/scripts/check.mjs ``` -For `hwlab-cloud-web`, the build/publish path refreshes and verifies the static -bundle before image build: +For `hwlab-cloud-web`, the build path refreshes and verifies the static bundle +before image build; publish still goes through the transaction command above: ```sh node web/hwlab-cloud-web/scripts/build.mjs -node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web --report reports/dev-gate/dev-artifacts-hwlab-cloud-web-.json +node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report ``` The runtime wrapper serves `/app/web/hwlab-cloud-web/dist` before the source diff --git a/docs/dev-deploy-apply.md b/docs/dev-deploy-apply.md index dbcfcef8..89d9444d 100644 --- a/docs/dev-deploy-apply.md +++ b/docs/dev-deploy-apply.md @@ -1,9 +1,11 @@ # HWLAB DEV Deploy Apply -This document records the DEV-only apply/preflight path for +This document records the DEV-only apply/preflight backend for `pikasTech/HWLAB#33` and the apply-plan/report path for -`pikasTech/HWLAB#50`. It targets only the D601 `hwlab-dev` namespace and must -not be used for PROD. +`pikasTech/HWLAB#50`. Real apply side effects are normally entered through +`scripts/dev-cd-apply.mjs`, which sets the transaction environment and +serializes publish/apply/verify with the DEV CD Lease lock. The apply backend +targets only the D601 `hwlab-dev` namespace and must not be used for PROD. ## Scope @@ -56,13 +58,22 @@ node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run ``` Only after CI publish evidence and executor prerequisites exist, a real DEV -apply requires explicit non-production confirmation: +CD transaction is: + +```sh +node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report +``` + +The underlying apply backend still requires explicit non-production +confirmation and a transaction environment: ```sh node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report ``` -The script refuses PROD flags, secret-read flags, heavyweight e2e flags, and +Direct `--apply` without `HWLAB_CD_TRANSACTION_ID` is rejected with +`cd-transaction-required`; this is a bypass guard, not a second lock. The +script refuses PROD flags, secret-read flags, heavyweight e2e flags, and force-push flags. It also refuses to apply if the DEV artifact catalog still contains skeleton-only unpublished images. @@ -176,7 +187,7 @@ node scripts/dev-deploy-apply.mjs --dry-run --write-report node scripts/validate-dev-gate-report.mjs ``` -Only after that approval may the DEV-only apply command be used. +Only after that approval may the DEV CD transaction command be used. ## Cloud Web Manual DEV Rollout @@ -202,7 +213,7 @@ freshness gates. ```sh node web/hwlab-cloud-web/scripts/check.mjs node web/hwlab-cloud-web/scripts/build.mjs -node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web --report reports/dev-gate/dev-artifacts-hwlab-cloud-web-.json +node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report ``` Probe the image locally before touching k3s: diff --git a/docs/reference/README.md b/docs/reference/README.md index 9e4984d7..e2fa18b8 100644 --- a/docs/reference/README.md +++ b/docs/reference/README.md @@ -20,7 +20,7 @@ | 文档治理与 docs-spec 本地权威 | [documentation-governance.md](documentation-governance.md) | | 架构和 M3 上位约束 | [architecture.md](architecture.md) | | DEV 运行态、端口、k3s、DB readiness 和环境边界 | [dev-runtime-boundary.md](dev-runtime-boundary.md) | -| 部署正规化、artifact 发布、回滚和 Cloud Web rollout | [deployment-publish.md](deployment-publish.md) | +| 部署正规化、DEV CD 单事务入口、artifact 发布、回滚和 Cloud Web rollout | [deployment-publish.md](deployment-publish.md) | | Cloud Workbench 默认界面和 UX 边界 | [cloud-workbench.md](cloud-workbench.md) | | Code Agent chat 同源通道 readiness 与真实回复判定 | [code-agent-chat-readiness.md](code-agent-chat-readiness.md) | | MVP E2E 验收测试与带编号测试报告 issue 规则 | [MVP-e2e-acceptance.md](MVP-e2e-acceptance.md) | diff --git a/docs/reference/deployment-publish.md b/docs/reference/deployment-publish.md index b9d4be01..44a52ac7 100644 --- a/docs/reference/deployment-publish.md +++ b/docs/reference/deployment-publish.md @@ -25,9 +25,44 @@ truth. | Stage | Current HWLAB state | Required next behavior | | --- | --- | --- | | 1. Long-term reference | This file records DEV deployment, Cloud Web publish, API/edge/health, frp, D601 registry, k3s rollout, rollback, and verification rules. `AGENTS.md` indexes it. | Keep process issues as sources only; update this reference when a deployment rule changes. | -| 2. CLI or controlled scripts | Existing controlled entrypoints cover contract planning, desired-state planning, artifact publish, DEV apply/report, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. | Converge check, plan, publish, apply, rollback, status, and verify into HWLAB CLI or a small controlled script surface backed by `deploy/deploy.json`. | +| 2. CLI or controlled scripts | `scripts/dev-cd-apply.mjs` is the only formal DEV CD side-effect entrypoint for publish, desired-state refresh, apply, and live verify. Other controlled scripts still cover contract planning, dry-run planning, preflight, single-step internals, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. | Converge check, plan, rollback, status, and verify into HWLAB CLI while preserving the single transaction lock for side effects backed by `deploy/deploy.json`. | | 3. UniDesk CI/CD plus images | DEV services are imageized and published to the D601 registry. D601 rollout still includes manual bridge steps for Cloud Web and frp/k3s execution. | Move build, publish, desired-state refresh, rollout, post-deploy validation, and report upload into standard UniDesk CI/CD jobs without replacing HWLAB runtime. | +## DEV CD Transaction + +The stable DEV CD side-effect command is: + +```sh +node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report +``` + +This command is the only formal path that may publish DEV artifacts, refresh +desired-state from the publish report, apply to `hwlab-dev`, and verify public +`16666/16667` in one run. It reads `deploy/deploy.json` at transaction start, +records the manifest hash and target commit, acquires +`Lease/hwlab-dev/hwlab-dev-cd-lock`, runs the internal steps serially, writes +`reports/dev-gate/dev-cd-apply.json`, and releases the Lease. + +The Lease is a deployment-plane mutex, not a desired-state source. Its +annotations record at least `promotionCommit`, `deployJsonHash`, +`ownerTaskId`, `transactionId`, `phase`, `startedAt`, `updatedAt`, +`ttlSeconds`, `liveBefore`, and `targetNamespace`. `deploy/deploy.json`, +`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml` +remain the only authoritative deploy inputs. + +If another transaction holds the Lease, the command must stop before publish or +apply with structured `deploy-lock-held` output that includes holder, +promotion commit, phase, target namespace, and retry seconds. Expired locks are +not silently overwritten; taking over a stale lock requires both +`--break-stale-lock` and `--confirm-dev`, and the previous holder is retained +in audit annotations/report fields. + +The old side-effect commands are now internal transaction steps. Direct manual +use of `scripts/dev-artifact-publish.mjs --publish` or +`scripts/dev-deploy-apply.mjs --apply` without `HWLAB_CD_TRANSACTION_ID` is +rejected with `cd-transaction-required`; preflight, build, dry-run, check, and +read-only modes remain available. + ## Artifact Publish DEV artifacts publish to the D601 local/internal registry: @@ -36,18 +71,22 @@ DEV artifacts publish to the D601 local/internal registry: 127.0.0.1:5000/hwlab/* ``` -The stable publish command is: +The single-step publish command is a transaction-internal side-effect step: ```sh node scripts/dev-artifact-publish.mjs --publish ``` -For a single Cloud Web image: +It only runs in side-effect mode when `HWLAB_CD_TRANSACTION_ID` is set by +`scripts/dev-cd-apply.mjs`. Use `node scripts/dev-cd-apply.mjs --apply +--confirm-dev --confirmed-non-production --write-report` for normal DEV CD. + +For Cloud Web source freshness before the transaction: ```sh node web/hwlab-cloud-web/scripts/check.mjs node web/hwlab-cloud-web/scripts/build.mjs -node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web +node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report ``` Cloud Web must build `web/hwlab-cloud-web/dist` before publish because the @@ -130,7 +169,12 @@ services may still use internal `6667`. ## Cloud Web Manual DEV Rollout -The verified manual path for `hwlab-cloud-web` is: +This section is retained as historical operational baseline for what the +transaction reproduces. New DEV side effects should use +`scripts/dev-cd-apply.mjs`; do not bypass the Lease with ad hoc publish, +`kubectl set image`, or restart commands. + +The previous verified manual path for `hwlab-cloud-web` was: ```sh cd /home/ubuntu/workspace/hwlab @@ -141,6 +185,9 @@ KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/h KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s ``` +That historical sequence is no longer the normal side-effect path; direct +`--publish` is rejected unless it is called by the transaction. + The rollout must use D601 native k3s: ```sh @@ -171,16 +218,18 @@ scroll, and an internal Markdown help surface. The frontend contract is ## Apply Boundary -`scripts/dev-deploy-apply.mjs` is the DEV-only apply/report path. A real apply -requires explicit non-production flags: +`scripts/dev-deploy-apply.mjs` is the DEV-only apply/report backend. A real +apply requires the transaction environment created by `scripts/dev-cd-apply.mjs` +plus explicit non-production flags: ```sh node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report ``` -The apply path must stop before mutation when artifact evidence, registry -digests, kube access, DEV health, or required runtime readiness are missing. -It must not read secret values or target PROD. +Direct manual `--apply` calls without `HWLAB_CD_TRANSACTION_ID` return +`cd-transaction-required`. The apply path must stop before mutation when +artifact evidence, registry digests, kube access, DEV health, or required +runtime readiness are missing. It must not read secret values or target PROD. Suspended template Jobs are immutable in `spec.template`. Only the explicit DEV allowlist may be replaced by the apply script: @@ -220,9 +269,10 @@ use the repo-owned scripts below instead of ad hoc shell fragments. | Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. | | Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. | | Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. | -| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` | CI job should run this and upload digest reports. | -| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Make refresh an explicit CLI/CD step after publish. | -| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep apply behind confirmation flags and route through CLI/CD approval. | +| Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. | +| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. | +| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Internal transaction step after publish; read/write desired-state only from repo files. | +| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` inside `HWLAB_CD_TRANSACTION_ID` | Internal transaction step; direct side-effect calls are rejected. | | Roll back DEV Deployment | `kubectl -n hwlab-dev rollout undo deployment/` with explicit D601 kubeconfig | Add `hwlab deploy rollback --env dev --service --to-revision `. | | Verify status | `curl` checks plus `node scripts/d601-k3s-readonly-observability.mjs` | Add `hwlab deploy verify --env dev --write-report`. | diff --git a/package.json b/package.json index e5fab95a..194113de 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "type": "module", "scripts": { "validate": "node scripts/validate-contract.mjs && node scripts/deploy-contract-plan.mjs --check && node scripts/deploy-desired-state-plan.mjs --check && node scripts/dev-runtime-migration.mjs --check && node --check scripts/artifact-runtime-readiness-guard.mjs && node --check scripts/src/artifact-runtime-readiness-guard.mjs && node --test scripts/artifact-runtime-readiness-guard.test.mjs", - "check": "node --check internal/protocol/index.mjs && node --check internal/agent/index.mjs && node --check internal/agent/runtime.mjs && node --check internal/audit/index.mjs && node --check internal/db/runtime-store.mjs && node --check internal/db/runtime-store.test.mjs && node --check internal/mvp-gate/summary.mjs && node --check internal/cloud/db-contract.mjs && node --check internal/dev-entrypoint/http.mjs && node --check internal/dev-entrypoint/http.test.mjs && node --check internal/cloud/code-agent-contract.mjs && node --check internal/cloud/json-rpc.mjs && node --check internal/cloud/code-agent-chat.mjs && node --check internal/cloud/m3-io-control.mjs && node --check internal/cloud/server.mjs && node --check internal/sim/model.mjs && node --check internal/sim/model.test.mjs && node --check internal/sim/http.mjs && node --check internal/sim/l2-runtime.mjs && node --check internal/patchpanel/model.mjs && node --check internal/patchpanel/runtime.mjs && node --check cmd/hwlab-cloud-api/main.mjs && node --check cmd/hwlab-edge-proxy/main.mjs && node --check cmd/hwlab-agent-mgr/main.mjs && node --check cmd/hwlab-agent-worker/main.mjs && node --check cmd/hwlab-box-simu/main.mjs && node --check cmd/hwlab-gateway-simu/main.mjs && node --check scripts/cloud-api-runtime-smoke.mjs && node --check scripts/code-agent-chat-smoke.mjs && node --check scripts/dev-edge-health-smoke.mjs && node --check scripts/src/dev-edge-health-smoke-lib.mjs && node --check scripts/validate-contract.mjs && node --check scripts/deploy-contract-plan.test.mjs && node --check scripts/deploy-desired-state-plan.mjs && node --check scripts/src/deploy-desired-state-plan.mjs && node --check scripts/artifact-runtime-readiness-guard.mjs && node --check scripts/src/artifact-runtime-readiness-guard.mjs && node --check scripts/artifact-runtime-readiness-guard.test.mjs && node --check scripts/report-lifecycle.mjs && node --check scripts/report-lifecycle.test.mjs && node --check scripts/validate-dev-gate-report.mjs && node --check scripts/dev-m3-hardware-loop-smoke.test.mjs && node --check scripts/m3-io-control-e2e.mjs && node --check scripts/src/m3-io-control-e2e.mjs && node --check scripts/m3-io-control-e2e.test.mjs && node --check scripts/dev-cloud-workbench-smoke.test.mjs && node --check scripts/validate-dev-m3-cardinality.mjs && node --check scripts/validate-dev-m3-cardinality.test.mjs && node --check scripts/validate-artifact-catalog.mjs && node --check scripts/refresh-artifact-catalog.mjs && node --check scripts/dev-artifact-publish.mjs && node --check scripts/src/dev-artifact-services.mjs && node --check scripts/src/registry-capabilities.mjs && node --check scripts/preflight-dev-base-image.mjs && node --check scripts/src/dev-base-image-preflight.mjs && node --check scripts/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.test.mjs && node --check scripts/src/dev-m4-agent-loop-smoke-lib.test.mjs && node --check scripts/d601-k3s-readonly-observability.mjs && node --check scripts/src/d601-k3s-readonly-observability.mjs && node --check scripts/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.test.mjs && node --check scripts/l2-runtime-contract-smoke.mjs && node --check scripts/patch-panel-runtime-smoke.mjs && node --check scripts/export-web-gate-summary.mjs && node --check scripts/l6-cli-web-smoke.mjs && node --check tools/hwlab-cli/bin/hwlab-cli.mjs && node --check tools/hwlab-cli/lib/cli.mjs && node --check web/hwlab-cloud-web/app.mjs && node --check web/hwlab-cloud-web/gate-summary.mjs && node --check web/hwlab-cloud-web/scripts/check.mjs && node --check web/hwlab-cloud-web/scripts/build.mjs && node --check web/hwlab-cloud-web/scripts/dist-contract.mjs && node scripts/validate-contract.mjs && node scripts/validate-dev-gate-report.mjs && node scripts/report-lifecycle.test.mjs && node scripts/validate-dev-m3-cardinality.mjs && node scripts/validate-artifact-catalog.mjs && node scripts/deploy-desired-state-plan.mjs --check && node scripts/dev-runtime-migration.mjs --check && node scripts/dev-evidence-blocker-aggregator.mjs --check && node scripts/l2-runtime-contract-smoke.mjs && node scripts/l6-cli-web-smoke.mjs && node web/hwlab-cloud-web/scripts/check.mjs && node scripts/code-agent-chat-smoke.mjs && node --test scripts/artifact-runtime-readiness-guard.test.mjs scripts/deploy-contract-plan.test.mjs scripts/dev-m3-hardware-loop-smoke.test.mjs scripts/validate-dev-m3-cardinality.test.mjs scripts/dev-cloud-workbench-smoke.test.mjs scripts/deploy-desired-state-plan.test.mjs scripts/src/dev-deploy-apply.test.mjs scripts/src/dev-runtime-migration.test.mjs scripts/src/dev-m4-agent-loop-smoke-lib.test.mjs scripts/src/dev-evidence-blocker-aggregator.test.mjs internal/agent/index.test.mjs internal/audit/index.test.mjs internal/db/schema.test.mjs internal/db/runtime-store.test.mjs internal/cloud/json-rpc.test.mjs internal/cloud/m3-io-control.test.mjs internal/cloud/server.test.mjs internal/dev-entrypoint/http.test.mjs internal/sim/model.test.mjs internal/patchpanel/model.test.mjs internal/patchpanel/runtime.test.mjs && node scripts/cloud-api-runtime-smoke.mjs && sh -n scripts/bootstrap-skills.sh scripts/worker-entrypoint.sh", + "check": "node --check internal/protocol/index.mjs && node --check internal/agent/index.mjs && node --check internal/agent/runtime.mjs && node --check internal/audit/index.mjs && node --check internal/db/runtime-store.mjs && node --check internal/db/runtime-store.test.mjs && node --check internal/mvp-gate/summary.mjs && node --check internal/cloud/db-contract.mjs && node --check internal/dev-entrypoint/http.mjs && node --check internal/dev-entrypoint/http.test.mjs && node --check internal/cloud/code-agent-contract.mjs && node --check internal/cloud/json-rpc.mjs && node --check internal/cloud/code-agent-chat.mjs && node --check internal/cloud/m3-io-control.mjs && node --check internal/cloud/server.mjs && node --check internal/sim/model.mjs && node --check internal/sim/model.test.mjs && node --check internal/sim/http.mjs && node --check internal/sim/l2-runtime.mjs && node --check internal/patchpanel/model.mjs && node --check internal/patchpanel/runtime.mjs && node --check cmd/hwlab-cloud-api/main.mjs && node --check cmd/hwlab-edge-proxy/main.mjs && node --check cmd/hwlab-agent-mgr/main.mjs && node --check cmd/hwlab-agent-worker/main.mjs && node --check cmd/hwlab-box-simu/main.mjs && node --check cmd/hwlab-gateway-simu/main.mjs && node --check scripts/cloud-api-runtime-smoke.mjs && node --check scripts/code-agent-chat-smoke.mjs && node --check scripts/dev-edge-health-smoke.mjs && node --check scripts/src/dev-edge-health-smoke-lib.mjs && node --check scripts/validate-contract.mjs && node --check scripts/deploy-contract-plan.test.mjs && node --check scripts/deploy-desired-state-plan.mjs && node --check scripts/src/deploy-desired-state-plan.mjs && node --check scripts/artifact-runtime-readiness-guard.mjs && node --check scripts/src/artifact-runtime-readiness-guard.mjs && node --check scripts/artifact-runtime-readiness-guard.test.mjs && node --check scripts/report-lifecycle.mjs && node --check scripts/report-lifecycle.test.mjs && node --check scripts/validate-dev-gate-report.mjs && node --check scripts/dev-cd-apply.mjs && node --check scripts/src/dev-cd-apply.mjs && node --check scripts/src/dev-cd-apply.test.mjs && node --check scripts/dev-m3-hardware-loop-smoke.test.mjs && node --check scripts/m3-io-control-e2e.mjs && node --check scripts/src/m3-io-control-e2e.mjs && node --check scripts/m3-io-control-e2e.test.mjs && node --check scripts/dev-cloud-workbench-smoke.test.mjs && node --check scripts/validate-dev-m3-cardinality.mjs && node --check scripts/validate-dev-m3-cardinality.test.mjs && node --check scripts/validate-artifact-catalog.mjs && node --check scripts/refresh-artifact-catalog.mjs && node --check scripts/dev-artifact-publish.mjs && node --check scripts/src/dev-artifact-services.mjs && node --check scripts/src/registry-capabilities.mjs && node --check scripts/preflight-dev-base-image.mjs && node --check scripts/src/dev-base-image-preflight.mjs && node --check scripts/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.mjs && node --check scripts/src/dev-evidence-blocker-aggregator.test.mjs && node --check scripts/src/dev-m4-agent-loop-smoke-lib.test.mjs && node --check scripts/d601-k3s-readonly-observability.mjs && node --check scripts/src/d601-k3s-readonly-observability.mjs && node --check scripts/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.mjs && node --check scripts/src/dev-runtime-migration.test.mjs && node --check scripts/l2-runtime-contract-smoke.mjs && node --check scripts/patch-panel-runtime-smoke.mjs && node --check scripts/export-web-gate-summary.mjs && node --check scripts/l6-cli-web-smoke.mjs && node --check tools/hwlab-cli/bin/hwlab-cli.mjs && node --check tools/hwlab-cli/lib/cli.mjs && node --check web/hwlab-cloud-web/app.mjs && node --check web/hwlab-cloud-web/gate-summary.mjs && node --check web/hwlab-cloud-web/scripts/check.mjs && node --check web/hwlab-cloud-web/scripts/build.mjs && node --check web/hwlab-cloud-web/scripts/dist-contract.mjs && node scripts/validate-contract.mjs && node scripts/validate-dev-gate-report.mjs && node scripts/report-lifecycle.test.mjs && node scripts/validate-dev-m3-cardinality.mjs && node scripts/validate-artifact-catalog.mjs && node scripts/deploy-desired-state-plan.mjs --check && node scripts/dev-runtime-migration.mjs --check && node scripts/dev-evidence-blocker-aggregator.mjs --check && node scripts/l2-runtime-contract-smoke.mjs && node scripts/l6-cli-web-smoke.mjs && node web/hwlab-cloud-web/scripts/check.mjs && node scripts/code-agent-chat-smoke.mjs && node --test scripts/artifact-runtime-readiness-guard.test.mjs scripts/deploy-contract-plan.test.mjs scripts/dev-m3-hardware-loop-smoke.test.mjs scripts/validate-dev-m3-cardinality.test.mjs scripts/dev-cloud-workbench-smoke.test.mjs scripts/deploy-desired-state-plan.test.mjs scripts/src/dev-cd-apply.test.mjs scripts/src/dev-deploy-apply.test.mjs scripts/src/dev-runtime-migration.test.mjs scripts/src/dev-m4-agent-loop-smoke-lib.test.mjs scripts/src/dev-evidence-blocker-aggregator.test.mjs internal/agent/index.test.mjs internal/audit/index.test.mjs internal/db/schema.test.mjs internal/db/runtime-store.test.mjs internal/cloud/json-rpc.test.mjs internal/cloud/m3-io-control.test.mjs internal/cloud/server.test.mjs internal/dev-entrypoint/http.test.mjs internal/sim/model.test.mjs internal/patchpanel/model.test.mjs internal/patchpanel/runtime.test.mjs && node scripts/cloud-api-runtime-smoke.mjs && sh -n scripts/bootstrap-skills.sh scripts/worker-entrypoint.sh", "dev-base-image:preflight": "node scripts/preflight-dev-base-image.mjs", "cloud-api:smoke": "node scripts/cloud-api-runtime-smoke.mjs", "m1:smoke": "node scripts/m1-contract-smoke.mjs", @@ -29,7 +29,8 @@ "web:build": "node web/hwlab-cloud-web/scripts/build.mjs", "artifact-catalog:refresh-blocked": "node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --blocked", "dev-artifact:preflight": "node scripts/dev-artifact-publish.mjs --preflight", - "dev-artifact:publish": "node scripts/dev-artifact-publish.mjs --publish", + "dev-cd:apply": "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report", + "dev-artifact:publish": "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report", "docs:validate:m3-rollout": "node scripts/validate-m3-rollout-runbook.mjs", "d601:k3s:readonly": "node scripts/d601-k3s-readonly-observability.mjs", "runner:issue-visibility:preflight": "node scripts/runner-issue-visibility-preflight.mjs", diff --git a/reports/dev-gate/dev-deploy-report.json b/reports/dev-gate/dev-deploy-report.json index 1daa83bc..e46e626c 100644 --- a/reports/dev-gate/dev-deploy-report.json +++ b/reports/dev-gate/dev-deploy-report.json @@ -1140,7 +1140,7 @@ "node scripts/validate-dev-gate-report.mjs" ], "afterHumanApproval": [ - "node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report" + "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report" ], "summary": "Run the dry-run plan and validator immediately before requesting human approval; live apply still requires the explicit DEV confirmation flags." }, diff --git a/scripts/dev-artifact-publish.mjs b/scripts/dev-artifact-publish.mjs index e69360bc..ac6bc0fa 100644 --- a/scripts/dev-artifact-publish.mjs +++ b/scripts/dev-artifact-publish.mjs @@ -16,6 +16,7 @@ import { resolveDevArtifactServices, serviceInventoryFromServices } from "./src/dev-artifact-services.mjs"; +import { requireDevCdTransactionForSideEffect } from "./src/dev-cd-transaction-guard.mjs"; import { runDevBaseImagePreflight } from "./src/dev-base-image-preflight.mjs"; import { probeRegistryCapabilities } from "./src/registry-capabilities.mjs"; import { inspectCloudWebDistFreshness } from "../web/hwlab-cloud-web/scripts/dist-contract.mjs"; @@ -1120,6 +1121,17 @@ async function main() { printHelp(); return; } + if (args.mode === "publish") { + const transactionGuard = requireDevCdTransactionForSideEffect({ + script: "scripts/dev-artifact-publish.mjs", + mode: "--publish" + }); + if (transactionGuard) { + console.log(JSON.stringify(transactionGuard, null, 2)); + process.exitCode = 2; + return; + } + } args.registryPrefix = validateRegistryPrefix(args.registryPrefix); const [baseImagePreflight, registryCapabilities] = await Promise.all([ diff --git a/scripts/dev-cd-apply.mjs b/scripts/dev-cd-apply.mjs new file mode 100644 index 00000000..43882411 --- /dev/null +++ b/scripts/dev-cd-apply.mjs @@ -0,0 +1,9 @@ +#!/usr/bin/env node +import { formatDevCdApplyFailure, runDevCdApply } from "./src/dev-cd-apply.mjs"; + +try { + process.exitCode = await runDevCdApply(process.argv.slice(2)); +} catch (error) { + process.stdout.write(`${JSON.stringify(formatDevCdApplyFailure(error), null, 2)}\n`); + process.exitCode = 1; +} diff --git a/scripts/src/dev-cd-apply.mjs b/scripts/src/dev-cd-apply.mjs new file mode 100644 index 00000000..aee2e340 --- /dev/null +++ b/scripts/src/dev-cd-apply.mjs @@ -0,0 +1,1432 @@ +import { spawn } from "node:child_process"; +import { createHash, randomUUID } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { request as httpRequest } from "node:http"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +import { activeReportLifecycle } from "../../internal/dev-report-lifecycle.mjs"; +import { DEV_ENDPOINT, ENVIRONMENT_DEV } from "../../internal/protocol/index.mjs"; +import { + buildKubectlCommandPrefix, + redactSensitiveText, + resolveDevKubeconfigSelection +} from "./dev-deploy-apply.mjs"; + +const defaultRepoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); +const defaultNamespace = "hwlab-dev"; +const defaultLockName = "hwlab-dev-cd-lock"; +const defaultTtlSeconds = 3600; +const defaultReportPath = "reports/dev-gate/dev-cd-apply.json"; +const artifactReportPath = "reports/dev-gate/dev-artifacts.json"; +const browserLiveUrl = "http://74.48.78.17:16666/health/live"; +const apiLiveUrl = `${DEV_ENDPOINT}/health/live`; +const lockAnnotationPrefix = "hwlab.pikastech.local"; +const lockAnnotationFields = { + promotionCommit: `${lockAnnotationPrefix}/promotionCommit`, + deployJsonHash: `${lockAnnotationPrefix}/deployJsonHash`, + ownerTaskId: `${lockAnnotationPrefix}/ownerTaskId`, + transactionId: `${lockAnnotationPrefix}/transactionId`, + phase: `${lockAnnotationPrefix}/phase`, + startedAt: `${lockAnnotationPrefix}/startedAt`, + updatedAt: `${lockAnnotationPrefix}/updatedAt`, + ttlSeconds: `${lockAnnotationPrefix}/ttlSeconds`, + liveBefore: `${lockAnnotationPrefix}/liveBefore`, + targetNamespace: `${lockAnnotationPrefix}/targetNamespace`, + targetRef: `${lockAnnotationPrefix}/targetRef`, + lockBackend: `${lockAnnotationPrefix}/lockBackend`, + releasedAt: `${lockAnnotationPrefix}/releasedAt`, + releaseStatus: `${lockAnnotationPrefix}/releaseStatus`, + staleLockBrokenAt: `${lockAnnotationPrefix}/staleLockBrokenAt`, + staleLockPrevious: `${lockAnnotationPrefix}/staleLockPrevious` +}; + +class DevCdApplyError extends Error { + constructor(message, details = {}) { + super(message); + this.name = "DevCdApplyError"; + Object.assign(this, details); + } +} + +export function parseArgs(argv) { + const args = { + apply: false, + confirmDev: false, + confirmedNonProduction: false, + writeReport: false, + reportPath: defaultReportPath, + targetRef: "origin/main", + kubeconfig: null, + kubeconfigSpecified: false, + lockName: defaultLockName, + targetNamespace: defaultNamespace, + ttlSeconds: defaultTtlSeconds, + ownerTaskId: null, + breakStaleLock: false, + skipLiveVerify: false, + help: false, + flags: new Set() + }; + + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--apply") { + args.apply = true; + args.flags.add(arg); + } else if (arg === "--confirm-dev") { + args.confirmDev = true; + args.flags.add(arg); + } else if (arg === "--confirmed-non-production") { + args.confirmedNonProduction = true; + args.flags.add(arg); + } else if (arg === "--write-report") { + args.writeReport = true; + args.flags.add(arg); + } else if (arg === "--break-stale-lock") { + args.breakStaleLock = true; + args.flags.add(arg); + } else if (arg === "--skip-live-verify") { + args.skipLiveVerify = true; + args.flags.add(arg); + } else if (arg === "--report") { + args.reportPath = readOption(argv, ++index, arg); + args.flags.add(arg); + } else if (arg === "--target-ref") { + args.targetRef = readOption(argv, ++index, arg); + args.flags.add(arg); + } else if (arg === "--kubeconfig") { + args.kubeconfig = readOption(argv, ++index, arg); + args.kubeconfigSpecified = true; + args.flags.add(arg); + } else if (arg.startsWith("--kubeconfig=")) { + args.kubeconfig = arg.slice("--kubeconfig=".length); + args.kubeconfigSpecified = true; + args.flags.add("--kubeconfig"); + } else if (arg === "--lock-name") { + args.lockName = readOption(argv, ++index, arg); + args.flags.add(arg); + } else if (arg === "--target-namespace") { + args.targetNamespace = readOption(argv, ++index, arg); + args.flags.add(arg); + } else if (arg === "--ttl-seconds") { + args.ttlSeconds = Number.parseInt(readOption(argv, ++index, arg), 10); + args.flags.add(arg); + } else if (arg === "--owner-task-id") { + args.ownerTaskId = readOption(argv, ++index, arg); + args.flags.add(arg); + } else if (arg === "--help" || arg === "-h") { + args.help = true; + } else { + throw new Error(`unknown argument ${arg}`); + } + } + + return args; +} + +function readOption(argv, index, name) { + const value = argv[index]; + if (!value || value.startsWith("--")) { + throw new Error(`${name} requires a value`); + } + return value; +} + +export function usage() { + return [ + "usage: node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production [--write-report]", + "", + "Single DEV CD transaction for latest-main publish, desired-state refresh, apply, and live verify.", + "", + "options:", + " --target-ref REF git ref to publish/apply; default: origin/main", + " --kubeconfig PATH DEV kubeconfig path; also honors HWLAB_DEV_KUBECONFIG/KUBECONFIG", + " --ttl-seconds SECONDS lock TTL; default: 3600", + " --owner-task-id ID lock holder; default: Code Queue/env/user derived", + " --break-stale-lock may take over an expired lock only with --confirm-dev", + " --report PATH default: reports/dev-gate/dev-cd-apply.json", + " --skip-live-verify test-only escape hatch; do not use for DEV acceptance" + ].join("\n"); +} + +function nowIso() { + return new Date().toISOString(); +} + +function parseJsonMaybe(value) { + try { + return JSON.parse(value); + } catch { + return null; + } +} + +function oneLine(value) { + return String(value ?? "").replace(/\s+/g, " ").trim(); +} + +function sha256(value) { + return `sha256:${createHash("sha256").update(value).digest("hex")}`; +} + +function shortCommit(value) { + return typeof value === "string" && value.length >= 7 ? value.slice(0, 7) : value; +} + +function commitMatches(actual, expected) { + if (typeof actual !== "string" || typeof expected !== "string") return false; + return actual === expected || actual.startsWith(expected) || expected.startsWith(actual); +} + +function shellCommand(command, args) { + return [command, ...args].map((part) => (/\s/u.test(part) ? JSON.stringify(part) : part)).join(" "); +} + +function defaultOwnerTaskId(env = process.env) { + return ( + env.CODE_QUEUE_TASK_ID || + env.HWLAB_OWNER_TASK_ID || + env.GITHUB_RUN_ID || + env.USER || + "unknown-dev-cd-runner" + ); +} + +function defaultRunCommand(command, args, options = {}) { + return new Promise((resolve) => { + const child = spawn(command, args, { + cwd: options.cwd ?? defaultRepoRoot, + env: options.env ?? process.env, + stdio: ["pipe", "pipe", "pipe"] + }); + let stdout = ""; + let stderr = ""; + const timeout = options.timeoutMs + ? setTimeout(() => child.kill("SIGTERM"), options.timeoutMs) + : null; + + child.stdin.on("error", () => {}); + child.stdin.end(options.input ?? ""); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + if (timeout) clearTimeout(timeout); + resolve({ code: 127, signal: null, stdout, stderr: error.message }); + }); + child.on("close", (code, signal) => { + if (timeout) clearTimeout(timeout); + resolve({ code: code ?? 1, signal, stdout, stderr }); + }); + }); +} + +async function readDeployJson(repoRoot) { + const relativePath = "deploy/deploy.json"; + const raw = await readFile(path.join(repoRoot, relativePath), "utf8"); + const manifest = JSON.parse(raw); + return { + path: relativePath, + hash: sha256(raw), + commitId: manifest.commitId ?? "unknown", + namespace: manifest.namespace ?? defaultNamespace, + environment: manifest.environment ?? "unknown", + endpoint: manifest.endpoint ?? null, + manifest + }; +} + +async function resolveTargetRef(ctx, targetRef) { + const target = await ctx.runCommand("git", ["rev-parse", "--verify", `${targetRef}^{commit}`], { + cwd: ctx.repoRoot, + timeoutMs: 10000 + }); + if (target.code !== 0) { + throw new DevCdApplyError(`target ref ${targetRef} could not be resolved`, { + code: "target-ref-unresolved", + targetRef, + stderr: redactSensitiveText(target.stderr) + }); + } + const head = await ctx.runCommand("git", ["rev-parse", "--verify", "HEAD^{commit}"], { + cwd: ctx.repoRoot, + timeoutMs: 10000 + }); + if (head.code !== 0) { + throw new DevCdApplyError("HEAD could not be resolved", { + code: "head-unresolved", + stderr: redactSensitiveText(head.stderr) + }); + } + const commitId = target.stdout.trim(); + const headCommitId = head.stdout.trim(); + return { + ref: targetRef, + commitId, + shortCommitId: shortCommit(commitId), + headCommitId, + headShortCommitId: shortCommit(headCommitId), + headMatchesTarget: commitId === headCommitId + }; +} + +function validateArgs(args) { + const blockers = []; + if (!args.apply) { + blockers.push({ + type: "safety_blocker", + scope: "apply-mode", + status: "open", + summary: "DEV CD transaction side effects require --apply." + }); + } + if (args.apply && (!args.confirmDev || !args.confirmedNonProduction)) { + blockers.push({ + type: "safety_blocker", + scope: "dev-confirmation", + status: "open", + summary: "DEV CD apply requires --confirm-dev and --confirmed-non-production." + }); + } + if (args.breakStaleLock && !args.confirmDev) { + blockers.push({ + type: "safety_blocker", + scope: "break-stale-lock-confirmation", + status: "open", + summary: "Breaking a stale DEV CD lock requires --break-stale-lock and --confirm-dev." + }); + } + if (!Number.isInteger(args.ttlSeconds) || args.ttlSeconds < 60 || args.ttlSeconds > 24 * 60 * 60) { + blockers.push({ + type: "safety_blocker", + scope: "lock-ttl", + status: "open", + summary: "--ttl-seconds must be an integer between 60 and 86400." + }); + } + for (const forbidden of ["--prod", "--production", "--read-secret", "--force-push"]) { + if (args.flags.has(forbidden)) { + blockers.push({ + type: "safety_blocker", + scope: forbidden.slice(2), + status: "open", + summary: `${forbidden} is forbidden for DEV CD apply.` + }); + } + } + if (blockers.length > 0) { + throw new DevCdApplyError("invalid DEV CD apply arguments", { + code: "invalid-arguments", + blockers + }); + } +} + +function lockAnnotationValue(value) { + if (value === undefined || value === null) return ""; + if (typeof value === "string") return value; + return JSON.stringify(value); +} + +export function buildLockAnnotations(lock) { + const annotations = {}; + for (const [field, key] of Object.entries(lockAnnotationFields)) { + if (Object.hasOwn(lock, field)) { + annotations[key] = lockAnnotationValue(lock[field]); + } + } + return annotations; +} + +function readAnnotation(annotations, field) { + return annotations?.[lockAnnotationFields[field]] ?? null; +} + +function readJsonAnnotation(annotations, field) { + const value = readAnnotation(annotations, field); + return value ? parseJsonMaybe(value) : null; +} + +export function parseDeployLock(lease) { + if (!lease) return null; + const annotations = lease.metadata?.annotations ?? {}; + const ttlSeconds = Number.parseInt( + readAnnotation(annotations, "ttlSeconds") || lease.spec?.leaseDurationSeconds || `${defaultTtlSeconds}`, + 10 + ); + const updatedAt = + readAnnotation(annotations, "updatedAt") || + lease.spec?.renewTime || + lease.metadata?.creationTimestamp || + null; + const transactionId = readAnnotation(annotations, "transactionId") || lease.spec?.holderIdentity || null; + const ownerTaskId = readAnnotation(annotations, "ownerTaskId") || lease.spec?.holderIdentity || null; + return { + lockBackend: readAnnotation(annotations, "lockBackend") || "Lease", + lockName: lease.metadata?.name ?? defaultLockName, + resourceVersion: lease.metadata?.resourceVersion ?? null, + holderIdentity: lease.spec?.holderIdentity ?? null, + promotionCommit: readAnnotation(annotations, "promotionCommit"), + deployJsonHash: readAnnotation(annotations, "deployJsonHash"), + ownerTaskId, + transactionId, + phase: readAnnotation(annotations, "phase") || "unknown", + startedAt: readAnnotation(annotations, "startedAt") || lease.spec?.acquireTime || lease.metadata?.creationTimestamp || null, + updatedAt, + ttlSeconds: Number.isInteger(ttlSeconds) ? ttlSeconds : defaultTtlSeconds, + liveBefore: readJsonAnnotation(annotations, "liveBefore"), + targetNamespace: readAnnotation(annotations, "targetNamespace") || lease.metadata?.namespace || defaultNamespace, + targetRef: readAnnotation(annotations, "targetRef"), + releasedAt: readAnnotation(annotations, "releasedAt"), + releaseStatus: readAnnotation(annotations, "releaseStatus"), + staleLockBrokenAt: readAnnotation(annotations, "staleLockBrokenAt"), + staleLockPrevious: readJsonAnnotation(annotations, "staleLockPrevious") + }; +} + +export function classifyDeployLock(lock, now = new Date()) { + if (!lock || lock.phase === "released") { + return { + held: false, + stale: false, + retryAfterSeconds: 0, + expiresAt: null + }; + } + const updatedAtMs = Number.isFinite(Date.parse(lock.updatedAt)) ? Date.parse(lock.updatedAt) : 0; + const expiresAtMs = updatedAtMs + lock.ttlSeconds * 1000; + const retryAfterSeconds = Math.max(0, Math.ceil((expiresAtMs - now.getTime()) / 1000)); + return { + held: retryAfterSeconds > 0, + stale: retryAfterSeconds <= 0, + retryAfterSeconds, + expiresAt: Number.isFinite(expiresAtMs) ? new Date(expiresAtMs).toISOString() : null + }; +} + +export function deployLockHeldFailure(lock, now = new Date()) { + const classification = classifyDeployLock(lock, now); + return { + ok: false, + status: "blocked", + error: "deploy-lock-held", + code: "deploy-lock-held", + holder: lock.ownerTaskId || lock.transactionId || lock.holderIdentity || "unknown", + promotionCommit: lock.promotionCommit ?? null, + deployJsonHash: lock.deployJsonHash ?? null, + transactionId: lock.transactionId ?? null, + phase: lock.phase ?? "unknown", + targetNamespace: lock.targetNamespace ?? defaultNamespace, + retryAfterSeconds: classification.retryAfterSeconds, + stale: classification.stale, + requiresBreakStaleLock: classification.stale, + lockName: lock.lockName ?? defaultLockName, + lockBackend: lock.lockBackend ?? "Lease", + summary: classification.stale + ? "A stale DEV CD transaction lock exists; rerun with --break-stale-lock --confirm-dev only after confirming the holder is inactive." + : "Another DEV CD transaction is already running; retry after the current holder releases the lock.", + mutationAttempted: false, + prodTouched: false + }; +} + +function isNotFound(result) { + return /not\s*found|notfound/i.test(`${result.stderr}\n${result.stdout}`); +} + +function isAlreadyExists(result) { + return /already\s*exists|alreadyexists/i.test(`${result.stderr}\n${result.stdout}`); +} + +function isConflict(result) { + return /conflict|object has been modified|operation cannot be fulfilled/i.test(`${result.stderr}\n${result.stdout}`); +} + +function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTaskId, startedAt, phase, liveBefore, staleLockPrevious = null }) { + const lock = { + promotionCommit: target.shortCommitId, + deployJsonHash: deployBefore.hash, + ownerTaskId, + transactionId, + phase, + startedAt, + updatedAt: startedAt, + ttlSeconds: args.ttlSeconds, + liveBefore, + targetNamespace: args.targetNamespace, + targetRef: target.ref, + lockBackend: "Lease" + }; + if (staleLockPrevious) { + lock.staleLockBrokenAt = startedAt; + lock.staleLockPrevious = staleLockPrevious; + } + return { + apiVersion: "coordination.k8s.io/v1", + kind: "Lease", + metadata: { + name: args.lockName, + namespace: args.targetNamespace, + labels: { + "app.kubernetes.io/part-of": "hwlab", + "hwlab.pikastech.local/profile": "dev", + "hwlab.pikastech.local/cd-lock": "true" + }, + annotations: buildLockAnnotations(lock) + }, + spec: { + holderIdentity: `${ownerTaskId}/${transactionId}`, + leaseDurationSeconds: args.ttlSeconds, + acquireTime: startedAt, + renewTime: startedAt + } + }; +} + +function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, phase, startedAt, updatedAt, liveBefore, staleLockPrevious = null }) { + const lock = { + promotionCommit: target.shortCommitId, + deployJsonHash: deployBefore.hash, + ownerTaskId, + transactionId, + phase, + startedAt, + updatedAt, + ttlSeconds: args.ttlSeconds, + liveBefore, + targetNamespace: args.targetNamespace, + targetRef: target.ref, + lockBackend: "Lease" + }; + if (staleLockPrevious) { + lock.staleLockBrokenAt = updatedAt; + lock.staleLockPrevious = staleLockPrevious; + } + return { + metadata: { + labels: { + "app.kubernetes.io/part-of": "hwlab", + "hwlab.pikastech.local/profile": "dev", + "hwlab.pikastech.local/cd-lock": "true" + }, + annotations: buildLockAnnotations(lock) + }, + spec: { + holderIdentity: `${ownerTaskId}/${transactionId}`, + leaseDurationSeconds: args.ttlSeconds, + acquireTime: startedAt, + renewTime: updatedAt + } + }; +} + +async function kubectl(ctx, kubectlContext, args, options = {}) { + const result = await ctx.runCommand(kubectlContext.executor, args, { + cwd: ctx.repoRoot, + env: kubectlContext.env, + input: options.input, + timeoutMs: options.timeoutMs ?? 30000 + }); + return { + ...result, + command: shellCommand("kubectl", args), + stdout: result.stdout ?? "", + stderr: result.stderr ?? "" + }; +} + +async function replaceLease(ctx, kubectlContext, args, lease, timeoutMs = 30000) { + return kubectl( + ctx, + kubectlContext, + ["-n", args.targetNamespace, "replace", "-f", "-", "-o", "json"], + { + input: JSON.stringify(lease), + timeoutMs + } + ); +} + +function mergeLease(lease, patch, { resetLockAnnotations = false } = {}) { + const baseAnnotations = { ...(lease.metadata?.annotations ?? {}) }; + if (resetLockAnnotations) { + for (const key of Object.values(lockAnnotationFields)) { + delete baseAnnotations[key]; + } + } + return { + ...lease, + metadata: { + ...lease.metadata, + ...(patch.metadata ?? {}), + labels: { + ...(lease.metadata?.labels ?? {}), + ...(patch.metadata?.labels ?? {}) + }, + annotations: { + ...baseAnnotations, + ...(patch.metadata?.annotations ?? {}) + } + }, + spec: { + ...(lease.spec ?? {}), + ...(patch.spec ?? {}) + } + }; +} + +async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefore, transactionId, ownerTaskId, liveBefore, now }) { + const startedAt = now.toISOString(); + const get = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "get", "lease", args.lockName, "-o", "json"]); + if (get.code === 0) { + const lease = JSON.parse(get.stdout); + const existing = parseDeployLock(lease); + const classification = classifyDeployLock(existing, now); + if (classification.held || (classification.stale && !args.breakStaleLock)) { + const failure = deployLockHeldFailure(existing, now); + throw new DevCdApplyError(failure.summary, { + code: "deploy-lock-held", + lockFailure: failure + }); + } + + const patch = lockPatch({ + args, + target, + deployBefore, + transactionId, + ownerTaskId, + phase: "publishing", + startedAt, + updatedAt: startedAt, + liveBefore, + staleLockPrevious: classification.stale ? existing : null + }); + const replaced = await replaceLease(ctx, kubectlContext, args, mergeLease(lease, patch, { resetLockAnnotations: true })); + if (isConflict(replaced)) { + return acquireDeployLock({ ctx, args, kubectlContext, target, deployBefore, transactionId, ownerTaskId, liveBefore, now }); + } + if (replaced.code !== 0) { + throw new DevCdApplyError("failed to acquire DEV CD Lease lock", { + code: "lock-acquire-failed", + stderr: redactSensitiveText(replaced.stderr || replaced.stdout) + }); + } + return { + acquired: true, + lock: parseDeployLock(JSON.parse(replaced.stdout)), + staleBreak: classification.stale ? existing : null, + command: replaced.command + }; + } + + if (!isNotFound(get)) { + throw new DevCdApplyError("failed to read DEV CD Lease lock", { + code: "lock-read-failed", + stderr: redactSensitiveText(get.stderr || get.stdout) + }); + } + + const manifest = buildLeaseManifest({ + args, + target, + deployBefore, + transactionId, + ownerTaskId, + startedAt, + phase: "publishing", + liveBefore + }); + const created = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "create", "-f", "-", "-o", "json"], { + input: JSON.stringify(manifest), + timeoutMs: 30000 + }); + if (created.code === 0) { + return { + acquired: true, + lock: parseDeployLock(JSON.parse(created.stdout)), + staleBreak: null, + command: created.command + }; + } + if (isAlreadyExists(created)) { + return acquireDeployLock({ ctx, args, kubectlContext, target, deployBefore, transactionId, ownerTaskId, liveBefore, now }); + } + throw new DevCdApplyError("failed to create DEV CD Lease lock", { + code: "lock-create-failed", + stderr: redactSensitiveText(created.stderr || created.stdout) + }); +} + +async function readOwnedLease({ ctx, args, kubectlContext, transactionId, timeoutMs = 15000 }) { + const get = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "get", "lease", args.lockName, "-o", "json"], { + timeoutMs + }); + if (get.code !== 0) { + throw new DevCdApplyError("failed to read DEV CD lock ownership", { + code: "lock-owner-read-failed", + stderr: redactSensitiveText(get.stderr || get.stdout) + }); + } + const lease = JSON.parse(get.stdout); + const lock = parseDeployLock(lease); + if (lock.transactionId !== transactionId) { + throw new DevCdApplyError("DEV CD lock ownership changed before transaction update", { + code: "lock-owner-mismatch", + holder: lock.ownerTaskId || lock.transactionId || "unknown", + phase: lock.phase + }); + } + return { lease, lock }; +} + +async function patchOwnedLease({ ctx, args, kubectlContext, lockState, transactionId, patch, errorCode, errorMessage }) { + const { lease } = await readOwnedLease({ ctx, args, kubectlContext, transactionId }); + const result = await replaceLease(ctx, kubectlContext, args, mergeLease(lease, patch)); + if (result.code !== 0) { + throw new DevCdApplyError(errorMessage, { + code: errorCode, + stderr: redactSensitiveText(result.stderr || result.stdout) + }); + } + lockState.lock = parseDeployLock(JSON.parse(result.stdout)); + return lockState.lock; +} + +async function updateDeployLockPhase({ ctx, args, kubectlContext, transactionId, lockState, phase, status = "running" }) { + const updatedAt = ctx.now().toISOString(); + const patch = { + metadata: { + annotations: { + [lockAnnotationFields.phase]: phase, + [lockAnnotationFields.updatedAt]: updatedAt, + [lockAnnotationFields.releaseStatus]: status + } + }, + spec: { + renewTime: updatedAt + } + }; + return await patchOwnedLease({ + ctx, + args, + kubectlContext, + transactionId, + lockState, + patch, + errorCode: "lock-phase-update-failed", + errorMessage: `failed to update DEV CD lock phase to ${phase}` + }); +} + +async function updateDeployLockLiveBefore({ ctx, args, kubectlContext, transactionId, lockState, liveBefore }) { + const updatedAt = ctx.now().toISOString(); + const patch = { + metadata: { + annotations: { + [lockAnnotationFields.liveBefore]: JSON.stringify(liveBefore), + [lockAnnotationFields.updatedAt]: updatedAt + } + }, + spec: { + renewTime: updatedAt + } + }; + return await patchOwnedLease({ + ctx, + args, + kubectlContext, + transactionId, + lockState, + patch, + errorCode: "lock-live-before-update-failed", + errorMessage: "failed to update DEV CD lock liveBefore evidence" + }); +} + +async function releaseDeployLock({ ctx, args, kubectlContext, transactionId, status }) { + const get = await kubectl(ctx, kubectlContext, ["-n", args.targetNamespace, "get", "lease", args.lockName, "-o", "json"], { + timeoutMs: 15000 + }); + if (get.code !== 0) { + return { + status: "release_read_failed", + command: get.command, + error: redactSensitiveText(get.stderr || get.stdout) + }; + } + const current = parseDeployLock(JSON.parse(get.stdout)); + if (current.transactionId !== transactionId) { + return { + status: "not_owner", + holder: current.ownerTaskId || current.transactionId || "unknown", + phase: current.phase + }; + } + const releasedAt = ctx.now().toISOString(); + const patch = { + metadata: { + annotations: { + [lockAnnotationFields.phase]: "released", + [lockAnnotationFields.updatedAt]: releasedAt, + [lockAnnotationFields.releasedAt]: releasedAt, + [lockAnnotationFields.releaseStatus]: status + } + }, + spec: { + holderIdentity: "", + leaseDurationSeconds: 1, + renewTime: releasedAt + } + }; + const result = await replaceLease(ctx, kubectlContext, args, mergeLease(JSON.parse(get.stdout), patch), 15000); + if (isConflict(result)) { + return { + status: "release_conflict", + command: result.command, + error: "Lease changed before release; lock was not overwritten." + }; + } + if (result.code !== 0) { + return { + status: "release_failed", + command: result.command, + error: redactSensitiveText(result.stderr || result.stdout) + }; + } + return { + status: "released", + command: result.command, + lock: parseDeployLock(JSON.parse(result.stdout)) + }; +} + +async function resolveKubectlContext(args, env, runCommand) { + const selection = resolveDevKubeconfigSelection({ flagValue: args.kubeconfig, env }); + const which = await runCommand("which", ["kubectl"], { timeoutMs: 5000 }); + const executor = which.code === 0 ? which.stdout.trim() : "kubectl"; + return { + executor, + kubeconfig: selection.kubeconfig, + kubeconfigSource: selection.source, + commandPrefix: buildKubectlCommandPrefix(selection.kubeconfig), + env: { + ...env, + KUBECONFIG: selection.kubeconfig + } + }; +} + +function commandEnv(ctx, transaction) { + return { + ...ctx.env, + HWLAB_CD_TRANSACTION_ID: transaction.transactionId, + HWLAB_CD_TRANSACTION_OWNER: transaction.ownerTaskId, + HWLAB_CD_LOCK_NAME: transaction.lockName, + HWLAB_CD_LOCK_NAMESPACE: transaction.targetNamespace + }; +} + +async function runStep(ctx, transaction, step) { + const startedAt = ctx.now().toISOString(); + const command = shellCommand(step.command, step.args); + const result = await ctx.runCommand(step.command, step.args, { + cwd: ctx.repoRoot, + env: commandEnv(ctx, transaction), + timeoutMs: step.timeoutMs ?? 300000 + }); + const finishedAt = ctx.now().toISOString(); + const stdout = redactSensitiveText(result.stdout ?? ""); + const stderr = redactSensitiveText(result.stderr ?? ""); + const parsedStdout = parseJsonMaybe(stdout); + return { + id: step.id, + phase: step.phase, + status: result.code === 0 ? "pass" : "blocked", + command, + code: result.code, + startedAt, + finishedAt, + stdoutJson: parsedStdout, + stdoutTail: parsedStdout ? null : stdout.trim().slice(-2000), + stderrTail: stderr.trim().slice(-2000), + reportPath: step.reportPath ?? null + }; +} + +function stepBlocker(stepResult) { + return { + type: "runtime_blocker", + scope: stepResult.id, + status: "open", + summary: `${stepResult.command} exited ${stepResult.code}.` + }; +} + +function healthCommit(json) { + return ( + json?.commit?.id || + json?.commitId || + json?.revision || + json?.image?.tag || + null + ); +} + +function healthImage(json) { + if (typeof json?.image === "string") return json.image; + return json?.image?.reference ?? null; +} + +function summarizeHealthJson(json, expectedServiceId, expectedCommit) { + const observedCommit = healthCommit(json); + return { + serviceId: json?.serviceId ?? json?.service?.id ?? null, + environment: json?.environment ?? null, + applicationStatus: json?.status ?? null, + ready: Object.hasOwn(json ?? {}, "ready") ? json.ready : null, + observedCommit, + image: healthImage(json), + imageTag: json?.image?.tag ?? null, + observedAt: json?.observedAt ?? null, + serviceMatches: (json?.serviceId ?? json?.service?.id) === expectedServiceId, + environmentMatches: json?.environment === ENVIRONMENT_DEV, + commitMatches: expectedCommit ? commitMatches(observedCommit, expectedCommit) : null, + blockerCodes: Array.isArray(json?.blockerCodes) ? json.blockerCodes : [] + }; +} + +async function defaultHttpGetJson(url, timeoutMs = 10000) { + return new Promise((resolve, reject) => { + const request = httpRequest(url, { method: "GET", timeout: timeoutMs }, (response) => { + response.setEncoding("utf8"); + let body = ""; + response.on("data", (chunk) => { + body += chunk; + }); + response.on("end", () => { + let json = null; + try { + json = JSON.parse(body); + } catch {} + resolve({ + statusCode: response.statusCode ?? 0, + body, + json + }); + }); + }); + request.on("timeout", () => { + request.destroy(new Error(`timeout after ${timeoutMs}ms`)); + }); + request.on("error", reject); + request.end(); + }); +} + +async function probeLiveEndpoint(ctx, { id, url, expectedServiceId, expectedCommit }) { + const startedAt = ctx.now().toISOString(); + try { + const response = await ctx.httpGetJson(url, 10000); + const summary = summarizeHealthJson(response.json, expectedServiceId, expectedCommit); + const reachable = response.statusCode >= 200 && response.statusCode < 300 && Boolean(response.json); + const identityMatches = reachable && summary.serviceMatches && summary.environmentMatches && (expectedCommit ? summary.commitMatches : true); + return { + id, + url, + status: identityMatches ? "pass" : "blocked", + httpStatus: response.statusCode, + reachable, + identityMatches, + startedAt, + finishedAt: ctx.now().toISOString(), + expectedServiceId, + expectedCommit: expectedCommit ?? null, + ...summary + }; + } catch (error) { + return { + id, + url, + status: "blocked", + reachable: false, + identityMatches: false, + startedAt, + finishedAt: ctx.now().toISOString(), + expectedServiceId, + expectedCommit: expectedCommit ?? null, + error: oneLine(error.message) + }; + } +} + +export async function verifyDevLive(ctx, { expectedCommit = null } = {}) { + const [cloudWeb, cloudApi] = await Promise.all([ + probeLiveEndpoint(ctx, { + id: "cloud-web-16666", + url: browserLiveUrl, + expectedServiceId: "hwlab-cloud-web", + expectedCommit + }), + probeLiveEndpoint(ctx, { + id: "cloud-api-16667", + url: apiLiveUrl, + expectedServiceId: "hwlab-cloud-api", + expectedCommit + }) + ]); + const endpoints = [cloudWeb, cloudApi]; + return { + status: endpoints.every((endpoint) => endpoint.status === "pass") ? "pass" : "blocked", + expectedCommit: expectedCommit ?? null, + endpoints, + summary: { + checked: endpoints.length, + reachable: endpoints.filter((endpoint) => endpoint.reachable).length, + identityMatches: endpoints.filter((endpoint) => endpoint.identityMatches).length, + commitMatches: expectedCommit + ? endpoints.filter((endpoint) => endpoint.commitMatches === true).length + : null, + ports: [16666, 16667] + } + }; +} + +function buildReport({ + args, + transaction, + target, + deployBefore, + deployAfter, + lockState, + steps, + blockers, + status, + startedAt, + finishedAt, + liveBefore, + liveVerify, + release +}) { + return { + $schema: "https://hwlab.pikastech.local/schemas/dev-cd-apply.schema.json", + $id: "https://hwlab.pikastech.local/reports/dev-gate/dev-cd-apply.json", + reportVersion: "v1", + issue: "pikasTech/HWLAB#274", + taskId: "dev-cd-apply", + commitId: target.shortCommitId, + acceptanceLevel: "dev_cd_apply", + devOnly: true, + prodDisabled: true, + status, + generatedAt: finishedAt, + reportLifecycle: activeReportLifecycle("Current DEV CD transaction report; the only DEV side-effect entrypoint is scripts/dev-cd-apply.mjs."), + supports: [ + "pikasTech/HWLAB#274", + "pikasTech/HWLAB#116", + "pikasTech/HWLAB#235", + "pikasTech/HWLAB#50", + "pikasTech/HWLAB#57", + "pikasTech/HWLAB#67", + "pikasTech/HWLAB#73" + ], + sourceContract: { + status: blockers.length ? "blocked" : "pass", + documents: [ + "docs/reference/deployment-publish.md", + "docs/reference/dev-runtime-boundary.md", + "docs/dev-artifact-publish.md", + "docs/dev-deploy-apply.md", + "docs/artifact-catalog.md" + ], + summary: "DEV CD publish/apply/verify side effects are serialized by one Kubernetes Lease and use deploy/deploy.json as the apply desired-state source." + }, + validationCommands: [ + "node --check scripts/dev-cd-apply.mjs", + "node --check scripts/src/dev-cd-apply.mjs", + "node --test scripts/src/dev-cd-apply.test.mjs", + "node --check scripts/dev-artifact-publish.mjs", + "node --check scripts/src/dev-deploy-apply.mjs", + "node --test scripts/src/dev-deploy-apply.test.mjs", + "node scripts/deploy-desired-state-plan.mjs --check", + "node scripts/validate-artifact-catalog.mjs", + "git diff --check" + ], + localSmoke: { + status: "not_run", + commands: ["npm run check"], + evidence: ["Local checks are listed as validation commands and are not implied by a DEV CD transaction."], + summary: "The transaction report records DEV publish/apply/verify evidence; local smoke is a separate gate." + }, + dryRun: { + status: "not_run", + commands: ["node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report"], + evidence: ["The transaction path owns live side effects; dry-run remains a preflight support mode."], + summary: "DEV CD mutation requires --apply and the transaction Lease lock." + }, + devPreconditions: { + status: blockers.length ? "blocked" : "pass", + requirements: [ + "HEAD must match the requested target ref before publish.", + "Lease/hwlab-dev-cd-lock must be acquired before publish/apply.", + "Legacy publish/apply side-effect scripts must run with HWLAB_CD_TRANSACTION_ID.", + "deploy/deploy.json, artifact catalog, and workloads must converge before apply.", + "Public 16666 and 16667 live health must be recorded in this report." + ], + summary: blockers.length ? "One or more DEV CD transaction preconditions failed." : "DEV CD transaction preconditions passed." + }, + transaction: { + transactionId: transaction.transactionId, + ownerTaskId: transaction.ownerTaskId, + targetNamespace: transaction.targetNamespace, + lockName: transaction.lockName, + lockBackend: "Lease", + startedAt, + finishedAt, + ttlSeconds: args.ttlSeconds, + phases: transaction.phases, + release + }, + devCdApply: { + mode: args.apply ? "apply" : "dry-run", + target: { + ref: target.ref, + promotionCommit: target.commitId, + shortCommitId: target.shortCommitId, + headCommitId: target.headCommitId, + headMatchesTarget: target.headMatchesTarget, + namespace: args.targetNamespace, + apiEndpoint: DEV_ENDPOINT, + browserEndpoint: "http://74.48.78.17:16666/" + }, + deployJson: { + before: deployBefore, + after: deployAfter + }, + lock: { + acquired: Boolean(lockState?.acquired), + current: lockState?.lock ?? null, + staleBreak: lockState?.staleBreak ?? null + }, + steps, + liveBefore, + liveVerify, + reportPaths: { + transaction: args.writeReport ? args.reportPath : null, + artifacts: artifactReportPath, + deployApply: "reports/dev-gate/dev-deploy-report.json" + }, + safety: { + prodTouched: false, + secretValuesRead: false, + secretValuesPrinted: false, + manualDbWrite: false, + oldPublicPortsUsed: false, + mutationAttempted: steps.some((step) => ["artifact-publish", "dev-deploy-apply"].includes(step.id)), + singleLock: "Lease/hwlab-dev/hwlab-dev-cd-lock" + } + }, + blockers, + notes: "DEV-only CD transaction evidence. This does not claim M3 PASS or runtime durability." + }; +} + +function buildArgumentFailure(error) { + const blockers = error.blockers ?? [{ + type: "safety_blocker", + scope: error.code ?? "dev-cd-apply", + status: "open", + summary: oneLine(error.message) + }]; + return { + ok: false, + status: "blocked", + error: error.code ?? "dev-cd-apply-failed", + blockers, + mutationAttempted: false, + prodTouched: false + }; +} + +export function formatDevCdApplyFailure(error) { + if (error?.lockFailure) return error.lockFailure; + if (error instanceof DevCdApplyError) return buildArgumentFailure(error); + return { + ok: false, + status: "failed", + error: "dev-cd-apply-failed", + summary: error instanceof Error ? error.message : String(error), + mutationAttempted: false, + prodTouched: false + }; +} + +export async function runDevCdApply(argv, io = {}) { + const stdout = io.stdout ?? process.stdout; + const env = io.env ?? process.env; + const ctx = { + repoRoot: io.repoRoot ?? defaultRepoRoot, + env, + runCommand: io.runCommand ?? defaultRunCommand, + httpGetJson: io.httpGetJson ?? defaultHttpGetJson, + now: io.now ?? (() => new Date()) + }; + + let args; + try { + args = parseArgs(argv); + if (args.help) { + stdout.write(`${usage()}\n`); + return 0; + } + validateArgs(args); + } catch (error) { + const failure = formatDevCdApplyFailure(error); + stdout.write(`${JSON.stringify(failure, null, 2)}\n`); + return 2; + } + + const startedAt = ctx.now().toISOString(); + const transaction = { + transactionId: randomUUID(), + ownerTaskId: args.ownerTaskId ?? defaultOwnerTaskId(env), + targetNamespace: args.targetNamespace, + lockName: args.lockName, + phases: [] + }; + const steps = []; + const blockers = []; + let lockState = null; + let release = null; + let target = null; + let deployBefore = null; + let deployAfter = null; + let liveBefore = null; + let liveVerify = null; + let status = "blocked"; + + try { + target = await resolveTargetRef(ctx, args.targetRef); + deployBefore = await readDeployJson(ctx.repoRoot); + const kubectlContext = await resolveKubectlContext(args, env, ctx.runCommand); + liveBefore = { + status: "pending", + reason: "liveBefore is captured immediately after Lease acquisition so publish/apply cannot race before the transaction lock." + }; + lockState = await acquireDeployLock({ + ctx, + args, + kubectlContext, + target, + deployBefore, + transactionId: transaction.transactionId, + ownerTaskId: transaction.ownerTaskId, + liveBefore, + now: ctx.now() + }); + liveBefore = await verifyDevLive(ctx, { expectedCommit: null }); + await updateDeployLockLiveBefore({ + ctx, + args, + kubectlContext, + transactionId: transaction.transactionId, + lockState, + liveBefore + }); + transaction.phases.push({ phase: "publishing", status: "entered", at: ctx.now().toISOString() }); + + if (!target.headMatchesTarget) { + blockers.push({ + type: "contract_blocker", + scope: "target-ref-head", + status: "open", + summary: `HEAD ${target.headShortCommitId} must match target ref ${target.ref} ${target.shortCommitId} before DEV CD publish.` + }); + } + + const plannedSteps = [ + { + id: "desired-state-check-before-publish", + phase: "publishing", + command: process.execPath, + args: ["scripts/deploy-desired-state-plan.mjs", "--check", "--pretty"], + timeoutMs: 120000 + }, + { + id: "artifact-publish", + phase: "publishing", + command: process.execPath, + args: ["scripts/dev-artifact-publish.mjs", "--publish"], + timeoutMs: 60 * 60 * 1000, + reportPath: artifactReportPath + }, + { + id: "artifact-catalog-refresh", + phase: "publishing", + command: process.execPath, + args: [ + "scripts/refresh-artifact-catalog.mjs", + "--target-ref", + target.ref, + "--publish-report", + artifactReportPath + ], + timeoutMs: 120000 + }, + { + id: "artifact-catalog-validate", + phase: "publishing", + command: process.execPath, + args: ["scripts/validate-artifact-catalog.mjs"], + timeoutMs: 120000 + }, + { + id: "desired-state-check-after-refresh", + phase: "publishing", + command: process.execPath, + args: [ + "scripts/deploy-desired-state-plan.mjs", + "--promotion-commit", + target.commitId, + "--check", + "--pretty" + ], + timeoutMs: 120000 + }, + { + id: "dev-deploy-apply", + phase: "applying", + command: process.execPath, + args: [ + "scripts/dev-deploy-apply.mjs", + "--apply", + "--confirm-dev", + "--confirmed-non-production", + "--write-report", + ...(args.kubeconfig ? ["--kubeconfig", args.kubeconfig] : []) + ], + timeoutMs: 20 * 60 * 1000, + reportPath: "reports/dev-gate/dev-deploy-report.json" + } + ]; + + if (blockers.length === 0) { + for (const step of plannedSteps) { + if (step.phase === "applying" && lockState.lock?.phase !== "applying") { + await updateDeployLockPhase({ + ctx, + args, + kubectlContext, + transactionId: transaction.transactionId, + lockState, + phase: "applying" + }); + transaction.phases.push({ phase: "applying", status: "entered", at: ctx.now().toISOString() }); + } + const result = await runStep(ctx, transaction, step); + steps.push(result); + if (result.status !== "pass") { + blockers.push(stepBlocker(result)); + break; + } + } + } + + if (blockers.length === 0) { + await updateDeployLockPhase({ + ctx, + args, + kubectlContext, + transactionId: transaction.transactionId, + lockState, + phase: "verifying" + }); + transaction.phases.push({ phase: "verifying", status: "entered", at: ctx.now().toISOString() }); + deployAfter = await readDeployJson(ctx.repoRoot); + liveVerify = args.skipLiveVerify + ? { status: "not_run", reason: "--skip-live-verify", endpoints: [], summary: { checked: 0 } } + : await verifyDevLive(ctx, { expectedCommit: target.shortCommitId }); + if (liveVerify.status !== "pass") { + blockers.push({ + type: "observability_blocker", + scope: "live-verify", + status: "open", + summary: "16666/16667 live health did not match the DEV CD promotion commit." + }); + } + } else { + deployAfter = deployBefore; + liveVerify = { status: "not_run", reason: "transaction blocked before live verify", endpoints: [], summary: { checked: 0 } }; + } + + status = blockers.length === 0 ? "pass" : "blocked"; + release = await releaseDeployLock({ + ctx, + args, + kubectlContext, + transactionId: transaction.transactionId, + status + }); + transaction.phases.push({ phase: "released", status: release.status, at: ctx.now().toISOString() }); + } catch (error) { + if (error?.lockFailure) { + stdout.write(`${JSON.stringify(error.lockFailure, null, 2)}\n`); + return 2; + } + blockers.push({ + type: "runtime_blocker", + scope: error.code ?? "dev-cd-apply", + status: "open", + summary: oneLine(error.message) + }); + status = "blocked"; + if (lockState?.acquired) { + const kubectlContext = await resolveKubectlContext(args, env, ctx.runCommand); + release = await releaseDeployLock({ + ctx, + args, + kubectlContext, + transactionId: transaction.transactionId, + status + }); + transaction.phases.push({ phase: "released", status: release.status, at: ctx.now().toISOString() }); + } + } + + const finishedAt = ctx.now().toISOString(); + const report = buildReport({ + args, + transaction, + target: target ?? { + ref: args.targetRef, + commitId: "unknown", + shortCommitId: "unknown", + headCommitId: "unknown", + headMatchesTarget: false + }, + deployBefore: deployBefore ?? { path: "deploy/deploy.json", hash: "unknown", commitId: "unknown" }, + deployAfter: deployAfter ?? deployBefore ?? { path: "deploy/deploy.json", hash: "unknown", commitId: "unknown" }, + lockState, + steps, + blockers, + status, + startedAt, + finishedAt, + liveBefore: liveBefore ?? { status: "not_run" }, + liveVerify: liveVerify ?? { status: "not_run", endpoints: [], summary: { checked: 0 } }, + release + }); + + if (args.writeReport) { + const absoluteReportPath = path.resolve(ctx.repoRoot, args.reportPath); + await mkdir(path.dirname(absoluteReportPath), { recursive: true }); + await writeFile(absoluteReportPath, `${JSON.stringify(report, null, 2)}\n`); + } + + stdout.write(`${JSON.stringify(report, null, 2)}\n`); + return status === "pass" ? 0 : 2; +} diff --git a/scripts/src/dev-cd-apply.test.mjs b/scripts/src/dev-cd-apply.test.mjs new file mode 100644 index 00000000..51ed4513 --- /dev/null +++ b/scripts/src/dev-cd-apply.test.mjs @@ -0,0 +1,382 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { execFile } from "node:child_process"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; + +import { + buildLockAnnotations, + classifyDeployLock, + deployLockHeldFailure, + parseDeployLock, + runDevCdApply, + verifyDevLive +} from "./dev-cd-apply.mjs"; + +const execFileAsync = promisify(execFile); +const repoRoot = path.resolve(path.dirname(new URL(import.meta.url).pathname), "../.."); + +function iso(offsetMs = 0) { + return new Date(Date.parse("2026-05-23T08:00:00.000Z") + offsetMs).toISOString(); +} + +function leaseFromLock(lock) { + return { + apiVersion: "coordination.k8s.io/v1", + kind: "Lease", + metadata: { + name: "hwlab-dev-cd-lock", + namespace: "hwlab-dev", + resourceVersion: "1", + annotations: buildLockAnnotations(lock) + }, + spec: { + holderIdentity: `${lock.ownerTaskId}/${lock.transactionId}`, + leaseDurationSeconds: lock.ttlSeconds, + acquireTime: lock.startedAt, + renewTime: lock.updatedAt + } + }; +} + +function parsePatch(args) { + const index = args.indexOf("-p"); + assert.notEqual(index, -1, "kubectl patch must include -p"); + return JSON.parse(args[index + 1]); +} + +async function makeRepo() { + const repoRoot = await fsTempDir(); + await mkdir(path.join(repoRoot, "deploy"), { recursive: true }); + await writeFile( + path.join(repoRoot, "deploy/deploy.json"), + `${JSON.stringify({ + manifestVersion: "v1", + environment: "dev", + namespace: "hwlab-dev", + endpoint: "http://74.48.78.17:16667", + commitId: "abc1234" + }, null, 2)}\n` + ); + return repoRoot; +} + +async function fsTempDir() { + return await import("node:fs/promises").then((fs) => fs.mkdtemp(path.join(os.tmpdir(), "hwlab-cd-"))); +} + +function makeHttpGetJson() { + return async (url) => { + if (url.includes("16666")) { + return { + statusCode: 200, + json: { + serviceId: "hwlab-cloud-web", + environment: "dev", + status: "ok", + revision: "abc1234", + image: { reference: "127.0.0.1:5000/hwlab/hwlab-cloud-web:abc1234", tag: "abc1234" }, + observedAt: iso() + } + }; + } + return { + statusCode: 200, + json: { + serviceId: "hwlab-cloud-api", + environment: "dev", + status: "degraded", + ready: false, + commit: { id: "abc1234", source: "runtime-env" }, + image: { reference: "127.0.0.1:5000/hwlab/hwlab-cloud-api:abc1234", tag: "abc1234" }, + blockerCodes: ["runtime_durable_adapter_auth_blocked"], + observedAt: iso() + } + }; + }; +} + +function makeRunCommand({ heldLock = null, commandLog = [] } = {}) { + let lease = heldLock ? leaseFromLock(heldLock) : null; + return async (command, args, options = {}) => { + commandLog.push({ command, args, env: options.env ?? {}, input: options.input ?? "" }); + if (command === "git" && args[0] === "rev-parse" && args.includes("origin/main^{commit}")) { + return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" }; + } + if (command === "git" && args[0] === "rev-parse" && args.includes("HEAD^{commit}")) { + return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" }; + } + if (command === "which" && args[0] === "kubectl") { + return { code: 0, stdout: "/usr/local/bin/kubectl\n", stderr: "" }; + } + if (command.includes("kubectl") && args.includes("get") && args.includes("lease")) { + if (!lease) return { code: 1, stdout: "", stderr: "Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found" }; + return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; + } + if (command.includes("kubectl") && args.includes("create")) { + lease = JSON.parse(options.input); + lease.metadata.resourceVersion = "2"; + return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; + } + if (command.includes("kubectl") && args.includes("replace")) { + const nextLease = JSON.parse(options.input); + assert.equal(nextLease.metadata?.resourceVersion, lease?.metadata?.resourceVersion); + lease = nextLease; + lease.metadata.resourceVersion = String(Number(lease.metadata.resourceVersion ?? "2") + 1); + return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; + } + if (command.includes("kubectl") && args.includes("patch")) { + assert.ok(lease, "lease must exist before patch"); + const patch = parsePatch(args); + lease.metadata.annotations = { + ...lease.metadata.annotations, + ...(patch.metadata?.annotations ?? {}) + }; + lease.spec = { + ...lease.spec, + ...(patch.spec ?? {}) + }; + lease.metadata.resourceVersion = String(Number(lease.metadata.resourceVersion ?? "2") + 1); + return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; + } + if (command === process.execPath || command.endsWith("/node")) { + return { code: 0, stdout: JSON.stringify({ ok: true, command: args.join(" ") }), stderr: "" }; + } + return { code: 0, stdout: "", stderr: "" }; + }; +} + +test("lock classifier reports held and stale states with retryAfterSeconds", () => { + const lock = { + ownerTaskId: "task-a", + transactionId: "tx-a", + promotionCommit: "abc1234", + deployJsonHash: "sha256:a", + phase: "applying", + startedAt: iso(0), + updatedAt: iso(0), + ttlSeconds: 300, + targetNamespace: "hwlab-dev" + }; + const parsed = parseDeployLock(leaseFromLock(lock)); + const held = classifyDeployLock(parsed, new Date(iso(120000))); + assert.equal(held.held, true); + assert.equal(held.stale, false); + assert.equal(held.retryAfterSeconds, 180); + + const stale = classifyDeployLock(parsed, new Date(iso(301000))); + assert.equal(stale.held, false); + assert.equal(stale.stale, true); + assert.equal(stale.retryAfterSeconds, 0); +}); + +test("deploy-lock-held failure is structured before publish/apply", () => { + const failure = deployLockHeldFailure({ + ownerTaskId: "task-a", + transactionId: "tx-a", + promotionCommit: "abc1234", + deployJsonHash: "sha256:a", + phase: "applying", + updatedAt: iso(0), + ttlSeconds: 300, + targetNamespace: "hwlab-dev", + lockName: "hwlab-dev-cd-lock", + lockBackend: "Lease" + }, new Date(iso(100000))); + + assert.equal(failure.ok, false); + assert.equal(failure.error, "deploy-lock-held"); + assert.equal(failure.holder, "task-a"); + assert.equal(failure.promotionCommit, "abc1234"); + assert.equal(failure.phase, "applying"); + assert.equal(failure.retryAfterSeconds, 200); + assert.equal(failure.mutationAttempted, false); +}); + +test("second transaction exits deploy-lock-held without running side effects", async () => { + const repoRoot = await makeRepo(); + const commandLog = []; + let output = ""; + const code = await runDevCdApply([ + "--apply", + "--confirm-dev", + "--confirmed-non-production", + "--owner-task-id", + "task-b", + "--kubeconfig", + "/tmp/kubeconfig" + ], { + repoRoot, + env: {}, + runCommand: makeRunCommand({ + commandLog, + heldLock: { + promotionCommit: "abc1234", + deployJsonHash: "sha256:a", + ownerTaskId: "task-a", + transactionId: "tx-a", + phase: "publishing", + startedAt: iso(0), + updatedAt: iso(0), + ttlSeconds: 300, + liveBefore: { status: "pass" }, + targetNamespace: "hwlab-dev", + targetRef: "origin/main", + lockBackend: "Lease" + } + }), + httpGetJson: makeHttpGetJson(), + now: () => new Date(iso(100000)), + stdout: { write: (chunk) => { output += chunk; } } + }); + + const failure = JSON.parse(output); + assert.equal(code, 2); + assert.equal(failure.error, "deploy-lock-held"); + assert.equal(failure.holder, "task-a"); + assert.equal(failure.retryAfterSeconds, 200); + assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); + assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); +}); + +test("stale lock requires explicit break-stale-lock confirmation", async () => { + const repoRoot = await makeRepo(); + let output = ""; + const code = await runDevCdApply([ + "--apply", + "--confirm-dev", + "--confirmed-non-production", + "--owner-task-id", + "task-b", + "--kubeconfig", + "/tmp/kubeconfig" + ], { + repoRoot, + env: {}, + runCommand: makeRunCommand({ + heldLock: { + promotionCommit: "abc1234", + deployJsonHash: "sha256:a", + ownerTaskId: "task-a", + transactionId: "tx-a", + phase: "verifying", + startedAt: iso(0), + updatedAt: iso(0), + ttlSeconds: 300, + liveBefore: { status: "pass" }, + targetNamespace: "hwlab-dev", + targetRef: "origin/main", + lockBackend: "Lease" + } + }), + httpGetJson: makeHttpGetJson(), + now: () => new Date(iso(301000)), + stdout: { write: (chunk) => { output += chunk; } } + }); + + const failure = JSON.parse(output); + assert.equal(code, 2); + assert.equal(failure.error, "deploy-lock-held"); + assert.equal(failure.stale, true); + assert.equal(failure.requiresBreakStaleLock, true); +}); + +test("transaction runs phases, allows internal side-effect env, releases lock, and reports live verify", async () => { + const repoRoot = await makeRepo(); + const commandLog = []; + let output = ""; + const code = await runDevCdApply([ + "--apply", + "--confirm-dev", + "--confirmed-non-production", + "--break-stale-lock", + "--owner-task-id", + "task-b", + "--kubeconfig", + "/tmp/kubeconfig", + "--write-report" + ], { + repoRoot, + env: {}, + runCommand: makeRunCommand({ + commandLog, + heldLock: { + promotionCommit: "old1234", + deployJsonHash: "sha256:old", + ownerTaskId: "task-a", + transactionId: "tx-a", + phase: "verifying", + startedAt: iso(0), + updatedAt: iso(0), + ttlSeconds: 300, + liveBefore: { status: "pass" }, + targetNamespace: "hwlab-dev", + targetRef: "origin/main", + lockBackend: "Lease" + } + }), + httpGetJson: makeHttpGetJson(), + now: () => new Date(iso(301000)), + stdout: { write: (chunk) => { output += chunk; } } + }); + + const report = JSON.parse(output); + assert.equal(code, 0); + assert.equal(report.status, "pass"); + assert.deepEqual( + report.transaction.phases.map((phase) => phase.phase), + ["publishing", "applying", "verifying", "released"] + ); + assert.equal(report.devCdApply.lock.staleBreak.ownerTaskId, "task-a"); + assert.equal(report.transaction.release.status, "released"); + assert.equal(report.devCdApply.liveVerify.status, "pass"); + assert.equal(report.devCdApply.liveVerify.summary.checked, 2); + assert.equal(report.devCdApply.liveVerify.endpoints.some((endpoint) => endpoint.id === "cloud-web-16666"), true); + assert.equal(report.devCdApply.liveVerify.endpoints.some((endpoint) => endpoint.id === "cloud-api-16667"), true); + assert.equal(report.devCdApply.reportPaths.transaction, "reports/dev-gate/dev-cd-apply.json"); + + const publishCall = commandLog.find((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")); + const applyCall = commandLog.find((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")); + assert.ok(publishCall?.env.HWLAB_CD_TRANSACTION_ID); + assert.equal(applyCall?.env.HWLAB_CD_TRANSACTION_ID, publishCall.env.HWLAB_CD_TRANSACTION_ID); + + const writtenReport = JSON.parse(await readFile(path.join(repoRoot, "reports/dev-gate/dev-cd-apply.json"), "utf8")); + assert.equal(writtenReport.devCdApply.liveVerify.summary.checked, 2); +}); + +test("live verify summarizes 16666 and 16667 health identity", async () => { + const result = await verifyDevLive({ + now: () => new Date(iso()), + httpGetJson: makeHttpGetJson() + }, { expectedCommit: "abc1234" }); + + assert.equal(result.status, "pass"); + assert.equal(result.summary.checked, 2); + assert.deepEqual(result.endpoints.map((endpoint) => endpoint.url), [ + "http://74.48.78.17:16666/health/live", + "http://74.48.78.17:16667/health/live" + ]); + assert.equal(result.endpoints.every((endpoint) => endpoint.commitMatches), true); +}); + +test("legacy artifact publish CLI rejects side effects without transaction env", async () => { + await assert.rejects( + execFileAsync(process.execPath, ["scripts/dev-artifact-publish.mjs", "--publish", "--no-report"], { + cwd: repoRoot, + env: { + ...process.env, + HWLAB_CD_TRANSACTION_ID: "" + } + }), + (error) => { + assert.equal(error.code, 2); + const payload = JSON.parse(error.stdout); + assert.equal(payload.error, "cd-transaction-required"); + assert.equal(payload.script, "scripts/dev-artifact-publish.mjs"); + assert.equal(payload.mutationAttempted, false); + return true; + } + ); +}); diff --git a/scripts/src/dev-cd-transaction-guard.mjs b/scripts/src/dev-cd-transaction-guard.mjs new file mode 100644 index 00000000..64a6f104 --- /dev/null +++ b/scripts/src/dev-cd-transaction-guard.mjs @@ -0,0 +1,36 @@ +const transactionEnvNames = [ + "HWLAB_CD_TRANSACTION_ID", + "HWLAB_CD_TRANSACTION_OWNER", + "HWLAB_CD_LOCK_NAME" +]; + +export function hasDevCdTransactionEnv(env = process.env) { + return typeof env.HWLAB_CD_TRANSACTION_ID === "string" && env.HWLAB_CD_TRANSACTION_ID.trim().length > 0; +} + +export function devCdTransactionGuardFailure({ + script, + mode, + requiredEnv = "HWLAB_CD_TRANSACTION_ID" +}) { + return { + ok: false, + status: "blocked", + error: "cd-transaction-required", + code: "cd-transaction-required", + script, + mode, + requiredEnv, + acceptedEnv: transactionEnvNames, + entrypoint: "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report", + summary: `${script} ${mode} is a DEV CD side-effect step and must run inside scripts/dev-cd-apply.mjs.`, + devOnly: true, + prodTouched: false, + mutationAttempted: false + }; +} + +export function requireDevCdTransactionForSideEffect({ env = process.env, script, mode }) { + if (hasDevCdTransactionEnv(env)) return null; + return devCdTransactionGuardFailure({ script, mode }); +} diff --git a/scripts/src/dev-deploy-apply.mjs b/scripts/src/dev-deploy-apply.mjs index b05aae05..b98c7b67 100755 --- a/scripts/src/dev-deploy-apply.mjs +++ b/scripts/src/dev-deploy-apply.mjs @@ -17,6 +17,7 @@ import { } from "../../internal/cloud/code-agent-contract.mjs"; import { activeReportLifecycle } from "../../internal/dev-report-lifecycle.mjs"; import { DEV_ENDPOINT, ENVIRONMENT_DEV, SERVICE_IDS } from "../../internal/protocol/index.mjs"; +import { requireDevCdTransactionForSideEffect } from "./dev-cd-transaction-guard.mjs"; const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); const reportPath = "reports/dev-gate/dev-deploy-report.json"; @@ -72,7 +73,7 @@ const requiredValidationCommands = [ "node --check scripts/src/dev-deploy-apply.mjs", "node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked" ]; -const applyCommand = "node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"; +const applyCommand = "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"; const dryRunPlanCommand = "node scripts/dev-deploy-apply.mjs --dry-run --write-report"; const blockedDryRunCommand = "node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report"; const forbiddenActions = [ @@ -1771,6 +1772,17 @@ export async function runDevDeployApply(argv, io = {}) { const args = parseArgs(argv); const blockers = []; validateArgs(args, blockers); + if (args.apply) { + const transactionGuard = requireDevCdTransactionForSideEffect({ + env: io.env ?? process.env, + script: "scripts/dev-deploy-apply.mjs", + mode: "--apply" + }); + if (transactionGuard) { + stdout.write(`${JSON.stringify(transactionGuard, null, 2)}\n`); + return 2; + } + } const gitHeadCommitIdPromise = gitHeadCommit(); const [deploy, catalog, devKustomization, namespaceDoc, workloads, services, healthContract] = await Promise.all([ diff --git a/scripts/src/dev-deploy-apply.test.mjs b/scripts/src/dev-deploy-apply.test.mjs index 65e9845a..33834adb 100644 --- a/scripts/src/dev-deploy-apply.test.mjs +++ b/scripts/src/dev-deploy-apply.test.mjs @@ -14,6 +14,11 @@ import { resolveApplySourceCommit, resolveDevKubeconfigSelection } from "./dev-deploy-apply.mjs"; +import { + devCdTransactionGuardFailure, + hasDevCdTransactionEnv, + requireDevCdTransactionForSideEffect +} from "./dev-cd-transaction-guard.mjs"; const codeAgentBaseUrl = "http://172.26.26.227:17680/v1/responses"; @@ -109,6 +114,33 @@ test("allowlisted suspended template Job image change plans replacement", () => assert.equal(decision.newImage, "127.0.0.1:5000/hwlab/hwlab-agent-worker:new5678"); }); +test("DEV CD transaction guard rejects direct legacy side-effect calls", () => { + assert.equal(hasDevCdTransactionEnv({}), false); + assert.equal(hasDevCdTransactionEnv({ HWLAB_CD_TRANSACTION_ID: "tx-123" }), true); + + const failure = requireDevCdTransactionForSideEffect({ + env: {}, + script: "scripts/dev-deploy-apply.mjs", + mode: "--apply" + }); + assert.deepEqual(failure, devCdTransactionGuardFailure({ + script: "scripts/dev-deploy-apply.mjs", + mode: "--apply" + })); + assert.equal(failure.error, "cd-transaction-required"); + assert.equal(failure.mutationAttempted, false); + assert.match(failure.entrypoint, /scripts\/dev-cd-apply\.mjs/u); + + assert.equal( + requireDevCdTransactionForSideEffect({ + env: { HWLAB_CD_TRANSACTION_ID: "tx-123" }, + script: "scripts/dev-deploy-apply.mjs", + mode: "--apply" + }), + null + ); +}); + test("matching allowlisted suspended template Job image does not replace", () => { const image = "127.0.0.1:5000/hwlab/hwlab-cli:73b379f"; const decision = decideDevTemplateJobReplacement({ diff --git a/scripts/validate-dev-gate-report.mjs b/scripts/validate-dev-gate-report.mjs index 0eb79616..825c8b98 100644 --- a/scripts/validate-dev-gate-report.mjs +++ b/scripts/validate-dev-gate-report.mjs @@ -20,6 +20,7 @@ const reportsDir = path.join(repoRoot, "reports/dev-gate"); const contractIssue = "pikasTech/HWLAB#31"; const issueFamily = Object.freeze({ + DEV_CD_APPLY: "pikasTech/HWLAB#274", DEV_DEPLOY_APPLY: "pikasTech/HWLAB#33", DEV_GATE_PREFLIGHT: "pikasTech/HWLAB#34", DEV_ARTIFACT_PUBLISH: "pikasTech/HWLAB#35", @@ -143,6 +144,32 @@ const requiredDevCloudWorkbenchDocs = [ "docs/reference/code-agent-chat-readiness.md" ]; const reportFamilyTemplates = new Map([ + [ + "dev-cd-apply", + { + issue: issueFamily.DEV_CD_APPLY, + requiredDocs: [ + "docs/reference/deployment-publish.md", + "docs/reference/dev-runtime-boundary.md", + "docs/dev-artifact-publish.md", + "docs/dev-deploy-apply.md", + "docs/artifact-catalog.md" + ], + requiredValidationCommands: [ + "node --check scripts/dev-cd-apply.mjs", + "node --check scripts/src/dev-cd-apply.mjs", + "node --test scripts/src/dev-cd-apply.test.mjs", + "node --check scripts/dev-artifact-publish.mjs", + "node --check scripts/src/dev-deploy-apply.mjs", + "node --test scripts/src/dev-deploy-apply.test.mjs", + "node scripts/deploy-desired-state-plan.mjs --check", + "node scripts/validate-artifact-catalog.mjs", + "git diff --check" + ], + requiredSmokeCommand: "npm run check", + requiredDryRunCommand: "node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report" + } + ], [ "dev-gate-report-contract", { @@ -714,6 +741,9 @@ async function validateReport(relativePath) { if (report.taskId === "dev-deploy-apply") { assertDevDeployApplyReport(report, label); } + if (report.taskId === "dev-cd-apply") { + assertDevCdApplyReport(report, label); + } } async function assertDocumentSet(documentsValue, label, requiredDocs) { @@ -937,8 +967,8 @@ function assertDevDeployApplyReport(report, label) { if (plan.conclusion.status === "ready") { assert.equal(plan.manualCommands.status, "ready", `${label}.devDeployApply.manualCommands.status`); assert.ok( - plan.manualCommands.afterHumanApproval.includes("node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"), - `${label}.devDeployApply.manualCommands.afterHumanApproval missing apply command` + plan.manualCommands.afterHumanApproval.includes("node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"), + `${label}.devDeployApply.manualCommands.afterHumanApproval missing transaction apply command` ); } @@ -963,6 +993,90 @@ function assertDevDeployApplyReport(report, label) { } } +function assertDevCdApplyReport(report, label) { + assert.equal(report.issue, issueFamily.DEV_CD_APPLY, `${label}.issue`); + assertStringArray(report.supports, `${label}.supports`, { minLength: 4 }); + for (const supportedIssue of [ + "pikasTech/HWLAB#274", + "pikasTech/HWLAB#116", + "pikasTech/HWLAB#235" + ]) { + assert.ok(report.supports.includes(supportedIssue), `${label}.supports missing ${supportedIssue}`); + } + + assertObject(report.transaction, `${label}.transaction`); + for (const field of ["transactionId", "ownerTaskId", "targetNamespace", "lockName", "lockBackend", "startedAt", "finishedAt", "ttlSeconds", "phases"]) { + assert.ok(Object.hasOwn(report.transaction, field), `${label}.transaction missing ${field}`); + } + assert.equal(report.transaction.targetNamespace, "hwlab-dev", `${label}.transaction.targetNamespace`); + assert.equal(report.transaction.lockName, "hwlab-dev-cd-lock", `${label}.transaction.lockName`); + assert.equal(report.transaction.lockBackend, "Lease", `${label}.transaction.lockBackend`); + assertString(report.transaction.transactionId, `${label}.transaction.transactionId`); + assertString(report.transaction.ownerTaskId, `${label}.transaction.ownerTaskId`); + assert.ok(Number.isInteger(report.transaction.ttlSeconds), `${label}.transaction.ttlSeconds`); + assertArray(report.transaction.phases, `${label}.transaction.phases`); + assert.ok( + report.transaction.phases.some((phase) => phase.phase === "publishing"), + `${label}.transaction.phases must include publishing` + ); + + assertObject(report.devCdApply, `${label}.devCdApply`); + for (const field of ["mode", "target", "deployJson", "lock", "steps", "liveBefore", "liveVerify", "reportPaths", "safety"]) { + assert.ok(Object.hasOwn(report.devCdApply, field), `${label}.devCdApply missing ${field}`); + } + assert.equal(report.devCdApply.mode, "apply", `${label}.devCdApply.mode`); + + assertObject(report.devCdApply.target, `${label}.devCdApply.target`); + assert.equal(report.devCdApply.target.namespace, "hwlab-dev", `${label}.devCdApply.target.namespace`); + assert.equal(report.devCdApply.target.apiEndpoint, "http://74.48.78.17:16667", `${label}.devCdApply.target.apiEndpoint`); + assert.equal(report.devCdApply.target.browserEndpoint, "http://74.48.78.17:16666/", `${label}.devCdApply.target.browserEndpoint`); + assertString(report.devCdApply.target.shortCommitId, `${label}.devCdApply.target.shortCommitId`); + + assertObject(report.devCdApply.deployJson, `${label}.devCdApply.deployJson`); + for (const side of ["before", "after"]) { + assertObject(report.devCdApply.deployJson[side], `${label}.devCdApply.deployJson.${side}`); + assert.equal(report.devCdApply.deployJson[side].path, "deploy/deploy.json", `${label}.devCdApply.deployJson.${side}.path`); + assertString(report.devCdApply.deployJson[side].hash, `${label}.devCdApply.deployJson.${side}.hash`); + assertString(report.devCdApply.deployJson[side].commitId, `${label}.devCdApply.deployJson.${side}.commitId`); + } + + assertObject(report.devCdApply.lock, `${label}.devCdApply.lock`); + assert.equal(typeof report.devCdApply.lock.acquired, "boolean", `${label}.devCdApply.lock.acquired`); + + assertArray(report.devCdApply.steps, `${label}.devCdApply.steps`); + for (const [index, step] of report.devCdApply.steps.entries()) { + const stepLabel = `${label}.devCdApply.steps[${index}]`; + assertObject(step, stepLabel); + for (const field of ["id", "phase", "status", "command", "code", "startedAt", "finishedAt"]) { + assert.ok(Object.hasOwn(step, field), `${stepLabel} missing ${field}`); + } + assert.ok(["publishing", "applying"].includes(step.phase), `${stepLabel}.phase`); + assert.ok(["pass", "blocked"].includes(step.status), `${stepLabel}.status`); + assertString(step.command, `${stepLabel}.command`); + } + + assertObject(report.devCdApply.liveVerify, `${label}.devCdApply.liveVerify`); + assert.ok(["pass", "blocked", "not_run"].includes(report.devCdApply.liveVerify.status), `${label}.devCdApply.liveVerify.status`); + assertArray(report.devCdApply.liveVerify.endpoints, `${label}.devCdApply.liveVerify.endpoints`); + assertObject(report.devCdApply.liveVerify.summary, `${label}.devCdApply.liveVerify.summary`); + if (report.devCdApply.liveVerify.status !== "not_run") { + const urls = report.devCdApply.liveVerify.endpoints.map((endpoint) => endpoint.url); + assert.ok(urls.includes("http://74.48.78.17:16666/health/live"), `${label}.devCdApply.liveVerify missing 16666 health`); + assert.ok(urls.includes("http://74.48.78.17:16667/health/live"), `${label}.devCdApply.liveVerify missing 16667 health`); + } + + assertObject(report.devCdApply.reportPaths, `${label}.devCdApply.reportPaths`); + assert.equal(report.devCdApply.reportPaths.artifacts, "reports/dev-gate/dev-artifacts.json", `${label}.devCdApply.reportPaths.artifacts`); + assert.equal(report.devCdApply.reportPaths.deployApply, "reports/dev-gate/dev-deploy-report.json", `${label}.devCdApply.reportPaths.deployApply`); + + assertObject(report.devCdApply.safety, `${label}.devCdApply.safety`); + assert.equal(report.devCdApply.safety.prodTouched, false, `${label}.devCdApply.safety.prodTouched`); + assert.equal(report.devCdApply.safety.secretValuesRead, false, `${label}.devCdApply.safety.secretValuesRead`); + assert.equal(report.devCdApply.safety.secretValuesPrinted, false, `${label}.devCdApply.safety.secretValuesPrinted`); + assert.equal(report.devCdApply.safety.oldPublicPortsUsed, false, `${label}.devCdApply.safety.oldPublicPortsUsed`); + assert.equal(report.devCdApply.safety.singleLock, "Lease/hwlab-dev/hwlab-dev-cd-lock", `${label}.devCdApply.safety.singleLock`); +} + function assertShaOrNotPublished(value, label) { assertString(value, label);