fix: serialize DEV CD apply

This commit is contained in:
Code Queue Review
2026-05-23 09:08:31 +00:00
parent dc031e0640
commit 24ccabbb89
15 changed files with 2140 additions and 34 deletions
+1
View File
@@ -45,6 +45,7 @@ HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥
- Cloud Web M3 只读护栏:`npm run web:m3-readonly`
- Cloud Workbench 布局/遮挡 smoke`node scripts/dev-cloud-workbench-layout-smoke.mjs --static`
- DEV artifact 发布预检:`npm run dev-artifact:preflight`
- DEV CD 单事务发布/应用/验证:`npm run dev-cd:apply`
- runner GitHub 可见性预检:`npm run runner:issue-visibility:preflight`
- D601 k3s 只读观测:`npm run d601:k3s:readonly`
+21 -7
View File
@@ -1,8 +1,11 @@
# HWLAB DEV Artifact Publish
`scripts/dev-artifact-publish.mjs` is the DEV-only build and publish path for
D601 local/internal registry artifacts. It does not deploy workloads, read
secrets, enable PROD, or push to GHCR/Docker Hub/other third-party registries.
`scripts/dev-artifact-publish.mjs` is the DEV-only artifact preflight/build
backend for D601 local/internal registry artifacts. Publish side effects are
normally entered through `scripts/dev-cd-apply.mjs`, which sets the transaction
environment and serializes publish/apply/verify with the DEV CD Lease lock.
The artifact script does not deploy workloads, read secrets, enable PROD, or
push to GHCR/Docker Hub/other third-party registries.
## Scope
@@ -58,24 +61,35 @@ Build only:
node scripts/dev-artifact-publish.mjs --build
```
Build and publish to the D601 local/internal registry:
Build and publish to the D601 local/internal registry inside the DEV CD
transaction:
```sh
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
The underlying single-step command is kept only for transaction internals and
specialized tests that explicitly inject the transaction environment:
```sh
node scripts/dev-artifact-publish.mjs --publish
```
Direct `--publish` without `HWLAB_CD_TRANSACTION_ID` is rejected with
`cd-transaction-required`; this is a bypass guard, not a second lock.
Daily Cloud Web source validation does not require ignored `dist/` to exist:
```sh
node web/hwlab-cloud-web/scripts/check.mjs
```
For `hwlab-cloud-web`, the build/publish path refreshes and verifies the static
bundle before image build:
For `hwlab-cloud-web`, the build path refreshes and verifies the static bundle
before image build; publish still goes through the transaction command above:
```sh
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web --report reports/dev-gate/dev-artifacts-hwlab-cloud-web-<tag>.json
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
The runtime wrapper serves `/app/web/hwlab-cloud-web/dist` before the source
+18 -7
View File
@@ -1,9 +1,11 @@
# HWLAB DEV Deploy Apply
This document records the DEV-only apply/preflight path for
This document records the DEV-only apply/preflight backend for
`pikasTech/HWLAB#33` and the apply-plan/report path for
`pikasTech/HWLAB#50`. It targets only the D601 `hwlab-dev` namespace and must
not be used for PROD.
`pikasTech/HWLAB#50`. Real apply side effects are normally entered through
`scripts/dev-cd-apply.mjs`, which sets the transaction environment and
serializes publish/apply/verify with the DEV CD Lease lock. The apply backend
targets only the D601 `hwlab-dev` namespace and must not be used for PROD.
## Scope
@@ -56,13 +58,22 @@ node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run
```
Only after CI publish evidence and executor prerequisites exist, a real DEV
apply requires explicit non-production confirmation:
CD transaction is:
```sh
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
The underlying apply backend still requires explicit non-production
confirmation and a transaction environment:
```sh
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
The script refuses PROD flags, secret-read flags, heavyweight e2e flags, and
Direct `--apply` without `HWLAB_CD_TRANSACTION_ID` is rejected with
`cd-transaction-required`; this is a bypass guard, not a second lock. The
script refuses PROD flags, secret-read flags, heavyweight e2e flags, and
force-push flags. It also refuses to apply if the DEV artifact catalog still
contains skeleton-only unpublished images.
@@ -176,7 +187,7 @@ node scripts/dev-deploy-apply.mjs --dry-run --write-report
node scripts/validate-dev-gate-report.mjs
```
Only after that approval may the DEV-only apply command be used.
Only after that approval may the DEV CD transaction command be used.
## Cloud Web Manual DEV Rollout
@@ -202,7 +213,7 @@ freshness gates.
```sh
node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web --report reports/dev-gate/dev-artifacts-hwlab-cloud-web-<tag>.json
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
Probe the image locally before touching k3s:
+1 -1
View File
@@ -20,7 +20,7 @@
| 文档治理与 docs-spec 本地权威 | [documentation-governance.md](documentation-governance.md) |
| 架构和 M3 上位约束 | [architecture.md](architecture.md) |
| DEV 运行态、端口、k3s、DB readiness 和环境边界 | [dev-runtime-boundary.md](dev-runtime-boundary.md) |
| 部署正规化、artifact 发布、回滚和 Cloud Web rollout | [deployment-publish.md](deployment-publish.md) |
| 部署正规化、DEV CD 单事务入口、artifact 发布、回滚和 Cloud Web rollout | [deployment-publish.md](deployment-publish.md) |
| Cloud Workbench 默认界面和 UX 边界 | [cloud-workbench.md](cloud-workbench.md) |
| Code Agent chat 同源通道 readiness 与真实回复判定 | [code-agent-chat-readiness.md](code-agent-chat-readiness.md) |
| MVP E2E 验收测试与带编号测试报告 issue 规则 | [MVP-e2e-acceptance.md](MVP-e2e-acceptance.md) |
+63 -13
View File
@@ -25,9 +25,44 @@ truth.
| Stage | Current HWLAB state | Required next behavior |
| --- | --- | --- |
| 1. Long-term reference | This file records DEV deployment, Cloud Web publish, API/edge/health, frp, D601 registry, k3s rollout, rollback, and verification rules. `AGENTS.md` indexes it. | Keep process issues as sources only; update this reference when a deployment rule changes. |
| 2. CLI or controlled scripts | Existing controlled entrypoints cover contract planning, desired-state planning, artifact publish, DEV apply/report, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. | Converge check, plan, publish, apply, rollback, status, and verify into HWLAB CLI or a small controlled script surface backed by `deploy/deploy.json`. |
| 2. CLI or controlled scripts | `scripts/dev-cd-apply.mjs` is the only formal DEV CD side-effect entrypoint for publish, desired-state refresh, apply, and live verify. Other controlled scripts still cover contract planning, dry-run planning, preflight, single-step internals, k3s read-only visibility, and Cloud Web checks. Rollback is documented as commands/report hints but is not yet one unified CLI command. | Converge check, plan, rollback, status, and verify into HWLAB CLI while preserving the single transaction lock for side effects backed by `deploy/deploy.json`. |
| 3. UniDesk CI/CD plus images | DEV services are imageized and published to the D601 registry. D601 rollout still includes manual bridge steps for Cloud Web and frp/k3s execution. | Move build, publish, desired-state refresh, rollout, post-deploy validation, and report upload into standard UniDesk CI/CD jobs without replacing HWLAB runtime. |
## DEV CD Transaction
The stable DEV CD side-effect command is:
```sh
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
This command is the only formal path that may publish DEV artifacts, refresh
desired-state from the publish report, apply to `hwlab-dev`, and verify public
`16666/16667` in one run. It reads `deploy/deploy.json` at transaction start,
records the manifest hash and target commit, acquires
`Lease/hwlab-dev/hwlab-dev-cd-lock`, runs the internal steps serially, writes
`reports/dev-gate/dev-cd-apply.json`, and releases the Lease.
The Lease is a deployment-plane mutex, not a desired-state source. Its
annotations record at least `promotionCommit`, `deployJsonHash`,
`ownerTaskId`, `transactionId`, `phase`, `startedAt`, `updatedAt`,
`ttlSeconds`, `liveBefore`, and `targetNamespace`. `deploy/deploy.json`,
`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml`
remain the only authoritative deploy inputs.
If another transaction holds the Lease, the command must stop before publish or
apply with structured `deploy-lock-held` output that includes holder,
promotion commit, phase, target namespace, and retry seconds. Expired locks are
not silently overwritten; taking over a stale lock requires both
`--break-stale-lock` and `--confirm-dev`, and the previous holder is retained
in audit annotations/report fields.
The old side-effect commands are now internal transaction steps. Direct manual
use of `scripts/dev-artifact-publish.mjs --publish` or
`scripts/dev-deploy-apply.mjs --apply` without `HWLAB_CD_TRANSACTION_ID` is
rejected with `cd-transaction-required`; preflight, build, dry-run, check, and
read-only modes remain available.
## Artifact Publish
DEV artifacts publish to the D601 local/internal registry:
@@ -36,18 +71,22 @@ DEV artifacts publish to the D601 local/internal registry:
127.0.0.1:5000/hwlab/*
```
The stable publish command is:
The single-step publish command is a transaction-internal side-effect step:
```sh
node scripts/dev-artifact-publish.mjs --publish
```
For a single Cloud Web image:
It only runs in side-effect mode when `HWLAB_CD_TRANSACTION_ID` is set by
`scripts/dev-cd-apply.mjs`. Use `node scripts/dev-cd-apply.mjs --apply
--confirm-dev --confirmed-non-production --write-report` for normal DEV CD.
For Cloud Web source freshness before the transaction:
```sh
node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
Cloud Web must build `web/hwlab-cloud-web/dist` before publish because the
@@ -130,7 +169,12 @@ services may still use internal `6667`.
## Cloud Web Manual DEV Rollout
The verified manual path for `hwlab-cloud-web` is:
This section is retained as historical operational baseline for what the
transaction reproduces. New DEV side effects should use
`scripts/dev-cd-apply.mjs`; do not bypass the Lease with ad hoc publish,
`kubectl set image`, or restart commands.
The previous verified manual path for `hwlab-cloud-web` was:
```sh
cd /home/ubuntu/workspace/hwlab
@@ -141,6 +185,9 @@ KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/h
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
```
That historical sequence is no longer the normal side-effect path; direct
`--publish` is rejected unless it is called by the transaction.
The rollout must use D601 native k3s:
```sh
@@ -171,16 +218,18 @@ scroll, and an internal Markdown help surface. The frontend contract is
## Apply Boundary
`scripts/dev-deploy-apply.mjs` is the DEV-only apply/report path. A real apply
requires explicit non-production flags:
`scripts/dev-deploy-apply.mjs` is the DEV-only apply/report backend. A real
apply requires the transaction environment created by `scripts/dev-cd-apply.mjs`
plus explicit non-production flags:
```sh
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```
The apply path must stop before mutation when artifact evidence, registry
digests, kube access, DEV health, or required runtime readiness are missing.
It must not read secret values or target PROD.
Direct manual `--apply` calls without `HWLAB_CD_TRANSACTION_ID` return
`cd-transaction-required`. The apply path must stop before mutation when
artifact evidence, registry digests, kube access, DEV health, or required
runtime readiness are missing. It must not read secret values or target PROD.
Suspended template Jobs are immutable in `spec.template`. Only the explicit
DEV allowlist may be replaced by the apply script:
@@ -220,9 +269,10 @@ use the repo-owned scripts below instead of ad hoc shell fragments.
| Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. |
| Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. |
| Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. |
| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` | CI job should run this and upload digest reports. |
| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Make refresh an explicit CLI/CD step after publish. |
| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep apply behind confirmation flags and route through CLI/CD approval. |
| Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. |
| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. |
| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Internal transaction step after publish; read/write desired-state only from repo files. |
| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` inside `HWLAB_CD_TRANSACTION_ID` | Internal transaction step; direct side-effect calls are rejected. |
| Roll back DEV Deployment | `kubectl -n hwlab-dev rollout undo deployment/<name>` with explicit D601 kubeconfig | Add `hwlab deploy rollback --env dev --service <id> --to-revision <n>`. |
| Verify status | `curl` checks plus `node scripts/d601-k3s-readonly-observability.mjs` | Add `hwlab deploy verify --env dev --write-report`. |
+3 -2
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -1140,7 +1140,7 @@
"node scripts/validate-dev-gate-report.mjs"
],
"afterHumanApproval": [
"node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"
"node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"
],
"summary": "Run the dry-run plan and validator immediately before requesting human approval; live apply still requires the explicit DEV confirmation flags."
},
+12
View File
@@ -16,6 +16,7 @@ import {
resolveDevArtifactServices,
serviceInventoryFromServices
} from "./src/dev-artifact-services.mjs";
import { requireDevCdTransactionForSideEffect } from "./src/dev-cd-transaction-guard.mjs";
import { runDevBaseImagePreflight } from "./src/dev-base-image-preflight.mjs";
import { probeRegistryCapabilities } from "./src/registry-capabilities.mjs";
import { inspectCloudWebDistFreshness } from "../web/hwlab-cloud-web/scripts/dist-contract.mjs";
@@ -1120,6 +1121,17 @@ async function main() {
printHelp();
return;
}
if (args.mode === "publish") {
const transactionGuard = requireDevCdTransactionForSideEffect({
script: "scripts/dev-artifact-publish.mjs",
mode: "--publish"
});
if (transactionGuard) {
console.log(JSON.stringify(transactionGuard, null, 2));
process.exitCode = 2;
return;
}
}
args.registryPrefix = validateRegistryPrefix(args.registryPrefix);
const [baseImagePreflight, registryCapabilities] = await Promise.all([
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env node
import { formatDevCdApplyFailure, runDevCdApply } from "./src/dev-cd-apply.mjs";
try {
process.exitCode = await runDevCdApply(process.argv.slice(2));
} catch (error) {
process.stdout.write(`${JSON.stringify(formatDevCdApplyFailure(error), null, 2)}\n`);
process.exitCode = 1;
}
File diff suppressed because it is too large Load Diff
+382
View File
@@ -0,0 +1,382 @@
import assert from "node:assert/strict";
import test from "node:test";
import { execFile } from "node:child_process";
import { mkdir, readFile, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import {
buildLockAnnotations,
classifyDeployLock,
deployLockHeldFailure,
parseDeployLock,
runDevCdApply,
verifyDevLive
} from "./dev-cd-apply.mjs";
const execFileAsync = promisify(execFile);
const repoRoot = path.resolve(path.dirname(new URL(import.meta.url).pathname), "../..");
function iso(offsetMs = 0) {
return new Date(Date.parse("2026-05-23T08:00:00.000Z") + offsetMs).toISOString();
}
function leaseFromLock(lock) {
return {
apiVersion: "coordination.k8s.io/v1",
kind: "Lease",
metadata: {
name: "hwlab-dev-cd-lock",
namespace: "hwlab-dev",
resourceVersion: "1",
annotations: buildLockAnnotations(lock)
},
spec: {
holderIdentity: `${lock.ownerTaskId}/${lock.transactionId}`,
leaseDurationSeconds: lock.ttlSeconds,
acquireTime: lock.startedAt,
renewTime: lock.updatedAt
}
};
}
function parsePatch(args) {
const index = args.indexOf("-p");
assert.notEqual(index, -1, "kubectl patch must include -p");
return JSON.parse(args[index + 1]);
}
async function makeRepo() {
const repoRoot = await fsTempDir();
await mkdir(path.join(repoRoot, "deploy"), { recursive: true });
await writeFile(
path.join(repoRoot, "deploy/deploy.json"),
`${JSON.stringify({
manifestVersion: "v1",
environment: "dev",
namespace: "hwlab-dev",
endpoint: "http://74.48.78.17:16667",
commitId: "abc1234"
}, null, 2)}\n`
);
return repoRoot;
}
async function fsTempDir() {
return await import("node:fs/promises").then((fs) => fs.mkdtemp(path.join(os.tmpdir(), "hwlab-cd-")));
}
function makeHttpGetJson() {
return async (url) => {
if (url.includes("16666")) {
return {
statusCode: 200,
json: {
serviceId: "hwlab-cloud-web",
environment: "dev",
status: "ok",
revision: "abc1234",
image: { reference: "127.0.0.1:5000/hwlab/hwlab-cloud-web:abc1234", tag: "abc1234" },
observedAt: iso()
}
};
}
return {
statusCode: 200,
json: {
serviceId: "hwlab-cloud-api",
environment: "dev",
status: "degraded",
ready: false,
commit: { id: "abc1234", source: "runtime-env" },
image: { reference: "127.0.0.1:5000/hwlab/hwlab-cloud-api:abc1234", tag: "abc1234" },
blockerCodes: ["runtime_durable_adapter_auth_blocked"],
observedAt: iso()
}
};
};
}
function makeRunCommand({ heldLock = null, commandLog = [] } = {}) {
let lease = heldLock ? leaseFromLock(heldLock) : null;
return async (command, args, options = {}) => {
commandLog.push({ command, args, env: options.env ?? {}, input: options.input ?? "" });
if (command === "git" && args[0] === "rev-parse" && args.includes("origin/main^{commit}")) {
return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" };
}
if (command === "git" && args[0] === "rev-parse" && args.includes("HEAD^{commit}")) {
return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" };
}
if (command === "which" && args[0] === "kubectl") {
return { code: 0, stdout: "/usr/local/bin/kubectl\n", stderr: "" };
}
if (command.includes("kubectl") && args.includes("get") && args.includes("lease")) {
if (!lease) return { code: 1, stdout: "", stderr: "Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found" };
return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" };
}
if (command.includes("kubectl") && args.includes("create")) {
lease = JSON.parse(options.input);
lease.metadata.resourceVersion = "2";
return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" };
}
if (command.includes("kubectl") && args.includes("replace")) {
const nextLease = JSON.parse(options.input);
assert.equal(nextLease.metadata?.resourceVersion, lease?.metadata?.resourceVersion);
lease = nextLease;
lease.metadata.resourceVersion = String(Number(lease.metadata.resourceVersion ?? "2") + 1);
return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" };
}
if (command.includes("kubectl") && args.includes("patch")) {
assert.ok(lease, "lease must exist before patch");
const patch = parsePatch(args);
lease.metadata.annotations = {
...lease.metadata.annotations,
...(patch.metadata?.annotations ?? {})
};
lease.spec = {
...lease.spec,
...(patch.spec ?? {})
};
lease.metadata.resourceVersion = String(Number(lease.metadata.resourceVersion ?? "2") + 1);
return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" };
}
if (command === process.execPath || command.endsWith("/node")) {
return { code: 0, stdout: JSON.stringify({ ok: true, command: args.join(" ") }), stderr: "" };
}
return { code: 0, stdout: "", stderr: "" };
};
}
test("lock classifier reports held and stale states with retryAfterSeconds", () => {
const lock = {
ownerTaskId: "task-a",
transactionId: "tx-a",
promotionCommit: "abc1234",
deployJsonHash: "sha256:a",
phase: "applying",
startedAt: iso(0),
updatedAt: iso(0),
ttlSeconds: 300,
targetNamespace: "hwlab-dev"
};
const parsed = parseDeployLock(leaseFromLock(lock));
const held = classifyDeployLock(parsed, new Date(iso(120000)));
assert.equal(held.held, true);
assert.equal(held.stale, false);
assert.equal(held.retryAfterSeconds, 180);
const stale = classifyDeployLock(parsed, new Date(iso(301000)));
assert.equal(stale.held, false);
assert.equal(stale.stale, true);
assert.equal(stale.retryAfterSeconds, 0);
});
test("deploy-lock-held failure is structured before publish/apply", () => {
const failure = deployLockHeldFailure({
ownerTaskId: "task-a",
transactionId: "tx-a",
promotionCommit: "abc1234",
deployJsonHash: "sha256:a",
phase: "applying",
updatedAt: iso(0),
ttlSeconds: 300,
targetNamespace: "hwlab-dev",
lockName: "hwlab-dev-cd-lock",
lockBackend: "Lease"
}, new Date(iso(100000)));
assert.equal(failure.ok, false);
assert.equal(failure.error, "deploy-lock-held");
assert.equal(failure.holder, "task-a");
assert.equal(failure.promotionCommit, "abc1234");
assert.equal(failure.phase, "applying");
assert.equal(failure.retryAfterSeconds, 200);
assert.equal(failure.mutationAttempted, false);
});
test("second transaction exits deploy-lock-held without running side effects", async () => {
const repoRoot = await makeRepo();
const commandLog = [];
let output = "";
const code = await runDevCdApply([
"--apply",
"--confirm-dev",
"--confirmed-non-production",
"--owner-task-id",
"task-b",
"--kubeconfig",
"/tmp/kubeconfig"
], {
repoRoot,
env: {},
runCommand: makeRunCommand({
commandLog,
heldLock: {
promotionCommit: "abc1234",
deployJsonHash: "sha256:a",
ownerTaskId: "task-a",
transactionId: "tx-a",
phase: "publishing",
startedAt: iso(0),
updatedAt: iso(0),
ttlSeconds: 300,
liveBefore: { status: "pass" },
targetNamespace: "hwlab-dev",
targetRef: "origin/main",
lockBackend: "Lease"
}
}),
httpGetJson: makeHttpGetJson(),
now: () => new Date(iso(100000)),
stdout: { write: (chunk) => { output += chunk; } }
});
const failure = JSON.parse(output);
assert.equal(code, 2);
assert.equal(failure.error, "deploy-lock-held");
assert.equal(failure.holder, "task-a");
assert.equal(failure.retryAfterSeconds, 200);
assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false);
assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false);
});
test("stale lock requires explicit break-stale-lock confirmation", async () => {
const repoRoot = await makeRepo();
let output = "";
const code = await runDevCdApply([
"--apply",
"--confirm-dev",
"--confirmed-non-production",
"--owner-task-id",
"task-b",
"--kubeconfig",
"/tmp/kubeconfig"
], {
repoRoot,
env: {},
runCommand: makeRunCommand({
heldLock: {
promotionCommit: "abc1234",
deployJsonHash: "sha256:a",
ownerTaskId: "task-a",
transactionId: "tx-a",
phase: "verifying",
startedAt: iso(0),
updatedAt: iso(0),
ttlSeconds: 300,
liveBefore: { status: "pass" },
targetNamespace: "hwlab-dev",
targetRef: "origin/main",
lockBackend: "Lease"
}
}),
httpGetJson: makeHttpGetJson(),
now: () => new Date(iso(301000)),
stdout: { write: (chunk) => { output += chunk; } }
});
const failure = JSON.parse(output);
assert.equal(code, 2);
assert.equal(failure.error, "deploy-lock-held");
assert.equal(failure.stale, true);
assert.equal(failure.requiresBreakStaleLock, true);
});
test("transaction runs phases, allows internal side-effect env, releases lock, and reports live verify", async () => {
const repoRoot = await makeRepo();
const commandLog = [];
let output = "";
const code = await runDevCdApply([
"--apply",
"--confirm-dev",
"--confirmed-non-production",
"--break-stale-lock",
"--owner-task-id",
"task-b",
"--kubeconfig",
"/tmp/kubeconfig",
"--write-report"
], {
repoRoot,
env: {},
runCommand: makeRunCommand({
commandLog,
heldLock: {
promotionCommit: "old1234",
deployJsonHash: "sha256:old",
ownerTaskId: "task-a",
transactionId: "tx-a",
phase: "verifying",
startedAt: iso(0),
updatedAt: iso(0),
ttlSeconds: 300,
liveBefore: { status: "pass" },
targetNamespace: "hwlab-dev",
targetRef: "origin/main",
lockBackend: "Lease"
}
}),
httpGetJson: makeHttpGetJson(),
now: () => new Date(iso(301000)),
stdout: { write: (chunk) => { output += chunk; } }
});
const report = JSON.parse(output);
assert.equal(code, 0);
assert.equal(report.status, "pass");
assert.deepEqual(
report.transaction.phases.map((phase) => phase.phase),
["publishing", "applying", "verifying", "released"]
);
assert.equal(report.devCdApply.lock.staleBreak.ownerTaskId, "task-a");
assert.equal(report.transaction.release.status, "released");
assert.equal(report.devCdApply.liveVerify.status, "pass");
assert.equal(report.devCdApply.liveVerify.summary.checked, 2);
assert.equal(report.devCdApply.liveVerify.endpoints.some((endpoint) => endpoint.id === "cloud-web-16666"), true);
assert.equal(report.devCdApply.liveVerify.endpoints.some((endpoint) => endpoint.id === "cloud-api-16667"), true);
assert.equal(report.devCdApply.reportPaths.transaction, "reports/dev-gate/dev-cd-apply.json");
const publishCall = commandLog.find((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs"));
const applyCall = commandLog.find((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs"));
assert.ok(publishCall?.env.HWLAB_CD_TRANSACTION_ID);
assert.equal(applyCall?.env.HWLAB_CD_TRANSACTION_ID, publishCall.env.HWLAB_CD_TRANSACTION_ID);
const writtenReport = JSON.parse(await readFile(path.join(repoRoot, "reports/dev-gate/dev-cd-apply.json"), "utf8"));
assert.equal(writtenReport.devCdApply.liveVerify.summary.checked, 2);
});
test("live verify summarizes 16666 and 16667 health identity", async () => {
const result = await verifyDevLive({
now: () => new Date(iso()),
httpGetJson: makeHttpGetJson()
}, { expectedCommit: "abc1234" });
assert.equal(result.status, "pass");
assert.equal(result.summary.checked, 2);
assert.deepEqual(result.endpoints.map((endpoint) => endpoint.url), [
"http://74.48.78.17:16666/health/live",
"http://74.48.78.17:16667/health/live"
]);
assert.equal(result.endpoints.every((endpoint) => endpoint.commitMatches), true);
});
test("legacy artifact publish CLI rejects side effects without transaction env", async () => {
await assert.rejects(
execFileAsync(process.execPath, ["scripts/dev-artifact-publish.mjs", "--publish", "--no-report"], {
cwd: repoRoot,
env: {
...process.env,
HWLAB_CD_TRANSACTION_ID: ""
}
}),
(error) => {
assert.equal(error.code, 2);
const payload = JSON.parse(error.stdout);
assert.equal(payload.error, "cd-transaction-required");
assert.equal(payload.script, "scripts/dev-artifact-publish.mjs");
assert.equal(payload.mutationAttempted, false);
return true;
}
);
});
+36
View File
@@ -0,0 +1,36 @@
const transactionEnvNames = [
"HWLAB_CD_TRANSACTION_ID",
"HWLAB_CD_TRANSACTION_OWNER",
"HWLAB_CD_LOCK_NAME"
];
export function hasDevCdTransactionEnv(env = process.env) {
return typeof env.HWLAB_CD_TRANSACTION_ID === "string" && env.HWLAB_CD_TRANSACTION_ID.trim().length > 0;
}
export function devCdTransactionGuardFailure({
script,
mode,
requiredEnv = "HWLAB_CD_TRANSACTION_ID"
}) {
return {
ok: false,
status: "blocked",
error: "cd-transaction-required",
code: "cd-transaction-required",
script,
mode,
requiredEnv,
acceptedEnv: transactionEnvNames,
entrypoint: "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report",
summary: `${script} ${mode} is a DEV CD side-effect step and must run inside scripts/dev-cd-apply.mjs.`,
devOnly: true,
prodTouched: false,
mutationAttempted: false
};
}
export function requireDevCdTransactionForSideEffect({ env = process.env, script, mode }) {
if (hasDevCdTransactionEnv(env)) return null;
return devCdTransactionGuardFailure({ script, mode });
}
+13 -1
View File
@@ -17,6 +17,7 @@ import {
} from "../../internal/cloud/code-agent-contract.mjs";
import { activeReportLifecycle } from "../../internal/dev-report-lifecycle.mjs";
import { DEV_ENDPOINT, ENVIRONMENT_DEV, SERVICE_IDS } from "../../internal/protocol/index.mjs";
import { requireDevCdTransactionForSideEffect } from "./dev-cd-transaction-guard.mjs";
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../..");
const reportPath = "reports/dev-gate/dev-deploy-report.json";
@@ -72,7 +73,7 @@ const requiredValidationCommands = [
"node --check scripts/src/dev-deploy-apply.mjs",
"node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked"
];
const applyCommand = "node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report";
const applyCommand = "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report";
const dryRunPlanCommand = "node scripts/dev-deploy-apply.mjs --dry-run --write-report";
const blockedDryRunCommand = "node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report";
const forbiddenActions = [
@@ -1771,6 +1772,17 @@ export async function runDevDeployApply(argv, io = {}) {
const args = parseArgs(argv);
const blockers = [];
validateArgs(args, blockers);
if (args.apply) {
const transactionGuard = requireDevCdTransactionForSideEffect({
env: io.env ?? process.env,
script: "scripts/dev-deploy-apply.mjs",
mode: "--apply"
});
if (transactionGuard) {
stdout.write(`${JSON.stringify(transactionGuard, null, 2)}\n`);
return 2;
}
}
const gitHeadCommitIdPromise = gitHeadCommit();
const [deploy, catalog, devKustomization, namespaceDoc, workloads, services, healthContract] = await Promise.all([
+32
View File
@@ -14,6 +14,11 @@ import {
resolveApplySourceCommit,
resolveDevKubeconfigSelection
} from "./dev-deploy-apply.mjs";
import {
devCdTransactionGuardFailure,
hasDevCdTransactionEnv,
requireDevCdTransactionForSideEffect
} from "./dev-cd-transaction-guard.mjs";
const codeAgentBaseUrl = "http://172.26.26.227:17680/v1/responses";
@@ -109,6 +114,33 @@ test("allowlisted suspended template Job image change plans replacement", () =>
assert.equal(decision.newImage, "127.0.0.1:5000/hwlab/hwlab-agent-worker:new5678");
});
test("DEV CD transaction guard rejects direct legacy side-effect calls", () => {
assert.equal(hasDevCdTransactionEnv({}), false);
assert.equal(hasDevCdTransactionEnv({ HWLAB_CD_TRANSACTION_ID: "tx-123" }), true);
const failure = requireDevCdTransactionForSideEffect({
env: {},
script: "scripts/dev-deploy-apply.mjs",
mode: "--apply"
});
assert.deepEqual(failure, devCdTransactionGuardFailure({
script: "scripts/dev-deploy-apply.mjs",
mode: "--apply"
}));
assert.equal(failure.error, "cd-transaction-required");
assert.equal(failure.mutationAttempted, false);
assert.match(failure.entrypoint, /scripts\/dev-cd-apply\.mjs/u);
assert.equal(
requireDevCdTransactionForSideEffect({
env: { HWLAB_CD_TRANSACTION_ID: "tx-123" },
script: "scripts/dev-deploy-apply.mjs",
mode: "--apply"
}),
null
);
});
test("matching allowlisted suspended template Job image does not replace", () => {
const image = "127.0.0.1:5000/hwlab/hwlab-cli:73b379f";
const decision = decideDevTemplateJobReplacement({
+116 -2
View File
@@ -20,6 +20,7 @@ const reportsDir = path.join(repoRoot, "reports/dev-gate");
const contractIssue = "pikasTech/HWLAB#31";
const issueFamily = Object.freeze({
DEV_CD_APPLY: "pikasTech/HWLAB#274",
DEV_DEPLOY_APPLY: "pikasTech/HWLAB#33",
DEV_GATE_PREFLIGHT: "pikasTech/HWLAB#34",
DEV_ARTIFACT_PUBLISH: "pikasTech/HWLAB#35",
@@ -143,6 +144,32 @@ const requiredDevCloudWorkbenchDocs = [
"docs/reference/code-agent-chat-readiness.md"
];
const reportFamilyTemplates = new Map([
[
"dev-cd-apply",
{
issue: issueFamily.DEV_CD_APPLY,
requiredDocs: [
"docs/reference/deployment-publish.md",
"docs/reference/dev-runtime-boundary.md",
"docs/dev-artifact-publish.md",
"docs/dev-deploy-apply.md",
"docs/artifact-catalog.md"
],
requiredValidationCommands: [
"node --check scripts/dev-cd-apply.mjs",
"node --check scripts/src/dev-cd-apply.mjs",
"node --test scripts/src/dev-cd-apply.test.mjs",
"node --check scripts/dev-artifact-publish.mjs",
"node --check scripts/src/dev-deploy-apply.mjs",
"node --test scripts/src/dev-deploy-apply.test.mjs",
"node scripts/deploy-desired-state-plan.mjs --check",
"node scripts/validate-artifact-catalog.mjs",
"git diff --check"
],
requiredSmokeCommand: "npm run check",
requiredDryRunCommand: "node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report"
}
],
[
"dev-gate-report-contract",
{
@@ -714,6 +741,9 @@ async function validateReport(relativePath) {
if (report.taskId === "dev-deploy-apply") {
assertDevDeployApplyReport(report, label);
}
if (report.taskId === "dev-cd-apply") {
assertDevCdApplyReport(report, label);
}
}
async function assertDocumentSet(documentsValue, label, requiredDocs) {
@@ -937,8 +967,8 @@ function assertDevDeployApplyReport(report, label) {
if (plan.conclusion.status === "ready") {
assert.equal(plan.manualCommands.status, "ready", `${label}.devDeployApply.manualCommands.status`);
assert.ok(
plan.manualCommands.afterHumanApproval.includes("node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"),
`${label}.devDeployApply.manualCommands.afterHumanApproval missing apply command`
plan.manualCommands.afterHumanApproval.includes("node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"),
`${label}.devDeployApply.manualCommands.afterHumanApproval missing transaction apply command`
);
}
@@ -963,6 +993,90 @@ function assertDevDeployApplyReport(report, label) {
}
}
function assertDevCdApplyReport(report, label) {
assert.equal(report.issue, issueFamily.DEV_CD_APPLY, `${label}.issue`);
assertStringArray(report.supports, `${label}.supports`, { minLength: 4 });
for (const supportedIssue of [
"pikasTech/HWLAB#274",
"pikasTech/HWLAB#116",
"pikasTech/HWLAB#235"
]) {
assert.ok(report.supports.includes(supportedIssue), `${label}.supports missing ${supportedIssue}`);
}
assertObject(report.transaction, `${label}.transaction`);
for (const field of ["transactionId", "ownerTaskId", "targetNamespace", "lockName", "lockBackend", "startedAt", "finishedAt", "ttlSeconds", "phases"]) {
assert.ok(Object.hasOwn(report.transaction, field), `${label}.transaction missing ${field}`);
}
assert.equal(report.transaction.targetNamespace, "hwlab-dev", `${label}.transaction.targetNamespace`);
assert.equal(report.transaction.lockName, "hwlab-dev-cd-lock", `${label}.transaction.lockName`);
assert.equal(report.transaction.lockBackend, "Lease", `${label}.transaction.lockBackend`);
assertString(report.transaction.transactionId, `${label}.transaction.transactionId`);
assertString(report.transaction.ownerTaskId, `${label}.transaction.ownerTaskId`);
assert.ok(Number.isInteger(report.transaction.ttlSeconds), `${label}.transaction.ttlSeconds`);
assertArray(report.transaction.phases, `${label}.transaction.phases`);
assert.ok(
report.transaction.phases.some((phase) => phase.phase === "publishing"),
`${label}.transaction.phases must include publishing`
);
assertObject(report.devCdApply, `${label}.devCdApply`);
for (const field of ["mode", "target", "deployJson", "lock", "steps", "liveBefore", "liveVerify", "reportPaths", "safety"]) {
assert.ok(Object.hasOwn(report.devCdApply, field), `${label}.devCdApply missing ${field}`);
}
assert.equal(report.devCdApply.mode, "apply", `${label}.devCdApply.mode`);
assertObject(report.devCdApply.target, `${label}.devCdApply.target`);
assert.equal(report.devCdApply.target.namespace, "hwlab-dev", `${label}.devCdApply.target.namespace`);
assert.equal(report.devCdApply.target.apiEndpoint, "http://74.48.78.17:16667", `${label}.devCdApply.target.apiEndpoint`);
assert.equal(report.devCdApply.target.browserEndpoint, "http://74.48.78.17:16666/", `${label}.devCdApply.target.browserEndpoint`);
assertString(report.devCdApply.target.shortCommitId, `${label}.devCdApply.target.shortCommitId`);
assertObject(report.devCdApply.deployJson, `${label}.devCdApply.deployJson`);
for (const side of ["before", "after"]) {
assertObject(report.devCdApply.deployJson[side], `${label}.devCdApply.deployJson.${side}`);
assert.equal(report.devCdApply.deployJson[side].path, "deploy/deploy.json", `${label}.devCdApply.deployJson.${side}.path`);
assertString(report.devCdApply.deployJson[side].hash, `${label}.devCdApply.deployJson.${side}.hash`);
assertString(report.devCdApply.deployJson[side].commitId, `${label}.devCdApply.deployJson.${side}.commitId`);
}
assertObject(report.devCdApply.lock, `${label}.devCdApply.lock`);
assert.equal(typeof report.devCdApply.lock.acquired, "boolean", `${label}.devCdApply.lock.acquired`);
assertArray(report.devCdApply.steps, `${label}.devCdApply.steps`);
for (const [index, step] of report.devCdApply.steps.entries()) {
const stepLabel = `${label}.devCdApply.steps[${index}]`;
assertObject(step, stepLabel);
for (const field of ["id", "phase", "status", "command", "code", "startedAt", "finishedAt"]) {
assert.ok(Object.hasOwn(step, field), `${stepLabel} missing ${field}`);
}
assert.ok(["publishing", "applying"].includes(step.phase), `${stepLabel}.phase`);
assert.ok(["pass", "blocked"].includes(step.status), `${stepLabel}.status`);
assertString(step.command, `${stepLabel}.command`);
}
assertObject(report.devCdApply.liveVerify, `${label}.devCdApply.liveVerify`);
assert.ok(["pass", "blocked", "not_run"].includes(report.devCdApply.liveVerify.status), `${label}.devCdApply.liveVerify.status`);
assertArray(report.devCdApply.liveVerify.endpoints, `${label}.devCdApply.liveVerify.endpoints`);
assertObject(report.devCdApply.liveVerify.summary, `${label}.devCdApply.liveVerify.summary`);
if (report.devCdApply.liveVerify.status !== "not_run") {
const urls = report.devCdApply.liveVerify.endpoints.map((endpoint) => endpoint.url);
assert.ok(urls.includes("http://74.48.78.17:16666/health/live"), `${label}.devCdApply.liveVerify missing 16666 health`);
assert.ok(urls.includes("http://74.48.78.17:16667/health/live"), `${label}.devCdApply.liveVerify missing 16667 health`);
}
assertObject(report.devCdApply.reportPaths, `${label}.devCdApply.reportPaths`);
assert.equal(report.devCdApply.reportPaths.artifacts, "reports/dev-gate/dev-artifacts.json", `${label}.devCdApply.reportPaths.artifacts`);
assert.equal(report.devCdApply.reportPaths.deployApply, "reports/dev-gate/dev-deploy-report.json", `${label}.devCdApply.reportPaths.deployApply`);
assertObject(report.devCdApply.safety, `${label}.devCdApply.safety`);
assert.equal(report.devCdApply.safety.prodTouched, false, `${label}.devCdApply.safety.prodTouched`);
assert.equal(report.devCdApply.safety.secretValuesRead, false, `${label}.devCdApply.safety.secretValuesRead`);
assert.equal(report.devCdApply.safety.secretValuesPrinted, false, `${label}.devCdApply.safety.secretValuesPrinted`);
assert.equal(report.devCdApply.safety.oldPublicPortsUsed, false, `${label}.devCdApply.safety.oldPublicPortsUsed`);
assert.equal(report.devCdApply.safety.singleLock, "Lease/hwlab-dev/hwlab-dev-cd-lock", `${label}.devCdApply.safety.singleLock`);
}
function assertShaOrNotPublished(value, label) {
assertString(value, label);