diff --git a/docs/reference/deployment-publish.md b/docs/reference/deployment-publish.md index 1e95f868..6b4c0920 100644 --- a/docs/reference/deployment-publish.md +++ b/docs/reference/deployment-publish.md @@ -30,55 +30,52 @@ truth. ## DEV CD Transaction -The stable DEV CD side-effect command is: +正式 DEV CD 写路径只有一个入口： ```sh node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report ``` -This command is the only formal path that may publish DEV artifacts, refresh -desired-state from the publish report, apply to `hwlab-dev`, and verify public -`16666/16667` in one run. It reads `deploy/deploy.json` at transaction start, -records the manifest hash and target commit, acquires -`Lease/hwlab-dev/hwlab-dev-cd-lock`, runs the internal steps serially, writes -`reports/dev-gate/dev-cd-apply.json`, and releases the Lease. +该命令是唯一允许一次性发布 DEV artifact、从发布报告刷新 desired +state、apply 到 `hwlab-dev`、并复验公开 `16666/16667` 的正式写路径。 +事务开始时读取 `deploy/deploy.json`，记录 manifest hash 和 target commit， +获取 `Lease/hwlab-dev/hwlab-dev-cd-lock`，串行执行内部步骤，按需写入 +`reports/dev-gate/dev-cd-apply.json`，最后释放 Lease。 -The Lease is a deployment-plane mutex, not a desired-state source. Its -annotations record at least `promotionCommit`, `deployJsonHash`, -`ownerTaskId`, `transactionId`, `phase`, `startedAt`, `updatedAt`, -`ttlSeconds`, `liveBefore`, and `targetNamespace`. `deploy/deploy.json`, -`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml` -remain the only authoritative deploy inputs. +Lease 只是发布平面的互斥锁，不是 desired-state 来源。Lease annotations +至少记录 `promotionCommit`、`deployJsonHash`、`ownerTaskId`、 +`transactionId`、`phase`、`startedAt`、`updatedAt`、`ttlSeconds`、 +`liveBefore` 和 `targetNamespace`。`deploy/deploy.json`、 +`deploy/artifact-catalog.dev.json`、`deploy/k8s/base/workloads.yaml` 仍是唯 +一权威部署输入。 -If another transaction holds the Lease, the command must stop before publish or -apply with structured `deploy-lock-held` output that includes holder, -promotion commit, phase, target namespace, and retry seconds. Expired locks are -not silently overwritten; taking over a stale lock requires both -`--break-stale-lock` and `--confirm-dev`, and the previous holder is retained -in audit annotations/report fields. +如果另一个事务持有 Lease，`--apply` 必须在 publish/apply 之前停止，并 +输出结构化 `deploy-lock-held`，包含 holder、promotion commit、phase、 +target namespace 和 retry seconds。过期锁不能静默覆盖；stale lock 只能 +在确认 holder 已失活后显式使用 `--break-stale-lock --confirm-dev`，并把 +previous holder 保留在审计 annotations/report 字段中。未过期 Lease 没有 +抢占入口，普通 status 输出也不得建议强制抢锁。 -`scripts/dev-cd-apply.mjs` also owns the read-only status surface. Running it -without arguments, with `--status`, or with `--dry-run` must return compact JSON -for the target ref, `deploy/deploy.json`, current Lease, and public -`16666/16667` live health. These modes must not acquire the Lease, publish -artifacts, apply Kubernetes workloads, or write reports. Normal `--apply` -stdout is also compact by default; the complete transaction record is preserved -through `--write-report`, or printed only when `--full-output` is explicit. +`scripts/dev-cd-apply.mjs` 同时提供只读状态面。无参数、`--status`、或 +`--dry-run` 都返回紧凑 JSON，覆盖 target ref/commit、 +`deploy/deploy.json` commit/hash、当前 Lease 状态摘要、以及公开 +`16666/16667` live health identity 摘要。这些模式 host commander 和 +runner 都可以执行，但必须保持只读：不获取或替换 Lease，不 publish +artifact，不执行 `kubectl apply/rollout`，不写 report，不修改 +`deploy.json`/catalog，也不读取或打印 Secret value。kubectl 或 kubeconfig +不可用时，状态面应返回 `lock.status=unavailable` 和脱敏原因，而不是把 +只读观测失败升级成写路径 blocker。 -When the current DEV apply must be stopped, the operator may use -`--force-interrupt-lock` together with `--apply --confirm-dev ---confirmed-non-production`. This is a DEV-only takeover path: it first tries -to send `TERM` to local `scripts/dev-cd-apply.mjs --apply` processes, then -replaces the Lease holder and records the interrupted holder and interrupt -attempt in Lease annotations and the transaction report. Do not use this flag -for ordinary contention; prefer waiting for the holder or breaking only a -confirmed stale lock. +正常 `--apply` stdout 默认也是紧凑摘要；完整事务记录继续通过 +`--write-report` 写入既有 report 路径，只有显式 `--full-output` 时才打印 +完整 JSON。DEV apply、rollout 和 live verification 仍由 host commander +统一执行；runner 默认只使用 status/dry-run 观测面，不竞争 DEV CD lock。 -The old side-effect commands are now internal transaction steps. Direct manual -use of `scripts/dev-artifact-publish.mjs --publish` or -`scripts/dev-deploy-apply.mjs --apply` without `HWLAB_CD_TRANSACTION_ID` is -rejected with `cd-transaction-required`; preflight, build, dry-run, check, and -read-only modes remain available. +旧的 side-effect 命令已经降为事务内部步骤。直接手动执行 +`scripts/dev-artifact-publish.mjs --publish` 或 +`scripts/dev-deploy-apply.mjs --apply` 且没有 `HWLAB_CD_TRANSACTION_ID` 时， +会以 `cd-transaction-required` 拒绝；preflight、build、dry-run、check 和 +read-only 模式仍可用。 ## Artifact Publish @@ -299,6 +296,7 @@ use the repo-owned scripts below instead of ad hoc shell fragments. | --- | --- | --- | | Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. | | Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. | +| Observe DEV CD status | `node scripts/dev-cd-apply.mjs`, `node scripts/dev-cd-apply.mjs --status`, or `node scripts/dev-cd-apply.mjs --dry-run` | Keep read-only for host and runners; no Lease mutation, artifact publish, apply, rollout, report write, or Secret read. | | Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. | | Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. | | Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. | diff --git a/scripts/src/dev-cd-apply.mjs b/scripts/src/dev-cd-apply.mjs index 38bcfabd..3cab1228 100644 --- a/scripts/src/dev-cd-apply.mjs +++ b/scripts/src/dev-cd-apply.mjs @@ -42,11 +42,7 @@ const lockAnnotationFields = { releasedAt: `${lockAnnotationPrefix}/releasedAt`, releaseStatus: `${lockAnnotationPrefix}/releaseStatus`, staleLockBrokenAt: `${lockAnnotationPrefix}/staleLockBrokenAt`, - staleLockPrevious: `${lockAnnotationPrefix}/staleLockPrevious`, - forceInterruptedAt: `${lockAnnotationPrefix}/forceInterruptedAt`, - forceInterruptedBy: `${lockAnnotationPrefix}/forceInterruptedBy`, - forceInterruptStatus: `${lockAnnotationPrefix}/forceInterruptStatus`, - forceInterruptedPrevious: `${lockAnnotationPrefix}/forceInterruptedPrevious` + staleLockPrevious: `${lockAnnotationPrefix}/staleLockPrevious` }; class DevCdApplyError extends Error { @@ -72,7 +68,6 @@ export function parseArgs(argv) { ttlSeconds: defaultTtlSeconds, ownerTaskId: null, breakStaleLock: false, - forceInterruptLock: false, skipLiveVerify: false, status: false, dryRun: false, @@ -98,9 +93,6 @@ export function parseArgs(argv) { } else if (arg === "--break-stale-lock") { args.breakStaleLock = true; args.flags.add(arg); - } else if (arg === "--force-interrupt-lock") { - args.forceInterruptLock = true; - args.flags.add(arg); } else if (arg === "--dry-run" || arg === "--status") { args.status = true; args.dryRun = arg === "--dry-run"; @@ -140,7 +132,15 @@ export function parseArgs(argv) { } else if (arg === "--help" || arg === "-h") { args.help = true; } else { - throw new Error(`unknown argument ${arg}`); + throw new DevCdApplyError(`unknown argument ${arg}`, { + code: "unknown-argument", + blockers: [{ + type: "safety_blocker", + scope: "unknown-argument", + status: "open", + summary: `${arg} is not a supported DEV CD option. Use --status/--dry-run for read-only observability or --apply with explicit DEV confirmations.` + }] + }); } } @@ -163,21 +163,19 @@ export function usage() { examples: [ "node scripts/dev-cd-apply.mjs --status", "node scripts/dev-cd-apply.mjs --dry-run", - "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report", - "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --force-interrupt-lock --write-report" + "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report" ], options: { "--status": "Read-only status: target ref, deploy.json, current Lease, and public live health.", "--dry-run": "Read-only alias for --status; does not acquire the Lease or run side effects.", "--apply": "Run the DEV CD transaction side effects.", - "--confirm-dev": "Required for DEV mutation and stale/force lock takeover.", + "--confirm-dev": "Required for DEV mutation and explicit stale-lock recovery.", "--confirmed-non-production": "Required for DEV mutation.", "--target-ref REF": "git ref to publish/apply; default: origin/main", "--kubeconfig PATH": "DEV kubeconfig path; also honors HWLAB_DEV_KUBECONFIG/KUBECONFIG", "--ttl-seconds SECONDS": "lock TTL; default: 3600", "--owner-task-id ID": "lock holder; default: Code Queue/env/user derived", "--break-stale-lock": "May take over an expired lock only with --confirm-dev.", - "--force-interrupt-lock": "DEV-only explicit takeover: try to interrupt the current local dev-cd apply process, then acquire the Lease.", "--report PATH": `default: ${defaultReportPath}`, "--write-report": "Write the full report to --report.", "--full-output": "Print the full report to stdout instead of the concise summary.", @@ -241,15 +239,17 @@ function defaultOwnerTaskId(env = process.env) { function defaultRunCommand(command, args, options = {}) { return new Promise((resolve) => { + const abortController = options.timeoutMs ? new AbortController() : null; const child = spawn(command, args, { cwd: options.cwd ?? defaultRepoRoot, env: options.env ?? process.env, - stdio: ["pipe", "pipe", "pipe"] + stdio: ["pipe", "pipe", "pipe"], + signal: abortController?.signal }); let stdout = ""; let stderr = ""; const timeout = options.timeoutMs - ? setTimeout(() => child.kill("SIGTERM"), options.timeoutMs) + ? setTimeout(() => abortController.abort(), options.timeoutMs) : null; child.stdin.on("error", () => {}); @@ -262,7 +262,12 @@ function defaultRunCommand(command, args, options = {}) { }); child.on("error", (error) => { if (timeout) clearTimeout(timeout); - resolve({ code: 127, signal: null, stdout, stderr: error.message }); + resolve({ + code: error.name === "AbortError" ? 124 : 127, + signal: null, + stdout, + stderr: error.message + }); }); child.on("close", (code, signal) => { if (timeout) clearTimeout(timeout); @@ -346,22 +351,6 @@ function validateArgs(args) { summary: "Breaking a stale DEV CD lock requires --break-stale-lock and --confirm-dev." }); } - if (args.forceInterruptLock && !args.confirmDev) { - blockers.push({ - type: "safety_blocker", - scope: "force-interrupt-confirmation", - status: "open", - summary: "Force-interrupting a DEV CD lock requires --force-interrupt-lock and --confirm-dev." - }); - } - if (args.forceInterruptLock && !args.apply) { - blockers.push({ - type: "safety_blocker", - scope: "force-interrupt-mode", - status: "open", - summary: "--force-interrupt-lock is only valid with --apply." - }); - } if (!Number.isInteger(args.ttlSeconds) || args.ttlSeconds < 60 || args.ttlSeconds > 24 * 60 * 60) { blockers.push({ type: "safety_blocker", @@ -446,11 +435,7 @@ export function parseDeployLock(lease) { releasedAt: readAnnotation(annotations, "releasedAt"), releaseStatus: readAnnotation(annotations, "releaseStatus"), staleLockBrokenAt: readAnnotation(annotations, "staleLockBrokenAt"), - staleLockPrevious: readJsonAnnotation(annotations, "staleLockPrevious"), - forceInterruptedAt: readAnnotation(annotations, "forceInterruptedAt"), - forceInterruptedBy: readAnnotation(annotations, "forceInterruptedBy"), - forceInterruptStatus: readAnnotation(annotations, "forceInterruptStatus"), - forceInterruptedPrevious: readJsonAnnotation(annotations, "forceInterruptedPrevious") + staleLockPrevious: readJsonAnnotation(annotations, "staleLockPrevious") }; } @@ -512,7 +497,7 @@ function isConflict(result) { return /conflict|object has been modified|operation cannot be fulfilled/i.test(`${result.stderr}\n${result.stdout}`); } -function compactInterruptedLock(lock) { +function compactLockSummary(lock) { if (!lock) return null; return { lockName: lock.lockName ?? defaultLockName, @@ -526,39 +511,14 @@ function compactInterruptedLock(lock) { updatedAt: lock.updatedAt ?? null, ttlSeconds: lock.ttlSeconds ?? null, targetNamespace: lock.targetNamespace ?? defaultNamespace, - targetRef: lock.targetRef ?? null + targetRef: lock.targetRef ?? null, + releasedAt: lock.releasedAt ?? null, + releaseStatus: lock.releaseStatus ?? null, + staleLockBrokenAt: lock.staleLockBrokenAt ?? null }; } -function compactForceInterruptRecord(record) { - if (!record) return null; - return { - status: record.status, - warning: record.warning ?? null, - processCandidates: (record.processCandidates ?? []).map((candidate) => ({ - pid: candidate.pid, - command: candidate.command - })), - attempts: (record.attempts ?? []).map((attempt) => ({ - pid: attempt.pid, - signal: attempt.signal, - status: attempt.status - })) - }; -} - -function attachForceInterruptAnnotations(lock, forceInterrupt, timestamp, ownerTaskId) { - if (!forceInterrupt) return; - lock.forceInterruptedAt = timestamp; - lock.forceInterruptedBy = ownerTaskId; - lock.forceInterruptStatus = forceInterrupt.status; - lock.forceInterruptedPrevious = { - previousLock: compactInterruptedLock(forceInterrupt.previousLock), - interrupt: compactForceInterruptRecord(forceInterrupt) - }; -} - -function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTaskId, startedAt, phase, liveBefore, staleLockPrevious = null, forceInterrupt = null }) { +function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTaskId, startedAt, phase, liveBefore, staleLockPrevious = null }) { const lock = { promotionCommit: target.shortCommitId, deployJsonHash: deployBefore.hash, @@ -577,7 +537,6 @@ function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTa lock.staleLockBrokenAt = startedAt; lock.staleLockPrevious = staleLockPrevious; } - attachForceInterruptAnnotations(lock, forceInterrupt, startedAt, ownerTaskId); return { apiVersion: "coordination.k8s.io/v1", kind: "Lease", @@ -600,7 +559,7 @@ function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTa }; } -function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, phase, startedAt, updatedAt, liveBefore, staleLockPrevious = null, forceInterrupt = null }) { +function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, phase, startedAt, updatedAt, liveBefore, staleLockPrevious = null }) { const lock = { promotionCommit: target.shortCommitId, deployJsonHash: deployBefore.hash, @@ -619,7 +578,6 @@ function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, pha lock.staleLockBrokenAt = updatedAt; lock.staleLockPrevious = staleLockPrevious; } - attachForceInterruptAnnotations(lock, forceInterrupt, updatedAt, ownerTaskId); return { metadata: { labels: { @@ -638,66 +596,6 @@ function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, pha }; } -export function findDevCdApplyProcessCandidates(psOutput, currentPid = process.pid) { - return String(psOutput ?? "") - .split(/\r?\n/u) - .map((line) => line.trim()) - .filter(Boolean) - .map((line) => { - const match = line.match(/^(\d+)\s+(.+)$/u); - if (!match) return null; - return { - pid: Number.parseInt(match[1], 10), - command: oneLine(redactSensitiveText(match[2])).slice(0, 500) - }; - }) - .filter((entry) => { - if (!entry || !Number.isInteger(entry.pid) || entry.pid === currentPid) return false; - return entry.command.includes("scripts/dev-cd-apply.mjs") && entry.command.includes("--apply"); - }); -} - -async function forceInterruptCurrentApplyProcess(ctx, existingLock) { - const ps = await ctx.runCommand("ps", ["-eo", "pid=,args="], { - cwd: ctx.repoRoot, - timeoutMs: 5000 - }); - if (ps.code !== 0) { - return { - status: "ps-failed", - previousLock: compactInterruptedLock(existingLock), - processCandidates: [], - attempts: [], - warning: oneLine(redactSensitiveText(ps.stderr || ps.stdout || "ps failed")) - }; - } - - const processCandidates = findDevCdApplyProcessCandidates(ps.stdout, process.pid); - const attempts = []; - for (const candidate of processCandidates) { - const result = await ctx.runCommand("kill", ["-TERM", String(candidate.pid)], { - cwd: ctx.repoRoot, - timeoutMs: 5000 - }); - attempts.push({ - pid: candidate.pid, - signal: "TERM", - status: result.code === 0 ? "sent" : "failed", - stderrTail: redactSensitiveText(result.stderr || result.stdout || "").trim().slice(-500) - }); - } - - return { - status: processCandidates.length > 0 ? "attempted" : "no-local-process-found", - previousLock: compactInterruptedLock(existingLock), - processCandidates, - attempts, - warning: processCandidates.length > 0 - ? null - : "No local scripts/dev-cd-apply.mjs --apply process was found; the Lease will still be taken over so the previous holder cannot continue lock-owned updates." - }; -} - async function kubectl(ctx, kubectlContext, args, options = {}) { const result = await ctx.runCommand(kubectlContext.executor, args, { cwd: ctx.repoRoot, @@ -728,7 +626,7 @@ async function replaceLease(ctx, kubectlContext, args, lease, timeoutMs = 30000) function mergeLease(lease, patch, { resetLockAnnotations = false } = {}) { const baseAnnotations = { ...(lease.metadata?.annotations ?? {}) }; if (resetLockAnnotations) { - for (const key of Object.values(lockAnnotationFields)) { + for (const key of Object.keys(baseAnnotations).filter((name) => name.startsWith(`${lockAnnotationPrefix}/`))) { delete baseAnnotations[key]; } } @@ -760,14 +658,14 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo const lease = JSON.parse(get.stdout); const existing = parseDeployLock(lease); const classification = classifyDeployLock(existing, now); - if (classification.held && !args.forceInterruptLock) { + if (classification.held) { const failure = deployLockHeldFailure(existing, now); throw new DevCdApplyError(failure.summary, { code: "deploy-lock-held", lockFailure: failure }); } - if (classification.stale && !args.breakStaleLock && !args.forceInterruptLock) { + if (classification.stale && !args.breakStaleLock) { const failure = deployLockHeldFailure(existing, now); throw new DevCdApplyError(failure.summary, { code: "deploy-lock-held", @@ -775,10 +673,6 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo }); } - const forceInterrupt = args.forceInterruptLock - ? await forceInterruptCurrentApplyProcess(ctx, existing) - : null; - const patch = lockPatch({ args, target, @@ -789,8 +683,7 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo startedAt, updatedAt: startedAt, liveBefore, - staleLockPrevious: classification.stale ? existing : null, - forceInterrupt + staleLockPrevious: classification.stale ? existing : null }); const replaced = await replaceLease(ctx, kubectlContext, args, mergeLease(lease, patch, { resetLockAnnotations: true })); if (isConflict(replaced)) { @@ -806,7 +699,6 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo acquired: true, lock: parseDeployLock(JSON.parse(replaced.stdout)), staleBreak: classification.stale ? existing : null, - forceInterrupt, command: replaced.command }; } @@ -837,7 +729,6 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo acquired: true, lock: parseDeployLock(JSON.parse(created.stdout)), staleBreak: null, - forceInterrupt: null, command: created.command }; } @@ -1410,6 +1301,10 @@ function summarizeDeployJson(deploy) { }; } +function deployJsonMatchesTarget(deploy, target) { + return deploy.commitId === target.shortCommitId || deploy.commitId === target.commitId; +} + function compactLiveEndpoint(endpoint) { return { id: endpoint.id, @@ -1481,7 +1376,8 @@ function summarizeLockStatus(lockRead, now) { current: null, retryAfterSeconds: null, expiresAt: null, - error: oneLine(lockRead.error) + reason: "kubectl or DEV kubeconfig unavailable; command output is redacted", + redactedReason: oneLine(lockRead.error).slice(0, 500) }; } const classification = classifyDeployLock(lockRead.lock, now); @@ -1494,12 +1390,12 @@ function summarizeLockStatus(lockRead, now) { : "unknown"; return { status, - current: compactInterruptedLock(lockRead.lock), + lockName: lockRead.lock?.lockName ?? lockRead.lockName ?? defaultLockName, + current: compactLockSummary(lockRead.lock), retryAfterSeconds: classification.retryAfterSeconds, expiresAt: classification.expiresAt, - takeover: { - stale: classification.stale ? "--apply --confirm-dev --confirmed-non-production --break-stale-lock" : null, - forceInterrupt: status === "held" ? "--apply --confirm-dev --confirmed-non-production --force-interrupt-lock" : null + allowedLockAction: { + breakStale: classification.stale ? "--apply --confirm-dev --confirmed-non-production --break-stale-lock" : null } }; } @@ -1513,7 +1409,7 @@ function buildStatusNextActions({ target, deployBefore, lock, liveVerify }) { actions.push(`deploy/deploy.json commitId is ${deployBefore.commitId}; target ref is ${target.shortCommitId}.`); } if (lock.status === "held") { - actions.push(`Wait ${lock.retryAfterSeconds}s or use explicit --force-interrupt-lock if the current apply must be stopped.`); + actions.push(`Wait for the current DEV CD holder to release the Lease or for the ${lock.retryAfterSeconds}s TTL to expire before retrying apply.`); } else if (lock.status === "stale") { actions.push("Use --break-stale-lock only after confirming the holder is inactive."); } else if (lock.status === "unavailable") { @@ -1538,7 +1434,13 @@ async function runDevCdStatus(args, ctx, stdout) { const liveVerify = args.skipLiveVerify ? { status: "not_run", reason: "--skip-live-verify", endpoints: [], summary: { checked: 0 } } : await verifyDevLive(ctx, { expectedCommit: null }); - const status = lock.status === "unavailable" || liveVerify.status !== "pass" ? "degraded" : "pass"; + const deployMatchesTarget = deployJsonMatchesTarget(deployBefore, target); + const status = ( + lock.status === "unavailable" || + liveVerify.status !== "pass" || + !target.headMatchesTarget || + !deployMatchesTarget + ) ? "degraded" : "pass"; const report = { ok: true, status, @@ -1555,7 +1457,10 @@ async function runDevCdStatus(args, ctx, stdout) { headMatchesTarget: target.headMatchesTarget, namespace: args.targetNamespace }, - deployJson: summarizeDeployJson(deployBefore), + deployJson: { + ...summarizeDeployJson(deployBefore), + matchesTarget: deployMatchesTarget + }, kubectl: { kubeconfigSource: kubectlContext.kubeconfigSource, kubeconfig: kubectlContext.kubeconfig ? "" : "", @@ -1592,9 +1497,8 @@ function summarizeDevCdApplyReport(report) { target: apply.target ?? null, lock: { acquired: Boolean(apply.lock?.acquired), - current: compactInterruptedLock(apply.lock?.current), - staleBreak: compactInterruptedLock(apply.lock?.staleBreak), - forceInterrupt: compactForceInterruptRecord(apply.lock?.forceInterrupt) + current: compactLockSummary(apply.lock?.current), + staleBreak: compactLockSummary(apply.lock?.staleBreak) }, steps: (apply.steps ?? []).map((step) => ({ id: step.id, @@ -1726,8 +1630,7 @@ function buildReport({ lock: { acquired: Boolean(lockState?.acquired), current: lockState?.lock ?? null, - staleBreak: lockState?.staleBreak ?? null, - forceInterrupt: lockState?.forceInterrupt ?? null + staleBreak: lockState?.staleBreak ?? null }, steps, liveBefore, diff --git a/scripts/src/dev-cd-apply.test.mjs b/scripts/src/dev-cd-apply.test.mjs index e47216f4..c346c78b 100644 --- a/scripts/src/dev-cd-apply.test.mjs +++ b/scripts/src/dev-cd-apply.test.mjs @@ -10,7 +10,6 @@ import { buildLockAnnotations, classifyDeployLock, deployLockHeldFailure, - findDevCdApplyProcessCandidates, parseDeployLock, runDevCdApply, verifyDevLive @@ -99,7 +98,7 @@ function makeHttpGetJson() { }; } -function makeRunCommand({ heldLock = null, commandLog = [], psOutput = "", killLog = [] } = {}) { +function makeRunCommand({ heldLock = null, commandLog = [] } = {}) { let lease = heldLock ? leaseFromLock(heldLock) : null; const assertLeaseMicroTime = (value) => { assert.match(value, /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{6}Z$/u); @@ -115,13 +114,6 @@ function makeRunCommand({ heldLock = null, commandLog = [], psOutput = "", killL if (command === "which" && args[0] === "kubectl") { return { code: 0, stdout: "/usr/local/bin/kubectl\n", stderr: "" }; } - if (command === "ps") { - return { code: 0, stdout: psOutput, stderr: "" }; - } - if (command === "kill") { - killLog.push({ signal: args[0], pid: args[1] }); - return { code: 0, stdout: "", stderr: "" }; - } if (command.includes("kubectl") && args.includes("get") && args.includes("lease")) { if (!lease) return { code: 1, stdout: "", stderr: "Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found" }; return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; @@ -191,6 +183,18 @@ function makeRunCommand({ heldLock = null, commandLog = [], psOutput = "", killL }; } +function assertNoReadOnlySideEffects(commandLog) { + const mutatingKubectlVerbs = new Set(["apply", "create", "replace", "patch", "delete", "rollout", "wait", "logs"]); + assert.equal(commandLog.some((entry) => entry.command === "ps"), false); + assert.equal(commandLog.some((entry) => entry.command === "kill"), false); + assert.equal(commandLog.some((entry) => + entry.command.includes("kubectl") && entry.args.some((arg) => mutatingKubectlVerbs.has(arg)) + ), false); + assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); + assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); + assert.equal(commandLog.some((entry) => entry.args.includes("scripts/refresh-artifact-catalog.mjs")), false); +} + test("lock classifier reports held and stale states with retryAfterSeconds", () => { const lock = { ownerTaskId: "task-a", @@ -222,7 +226,8 @@ test("dev-cd status and dry-run are read-only compact JSON", async () => { const code = await runDevCdApply([ "--dry-run", "--kubeconfig", - "/tmp/kubeconfig" + "/tmp/kubeconfig", + "--write-report" ], { repoRoot, env: {}, @@ -239,22 +244,179 @@ test("dev-cd status and dry-run are read-only compact JSON", async () => { assert.equal(status.prodTouched, false); assert.equal(status.lock.status, "absent"); assert.equal(status.deployJson.manifest, undefined); + assert.equal(status.deployJson.matchesTarget, true); assert.equal(status.live.summary.checked, 2); - assert.equal(commandLog.some((entry) => entry.args.includes("create")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("replace")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); - assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); + assertNoReadOnlySideEffects(commandLog); + await assert.rejects( + readFile(path.join(repoRoot, "reports/dev-gate/dev-cd-apply.json"), "utf8"), + { code: "ENOENT" } + ); }); -test("process candidate discovery only returns dev-cd apply processes", () => { - const psOutput = [ - "101 node scripts/dev-cd-apply.mjs --apply --confirm-dev", - "202 node scripts/dev-cd-apply.mjs --status", - "303 node scripts/dev-deploy-apply.mjs --apply", - "404 node scripts/dev-cd-apply.mjs --apply --confirm-dev" - ].join("\n"); - const candidates = findDevCdApplyProcessCandidates(psOutput, 404); - assert.deepEqual(candidates.map((candidate) => candidate.pid), [101]); +test("dev-cd default status is read-only compact JSON", async () => { + const repoRoot = await makeRepo(); + const commandLog = []; + let output = ""; + const code = await runDevCdApply([], { + repoRoot, + env: {}, + runCommand: makeRunCommand({ commandLog }), + httpGetJson: makeHttpGetJson(), + now: () => new Date(iso(100000)), + stdout: { write: (chunk) => { output += chunk; } } + }); + + const status = JSON.parse(output); + assert.equal(code, 0); + assert.equal(status.mode, "status"); + assert.equal(status.mutationAttempted, false); + assert.equal(status.prodTouched, false); + assert.equal(status.target.shortCommitId, "abc1234"); + assert.equal(status.deployJson.hash.startsWith("sha256:"), true); + assert.equal(status.deployJson.matchesTarget, true); + assert.equal(status.lock.status, "absent"); + assertNoReadOnlySideEffects(commandLog); +}); + +test("dev-cd status summarizes held, released, and stale locks without unsafe takeover hints", async () => { + const cases = [ + { + name: "held", + phase: "applying", + now: iso(100000), + expectedStatus: "held", + expectedRetryAfterSeconds: 200 + }, + { + name: "released", + phase: "released", + releasedAt: iso(50000), + releaseStatus: "pass", + now: iso(100000), + expectedStatus: "released", + expectedRetryAfterSeconds: 0 + }, + { + name: "stale", + phase: "verifying", + now: iso(301000), + expectedStatus: "stale", + expectedRetryAfterSeconds: 0 + } + ]; + + for (const item of cases) { + const repoRoot = await makeRepo(); + const commandLog = []; + let output = ""; + const code = await runDevCdApply(["--status", "--kubeconfig", "/tmp/kubeconfig"], { + repoRoot, + env: {}, + runCommand: makeRunCommand({ + commandLog, + heldLock: { + promotionCommit: "abc1234", + deployJsonHash: "sha256:a", + ownerTaskId: `task-${item.name}`, + transactionId: `tx-${item.name}`, + phase: item.phase, + startedAt: iso(0), + updatedAt: iso(0), + ttlSeconds: 300, + liveBefore: { status: "pass" }, + targetNamespace: "hwlab-dev", + targetRef: "origin/main", + lockBackend: "Lease", + ...(item.releasedAt ? { releasedAt: item.releasedAt } : {}), + ...(item.releaseStatus ? { releaseStatus: item.releaseStatus } : {}) + } + }), + httpGetJson: makeHttpGetJson(), + now: () => new Date(item.now), + stdout: { write: (chunk) => { output += chunk; } } + }); + + const status = JSON.parse(output); + assert.equal(code, 0); + assert.equal(status.lock.status, item.expectedStatus); + assert.equal(status.lock.lockName, "hwlab-dev-cd-lock"); + assert.equal(status.lock.current.ownerTaskId, `task-${item.name}`); + assert.equal(status.lock.retryAfterSeconds, item.expectedRetryAfterSeconds); + assert.equal(JSON.stringify(status).includes("force-interrupt"), false); + assert.equal(JSON.stringify(status.nextActions).includes("force"), false); + if (item.expectedStatus === "stale") { + assert.match(status.lock.allowedLockAction.breakStale, /--break-stale-lock/u); + } + assertNoReadOnlySideEffects(commandLog); + } +}); + +test("dev-cd status reports unavailable kubectl with redacted reason", async () => { + const repoRoot = await makeRepo(); + const commandLog = []; + let output = ""; + const runCommand = async (command, args, options = {}) => { + commandLog.push({ command, args, env: options.env ?? {}, input: options.input ?? "" }); + if (command === "git" && args[0] === "rev-parse" && args.includes("origin/main^{commit}")) { + return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" }; + } + if (command === "git" && args[0] === "rev-parse" && args.includes("HEAD^{commit}")) { + return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" }; + } + if (command === "which" && args[0] === "kubectl") { + return { code: 1, stdout: "", stderr: "kubectl missing" }; + } + if (command === "kubectl") { + return { + code: 1, + stdout: "", + stderr: "connection failed token=SECRET_TOKEN password=SECRET_PASSWORD postgresql://user:SECRET_DB@db" + }; + } + return { code: 0, stdout: "", stderr: "" }; + }; + + const code = await runDevCdApply(["--status"], { + repoRoot, + env: {}, + runCommand, + httpGetJson: makeHttpGetJson(), + now: () => new Date(iso(100000)), + stdout: { write: (chunk) => { output += chunk; } } + }); + + const status = JSON.parse(output); + const text = JSON.stringify(status); + assert.equal(code, 0); + assert.equal(status.status, "degraded"); + assert.equal(status.lock.status, "unavailable"); + assert.equal(status.lock.reason, "kubectl or DEV kubeconfig unavailable; command output is redacted"); + assert.equal(text.includes("SECRET_TOKEN"), false); + assert.equal(text.includes("SECRET_PASSWORD"), false); + assert.equal(text.includes("SECRET_DB"), false); + assert.match(status.lock.redactedReason, //u); + assertNoReadOnlySideEffects(commandLog); +}); + +test("force-interrupt option is rejected before process discovery or lease mutation", async () => { + const repoRoot = await makeRepo(); + const commandLog = []; + let output = ""; + const code = await runDevCdApply(["--force-interrupt-lock"], { + repoRoot, + env: {}, + runCommand: makeRunCommand({ commandLog }), + httpGetJson: makeHttpGetJson(), + now: () => new Date(iso(100000)), + stdout: { write: (chunk) => { output += chunk; } } + }); + + const failure = JSON.parse(output); + assert.equal(code, 2); + assert.equal(failure.error, "unknown-argument"); + assert.equal(failure.mutationAttempted, false); + assert.equal(failure.prodTouched, false); + assert.deepEqual(commandLog, []); }); test("deploy-lock-held failure is structured before publish/apply", () => { @@ -489,55 +651,6 @@ test("apply stdout is concise by default while write-report keeps the full repor assert.equal(writtenReport.devCdApply.deployJson.before.manifest.environment, "dev"); }); -test("force-interrupt-lock terminates local apply candidate and takes a held lock", async () => { - const repoRoot = await makeRepo(); - const killLog = []; - let output = ""; - const code = await runDevCdApply([ - "--apply", - "--confirm-dev", - "--confirmed-non-production", - "--force-interrupt-lock", - "--owner-task-id", - "task-force", - "--kubeconfig", - "/tmp/kubeconfig", - "--full-output" - ], { - repoRoot, - env: {}, - runCommand: makeRunCommand({ - killLog, - psOutput: "555 node scripts/dev-cd-apply.mjs --apply --confirm-dev\n666 node scripts/dev-cd-apply.mjs --status\n", - heldLock: { - promotionCommit: "abc1234", - deployJsonHash: "sha256:a", - ownerTaskId: "task-a", - transactionId: "tx-a", - phase: "applying", - startedAt: iso(0), - updatedAt: iso(0), - ttlSeconds: 300, - liveBefore: { status: "pass" }, - targetNamespace: "hwlab-dev", - targetRef: "origin/main", - lockBackend: "Lease" - } - }), - httpGetJson: makeHttpGetJson(), - now: () => new Date(iso(100000)), - stdout: { write: (chunk) => { output += chunk; } } - }); - - const report = JSON.parse(output); - assert.equal(code, 0); - assert.deepEqual(killLog, [{ signal: "-TERM", pid: "555" }]); - assert.equal(report.devCdApply.lock.forceInterrupt.status, "attempted"); - assert.equal(report.devCdApply.lock.forceInterrupt.processCandidates[0].pid, 555); - assert.equal(report.devCdApply.lock.current.forceInterruptStatus, "attempted"); - assert.equal(report.devCdApply.lock.current.forceInterruptedBy, "task-force"); -}); - test("live verify summarizes 16666 and 16667 health identity", async () => { const result = await verifyDevLive({ now: () => new Date(iso()),