Merge pull request #329 from pikasTech/fix/dev-cd-status-surface-274

feat: add read-only DEV CD status surface
This commit is contained in:
Lyon
2026-05-23 18:45:57 +08:00
committed by GitHub
3 changed files with 281 additions and 267 deletions
+37 -39
View File
@@ -30,55 +30,52 @@ truth.
## DEV CD Transaction ## DEV CD Transaction
The stable DEV CD side-effect command is: 正式 DEV CD 写路径只有一个入口:
```sh ```sh
node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
``` ```
This command is the only formal path that may publish DEV artifacts, refresh 该命令是唯一允许一次性发布 DEV artifact、从发布报告刷新 desired
desired-state from the publish report, apply to `hwlab-dev`, and verify public state、apply `hwlab-dev`、并复验公开 `16666/16667` 的正式写路径。
`16666/16667` in one run. It reads `deploy/deploy.json` at transaction start, 事务开始时读取 `deploy/deploy.json`,记录 manifest hash 和 target commit
records the manifest hash and target commit, acquires 获取 `Lease/hwlab-dev/hwlab-dev-cd-lock`,串行执行内部步骤,按需写入
`Lease/hwlab-dev/hwlab-dev-cd-lock`, runs the internal steps serially, writes `reports/dev-gate/dev-cd-apply.json`,最后释放 Lease。
`reports/dev-gate/dev-cd-apply.json`, and releases the Lease.
The Lease is a deployment-plane mutex, not a desired-state source. Its Lease 只是发布平面的互斥锁,不是 desired-state 来源。Lease annotations
annotations record at least `promotionCommit`, `deployJsonHash`, 至少记录 `promotionCommit``deployJsonHash``ownerTaskId`
`ownerTaskId`, `transactionId`, `phase`, `startedAt`, `updatedAt`, `transactionId``phase``startedAt``updatedAt``ttlSeconds`
`ttlSeconds`, `liveBefore`, and `targetNamespace`. `deploy/deploy.json`, `liveBefore` `targetNamespace``deploy/deploy.json`
`deploy/artifact-catalog.dev.json`, and `deploy/k8s/base/workloads.yaml` `deploy/artifact-catalog.dev.json``deploy/k8s/base/workloads.yaml` 仍是唯
remain the only authoritative deploy inputs. 一权威部署输入。
If another transaction holds the Lease, the command must stop before publish or 如果另一个事务持有 Lease`--apply` 必须在 publish/apply 之前停止,并
apply with structured `deploy-lock-held` output that includes holder, 输出结构化 `deploy-lock-held`,包含 holder、promotion commit、phase、
promotion commit, phase, target namespace, and retry seconds. Expired locks are target namespace retry seconds。过期锁不能静默覆盖;stale lock 只能
not silently overwritten; taking over a stale lock requires both 在确认 holder 已失活后显式使用 `--break-stale-lock --confirm-dev`,并把
`--break-stale-lock` and `--confirm-dev`, and the previous holder is retained previous holder 保留在审计 annotations/report 字段中。未过期 Lease 没有
in audit annotations/report fields. 抢占入口,普通 status 输出也不得建议强制抢锁。
`scripts/dev-cd-apply.mjs` also owns the read-only status surface. Running it `scripts/dev-cd-apply.mjs` 同时提供只读状态面。无参数、`--status`、或
without arguments, with `--status`, or with `--dry-run` must return compact JSON `--dry-run` 都返回紧凑 JSON,覆盖 target ref/commit、
for the target ref, `deploy/deploy.json`, current Lease, and public `deploy/deploy.json` commit/hash、当前 Lease 状态摘要、以及公开
`16666/16667` live health. These modes must not acquire the Lease, publish `16666/16667` live health identity 摘要。这些模式 host commander 和
artifacts, apply Kubernetes workloads, or write reports. Normal `--apply` runner 都可以执行,但必须保持只读:不获取或替换 Lease,不 publish
stdout is also compact by default; the complete transaction record is preserved artifact,不执行 `kubectl apply/rollout`,不写 report,不修改
through `--write-report`, or printed only when `--full-output` is explicit. `deploy.json`/catalog,也不读取或打印 Secret value。kubectl 或 kubeconfig
不可用时,状态面应返回 `lock.status=unavailable` 和脱敏原因,而不是把
只读观测失败升级成写路径 blocker。
When the current DEV apply must be stopped, the operator may use 正常 `--apply` stdout 默认也是紧凑摘要;完整事务记录继续通过
`--force-interrupt-lock` together with `--apply --confirm-dev `--write-report` 写入既有 report 路径,只有显式 `--full-output` 时才打印
--confirmed-non-production`. This is a DEV-only takeover path: it first tries 完整 JSON。DEV apply、rollout 和 live verification 仍由 host commander
to send `TERM` to local `scripts/dev-cd-apply.mjs --apply` processes, then 统一执行;runner 默认只使用 status/dry-run 观测面,不竞争 DEV CD lock。
replaces the Lease holder and records the interrupted holder and interrupt
attempt in Lease annotations and the transaction report. Do not use this flag
for ordinary contention; prefer waiting for the holder or breaking only a
confirmed stale lock.
The old side-effect commands are now internal transaction steps. Direct manual 旧的 side-effect 命令已经降为事务内部步骤。直接手动执行
use of `scripts/dev-artifact-publish.mjs --publish` or `scripts/dev-artifact-publish.mjs --publish`
`scripts/dev-deploy-apply.mjs --apply` without `HWLAB_CD_TRANSACTION_ID` is `scripts/dev-deploy-apply.mjs --apply` 且没有 `HWLAB_CD_TRANSACTION_ID` 时,
rejected with `cd-transaction-required`; preflight, build, dry-run, check, and 会以 `cd-transaction-required` 拒绝;preflightbuilddry-runcheck
read-only modes remain available. read-only 模式仍可用。
## Artifact Publish ## Artifact Publish
@@ -299,6 +296,7 @@ use the repo-owned scripts below instead of ad hoc shell fragments.
| --- | --- | --- | | --- | --- | --- |
| Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. | | Check deployment contract and FRP/endpoint drift | `node scripts/deploy-contract-plan.mjs --check` | Fold into `hwlab deploy check --env dev`. |
| Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. | | Plan desired image/workload state | `node scripts/deploy-desired-state-plan.mjs --plan --pretty` and `node scripts/deploy-desired-state-plan.mjs --promotion-commit <sha> --check` | Fold into `hwlab deploy plan --env dev --file deploy/deploy.json` with a promotion commit gate. |
| Observe DEV CD status | `node scripts/dev-cd-apply.mjs`, `node scripts/dev-cd-apply.mjs --status`, or `node scripts/dev-cd-apply.mjs --dry-run` | Keep read-only for host and runners; no Lease mutation, artifact publish, apply, rollout, report write, or Secret read. |
| Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. | | Preflight artifact publish | `node scripts/dev-artifact-publish.mjs --preflight` | Keep as backend for `hwlab artifact publish --preflight`. |
| Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. | | Publish/apply/verify DEV | `node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` | Keep as the single side-effect transaction and run from repo-owned DEV CI/CD. |
| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. | | Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. |
+59 -156
View File
@@ -42,11 +42,7 @@ const lockAnnotationFields = {
releasedAt: `${lockAnnotationPrefix}/releasedAt`, releasedAt: `${lockAnnotationPrefix}/releasedAt`,
releaseStatus: `${lockAnnotationPrefix}/releaseStatus`, releaseStatus: `${lockAnnotationPrefix}/releaseStatus`,
staleLockBrokenAt: `${lockAnnotationPrefix}/staleLockBrokenAt`, staleLockBrokenAt: `${lockAnnotationPrefix}/staleLockBrokenAt`,
staleLockPrevious: `${lockAnnotationPrefix}/staleLockPrevious`, staleLockPrevious: `${lockAnnotationPrefix}/staleLockPrevious`
forceInterruptedAt: `${lockAnnotationPrefix}/forceInterruptedAt`,
forceInterruptedBy: `${lockAnnotationPrefix}/forceInterruptedBy`,
forceInterruptStatus: `${lockAnnotationPrefix}/forceInterruptStatus`,
forceInterruptedPrevious: `${lockAnnotationPrefix}/forceInterruptedPrevious`
}; };
class DevCdApplyError extends Error { class DevCdApplyError extends Error {
@@ -72,7 +68,6 @@ export function parseArgs(argv) {
ttlSeconds: defaultTtlSeconds, ttlSeconds: defaultTtlSeconds,
ownerTaskId: null, ownerTaskId: null,
breakStaleLock: false, breakStaleLock: false,
forceInterruptLock: false,
skipLiveVerify: false, skipLiveVerify: false,
status: false, status: false,
dryRun: false, dryRun: false,
@@ -98,9 +93,6 @@ export function parseArgs(argv) {
} else if (arg === "--break-stale-lock") { } else if (arg === "--break-stale-lock") {
args.breakStaleLock = true; args.breakStaleLock = true;
args.flags.add(arg); args.flags.add(arg);
} else if (arg === "--force-interrupt-lock") {
args.forceInterruptLock = true;
args.flags.add(arg);
} else if (arg === "--dry-run" || arg === "--status") { } else if (arg === "--dry-run" || arg === "--status") {
args.status = true; args.status = true;
args.dryRun = arg === "--dry-run"; args.dryRun = arg === "--dry-run";
@@ -140,7 +132,15 @@ export function parseArgs(argv) {
} else if (arg === "--help" || arg === "-h") { } else if (arg === "--help" || arg === "-h") {
args.help = true; args.help = true;
} else { } else {
throw new Error(`unknown argument ${arg}`); throw new DevCdApplyError(`unknown argument ${arg}`, {
code: "unknown-argument",
blockers: [{
type: "safety_blocker",
scope: "unknown-argument",
status: "open",
summary: `${arg} is not a supported DEV CD option. Use --status/--dry-run for read-only observability or --apply with explicit DEV confirmations.`
}]
});
} }
} }
@@ -163,21 +163,19 @@ export function usage() {
examples: [ examples: [
"node scripts/dev-cd-apply.mjs --status", "node scripts/dev-cd-apply.mjs --status",
"node scripts/dev-cd-apply.mjs --dry-run", "node scripts/dev-cd-apply.mjs --dry-run",
"node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report", "node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report"
"node scripts/dev-cd-apply.mjs --apply --confirm-dev --confirmed-non-production --force-interrupt-lock --write-report"
], ],
options: { options: {
"--status": "Read-only status: target ref, deploy.json, current Lease, and public live health.", "--status": "Read-only status: target ref, deploy.json, current Lease, and public live health.",
"--dry-run": "Read-only alias for --status; does not acquire the Lease or run side effects.", "--dry-run": "Read-only alias for --status; does not acquire the Lease or run side effects.",
"--apply": "Run the DEV CD transaction side effects.", "--apply": "Run the DEV CD transaction side effects.",
"--confirm-dev": "Required for DEV mutation and stale/force lock takeover.", "--confirm-dev": "Required for DEV mutation and explicit stale-lock recovery.",
"--confirmed-non-production": "Required for DEV mutation.", "--confirmed-non-production": "Required for DEV mutation.",
"--target-ref REF": "git ref to publish/apply; default: origin/main", "--target-ref REF": "git ref to publish/apply; default: origin/main",
"--kubeconfig PATH": "DEV kubeconfig path; also honors HWLAB_DEV_KUBECONFIG/KUBECONFIG", "--kubeconfig PATH": "DEV kubeconfig path; also honors HWLAB_DEV_KUBECONFIG/KUBECONFIG",
"--ttl-seconds SECONDS": "lock TTL; default: 3600", "--ttl-seconds SECONDS": "lock TTL; default: 3600",
"--owner-task-id ID": "lock holder; default: Code Queue/env/user derived", "--owner-task-id ID": "lock holder; default: Code Queue/env/user derived",
"--break-stale-lock": "May take over an expired lock only with --confirm-dev.", "--break-stale-lock": "May take over an expired lock only with --confirm-dev.",
"--force-interrupt-lock": "DEV-only explicit takeover: try to interrupt the current local dev-cd apply process, then acquire the Lease.",
"--report PATH": `default: ${defaultReportPath}`, "--report PATH": `default: ${defaultReportPath}`,
"--write-report": "Write the full report to --report.", "--write-report": "Write the full report to --report.",
"--full-output": "Print the full report to stdout instead of the concise summary.", "--full-output": "Print the full report to stdout instead of the concise summary.",
@@ -241,15 +239,17 @@ function defaultOwnerTaskId(env = process.env) {
function defaultRunCommand(command, args, options = {}) { function defaultRunCommand(command, args, options = {}) {
return new Promise((resolve) => { return new Promise((resolve) => {
const abortController = options.timeoutMs ? new AbortController() : null;
const child = spawn(command, args, { const child = spawn(command, args, {
cwd: options.cwd ?? defaultRepoRoot, cwd: options.cwd ?? defaultRepoRoot,
env: options.env ?? process.env, env: options.env ?? process.env,
stdio: ["pipe", "pipe", "pipe"] stdio: ["pipe", "pipe", "pipe"],
signal: abortController?.signal
}); });
let stdout = ""; let stdout = "";
let stderr = ""; let stderr = "";
const timeout = options.timeoutMs const timeout = options.timeoutMs
? setTimeout(() => child.kill("SIGTERM"), options.timeoutMs) ? setTimeout(() => abortController.abort(), options.timeoutMs)
: null; : null;
child.stdin.on("error", () => {}); child.stdin.on("error", () => {});
@@ -262,7 +262,12 @@ function defaultRunCommand(command, args, options = {}) {
}); });
child.on("error", (error) => { child.on("error", (error) => {
if (timeout) clearTimeout(timeout); if (timeout) clearTimeout(timeout);
resolve({ code: 127, signal: null, stdout, stderr: error.message }); resolve({
code: error.name === "AbortError" ? 124 : 127,
signal: null,
stdout,
stderr: error.message
});
}); });
child.on("close", (code, signal) => { child.on("close", (code, signal) => {
if (timeout) clearTimeout(timeout); if (timeout) clearTimeout(timeout);
@@ -346,22 +351,6 @@ function validateArgs(args) {
summary: "Breaking a stale DEV CD lock requires --break-stale-lock and --confirm-dev." summary: "Breaking a stale DEV CD lock requires --break-stale-lock and --confirm-dev."
}); });
} }
if (args.forceInterruptLock && !args.confirmDev) {
blockers.push({
type: "safety_blocker",
scope: "force-interrupt-confirmation",
status: "open",
summary: "Force-interrupting a DEV CD lock requires --force-interrupt-lock and --confirm-dev."
});
}
if (args.forceInterruptLock && !args.apply) {
blockers.push({
type: "safety_blocker",
scope: "force-interrupt-mode",
status: "open",
summary: "--force-interrupt-lock is only valid with --apply."
});
}
if (!Number.isInteger(args.ttlSeconds) || args.ttlSeconds < 60 || args.ttlSeconds > 24 * 60 * 60) { if (!Number.isInteger(args.ttlSeconds) || args.ttlSeconds < 60 || args.ttlSeconds > 24 * 60 * 60) {
blockers.push({ blockers.push({
type: "safety_blocker", type: "safety_blocker",
@@ -446,11 +435,7 @@ export function parseDeployLock(lease) {
releasedAt: readAnnotation(annotations, "releasedAt"), releasedAt: readAnnotation(annotations, "releasedAt"),
releaseStatus: readAnnotation(annotations, "releaseStatus"), releaseStatus: readAnnotation(annotations, "releaseStatus"),
staleLockBrokenAt: readAnnotation(annotations, "staleLockBrokenAt"), staleLockBrokenAt: readAnnotation(annotations, "staleLockBrokenAt"),
staleLockPrevious: readJsonAnnotation(annotations, "staleLockPrevious"), staleLockPrevious: readJsonAnnotation(annotations, "staleLockPrevious")
forceInterruptedAt: readAnnotation(annotations, "forceInterruptedAt"),
forceInterruptedBy: readAnnotation(annotations, "forceInterruptedBy"),
forceInterruptStatus: readAnnotation(annotations, "forceInterruptStatus"),
forceInterruptedPrevious: readJsonAnnotation(annotations, "forceInterruptedPrevious")
}; };
} }
@@ -512,7 +497,7 @@ function isConflict(result) {
return /conflict|object has been modified|operation cannot be fulfilled/i.test(`${result.stderr}\n${result.stdout}`); return /conflict|object has been modified|operation cannot be fulfilled/i.test(`${result.stderr}\n${result.stdout}`);
} }
function compactInterruptedLock(lock) { function compactLockSummary(lock) {
if (!lock) return null; if (!lock) return null;
return { return {
lockName: lock.lockName ?? defaultLockName, lockName: lock.lockName ?? defaultLockName,
@@ -526,39 +511,14 @@ function compactInterruptedLock(lock) {
updatedAt: lock.updatedAt ?? null, updatedAt: lock.updatedAt ?? null,
ttlSeconds: lock.ttlSeconds ?? null, ttlSeconds: lock.ttlSeconds ?? null,
targetNamespace: lock.targetNamespace ?? defaultNamespace, targetNamespace: lock.targetNamespace ?? defaultNamespace,
targetRef: lock.targetRef ?? null targetRef: lock.targetRef ?? null,
releasedAt: lock.releasedAt ?? null,
releaseStatus: lock.releaseStatus ?? null,
staleLockBrokenAt: lock.staleLockBrokenAt ?? null
}; };
} }
function compactForceInterruptRecord(record) { function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTaskId, startedAt, phase, liveBefore, staleLockPrevious = null }) {
if (!record) return null;
return {
status: record.status,
warning: record.warning ?? null,
processCandidates: (record.processCandidates ?? []).map((candidate) => ({
pid: candidate.pid,
command: candidate.command
})),
attempts: (record.attempts ?? []).map((attempt) => ({
pid: attempt.pid,
signal: attempt.signal,
status: attempt.status
}))
};
}
function attachForceInterruptAnnotations(lock, forceInterrupt, timestamp, ownerTaskId) {
if (!forceInterrupt) return;
lock.forceInterruptedAt = timestamp;
lock.forceInterruptedBy = ownerTaskId;
lock.forceInterruptStatus = forceInterrupt.status;
lock.forceInterruptedPrevious = {
previousLock: compactInterruptedLock(forceInterrupt.previousLock),
interrupt: compactForceInterruptRecord(forceInterrupt)
};
}
function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTaskId, startedAt, phase, liveBefore, staleLockPrevious = null, forceInterrupt = null }) {
const lock = { const lock = {
promotionCommit: target.shortCommitId, promotionCommit: target.shortCommitId,
deployJsonHash: deployBefore.hash, deployJsonHash: deployBefore.hash,
@@ -577,7 +537,6 @@ function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTa
lock.staleLockBrokenAt = startedAt; lock.staleLockBrokenAt = startedAt;
lock.staleLockPrevious = staleLockPrevious; lock.staleLockPrevious = staleLockPrevious;
} }
attachForceInterruptAnnotations(lock, forceInterrupt, startedAt, ownerTaskId);
return { return {
apiVersion: "coordination.k8s.io/v1", apiVersion: "coordination.k8s.io/v1",
kind: "Lease", kind: "Lease",
@@ -600,7 +559,7 @@ function buildLeaseManifest({ args, target, deployBefore, transactionId, ownerTa
}; };
} }
function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, phase, startedAt, updatedAt, liveBefore, staleLockPrevious = null, forceInterrupt = null }) { function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, phase, startedAt, updatedAt, liveBefore, staleLockPrevious = null }) {
const lock = { const lock = {
promotionCommit: target.shortCommitId, promotionCommit: target.shortCommitId,
deployJsonHash: deployBefore.hash, deployJsonHash: deployBefore.hash,
@@ -619,7 +578,6 @@ function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, pha
lock.staleLockBrokenAt = updatedAt; lock.staleLockBrokenAt = updatedAt;
lock.staleLockPrevious = staleLockPrevious; lock.staleLockPrevious = staleLockPrevious;
} }
attachForceInterruptAnnotations(lock, forceInterrupt, updatedAt, ownerTaskId);
return { return {
metadata: { metadata: {
labels: { labels: {
@@ -638,66 +596,6 @@ function lockPatch({ args, target, deployBefore, transactionId, ownerTaskId, pha
}; };
} }
export function findDevCdApplyProcessCandidates(psOutput, currentPid = process.pid) {
return String(psOutput ?? "")
.split(/\r?\n/u)
.map((line) => line.trim())
.filter(Boolean)
.map((line) => {
const match = line.match(/^(\d+)\s+(.+)$/u);
if (!match) return null;
return {
pid: Number.parseInt(match[1], 10),
command: oneLine(redactSensitiveText(match[2])).slice(0, 500)
};
})
.filter((entry) => {
if (!entry || !Number.isInteger(entry.pid) || entry.pid === currentPid) return false;
return entry.command.includes("scripts/dev-cd-apply.mjs") && entry.command.includes("--apply");
});
}
async function forceInterruptCurrentApplyProcess(ctx, existingLock) {
const ps = await ctx.runCommand("ps", ["-eo", "pid=,args="], {
cwd: ctx.repoRoot,
timeoutMs: 5000
});
if (ps.code !== 0) {
return {
status: "ps-failed",
previousLock: compactInterruptedLock(existingLock),
processCandidates: [],
attempts: [],
warning: oneLine(redactSensitiveText(ps.stderr || ps.stdout || "ps failed"))
};
}
const processCandidates = findDevCdApplyProcessCandidates(ps.stdout, process.pid);
const attempts = [];
for (const candidate of processCandidates) {
const result = await ctx.runCommand("kill", ["-TERM", String(candidate.pid)], {
cwd: ctx.repoRoot,
timeoutMs: 5000
});
attempts.push({
pid: candidate.pid,
signal: "TERM",
status: result.code === 0 ? "sent" : "failed",
stderrTail: redactSensitiveText(result.stderr || result.stdout || "").trim().slice(-500)
});
}
return {
status: processCandidates.length > 0 ? "attempted" : "no-local-process-found",
previousLock: compactInterruptedLock(existingLock),
processCandidates,
attempts,
warning: processCandidates.length > 0
? null
: "No local scripts/dev-cd-apply.mjs --apply process was found; the Lease will still be taken over so the previous holder cannot continue lock-owned updates."
};
}
async function kubectl(ctx, kubectlContext, args, options = {}) { async function kubectl(ctx, kubectlContext, args, options = {}) {
const result = await ctx.runCommand(kubectlContext.executor, args, { const result = await ctx.runCommand(kubectlContext.executor, args, {
cwd: ctx.repoRoot, cwd: ctx.repoRoot,
@@ -728,7 +626,7 @@ async function replaceLease(ctx, kubectlContext, args, lease, timeoutMs = 30000)
function mergeLease(lease, patch, { resetLockAnnotations = false } = {}) { function mergeLease(lease, patch, { resetLockAnnotations = false } = {}) {
const baseAnnotations = { ...(lease.metadata?.annotations ?? {}) }; const baseAnnotations = { ...(lease.metadata?.annotations ?? {}) };
if (resetLockAnnotations) { if (resetLockAnnotations) {
for (const key of Object.values(lockAnnotationFields)) { for (const key of Object.keys(baseAnnotations).filter((name) => name.startsWith(`${lockAnnotationPrefix}/`))) {
delete baseAnnotations[key]; delete baseAnnotations[key];
} }
} }
@@ -760,14 +658,14 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo
const lease = JSON.parse(get.stdout); const lease = JSON.parse(get.stdout);
const existing = parseDeployLock(lease); const existing = parseDeployLock(lease);
const classification = classifyDeployLock(existing, now); const classification = classifyDeployLock(existing, now);
if (classification.held && !args.forceInterruptLock) { if (classification.held) {
const failure = deployLockHeldFailure(existing, now); const failure = deployLockHeldFailure(existing, now);
throw new DevCdApplyError(failure.summary, { throw new DevCdApplyError(failure.summary, {
code: "deploy-lock-held", code: "deploy-lock-held",
lockFailure: failure lockFailure: failure
}); });
} }
if (classification.stale && !args.breakStaleLock && !args.forceInterruptLock) { if (classification.stale && !args.breakStaleLock) {
const failure = deployLockHeldFailure(existing, now); const failure = deployLockHeldFailure(existing, now);
throw new DevCdApplyError(failure.summary, { throw new DevCdApplyError(failure.summary, {
code: "deploy-lock-held", code: "deploy-lock-held",
@@ -775,10 +673,6 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo
}); });
} }
const forceInterrupt = args.forceInterruptLock
? await forceInterruptCurrentApplyProcess(ctx, existing)
: null;
const patch = lockPatch({ const patch = lockPatch({
args, args,
target, target,
@@ -789,8 +683,7 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo
startedAt, startedAt,
updatedAt: startedAt, updatedAt: startedAt,
liveBefore, liveBefore,
staleLockPrevious: classification.stale ? existing : null, staleLockPrevious: classification.stale ? existing : null
forceInterrupt
}); });
const replaced = await replaceLease(ctx, kubectlContext, args, mergeLease(lease, patch, { resetLockAnnotations: true })); const replaced = await replaceLease(ctx, kubectlContext, args, mergeLease(lease, patch, { resetLockAnnotations: true }));
if (isConflict(replaced)) { if (isConflict(replaced)) {
@@ -806,7 +699,6 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo
acquired: true, acquired: true,
lock: parseDeployLock(JSON.parse(replaced.stdout)), lock: parseDeployLock(JSON.parse(replaced.stdout)),
staleBreak: classification.stale ? existing : null, staleBreak: classification.stale ? existing : null,
forceInterrupt,
command: replaced.command command: replaced.command
}; };
} }
@@ -837,7 +729,6 @@ async function acquireDeployLock({ ctx, args, kubectlContext, target, deployBefo
acquired: true, acquired: true,
lock: parseDeployLock(JSON.parse(created.stdout)), lock: parseDeployLock(JSON.parse(created.stdout)),
staleBreak: null, staleBreak: null,
forceInterrupt: null,
command: created.command command: created.command
}; };
} }
@@ -1410,6 +1301,10 @@ function summarizeDeployJson(deploy) {
}; };
} }
function deployJsonMatchesTarget(deploy, target) {
return deploy.commitId === target.shortCommitId || deploy.commitId === target.commitId;
}
function compactLiveEndpoint(endpoint) { function compactLiveEndpoint(endpoint) {
return { return {
id: endpoint.id, id: endpoint.id,
@@ -1481,7 +1376,8 @@ function summarizeLockStatus(lockRead, now) {
current: null, current: null,
retryAfterSeconds: null, retryAfterSeconds: null,
expiresAt: null, expiresAt: null,
error: oneLine(lockRead.error) reason: "kubectl or DEV kubeconfig unavailable; command output is redacted",
redactedReason: oneLine(lockRead.error).slice(0, 500)
}; };
} }
const classification = classifyDeployLock(lockRead.lock, now); const classification = classifyDeployLock(lockRead.lock, now);
@@ -1494,12 +1390,12 @@ function summarizeLockStatus(lockRead, now) {
: "unknown"; : "unknown";
return { return {
status, status,
current: compactInterruptedLock(lockRead.lock), lockName: lockRead.lock?.lockName ?? lockRead.lockName ?? defaultLockName,
current: compactLockSummary(lockRead.lock),
retryAfterSeconds: classification.retryAfterSeconds, retryAfterSeconds: classification.retryAfterSeconds,
expiresAt: classification.expiresAt, expiresAt: classification.expiresAt,
takeover: { allowedLockAction: {
stale: classification.stale ? "--apply --confirm-dev --confirmed-non-production --break-stale-lock" : null, breakStale: classification.stale ? "--apply --confirm-dev --confirmed-non-production --break-stale-lock" : null
forceInterrupt: status === "held" ? "--apply --confirm-dev --confirmed-non-production --force-interrupt-lock" : null
} }
}; };
} }
@@ -1513,7 +1409,7 @@ function buildStatusNextActions({ target, deployBefore, lock, liveVerify }) {
actions.push(`deploy/deploy.json commitId is ${deployBefore.commitId}; target ref is ${target.shortCommitId}.`); actions.push(`deploy/deploy.json commitId is ${deployBefore.commitId}; target ref is ${target.shortCommitId}.`);
} }
if (lock.status === "held") { if (lock.status === "held") {
actions.push(`Wait ${lock.retryAfterSeconds}s or use explicit --force-interrupt-lock if the current apply must be stopped.`); actions.push(`Wait for the current DEV CD holder to release the Lease or for the ${lock.retryAfterSeconds}s TTL to expire before retrying apply.`);
} else if (lock.status === "stale") { } else if (lock.status === "stale") {
actions.push("Use --break-stale-lock only after confirming the holder is inactive."); actions.push("Use --break-stale-lock only after confirming the holder is inactive.");
} else if (lock.status === "unavailable") { } else if (lock.status === "unavailable") {
@@ -1538,7 +1434,13 @@ async function runDevCdStatus(args, ctx, stdout) {
const liveVerify = args.skipLiveVerify const liveVerify = args.skipLiveVerify
? { status: "not_run", reason: "--skip-live-verify", endpoints: [], summary: { checked: 0 } } ? { status: "not_run", reason: "--skip-live-verify", endpoints: [], summary: { checked: 0 } }
: await verifyDevLive(ctx, { expectedCommit: null }); : await verifyDevLive(ctx, { expectedCommit: null });
const status = lock.status === "unavailable" || liveVerify.status !== "pass" ? "degraded" : "pass"; const deployMatchesTarget = deployJsonMatchesTarget(deployBefore, target);
const status = (
lock.status === "unavailable" ||
liveVerify.status !== "pass" ||
!target.headMatchesTarget ||
!deployMatchesTarget
) ? "degraded" : "pass";
const report = { const report = {
ok: true, ok: true,
status, status,
@@ -1555,7 +1457,10 @@ async function runDevCdStatus(args, ctx, stdout) {
headMatchesTarget: target.headMatchesTarget, headMatchesTarget: target.headMatchesTarget,
namespace: args.targetNamespace namespace: args.targetNamespace
}, },
deployJson: summarizeDeployJson(deployBefore), deployJson: {
...summarizeDeployJson(deployBefore),
matchesTarget: deployMatchesTarget
},
kubectl: { kubectl: {
kubeconfigSource: kubectlContext.kubeconfigSource, kubeconfigSource: kubectlContext.kubeconfigSource,
kubeconfig: kubectlContext.kubeconfig ? "<set>" : "<unset>", kubeconfig: kubectlContext.kubeconfig ? "<set>" : "<unset>",
@@ -1592,9 +1497,8 @@ function summarizeDevCdApplyReport(report) {
target: apply.target ?? null, target: apply.target ?? null,
lock: { lock: {
acquired: Boolean(apply.lock?.acquired), acquired: Boolean(apply.lock?.acquired),
current: compactInterruptedLock(apply.lock?.current), current: compactLockSummary(apply.lock?.current),
staleBreak: compactInterruptedLock(apply.lock?.staleBreak), staleBreak: compactLockSummary(apply.lock?.staleBreak)
forceInterrupt: compactForceInterruptRecord(apply.lock?.forceInterrupt)
}, },
steps: (apply.steps ?? []).map((step) => ({ steps: (apply.steps ?? []).map((step) => ({
id: step.id, id: step.id,
@@ -1726,8 +1630,7 @@ function buildReport({
lock: { lock: {
acquired: Boolean(lockState?.acquired), acquired: Boolean(lockState?.acquired),
current: lockState?.lock ?? null, current: lockState?.lock ?? null,
staleBreak: lockState?.staleBreak ?? null, staleBreak: lockState?.staleBreak ?? null
forceInterrupt: lockState?.forceInterrupt ?? null
}, },
steps, steps,
liveBefore, liveBefore,
+185 -72
View File
@@ -10,7 +10,6 @@ import {
buildLockAnnotations, buildLockAnnotations,
classifyDeployLock, classifyDeployLock,
deployLockHeldFailure, deployLockHeldFailure,
findDevCdApplyProcessCandidates,
parseDeployLock, parseDeployLock,
runDevCdApply, runDevCdApply,
verifyDevLive verifyDevLive
@@ -99,7 +98,7 @@ function makeHttpGetJson() {
}; };
} }
function makeRunCommand({ heldLock = null, commandLog = [], psOutput = "", killLog = [] } = {}) { function makeRunCommand({ heldLock = null, commandLog = [] } = {}) {
let lease = heldLock ? leaseFromLock(heldLock) : null; let lease = heldLock ? leaseFromLock(heldLock) : null;
const assertLeaseMicroTime = (value) => { const assertLeaseMicroTime = (value) => {
assert.match(value, /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{6}Z$/u); assert.match(value, /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{6}Z$/u);
@@ -115,13 +114,6 @@ function makeRunCommand({ heldLock = null, commandLog = [], psOutput = "", killL
if (command === "which" && args[0] === "kubectl") { if (command === "which" && args[0] === "kubectl") {
return { code: 0, stdout: "/usr/local/bin/kubectl\n", stderr: "" }; return { code: 0, stdout: "/usr/local/bin/kubectl\n", stderr: "" };
} }
if (command === "ps") {
return { code: 0, stdout: psOutput, stderr: "" };
}
if (command === "kill") {
killLog.push({ signal: args[0], pid: args[1] });
return { code: 0, stdout: "", stderr: "" };
}
if (command.includes("kubectl") && args.includes("get") && args.includes("lease")) { if (command.includes("kubectl") && args.includes("get") && args.includes("lease")) {
if (!lease) return { code: 1, stdout: "", stderr: "Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found" }; if (!lease) return { code: 1, stdout: "", stderr: "Error from server (NotFound): leases.coordination.k8s.io \"hwlab-dev-cd-lock\" not found" };
return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" }; return { code: 0, stdout: `${JSON.stringify(lease)}\n`, stderr: "" };
@@ -191,6 +183,18 @@ function makeRunCommand({ heldLock = null, commandLog = [], psOutput = "", killL
}; };
} }
function assertNoReadOnlySideEffects(commandLog) {
const mutatingKubectlVerbs = new Set(["apply", "create", "replace", "patch", "delete", "rollout", "wait", "logs"]);
assert.equal(commandLog.some((entry) => entry.command === "ps"), false);
assert.equal(commandLog.some((entry) => entry.command === "kill"), false);
assert.equal(commandLog.some((entry) =>
entry.command.includes("kubectl") && entry.args.some((arg) => mutatingKubectlVerbs.has(arg))
), false);
assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false);
assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false);
assert.equal(commandLog.some((entry) => entry.args.includes("scripts/refresh-artifact-catalog.mjs")), false);
}
test("lock classifier reports held and stale states with retryAfterSeconds", () => { test("lock classifier reports held and stale states with retryAfterSeconds", () => {
const lock = { const lock = {
ownerTaskId: "task-a", ownerTaskId: "task-a",
@@ -222,7 +226,8 @@ test("dev-cd status and dry-run are read-only compact JSON", async () => {
const code = await runDevCdApply([ const code = await runDevCdApply([
"--dry-run", "--dry-run",
"--kubeconfig", "--kubeconfig",
"/tmp/kubeconfig" "/tmp/kubeconfig",
"--write-report"
], { ], {
repoRoot, repoRoot,
env: {}, env: {},
@@ -239,22 +244,179 @@ test("dev-cd status and dry-run are read-only compact JSON", async () => {
assert.equal(status.prodTouched, false); assert.equal(status.prodTouched, false);
assert.equal(status.lock.status, "absent"); assert.equal(status.lock.status, "absent");
assert.equal(status.deployJson.manifest, undefined); assert.equal(status.deployJson.manifest, undefined);
assert.equal(status.deployJson.matchesTarget, true);
assert.equal(status.live.summary.checked, 2); assert.equal(status.live.summary.checked, 2);
assert.equal(commandLog.some((entry) => entry.args.includes("create")), false); assertNoReadOnlySideEffects(commandLog);
assert.equal(commandLog.some((entry) => entry.args.includes("replace")), false); await assert.rejects(
assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-artifact-publish.mjs")), false); readFile(path.join(repoRoot, "reports/dev-gate/dev-cd-apply.json"), "utf8"),
assert.equal(commandLog.some((entry) => entry.args.includes("scripts/dev-deploy-apply.mjs")), false); { code: "ENOENT" }
);
}); });
test("process candidate discovery only returns dev-cd apply processes", () => { test("dev-cd default status is read-only compact JSON", async () => {
const psOutput = [ const repoRoot = await makeRepo();
"101 node scripts/dev-cd-apply.mjs --apply --confirm-dev", const commandLog = [];
"202 node scripts/dev-cd-apply.mjs --status", let output = "";
"303 node scripts/dev-deploy-apply.mjs --apply", const code = await runDevCdApply([], {
"404 node scripts/dev-cd-apply.mjs --apply --confirm-dev" repoRoot,
].join("\n"); env: {},
const candidates = findDevCdApplyProcessCandidates(psOutput, 404); runCommand: makeRunCommand({ commandLog }),
assert.deepEqual(candidates.map((candidate) => candidate.pid), [101]); httpGetJson: makeHttpGetJson(),
now: () => new Date(iso(100000)),
stdout: { write: (chunk) => { output += chunk; } }
});
const status = JSON.parse(output);
assert.equal(code, 0);
assert.equal(status.mode, "status");
assert.equal(status.mutationAttempted, false);
assert.equal(status.prodTouched, false);
assert.equal(status.target.shortCommitId, "abc1234");
assert.equal(status.deployJson.hash.startsWith("sha256:"), true);
assert.equal(status.deployJson.matchesTarget, true);
assert.equal(status.lock.status, "absent");
assertNoReadOnlySideEffects(commandLog);
});
test("dev-cd status summarizes held, released, and stale locks without unsafe takeover hints", async () => {
const cases = [
{
name: "held",
phase: "applying",
now: iso(100000),
expectedStatus: "held",
expectedRetryAfterSeconds: 200
},
{
name: "released",
phase: "released",
releasedAt: iso(50000),
releaseStatus: "pass",
now: iso(100000),
expectedStatus: "released",
expectedRetryAfterSeconds: 0
},
{
name: "stale",
phase: "verifying",
now: iso(301000),
expectedStatus: "stale",
expectedRetryAfterSeconds: 0
}
];
for (const item of cases) {
const repoRoot = await makeRepo();
const commandLog = [];
let output = "";
const code = await runDevCdApply(["--status", "--kubeconfig", "/tmp/kubeconfig"], {
repoRoot,
env: {},
runCommand: makeRunCommand({
commandLog,
heldLock: {
promotionCommit: "abc1234",
deployJsonHash: "sha256:a",
ownerTaskId: `task-${item.name}`,
transactionId: `tx-${item.name}`,
phase: item.phase,
startedAt: iso(0),
updatedAt: iso(0),
ttlSeconds: 300,
liveBefore: { status: "pass" },
targetNamespace: "hwlab-dev",
targetRef: "origin/main",
lockBackend: "Lease",
...(item.releasedAt ? { releasedAt: item.releasedAt } : {}),
...(item.releaseStatus ? { releaseStatus: item.releaseStatus } : {})
}
}),
httpGetJson: makeHttpGetJson(),
now: () => new Date(item.now),
stdout: { write: (chunk) => { output += chunk; } }
});
const status = JSON.parse(output);
assert.equal(code, 0);
assert.equal(status.lock.status, item.expectedStatus);
assert.equal(status.lock.lockName, "hwlab-dev-cd-lock");
assert.equal(status.lock.current.ownerTaskId, `task-${item.name}`);
assert.equal(status.lock.retryAfterSeconds, item.expectedRetryAfterSeconds);
assert.equal(JSON.stringify(status).includes("force-interrupt"), false);
assert.equal(JSON.stringify(status.nextActions).includes("force"), false);
if (item.expectedStatus === "stale") {
assert.match(status.lock.allowedLockAction.breakStale, /--break-stale-lock/u);
}
assertNoReadOnlySideEffects(commandLog);
}
});
test("dev-cd status reports unavailable kubectl with redacted reason", async () => {
const repoRoot = await makeRepo();
const commandLog = [];
let output = "";
const runCommand = async (command, args, options = {}) => {
commandLog.push({ command, args, env: options.env ?? {}, input: options.input ?? "" });
if (command === "git" && args[0] === "rev-parse" && args.includes("origin/main^{commit}")) {
return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" };
}
if (command === "git" && args[0] === "rev-parse" && args.includes("HEAD^{commit}")) {
return { code: 0, stdout: "abc1234abc1234abc1234abc1234abc1234abc1\n", stderr: "" };
}
if (command === "which" && args[0] === "kubectl") {
return { code: 1, stdout: "", stderr: "kubectl missing" };
}
if (command === "kubectl") {
return {
code: 1,
stdout: "",
stderr: "connection failed token=SECRET_TOKEN password=SECRET_PASSWORD postgresql://user:SECRET_DB@db"
};
}
return { code: 0, stdout: "", stderr: "" };
};
const code = await runDevCdApply(["--status"], {
repoRoot,
env: {},
runCommand,
httpGetJson: makeHttpGetJson(),
now: () => new Date(iso(100000)),
stdout: { write: (chunk) => { output += chunk; } }
});
const status = JSON.parse(output);
const text = JSON.stringify(status);
assert.equal(code, 0);
assert.equal(status.status, "degraded");
assert.equal(status.lock.status, "unavailable");
assert.equal(status.lock.reason, "kubectl or DEV kubeconfig unavailable; command output is redacted");
assert.equal(text.includes("SECRET_TOKEN"), false);
assert.equal(text.includes("SECRET_PASSWORD"), false);
assert.equal(text.includes("SECRET_DB"), false);
assert.match(status.lock.redactedReason, /<redacted>/u);
assertNoReadOnlySideEffects(commandLog);
});
test("force-interrupt option is rejected before process discovery or lease mutation", async () => {
const repoRoot = await makeRepo();
const commandLog = [];
let output = "";
const code = await runDevCdApply(["--force-interrupt-lock"], {
repoRoot,
env: {},
runCommand: makeRunCommand({ commandLog }),
httpGetJson: makeHttpGetJson(),
now: () => new Date(iso(100000)),
stdout: { write: (chunk) => { output += chunk; } }
});
const failure = JSON.parse(output);
assert.equal(code, 2);
assert.equal(failure.error, "unknown-argument");
assert.equal(failure.mutationAttempted, false);
assert.equal(failure.prodTouched, false);
assert.deepEqual(commandLog, []);
}); });
test("deploy-lock-held failure is structured before publish/apply", () => { test("deploy-lock-held failure is structured before publish/apply", () => {
@@ -489,55 +651,6 @@ test("apply stdout is concise by default while write-report keeps the full repor
assert.equal(writtenReport.devCdApply.deployJson.before.manifest.environment, "dev"); assert.equal(writtenReport.devCdApply.deployJson.before.manifest.environment, "dev");
}); });
test("force-interrupt-lock terminates local apply candidate and takes a held lock", async () => {
const repoRoot = await makeRepo();
const killLog = [];
let output = "";
const code = await runDevCdApply([
"--apply",
"--confirm-dev",
"--confirmed-non-production",
"--force-interrupt-lock",
"--owner-task-id",
"task-force",
"--kubeconfig",
"/tmp/kubeconfig",
"--full-output"
], {
repoRoot,
env: {},
runCommand: makeRunCommand({
killLog,
psOutput: "555 node scripts/dev-cd-apply.mjs --apply --confirm-dev\n666 node scripts/dev-cd-apply.mjs --status\n",
heldLock: {
promotionCommit: "abc1234",
deployJsonHash: "sha256:a",
ownerTaskId: "task-a",
transactionId: "tx-a",
phase: "applying",
startedAt: iso(0),
updatedAt: iso(0),
ttlSeconds: 300,
liveBefore: { status: "pass" },
targetNamespace: "hwlab-dev",
targetRef: "origin/main",
lockBackend: "Lease"
}
}),
httpGetJson: makeHttpGetJson(),
now: () => new Date(iso(100000)),
stdout: { write: (chunk) => { output += chunk; } }
});
const report = JSON.parse(output);
assert.equal(code, 0);
assert.deepEqual(killLog, [{ signal: "-TERM", pid: "555" }]);
assert.equal(report.devCdApply.lock.forceInterrupt.status, "attempted");
assert.equal(report.devCdApply.lock.forceInterrupt.processCandidates[0].pid, 555);
assert.equal(report.devCdApply.lock.current.forceInterruptStatus, "attempted");
assert.equal(report.devCdApply.lock.current.forceInterruptedBy, "task-force");
});
test("live verify summarizes 16666 and 16667 health identity", async () => { test("live verify summarizes 16666 and 16667 health identity", async () => {
const result = await verifyDevLive({ const result = await verifyDevLive({
now: () => new Date(iso()), now: () => new Date(iso()),