docs: add DEV runtime hotfix runbook and audit

Refs #462\nRefs #465\nRefs #7\nRefs #239
This commit is contained in:
Lyon
2026-05-24 20:26:21 +08:00
committed by GitHub
parent 0163390dc3
commit 9d143874f8
7 changed files with 860 additions and 2 deletions
+2
View File
@@ -37,6 +37,7 @@ HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥
- DEV 运行态、端口、k3s 和 DB DNS 边界:[docs/reference/dev-runtime-boundary.md](docs/reference/dev-runtime-boundary.md)
- 部署正规化、`deploy.json` DEV CD 路径、SecretRef preflight、runner/host 边界和镜像发布:[docs/reference/deployment-publish.md](docs/reference/deployment-publish.md)
- Code Agent 对话就绪与真实回复判定:[docs/reference/code-agent-chat-readiness.md](docs/reference/code-agent-chat-readiness.md)
- DEV runtime hotfix runbook 与只读审计:[docs/reference/dev-runtime-hotfix-runbook.md](docs/reference/dev-runtime-hotfix-runbook.md)
- Gateway 主动出站 demo、poll/result 和本地 smoke[docs/reference/gateway-outbound-demo.md](docs/reference/gateway-outbound-demo.md)
- MVP E2E 验收测试与带编号测试报告 issue 规则:[docs/reference/MVP-e2e-acceptance.md](docs/reference/MVP-e2e-acceptance.md)
- 指挥官协作、PR 和 runner 交接:[docs/reference/commander-collaboration.md](docs/reference/commander-collaboration.md)
@@ -64,6 +65,7 @@ HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥
- CI/CD job 查询:`node tools/hwlab-cli/bin/hwlab-cli.mjs cicd status|logs|report <jobId>`
- runner GitHub 可见性预检:`npm run runner:issue-visibility:preflight`
- D601 k3s 只读观测:`npm run d601:k3s:readonly`
- DEV runtime hotfix 只读审计计划:`npm run dev-runtime:hotfix-audit`
- Gateway 主动出站本地 smoke`npm run gateway:demo:smoke`;经本地 edge-proxy 验证用 `npm run gateway:demo:edge-smoke`
## D601 k3s 只读观测
+14
View File
@@ -163,6 +163,20 @@ contracts, docs, and tests so future rollouts reproduce the same env and safety
boundaries. The runner must not re-execute the live hotfix, restart services,
read Secrets, mutate PROD, or claim live evidence from source-only changes.
DEV pod/Deployment file overrides, ConfigMap mounts, and pod-local edits are
governed by [dev-runtime-hotfix-runbook.md](dev-runtime-hotfix-runbook.md). The
stable audit entry is:
```sh
node scripts/dev-runtime-hotfix-audit.mjs --pretty
node scripts/dev-runtime-hotfix-audit.mjs --collect-readonly --pretty
```
The default mode only prints a read-only plan. `--collect-readonly` may run
`kubectl get` and `kubectl exec` checks with
`KUBECONFIG=/etc/rancher/k3s/k3s.yaml`; it must not apply, patch, rollout,
restart, delete, create, read Secret resources, or print secret values.
## Durable Runtime Readiness Contract
DB live readiness and durable runtime readiness are separate gates. A
@@ -0,0 +1,106 @@
# HWLAB DEV Runtime Hotfix Runbook
本文定义 `hwlab-dev` 运行态热修的最低边界、审计字段、只读确认和回滚口径。它只适用于 DEV 能力边界或阻塞排障,不是发布验收手册,也不把运行态覆盖当作源码真相。
本规则承接 [pikasTech/HWLAB#462](https://github.com/pikasTech/HWLAB/issues/462) 和 [pikasTech/HWLAB#465](https://github.com/pikasTech/HWLAB/issues/465),服务 [pikasTech/HWLAB#7](https://github.com/pikasTech/HWLAB/issues/7) / [pikasTech/HWLAB#239](https://github.com/pikasTech/HWLAB/issues/239) 的“用户工作台 + Code Agent 真实可用”主线。它不改变 `DC-DCSN-P0-2026-003` / [pikasTech/HWLAB#78](https://github.com/pikasTech/HWLAB/issues/78) 的 M3 上位约束:hotfix runbook、审计输出、SOURCE、LOCAL、DRY-RUN 或只读观察都不能被包装成 M3 DEV-LIVE 通过。
## 允许条件
DEV runtime hotfix 只能在以下条件同时满足时使用:
- 目标是 `hwlab-dev`,不是 PROD、真实用户生产面或真实硬件生产控制面。
- 目的是打通能力边界、定位阻塞层或恢复用户工作台关键路径;不能为了绕过源码、测试、artifact 或 CD 流程而热修。
- 操作前已经明确该任务不会抢占 M3 虚拟硬件可信闭环主线,也不会把热修证据升级成 M3 验收。
- 操作后必须创建或关联源码化 follow-up。#465 的持久化拆分是固定分工:#460 负责 Code Agent -> PC gateway `shell.exec` 能力路由,#461 负责 hotfix runner 源码化,#462 负责本 runbook 与只读审计,#463 负责 JSON-RPC `serviceId` 结构化诊断,#464 负责 DEV-LIVE smoke/harness。
## D601 k3s 前置
D601 裸 `kubectl` 不可信。所有只读观测、热修、回滚和复核都必须显式指定原生 k3s kubeconfig,并先确认节点:
```sh
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get nodes -o jsonpath='{.items[*].metadata.name}'
```
结果必须包含 `d601`。出现 `docker-desktop``desktop-control-plane``127.0.0.1:11700` 或第二套 `hwlab-dev` 控制面时,停止 hotfix、回滚和验收判断。此边界与 [dev-runtime-boundary.md](dev-runtime-boundary.md) 和 [deployment-publish.md](deployment-publish.md) 保持一致。
## 必须记录
每次 DEV hotfix 至少记录以下字段,写入 PR/issue/comment,不写入仓库 `reports/**`
| 字段 | 说明 |
| --- | --- |
| Deployment | namespace、name、generation、template annotation、容器 image。 |
| Pod | 新旧 pod 名、phase、node、用于验证的 running pod。 |
| ConfigMap | 名称、key 名、用途;不得把 data 全量贴入 issue。 |
| annotation | 记录热修来源和目的的 Deployment template annotation。 |
| mount/subPath | volume 名、ConfigMap 名、mountPath、subPath、readOnly。 |
| traceId / operationId | 能力验证请求的 trace 与 operation 标识。 |
| 验证命令 | `grep` marker、`node --check`、API 返回字段等只读或最小验证命令。 |
| 回滚方式 | 移除 volume/mount/annotation 的方法,或用源码 artifact/CD 覆盖的方法。 |
| 源码化 follow-up | 关联 issue/PR,说明热修不会作为长期 runtime truth。 |
#465 的典型对象是:
| 项 | 值 |
| --- | --- |
| Deployment | `hwlab-dev/hwlab-cloud-api` |
| ConfigMap | `hwlab-cloud-api-code-agent-hotfix` |
| 覆盖文件 | `/app/internal/cloud/code-agent-chat.mjs` |
| subPath | `code-agent-chat.mjs` |
| marker | `runtime-hotfix-pc-gateway-shell` |
| annotation | `hwlab.pikastech.local/pc-gateway-shell-hotfix` |
| 能力路径 | `/v1/agent/chat -> /v1/rpc/hardware.invoke.shell -> PC gateway -> Windows cmd` |
## 只读审计入口
仓库内只读入口是:
```sh
node scripts/dev-runtime-hotfix-audit.mjs --pretty
node scripts/dev-runtime-hotfix-audit.mjs --collect-readonly --pretty
```
默认 `--plan` 模式只输出 compact JSON 计划,不执行 `kubectl``--collect-readonly` 只允许 `kubectl get``kubectl exec` 形态的读检查,用于判断当前 DEV 是否仍被 hotfix 覆盖;它不会执行 `apply``patch``rollout``restart``delete``create``set image`,也不会读取 Secret resources 或 secret values。
输出分类固定包含以下语义:
| 分类 | 含义 |
| --- | --- |
| `hotfix-configmap-present` | 目标 ConfigMap 仍存在。 |
| `deployment-mounts-hotfix` | Deployment template 仍通过 volumeMount/subPath 覆盖目标文件。 |
| `pod-loads-hotfix-marker` | running pod 内目标文件仍包含 hotfix marker。 |
| `source-artifact-expected` | 正式收口应回到 source/PR/artifact/CD。 |
| `rollback-required` | 当前仍检测到热修残留或覆盖,不能把运行态当源码真相。 |
| `no-hotfix-detected` | 只读审计未发现该 hotfix ConfigMap 或 mount 覆盖。 |
| `unknown-needs-manual-readonly-check` | 无法确认节点、对象或 pod 状态,需要人工继续只读核对。 |
## 回滚口径
优先回滚方式是用源码化 artifact/CD 覆盖 runtime hotfix#460/#461 合并并发布后,Deployment 应消费正式镜像,不再通过 ConfigMap 覆盖 `/app/internal/cloud/code-agent-chat.mjs`。只读确认口径是:
```sh
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get nodes -o jsonpath='{.items[*].metadata.name}'
kubectl -n hwlab-dev get deployment hwlab-cloud-api -o json
kubectl -n hwlab-dev get configmap hwlab-cloud-api-code-agent-hotfix -o json
node scripts/dev-runtime-hotfix-audit.mjs --collect-readonly --pretty
```
如果必须直接移除运行态覆盖,授权操作者应先从 `kubectl -n hwlab-dev get deployment hwlab-cloud-api -o json` 中定位精确 indexes,再移除:
- 指向 hotfix ConfigMap 的 `spec.template.spec.volumes[]`
- 指向 `/app/internal/cloud/code-agent-chat.mjs``containers[].volumeMounts[]`
- `hwlab.pikastech.local/pc-gateway-shell-hotfix` 等 hotfix annotation
- 不再需要的 `hwlab-cloud-api-code-agent-hotfix` ConfigMap。
直接 `kubectl patch``kubectl delete configmap``kubectl rollout status` 或等价写操作必须由明确授权的 DEV 操作者执行,并且仍要使用 `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` 与节点 `d601` 确认。本 runbook 的审计脚本不执行这些写操作。回滚后再次运行 `scripts/dev-runtime-hotfix-audit.mjs --collect-readonly`,期望分类包含 `no-hotfix-detected`,且不包含 `deployment-mounts-hotfix``pod-loads-hotfix-marker``rollback-required`
## 禁止误用
- 不得把 ConfigMap/pod 覆盖当作 source truth;正式修复必须回到 source、PR、artifact 和 repo-owned CD。
- 不得把 #462 runbook 或只读审计包装成 #460/#461 源码化完成。
- 不得把 #462 runbook 或只读审计包装成 #464 DEV-LIVE smoke/harness 完成。
- 不得把热修期间的 stdout、trace 或 operation 扩大为 M3 trusted evidenceM3 判定仍按 [m3-loop-rollout-runbook.md](m3-loop-rollout-runbook.md)。
- 不得读取或打印 Secret value、kubeconfig token、DB URL 密码、provider token。
- 不得新增或修改仓库 `reports/**`;过程证据写入 PR/issue/comment,临时 JSON 只能放 `/tmp``.state` 或 CI artifact。
+3 -2
View File
File diff suppressed because one or more lines are too long
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env node
import {
formatDevRuntimeHotfixAuditFailure,
runDevRuntimeHotfixAuditCli
} from "./src/dev-runtime-hotfix-audit.mjs";
try {
process.exitCode = await runDevRuntimeHotfixAuditCli(process.argv.slice(2));
} catch (error) {
process.stdout.write(`${JSON.stringify(formatDevRuntimeHotfixAuditFailure(error), null, 2)}\n`);
process.exitCode = 1;
}
+565
View File
@@ -0,0 +1,565 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
const execFileAsync = promisify(execFile);
const defaultKubeconfig = "/etc/rancher/k3s/k3s.yaml";
const defaultNamespace = "hwlab-dev";
const defaultDeployment = "hwlab-cloud-api";
const defaultConfigMap = "hwlab-cloud-api-code-agent-hotfix";
const defaultMountPath = "/app/internal/cloud/code-agent-chat.mjs";
const defaultSubPath = "code-agent-chat.mjs";
const defaultMarker = "runtime-hotfix-pc-gateway-shell";
const defaultAnnotationKey = "hwlab.pikastech.local/pc-gateway-shell-hotfix";
const defaultPodLabelSelector = "app.kubernetes.io/name=hwlab-cloud-api";
const forbiddenKubectlVerbs = new Set([
"apply",
"patch",
"rollout",
"restart",
"delete",
"create",
"replace",
"scale",
"set",
"edit",
"annotate",
"label"
]);
const allowedKubectlVerbs = new Set(["get", "describe", "exec", "auth", "config", "version"]);
const classificationOrder = Object.freeze([
"hotfix-configmap-present",
"deployment-mounts-hotfix",
"pod-loads-hotfix-marker",
"source-artifact-expected",
"rollback-required",
"no-hotfix-detected",
"unknown-needs-manual-readonly-check"
]);
export async function runDevRuntimeHotfixAuditCli(argv = process.argv.slice(2), options = {}) {
const args = parseArgs(argv);
if (args.help) {
process.stdout.write(`${usage()}\n`);
return 0;
}
const report = await buildDevRuntimeHotfixAuditReport(args, options);
process.stdout.write(`${JSON.stringify(report, null, args.pretty ? 2 : 0)}\n`);
return 0;
}
export function parseArgs(argv = []) {
const args = {
collectReadonly: false,
pretty: false,
timeoutMs: 5000,
kubeconfig: defaultKubeconfig,
namespace: defaultNamespace,
deployment: defaultDeployment,
configMap: defaultConfigMap,
mountPath: defaultMountPath,
subPath: defaultSubPath,
marker: defaultMarker,
annotationKey: defaultAnnotationKey,
podLabelSelector: defaultPodLabelSelector,
help: false
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--collect-readonly") args.collectReadonly = true;
else if (arg === "--plan" || arg === "--dry-run") args.collectReadonly = false;
else if (arg === "--pretty") args.pretty = true;
else if (arg === "--timeout-ms") args.timeoutMs = parsePositiveInt(requireValue(argv[++index], arg), arg);
else if (arg === "--kubeconfig") args.kubeconfig = requireValue(argv[++index], arg);
else if (arg === "--namespace") args.namespace = requireValue(argv[++index], arg);
else if (arg === "--deployment") args.deployment = requireValue(argv[++index], arg);
else if (arg === "--configmap") args.configMap = requireValue(argv[++index], arg);
else if (arg === "--mount-path") args.mountPath = requireValue(argv[++index], arg);
else if (arg === "--sub-path") args.subPath = requireValue(argv[++index], arg);
else if (arg === "--marker") args.marker = requireValue(argv[++index], arg);
else if (arg === "--annotation-key") args.annotationKey = requireValue(argv[++index], arg);
else if (arg === "--pod-label-selector") args.podLabelSelector = requireValue(argv[++index], arg);
else if (arg === "--help" || arg === "-h") args.help = true;
else throw new Error(`unknown argument: ${arg}`);
}
return args;
}
export async function buildDevRuntimeHotfixAuditReport(args = {}, options = {}) {
const parsed = { ...parseArgs([]), ...args };
const report = {
issueRefs: ["pikasTech/HWLAB#462", "pikasTech/HWLAB#465", "pikasTech/HWLAB#7", "pikasTech/HWLAB#239"],
relatedDivisionOfWork: {
capabilityRouting: "pikasTech/HWLAB#460",
sourceRunner: "pikasTech/HWLAB#461",
runtimeHotfixRunbookAndAudit: "pikasTech/HWLAB#462",
serviceIdDiagnostics: "pikasTech/HWLAB#463",
devLiveSmoke: "pikasTech/HWLAB#464"
},
mode: parsed.collectReadonly ? "collect-readonly" : "plan",
target: {
environment: "DEV",
namespace: parsed.namespace,
deployment: parsed.deployment,
configMap: parsed.configMap,
annotationKey: parsed.annotationKey,
mountPath: parsed.mountPath,
subPath: parsed.subPath,
marker: parsed.marker,
kubeconfig: parsed.kubeconfig,
requiredNode: "d601"
},
safety: {
prodAllowed: false,
secretValuesRead: false,
secretResourcesRead: false,
k8sWriteAttempted: false,
k8sWriteCommandsExecuted: [],
defaultExecutesKubectl: parsed.collectReadonly,
allowedKubectlVerbs: [...allowedKubectlVerbs].sort(),
forbiddenKubectlVerbs: [...forbiddenKubectlVerbs].sort()
},
requiredOperatorEvidence: [
"Deployment",
"pod",
"ConfigMap",
"annotation",
"volume",
"mountPath",
"subPath",
"traceId",
"operationId",
"validationCommand",
"rollbackMethod",
"sourceFollowUp"
],
commandPlan: buildCommandPlan(parsed),
checks: null,
classifications: [],
conclusion: null
};
if (!parsed.collectReadonly) {
report.classifications = orderClassifications([
"source-artifact-expected",
"unknown-needs-manual-readonly-check"
]);
report.conclusion = {
status: "unknown",
summary: "Plan mode did not query D601; run --collect-readonly on D601 with the native k3s kubeconfig to classify the current hotfix state."
};
return report;
}
const checks = await collectReadonlyChecks(parsed, options);
report.checks = checks;
report.classifications = classifyHotfixState(checks);
report.conclusion = summarizeClassifications(report.classifications);
return report;
}
async function collectReadonlyChecks(args, options) {
const runCommand = options.runCommand ?? defaultRunCommand;
const nodeProbe = await runKubectl(args, ["get", "nodes", "-o", "json"], runCommand);
const nodeJson = parseJson(nodeProbe);
const nodeNames = (nodeJson?.items ?? []).map((item) => item?.metadata?.name).filter(Boolean);
const configMapProbe = await runKubectl(args, ["-n", args.namespace, "get", "configmap", args.configMap, "-o", "json"], runCommand);
const configMapJson = parseJson(configMapProbe);
const deploymentProbe = await runKubectl(args, ["-n", args.namespace, "get", "deployment", args.deployment, "-o", "json"], runCommand);
const deploymentJson = parseJson(deploymentProbe);
const deployment = summarizeDeployment(deploymentJson, args);
const podsProbe = await runKubectl(args, [
"-n",
args.namespace,
"get",
"pods",
"-l",
args.podLabelSelector,
"--field-selector=status.phase=Running",
"--sort-by=.metadata.creationTimestamp",
"-o",
"json"
], runCommand);
const podsJson = parseJson(podsProbe);
const pod = summarizeSelectedPod(podsJson);
let markerProbe = null;
let syntaxProbe = null;
if (pod?.name) {
markerProbe = await runKubectl(args, [
"-n",
args.namespace,
"exec",
pod.name,
"--",
"sh",
"-lc",
`grep -n ${shellQuote(args.marker)} ${shellQuote(args.mountPath)} | head`
], runCommand);
syntaxProbe = await runKubectl(args, [
"-n",
args.namespace,
"exec",
pod.name,
"--",
"sh",
"-lc",
`node --check ${shellQuote(args.mountPath)}`
], runCommand);
}
return {
node: {
ok: nodeProbe.ok,
command: nodeProbe.command,
names: nodeNames,
requiredNodePresent: nodeNames.includes("d601"),
probe: compactProbe(nodeProbe, { includeStdout: false })
},
configMap: {
command: configMapProbe.command,
present: configMapProbe.ok && Boolean(configMapJson),
unknown: !configMapProbe.ok && !looksNotFound(configMapProbe),
keys: configMapJson ? Object.keys(configMapJson.data ?? {}).sort() : [],
probe: compactProbe(configMapProbe, { includeStdout: false })
},
deployment: {
command: deploymentProbe.command,
present: deploymentProbe.ok && Boolean(deploymentJson),
unknown: !deploymentProbe.ok || !deploymentJson,
...deployment,
probe: compactProbe(deploymentProbe, { includeStdout: false })
},
pod: {
command: podsProbe.command,
selected: pod,
unknown: !podsProbe.ok || !podsJson,
marker: markerProbe ? {
command: markerProbe.command,
found: markerProbe.ok,
matchCount: markerProbe.ok ? markerProbe.stdout.split("\n").filter(Boolean).length : 0,
probe: compactProbe(markerProbe, { includeStdout: false })
} : null,
syntax: syntaxProbe ? {
command: syntaxProbe.command,
ok: syntaxProbe.ok,
probe: compactProbe(syntaxProbe)
} : null,
probe: compactProbe(podsProbe, { includeStdout: false })
}
};
}
function summarizeDeployment(document, args) {
if (!document) {
return {
generation: null,
annotations: {},
hotfixVolumes: [],
hotfixMounts: [],
mountsHotfix: false
};
}
const volumes = document.spec?.template?.spec?.volumes ?? [];
const containers = document.spec?.template?.spec?.containers ?? [];
const hotfixVolumeNames = new Set(
volumes
.filter((volume) => volume?.configMap?.name === args.configMap)
.map((volume) => volume.name)
.filter(Boolean)
);
const hotfixMounts = containers.flatMap((container, containerIndex) =>
(container.volumeMounts ?? [])
.map((mount, mountIndex) => ({ container, containerIndex, mount, mountIndex }))
.filter(({ mount }) =>
hotfixVolumeNames.has(mount.name) ||
mount.mountPath === args.mountPath ||
mount.subPath === args.subPath
)
.map(({ container, containerIndex, mount, mountIndex }) => ({
container: container.name ?? `container-${containerIndex}`,
containerIndex,
mountIndex,
name: mount.name,
mountPath: mount.mountPath ?? null,
subPath: mount.subPath ?? null,
readOnly: mount.readOnly === true
}))
);
const annotations = document.spec?.template?.metadata?.annotations ?? {};
const hotfixAnnotations = Object.fromEntries(
Object.entries(annotations).filter(([key]) => key === args.annotationKey || key.includes("hotfix"))
);
return {
generation: document.metadata?.generation ?? null,
observedImageTags: containers.map((container) => ({
container: container.name,
image: container.image
})),
annotations: hotfixAnnotations,
hotfixVolumes: volumes
.filter((volume) => hotfixVolumeNames.has(volume.name))
.map((volume, volumeIndex) => ({
name: volume.name,
volumeIndex,
configMap: volume.configMap?.name,
keys: (volume.configMap?.items ?? []).map((item) => item.key).filter(Boolean)
})),
hotfixMounts,
mountsHotfix: hotfixMounts.some((mount) =>
mount.mountPath === args.mountPath &&
mount.subPath === args.subPath
)
};
}
function summarizeSelectedPod(document) {
const pods = document?.items ?? [];
if (pods.length === 0) return null;
const pod = pods[pods.length - 1];
return {
name: pod.metadata?.name ?? null,
phase: pod.status?.phase ?? null,
nodeName: pod.spec?.nodeName ?? null,
creationTimestamp: pod.metadata?.creationTimestamp ?? null
};
}
export function classifyHotfixState(checks) {
const classifications = new Set(["source-artifact-expected"]);
const unknown = !checks ||
checks.node?.requiredNodePresent !== true ||
checks.configMap?.unknown === true ||
checks.deployment?.unknown === true ||
checks.pod?.unknown === true;
if (checks?.configMap?.present === true) classifications.add("hotfix-configmap-present");
if (checks?.deployment?.mountsHotfix === true) classifications.add("deployment-mounts-hotfix");
if (checks?.pod?.marker?.found === true) classifications.add("pod-loads-hotfix-marker");
const hotfixDetected = classifications.has("hotfix-configmap-present") ||
classifications.has("deployment-mounts-hotfix") ||
classifications.has("pod-loads-hotfix-marker");
if (hotfixDetected) classifications.add("rollback-required");
if (!hotfixDetected && !unknown) classifications.add("no-hotfix-detected");
if (unknown) classifications.add("unknown-needs-manual-readonly-check");
return orderClassifications([...classifications]);
}
function summarizeClassifications(classifications) {
if (classifications.includes("rollback-required")) {
return {
status: "hotfix-detected",
summary: "DEV runtime still shows hotfix residue or active coverage; remove the override or replace it with a source artifact before treating source/CD as the runtime truth."
};
}
if (classifications.includes("no-hotfix-detected")) {
return {
status: "clear",
summary: "No ConfigMap or Deployment mount hotfix was detected by the read-only audit; keep official fixes in source/PR/artifact/CD."
};
}
return {
status: "unknown",
summary: "The audit could not fully classify current DEV hotfix state; run the listed read-only checks on D601 before rollback or source verification."
};
}
function orderClassifications(values) {
const set = new Set(values);
return classificationOrder.filter((item) => set.has(item));
}
function buildCommandPlan(args) {
return {
readonly: [
`${kubePrefix(args)} kubectl get nodes -o jsonpath='{.items[*].metadata.name}'`,
`${kubePrefix(args)} kubectl -n ${args.namespace} get configmap ${args.configMap} -o json`,
`${kubePrefix(args)} kubectl -n ${args.namespace} get deployment ${args.deployment} -o json`,
`${kubePrefix(args)} kubectl -n ${args.namespace} get pods -l ${shellQuote(args.podLabelSelector)} --field-selector=status.phase=Running --sort-by=.metadata.creationTimestamp -o json`,
`${kubePrefix(args)} kubectl -n ${args.namespace} exec <pod> -- sh -lc ${shellQuote(`grep -n ${shellQuote(args.marker)} ${shellQuote(args.mountPath)} | head`)}`,
`${kubePrefix(args)} kubectl -n ${args.namespace} exec <pod> -- sh -lc ${shellQuote(`node --check ${shellQuote(args.mountPath)}`)}`
],
rollbackNotExecutedByThisScript: [
"Use source/PR/artifact/CD to replace the runtime override whenever possible.",
`If an authorized DEV rollback must remove the override directly, first locate the hotfix volumeMount, volume, and annotation indexes from the read-only deployment JSON for ${args.deployment}.`,
"Then execute the smallest explicit DEV-only rollback outside this audit script, with KUBECONFIG pinned to D601 native k3s, and re-run this audit in --collect-readonly mode."
],
forbiddenDefaultOperations: [
"kubectl apply",
"kubectl patch",
"kubectl rollout",
"kubectl rollout restart",
"kubectl delete",
"kubectl create",
"kubectl set image"
]
};
}
async function runKubectl(args, kubectlArgs, runCommand) {
assertReadonlyKubectlArgs(kubectlArgs);
const displayCommand = `${kubePrefix(args)} kubectl ${kubectlArgs.map(shellDisplay).join(" ")}`;
const result = await runCommand({
command: "env",
args: [`KUBECONFIG=${args.kubeconfig}`, "kubectl", ...kubectlArgs],
timeoutMs: args.timeoutMs,
displayCommand
});
const probe = normalizeCommandResult(result, displayCommand);
Object.defineProperty(probe, "rawStdout", {
value: String(result.stdout ?? "").trim(),
enumerable: false
});
return probe;
}
export function assertReadonlyKubectlArgs(kubectlArgs) {
const verb = kubectlArgs.find((arg) => allowedKubectlVerbs.has(arg) || forbiddenKubectlVerbs.has(arg));
const forbidden = kubectlArgs.find((arg) => forbiddenKubectlVerbs.has(arg));
if (forbidden) {
throw new Error(`kubectl write verb is forbidden in hotfix audit: ${forbidden}`);
}
if (!verb || !allowedKubectlVerbs.has(verb)) {
throw new Error(`kubectl command is not in the read-only allowlist: ${kubectlArgs.join(" ")}`);
}
}
async function defaultRunCommand({ command, args, timeoutMs }) {
try {
const result = await execFileAsync(command, args, {
timeout: timeoutMs,
maxBuffer: 1024 * 1024
});
return {
ok: true,
exitCode: 0,
stdout: result.stdout,
stderr: result.stderr
};
} catch (error) {
return {
ok: false,
exitCode: typeof error.code === "number" ? error.code : 1,
stdout: String(error.stdout ?? ""),
stderr: String(error.stderr ?? ""),
error: error.message
};
}
}
function normalizeCommandResult(result, command) {
return {
ok: result.ok === true || result.exitCode === 0,
command,
exitCode: Number.isInteger(result.exitCode) ? result.exitCode : result.ok === true ? 0 : 1,
stdout: redact(String(result.stdout ?? "").trim()),
stderr: redact(String(result.stderr ?? "").trim()),
error: result.error ? redact(String(result.error)) : undefined
};
}
function compactProbe(probe, { includeStdout = true } = {}) {
return {
ok: probe.ok,
command: probe.command,
exitCode: probe.exitCode,
stdout: includeStdout && probe.stdout.length <= 200 ? probe.stdout : undefined,
stderr: probe.stderr ? oneLine(probe.stderr) : undefined,
error: probe.error ? oneLine(probe.error) : undefined
};
}
function parseJson(probe) {
if (!probe.ok) return null;
const raw = probe.rawStdout ?? probe.stdout;
try {
return raw ? JSON.parse(raw) : null;
} catch {
return null;
}
}
function looksNotFound(probe) {
return /notfound|not found|notfound/i.test(`${probe.stderr ?? ""} ${probe.error ?? ""}`);
}
function requireValue(value, flag) {
if (!value) throw new Error(`${flag} requires a value`);
return value;
}
function parsePositiveInt(value, flag) {
const parsed = Number.parseInt(value, 10);
if (!Number.isInteger(parsed) || parsed <= 0) throw new Error(`${flag} must be a positive integer`);
return parsed;
}
function kubePrefix(args) {
return `KUBECONFIG=${args.kubeconfig}`;
}
function shellQuote(value) {
return `'${String(value).replaceAll("'", "'\\''")}'`;
}
function shellDisplay(value) {
return /\s|["'\\]/u.test(String(value)) ? shellQuote(value) : String(value);
}
function oneLine(value) {
return redact(String(value)).replace(/\s+/g, " ").trim().slice(0, 500);
}
function redact(value) {
return String(value)
.replace(/(bearer\s+)[A-Za-z0-9._~+/=-]+/giu, "$1[REDACTED]")
.replace(/((?:token|password|client-key-data|client-certificate-data|certificate-authority-data)\s*[:=]\s*)\S+/giu, "$1[REDACTED]")
.replace(/(authorization:\s*)\S+/giu, "$1[REDACTED]")
.slice(0, 4000);
}
function usage() {
return [
"Usage: node scripts/dev-runtime-hotfix-audit.mjs [--plan|--collect-readonly] [options]",
"",
"Default mode is --plan: it prints the read-only audit plan and performs no kubectl calls.",
"",
"Options:",
" --collect-readonly Run read-only kubectl get/exec checks; never apply/patch/rollout/restart",
" --plan, --dry-run Print plan only (default)",
" --pretty Pretty-print JSON",
" --timeout-ms <ms> Per kubectl command timeout",
" --kubeconfig <path> Kubeconfig path, default /etc/rancher/k3s/k3s.yaml",
" --namespace <name> Namespace, default hwlab-dev",
" --deployment <name> Deployment, default hwlab-cloud-api",
" --configmap <name> Hotfix ConfigMap name",
" --mount-path <path> Expected mounted file path",
" --sub-path <name> Expected subPath key",
" --marker <text> Runtime marker to grep in the mounted file",
" --annotation-key <key> Deployment hotfix annotation key",
" --pod-label-selector <sel> Pod selector for the Deployment",
" --help Show this help"
].join("\n");
}
export function formatDevRuntimeHotfixAuditFailure(error) {
return {
status: "failed",
script: "dev-runtime-hotfix-audit",
error: {
message: error instanceof Error ? error.message : String(error)
},
safety: {
k8sWriteAttempted: false,
secretValuesRead: false
}
};
}
@@ -0,0 +1,158 @@
import assert from "node:assert/strict";
import test from "node:test";
import {
assertReadonlyKubectlArgs,
buildDevRuntimeHotfixAuditReport,
parseArgs
} from "./dev-runtime-hotfix-audit.mjs";
test("plan mode emits a bounded read-only plan and runs no kubectl commands", async () => {
let commandCount = 0;
const report = await buildDevRuntimeHotfixAuditReport(parseArgs(["--plan"]), {
runCommand: async () => {
commandCount += 1;
throw new Error("plan mode must not execute commands");
}
});
assert.equal(commandCount, 0);
assert.equal(report.mode, "plan");
assert.equal(report.safety.k8sWriteAttempted, false);
assert.deepEqual(report.classifications, [
"source-artifact-expected",
"unknown-needs-manual-readonly-check"
]);
assert.ok(report.commandPlan.readonly.every((command) => command.includes("KUBECONFIG=/etc/rancher/k3s/k3s.yaml")));
assert.ok(report.commandPlan.forbiddenDefaultOperations.some((command) => command.includes("kubectl patch")));
});
test("collect-readonly classifies active ConfigMap mount and pod marker without write verbs", async () => {
const commands = [];
const report = await buildDevRuntimeHotfixAuditReport(parseArgs(["--collect-readonly"]), {
runCommand: async ({ args, displayCommand }) => {
commands.push(displayCommand);
return fixtureResult(args);
}
});
assert.equal(report.mode, "collect-readonly");
assert.equal(report.safety.k8sWriteAttempted, false);
assert.ok(report.classifications.includes("hotfix-configmap-present"));
assert.ok(report.classifications.includes("deployment-mounts-hotfix"));
assert.ok(report.classifications.includes("pod-loads-hotfix-marker"));
assert.ok(report.classifications.includes("rollback-required"));
assert.equal(report.conclusion.status, "hotfix-detected");
assert.ok(commands.length > 0);
assertNoWriteVerbs(commands);
});
test("collect-readonly reports no-hotfix-detected when ConfigMap and mount are absent", async () => {
const report = await buildDevRuntimeHotfixAuditReport(parseArgs(["--collect-readonly"]), {
runCommand: async ({ args }) => fixtureResult(args, { noHotfix: true })
});
assert.deepEqual(report.classifications, [
"source-artifact-expected",
"no-hotfix-detected"
]);
assert.equal(report.conclusion.status, "clear");
});
test("collect-readonly keeps unknown classification when D601 node cannot be proven", async () => {
const report = await buildDevRuntimeHotfixAuditReport(parseArgs(["--collect-readonly"]), {
runCommand: async ({ args }) => {
if (args.includes("nodes")) {
return { ok: false, exitCode: 1, stdout: "", stderr: "unable to connect to the server" };
}
return fixtureResult(args);
}
});
assert.ok(report.classifications.includes("unknown-needs-manual-readonly-check"));
assert.notEqual(report.conclusion.status, "clear");
});
test("kubectl write verbs are rejected by the audit helper", () => {
assert.throws(() => assertReadonlyKubectlArgs(["-n", "hwlab-dev", "patch", "deployment", "hwlab-cloud-api"]), /forbidden/u);
assert.throws(() => assertReadonlyKubectlArgs(["-n", "hwlab-dev", "rollout", "restart", "deployment", "hwlab-cloud-api"]), /forbidden/u);
});
function fixtureResult(args, options = {}) {
const text = args.join(" ");
if (text.includes("get nodes")) {
return okJson({ items: [{ metadata: { name: "d601" } }] });
}
if (text.includes("get configmap")) {
if (options.noHotfix) {
return { ok: false, exitCode: 1, stdout: "", stderr: "Error from server (NotFound): configmaps \"hwlab-cloud-api-code-agent-hotfix\" not found" };
}
return okJson({ data: { "code-agent-chat.mjs": "// runtime-hotfix-pc-gateway-shell" } });
}
if (text.includes("get deployment")) {
return okJson(deploymentFixture({ noHotfix: options.noHotfix }));
}
if (text.includes("get pods")) {
return okJson({
items: [{
metadata: { name: "hwlab-cloud-api-abc", creationTimestamp: "2026-05-24T00:00:00Z" },
status: { phase: "Running" },
spec: { nodeName: "d601" }
}]
});
}
if (text.includes("exec") && text.includes("grep")) {
return options.noHotfix
? { ok: false, exitCode: 1, stdout: "", stderr: "" }
: { ok: true, exitCode: 0, stdout: "1:runtime-hotfix-pc-gateway-shell\n", stderr: "" };
}
if (text.includes("exec") && text.includes("node --check")) {
return { ok: true, exitCode: 0, stdout: "", stderr: "" };
}
throw new Error(`unexpected fixture command: ${text}`);
}
function deploymentFixture({ noHotfix = false } = {}) {
return {
metadata: { generation: 9 },
spec: {
template: {
metadata: {
annotations: noHotfix ? {} : {
"hwlab.pikastech.local/pc-gateway-shell-hotfix": "runtime-hotfix"
}
},
spec: {
volumes: noHotfix ? [] : [{
name: "code-agent-hotfix",
configMap: {
name: "hwlab-cloud-api-code-agent-hotfix",
items: [{ key: "code-agent-chat.mjs", path: "code-agent-chat.mjs" }]
}
}],
containers: [{
name: "hwlab-cloud-api",
image: "127.0.0.1:5000/hwlab/hwlab-cloud-api:dev",
volumeMounts: noHotfix ? [] : [{
name: "code-agent-hotfix",
mountPath: "/app/internal/cloud/code-agent-chat.mjs",
subPath: "code-agent-chat.mjs",
readOnly: true
}]
}]
}
}
}
};
}
function okJson(value) {
return { ok: true, exitCode: 0, stdout: JSON.stringify(value), stderr: "" };
}
function assertNoWriteVerbs(commands) {
const forbidden = /\bkubectl\s+(?:apply|patch|rollout|delete|create|replace|scale|set|edit|annotate|label)\b/u;
for (const command of commands) {
assert.doesNotMatch(command, forbidden, command);
}
}