fix(cicd): 恢复 PaC 公共状态观察器

This commit is contained in:
Codex
2026-07-16 03:11:03 +02:00
parent b21431aa9d
commit a0491e9dc5
7 changed files with 226 additions and 64 deletions
+10
View File
@@ -116,6 +116,11 @@ bun scripts/cli.ts hwlab nodes control-plane legacy-cicd --help
- 事故处理中曾在故障基线上推进共享 Repository 多分支改造,`#2223``#2226` 的不完整实现分别通过 `#2225``#2228` 回退;误合并的无关 PR `#2227` 通过 `#2229` 回退,不得让事故恢复夹带无关语义;
- 基线恢复后,再通过独立 PR `#2230``#2231``#2233` 完整实现共享 Repository、多分支参数、release Argo 自动同步和资源镜像 ref,并分别通过正常自动事件与 release canary 验收;
- 本案例的固定结论是:生产自动链故障的第一目标是缩小变更面并恢复已知健康状态,不是让当前失败变更通过更多补丁勉强工作。
- PaC 公共 status renderer 故障必须先区分交付失败与观察失败:
- 先用 PipelineRun、Argo 与 runtime 独立事实确认自动交付是否成功,不得把 `parse-failure` 或 status timeout 直接表述为交付链失败;
- 引用缺失变量导致非法 JSON 时,优先从最后已知正确实现恢复该赋值,不得整体回退并撤销仍然正确的 fail-closed 业务语义;
- `status` 不得把慢 TaskRun 日志作为同步必需项,必须复用一次 TaskRun 读取并输出有界阶段标记;精确日志证据留给 id-specific `history``debug-step`
- PipelineRun 成功、Argo `Synced/Healthy` 且 runtime ready 时,同步日志证据延后只能是 `blocking=false` warning,不能制造 false outage;详细案例见 [references/incident-recovery.md](references/incident-recovery.md)。
- PK01 的 CI/CD 默认只允许只读诊断:
- 普通 PR merge、自动滚动、环境恢复或跨服务交付不构成 PK01 变更授权;
- 只有用户在当前请求中明确要求修改 PK01,才允许提交或推广会改变 PK01 版本、配置、Secret 绑定、边缘路由、容器或运行面状态的 source 变更。
@@ -164,6 +169,11 @@ bun scripts/cli.ts hwlab nodes control-plane legacy-cicd --help
- 仍要求 PipelineRun 成功、原 Argo `Synced/Healthy`、runtime ready、sha256 digest pin 与 health ready
- 证据缺失、冲突或 source commit 不匹配必须 fail-closed,不得等价为 delivery disabled,也不得伪造新 artifact、GitOps revision 或 runtime source commit。
- PaC `status|closeout|history``--json` 是有界 machine contract`--full` 展开单条详情,`--raw` 才展开目标侧原始 payload。`history --id --full` 不得重复嵌套同一 rows/consumer payload或依赖 `/tmp/unidesk-cli-output` 才可读。
- PaC `status` 必须在 `observability.readOnlyCapture.timeoutMs` 内完成:
- 目标侧通过 `pac-status-progress` 披露最后阶段,CLI 将其投影为 `observation.remoteStage`
- PipelineRun 与 TaskRun 原始对象各只读取一次并复用,禁止 status 为同一 run 串行读取每个 Pod 日志;
- status 的 TaskRun 日志证据模式固定为 `taskrun-status-only`,需要日志时使用 id-specific 下钻;
- 不得通过提高 YAML timeout 掩盖重复采集、同步日志或无界远端查询。
- PaC `debug-step --id <pipelinerun>` 必须读取目标侧真实 PipelineRun、TaskRun condition/result 与合同 Task 日志:
- 默认输出每个 terminal role 的 presence、status、source 与首个断点;
- 同时披露 artifact catalog、digest、GitOps commit 和 collector 读错;
@@ -107,3 +107,39 @@
- 先恢复最后已知健康状态;
- 再做独立前向修复;
- 每个 PR 必须单一职责、可独立验证、可独立回退。
## 案例:PaC status 公共 renderer 生成非法 JSON
- 现象分层:
- SelfMedia PR `#82` 合并后 `14s` 自动触发 PipelineRun
- PipelineRun 对 source commit `8c03bcca554457315bc16bad01e5a96764276472` 执行成功,耗时 `256s`
- Argo 与 runtime 正常,`pac-read-parse-failure` 只发生在共享 status 观察器;
- 因此这是公共观察面事故,不是 SelfMedia 私有 YAML 或自动交付失败。
- 直接根因:
- `80ca187c` 中 `bootstrap_ready` 的赋值原本正确;
- `1f3283ff` 在回退 admission 业务阻断时删除该赋值;
- `2b385f08` 恢复 consumer bootstrap fail-closed 引用,却没有恢复赋值;
- shell 最终拼出非法 `"ok":,`,影响所有复用共享 renderer 的 PaC consumer。
- 回退决策:
- 不整体回退 `2b385f08`,因为整体回退会再次丢失正确的 bootstrap fail-closed 语义;
- 只从最后已知正确实现恢复 `bootstrap_ready` 赋值,并实际执行三种 bootstrap fixture 后 `JSON.parse` 输出;
- 这是对致因遗漏的最小语义回退,不是在错误基础上继续扩展 admission 设计。
- 恢复期间暴露的第二断点:
- status 在 `25000ms` YAML 预算内超时,阶段标记锁定在 `artifact-evidence`
- 共享 collector 对同一 TaskRun 重复读取,并为成功 run 同步串行执行 `kubectl logs`
- NC01 上同一条有界日志读取实测需要 `18s` 到 `38.5s`,不能作为 status 同步必需项。
- 公共面修复:
- status 只读取一次 TaskRun JSON,并在 task summary 与 artifact terminal record 间复用;
- status 使用 `taskrun-status-only`,不再同步读取 Pod 日志;
- history collector 收敛为一次 selector 聚合,保留失败日志下钻;
- 目标侧输出无敏感值的 `pac-status-progress`CLI 在 timeout observation 中显示 `remoteStage`
- PipelineRun 成功、Argo `Synced/Healthy` 且 runtime ready 时,缺少同步日志证据降级为 `pac-artifact-log-evidence-deferred` 非阻断告警。
- 验收结论:
- 原 `status --target NC01 --consumer selfmedia-nc01 --json` 在 `14.5s` 内返回合法 JSON
- 输出 `ok=true`、`remoteStage=complete`、PipelineRun `Succeeded`、Argo `Synced/Healthy`、runtime `1/1`
- 全程未修改 SelfMedia 私有 YAML,未人工执行 PipelineRun、mirror、trigger、sync、flush、Argo sync 或 runtime patch。
- 长期结论:
- 先区分 delivery failure 与 observer failure
- 修复共享 renderer 必须取得公共服务面授权;
- 先恢复最后已知正确语义,再删除重复、无界和同步日志读取;
- status 负责快速结构化摘要,id-specific 入口负责慢日志证据。
@@ -5,7 +5,7 @@
- MDTODO`R4.1.1`
- GitHub issue`pikasTech/unidesk#2039`
- 状态:`in_progress`
- 原因:源码修复定向验证已完成,尚未合并 PR,也未执行合并后的 NC01 原入口运行面验收。
- 原因:公共 status renderer 源码修复定向验证与合并前 NC01 canary 已完成,尚未合并 PR,也未执行合并后的原入口验收。
## 问题事实
@@ -56,7 +56,7 @@
- `sh -n scripts/src/platform-infra-pipelines-as-code-remote.sh`
- `bun test scripts/src/platform-infra-pac-provenance.test.ts scripts/src/platform-infra-pipelines-as-code-bootstrap.test.ts`
- 结果:`23 pass, 0 fail, 239 expect()`
- 历史结果:`23 pass, 0 fail, 239 expect()`
- `node` 调用 `runPacStatusFixtureChecks()`
- 结果:`ok=true, checks=34`
- `bun scripts/cli.ts platform-infra pipelines-as-code plan --target NC01`
@@ -76,8 +76,8 @@
## 当前阻塞
- 无源码定向验证阻塞。
- 运行面验收仍等待 PR 合并及主代理执行受控 bootstrap,因此任务保持 `in_progress`
- 无源码定向验证或合并前 canary 阻塞。
- 仍等待本次公共 status renderer PR 合并及合并后原入口 canary,因此任务保持 `in_progress`
## 启动 backlog 后续
@@ -103,3 +103,42 @@
- `bun scripts/cli.ts platform-infra pipelines-as-code plan --target NC01 --consumer hwlab-nc01-v03 --json`:成功披露 watcher probes`warnings=[]`
- `git diff --check`:通过。
- 合并后只允许由 PaC migrated consumer 的正常 PR merge 自动链触发交付;不得人工执行 PipelineRun、mirror、trigger、sync、flush、Argo sync、runtime patch 或删除历史业务对象。
## 公共 status renderer 事故
- SelfMedia 自动交付事实:
- PR `#82` 合并后 `14s` 自动触发 PipelineRun
- source commit 为 `8c03bcca554457315bc16bad01e5a96764276472`
- PipelineRun `Succeeded`,耗时 `256s`Argo 与 runtime 正常;
- 因此 `pac-read-parse-failure` 是公共观察面事故,不是 SelfMedia 私有 YAML 或自动交付失败。
- 源码溯源:
- `80ca187c``bootstrap_ready` 赋值正确;
- `1f3283ff` 在回退 admission 业务阻断时删除赋值;
- `2b385f08` 恢复 consumer bootstrap fail-closed 引用,但漏恢复赋值;
- renderer 最终生成非法 `"ok":,`,影响所有复用共享脚本的 PaC consumer。
- 最小语义回退:
- 不整体回退 `2b385f08`,避免再次丢失正确的 bootstrap fail-closed 语义;
- 只恢复最后已知正确的 `bootstrap_ready` 赋值;
- 实际执行未配置、ready 和 not-ready 三种 fixture,并对最终 JSON 调用 `JSON.parse`
- 第二断点与恢复:
- 原 status 连续在 YAML `25000ms` 预算超时;
- `pac-status-progress` 将最后阶段锁定为 `artifact-evidence`
- collector 对同一 TaskRun 重复读取,并同步拉取成功 run 日志;
- 同一 NC01 日志查询实测耗时约 `18s``38.5s`
- 公共面修复:
- status 只读取一次 TaskRun JSON,并由 task summary 与 artifact terminal record 复用;
- status 使用 `taskrun-status-only`,不再同步拉 Pod 日志;
- history 日志采集收敛为一次 selector 聚合,精确日志继续由 id-specific 入口承担;
- CLI 将最后阶段投影为 `observation.remoteStage`
- PipelineRun 成功、Argo `Synced/Healthy` 且 runtime ready 时,缺少同步日志证据降级为 `pac-artifact-log-evidence-deferred` 非阻断告警;
- 默认 compact JSON 删除静态 capabilities 冗余,保持在 CLI 输出预算内。
- 合并前 canary
- `bun scripts/cli.ts platform-infra pipelines-as-code status --target NC01 --consumer selfmedia-nc01 --json`
- 结果:`ok=true``remoteStage=complete`,远端读取约 `14.5s`
- PipelineRun `Succeeded`Argo `Synced/Healthy`runtime `1/1`
- 全程未修改 SelfMedia 私有 YAML,未人工执行 PipelineRun、mirror、trigger、sync、flush、Argo sync 或 runtime patch。
- 本轮验证:
- bootstrap 与 failure evidence 定向测试:`23 pass, 0 fail, 156 expect()`
- 加入 provenance 测试后:`33 pass, 1 fail`
- 唯一失败是旧 consumer 列表断言缺少现有 `agentrun-nc01-release``pikaoa-nc01`,与本次补丁无关,未扩大范围修改;
- `sh -n``git diff --check` 与 skill `quick_validate.py` 均通过。
@@ -61,7 +61,7 @@ describe("PaC 失败证据合同", () => {
action: "history",
targetId: "NC01",
consumerId: "agentrun-nc01-v02",
result: { exitCode: 124, stdout: "", stderr: "UNIDESK_SSH_RUNTIME_TIMEOUT" },
result: { exitCode: 124, stdout: "", stderr: '{"event":"pac-status-progress","stage":"artifact-evidence","valuesPrinted":false}\nUNIDESK_SSH_RUNTIME_TIMEOUT' },
parsed: null,
elapsedMs: 25001,
timeoutMs: 25000,
@@ -74,6 +74,7 @@ describe("PaC 失败证据合同", () => {
target: "NC01",
consumer: "agentrun-nc01-v02",
stage: "history-remote-observe",
remoteStage: "artifact-evidence",
elapsedMs: 25001,
timeoutMs: 25000,
configPath: "config/platform-infra/pipelines-as-code.yaml#observability.readOnlyCapture.timeoutMs",
@@ -156,10 +157,18 @@ describe("PaC 失败证据合同", () => {
expect(JSON.stringify(observation)).not.toContain("process.env");
});
test("collector 遍历全部 matching TaskRunterminal record 仍限定合同任务", () => {
test("collector 以单次有界 selector 拉取 PipelineRun 日志terminal record 仍限定合同任务", () => {
const remote = readFileSync(resolve(import.meta.dir, "platform-infra-pipelines-as-code-remote.sh"), "utf8");
expect(remote).toContain("for (const item of matching)");
expect(remote).toContain("const contractTaskRuns = matching.filter((item) => taskTerminalRecord(item) !== null)");
expect(remote).toContain("'--tail=320', '--max-log-requests=20'");
expect(remote.match(/cp\.spawnSync\('kubectl', \['-n', namespace, 'logs'/gu)).toHaveLength(2);
expect(remote).toContain("if (process.env.UNIDESK_PAC_ACTION !== 'status') {");
expect(remote).toContain("let mode = 'taskrun-status-only';");
expect(remote).toContain('payload_file="${UNIDESK_PAC_TASKRUN_FILE:-}"');
expect(remote).toContain('task_file="${UNIDESK_PAC_TASKRUN_FILE:-}"');
expect(remote).toContain('export UNIDESK_PAC_TASKRUN_FILE="$task_file"');
expect(remote).not.toContain("for (const item of matching)");
expect(remote).not.toContain("for (const item of taskRuns)");
expect(remote).toContain("Object.keys(record(row.firstBreak)).length > 0");
expect(remote).toContain("traceId: row.traceId || null");
});
@@ -1,4 +1,5 @@
import { expect, test } from "bun:test";
import { spawnSync } from "node:child_process";
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
import type { GiteaConfig, GiteaMirrorRepository } from "./platform-infra-gitea-config";
@@ -216,11 +217,36 @@ test("PaC apply checks the Gitea repository before dry-run return and cluster mu
expect(remote).toContain('"admissionProvenance":%s');
expect(remote).toContain('"error":"gitea-repository-missing"');
const statusAction = remote.slice(remote.indexOf("status_action() {"), remote.indexOf("history_action() {"));
const bootstrapReadyAssignment = statusAction.split("\n").find((line) => line.includes("bootstrap_ready=$("));
expect(bootstrapReadyAssignment).toBeDefined();
expect(remote).toContain("status_progress() {");
expect(statusAction).toContain("status_progress artifact-evidence");
expect(statusAction).toContain("status_progress diagnostics");
expect(statusAction).toContain("status_progress complete");
expect(statusAction).toContain("output.code === 'pac-gitops-missing'");
expect(statusAction).toContain("output.code = 'pac-artifact-log-evidence-deferred'");
expect(statusAction).toContain("blocking: false");
expect(statusAction).toContain('[ "$bootstrap_ready" = "true" ]');
expect(statusAction).toContain("output.code = 'pac-consumer-bootstrap-not-ready'");
expect(statusAction).toContain("output.ok = false");
expect(statusAction).toContain("code: 'pac-admission-provenance-not-ready'");
expect(statusAction).toContain("blocking: false");
for (const fixture of [
{ state: { configured: false, ready: null }, expected: true },
{ state: { configured: true, ready: true }, expected: true },
{ state: { configured: true, ready: false }, expected: false },
]) {
const smoke = spawnSync("sh", ["-eu", "-c", [
"consumer_bootstrap=$UNIDESK_PAC_BOOTSTRAP_STATE",
bootstrapReadyAssignment!.trim(),
`printf '{"ok":%s}\\n' "$( [ "$bootstrap_ready" = "true" ] && echo true || echo false )"`,
].join("\n")], {
encoding: "utf8",
env: { ...process.env, UNIDESK_PAC_BOOTSTRAP_STATE: JSON.stringify(fixture.state) },
});
expect(smoke).toMatchObject({ status: 0, stderr: "" });
expect(JSON.parse(smoke.stdout)).toEqual({ ok: fixture.expected });
}
});
test("PaC release manifest renders YAML-owned watcher startup backlog probes", () => {
@@ -777,33 +777,15 @@ function envReuseForPipelineRun(namespace, name, taskRuns) {
const terminalRecords = contractTaskRuns.map(taskTerminalRecord).filter(Boolean);
const logErrors = [];
let logs = '';
for (const item of matching) {
const podName = item.status?.podName;
if (!podName) {
logErrors.push({ taskRun: item.metadata?.name || null, reason: 'pod-name-missing' });
continue;
}
const result = cp.spawnSync('kubectl', ['-n', namespace, 'logs', podName, '--all-containers=true'], {
encoding: 'utf8',
timeout: 12000,
maxBuffer: 4 * 1024 * 1024,
});
if (result.error || result.status !== 0) {
logErrors.push({ taskRun: item.metadata?.name || null, reason: result.error ? String(result.error.message || result.error) : String(result.stderr || '').slice(0, 240) });
continue;
}
logs += `${result.stdout || ''}\n`;
}
if (contractTaskRuns.length === 0) {
state.source = 'pipeline-run-log-fallback';
const result = cp.spawnSync('kubectl', ['-n', namespace, 'logs', '-l', `tekton.dev/pipelineRun=${name}`, '--all-containers=true', '--tail=320'], {
encoding: 'utf8',
timeout: 12000,
maxBuffer: 2 * 1024 * 1024,
});
if (result.error || result.status !== 0) {
return { ...state, source: 'logs-unavailable', collector: { mode: state.source, contractTaskCount: 0, terminalRecordCount: 0, logErrorCount: 1, valuesPrinted: false } };
}
const result = cp.spawnSync('kubectl', ['-n', namespace, 'logs', '-l', `tekton.dev/pipelineRun=${name}`, '--all-containers=true', '--tail=320', '--max-log-requests=20'], {
encoding: 'utf8',
timeout: 12000,
maxBuffer: 4 * 1024 * 1024,
});
if (result.error || result.status !== 0) {
state.source = 'logs-unavailable';
logErrors.push({ taskRun: null, reason: result.error ? String(result.error.message || result.error) : String(result.stderr || '').slice(0, 240) });
} else {
logs = result.stdout || '';
}
const lines = logs.split(/\r?\n/u);
@@ -1085,8 +1067,13 @@ NODE
}
task_rows() {
payload_file=$(mktemp)
kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" get taskrun -o json >"$payload_file" 2>/dev/null || printf '{"items":[]}' >"$payload_file"
payload_file="${UNIDESK_PAC_TASKRUN_FILE:-}"
remove_payload=false
if [ -z "$payload_file" ]; then
payload_file=$(mktemp)
remove_payload=true
kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" get taskrun -o json >"$payload_file" 2>/dev/null || printf '{"items":[]}' >"$payload_file"
fi
node - "$payload_file" <<'NODE'
const fs = require('node:fs');
const input = fs.readFileSync(process.argv[2], 'utf8') || '{"items":[]}';
@@ -1127,7 +1114,7 @@ const rows = (data.items || [])
});
process.stdout.write(JSON.stringify(rows));
NODE
rm -f "$payload_file"
if [ "$remove_payload" = true ]; then rm -f "$payload_file"; fi
}
artifact_summary() {
@@ -1135,8 +1122,13 @@ artifact_summary() {
printf '{}'
return
fi
task_file=$(mktemp)
kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" get taskrun -l "tekton.dev/pipelineRun=$UNIDESK_PAC_TARGET_PIPELINERUN" -o json >"$task_file" 2>/dev/null || printf '{"items":[]}' >"$task_file"
task_file="${UNIDESK_PAC_TASKRUN_FILE:-}"
remove_task_file=false
if [ -z "$task_file" ]; then
task_file=$(mktemp)
remove_task_file=true
kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" get taskrun -l "tekton.dev/pipelineRun=$UNIDESK_PAC_TARGET_PIPELINERUN" -o json >"$task_file" 2>/dev/null || printf '{"items":[]}' >"$task_file"
fi
node - "$task_file" <<'NODE'
const fs = require('node:fs');
const cp = require('node:child_process');
@@ -1149,33 +1141,19 @@ const contractTaskRuns = taskRuns.filter((item) => taskTerminalRecord(item) !==
const terminalRecords = contractTaskRuns.map(taskTerminalRecord).filter(Boolean);
const logErrors = [];
let logText = '';
for (const item of taskRuns) {
const podName = item.status?.podName;
if (!podName) {
logErrors.push({ taskRun: item.metadata?.name || null, reason: 'pod-name-missing' });
continue;
}
const result = cp.spawnSync('kubectl', ['-n', namespace, 'logs', podName, '--all-containers=true'], {
let mode = 'taskrun-status-only';
if (process.env.UNIDESK_PAC_ACTION !== 'status') {
mode = 'pipeline-run-logs';
const result = cp.spawnSync('kubectl', ['-n', namespace, 'logs', '-l', `tekton.dev/pipelineRun=${pipelineRun}`, '--all-containers=true', '--tail=320', '--max-log-requests=20'], {
encoding: 'utf8',
timeout: 12000,
maxBuffer: 4 * 1024 * 1024,
});
if (result.error || result.status !== 0) {
logErrors.push({ taskRun: item.metadata?.name || null, reason: result.error ? String(result.error.message || result.error) : String(result.stderr || '').slice(0, 240) });
continue;
if (!result.error && result.status === 0) logText = result.stdout || '';
else {
mode = 'logs-unavailable';
logErrors.push({ taskRun: null, reason: result.error ? String(result.error.message || result.error) : String(result.stderr || '').slice(0, 240) });
}
logText += `${result.stdout || ''}\n`;
}
let mode = 'pipeline-task-logs';
if (contractTaskRuns.length === 0) {
mode = 'pipeline-run-log-fallback';
const fallback = cp.spawnSync('kubectl', ['-n', namespace, 'logs', '-l', `tekton.dev/pipelineRun=${pipelineRun}`, '--all-containers=true', '--tail=320'], {
encoding: 'utf8',
timeout: 12000,
maxBuffer: 2 * 1024 * 1024,
});
if (!fallback.error && fallback.status === 0) logText = fallback.stdout || '';
else logErrors.push({ taskRun: null, reason: fallback.error ? String(fallback.error.message || fallback.error) : String(fallback.stderr || '').slice(0, 240) });
}
const records = [...parsePacLogRecords(logText), ...terminalRecords];
const out = extractPacArtifactEvidence(records, logText);
@@ -1216,7 +1194,7 @@ out.collector = {
};
process.stdout.write(JSON.stringify(out));
NODE
rm -f "$task_file"
if [ "$remove_task_file" = true ]; then rm -f "$task_file"; fi
}
runtime_summary() {
@@ -1694,20 +1672,43 @@ NODE
rm -f "$sa_file" "$role_binding_file" "$argo_secret_file"
}
status_progress() {
printf '{"event":"pac-status-progress","stage":"%s","valuesPrinted":false}\n' "$1" >&2
}
status_action() {
status_progress control-plane
crd=$(kubectl get crd repositories.pipelinesascode.tekton.dev -o name 2>/dev/null || true)
controller_ready=$(kubectl -n "$UNIDESK_PAC_RELEASE_NAMESPACE" get deploy "$UNIDESK_PAC_CONTROLLER_SERVICE_NAME" -o jsonpath='{.status.readyReplicas}/{.spec.replicas}' 2>/dev/null || echo "0/0")
repository_condition=$(condition_status "$UNIDESK_PAC_TARGET_NAMESPACE" repository "$UNIDESK_PAC_REPOSITORY_NAME")
status_progress admission-provenance
prepare_admission_provenance_state
status_progress consumer-bootstrap
consumer_bootstrap=$(consumer_bootstrap_state)
bootstrap_ready=$(UNIDESK_PAC_BOOTSTRAP_STATE="$consumer_bootstrap" node -e 'const s=JSON.parse(process.env.UNIDESK_PAC_BOOTSTRAP_STATE||"{}"); process.stdout.write(s.configured===true&&s.ready!==true?"false":"true")')
status_progress pipeline-runs
pipelines=$(pipeline_rows)
latest=$(printf '%s' "$pipelines" | node -e 'const fs=require("fs"); const a=JSON.parse(fs.readFileSync(0,"utf8")||"[]"); process.stdout.write(a[0]?.name||"")')
export UNIDESK_PAC_TARGET_PIPELINERUN="$latest"
status_progress task-runs
task_file=$(mktemp)
if [ -n "$latest" ]; then
kubectl -n "$UNIDESK_PAC_TARGET_NAMESPACE" get taskrun -l "tekton.dev/pipelineRun=$latest" -o json >"$task_file" 2>/dev/null || printf '{"items":[]}' >"$task_file"
else
printf '{"items":[]}' >"$task_file"
fi
export UNIDESK_PAC_TASKRUN_FILE="$task_file"
tasks=$(task_rows)
status_progress artifact-evidence
artifact=$(json_normalize "$(artifact_summary)")
unset UNIDESK_PAC_TASKRUN_FILE
rm -f "$task_file"
status_progress webhooks
hooks=$(hook_summary)
status_progress argo
argo=$(kubectl -n "$UNIDESK_PAC_ARGO_NAMESPACE" get application "$UNIDESK_PAC_ARGO_APPLICATION" -o json 2>/dev/null | node -e 'const fs=require("fs"); const s=fs.readFileSync(0,"utf8").trim(); if(!s){process.stdout.write("{}"); process.exit(0)} const a=JSON.parse(s); const source=a.spec?.source || a.spec?.sources?.[0] || {}; let repoURL=source.repoURL||null; try { const url=new URL(repoURL); url.username=""; url.password=""; repoURL=url.toString(); } catch {} process.stdout.write(JSON.stringify({sync:a.status?.sync?.status||null, health:a.status?.health?.status||null, revision:a.status?.sync?.revision||null, repoURL, targetRevision:source.targetRevision||null}))' || echo '{}')
argo=$(json_normalize "$argo")
status_progress gitops-revision
revision_relation=$(gitops_revision_relation "$artifact" "$argo")
artifact=$(UNIDESK_PAC_ARTIFACT_JSON="$artifact" UNIDESK_PAC_REVISION_RELATION_JSON="$revision_relation" node <<'NODE'
const artifact = JSON.parse(process.env.UNIDESK_PAC_ARTIFACT_JSON || '{}');
@@ -1721,13 +1722,41 @@ const revisionRelation = JSON.parse(process.env.UNIDESK_PAC_REVISION_RELATION_JS
process.stdout.write(JSON.stringify({ ...argo, revisionRelation }));
NODE
)
status_progress runtime
runtime=$(json_normalize "$(runtime_summary)")
status_progress diagnostics
diagnostics=$(cicd_diagnostics "$pipelines" "$artifact" "$argo" "$runtime")
diagnostics=$(UNIDESK_PAC_DIAGNOSTICS="$diagnostics" UNIDESK_PAC_STATE="$UNIDESK_PAC_ADMISSION_STATE_JSON" UNIDESK_PAC_BOOTSTRAP_STATE="$consumer_bootstrap" node <<'NODE'
diagnostics=$(UNIDESK_PAC_DIAGNOSTICS="$diagnostics" UNIDESK_PAC_STATE="$UNIDESK_PAC_ADMISSION_STATE_JSON" UNIDESK_PAC_BOOTSTRAP_STATE="$consumer_bootstrap" UNIDESK_PAC_PIPELINES_JSON="$pipelines" UNIDESK_PAC_ARTIFACT_JSON="$artifact" UNIDESK_PAC_ARGO_JSON="$argo" UNIDESK_PAC_RUNTIME_JSON="$runtime" node <<'NODE'
const diagnostics = JSON.parse(process.env.UNIDESK_PAC_DIAGNOSTICS || '{}');
const admissionProvenance = JSON.parse(process.env.UNIDESK_PAC_STATE || '{}');
const consumerBootstrap = JSON.parse(process.env.UNIDESK_PAC_BOOTSTRAP_STATE || '{}');
const pipelineRuns = JSON.parse(process.env.UNIDESK_PAC_PIPELINES_JSON || '[]');
const artifact = JSON.parse(process.env.UNIDESK_PAC_ARTIFACT_JSON || '{}');
const argo = JSON.parse(process.env.UNIDESK_PAC_ARGO_JSON || '{}');
const runtime = JSON.parse(process.env.UNIDESK_PAC_RUNTIME_JSON || '{}');
const output = { ...diagnostics, admissionProvenance, consumerBootstrap, valuesPrinted: false };
const latest = Array.isArray(pipelineRuns) ? pipelineRuns[0] || {} : {};
const runtimeReady = Number.isInteger(runtime.readyReplicas)
&& Number.isInteger(runtime.replicas)
&& runtime.replicas > 0
&& runtime.readyReplicas >= runtime.replicas;
if (output.code === 'pac-gitops-missing'
&& artifact.collector?.mode === 'taskrun-status-only'
&& latest.status === 'True'
&& argo.sync === 'Synced'
&& argo.health === 'Healthy'
&& runtimeReady) {
output.ok = true;
output.code = 'pac-artifact-log-evidence-deferred';
output.phase = 'read-only-evidence-deferred';
output.hint = 'PipelineRun, Argo, and runtime are ready; use id-specific history when synchronous TaskRun log evidence is required';
output.warnings = [...(Array.isArray(output.warnings) ? output.warnings : []), {
code: 'pac-artifact-log-evidence-deferred',
blocking: false,
pipelineRun: latest.name || null,
valuesPrinted: false,
}];
}
if (process.env.UNIDESK_PAC_DELIVERY_PROVENANCE_REQUIRED === '1' && admissionProvenance.ready !== true) {
output.warnings = [...(Array.isArray(output.warnings) ? output.warnings : []), {
code: 'pac-admission-provenance-not-ready',
@@ -1745,6 +1774,7 @@ if (consumerBootstrap.configured === true && consumerBootstrap.ready !== true) {
process.stdout.write(JSON.stringify(output));
NODE
)
status_progress complete
printf '{"ok":%s,"crdPresent":%s,"controllerReady":"%s","admissionProvenance":%s,"consumerBootstrap":%s,"repository":{"name":"%s","repo":"%s/%s","url":"%s","condition":"%s"},"repositoryCondition":"%s","webhooks":%s,"pipelineRuns":%s,"taskRuns":%s,"artifact":%s,"argo":%s,"runtime":%s,"diagnostics":%s,"valuesPrinted":false}\n' \
"$( [ -n "$crd" ] && [ "$controller_ready" != "0/0" ] && [ "$bootstrap_ready" = "true" ] && echo true || echo false )" \
"$( [ -n "$crd" ] && echo true || echo false )" \
@@ -308,6 +308,7 @@ interface PacReadOnlyCaptureObservation {
readonly target: string;
readonly consumer: string;
readonly stage: string;
readonly remoteStage: string | null;
readonly elapsedMs: number;
readonly timeoutMs: number;
readonly configPath: string;
@@ -1922,12 +1923,24 @@ export function pacReadOnlyCaptureObservation(options: {
? "parse-failure"
: "succeeded";
const firstBreakCode = status === "succeeded" ? "none" : `pac-read-${status}`;
const remoteStage = options.result.stderr.split(/\r?\n/u).reduce<string | null>((last, line) => {
if (!line.startsWith("{")) return last;
try {
const event = JSON.parse(line) as Record<string, unknown>;
return event.event === "pac-status-progress" && typeof event.stage === "string" && /^[a-z0-9-]{1,80}$/u.test(event.stage)
? event.stage
: last;
} catch {
return last;
}
}, null);
return {
ok: status === "succeeded",
status,
target: options.targetId,
consumer: options.consumerId,
stage: `${options.action}-remote-observe`,
remoteStage,
elapsedMs: options.elapsedMs,
timeoutMs: options.timeoutMs,
configPath: options.configPath,
@@ -2595,7 +2608,6 @@ function compactStatusJson(result: Record<string, unknown>): Record<string, unkn
argoApplication: consumer.argoApplication,
},
deliveryAuthority: result.deliveryAuthority,
capabilities: record(result.config).capabilities,
warnings: result.warnings,
deliveryBudget: result.deliveryBudget,
observation: result.observation,