refactor: primitive G14 CI without Docker fallback

This commit is contained in:
Codex
2026-05-27 00:53:37 +08:00
parent e56fece528
commit 97846207a3
7 changed files with 122 additions and 253 deletions
+11 -11
View File
@@ -102,12 +102,12 @@ The preflight checks:
`registryCapabilities` appears in both the preflight report and the artifact
publish report. The dimensions are:
- `process-http-access`: runner-process HTTP access to `/v2/`. A failed
`127.0.0.1` request can happen while Docker daemon push access still works,
so this dimension is diagnostic and may be `degraded` without blocking
publish.
- `process-http-access`: runner/BuildKit-side HTTP access to `/v2/`. This is
the publish-path registry API preflight for the current BuildKit-based G14
pipeline.
- `docker-daemon-push-access`: read-only Docker daemon evidence for the
local/internal registry target. This is the publish-path capability.
local/internal registry target. This is a legacy diagnostic only; it is not a
publish-path capability and must not block BuildKit publish readiness.
- `k3s-pull-access`: read-only native-k3s `kubectl` evidence for whether the
DEV cluster can inspect image pull state. This is the deploy-path capability.
@@ -115,12 +115,12 @@ publish report. The dimensions are:
`ready` requires all checks to pass and no open blocker.
`degraded` means a non-publish-path capability is observable but incomplete,
such as runner-process loopback HTTP refusing `127.0.0.1:5000/v2/` while Docker
daemon registry visibility is still present. `blocked` is expected until the
real DEV runtime path can be proven. Each blocker includes a `type`, `scope`,
`summary`, and `nextTask` so the next task is the smallest repair needed before
rerunning the preflight.
`degraded` means a non-required capability is observable but incomplete, such as
legacy Docker daemon registry visibility being unavailable after the G14 publish
path has moved to BuildKit. `blocked` is expected until the required real DEV
runtime path can be proven. Each blocker includes a `type`, `scope`, `summary`,
and `nextTask` so the next task is the smallest repair needed before rerunning
the preflight.
The DB gate has two common blocked scopes:
+8 -8
View File
@@ -5,7 +5,7 @@ G14 是 HWLAB 当前 DEV/PROD 原生 k8s 与 GitOps 运行面目标。G14 CI/CD
## 目标模型
- Source of truth:业务版本以 Git source commit 为唯一身份;镜像 tag、OCI labels、runtime annotation 和 Argo CD desired state 都必须记录同一个 source commit。
- CITekton 在 G14 k3s 内运行 `hwlab-g14-ci-image-publish` Pipeline,由 `scripts/g14-gitops-render.mjs` 直接生成原生 task;最小校验固定为 `repo-reports-guard``g14-contract-check``codex-api-forwarder-check`,随后按 component plan 做 per-service BuildKit publish 与 GitOps promote。
- CITekton 在 G14 k3s 内运行 `hwlab-g14-ci-image-publish` Pipeline,由 `scripts/g14-gitops-render.mjs` 直接生成原生 task;最小校验固定为 `repo-reports-guard``g14-contract-check``codex-api-forwarder-check`,随后按 component plan 做 per-service BuildKit publish 与 GitOps promote;没有 `CI.json` runner、DIND 单任务发布或 Docker fallback
- Artifact:镜像使用 commit tag,例如 `127.0.0.1:5000/hwlab/hwlab-cloud-api:<shortCommit>`digest 由 registry 返回,CI report 只作为审计证据,不作为 CD 真相。
- Branch split`G14` 是源码监控分支;`G14-gitops` 是 Tekton promotion 写入的生成分支,保存 `deploy/gitops/g14/**` desired state。
- CDArgo CD 只消费 `G14-gitops:deploy/gitops/g14/runtime-dev``deploy/gitops/g14/runtime-prod` 的 Git desired state,不重新构建镜像,不读取 D601 状态,不获取 legacy DEV CD Lease。
@@ -25,12 +25,12 @@ G14 是 HWLAB 当前 DEV/PROD 原生 k8s 与 GitOps 运行面目标。G14 CI/CD
- 默认输出到 `deploy/gitops/g14/`
- 默认 GitOps 生成分支是 `G14-gitops`Pipeline 成功后把本次 source commit 对应的 `deploy/gitops/g14/**` 推送到该分支,避免把生成提交继续写回 `G14`
- 默认 registry prefix 是 `127.0.0.1:5000/hwlab`,用于 G14 单节点 k3s 的 node-local registry。
- 默认 CI/CD proxy 是 G14 本机 `http://127.0.0.1:10808` / `socks5h://127.0.0.1:10808`。Tekton CI step、Docker-in-Docker sidecar 和 publish step 都注入 proxy/no_proxypublish step 必须先在 DIND 内 `docker pull $HWLAB_DEV_BASE_IMAGE`,再进入 DEV base-image preflight、build 和 registry push,保证 npm、base image 拉取和 registry push 走 G14 本机代理边界
- `prepare-source` 和最小原语校验 task 不允许每次运行时重新 `apk add``apt-get install` 或临时下载 browser/runtime 依赖。当前固定工具镜像是 `127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-v1`,包含 Node 22、npm、Git、OpenSSH、curlPython3 和 Docker CLI。脚本启动时必须输出工具与 proxy preflight 结构化日志;缺少工具时应先重建并推送工具镜像,再修改 `HWLAB_G14_CI_TOOLS_IMAGE`/render 默认值,不能回退到 runtime 安装。
- 默认 CI/CD proxy 是 G14 本机 `http://127.0.0.1:10808` / `socks5h://127.0.0.1:10808`。Tekton CI step、BuildKit sidecar 和 publish step 都注入 proxy/no_proxy服务镜像构建只允许通过 Pod 内 BuildKit Unix socket 直接 push 到 G14 本地 registry,不能回退到 Docker daemon、DIND 或 host Docker
- `prepare-source` 和最小原语校验 task 不允许每次运行时重新 `apk add``apt-get install` 或临时下载 browser/runtime 依赖。当前固定工具镜像是 `127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-v1`,包含 Node 22、npm、Git、OpenSSH、curlPython3。脚本启动时必须输出工具与 proxy preflight 结构化日志;缺少工具时应先重建并推送工具镜像,再修改 `HWLAB_G14_CI_TOOLS_IMAGE`/render 默认值,不能回退到 runtime 安装。
- G14 host 只用于 source workspace、GitOps render、k3s 控制和轻量语法/静态合同检查;不要把 host 当成浏览器执行面。低频 browser smoke 已不再属于默认 primitive CI。若确实需要一次性布局、移动端或交互验证,必须显式在 G14 k3s/Tekton 的专用 Playwright 镜像内运行,不能回退成 host 上强装 browser 的长期方案。
- Poller、control-plane reconciler、image publish 和 GitOps promote step 都不允许每次运行时 `apk add` / `apt-get install`。当前 G14 registry 固定工具镜像是 `127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-v1`,包含 Node 22、npm、Git、OpenSSH、curlPython3 和 Docker CLI;生成的脚本只做 proxy preflight 与工具存在性检查。若需要升级工具,先在 G14 构建/推送新的工具镜像,再修改 `HWLAB_G14_CI_TOOLS_IMAGE`/render 默认值并由 reconciler apply。
- Poller、control-plane reconciler、image publish 和 GitOps promote step 都不允许每次运行时 `apk add` / `apt-get install`。当前 G14 registry 固定工具镜像是 `127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-v1`,包含 Node 22、npm、Git、OpenSSH、curlPython3;生成的脚本只做 proxy preflight 与工具存在性检查。若需要升级工具,先在 G14 构建/推送新的工具镜像,再修改 `HWLAB_G14_CI_TOOLS_IMAGE`/render 默认值并由 reconciler apply。
- 服务镜像构建的默认 parent/base image 不得从 Docker Hub 反复拉取。当前 `node:20-bookworm-slim` 已镜像到 G14 registry 的 allowlist 名称:`127.0.0.1:5000/hwlab/hwlab-node20-base:20-bookworm-slim`render 默认 `base-image` 和 poller `BASE_IMAGE` 都指向这个本地镜像。需要升级 parent image 时,先通过 G14 proxy 拉取并推送到 G14 registry,再修改 `HWLAB_G14_DEV_BASE_IMAGE`/render 默认值;image publish step 只允许从本地 registry pull base image,且 tag 必须符合 publish gate 的 `hwlab-node20-base`/`hwlab-dev-base`/`hwlab-node-runtime-base` allowlist。
- 任何依赖下载阶段都必须有可观测诊断。生成的 Tekton 脚本在 apt/apk、npm、Playwright browser、Docker base image pull 和 GitOps promote 之前输出结构化 `dependency-proxy-probe``dependency-curl-probe``dependency-download-*` 日志,至少包含 phase、目标 URL/镜像、脱敏 proxy、首包耗时、总耗时、下载字节数和速度。CI/CD 卡在下载时,先用这些日志判断是 proxy 不可达、目标源慢、DNS/首包慢还是下载吞吐低,再决定是否切换 G14 代理节点或预热镜像;不能只凭 PipelineRun Running 时长判断业务测试失败。
- 任何依赖下载阶段都必须有可观测诊断。生成的 Tekton 脚本在 npm、BuildKit base image/local registry probe 和 GitOps promote 之前输出结构化 `dependency-proxy-probe``dependency-curl-probe``dependency-download-*` 日志,至少包含 phase、目标 URL/镜像、脱敏 proxy、首包耗时、总耗时、下载字节数和速度。CI/CD 卡在下载时,先用这些日志判断是 proxy 不可达、目标源慢、DNS/首包慢还是下载吞吐低,再决定是否切换 G14 代理节点或预热镜像;不能只凭 PipelineRun Running 时长判断业务测试失败。
常用命令:
@@ -66,11 +66,11 @@ HWLAB 是 monorepoG14 CI/CD 加速必须按组件输入判断构建和滚动
- `scripts/g14-ci-plan.mjs` 是只读 planner,默认读取 `deploy/deploy.json``deploy/artifact-catalog.dev.json``scripts/src/g14-ci-plan-lib.mjs` 内建 component model,输出 `affectedServices``reusedServices``componentCommitId``componentInputHash``dockerfileHash``baseImageDigest``buildArgsHash` 和原因;它不得修改 deploy、catalog 或 GitOps 文件。
- 服务清单兼容顺序固定为:显式 `--services``deploy.services[]``deploy.k3s.serviceMappings[]``internal/protocol.SERVICE_IDS`。因此旧 `deploy/deploy.json` 形态和当前完整 `deploy.services[]` 形态都必须能被 planner 识别。
- 组件边界固定由 `scripts/src/g14-ci-plan-lib.mjs` 的内建 service-path model 定义;如需新增或调整 `componentPaths``sharedPaths``runtimeDeps``buildSystemPaths`,直接修改 planner 库和对应测试,不再额外维护 repo-local commands/forbidden skeleton。
- `scripts/g14-artifact-publish.mjs` 默认启用组件级 lazy build:先运行 planner,再只构建/推送 `affectedServices``reusedServices` 必须从 `deploy/artifact-catalog.dev.json` 复用已有 sha256 digest;如果 catalog 没有可验证 digest,必须结构化阻塞并提示先跑全量 publish。`--full-build` 是兼容回退开关,用于强制构建/发布每个 required service
- `scripts/g14-artifact-publish.mjs` 默认启用组件级 lazy build:先运行 planner,再只构建/推送 `affectedServices``reusedServices` 必须从 `deploy/artifact-catalog.dev.json` 复用已有 sha256 digest;如果 catalog 没有可验证 digest,必须结构化阻塞并修复 catalog 或显式扩大 changed service 范围,不能退回 Docker 或 legacy full-build 路线
- `scripts/g14-artifact-publish.mjs` 的 publish report 必须携带 planner 的 per-service 元数据;`scripts/refresh-artifact-catalog.mjs` 把这些字段复制进 `deploy/artifact-catalog.dev.json``deploy/deploy.json` 的对应 service。旧字段 `commitId``image``imageTag``digest``publishState` 语义不变,旧检查脚本不得因为新增字段失败。
- `scripts/g14-gitops-render.mjs` 默认启用混合 desired stateworkload 的 container image、`HWLAB_IMAGE``HWLAB_IMAGE_TAG` 和 pod template `source-commit` 来自 `deploy.services[]` 的 per-service image/env;全局 GitOps metadata 仍记录本次 source commit。`--legacy-source-images` `HWLAB_G14_USE_DEPLOY_IMAGES=0` 是兼容回退开关,用于把所有 workload image 渲染为同一个 source commit tag。
- `scripts/g14-gitops-render.mjs` 只支持混合 desired stateworkload 的 container image、`HWLAB_IMAGE``HWLAB_IMAGE_TAG` 和 pod template `source-commit` 来自 `deploy.services[]` 的 per-service image/env;全局 GitOps metadata 仍记录本次 source commit。`--legacy-source-images` `HWLAB_G14_USE_DEPLOY_IMAGES=0` 已废弃,不得把所有 workload image 回退渲染为同一个 source commit tag。
- G14 Tekton promotion 在推送 `G14-gitops` 前,必须用 publish report 刷新 workspace 内的 `deploy/deploy.json``deploy/artifact-catalog.dev.json``deploy/k8s/base/workloads.yaml`,并把该 per-service artifact truth 以 `chore: promote G14 GitOps source <short>` 提交回 `G14` source branchpoller 必须跳过这类生成提交,避免自触发循环。这样下一轮 planner 才能从最新 catalog 复用上一轮已构建服务,避免把未变服务回滚到旧 digest。
- 后续 Tekton 并发化只能以 planner 输出作为输入,先做到每个 changed service 一个构建任务;unchanged service 复用 catalog digest,且不能改 pod template,避免无意义 rollout。没有完整 per-service desired state 证据时,必须使用 `--full-build``--legacy-source-images` 回退,而不是猜测复用
- Tekton 并发化只能以 planner 输出作为输入;每个 service 都有独立 TaskRunchanged service 启动 BuildKitunchanged service 只写 reuse result 并复用 catalog digest,且不能改 pod template,避免无意义 rollout。没有完整 per-service desired state 证据时,必须修复 planner/catalog 证据,不能使用 `--full-build``--legacy-source-images`、DIND 或 Docker fallback 回退。
## 加速判定与当前瓶颈
+12 -114
View File
@@ -20,7 +20,6 @@ import {
resolveDevArtifactServices,
serviceInventoryFromServices
} from "./src/dev-artifact-services.mjs";
import { runDevBaseImagePreflight } from "./src/dev-base-image-preflight.mjs";
import { probeRegistryCapabilities } from "./src/registry-capabilities.mjs";
import { inspectCloudWebDistFreshness } from "../web/hwlab-cloud-web/scripts/dist-contract.mjs";
import { ensureNotRepoReportsPath, tempReportPath } from "./src/report-paths.mjs";
@@ -31,7 +30,7 @@ const defaultRegistryPrefix =
process.env.HWLAB_DEV_REGISTRY_PREFIX || process.env.HWLAB_DEV_REGISTRY || "127.0.0.1:5000/hwlab";
const defaultBaseImage = process.env.HWLAB_DEV_BASE_IMAGE || null;
const defaultReportPath = tempReportPath("dev-artifacts.json");
const defaultBuildBackend = process.env.HWLAB_ARTIFACT_BUILD_BACKEND || "buildkit";
const buildBackend = "buildkit";
const defaultBuildkitCommand = process.env.HWLAB_BUILDKIT_BUILDCTL || "buildctl-daemonless.sh";
const defaultBuildkitAddr = process.env.HWLAB_BUILDKIT_ADDR || null;
const catalogPath = "deploy/artifact-catalog.dev.json";
@@ -83,7 +82,7 @@ function parseArgs(argv) {
services: [...SERVICE_IDS],
servicesExplicit: false,
affectedOnly: true,
buildBackend: defaultBuildBackend,
buildBackend,
buildkitCommand: defaultBuildkitCommand,
buildkitAddr: defaultBuildkitAddr,
externalServiceReportDir: null,
@@ -107,8 +106,6 @@ function parseArgs(argv) {
args.servicesExplicit = true;
}
else if (arg === "--affected-only") args.affectedOnly = true;
else if (arg === "--full-build") args.affectedOnly = false;
else if (arg === "--build-backend") args.buildBackend = readOption(argv, ++index, arg);
else if (arg === "--buildkit-command") args.buildkitCommand = readOption(argv, ++index, arg);
else if (arg === "--buildkit-addr") args.buildkitAddr = readOption(argv, ++index, arg);
else if (arg === "--external-service-report-dir") args.externalServiceReportDir = readOption(argv, ++index, arg);
@@ -197,9 +194,7 @@ function printHelp() {
" --registry-prefix PREFIX default: 127.0.0.1:5000/hwlab",
" --base-image IMAGE override HWLAB_DEV_BASE_IMAGE for this run; must already be local",
" --services LIST comma-separated service IDs; default: all frozen DEV services",
" --affected-only default: build/publish only g14-ci-plan affected services and reuse unchanged catalog digests",
" --full-build build/publish every required service even when planner would reuse prior digests",
" --build-backend NAME buildkit only",
" --affected-only default and only supported scope: build/publish affected services and reuse unchanged catalog digests",
" --buildkit-command PATH buildctl-daemonless.sh/buildctl command for buildkit backend",
" --buildkit-addr ADDR optional buildctl --addr endpoint for a sidecar BuildKit daemon",
" --external-service-report-dir DIR collect affected service artifacts from per-service publish reports",
@@ -207,7 +202,7 @@ function printHelp() {
" --tekton-results-dir DIR write single-service build result files for Tekton results",
` --report PATH default: ${defaultReportPath}`,
" --no-report print JSON without updating the temporary artifact JSON",
" --quiet-build keep docker build quiet; default keeps build output visible",
" --quiet-build keep BuildKit progress quiet; default keeps build output visible",
" --concurrency N parallel service build/push workers; default: 4, max: 8"
].join("\n"));
}
@@ -230,16 +225,6 @@ function blocker({ type, scope, summary, next }) {
};
}
function preflightEnvForBaseImage(baseImage) {
if (!baseImage) {
return process.env;
}
return {
...process.env,
HWLAB_DEV_BASE_IMAGE: baseImage
};
}
function buildkitBaseImagePreflight(baseImage) {
const requestedImage = baseImage || defaultBaseImage || "127.0.0.1:5000/hwlab/hwlab-node20-base:20-bookworm-slim";
return {
@@ -337,10 +322,6 @@ function dockerfileQuote(value) {
return JSON.stringify(String(value));
}
function validBuildBackend(value) {
return value === "buildkit";
}
function buildkitCacheRef(registryPrefix, serviceId) {
return `${registryPrefix}/cache/${serviceId}`;
}
@@ -1021,12 +1002,6 @@ function imageRepository(registryPrefix, serviceId) {
return `${registryPrefix}/${serviceId}`;
}
function inspectDigest(pushOutput) {
const matches = [...pushOutput.matchAll(/digest:\s+(sha256:[a-f0-9]{64})/giu)];
if (!matches.length) return null;
return matches[matches.length - 1][1];
}
function tailText(value, maxLength = 2500) {
const text = value.trim();
if (text.length <= maxLength) return text;
@@ -1045,7 +1020,7 @@ function sourceStateBlocker(service) {
async function preflight({ args, services, catalog, deployManifest, baseImagePreflight, registryCapabilities, buildServiceIds = new Set() }) {
const blockers = [];
let registryPrefix = args.registryPrefix;
const buildRequired = (args.mode === "build" || args.mode === "publish") && (buildServiceIds.size > 0 || !args.affectedOnly);
const buildRequired = (args.mode === "build" || args.mode === "publish") && buildServiceIds.size > 0;
if (baseImagePreflight.publishUsable) {
args.baseImage = baseImagePreflight.localTag;
@@ -1093,7 +1068,7 @@ async function preflight({ args, services, catalog, deployManifest, baseImagePre
}
}
if (buildRequired && args.buildBackend === "buildkit") {
if (buildRequired) {
const buildkitPreflight = await buildkitCommandPreflight(args.buildkitCommand);
if (!buildkitPreflight.ok) {
blockers.push(
@@ -1178,7 +1153,7 @@ async function buildService({ args, repo, commitId, shortCommit, service, buildC
blocker: blocker({
type: "environment_blocker",
scope: service.serviceId,
summary: "cloud web dist build failed before Docker build",
summary: "cloud web dist build failed before BuildKit image build",
next: "Run node web/hwlab-cloud-web/scripts/build.mjs, fix the static asset build, then rerun --publish."
}),
logTail: tailText(`${build.stdout}\n${build.stderr}`),
@@ -1200,7 +1175,7 @@ async function buildService({ args, repo, commitId, shortCommit, service, buildC
blocker: blocker({
type: "contract_blocker",
scope: service.serviceId,
summary: `cloud web dist is stale or incomplete before Docker build: ${distFreshness.mismatches.join(", ")}`,
summary: `cloud web dist is stale or incomplete before BuildKit image build: ${distFreshness.mismatches.join(", ")}`,
next: "Run node web/hwlab-cloud-web/scripts/build.mjs and node web/hwlab-cloud-web/scripts/check.mjs before publishing hwlab-cloud-web."
})
};
@@ -1379,54 +1354,6 @@ async function runWithInput(command, args, input) {
};
}
async function publishService(artifact) {
if (artifact.status !== "built") {
return artifact;
}
const result = await run("docker", ["push", artifact.image]);
if (result.code !== 0) {
return {
...artifact,
status: "publish_failed",
blocker: blocker({
type: "network_blocker",
scope: artifact.serviceId,
summary: `docker push failed for ${artifact.image}`,
next: "Verify the G14 local/internal registry is reachable from the Docker daemon and rerun --publish."
}),
logTail: tailText(`${result.stdout}\n${result.stderr}`),
publishDurationMs: result.durationMs
};
}
const digest = inspectDigest(`${result.stdout}\n${result.stderr}`);
if (!digest) {
return {
...artifact,
status: "published_unverified_digest",
digest: "not_published",
blocker: blocker({
type: "environment_blocker",
scope: artifact.serviceId,
summary: `docker push completed for ${artifact.image} but did not return an immutable digest`,
next: "Verify registry digest reporting, then rerun --publish; do not update the catalog with a digest until a sha256 registry digest is observed."
}),
pushLogTail: tailText(`${result.stdout}\n${result.stderr}`, 1200),
publishDurationMs: result.durationMs
};
}
return {
...artifact,
status: "published",
digest,
repositoryDigest: `${repositoryFromImageRef(artifact.image)}@${digest}`,
pushLogTail: tailText(`${result.stdout}\n${result.stderr}`, 1200),
publishDurationMs: result.durationMs
};
}
function repositoryFromImageRef(image) {
const lastColon = image.lastIndexOf(":");
return lastColon === -1 ? image : image.slice(0, lastColon);
@@ -1736,7 +1663,6 @@ function publishPlanFor({ args, commitId, shortCommit, mode, artifacts, fatalBlo
repositoryDigest: artifact.repositoryDigest ?? null,
digestStatus: /^sha256:[a-f0-9]{64}$/u.test(digest) ? "real" : "placeholder",
status: artifact.status,
dockerBuildDurationMs: artifact.dockerBuildDurationMs ?? null,
cloudWebBuildDurationMs: artifact.cloudWebBuildDurationMs ?? null,
publishDurationMs: artifact.publishDurationMs ?? null,
componentCommitId: artifact.componentCommitId ?? null,
@@ -1926,7 +1852,6 @@ function createReport({ args, repo, commitId, shortCommit, mode, services, artif
publishedCount,
reusedCount,
timings: {
dockerBuildDurationMs: sumDurations(artifacts, "dockerBuildDurationMs"),
buildkitBuildDurationMs: sumDurations(artifacts, "buildkitBuildDurationMs"),
cloudWebBuildDurationMs: sumDurations(artifacts, "cloudWebBuildDurationMs"),
publishDurationMs: sumDurations(artifacts, "publishDurationMs")
@@ -1953,7 +1878,6 @@ function createReport({ args, repo, commitId, shortCommit, mode, services, artif
artifactScope: artifact.artifactScope,
notPublishedReason: artifactNotPublishedReason(artifact, mode, fatalBlocked),
localImageId: artifact.localImageId,
dockerBuildDurationMs: artifact.dockerBuildDurationMs ?? null,
buildkitBuildDurationMs: artifact.buildkitBuildDurationMs ?? null,
buildBackend: artifact.buildBackend ?? args.buildBackend,
buildkitCacheRef: artifact.buildkitCacheRef ?? null,
@@ -1969,7 +1893,6 @@ function createReport({ args, repo, commitId, shortCommit, mode, services, artif
ciReason: artifact.ciReason ?? [],
reuse: artifact.reuse ?? null,
reusedFrom: artifact.reusedFrom ?? null,
dockerBuildLogTail: artifact.dockerBuildLogTail ?? null,
logTail: artifact.logTail ?? null,
pushLogTail: artifact.pushLogTail ?? null
})),
@@ -1991,9 +1914,6 @@ async function main() {
printHelp();
return;
}
if (!validBuildBackend(args.buildBackend)) {
throw new Error(`unsupported build backend ${args.buildBackend}; expected buildkit`);
}
if (args.mode === "publish") {
const identityGuard = requireG14ArtifactPublishIdentity({
script: cliEntrypoint,
@@ -2008,11 +1928,7 @@ async function main() {
args.registryPrefix = validateRegistryPrefix(args.registryPrefix);
const [baseImagePreflight, registryCapabilities] = await Promise.all([
args.buildBackend === "buildkit"
? buildkitBaseImagePreflight(args.baseImage)
: runDevBaseImagePreflight({
env: preflightEnvForBaseImage(args.baseImage)
}),
buildkitBaseImagePreflight(args.baseImage),
probeRegistryCapabilities({
registryPrefix: args.registryPrefix
})
@@ -2041,9 +1957,7 @@ async function main() {
const catalogByServiceId = new Map((catalog?.services ?? []).map((service) => [service.serviceId, service]));
const affectedServiceIds = new Set(ciPlan?.affectedServices ?? []);
const buildServiceIds = new Set(
args.affectedOnly
? services.filter((service) => service.artifactRequired && affectedServiceIds.has(service.serviceId)).map((service) => service.serviceId)
: services.filter((service) => service.artifactRequired).map((service) => service.serviceId)
services.filter((service) => service.artifactRequired && affectedServiceIds.has(service.serviceId)).map((service) => service.serviceId)
);
const preflightBuildServiceIds = args.externalServiceReportDir || args.externalServiceResultsEnv ? new Set() : buildServiceIds;
@@ -2056,7 +1970,7 @@ async function main() {
status: service.artifactRequired ? "preflight_only" : "not_required"
}));
let externalArtifacts = new Map();
if (args.affectedOnly && (args.mode === "build" || args.mode === "publish")) {
if (args.mode === "build" || args.mode === "publish") {
if (args.externalServiceReportDir && buildServiceIds.size > 0) {
const externalResult = await readExternalServiceArtifacts({
dir: args.externalServiceReportDir,
@@ -2104,7 +2018,7 @@ async function main() {
type: "environment_blocker",
scope: service.serviceId,
summary: `${service.serviceId} cannot be reused because deploy/artifact-catalog.dev.json has no verified digest`,
next: "Run the default full publish once, or refresh the service catalog from a successful publish report before using --affected-only."
next: "Refresh the service catalog with a verified sha256 digest for this service, or create a service-scoped source change so the planner publishes it through the per-service BuildKit task."
}));
}
return reused;
@@ -2126,25 +2040,10 @@ async function main() {
serviceId: service.serviceId,
status: artifact.status,
image: artifact.image,
dockerBuildDurationMs: artifact.dockerBuildDurationMs ?? null,
buildkitBuildDurationMs: artifact.buildkitBuildDurationMs ?? null,
buildBackend: artifact.buildBackend ?? args.buildBackend,
cloudWebBuildDurationMs: artifact.cloudWebBuildDurationMs ?? null
});
if (args.mode === "publish" && artifact.status === "built") {
emit("publish_start", { serviceId: service.serviceId, image: artifact.image });
const published = await publishService(artifact);
artifactByServiceId.set(service.serviceId, published);
if (published.blocker) blockers.push(published.blocker);
emit("publish_done", {
serviceId: service.serviceId,
status: published.status,
image: published.image,
digest: published.digest,
publishDurationMs: published.publishDurationMs ?? null
});
}
});
artifacts = services.map((service) => artifactByServiceId.get(service.serviceId));
}
@@ -2184,7 +2083,6 @@ async function main() {
buildCreatedAt: service.buildCreatedAt,
digest: service.digest,
distFreshness: service.distFreshness,
dockerBuildDurationMs: service.dockerBuildDurationMs,
buildkitBuildDurationMs: service.buildkitBuildDurationMs,
buildBackend: service.buildBackend,
buildkitCacheRef: service.buildkitCacheRef,
+69 -100
View File
@@ -24,7 +24,6 @@ const defaultNoProxy = process.env.HWLAB_G14_NO_PROXY || "hyueapi.com,.hyueapi.c
const ciToolsRunnerImage = process.env.HWLAB_G14_CI_TOOLS_IMAGE || "127.0.0.1:5000/hwlab/hwlab-ci-node-tools:node22-alpine-v1";
const buildkitRunnerImage = process.env.HWLAB_G14_BUILDKIT_IMAGE || "moby/buildkit:rootless";
const defaultDevBaseImage = process.env.HWLAB_G14_DEV_BASE_IMAGE || "127.0.0.1:5000/hwlab/hwlab-node20-base:20-bookworm-slim";
const tektonSingleImagePublish = process.env.HWLAB_G14_TEKTON_SINGLE_IMAGE_PUBLISH === "1";
const defaultServiceIds = Object.freeze([
"hwlab-cloud-api",
"hwlab-cloud-web",
@@ -61,6 +60,66 @@ const primitiveValidationTasks = Object.freeze([
"render_check_revision=$(node --input-type=module -e \"import fs from 'node:fs'; const data = JSON.parse(fs.readFileSync('deploy/gitops/g14/source.json', 'utf8')); process.stdout.write(data.sourceCommit);\")",
"test -n \"$render_check_revision\"",
"node scripts/g14-gitops-render.mjs --check --source-revision \"$render_check_revision\"",
`node --input-type=module <<'NODE'
import { readdirSync, readFileSync, statSync } from "node:fs";
import path from "node:path";
const root = process.cwd();
const ciFileName = "CI" + ".json";
const forbiddenCiFragments = [
"ci" + "-json",
"HWLAB_G14_TEKTON" + "_SINGLE_IMAGE_PUBLISH",
"single" + "-dind",
"docker" + ":29",
"DOCKER" + "_HOST",
"Docker" + "-in-Docker",
"D" + "IND",
"docker" + " push",
"--build" + "-backend",
"HWLAB_ARTIFACT" + "_BUILD_BACKEND",
"--legacy" + "-source-images",
"HWLAB_G14_USE_DEPLOY_IMAGES" + "=0"
];
const scanTargets = [
"scripts/g14-gitops-render.mjs",
"scripts/artifact-publish.mjs",
"scripts/g14-artifact-publish.mjs",
"scripts/src/g14-ci-plan-lib.mjs",
"deploy/gitops/g14/tekton"
];
const skipDirs = new Set([".git", "node_modules", ".worktree"]);
function walkFiles(relativePath) {
const absolutePath = path.join(root, relativePath);
let stat;
try {
stat = statSync(absolutePath);
} catch {
return [];
}
if (stat.isFile()) return [relativePath];
if (!stat.isDirectory()) return [];
const files = [];
for (const entry of readdirSync(absolutePath, { withFileTypes: true })) {
if (entry.isDirectory() && skipDirs.has(entry.name)) continue;
files.push(...walkFiles(path.join(relativePath, entry.name)));
}
return files;
}
const ciFiles = walkFiles(".").filter((filePath) => path.basename(filePath) === ciFileName);
const violations = [];
for (const filePath of scanTargets.flatMap(walkFiles)) {
const text = readFileSync(path.join(root, filePath), "utf8");
for (const fragment of forbiddenCiFragments) {
if (text.includes(fragment)) violations.push({ filePath, fragment });
}
}
if (ciFiles.length || violations.length) {
console.error(JSON.stringify({ status: "failed", ciFiles, violations }, null, 2));
process.exit(1);
}
NODE`,
"node --check scripts/artifact-publish.mjs",
"node --check scripts/g14-artifact-publish.mjs"
]
@@ -100,7 +159,7 @@ function parseArgs(argv) {
webEndpoint: defaultWebEndpoint,
prodRuntimeEndpoint: defaultProdRuntimeEndpoint,
prodWebEndpoint: defaultProdWebEndpoint,
useDeployImages: process.env.HWLAB_G14_USE_DEPLOY_IMAGES !== "0",
useDeployImages: true,
check: false,
write: true,
help: false
@@ -118,7 +177,6 @@ function parseArgs(argv) {
else if (arg === "--prod-runtime-endpoint") args.prodRuntimeEndpoint = readOption(argv, ++index, arg);
else if (arg === "--prod-web-endpoint") args.prodWebEndpoint = readOption(argv, ++index, arg);
else if (arg === "--use-deploy-images") args.useDeployImages = true;
else if (arg === "--legacy-source-images") args.useDeployImages = false;
else if (arg === "--check") {
args.check = true;
args.write = false;
@@ -152,8 +210,7 @@ function usage() {
` --web-endpoint URL default: ${defaultWebEndpoint}`,
` --prod-runtime-endpoint URL default: ${defaultProdRuntimeEndpoint}`,
` --prod-web-endpoint URL default: ${defaultProdWebEndpoint}`,
" --use-deploy-images default: render workload images/env from deploy/deploy.json per service",
" --legacy-source-images render every workload with the source revision tag instead of per-service desired images",
" --use-deploy-images default and only supported mode: render workload images/env from deploy/deploy.json per service",
" --check verify generated files are current without writing",
" --no-write plan and print generated file list without writing"
].join("\n");
@@ -669,39 +726,6 @@ function primitiveValidationTask(task) {
};
}
function publishScript() {
return `#!/bin/sh
set -eu
${dependencyProxyProbeScript("image-publish-proxy-preflight", "http://deb.debian.org/debian/dists/bookworm/InRelease")}
echo '{"event":"ci-base-image","phase":"image-publish","image":"${ciToolsRunnerImage}","policy":"no-runtime-apk"}'
for tool in node npm git python3 ssh docker; do command -v "$tool" >/dev/null 2>&1 || { echo '{"event":"ci-base-image","phase":"image-publish","ok":false,"reason":"missing-tool","tool":"'"$tool"'"}'; exit 31; }; done
export DOCKER_HOST=unix:///var/run/docker.sock
export HWLAB_CI_ARTIFACT_RUN_ID="$(context.pipelineRun.name)"
export HWLAB_CI_ARTIFACT_RUN_OWNER="tekton"
export HWLAB_DEV_REGISTRY_PREFIX="$(params.registry-prefix)"
export HWLAB_DEV_REGISTRY_K8S_NATIVE=1
if [ -n "$(params.base-image)" ]; then export HWLAB_DEV_BASE_IMAGE="$(params.base-image)"; fi
for i in $(seq 1 90); do docker info >/dev/null 2>&1 && break; sleep 2; done
docker info >/dev/null
if [ -n "\${HWLAB_DEV_BASE_IMAGE:-}" ]; then
echo '{"event":"dependency-local-registry-probe-start","phase":"image-publish-pre-base-image","target":"http://127.0.0.1:5000/v2/"}'
registry_started_at="$(date +%s)"
curl -fsS --max-time 10 http://127.0.0.1:5000/v2/ >/dev/null
registry_finished_at="$(date +%s)"
echo '{"event":"dependency-local-registry-probe","phase":"image-publish-pre-base-image","ok":true,"durationSeconds":'"$((registry_finished_at - registry_started_at))"'}'
started_at="$(date +%s)"
echo '{"event":"dependency-download-start","phase":"image-publish-docker-pull","image":"'"$HWLAB_DEV_BASE_IMAGE"'"}'
docker pull "$HWLAB_DEV_BASE_IMAGE"
finished_at="$(date +%s)"
echo '{"event":"dependency-download-complete","phase":"image-publish-docker-pull","image":"'"$HWLAB_DEV_BASE_IMAGE"'","durationSeconds":'"$((finished_at - started_at))"'}'
fi
cd /workspace/source/repo
test "$(git rev-parse HEAD)" = "$(params.revision)"
node scripts/g14-artifact-publish.mjs --publish --registry-prefix "$(params.registry-prefix)" --report /workspace/source/dev-artifacts.json --quiet-build --concurrency "$(params.concurrency)" --services "$(params.services)"
node scripts/refresh-artifact-catalog.mjs --target-ref HEAD --publish-report /workspace/source/dev-artifacts.json --no-write
`;
}
function planArtifactsScript() {
return `#!/bin/sh
set -eu
@@ -801,7 +825,7 @@ if [ "$buildkit_ready" != "1" ]; then
echo '{"event":"buildkit-sidecar-not-ready","serviceId":"'"$(params.service-id)"'","addr":"'"$HWLAB_BUILDKIT_ADDR"'"}' >&2
exit 1
fi
node scripts/g14-artifact-publish.mjs --publish --registry-prefix "$(params.registry-prefix)" --no-report --tekton-results-dir /tekton/results --quiet-build --concurrency 1 --services "$(params.service-id)" --build-backend buildkit --buildkit-command /workspace/buildkit-bin/buildctl --buildkit-addr "$HWLAB_BUILDKIT_ADDR"
node scripts/g14-artifact-publish.mjs --publish --registry-prefix "$(params.registry-prefix)" --no-report --tekton-results-dir /tekton/results --quiet-build --concurrency 1 --services "$(params.service-id)" --buildkit-command /workspace/buildkit-bin/buildctl --buildkit-addr "$HWLAB_BUILDKIT_ADDR"
`;
}
@@ -1101,8 +1125,7 @@ const pipelineRun = {
{ name: "revision", value: revision },
{ name: "registry-prefix", value: env("REGISTRY_PREFIX", "127.0.0.1:5000/hwlab") },
{ name: "services", value: env("SERVICES", "${defaultServicesParam}") },
{ name: "base-image", value: env("BASE_IMAGE", "${defaultDevBaseImage}") },
{ name: "concurrency", value: env("CONCURRENCY", "4") }
{ name: "base-image", value: env("BASE_IMAGE", "${defaultDevBaseImage}") }
],
workspaces: [
{ name: "source", volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: "8Gi" } } } } },
@@ -1290,56 +1313,6 @@ function buildkitSidecar() {
};
}
function dockerRunVolume() {
return { name: "docker-run", emptyDir: {} };
}
function dockerSidecar() {
return {
name: "docker",
image: "docker:29-dind",
command: ["dockerd"],
args: ["--host=unix:///var/run/docker.sock", "--storage-driver=vfs"],
env: [{ name: "DOCKER_TLS_CERTDIR", value: "" }, ...proxyEnv()],
securityContext: { privileged: true },
volumeMounts: [{ name: "docker-run", mountPath: "/var/run" }]
};
}
function legacyImagePublishTask() {
return {
name: "image-publish",
runAfter: primitiveValidationTaskNames(),
workspaces: [{ name: "source", workspace: "source" }],
taskSpec: {
params: [
{ name: "revision" },
{ name: "registry-prefix" },
{ name: "services" },
{ name: "base-image" },
{ name: "concurrency" }
],
workspaces: [{ name: "source" }],
volumes: [dockerRunVolume()],
sidecars: [dockerSidecar()],
steps: [{
name: "publish",
image: ciToolsRunnerImage,
env: [{ name: "DOCKER_HOST", value: "unix:///var/run/docker.sock" }, ...proxyEnv()],
volumeMounts: [{ name: "docker-run", mountPath: "/var/run" }],
script: publishScript()
}]
},
params: [
{ name: "revision", value: "$(params.revision)" },
{ name: "registry-prefix", value: "$(params.registry-prefix)" },
{ name: "services", value: "$(params.services)" },
{ name: "base-image", value: "$(params.base-image)" },
{ name: "concurrency", value: "$(params.concurrency)" }
]
};
}
function planArtifactsTask() {
return {
name: "plan-artifacts",
@@ -1432,7 +1405,6 @@ function collectArtifactsTask() {
}
function imagePublishTaskSet() {
if (tektonSingleImagePublish) return [legacyImagePublishTask()];
return [
planArtifactsTask(),
...defaultServiceIds.map(perServiceBuildTask),
@@ -1454,7 +1426,7 @@ function tektonPipeline() {
annotations: {
"hwlab.pikastech.local/source-config": "scripts/g14-gitops-render.mjs#tekton-native-primitive-ci",
"hwlab.pikastech.local/ci-contract": "tekton-native-primitive-tasks",
"hwlab.pikastech.local/policy": tektonSingleImagePublish ? "single-dind-image-publish-compat" : "native-per-service-taskrun-image-publish"
"hwlab.pikastech.local/policy": "native-per-service-taskrun-image-publish"
}
},
spec: {
@@ -1465,8 +1437,7 @@ function tektonPipeline() {
{ name: "revision", type: "string", description: "Full git commit SHA; the only source truth for image tags." },
{ name: "registry-prefix", type: "string", default: defaultRegistryPrefix },
{ name: "services", type: "string", default: defaultServicesParam },
{ name: "base-image", type: "string", default: defaultDevBaseImage },
{ name: "concurrency", type: "string", default: "4" }
{ name: "base-image", type: "string", default: defaultDevBaseImage }
],
workspaces: [
{ name: "source" },
@@ -1498,7 +1469,7 @@ function tektonPipeline() {
...imagePublishTaskSet(),
{
name: "gitops-promote",
runAfter: [tektonSingleImagePublish ? "image-publish" : "collect-artifacts"],
runAfter: ["collect-artifacts"],
workspaces: [
{ name: "source", workspace: "source" },
{ name: "git-ssh", workspace: "git-ssh" }
@@ -1556,8 +1527,7 @@ function tektonPipelineRunTemplate({ source, args }) {
{ name: "gitops-branch", value: args.gitopsBranch },
{ name: "revision", value: source.full },
{ name: "registry-prefix", value: args.registryPrefix },
{ name: "base-image", value: defaultDevBaseImage },
{ name: "concurrency", value: "4" }
{ name: "base-image", value: defaultDevBaseImage }
],
workspaces: [
{ name: "source", volumeClaimTemplate: { spec: { accessModes: ["ReadWriteOnce"], resources: { requests: { storage: "8Gi" } } } } },
@@ -1616,8 +1586,7 @@ function tektonPollerCronJob(args) {
{ name: "GITOPS_BRANCH", value: args.gitopsBranch },
{ name: "REGISTRY_PREFIX", value: args.registryPrefix },
{ name: "SERVICES", value: defaultServicesParam },
{ name: "BASE_IMAGE", value: defaultDevBaseImage },
{ name: "CONCURRENCY", value: "4" }
{ name: "BASE_IMAGE", value: defaultDevBaseImage }
],
volumeMounts: [{ name: "git-ssh", mountPath: "/workspace/git-ssh", readOnly: true }],
command: ["/bin/sh", "-c"],
+3 -3
View File
@@ -739,17 +739,17 @@ function validateRegistryCapabilities(reporter, registryCapabilities) {
"registry-capability-classification",
"registry",
registryCapabilities.classification,
`Registry capability classification is ${registryCapabilities.classification}: process HTTP, Docker daemon push path, and k3s pull path are reported separately.`,
`Registry capability classification is ${registryCapabilities.classification}: BuildKit publish registry API, legacy Docker diagnostic, and k3s pull path are reported separately.`,
[registryCapabilities]
);
const dockerDaemonPushAccess = registryCapabilities.dimensions.dockerDaemonPushAccess;
if (dockerDaemonPushAccess.status === "blocked") {
if (dockerDaemonPushAccess.requiredForPublish && dockerDaemonPushAccess.status === "blocked") {
reporter.block({
type: "network_blocker",
scope: "docker-daemon-push-access",
summary: dockerDaemonPushAccess.summary,
nextTask: "Restore Docker daemon access to the D601 local/internal registry before claiming DEV artifact publish readiness."
nextTask: "Restore the required publish-path registry access before claiming DEV artifact publish readiness."
});
}
+1 -1
View File
@@ -158,7 +158,7 @@ export async function createG14CiPlan(options = {}) {
plannerMutatesDeployJson: false,
serviceIdSource: serviceIdResolution.source,
defaultComponentLazyBuild: true,
legacyFullBuildFallback: true
fullBuildFallback: false
},
inputs: {
registryPrefix,
+18 -16
View File
@@ -157,13 +157,13 @@ async function probeProcessHttpAccess(registryPrefix, timeoutMs) {
return {
id: "process-http-access",
status: passed ? "pass" : "degraded",
requiredForPublish: false,
requiredForPublish: true,
requiredForDeploy: false,
endpoint,
probeKind: "runner-process-http-v2",
summary: passed
? `Runner process can reach ${endpoint}.`
: `Runner process cannot reach ${endpoint}; this is process HTTP access only and does not prove Docker daemon push failure.`,
? `Runner or BuildKit-side process can reach ${endpoint}.`
: `Runner process cannot reach ${endpoint}; BuildKit publish may not be able to reach the registry from this execution surface.`,
evidence: {
probe,
publishFailure: false
@@ -176,12 +176,12 @@ async function probeDockerDaemonPushAccess(registryPrefix, timeoutMs) {
if (!dockerVersion.ok) {
return {
id: "docker-daemon-push-access",
status: "blocked",
requiredForPublish: true,
status: "degraded",
requiredForPublish: false,
requiredForDeploy: false,
endpoint: registryPrefix,
probeKind: "docker-cli-read-only",
summary: "Docker CLI is not available, so Docker daemon push access cannot be assessed.",
summary: "Docker CLI is not available; legacy Docker daemon registry visibility is diagnostic only and is not required for BuildKit publish.",
evidence: {
dockerVersion: summarizeCommand(dockerVersion),
pushAttempted: false,
@@ -194,12 +194,12 @@ async function probeDockerDaemonPushAccess(registryPrefix, timeoutMs) {
if (!dockerPs.ok) {
return {
id: "docker-daemon-push-access",
status: "blocked",
requiredForPublish: true,
status: "degraded",
requiredForPublish: false,
requiredForDeploy: false,
endpoint: registryPrefix,
probeKind: "docker-ps-read-only",
summary: "Docker daemon is not readable, so Docker daemon push access cannot be assessed.",
summary: "Docker daemon is not readable; legacy Docker daemon registry visibility is diagnostic only and is not required for BuildKit publish.",
evidence: {
dockerVersion: summarizeCommand(dockerVersion),
dockerPs: summarizeCommand(dockerPs),
@@ -217,16 +217,16 @@ async function probeDockerDaemonPushAccess(registryPrefix, timeoutMs) {
const registryAccepted = registryVisible || k8sNativeLoopbackRegistry;
return {
id: "docker-daemon-push-access",
status: registryAccepted ? "pass" : "blocked",
requiredForPublish: true,
status: registryAccepted ? "pass" : "degraded",
requiredForPublish: false,
requiredForDeploy: false,
endpoint: registryPrefix,
probeKind: "docker-ps-read-only",
summary: registryVisible
? "Docker daemon can see a local/internal registry target for publish preflight purposes; no push was attempted."
? "Docker daemon can see a local/internal registry target; this legacy diagnostic did not push and is not required for BuildKit publish."
: (k8sNativeLoopbackRegistry
? "Docker daemon did not show the registry container, but HWLAB_DEV_REGISTRY_K8S_NATIVE=1 declares the loopback registry is owned by Kubernetes and reachable through the node network."
: "Docker daemon did not show a registry container publishing port 5000."),
: "Docker daemon did not show a registry container publishing port 5000; this no longer blocks the BuildKit publish path."),
evidence: {
dockerVersion: summarizeCommand(dockerVersion),
registryPrefix,
@@ -364,7 +364,9 @@ async function probeK3sPullAccess(registryPrefix, timeoutMs) {
}
function capabilityClassification(dimensions) {
const values = Object.values(dimensions).map((dimension) => dimension.status);
const values = Object.values(dimensions)
.filter((dimension) => dimension.requiredForPublish || dimension.requiredForDeploy)
.map((dimension) => dimension.status);
if (values.includes("blocked")) return "blocked";
if (values.includes("degraded")) return "degraded";
return "pass";
@@ -392,8 +394,8 @@ export async function probeRegistryCapabilities({
classification: capabilityClassification(dimensions),
dimensions,
interpretation: {
processHttpAccess: "Runner process HTTP access to /v2/ is diagnostic only and must not be treated as Docker daemon push access.",
dockerDaemonPushAccess: "Docker daemon push access is the publish-path capability. This probe is read-only and does not push.",
processHttpAccess: "Runner/BuildKit-side process HTTP access to /v2/ is the publish-path registry API preflight.",
dockerDaemonPushAccess: "Docker daemon registry visibility is a legacy read-only diagnostic only. It is not required for the BuildKit publish path and does not push.",
k3sPullAccess: "k3s pull access is the deploy-path capability. This probe is read-only and does not create pods or read secrets."
}
};