Merge pull request #2800 from pikasTech/fix/2744-global-git-mirror-retire
refactor: 退役剩余 git mirror consumer
This commit is contained in:
@@ -299,6 +299,7 @@ bun scripts/cli.ts hwlab nodes control-plane legacy-cicd --help
|
||||
- source branch、snapshot、GitOps branch、artifact catalog、Argo 和 runtime checkout 必须使用同一 Gitea repository:
|
||||
- CLI plan 与 Pipeline 必须从同一个 `gitops_read_url` 恢复 catalog;
|
||||
- promotion 只通过 YAML 声明的 Gitea SecretRef 写回同一仓库;
|
||||
- AgentRun 不得创建 managed-repository reconciler 或挂载 legacy mirror cache;resource bundle 直接读取 owning YAML 声明的 Gitea repository;
|
||||
- 禁止 `git-mirror-http`、`git-mirror-write`、mirror cache、sync/flush Job、fallback 或第二 authority。
|
||||
- CLI 必须组合 `config/platform-infra/gitea.yaml` 与 `config/platform-infra/pipelines-as-code.yaml`,按 consumer、node、lane、upstream repository、branch 和 Gitea repository 精确解析 delivery authority。不得用 URL 片段或 repo 专属条件判断迁移;当前选中对象零匹配或多匹配时返回 `unknown`、`mutation=false` 并在 mutation 前 fail-closed,非选中对象和全局一致性错误只进入 `blocking=false` warning。
|
||||
- PaC 与 `unknown` authority 的 help、plan、status、失败态 `Next`、`REPAIR` 和实际执行 guard 都不得包含或执行 legacy source mutation。`trigger-current|refresh|sync|flush` 只保留退役诊断,不得创建 Job 或写 ref;平台 bootstrap、Secret 与配置维护使用独立 scoped help,不得充当 source delivery recovery。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# AgentRun 控制面
|
||||
|
||||
AgentRun YAML-only lane 以 `config/agentrun.yaml` 为部署真相;node/lane、source workspace/branch、image build、GitOps branch/path、runtime namespace、Secret、外置数据库、manager env、git-mirror 和 edge 暴露都从 YAML 进入 CLI。
|
||||
AgentRun YAML-only lane 以 `config/agentrun.yaml` 为部署真相;node/lane、source workspace/branch、image build、GitOps branch/path、runtime namespace、Secret、外置数据库、manager env、Gitea authority 和 edge 暴露都从 YAML 进入 CLI。
|
||||
|
||||
## 明确 legacy/manual authority 的 YAML-only lane
|
||||
|
||||
@@ -23,7 +23,7 @@ bun scripts/cli.ts platform-infra pipelines-as-code status --target JD01 [--full
|
||||
- `apply`: 按 YAML 渲染并 apply Tekton RBAC/Pipeline、Argo AppProject/Application 和 runtime namespace。
|
||||
- `secret-sync`: 按 YAML 的 Secret sourceRef/keyMapping 同步 runtime Secret 和外置 DB Secret,只输出 fingerprint。
|
||||
- `restart`: patch manager Deployment restart annotation 并等待 rollout,用于 Secret export/DB 连接串变化后让 workload 读取新 Secret;不要手工删除 Pod。
|
||||
- `trigger-current`: legacy v0.2 lane source authority 只读 k8s git-mirror snapshot。confirmed 运行先触发受控 `git-mirror sync`,为 source branch tip 创建 `refs/unidesk/snapshots/agentrun-yaml-lane/<branch>/<commit>`,再从该 snapshot 构建并推送 YAML 声明的 image,渲染 GitOps/artifact catalog,flush git-mirror 并创建 provenance PipelineRun。迁移到 PaC/Gitea 的 lane 不再通过该自维护触发器提交 CI。
|
||||
- `trigger-current`: 已退役。PaC/Gitea lane 只允许 `release plan` 后由 `release trigger --confirm` 发送一次 webhook,不再创建 legacy sync/flush Job 或自维护 PipelineRun。
|
||||
- `cleanup-runners`:
|
||||
- 只读分类 YAML 选中 lane 中匹配 `deployment.runner.retention.selectors` 的 runner Job/Pod;
|
||||
- 默认有界显示 observation、capacity、最近活跃、heartbeat、业务进展、Pod waiting reason 和 `stale-pending`;
|
||||
@@ -80,9 +80,7 @@ bun scripts/cli.ts agentrun provider-profile validate --node <NODE> --lane <lane
|
||||
- 给出 HWLAB web-probe 原入口验收指引;
|
||||
- 不替代真实 Workbench 用户路径验证。
|
||||
|
||||
Legacy YAML-only lane 的长步骤必须由 CLI 拆成短提交和状态轮询:k8s git-mirror snapshot sync、image build、GitOps publish、git-mirror flush 和 PipelineRun 创建不得塞进一个顶层 `trans` 长连接。GitOps publish 必须使用隔离临时 clone/worktree,不能切换或污染 YAML 声明的固定 source workspace。
|
||||
|
||||
AgentRun YAML-only lane closeout 必须同时看当前 k8s git-mirror source snapshot、目标 PipelineRun、GitOps revision、Argo revision 和 manager source commit。发布过程中如果 source branch 被并行 PR 推进,`status --pipeline-run <name>` 会通过 `summary.branchDrift` / `alignment.branchDrift` 标记目标 PipelineRun 是否已被当前 snapshot tip supersede。最终只用最新 PipelineRun 的 `status` 中 `aligned=true`、`blockers=[]`、`argoSyncedToGitops=true` 和 `managerSourceMatchesExpected=true` 收口。
|
||||
AgentRun PaC lane closeout 只核对 Gitea source commit、目标 PipelineRun、GitOps revision、Argo revision 和 manager source commit。发布过程中如果 source branch 被并行 PR 推进,`status --pipeline-run <name>` 会通过 `summary.branchDrift` / `alignment.branchDrift` 标记目标 PipelineRun 是否已被当前 Gitea branch tip supersede。最终只用最新 PipelineRun 的 `status` 中 `aligned=true`、`blockers=[]`、`argoSyncedToGitops=true` 和 `managerSourceMatchesExpected=true` 收口。
|
||||
|
||||
## JD01/NC01 v0.2 Gitea / Pipelines-as-Code lane
|
||||
|
||||
@@ -128,16 +126,11 @@ bun scripts/cli.ts agentrun control-plane cleanup-released-pvs [--limit N] [--dr
|
||||
|
||||
AgentRun compact JSON 关键字段在 `.data.summary.sourceCommit`、`.data.summary.expectedPipelineRun`、`.data.summary.runtimeAlignment`、`.data.summary.gitMirror`、`.data.summary.ci.pipelineRun`、`.data.summary.argo` 和 `.data.alignment`,不要假设存在 `.data.status`。
|
||||
|
||||
## AgentRun legacy Git mirror
|
||||
## AgentRun legacy Git mirror 退役诊断
|
||||
|
||||
```bash
|
||||
bun scripts/cli.ts agentrun git-mirror status [--full|--raw]
|
||||
bun scripts/cli.ts agentrun git-mirror sync [--dry-run|--confirm] [--wait]
|
||||
bun scripts/cli.ts agentrun git-mirror flush [--dry-run|--confirm] [--wait]
|
||||
```
|
||||
|
||||
- `status`: 返回 `localV01`/`githubV01`/`localGitops`/`githubGitops`/`pendingFlush`/`githubInSync`。
|
||||
- `sync`: 拉取 GitHub `v0.1` 和 `v0.1-gitops` refs。
|
||||
- `flush`: 推送本地 `v0.1-gitops` 到 GitHub。
|
||||
|
||||
AgentRun mirror 与 HWLAB mirror 共用 `devops-infra` 服务和 cache PVC,但 repo path、refs、status 文件和 CLI 命令彼此独立。
|
||||
该入口只允许返回退役状态;不得 sync、flush、创建 reconciler、挂载 cache PVC 或重建 legacy runtime。
|
||||
终态只接受 legacy Deployment、Service、reconciler、cache PVC、Secret 和 sync/flush Job 全部不存在;历史 ref 或 status 字段不得触发恢复动作。
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
`git-mirror-http`、`git-mirror-write`、mirror cache、sync/flush Job 和相关 Secret 已退出 CI/CD 运行架构。它们不得作为 source cache、GitOps write endpoint、故障 fallback 或第二 authority 保留。
|
||||
|
||||
AgentRun managed-repository reconciler 也不得继续挂载 legacy mirror cache。PaC/Gitea lane 的 source、resource bundle、artifact catalog 与 GitOps branch 统一直接读取 Gitea;历史 reconciler Deployment、ConfigMap、ServiceAccount 及其终态 TaskRun 只属于受控退役范围。
|
||||
|
||||
## 稳定规则
|
||||
|
||||
- source branch、immutable snapshot、GitOps branch、artifact catalog、Argo 和 runtime checkout 统一使用 owning YAML 声明的 Gitea repository。
|
||||
|
||||
@@ -336,7 +336,7 @@ controlPlane:
|
||||
syncJobPrefix: git-mirror-agentrun-nc01-v02-sync-manual
|
||||
flushJobPrefix: git-mirror-agentrun-nc01-v02-flush-manual
|
||||
repositoryReconciler:
|
||||
enabled: true
|
||||
enabled: false
|
||||
deploymentName: agentrun-nc01-v02-managed-repository-reconciler
|
||||
configMapName: agentrun-nc01-v02-managed-repository-reconciler
|
||||
serviceAccountName: agentrun-nc01-v02-managed-repository-reconciler
|
||||
|
||||
@@ -97,6 +97,7 @@
|
||||
- PaC、Tekton、Argo 和 runtime checkout 必须读取同一 Gitea 仓库中的对应 branch;
|
||||
- GitOps promotion 必须通过 Gitea SecretRef 写回同一仓库,禁止再写 legacy `git-mirror`;
|
||||
- `git-mirror-http`、`git-mirror-write`、mirror cache、sync/flush Job 和相关运行时入口必须退役,不得作为 fallback、缓存或第二 authority 保留;
|
||||
- AgentRun managed-repository reconciler 不得挂载或维持 legacy mirror cache;PaC/Gitea lane 的 source、resource bundle 与 GitOps 读取均直接使用 owning YAML 声明的 Gitea repository;
|
||||
- GitHub 只作为上游 source 输入,不能被 Pipeline 或 runtime 直接提升为第二 GitOps authority;
|
||||
- 输出应说明 owner/repo、read URL、write URL 和同步来源,但不得输出 Git credential、SSH private key 或可复制 token。
|
||||
|
||||
|
||||
@@ -12,7 +12,6 @@ import {
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import {
|
||||
agentRunManagedRepositoryDesiredConfig,
|
||||
agentRunManagedRepositoryRenderHash,
|
||||
renderAgentRunManagedRepositoryReconcilerFragment,
|
||||
} from "./agentrun-managed-repository-reconciler";
|
||||
@@ -29,48 +28,10 @@ const controllerPath = new URL("../native/agentrun/managed-repository-reconciler
|
||||
const askPassPath = new URL("../native/agentrun/git-askpass.mjs", import.meta.url).pathname;
|
||||
|
||||
describe("AgentRun YAML-owned managed repository reconciler", () => {
|
||||
test("renders one narrow controller from the lane repository list", () => {
|
||||
test("does not render the retired managed repository controller", () => {
|
||||
const spec = resolveAgentRunLaneTarget({ node: "NC01", lane: "nc01-v02" }).spec;
|
||||
const desired = agentRunManagedRepositoryDesiredConfig(spec);
|
||||
expect(desired.repositories.map((item) => item.repository)).toEqual([
|
||||
"pikasTech/agentrun",
|
||||
"pikasTech/unidesk",
|
||||
"pikasTech/agent_skills",
|
||||
]);
|
||||
expect(desired.repositories.every((item) => item.remote.startsWith("https://github.com/"))).toBe(true);
|
||||
expect(desired.remoteAuth).toMatchObject({
|
||||
authMode: "github-https-token",
|
||||
host: "github.com",
|
||||
proxy: { host: "127.0.0.1", port: 10808 },
|
||||
});
|
||||
expect(desired.lifecycle).toEqual({
|
||||
undeclaredRepositoryPolicy: "retain",
|
||||
managedRefs: "source-branch-only",
|
||||
});
|
||||
|
||||
const documents = renderAgentRunManagedRepositoryReconcilerFragment("NC01")
|
||||
.split(/^---\s*$/mu)
|
||||
.map((item) => Bun.YAML.parse(item) as any);
|
||||
expect(documents.map((item) => item.kind)).toEqual(["ServiceAccount", "ConfigMap", "Deployment"]);
|
||||
expect(documents[0].automountServiceAccountToken).toBe(false);
|
||||
expect(documents[1].metadata.annotations["unidesk.ai/undeclared-repository-policy"]).toBe("retain");
|
||||
expect(JSON.parse(documents[1].data["config.json"]).repositories).toEqual(desired.repositories);
|
||||
expect(documents[2].spec.strategy.type).toBe("Recreate");
|
||||
expect(documents[2].spec.template.spec.automountServiceAccountToken).toBe(false);
|
||||
expect(documents[2].spec.template.spec.hostNetwork).toBe(spec.gitMirror.repositoryReconciler.hostNetwork);
|
||||
expect(documents[2].spec.template.spec.dnsPolicy).toBe(spec.gitMirror.repositoryReconciler.dnsPolicy);
|
||||
expect(documents[2].spec.template.spec.containers[0].command.slice(0, 2)).toEqual(["tini", "--"]);
|
||||
expect(documents[2].spec.template.spec.containers[0].readinessProbe.exec.command).toContain("--health");
|
||||
const authVolume = documents[2].spec.template.spec.volumes.find((item: any) => item.name === "git-auth");
|
||||
expect(authVolume.projected.sources).toHaveLength(2);
|
||||
expect(authVolume.projected.sources[1].secret).toEqual({
|
||||
name: "gitea-github-sync-secrets",
|
||||
items: [{ key: "github-token", path: "token", mode: 256 }],
|
||||
});
|
||||
expect(JSON.stringify(authVolume)).not.toContain("gitea-username");
|
||||
expect(JSON.stringify(authVolume)).not.toContain("gitea-password");
|
||||
expect(JSON.stringify(authVolume)).not.toContain("github-webhook-secret");
|
||||
expect(documents[2].spec.template.spec.volumes.some((item: any) => item.name === "git-ssh")).toBe(false);
|
||||
expect(spec.gitMirror.repositoryReconciler.enabled).toBe(false);
|
||||
expect(renderAgentRunManagedRepositoryReconcilerFragment("NC01")).toBe("");
|
||||
expect(renderAgentRunManagedRepositoryReconcilerFragment("JD01")).toBe("");
|
||||
|
||||
const controllerSource = readFileSync(controllerPath, "utf8");
|
||||
@@ -80,8 +41,6 @@ describe("AgentRun YAML-owned managed repository reconciler", () => {
|
||||
});
|
||||
|
||||
test("registers after PaC-owned admission objects without identity or separator conflicts", () => {
|
||||
const spec = resolveAgentRunLaneTarget({ node: "NC01", lane: "nc01-v02" }).spec;
|
||||
const desired = agentRunManagedRepositoryDesiredConfig(spec);
|
||||
const fragment = renderPlatformInfraGiteaDesiredFragments("NC01");
|
||||
expect(fragment).not.toContain("---\n---");
|
||||
const documents = fragment
|
||||
@@ -94,11 +53,10 @@ describe("AgentRun YAML-owned managed repository reconciler", () => {
|
||||
"RoleBinding",
|
||||
"Role",
|
||||
"RoleBinding",
|
||||
"Role",
|
||||
"RoleBinding",
|
||||
"ValidatingAdmissionPolicy",
|
||||
"ValidatingAdmissionPolicyBinding",
|
||||
"ServiceAccount",
|
||||
"ConfigMap",
|
||||
"Deployment",
|
||||
]);
|
||||
const identities = documents.map((item) => [
|
||||
item.apiVersion,
|
||||
@@ -108,20 +66,7 @@ describe("AgentRun YAML-owned managed repository reconciler", () => {
|
||||
].join("/"));
|
||||
expect(new Set(identities).size).toBe(documents.length);
|
||||
|
||||
const managed = documents.slice(-3);
|
||||
expect(managed.map((item) => item.metadata.name)).toEqual([
|
||||
spec.gitMirror.repositoryReconciler.serviceAccountName,
|
||||
spec.gitMirror.repositoryReconciler.configMapName,
|
||||
spec.gitMirror.repositoryReconciler.deploymentName,
|
||||
]);
|
||||
const config = JSON.parse(managed[1].data["config.json"]);
|
||||
expect(config.desiredHash).toBe(desired.desiredHash);
|
||||
expect(managed[1].metadata.annotations["unidesk.ai/managed-repository-desired-sha"]).toBe(desired.desiredHash);
|
||||
expect(managed[2].metadata.annotations["unidesk.ai/managed-repository-desired-sha"]).toBe(desired.desiredHash);
|
||||
expect(managed[1].metadata.annotations["unidesk.ai/managed-repository-render-sha"]).toMatch(/^[0-9a-f]{64}$/u);
|
||||
expect(managed[2].metadata.annotations["unidesk.ai/managed-repository-render-sha"]).toBe(
|
||||
managed[1].metadata.annotations["unidesk.ai/managed-repository-render-sha"],
|
||||
);
|
||||
expect(fragment).not.toContain("managed-repository-reconciler");
|
||||
});
|
||||
|
||||
test("requires every repository entry to own a matching remote", () => {
|
||||
@@ -188,7 +133,8 @@ describe("AgentRun YAML-owned managed repository reconciler", () => {
|
||||
test("renders host networking only from the explicit reconciler fields", () => {
|
||||
const temporary = mkdtempSync("/tmp/unidesk-agentrun-repository-network-");
|
||||
try {
|
||||
const source = readFileSync(new URL("../../config/agentrun.yaml", import.meta.url), "utf8");
|
||||
const source = readFileSync(new URL("../../config/agentrun.yaml", import.meta.url), "utf8")
|
||||
.replace("repositoryReconciler:\n enabled: false", "repositoryReconciler:\n enabled: true");
|
||||
const nonLoopbackProxy = source.replace(
|
||||
"githubProxy:\n host: 127.0.0.1\n port: 10808",
|
||||
"githubProxy:\n host: proxy.internal.example\n port: 10808",
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { listAgentRunLaneTargetsForNode } from "./agentrun-lanes";
|
||||
import { rootPath } from "./config";
|
||||
import { runCommand, type CommandResult } from "./command";
|
||||
import { egressBenchmarkCompactResult, egressBenchmarkDryRun, egressBenchmarkStartScript, egressBenchmarkStatusScript, type EgressBenchmarkSpec } from "./egress-proxy-benchmark";
|
||||
@@ -456,6 +457,11 @@ function retireLegacySourceRuntime(node: ControlPlaneNodeSpec, target: ControlPl
|
||||
const transportSecretName = target.gitMirror.githubTransport.mode === "https"
|
||||
? target.gitMirror.githubTransport.tokenSecretName
|
||||
: target.gitMirror.secretName;
|
||||
const sharedAgentRunReconcilers = listAgentRunLaneTargetsForNode(node.id)
|
||||
.map(({ spec }) => spec)
|
||||
.filter((spec) => spec.gitMirror.namespace === target.gitMirror.namespace
|
||||
&& spec.gitMirror.cachePvc === target.gitMirror.cachePvcName)
|
||||
.map((spec) => spec.gitMirror.repositoryReconciler);
|
||||
const resources = [
|
||||
`deployment/${target.gitMirror.serviceReadName}`,
|
||||
`deployment/${target.gitMirror.serviceWriteName}`,
|
||||
@@ -464,6 +470,11 @@ function retireLegacySourceRuntime(node: ControlPlaneNodeSpec, target: ControlPl
|
||||
`configmap/${target.gitMirror.syncConfigMapName}`,
|
||||
`pvc/${target.gitMirror.cachePvcName}`,
|
||||
`secret/${transportSecretName}`,
|
||||
...sharedAgentRunReconcilers.flatMap((reconciler) => [
|
||||
`deployment/${reconciler.deploymentName}`,
|
||||
`configmap/${reconciler.configMapName}`,
|
||||
`serviceaccount/${reconciler.serviceAccountName}`,
|
||||
]),
|
||||
];
|
||||
const jobPrefixes = [target.gitMirror.syncJobPrefix, target.gitMirror.flushJobPrefix];
|
||||
const plan = {
|
||||
@@ -471,7 +482,8 @@ function retireLegacySourceRuntime(node: ControlPlaneNodeSpec, target: ControlPl
|
||||
authority: "gitea",
|
||||
resources,
|
||||
jobPrefixes,
|
||||
excluded: ["pipeline", "pipelinerun", "argo", "runtime-workloads", "registry"],
|
||||
terminalTaskRunsUsingCachePvc: target.gitMirror.cachePvcName,
|
||||
excluded: ["pipeline", "pipelinerun", "active-taskrun", "argo", "runtime-workloads", "registry"],
|
||||
};
|
||||
if (!options.confirm) {
|
||||
return {
|
||||
@@ -491,13 +503,36 @@ function retireLegacySourceRuntime(node: ControlPlaneNodeSpec, target: ControlPl
|
||||
const script = `
|
||||
set -eu
|
||||
namespace=${shQuote(namespace)}
|
||||
kubectl -n "$namespace" delete ${resourceArgs} --ignore-not-found --wait=true >/tmp/unidesk-legacy-source-runtime-delete.out
|
||||
cache_pvc=${shQuote(target.gitMirror.cachePvcName)}
|
||||
kubectl -n "$namespace" delete ${resourceArgs} --ignore-not-found --wait=false >/tmp/unidesk-legacy-source-runtime-delete.out
|
||||
for prefix in ${prefixArgs}; do
|
||||
kubectl -n "$namespace" get jobs -o name 2>/dev/null | while IFS= read -r resource; do
|
||||
name="\${resource#job.batch/}"
|
||||
case "$name" in "$prefix"*) kubectl -n "$namespace" delete "$resource" --ignore-not-found --wait=true >/dev/null ;; esac
|
||||
done
|
||||
done
|
||||
kubectl -n "$namespace" get pods -o jsonpath='{range .items[*]}{.status.phase}{"\\t"}{.metadata.ownerReferences[0].kind}{"\\t"}{.metadata.ownerReferences[0].name}{"\\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{","}{end}{"\\n"}{end}' 2>/dev/null |
|
||||
while IFS="$(printf '\\t')" read -r phase owner_kind owner_name claims; do
|
||||
case "$claims" in
|
||||
*"$cache_pvc"*)
|
||||
case "$phase:$owner_kind" in
|
||||
Succeeded:TaskRun|Failed:TaskRun)
|
||||
kubectl -n "$namespace" delete taskrun "$owner_name" --ignore-not-found --wait=false >/dev/null
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
done
|
||||
attempt=0
|
||||
while [ "$attempt" -lt 12 ]; do
|
||||
remaining=0
|
||||
for resource in ${resourceArgs}; do
|
||||
if kubectl -n "$namespace" get "$resource" >/dev/null 2>&1; then remaining=$((remaining + 1)); fi
|
||||
done
|
||||
[ "$remaining" -eq 0 ] && break
|
||||
attempt=$((attempt + 1))
|
||||
sleep 1
|
||||
done
|
||||
remaining=0
|
||||
for resource in ${resourceArgs}; do
|
||||
if kubectl -n "$namespace" get "$resource" >/dev/null 2>&1; then remaining=$((remaining + 1)); fi
|
||||
|
||||
Reference in New Issue
Block a user