refactor(platform-infra): retire cross-region k3s topology

This commit is contained in:
pikastech
2026-07-20 10:15:38 +02:00
parent e97d1f1fc4
commit 3295aa64a0
13 changed files with 604 additions and 2082 deletions
+16 -31
View File
@@ -348,44 +348,29 @@ bun scripts/cli.ts hwlab nodes control-plane legacy-cicd --help
- 一旦发现 CI/CD CLI 被误用且可能写入错误状态、产生伪证据或绕过目标运行面,必须立刻先把用法改成更符合直觉的公开入口并更新本 skill/reference,再继续验证或交付;不要只靠口头记忆、隐藏 flag、手动约定或后续小心来避免复发。内部 in-cluster 模式必须只由目标 k8s Job/Pod 调用,操作者从本机只能用公开入口提交目标侧 Job 或读取目标侧摘要。
- PaC migrated consumer 的旧手动 publish/debug 命令不得作为正式 closeout 引导;调查命令默认只能指向只读的 `platform-infra pipelines-as-code status|history``closeout` 仅保留只读兼容能力,不得作为主代理、子代理或操作者的 PR 合并后步骤。需要保留的旧 mutation 命令只能属于 YAML 明确的 legacy authority 或独立平台维护,不得标记或引导 manual recovery;单步确认的自动链缺陷必须回到 YAML/controller/源码修复。
- Secret 只通过 YAML sourceRef/targetKey 和受控 CLI 下发;输出只披露 presence/fingerprint。
- 多主机 k3s 节点变更
- CI/CD、业务 renderer、PipelineRun 和普通 GitOps workload 不得直接写 WireGuard 配置、k3s systemd unit、agent token 文件或节点 ServiceLB 标签。
- 节点加入、状态和调度 smoke 只走 `platform-infra k3s-cluster plan|apply|status|smoke`,具体 authority 和控制通道见 `docs/reference/platform-infra.md#多主机-k3s-集群`
- 集群 apply 的 control-plane 阶段只允许组合 `hwlab nodes control-plane infra k3s-config` 窄域;禁止调用全量 `infra apply`,避免节点加入顺带覆盖 registry、Tekton、Argo、git-mirror 或运行面对象。
- k3s drop-in 未变化且节点容量已收敛时必须 `mutation=false`;禁止无条件重启、重装、强写或通过 CI 自动覆盖同一节点配置。
- WireGuard 配置一致时不得 restart;k3s API/容量暂时不可读时保持 unknown 并停止 mutation,禁止把控制通道抖动解释为配置漂移
- worker agent 只在 YAML 派生摘要变化或 service 未运行时安装/恢复,不得在每次集群 apply 中先卸载再重装
- worker 调度资格和 taint 只认集群 owning YAML
- `schedulable=false` 必须由 agent 注册参数与受控 `cordon`/taint 双重收敛;
- CI、PipelineRun、GitOps 和业务 renderer 禁止 `uncordon`、删除该 taint
或添加 toleration/`nodeName` 将业务 Pod 强制放回不可调度 worker
- 清空不可调度 worker 只走 `platform-infra k3s-cluster drain --confirm`
保留 DaemonSet/节点系统 Pod,且节点重注册后必须先恢复调度隔离。
- `Type=notify` agent 的 `activating` 且 MainPID 存在属于运行中;节点 Ready 时不得仅因 systemd 未进入 `active` 而重复重装。
- worker 必须先由集群入口按 `hostProxyConfigRef` 部署并验证 proxy client,再启动 agentcontainerd 通过该 proxy 直接访问上游 registry,禁止用公共 registry mirror 代替出网链路。
- D601 独立 k3s 集群
- CI/CD、业务 renderer、PipelineRun 和 GitOps 不得写 k3s systemd unit
不得把 D601 注册为 NC01 agent,也不得创建跨地域 CNI 或共享 datastore
- D601 只走
`platform-infra k3s-cluster plan|apply|status|smoke --target d601`
NC01 只走自己的 control-plane 入口,两者不得互相组合
- D601 server 的 YAML 摘要变化且节点健康时禁止重装、重启或强写
- D601 默认是空集群。任何业务进入 D601 必须有独立 workload owning
YAML 和显式 target,禁止继承 NC01 的 PaC、Tekton、Argo、GitOps、
DaemonSet、namespace 或默认 placement
- D601 先通过 host proxy 让 server/containerd 出网,禁止公共 registry
mirror。
- 集群自有 artifact registry 的异地拉取是明确例外:
- 内部 publish endpoint 保持唯一写入 authority
- 公网 HTTPS 只允许鉴权 `GET`/`HEAD`,不得成为 push、cache export 或删除入口;
- worker 保持既有镜像引用,由集群 owning YAML 渲染 endpoint 与 SecretRef
- registry 大流量不得经过 WireGuard/Flannel,也不得使用 CDN 或公共镜像站;
- D601 保持既有镜像引用,由集群 owning YAML 渲染 endpoint 与 SecretRef
- registry 大流量不得经过 Kubernetes 数据面,也不得使用 CDN 或公共镜像站;
- bcrypt hash 必须以 Compose-safe 字面量写入 Secret source,且
public-edge 用脱敏 material fingerprint 驱动凭据变更后的 recreate
候选校验不得用会保留引号的 `docker run --env-file` 改变 Secret 语义;
- 验收统一走 `platform-infra k3s-cluster registry-smoke`,不手工 patch Caddy 或 containerd。
- worker 安装使用短启动并由 control-plane Node Ready 轮询收口,禁止让 `trans`/SSH 长挂等待 systemd notify。
- worker 加入集群不改变 PaC consumer 的 source commit authority,也不授权人工 PipelineRun、Argo sync 或业务 rollout
- NC01 与 D601 是永久分离的调度域:
- `config/platform-infra/k3s-clusters.yaml` 必须保持
`topologyMode=separated`,D601 只作为独立单节点 k3s 集群存在;
- CI/CD、PaC、Tekton、Argo、GitOps、VM bootstrap 和业务 renderer
禁止把 D601 注册为 NC01 agent,禁止恢复跨地域 WireGuard peer
禁止把 NC01 workload、DaemonSet 或 namespace 默认复制到 D601
- 拆分和验收只走
`platform-infra k3s-cluster separate|status|smoke`
- 跨地域 artifact 传输使用鉴权、只读的公网 HTTPS registry
不得通过 Kubernetes Pod 网络、Service、DNS 或 VXLAN
- 如未来确需跨地域计算,必须建立独立 workload 级交付规格并显式选择
D601 集群,不得通过重新组成单一集群获得隐式调度。
- 跨地域 artifact 传输使用鉴权、只读的公网 HTTPS registry,不得通过
Kubernetes Pod 网络、Service、DNS 或 VXLAN
- 长命令用异步 job 或短轮询;不要长时间挂住 trans/ssh。
- Web 哨兵 YAML cadence 变更不新增发布 orchestrator:提交前用 `web-probe sentinel validate --local`;合并后用只读 `web-probe sentinel control-plane status --wait`。线上 desired schedule 必须来自 GitOps 清单,不能用操作者旧 worktree 的本地渲染值做阻塞门禁;收敛差异统一为非阻塞 warning。
+14 -32
View File
@@ -88,38 +88,20 @@ description: UniDesk YAML-first 运维正规化技能。用户提到 ymal-first/
- 非核心字段可以使用显式领域默认值,但必须同时生成 `blocking=false` warning
- 不得生成不可见默认值,也不得把合并后的大对象写成新的 source of truth。
- 新 VPS 或裸节点 bootstrap 的出网必须优先走 YAML-first 声明的 0 依赖 host proxy client:通过 `trans` 分发零依赖客户端,先让 host route 能出网,再安装 k3s、containerd、Docker、包管理器依赖和 Git/gitrepo 同步;不得把 provider-gateway WebSocket egress 当作大二进制或依赖下载的数据面。
- 多主机 k3s 集群:
- 跨主机拓扑、WireGuard、agent token、worker host proxy 引用、集群内 artifact registry 映射、ServiceLB 节点和调度 smoke 归独立集群 owning YAML。
- worker host proxy 归 `config/platform-infra/host-proxy.yaml`;集群 apply 必须先确认 proxy client 连接声明的 `vpn-server` 并通过外网 probe,再启动 agent。
- worker proxy 前必须先收敛 WireGuard;配置一致时不重启,只有 unit 未运行或握手过期且隧道探针失败时才允许恢复 worker 接口
- containerd 通过 host proxy 直接访问公共 registry,禁止为公共依赖新增 registry mirror;集群内 artifact registry 映射不属于公共镜像代理。
- 集群自有 artifact registry 可由同一 owning YAML 声明公网可信 TLS 的 pull-only endpoint
- 内部 endpoint 继续拥有 push
- public-edge 只允许鉴权 `GET`/`HEAD`
- containerd 凭据与 Caddy hash 必须来自同一 SecretRef
- 镜像层绕开 WireGuard/Flannel 和 CDN
- registry 配置变化不得触发 worker agent 卸载重装。
- 既有 control-plane 的 k3s renderer 通过 `clusterConfigRef` 消费集群网络参数,不得由集群 CLI 再写第二个 systemd drop-in
- 无固定公网地址 worker 使用宿主内网 SSH 作为部署控制通道并主动连接 WireGuard endpoint,不使用 FRP 代替三层集群网络。
- control-plane drop-in 只走 `hwlab nodes control-plane infra k3s-config` 窄域:
- 渲染结果一致且节点容量已收敛时返回 `mutation=false`
- 只在必要时写入 drop-in、reload 和重启 k3s
- 确需重启 server 时,先停止服务并备份 k3s 派生
`server/cred/passwd`;禁止用该文件与 datastore 的
mtime/freshness 预判阻断变更,备份只输出
`blocking=false` warning
- WireGuard 内容一致时不重启,API/容量读取暂时失败时保持 unknown 并停止 mutation
- worker agent 使用 YAML 派生摘要决定是否重装,禁止每次 apply 先卸载再覆盖;
- 禁止借用全量 infra apply 顺带覆盖 registry、CI/CD 或业务运行面。
- 日常入口和验收判定统一见 `docs/reference/platform-infra.md#多主机-k3s-集群`
- NC01-D601 的 owning YAML 必须声明 `topologyMode=separated`
- D601 独立 server 的节点名、标签、禁用组件和 kubeconfig mode 必须
位于同一 YAML 的 `k3s.standalone`
- Hyper-V YAML 只能用 `management.mode=standalone` 引用该片段;
- parser、bootstrap、CI/CD 和 GitOps 禁止通过隐藏默认值把 D601
恢复为 agent、恢复 WireGuard peer或共享调度域;
- 独立集群间只共享显式声明的只读 artifact endpoint,不共享 Pod、
Service、DNS、CNI、datastore 或默认 workload。
- D601 独立 k3s 集群:
- `config/platform-infra/k3s-clusters.yaml#targets.d601` 只声明独立
server、节点、host proxy 和只读 artifact registry,不得出现 NC01、
agent token、跨地域 CNI 或共享调度字段
- Hyper-V YAML 以 `management.mode=standalone` 引用 D601 k3s 片段;
parser、bootstrap、CI/CD 和 GitOps 禁止用隐藏默认值恢复 agent。
- NC01 与 D601 不共享 Pod、Service、DNS、CNI、datastore、DaemonSet、
namespace 或默认 workload。
- D601 containerd 通过 host proxy 访问公共 registry,禁止公共镜像站
集群自有 artifact registry 只通过鉴权、只读公网 HTTPS 拉取。
- registry 内部 endpoint 继续拥有 pushpublic-edge 只允许
`GET`/`HEAD`containerd 凭据与 Caddy hash 来自同一 SecretRef
- 日常入口和验收见
`docs/reference/platform-infra.md#d601-独立-k3s-集群`
- 新节点 node-local registry 必须支持从零部署:优先由 YAML 渲染为 k3s workload/PVChost Docker registry 只能是 legacy/迁移前状态。需要证明从零能力时先停旧 registry,不从旧 registry seed,并在状态中披露 `seededFromOldRegistry``zeroSeeded`、PVC、workload、endpoint 与 runtime/tools image preload 结果。
- YAML 配置必须可复用、可继承、可组合;节点/lane 只能作为变量或 YAML value 参与渲染,禁止把具体 node/lane 写入 YAML 文件名、parser/renderer/helper 函数名或长期 schema 名。发现不同节点复制大段配置时,先抽 baseline/configRef/override,再继续扩展节点。
- CLI `plan/status` 应输出 redacted 配置引用图:每个 ref 的文件、path、presence、摘要 hash、缺失字段和下一步 drill-down 命令。不要默认 dump 展开后的完整 YAML 或 Secret。
-1
View File
@@ -316,7 +316,6 @@ nodes:
serviceName: k3s
dropInPath: /etc/systemd/system/k3s.service.d/20-unidesk-node-config.conf
nodeStatusName: v2202607378382480008
clusterConfigRef: config/platform-infra/k3s-clusters.yaml#targets.nc01-d601.controlPlane
execStartPre: []
install:
enabled: true
+1 -1
View File
@@ -127,7 +127,7 @@ targets:
route: D601-VM
sourceRef: sources.d601-nc01-vpn-server-shadowsocks
files:
k3sSystemdDropIn: /etc/systemd/system/k3s-agent.service.d/10-unidesk-proxy.conf
k3sSystemdDropIn: /etc/systemd/system/k3s.service.d/10-unidesk-proxy.conf
env:
noProxyRefs:
common: noProxy.common
+1 -1
View File
@@ -74,7 +74,7 @@ targets:
k3s:
management:
mode: standalone
configRef: config/platform-infra/k3s-clusters.yaml#targets.nc01-d601.k3s.standalone
configRef: config/platform-infra/k3s-clusters.yaml#targets.d601.k3s
version: v1.36.2+k3s1
installMirror: cn
binaryUrl: https://github.com/k3s-io/k3s/releases/download/v1.36.2%2Bk3s1/k3s
+18 -81
View File
@@ -4,77 +4,32 @@ metadata:
owner: unidesk
specRef: project-management/PJ2026-01/specs/PJ2026-01060314-hyperv-ubuntu-vm.md
defaults:
targetId: nc01-d601
targetId: d601
targets:
nc01-d601:
d601:
enabled: true
clusterName: nc01
topologyMode: separated
controlPlane:
id: NC01
configRef: config/hwlab-node-control-plane.yaml#nodes.NC01
lane: v03
nodeName: v2202607378382480008
publicAddress: 152.53.229.148
workers:
D601-VM:
id: D601-VM
bootstrapConfigRef: config/platform-infra/hyperv-vms.yaml#targets.D601-VM
hostProxyConfigRef: config/platform-infra/host-proxy.yaml#targets.D601-VM
nodeName: d601-vm
labels:
unidesk.ai/node-id: D601-VM
unidesk.ai/provider-id: D601-VM
unidesk.ai/runtime-plane: worker
scheduling:
schedulable: false
taints:
- key: unidesk.ai/business-scheduling
value: disabled
effect: NoSchedule
wireguard:
interfaceName: wg0
packageName: wireguard-tools
mtu: 1380
listenPort: 51820
controlPlaneAddress: 10.89.0.1/24
workerAddress: 10.89.0.2/24
endpoint: 152.53.229.148:51820
persistentKeepaliveSeconds: 10
tcpTuning:
congestionControl: bbr
defaultQdisc: fq
mtuProbing: 1
sysctlPath: /etc/sysctl.d/90-unidesk-k3s-cross-region.conf
secrets:
sourceRef: /root/.unidesk/.state/secrets/platform-infra/k3s-clusters/nc01-d601.env
controlPlanePrivateKey: NC01_WIREGUARD_PRIVATE_KEY
controlPlanePublicKey: NC01_WIREGUARD_PUBLIC_KEY
workerPrivateKey: D601_VM_WIREGUARD_PRIVATE_KEY
workerPublicKey: D601_VM_WIREGUARD_PUBLIC_KEY
agentToken: K3S_AGENT_TOKEN
clusterName: d601
node:
id: D601-VM
bootstrapConfigRef: config/platform-infra/hyperv-vms.yaml#targets.D601-VM
hostProxyConfigRef: config/platform-infra/host-proxy.yaml#targets.D601-VM
nodeName: d601-vm
labels:
unidesk.ai/node-id: D601-VM
unidesk.ai/provider-id: D601-VM
unidesk.ai/runtime-plane: standalone
k3s:
serverUrl: https://10.89.0.1:6443
agentTokenFile: /etc/unidesk/k3s-agent-token
flannelInterface: wg0
writeKubeconfigMode: "0644"
maxPods: 500
standalone:
clusterName: d601
nodeName: d601-vm
writeKubeconfigMode: "0644"
labels:
unidesk.ai/node-id: D601-VM
unidesk.ai/provider-id: D601-VM
unidesk.ai/runtime-plane: standalone
disableComponents:
- traefik
- servicelb
- metrics-server
disableComponents:
- traefik
- servicelb
- metrics-server
registryMirror:
source: 127.0.0.1:5000
endpoint: https://registry.hwpod.com
auth:
sourceRef: /root/.unidesk/.state/secrets/platform-infra/k3s-clusters/nc01-d601-artifact-registry-pull.env
sourceRef: /root/.unidesk/.state/secrets/platform-infra/k3s-clusters/d601-artifact-registry-pull.env
usernameKey: ARTIFACT_REGISTRY_PULL_USERNAME
passwordKey: ARTIFACT_REGISTRY_PULL_PASSWORD
passwordHashKey: ARTIFACT_REGISTRY_PULL_PASSWORD_HASH
@@ -98,21 +53,3 @@ targets:
expectedBytes: 195443636
concurrentPulls: 2
timeoutSeconds: 900
serviceLb:
enabledNodeLabel: svccontroller.k3s.cattle.io/enablelb
enabledNode: v2202607378382480008
workerDns:
namespace: kube-system
deploymentName: unidesk-worker-coredns
serviceName: unidesk-worker-kube-dns
clusterIp: 10.43.0.53
image: docker.io/coredns/coredns:1.14.4
configMapName: coredns
customConfigMapName: coredns-custom
replicas: 1
timeoutSeconds: 120
smoke:
namespace: platform-infra-smoke
jobName: d601-runtime-plane-smoke
image: docker.io/library/busybox:1.36.1
timeoutSeconds: 180
+1 -1
View File
@@ -70,7 +70,7 @@ targets:
sites:
- id: artifact-registry
configRef: config/platform-infra/k3s-clusters.yaml
path: targets.nc01-d601.k3s.registryMirror
path: targets.d601.k3s.registryMirror
- id: pikaoa-development
configRef: config/pikaoa.yaml
path: developmentRuntime.targets.NC01
+34 -127
View File
@@ -10,125 +10,33 @@
- UniDesk 拥有的 image version、namespace、endpoint、FRP port 与 profile 列表不得隐藏在 helper 常量里;外部工具要求的 TOML、JSON 和 env 仅是边缘渲染格式。
- 新节点出网 bootstrap 遵循 `docs/reference/yaml-first-ops.md` 的零依赖 host proxy 边界,不得依赖尚未安装的 Docker、k3s、containerd 或包管理器网络能力。
## k3s 集群拓扑
## D601 独立 k3s 集群
- 权威配置:
- 集群边界、拓扑模式、历史 WireGuard 迁移参数、独立 server 角色、
registry endpoint、ServiceLB 节点和 smoke 归
`config/platform-infra/k3s-clusters.yaml`
- worker host proxy 的服务端来源、客户端工件、Secret 引用、代理环境和 systemd drop-in 归 `config/platform-infra/host-proxy.yaml`
- Hyper-V VM 资源和宿主内网 SSH 归 `config/platform-infra/hyperv-vms.yaml`
- 既有 control-plane 的 k3s systemd drop-in 继续归 `config/hwlab-node-control-plane.yaml`,并通过 `clusterConfigRef` 消费集群网络参数。
- 受控入口:
- `platform-infra k3s-cluster secret-init --target <id> --confirm` 只在 Secret source 不存在时生成 WireGuard key pair 和 k3s agent token,存在时拒绝覆盖。
- `platform-infra k3s-cluster plan|apply|separate|drain|status|smoke
--target <id>` 是部署、拓扑拆分、清空不可调度 worker和验收入口
- `topologyMode=separated` 时,`apply --confirm`
`separate --confirm` 都只执行受控拆分:
- 先证明旧 worker 的非 DaemonSet Pod 为零;
- 再清理旧 agent datastore、CNI 和跨地域 WireGuard 配置;
- 在 D601 安装并等待独立单节点 k3s server Ready
- 最后从 NC01 删除旧 Node 和对端 WireGuard peer
- 拆分不得先删除 NC01 Node 或隧道;D601 独立 server 未 Ready 时必须
保留旧控制面侧事实并返回失败。
- 无固定公网地址的 worker 由宿主内网 SSH 控制,主动连接 control-plane WireGuard endpointworker provider route 只作为部署后补充验收入口。
- 集群 apply 先经宿主内网 SSH 分发零依赖 proxy client,确认它连接声明的 `vpn-server` 且外网 probe 成功,再启动或恢复 `k3s-agent`
- 集群 apply 在 proxy 前先收敛 worker WireGuard;内容一致时不强写,只有 unit 未运行或握手过期且隧道探针失败时才恢复 worker 接口
- worker WireGuard 的业务探针固定使用 control-plane API `:6443`,不以可能被策略丢弃的 ICMP 判断隧道健康。
- NAT 后 worker 的 WireGuard `PersistentKeepalive` 周期由集群 YAML 声明;跨地域节点优先用较短周期维持映射,但不得据此宣称降低物理 RTT。
- 跨地域 TCP 调优由集群 YAML 的 `tcpTuning` 统一声明并受控应用到两端:
- 高 RTT、持续丢包链路优先使用内核支持的 BBR、`fq`
`tcp_mtu_probing=1`,同时保留实际吞吐复测;
- sysctl 文件和 live 值均先比较后收敛,不得靠裸 `sysctl`
预拉镜像、导入镜像或反复重启 Pod 掩盖数据面问题;
- status 必须披露两端 live tuning 是否与 YAML 一致。
- 大型 proxy 工件由 Windows 宿主按 YAML `upstreamUrl` 获取并校验 SHA-256Provider/trans 只承载控制命令,不作为大二进制数据面。
- WireGuard 只承载集群三层网络;D601 worker proxy 使用 `vpn-server`
NC01 公网 Shadowsocks TCP 入口,不把应用出网 TCP 再套入 WireGuard
也不与集群 WireGuard 竞争 UDP 链路。
- proxy client 协议实现版本由 host-proxy YAML 声明,并与所连接的 `vpn-server` 协议实现版本保持一致。
- control-plane 节点配置只组合 `hwlab nodes control-plane infra k3s-config plan|status|apply`
- 该窄域只比较并安装 owning YAML 渲染的 k3s systemd drop-in
- 只在 drop-in 或节点 pod 容量需要收敛时重启 k3s;
- 确需重启 k3s server 时,受控入口先停止服务并备份
`/var/lib/rancher/k3s/server/cred/passwd` 派生文件,再启动服务;
- 禁止用 `passwd``state.db` 的文件 mtime、datastore freshness
预判或其他凭据新旧检查阻断 YAML-first 变更;备份动作必须以
`blocking=false` warning 披露,只有真实 stop/start、Node Ready
或期望状态收敛失败才可使 apply 失败;
- 同一入口等待节点 Ready;
- 禁止调用 HWLAB 全量 `infra apply`,也不得触碰 Kubernetes manifests、registry、Tekton、Argo、git-mirror 或业务 rollout。
- 运行边界:
- NC01 与 D601 必须保持两个独立 k3s 集群,禁止把 D601 重新注册为
NC01 agent,也禁止把 NC01 注册到 D601。
- D601 独立集群只运行一个 server 节点,初始业务 Pod 数必须为零;
是否承载新业务必须由新的 workload owning YAML 显式选择,不得继承
NC01 的 GitOps、PipelineRun、DaemonSet 或默认 namespace。
- NC01 保留自身 control-plane 与运行面;拆分后可保留无 peer 的本地
WireGuard 接口以避免重写生产 control-plane 地址,但不得恢复 D601 peer。
- 跨地域连接只用于受控 SSH/Provider 控制通道和公网 HTTPS
artifact pull,不再承担 Kubernetes API、Pod、Service、DNS 或 VXLAN 数据面。
- `topologyMode=separated` 是 owning YAML 的架构事实。CI/CD、GitOps、
VM bootstrap 和普通集群 apply 禁止改回 joined、启动 `k3s-agent`
写入 agent token 或渲染跨地域 Flannel/WireGuard peer。
- 下列 worker 规则只适用于仍由 owning YAML 明确声明为 joined 的其他
集群,不得用于 NC01-D601
- control-plane 保持唯一 k3s server,同时承担 workload
- worker 只运行 `k3s-agent`,加入前卸载旧独立 server
- worker 是否参与业务调度及其 taint 必须由集群 owning YAML 的
`workers.<id>.scheduling` 声明。
- joined worker 的调度规则:
- `schedulable=false` 时必须同时声明 `NoSchedule``NoExecute` taint
- agent 注册参数和 control-plane 的 `cordon`/taint 收敛必须消费同一声明,
防止节点重装、重注册或 CI 自动覆盖后短暂接收业务 Pod;
- `drain --confirm` 只允许清空声明为不可调度的 worker,并保留
DaemonSet/节点系统 Pod;普通 apply、CI 和业务 renderer 禁止隐式
`uncordon` 或移除 owning YAML 声明的 taint
- drain 与 Node Ready 等待必须由本地 CLI 组合短于 `trans` 上限的
目标侧操作和短轮询;节点已不可调度且用户要求彻底清空时允许
`--disable-eviction` 直接删除非 DaemonSet Pod,但不得扩大到其他节点。
- 不可调度且无业务 Pod 的 worker 不要求 PodCIDR 代理 probe 成功;
host 本地代理健康与外网 probe 仍须成功,空 DNS Deployment、Service
和 Endpoints 必须由 drain 一并清理。
- Flannel 使用 WireGuard 接口承载 VXLAN,不能只用 FRP 转发 Kubernetes API。
- 参与业务调度的跨地域 worker DNS 使用集群 YAML 声明的独立
CoreDNS Deployment 和固定 ClusterIP;不可调度 worker 必须移除该
非 DaemonSet DNS Pod
- Deployment 固定调度到 workerService 使用 `internalTrafficPolicy: Local`
- worker kubelet 只在本地 DNS Ready 后通过 `cluster-dns` 参数切换;
- 禁止 patch k3s Addon 拥有的 `coredns` Deployment 或 `kube-dns` Service
- 受控 apply 先比较声明态哈希,内容一致时跳过 `kubectl apply`,不得触发无意义 rollout。
- ServiceLB 只在 owning YAML 指定的公网节点启用,防止无公网地址 worker 接管 hostPort。
- worker 的 containerd 外网访问继承 host proxy systemd 环境,直接访问上游 registry;禁止为公共依赖新增 registry mirror。
- host proxy 的 Pod 访问监听地址必须由 owning YAML 按目标节点 PodCIDR
声明为该节点 `cni0` 地址;禁止复用其他节点的 `10.42.x.1`
否则 sing-box 会因绑定不存在地址持续重启。
- `127.0.0.1:5000` 只表示 UniDesk 集群内 artifact registryworker 的 `registries.yaml` 可把该本地引用映射到集群 endpoint,不得把它扩展为公共镜像代理。
- 幂等重复执行不得强写:
- WireGuard、systemd drop-in、agent unit、token 文件和节点标签均先比较声明态与运行态;
- 内容一致时保持 `mutation=false`,不得无条件重启、重装或覆盖;
- WireGuard 只在配置变化或 unit 未运行时重启/启动,避免短暂断网触发 control-plane 级联重启;
- worker agent 以 owning YAML、版本、参数、集群内 registry 映射和 token 的脱敏摘要判断是否需要重装;
- `Type=notify` agent 在节点 Ready 时可能保持 `activating`;摘要一致、状态为 `active|activating` 且 MainPID 存在即视为运行中,禁止因此重复重装;
- worker 安装只做短启动,终态由 control-plane 的 Node Ready 轮询判定,不得让宿主 SSH 会话等待 systemd notify
- k3s API 暂时不可读属于 unknown,不得当作容量漂移触发 restart;
- 集群 CLI 只组合各自 owning domain,不借用全量基础设施入口扩大写入范围。
- 验收:
- `topologyMode=separated``status` 必须同时证明:
- D601 只有一个 Ready 节点;
- D601 `k3s` server active 且旧 `k3s-agent` inactive
- D601 非 `kube-system` Pod 为零;
- NC01 不再存在 D601 Node。
- joined 拓扑的 `status` 才验证 WireGuard handshake、host proxy、
server/agent、两个 Ready node、worker InternalIP 和旧 server 退役。
- 仅当 worker 的 `scheduling.schedulable=true` 时,
`smoke --confirm` 才可把 Job 定向调度到 worker并验证集群 DNS与
Kubernetes ClusterIP;不可调度 worker 的 smoke 只读验证 taint、
cordon、Node Ready 和非 DaemonSet Pod 为零,不得创建带 toleration
的例外 Job。
- smoke 等待必须使用短 `kubectl get` 轮询,禁止让单次 `trans` 挂住等待 Job 条件。
- 基础 smoke 要求完整服务 FQDN 的解析结果包含 Pod 注入的 Kubernetes Service IP,并通过 API `ClusterIP:443` TCP connect 验证 Service 数据面;不得接受 `nslookup` 的空应答,也不把容器 TLS 客户端能力混入判定。
- Secret 输出只显示 sourceRef、presence、key 名和 fingerprint,不显示值。
- `config/platform-infra/k3s-clusters.yaml#targets.d601` 是 D601 单节点集群的
唯一拓扑真相,只包含节点、server、出网 proxy 引用和只读 artifact
registry;不得出现 NC01、agent token、跨地域 CNI 或共享调度参数。
- NC01 和 D601 是两个完全独立的 Kubernetes 调度域。禁止以 agent、
WireGuard、Flannel/VXLAN、共享 datastore 或隐藏 bootstrap 参数重新连接
- D601 只运行 `k3s` server,不运行 `k3s-agent`。新业务必须由自己的
owning YAML 显式选择 D601NC01 的 PaC、Tekton、Argo、GitOps、
DaemonSet 和 namespace 不得默认复制。
- 跨地域只保留两类边界:
- 宿主 SSH/Provider 控制通道;
- 鉴权、只读的公网 HTTPS artifact registry
- D601 的公共依赖出网由 `config/platform-infra/host-proxy.yaml` 声明,
proxy drop-in 必须归 `k3s.service`;禁止公共 registry mirror。
- 受控入口为
`platform-infra k3s-cluster plan|apply|status|smoke|registry-smoke
--target d601`。`apply` 先收敛 host proxy,再比较 server 声明摘要
内容一致时不得重装或重启
- `status``smoke` 必须证明:
- 只有一个 Ready 节点;
- server active 且 agent inactive
- 全部 `kube-system` Deployment Ready
- `kube-system` Pod 为零
- NC01 节点配置独立归 `config/hwlab-node-control-plane.yaml`,不得引用
D601 集群配置。确需重启 NC01 k3s 时,受控入口先备份并移走派生
`server/cred/passwd`,该 freshness 事实只能作为非阻塞 warning。
## Hyper-V Ubuntu Provider VM
- `config/platform-infra/hyperv-vms.yaml` 是 Windows Hyper-V Ubuntu Provider VM 的唯一配置真相,完整需求见 `project-management/PJ2026-01/specs/PJ2026-01060314-hyperv-ubuntu-vm.md`
@@ -228,7 +136,7 @@
- NC01 集群内 registry 的内部 endpoint 是唯一写入 authority
- CI、BuildKit 和 publish 继续使用内部 `127.0.0.1:5000` 语义;
- 公网 `registry.hwpod.com` 只用于异地 worker 拉取,不接受 push 或删除。
- 公网 `registry.hwpod.com` 只用于 D601 独立集群拉取,不接受 push 或删除。
- 公网入口归共享 public-edge 管理:
- hostname、DNS、upstream、允许方法和鉴权 SecretRef 来自集群 owning YAML
- Caddy 只允许 `GET``HEAD`,其余方法返回拒绝;
@@ -239,24 +147,23 @@
后仅按 key 继承环境,禁止用语义不同的 `docker run --env-file`,也禁止
手工重启容器;
- 可信 TLS 由 public-edge 管理,镜像层不得经过 CDN 代理。
- worker containerd 归集群 CLI 管理:
- D601 containerd 归集群 CLI 管理:
- 既有 `127.0.0.1:5000/...` 镜像引用保持不变;
- `registries.yaml` 把该引用映射到 YAML 声明的公网 HTTPS endpoint
- 用户名和密码只从同一 SecretRef 渲染;
- registry 配置单独变化时只重启 agent 使配置生效,不卸载或重装 k3s。
- registry 配置变化时由 server 受控加载,不卸载或重装 k3s。
- 受控入口:
```bash
bun scripts/cli.ts platform-infra k3s-cluster registry-credential-init --target nc01-d601 --confirm
bun scripts/cli.ts platform-infra k3s-cluster apply --target nc01-d601 --confirm
bun scripts/cli.ts platform-infra k3s-cluster registry-smoke --target nc01-d601 --confirm
bun scripts/cli.ts platform-infra k3s-cluster registry-credential-init --target d601 --confirm
bun scripts/cli.ts platform-infra k3s-cluster apply --target d601 --confirm
bun scripts/cli.ts platform-infra k3s-cluster registry-smoke --target d601 --confirm
```
- `registry-smoke` 必须验证:
- 未鉴权拒绝、有效凭据、manifest digest、blob range 和写方法拒绝;
- YAML 选定的真实镜像冷拉取、并发拉取和重复拉取;
- 拉取期间 Kubernetes API 连通,前后 control-plane 与 worker 均为 Ready
- Provider 控制通道能返回结构化终态;
- YAML 选定的真实镜像拉取和摘要
- 拉取后 D601 Node 仍为 Ready
- 结束后不保留临时 Kubernetes workload。
## Gitea 与 Pipelines-as-Code 边界
@@ -32,7 +32,11 @@
### 3.1 HYPERV-VM-REQ-001 YAML 真相
虚拟机名称、宿主 route、CPU、内存、磁盘、镜像 URL 与摘要、网络、Ubuntu 用户、k3s 版本和 Provider ID 必须来自 `config/platform-infra/hyperv-vms.yaml`。跨主机集群拓扑、WireGuard、k3s server/agent 角色、SecretRef、registry mirror 和调度 smoke 必须来自独立的集群 owning YAML。实现只能校验、解析引用和渲染,不能维护隐藏运行参数。
虚拟机名称、宿主 route、CPU、内存、磁盘、镜像 URL 与摘要、网络、
Ubuntu 用户、k3s 版本和 Provider ID 必须来自
`config/platform-infra/hyperv-vms.yaml`。D601 独立 server、host proxy、
registry endpoint 和拉取验收必须来自集群 owning YAML。实现只能校验、
解析引用和渲染,不能维护隐藏运行参数。
### 3.2 HYPERV-VM-REQ-002 稳定资源边界
@@ -79,23 +83,20 @@ Provider TCP 数据池恢复必须满足:
D601-VM 与 NC01 必须作为两个独立集群运行:
- `config/platform-infra/k3s-clusters.yaml` 必须显式声明
`topologyMode=separated` 和 D601 `k3s.standalone` 参数。
- `config/platform-infra/k3s-clusters.yaml#targets.d601` 只声明 D601
独立 server,不得出现 NC01、agent token 或跨地域 CNI 参数。
- `config/platform-infra/hyperv-vms.yaml` 必须以
`management.mode=standalone` 引用独立集群配置。
- D601 只运行单节点 k3s server;旧 `k3s-agent`、agent datastore、CNI
和跨地域 WireGuard peer 必须由受控拆分入口退役
- 拆分前必须证明 D601 非 DaemonSet Pod 为零;D601 独立 server Ready
后,才能从 NC01 删除旧 Node 和 WireGuard peer
- 拆分后 D601 非 `kube-system` Pod 必须为零,NC01 不得存在 D601 Node。
- CI/CD、GitOps、VM bootstrap 和业务 renderer 禁止重新安装 agent、
恢复 agent token、重建跨地域 CNI 或把 NC01 workload 自动投递到 D601。
- D601 只运行单节点 k3s server`k3s-agent` 必须 inactive。
- D601 非 `kube-system` Pod 必须为零,全部系统 Deployment 必须 Ready
- CI/CD、GitOps、VM bootstrap 和业务 renderer 禁止安装 agent、创建
跨地域 CNI 或把 NC01 workload 自动投递到 D601
- D601 host proxy 可继续为独立集群提供公共依赖出网,不得依赖公共镜像站
或新增公共 registry mirror。
- 集群自有 artifact registry 可以通过公网可信 TLS 向异地 worker 提供鉴权只读拉取:
- 集群自有 artifact registry 可以通过公网可信 TLS 向 D601 提供鉴权只读拉取:
- 内部 CI/publish 继续使用集群内写入 endpoint,公网入口不得成为第二个 push authority。
- 公网边缘只允许 Registry V2 拉取所需的 `GET``HEAD`,其他方法必须在反向代理层拒绝。
- endpoint、DNS、上游、允许方法、凭据 `sourceRef`、镜像摘要和拉取验收参数必须来自集群 owning YAML。
- D601 containerd 继续消费既有镜像引用,只把该内部引用映射到声明的公网 HTTPS endpoint。
- artifact 大文件不得经过 WireGuard、Flannel/VXLAN 或受传输时长和对象大小限制的 CDN。
- artifact 大文件不得经过 Kubernetes 数据面或受传输时长和对象大小限制的 CDN。
- 宿主内网 SSH 可以作为 D601 的稳定部署控制通道,provider route 只作为部署后验收入口。
+3 -3
View File
@@ -894,9 +894,9 @@ function platformInfraHelpSummary(): unknown {
"bun scripts/cli.ts platform-infra hyperv-vm plan --target D601-VM",
"bun scripts/cli.ts platform-infra hyperv-vm apply --target D601-VM --confirm",
"bun scripts/cli.ts platform-infra hyperv-vm status --target D601-VM",
"bun scripts/cli.ts platform-infra k3s-cluster plan --target nc01-d601",
"bun scripts/cli.ts platform-infra k3s-cluster apply --target nc01-d601 --confirm",
"bun scripts/cli.ts platform-infra k3s-cluster smoke --target nc01-d601 --confirm",
"bun scripts/cli.ts platform-infra k3s-cluster plan --target d601",
"bun scripts/cli.ts platform-infra k3s-cluster apply --target d601 --confirm",
"bun scripts/cli.ts platform-infra k3s-cluster smoke --target d601",
"bun scripts/cli.ts platform-infra hyperv-vm verify --target D601-VM [--repair-provider-channel --confirm]",
"bun scripts/cli.ts platform-infra langbot plan",
"bun scripts/cli.ts platform-infra langbot apply --confirm",
+1 -6
View File
@@ -201,12 +201,7 @@ export function prepareHostProxyGuestBundle(targetId: string): HostProxyGuestBun
remoteArchivePath: target.source.client.archiveInstallPath,
archiveSha256: target.source.client.archiveSha256,
prepareScript: remotePrepareScript(target.source.client),
applyScript: `${remoteApplyScript(target, material.clientConfigJson)}
if [ "$changed" = true ] && [ -s /etc/systemd/system/k3s-agent.service ]; then
systemctl daemon-reload
systemctl restart --no-block k3s-agent
fi
`,
applyScript: remoteApplyScript(target, material.clientConfigJson),
statusScript: remoteStatusScript(target),
sourceFingerprint: target.source.fingerprint,
materialFingerprint: material.fingerprint,
+54 -258
View File
@@ -6,69 +6,23 @@ export const K3S_CLUSTER_CONFIG_LABEL = "config/platform-infra/k3s-clusters.yaml
export interface K3sClusterTarget {
id: string;
clusterName: string;
topologyMode: "joined" | "separated";
controlPlane: {
id: string;
configRef: string;
lane: string;
nodeName: string;
publicAddress: string;
route: string;
kubeRoute: string;
};
worker: {
node: {
id: string;
bootstrapConfigRef: string;
hostProxyConfigRef: string;
hostProxyTargetId: string;
nodeName: string;
labels: Record<string, string>;
scheduling: {
schedulable: boolean;
taints: Array<{ key: string; value: string; effect: "NoSchedule" | "PreferNoSchedule" | "NoExecute" }>;
};
windowsRoute: string;
guestAddress: string;
guestUser: string;
stateDir: string;
k3sVersion: string;
};
wireguard: {
interfaceName: string;
packageName: string;
mtu: number;
listenPort: number;
controlPlaneAddress: string;
workerAddress: string;
endpoint: string;
persistentKeepaliveSeconds: number;
};
tcpTuning: {
congestionControl: "bbr" | "cubic";
defaultQdisc: "fq" | "fq_codel";
mtuProbing: 1 | 2;
sysctlPath: string;
};
secrets: {
sourceRef: string;
controlPlanePrivateKey: string;
controlPlanePublicKey: string;
workerPrivateKey: string;
workerPublicKey: string;
agentToken: string;
};
k3s: {
serverUrl: string;
agentTokenFile: string;
flannelInterface: string;
writeKubeconfigMode: "0600" | "0640" | "0644";
maxPods: number;
standalone: {
clusterName: string;
nodeName: string;
writeKubeconfigMode: string;
labels: Record<string, string>;
disableComponents: string[];
};
disableComponents: string[];
registryMirror: {
source: string;
endpoint: string;
@@ -86,23 +40,13 @@ export interface K3sClusterTarget {
concurrentPulls: number;
timeoutSeconds: number;
};
serviceLb: { enabledNodeLabel: string; enabledNode: string };
workerDns: {
namespace: string;
deploymentName: string;
serviceName: string;
clusterIp: string;
image: string;
configMapName: string;
customConfigMapName: string;
replicas: number;
timeoutSeconds: number;
};
};
smoke: { namespace: string; jobName: string; image: string; timeoutSeconds: number };
}
export function readK3sClusterConfig(targetId?: string): { defaultTargetId: string; target: K3sClusterTarget } {
export function readK3sClusterConfig(targetId?: string): {
defaultTargetId: string;
target: K3sClusterTarget;
} {
const root = record(Bun.YAML.parse(readFileSync(rootPath(K3S_CLUSTER_CONFIG_LABEL), "utf8")), K3S_CLUSTER_CONFIG_LABEL);
if (root.kind !== "platform-infra-k3s-clusters") throw new Error(`${K3S_CLUSTER_CONFIG_LABEL}.kind must be platform-infra-k3s-clusters`);
const defaults = record(root.defaults, `${K3S_CLUSTER_CONFIG_LABEL}.defaults`);
@@ -110,164 +54,63 @@ export function readK3sClusterConfig(targetId?: string): { defaultTargetId: stri
const id = targetId ?? defaultTargetId;
const raw = record(record(root.targets, `${K3S_CLUSTER_CONFIG_LABEL}.targets`)[id], `${K3S_CLUSTER_CONFIG_LABEL}.targets.${id}`);
if (raw.enabled !== true) throw new Error(`${K3S_CLUSTER_CONFIG_LABEL}.targets.${id}.enabled must be true`);
const control = record(raw.controlPlane, `targets.${id}.controlPlane`);
const workers = record(raw.workers, `targets.${id}.workers`);
const workerEntries = Object.entries(workers);
if (workerEntries.length !== 1) throw new Error(`targets.${id}.workers must contain exactly one worker`);
const [workerId, workerRaw] = workerEntries[0] ?? [];
const worker = record(workerRaw, `targets.${id}.workers.${workerId}`);
const wireguard = record(raw.wireguard, `targets.${id}.wireguard`);
const tcpTuning = record(raw.tcpTuning, `targets.${id}.tcpTuning`);
const secrets = record(raw.secrets, `targets.${id}.secrets`);
const node = record(raw.node, `targets.${id}.node`);
const k3s = record(raw.k3s, `targets.${id}.k3s`);
const standalone = record(k3s.standalone, `targets.${id}.k3s.standalone`);
const mirror = record(k3s.registryMirror, `targets.${id}.k3s.registryMirror`);
const mirrorAuth = record(mirror.auth, `targets.${id}.k3s.registryMirror.auth`);
const registryPullSmoke = record(k3s.registryPullSmoke, `targets.${id}.k3s.registryPullSmoke`);
const serviceLb = record(k3s.serviceLb, `targets.${id}.k3s.serviceLb`);
const workerDns = record(k3s.workerDns, `targets.${id}.k3s.workerDns`);
const smoke = record(raw.smoke, `targets.${id}.smoke`);
const controlSource = resolveRef(text(control, "configRef", `targets.${id}.controlPlane`));
const bootstrapSource = resolveRef(text(worker, "bootstrapConfigRef", `targets.${id}.workers[0]`));
const host = record(bootstrapSource.host, "worker bootstrap host");
const network = record(bootstrapSource.network, "worker bootstrap network");
const ubuntu = record(bootstrapSource.ubuntu, "worker bootstrap ubuntu");
const bootstrapK3s = record(bootstrapSource.k3s, "worker bootstrap k3s");
const labels = stringMap(record(worker.labels, `targets.${id}.workers.${workerId}.labels`), `targets.${id}.workers.${workerId}.labels`);
const scheduling = record(worker.scheduling, `targets.${id}.workers.${workerId}.scheduling`);
const taints = objectArray(scheduling.taints, `targets.${id}.workers.${workerId}.scheduling.taints`).map((item, index) => ({
key: kubernetesLabelKey(item, "key", `targets.${id}.workers.${workerId}.scheduling.taints[${index}]`),
value: kubernetesLabelValue(item, "value", `targets.${id}.workers.${workerId}.scheduling.taints[${index}]`),
effect: enumText(item, "effect", `targets.${id}.workers.${workerId}.scheduling.taints[${index}]`, ["NoSchedule", "PreferNoSchedule", "NoExecute"]),
}));
const target: K3sClusterTarget = {
id,
clusterName: text(raw, "clusterName", `targets.${id}`),
topologyMode: enumText(raw, "topologyMode", `targets.${id}`, ["joined", "separated"]),
controlPlane: {
id: text(control, "id", `targets.${id}.controlPlane`),
configRef: text(control, "configRef", `targets.${id}.controlPlane`),
lane: text(control, "lane", `targets.${id}.controlPlane`),
nodeName: text(control, "nodeName", `targets.${id}.controlPlane`),
publicAddress: text(control, "publicAddress", `targets.${id}.controlPlane`),
route: text(controlSource, "route", "control plane configRef"),
kubeRoute: text(controlSource, "kubeRoute", "control plane configRef"),
},
worker: {
id: text(worker, "id", `targets.${id}.workers.${workerId}`),
bootstrapConfigRef: text(worker, "bootstrapConfigRef", `targets.${id}.workers.${workerId}`),
hostProxyConfigRef: text(worker, "hostProxyConfigRef", `targets.${id}.workers.${workerId}`),
hostProxyTargetId: hostProxyTargetId(text(worker, "hostProxyConfigRef", `targets.${id}.workers.${workerId}`)),
nodeName: text(worker, "nodeName", `targets.${id}.workers.${workerId}`),
labels,
scheduling: {
schedulable: boolean(scheduling, "schedulable", `targets.${id}.workers.${workerId}.scheduling`),
taints,
const pullSmoke = record(k3s.registryPullSmoke, `targets.${id}.k3s.registryPullSmoke`);
const bootstrap = resolveRef(text(node, "bootstrapConfigRef", `targets.${id}.node`));
const host = record(bootstrap.host, "node bootstrap host");
const network = record(bootstrap.network, "node bootstrap network");
const ubuntu = record(bootstrap.ubuntu, "node bootstrap ubuntu");
const bootstrapK3s = record(bootstrap.k3s, "node bootstrap k3s");
const hostProxyConfigRef = text(node, "hostProxyConfigRef", `targets.${id}.node`);
return {
defaultTargetId,
target: {
id,
clusterName: kubernetesName(raw, "clusterName", `targets.${id}`),
node: {
id: text(node, "id", `targets.${id}.node`),
bootstrapConfigRef: text(node, "bootstrapConfigRef", `targets.${id}.node`),
hostProxyConfigRef,
hostProxyTargetId: hostProxyTargetId(hostProxyConfigRef),
nodeName: kubernetesName(node, "nodeName", `targets.${id}.node`),
labels: stringMap(record(node.labels, `targets.${id}.node.labels`), `targets.${id}.node.labels`),
windowsRoute: text(host, "route", "node bootstrap host"),
guestAddress: text(network, "address", "node bootstrap network"),
guestUser: text(ubuntu, "user", "node bootstrap ubuntu"),
stateDir: text(host, "stateDir", "node bootstrap host"),
k3sVersion: text(bootstrapK3s, "version", "node bootstrap k3s"),
},
windowsRoute: text(host, "route", "worker bootstrap host"),
guestAddress: text(network, "address", "worker bootstrap network"),
guestUser: text(ubuntu, "user", "worker bootstrap ubuntu"),
stateDir: text(host, "stateDir", "worker bootstrap host"),
k3sVersion: text(bootstrapK3s, "version", "worker bootstrap k3s"),
},
wireguard: {
interfaceName: text(wireguard, "interfaceName", `targets.${id}.wireguard`),
packageName: text(wireguard, "packageName", `targets.${id}.wireguard`),
mtu: integer(wireguard, "mtu", `targets.${id}.wireguard`),
listenPort: integer(wireguard, "listenPort", `targets.${id}.wireguard`),
controlPlaneAddress: cidr(wireguard, "controlPlaneAddress", `targets.${id}.wireguard`),
workerAddress: cidr(wireguard, "workerAddress", `targets.${id}.wireguard`),
endpoint: text(wireguard, "endpoint", `targets.${id}.wireguard`),
persistentKeepaliveSeconds: integer(wireguard, "persistentKeepaliveSeconds", `targets.${id}.wireguard`),
},
tcpTuning: {
congestionControl: enumText(tcpTuning, "congestionControl", `targets.${id}.tcpTuning`, ["bbr", "cubic"]),
defaultQdisc: enumText(tcpTuning, "defaultQdisc", `targets.${id}.tcpTuning`, ["fq", "fq_codel"]),
mtuProbing: oneOfInteger(tcpTuning, "mtuProbing", `targets.${id}.tcpTuning`, [1, 2]),
sysctlPath: text(tcpTuning, "sysctlPath", `targets.${id}.tcpTuning`),
},
secrets: {
sourceRef: text(secrets, "sourceRef", `targets.${id}.secrets`),
controlPlanePrivateKey: text(secrets, "controlPlanePrivateKey", `targets.${id}.secrets`),
controlPlanePublicKey: text(secrets, "controlPlanePublicKey", `targets.${id}.secrets`),
workerPrivateKey: text(secrets, "workerPrivateKey", `targets.${id}.secrets`),
workerPublicKey: text(secrets, "workerPublicKey", `targets.${id}.secrets`),
agentToken: text(secrets, "agentToken", `targets.${id}.secrets`),
},
k3s: {
serverUrl: text(k3s, "serverUrl", `targets.${id}.k3s`),
agentTokenFile: text(k3s, "agentTokenFile", `targets.${id}.k3s`),
flannelInterface: text(k3s, "flannelInterface", `targets.${id}.k3s`),
maxPods: integer(k3s, "maxPods", `targets.${id}.k3s`),
standalone: {
clusterName: kubernetesName(standalone, "clusterName", "k3s.standalone"),
nodeName: kubernetesName(standalone, "nodeName", "k3s.standalone"),
writeKubeconfigMode: enumText(standalone, "writeKubeconfigMode", "k3s.standalone", ["0600", "0640", "0644"]),
labels: stringMap(record(standalone.labels, "k3s.standalone.labels"), "k3s.standalone.labels"),
disableComponents: stringArray(standalone.disableComponents, "k3s.standalone.disableComponents"),
},
registryMirror: {
source: text(mirror, "source", "registryMirror"),
endpoint: httpsUrl(mirror, "endpoint", "registryMirror"),
auth: {
sourceRef: text(mirrorAuth, "sourceRef", "registryMirror.auth"),
usernameKey: text(mirrorAuth, "usernameKey", "registryMirror.auth"),
passwordKey: text(mirrorAuth, "passwordKey", "registryMirror.auth"),
passwordHashKey: text(mirrorAuth, "passwordHashKey", "registryMirror.auth"),
k3s: {
writeKubeconfigMode: enumText(k3s, "writeKubeconfigMode", `targets.${id}.k3s`, ["0600", "0640", "0644"]),
maxPods: integer(k3s, "maxPods", `targets.${id}.k3s`),
disableComponents: stringArray(k3s.disableComponents, `targets.${id}.k3s.disableComponents`),
registryMirror: {
source: text(mirror, "source", "registryMirror"),
endpoint: httpsUrl(mirror, "endpoint", "registryMirror"),
auth: {
sourceRef: text(mirrorAuth, "sourceRef", "registryMirror.auth"),
usernameKey: text(mirrorAuth, "usernameKey", "registryMirror.auth"),
passwordKey: text(mirrorAuth, "passwordKey", "registryMirror.auth"),
passwordHashKey: text(mirrorAuth, "passwordHashKey", "registryMirror.auth"),
},
},
registryPullSmoke: {
image: text(pullSmoke, "image", "registryPullSmoke"),
expectedDigest: sha256Digest(pullSmoke, "expectedDigest", "registryPullSmoke"),
expectedBytes: integer(pullSmoke, "expectedBytes", "registryPullSmoke"),
concurrentPulls: integer(pullSmoke, "concurrentPulls", "registryPullSmoke"),
timeoutSeconds: integer(pullSmoke, "timeoutSeconds", "registryPullSmoke"),
},
},
registryPullSmoke: {
image: text(registryPullSmoke, "image", "registryPullSmoke"),
expectedDigest: sha256Digest(registryPullSmoke, "expectedDigest", "registryPullSmoke"),
expectedBytes: integer(registryPullSmoke, "expectedBytes", "registryPullSmoke"),
concurrentPulls: integer(registryPullSmoke, "concurrentPulls", "registryPullSmoke"),
timeoutSeconds: integer(registryPullSmoke, "timeoutSeconds", "registryPullSmoke"),
},
serviceLb: { enabledNodeLabel: text(serviceLb, "enabledNodeLabel", "serviceLb"), enabledNode: text(serviceLb, "enabledNode", "serviceLb") },
workerDns: {
namespace: kubernetesName(workerDns, "namespace", "workerDns"),
deploymentName: kubernetesName(workerDns, "deploymentName", "workerDns"),
serviceName: kubernetesName(workerDns, "serviceName", "workerDns"),
clusterIp: ipv4Address(workerDns, "clusterIp", "workerDns"),
image: text(workerDns, "image", "workerDns"),
configMapName: kubernetesName(workerDns, "configMapName", "workerDns"),
customConfigMapName: kubernetesName(workerDns, "customConfigMapName", "workerDns"),
replicas: integer(workerDns, "replicas", "workerDns"),
timeoutSeconds: integer(workerDns, "timeoutSeconds", "workerDns"),
},
},
smoke: {
namespace: text(smoke, "namespace", `targets.${id}.smoke`),
jobName: text(smoke, "jobName", `targets.${id}.smoke`),
image: text(smoke, "image", `targets.${id}.smoke`),
timeoutSeconds: integer(smoke, "timeoutSeconds", `targets.${id}.smoke`),
},
};
if (target.k3s.flannelInterface !== target.wireguard.interfaceName) throw new Error(`targets.${id}.k3s.flannelInterface must match wireguard.interfaceName`);
if (target.k3s.serviceLb.enabledNode !== target.controlPlane.nodeName) throw new Error(`targets.${id}.k3s.serviceLb.enabledNode must match controlPlane.nodeName`);
if (!target.worker.scheduling.schedulable && !target.worker.scheduling.taints.some((taint) => taint.effect === "NoSchedule" || taint.effect === "NoExecute")) {
throw new Error(`targets.${id}.workers.${workerId}.scheduling must declare a NoSchedule or NoExecute taint when schedulable=false`);
}
if (!target.tcpTuning.sysctlPath.startsWith("/etc/sysctl.d/")) throw new Error(`targets.${id}.tcpTuning.sysctlPath must stay under /etc/sysctl.d`);
return { defaultTargetId, target };
}
export function clusterServerArgs(configRef: string): string[] {
const match = configRef.match(/^config\/platform-infra\/k3s-clusters\.yaml#targets\.([A-Za-z0-9._-]+)\.controlPlane$/u);
if (match === null) throw new Error(`unsupported k3s clusterConfigRef: ${configRef}`);
const { target } = readK3sClusterConfig(match[1]);
return [
"--node-ip", address(target.wireguard.controlPlaneAddress),
"--node-external-ip", target.controlPlane.publicAddress,
"--flannel-iface", target.k3s.flannelInterface,
"--tls-san", address(target.wireguard.controlPlaneAddress),
"--tls-san", target.controlPlane.publicAddress,
"--agent-token-file", target.k3s.agentTokenFile,
];
}
export function address(value: string): string {
return value.split("/", 1)[0] ?? value;
throw new Error(`external k3s cluster attachment is retired; remove clusterConfigRef: ${configRef}`);
}
function resolveRef(value: string): Record<string, unknown> {
@@ -295,17 +138,6 @@ function integer(value: Record<string, unknown>, key: string, label: string): nu
return Number(result);
}
function boolean(value: Record<string, unknown>, key: string, label: string): boolean {
const result = value[key];
if (typeof result !== "boolean") throw new Error(`${label}.${key} must be a boolean`);
return result;
}
function objectArray(value: unknown, label: string): Array<Record<string, unknown>> {
if (!Array.isArray(value)) throw new Error(`${label} must be an array`);
return value.map((item, index) => record(item, `${label}[${index}]`));
}
function httpsUrl(value: Record<string, unknown>, key: string, label: string): string {
const result = text(value, key, label);
const parsed = new URL(result);
@@ -327,26 +159,6 @@ function enumText<const T extends string>(value: Record<string, unknown>, key: s
return result as T;
}
function oneOfInteger<const T extends number>(value: Record<string, unknown>, key: string, label: string, allowed: readonly T[]): T {
const result = integer(value, key, label);
if (!allowed.includes(result as T)) throw new Error(`${label}.${key} must be one of ${allowed.join(", ")}`);
return result as T;
}
function cidr(value: Record<string, unknown>, key: string, label: string): string {
const result = text(value, key, label);
if (!/^\d{1,3}(?:\.\d{1,3}){3}\/\d{1,2}$/u.test(result)) throw new Error(`${label}.${key} must be an IPv4 CIDR`);
return result;
}
function ipv4Address(value: Record<string, unknown>, key: string, label: string): string {
const result = text(value, key, label);
if (!/^\d{1,3}(?:\.\d{1,3}){3}$/u.test(result) || result.split(".").some((part) => Number(part) > 255)) {
throw new Error(`${label}.${key} must be an IPv4 address`);
}
return result;
}
function kubernetesName(value: Record<string, unknown>, key: string, label: string): string {
const result = text(value, key, label);
if (!/^[a-z0-9](?:[-a-z0-9]*[a-z0-9])?$/u.test(result) || result.length > 63) {
@@ -355,22 +167,6 @@ function kubernetesName(value: Record<string, unknown>, key: string, label: stri
return result;
}
function kubernetesLabelKey(value: Record<string, unknown>, key: string, label: string): string {
const result = text(value, key, label);
if (!/^(?:[a-z0-9](?:[-a-z0-9.]*[a-z0-9])?\/)?[A-Za-z0-9](?:[-A-Za-z0-9_.]*[A-Za-z0-9])?$/u.test(result) || result.length > 253) {
throw new Error(`${label}.${key} must be a Kubernetes label key`);
}
return result;
}
function kubernetesLabelValue(value: Record<string, unknown>, key: string, label: string): string {
const result = text(value, key, label);
if (!/^[A-Za-z0-9](?:[-A-Za-z0-9_.]*[A-Za-z0-9])?$/u.test(result) || result.length > 63) {
throw new Error(`${label}.${key} must be a Kubernetes label value`);
}
return result;
}
function stringMap(value: Record<string, unknown>, label: string): Record<string, string> {
return Object.fromEntries(Object.entries(value).map(([key, item]) => {
if (typeof item !== "string" || item.length === 0) throw new Error(`${label}.${key} must be a non-empty string`);
@@ -387,6 +183,6 @@ function stringArray(value: unknown, label: string): string[] {
function hostProxyTargetId(ref: string): string {
const match = ref.match(/^config\/platform-infra\/host-proxy\.yaml#targets\.([A-Za-z0-9._-]+)$/u);
if (match === null) throw new Error(`worker hostProxyConfigRef must point at config/platform-infra/host-proxy.yaml#targets.<id>`);
if (match === null) throw new Error(`hostProxyConfigRef must point at config/platform-infra/host-proxy.yaml#targets.<id>`);
return match[1] ?? "";
}
File diff suppressed because it is too large Load Diff