refactor(platform-infra): retire cross-region k3s topology
This commit is contained in:
@@ -348,44 +348,29 @@ bun scripts/cli.ts hwlab nodes control-plane legacy-cicd --help
|
||||
- 一旦发现 CI/CD CLI 被误用且可能写入错误状态、产生伪证据或绕过目标运行面,必须立刻先把用法改成更符合直觉的公开入口并更新本 skill/reference,再继续验证或交付;不要只靠口头记忆、隐藏 flag、手动约定或后续小心来避免复发。内部 in-cluster 模式必须只由目标 k8s Job/Pod 调用,操作者从本机只能用公开入口提交目标侧 Job 或读取目标侧摘要。
|
||||
- PaC migrated consumer 的旧手动 publish/debug 命令不得作为正式 closeout 引导;调查命令默认只能指向只读的 `platform-infra pipelines-as-code status|history`。`closeout` 仅保留只读兼容能力,不得作为主代理、子代理或操作者的 PR 合并后步骤。需要保留的旧 mutation 命令只能属于 YAML 明确的 legacy authority 或独立平台维护,不得标记或引导 manual recovery;单步确认的自动链缺陷必须回到 YAML/controller/源码修复。
|
||||
- Secret 只通过 YAML sourceRef/targetKey 和受控 CLI 下发;输出只披露 presence/fingerprint。
|
||||
- 多主机 k3s 节点变更:
|
||||
- CI/CD、业务 renderer、PipelineRun 和普通 GitOps workload 不得直接写 WireGuard 配置、k3s systemd unit、agent token 文件或节点 ServiceLB 标签。
|
||||
- 节点加入、状态和调度 smoke 只走 `platform-infra k3s-cluster plan|apply|status|smoke`,具体 authority 和控制通道见 `docs/reference/platform-infra.md#多主机-k3s-集群`。
|
||||
- 集群 apply 的 control-plane 阶段只允许组合 `hwlab nodes control-plane infra k3s-config` 窄域;禁止调用全量 `infra apply`,避免节点加入顺带覆盖 registry、Tekton、Argo、git-mirror 或运行面对象。
|
||||
- k3s drop-in 未变化且节点容量已收敛时必须 `mutation=false`;禁止无条件重启、重装、强写或通过 CI 自动覆盖同一节点配置。
|
||||
- WireGuard 配置一致时不得 restart;k3s API/容量暂时不可读时保持 unknown 并停止 mutation,禁止把控制通道抖动解释为配置漂移。
|
||||
- worker agent 只在 YAML 派生摘要变化或 service 未运行时安装/恢复,不得在每次集群 apply 中先卸载再重装。
|
||||
- worker 调度资格和 taint 只认集群 owning YAML:
|
||||
- `schedulable=false` 必须由 agent 注册参数与受控 `cordon`/taint 双重收敛;
|
||||
- CI、PipelineRun、GitOps 和业务 renderer 禁止 `uncordon`、删除该 taint,
|
||||
或添加 toleration/`nodeName` 将业务 Pod 强制放回不可调度 worker;
|
||||
- 清空不可调度 worker 只走 `platform-infra k3s-cluster drain --confirm`,
|
||||
保留 DaemonSet/节点系统 Pod,且节点重注册后必须先恢复调度隔离。
|
||||
- `Type=notify` agent 的 `activating` 且 MainPID 存在属于运行中;节点 Ready 时不得仅因 systemd 未进入 `active` 而重复重装。
|
||||
- worker 必须先由集群入口按 `hostProxyConfigRef` 部署并验证 proxy client,再启动 agent;containerd 通过该 proxy 直接访问上游 registry,禁止用公共 registry mirror 代替出网链路。
|
||||
- D601 独立 k3s 集群:
|
||||
- CI/CD、业务 renderer、PipelineRun 和 GitOps 不得写 k3s systemd unit,
|
||||
不得把 D601 注册为 NC01 agent,也不得创建跨地域 CNI 或共享 datastore。
|
||||
- D601 只走
|
||||
`platform-infra k3s-cluster plan|apply|status|smoke --target d601`;
|
||||
NC01 只走自己的 control-plane 入口,两者不得互相组合。
|
||||
- D601 server 的 YAML 摘要未变化且节点健康时禁止重装、重启或强写。
|
||||
- D601 默认是空集群。任何业务进入 D601 必须有独立 workload owning
|
||||
YAML 和显式 target,禁止继承 NC01 的 PaC、Tekton、Argo、GitOps、
|
||||
DaemonSet、namespace 或默认 placement。
|
||||
- D601 先通过 host proxy 让 server/containerd 出网,禁止公共 registry
|
||||
mirror。
|
||||
- 集群自有 artifact registry 的异地拉取是明确例外:
|
||||
- 内部 publish endpoint 保持唯一写入 authority;
|
||||
- 公网 HTTPS 只允许鉴权 `GET`/`HEAD`,不得成为 push、cache export 或删除入口;
|
||||
- worker 保持既有镜像引用,由集群 owning YAML 渲染 endpoint 与 SecretRef;
|
||||
- registry 大流量不得经过 WireGuard/Flannel,也不得使用 CDN 或公共镜像站;
|
||||
- D601 保持既有镜像引用,由集群 owning YAML 渲染 endpoint 与 SecretRef;
|
||||
- registry 大流量不得经过 Kubernetes 数据面,也不得使用 CDN 或公共镜像站;
|
||||
- bcrypt hash 必须以 Compose-safe 字面量写入 Secret source,且
|
||||
public-edge 用脱敏 material fingerprint 驱动凭据变更后的 recreate;
|
||||
候选校验不得用会保留引号的 `docker run --env-file` 改变 Secret 语义;
|
||||
- 验收统一走 `platform-infra k3s-cluster registry-smoke`,不手工 patch Caddy 或 containerd。
|
||||
- worker 安装使用短启动并由 control-plane Node Ready 轮询收口,禁止让 `trans`/SSH 长挂等待 systemd notify。
|
||||
- worker 加入集群不改变 PaC consumer 的 source commit authority,也不授权人工 PipelineRun、Argo sync 或业务 rollout。
|
||||
- NC01 与 D601 是永久分离的调度域:
|
||||
- `config/platform-infra/k3s-clusters.yaml` 必须保持
|
||||
`topologyMode=separated`,D601 只作为独立单节点 k3s 集群存在;
|
||||
- CI/CD、PaC、Tekton、Argo、GitOps、VM bootstrap 和业务 renderer
|
||||
禁止把 D601 注册为 NC01 agent,禁止恢复跨地域 WireGuard peer,
|
||||
禁止把 NC01 workload、DaemonSet 或 namespace 默认复制到 D601;
|
||||
- 拆分和验收只走
|
||||
`platform-infra k3s-cluster separate|status|smoke`;
|
||||
- 跨地域 artifact 传输使用鉴权、只读的公网 HTTPS registry,
|
||||
不得通过 Kubernetes Pod 网络、Service、DNS 或 VXLAN;
|
||||
- 如未来确需跨地域计算,必须建立独立 workload 级交付规格并显式选择
|
||||
D601 集群,不得通过重新组成单一集群获得隐式调度。
|
||||
- 跨地域 artifact 传输使用鉴权、只读的公网 HTTPS registry,不得通过
|
||||
Kubernetes Pod 网络、Service、DNS 或 VXLAN。
|
||||
- 长命令用异步 job 或短轮询;不要长时间挂住 trans/ssh。
|
||||
- Web 哨兵 YAML cadence 变更不新增发布 orchestrator:提交前用 `web-probe sentinel validate --local`;合并后用只读 `web-probe sentinel control-plane status --wait`。线上 desired schedule 必须来自 GitOps 清单,不能用操作者旧 worktree 的本地渲染值做阻塞门禁;收敛差异统一为非阻塞 warning。
|
||||
|
||||
|
||||
@@ -88,38 +88,20 @@ description: UniDesk YAML-first 运维正规化技能。用户提到 ymal-first/
|
||||
- 非核心字段可以使用显式领域默认值,但必须同时生成 `blocking=false` warning;
|
||||
- 不得生成不可见默认值,也不得把合并后的大对象写成新的 source of truth。
|
||||
- 新 VPS 或裸节点 bootstrap 的出网必须优先走 YAML-first 声明的 0 依赖 host proxy client:通过 `trans` 分发零依赖客户端,先让 host route 能出网,再安装 k3s、containerd、Docker、包管理器依赖和 Git/gitrepo 同步;不得把 provider-gateway WebSocket egress 当作大二进制或依赖下载的数据面。
|
||||
- 多主机 k3s 集群:
|
||||
- 跨主机拓扑、WireGuard、agent token、worker host proxy 引用、集群内 artifact registry 映射、ServiceLB 节点和调度 smoke 归独立集群 owning YAML。
|
||||
- worker host proxy 归 `config/platform-infra/host-proxy.yaml`;集群 apply 必须先确认 proxy client 连接声明的 `vpn-server` 并通过外网 probe,再启动 agent。
|
||||
- worker proxy 前必须先收敛 WireGuard;配置一致时不重启,只有 unit 未运行或握手过期且隧道探针失败时才允许恢复 worker 接口。
|
||||
- containerd 通过 host proxy 直接访问公共 registry,禁止为公共依赖新增 registry mirror;集群内 artifact registry 映射不属于公共镜像代理。
|
||||
- 集群自有 artifact registry 可由同一 owning YAML 声明公网可信 TLS 的 pull-only endpoint:
|
||||
- 内部 endpoint 继续拥有 push;
|
||||
- public-edge 只允许鉴权 `GET`/`HEAD`;
|
||||
- containerd 凭据与 Caddy hash 必须来自同一 SecretRef;
|
||||
- 镜像层绕开 WireGuard/Flannel 和 CDN;
|
||||
- registry 配置变化不得触发 worker agent 卸载重装。
|
||||
- 既有 control-plane 的 k3s renderer 通过 `clusterConfigRef` 消费集群网络参数,不得由集群 CLI 再写第二个 systemd drop-in。
|
||||
- 无固定公网地址 worker 使用宿主内网 SSH 作为部署控制通道并主动连接 WireGuard endpoint,不使用 FRP 代替三层集群网络。
|
||||
- control-plane drop-in 只走 `hwlab nodes control-plane infra k3s-config` 窄域:
|
||||
- 渲染结果一致且节点容量已收敛时返回 `mutation=false`;
|
||||
- 只在必要时写入 drop-in、reload 和重启 k3s;
|
||||
- 确需重启 server 时,先停止服务并备份 k3s 派生
|
||||
`server/cred/passwd`;禁止用该文件与 datastore 的
|
||||
mtime/freshness 预判阻断变更,备份只输出
|
||||
`blocking=false` warning;
|
||||
- WireGuard 内容一致时不重启,API/容量读取暂时失败时保持 unknown 并停止 mutation;
|
||||
- worker agent 使用 YAML 派生摘要决定是否重装,禁止每次 apply 先卸载再覆盖;
|
||||
- 禁止借用全量 infra apply 顺带覆盖 registry、CI/CD 或业务运行面。
|
||||
- 日常入口和验收判定统一见 `docs/reference/platform-infra.md#多主机-k3s-集群`。
|
||||
- NC01-D601 的 owning YAML 必须声明 `topologyMode=separated`:
|
||||
- D601 独立 server 的节点名、标签、禁用组件和 kubeconfig mode 必须
|
||||
位于同一 YAML 的 `k3s.standalone`;
|
||||
- Hyper-V YAML 只能用 `management.mode=standalone` 引用该片段;
|
||||
- parser、bootstrap、CI/CD 和 GitOps 禁止通过隐藏默认值把 D601
|
||||
恢复为 agent、恢复 WireGuard peer或共享调度域;
|
||||
- 独立集群间只共享显式声明的只读 artifact endpoint,不共享 Pod、
|
||||
Service、DNS、CNI、datastore 或默认 workload。
|
||||
- D601 独立 k3s 集群:
|
||||
- `config/platform-infra/k3s-clusters.yaml#targets.d601` 只声明独立
|
||||
server、节点、host proxy 和只读 artifact registry,不得出现 NC01、
|
||||
agent token、跨地域 CNI 或共享调度字段。
|
||||
- Hyper-V YAML 以 `management.mode=standalone` 引用 D601 k3s 片段;
|
||||
parser、bootstrap、CI/CD 和 GitOps 禁止用隐藏默认值恢复 agent。
|
||||
- NC01 与 D601 不共享 Pod、Service、DNS、CNI、datastore、DaemonSet、
|
||||
namespace 或默认 workload。
|
||||
- D601 containerd 通过 host proxy 访问公共 registry,禁止公共镜像站;
|
||||
集群自有 artifact registry 只通过鉴权、只读公网 HTTPS 拉取。
|
||||
- registry 内部 endpoint 继续拥有 push,public-edge 只允许
|
||||
`GET`/`HEAD`;containerd 凭据与 Caddy hash 来自同一 SecretRef。
|
||||
- 日常入口和验收见
|
||||
`docs/reference/platform-infra.md#d601-独立-k3s-集群`。
|
||||
- 新节点 node-local registry 必须支持从零部署:优先由 YAML 渲染为 k3s workload/PVC;host Docker registry 只能是 legacy/迁移前状态。需要证明从零能力时先停旧 registry,不从旧 registry seed,并在状态中披露 `seededFromOldRegistry`、`zeroSeeded`、PVC、workload、endpoint 与 runtime/tools image preload 结果。
|
||||
- YAML 配置必须可复用、可继承、可组合;节点/lane 只能作为变量或 YAML value 参与渲染,禁止把具体 node/lane 写入 YAML 文件名、parser/renderer/helper 函数名或长期 schema 名。发现不同节点复制大段配置时,先抽 baseline/configRef/override,再继续扩展节点。
|
||||
- CLI `plan/status` 应输出 redacted 配置引用图:每个 ref 的文件、path、presence、摘要 hash、缺失字段和下一步 drill-down 命令。不要默认 dump 展开后的完整 YAML 或 Secret。
|
||||
|
||||
@@ -316,7 +316,6 @@ nodes:
|
||||
serviceName: k3s
|
||||
dropInPath: /etc/systemd/system/k3s.service.d/20-unidesk-node-config.conf
|
||||
nodeStatusName: v2202607378382480008
|
||||
clusterConfigRef: config/platform-infra/k3s-clusters.yaml#targets.nc01-d601.controlPlane
|
||||
execStartPre: []
|
||||
install:
|
||||
enabled: true
|
||||
|
||||
@@ -127,7 +127,7 @@ targets:
|
||||
route: D601-VM
|
||||
sourceRef: sources.d601-nc01-vpn-server-shadowsocks
|
||||
files:
|
||||
k3sSystemdDropIn: /etc/systemd/system/k3s-agent.service.d/10-unidesk-proxy.conf
|
||||
k3sSystemdDropIn: /etc/systemd/system/k3s.service.d/10-unidesk-proxy.conf
|
||||
env:
|
||||
noProxyRefs:
|
||||
common: noProxy.common
|
||||
|
||||
@@ -74,7 +74,7 @@ targets:
|
||||
k3s:
|
||||
management:
|
||||
mode: standalone
|
||||
configRef: config/platform-infra/k3s-clusters.yaml#targets.nc01-d601.k3s.standalone
|
||||
configRef: config/platform-infra/k3s-clusters.yaml#targets.d601.k3s
|
||||
version: v1.36.2+k3s1
|
||||
installMirror: cn
|
||||
binaryUrl: https://github.com/k3s-io/k3s/releases/download/v1.36.2%2Bk3s1/k3s
|
||||
|
||||
@@ -4,77 +4,32 @@ metadata:
|
||||
owner: unidesk
|
||||
specRef: project-management/PJ2026-01/specs/PJ2026-01060314-hyperv-ubuntu-vm.md
|
||||
defaults:
|
||||
targetId: nc01-d601
|
||||
targetId: d601
|
||||
targets:
|
||||
nc01-d601:
|
||||
d601:
|
||||
enabled: true
|
||||
clusterName: nc01
|
||||
topologyMode: separated
|
||||
controlPlane:
|
||||
id: NC01
|
||||
configRef: config/hwlab-node-control-plane.yaml#nodes.NC01
|
||||
lane: v03
|
||||
nodeName: v2202607378382480008
|
||||
publicAddress: 152.53.229.148
|
||||
workers:
|
||||
D601-VM:
|
||||
id: D601-VM
|
||||
bootstrapConfigRef: config/platform-infra/hyperv-vms.yaml#targets.D601-VM
|
||||
hostProxyConfigRef: config/platform-infra/host-proxy.yaml#targets.D601-VM
|
||||
nodeName: d601-vm
|
||||
labels:
|
||||
unidesk.ai/node-id: D601-VM
|
||||
unidesk.ai/provider-id: D601-VM
|
||||
unidesk.ai/runtime-plane: worker
|
||||
scheduling:
|
||||
schedulable: false
|
||||
taints:
|
||||
- key: unidesk.ai/business-scheduling
|
||||
value: disabled
|
||||
effect: NoSchedule
|
||||
wireguard:
|
||||
interfaceName: wg0
|
||||
packageName: wireguard-tools
|
||||
mtu: 1380
|
||||
listenPort: 51820
|
||||
controlPlaneAddress: 10.89.0.1/24
|
||||
workerAddress: 10.89.0.2/24
|
||||
endpoint: 152.53.229.148:51820
|
||||
persistentKeepaliveSeconds: 10
|
||||
tcpTuning:
|
||||
congestionControl: bbr
|
||||
defaultQdisc: fq
|
||||
mtuProbing: 1
|
||||
sysctlPath: /etc/sysctl.d/90-unidesk-k3s-cross-region.conf
|
||||
secrets:
|
||||
sourceRef: /root/.unidesk/.state/secrets/platform-infra/k3s-clusters/nc01-d601.env
|
||||
controlPlanePrivateKey: NC01_WIREGUARD_PRIVATE_KEY
|
||||
controlPlanePublicKey: NC01_WIREGUARD_PUBLIC_KEY
|
||||
workerPrivateKey: D601_VM_WIREGUARD_PRIVATE_KEY
|
||||
workerPublicKey: D601_VM_WIREGUARD_PUBLIC_KEY
|
||||
agentToken: K3S_AGENT_TOKEN
|
||||
clusterName: d601
|
||||
node:
|
||||
id: D601-VM
|
||||
bootstrapConfigRef: config/platform-infra/hyperv-vms.yaml#targets.D601-VM
|
||||
hostProxyConfigRef: config/platform-infra/host-proxy.yaml#targets.D601-VM
|
||||
nodeName: d601-vm
|
||||
labels:
|
||||
unidesk.ai/node-id: D601-VM
|
||||
unidesk.ai/provider-id: D601-VM
|
||||
unidesk.ai/runtime-plane: standalone
|
||||
k3s:
|
||||
serverUrl: https://10.89.0.1:6443
|
||||
agentTokenFile: /etc/unidesk/k3s-agent-token
|
||||
flannelInterface: wg0
|
||||
writeKubeconfigMode: "0644"
|
||||
maxPods: 500
|
||||
standalone:
|
||||
clusterName: d601
|
||||
nodeName: d601-vm
|
||||
writeKubeconfigMode: "0644"
|
||||
labels:
|
||||
unidesk.ai/node-id: D601-VM
|
||||
unidesk.ai/provider-id: D601-VM
|
||||
unidesk.ai/runtime-plane: standalone
|
||||
disableComponents:
|
||||
- traefik
|
||||
- servicelb
|
||||
- metrics-server
|
||||
disableComponents:
|
||||
- traefik
|
||||
- servicelb
|
||||
- metrics-server
|
||||
registryMirror:
|
||||
source: 127.0.0.1:5000
|
||||
endpoint: https://registry.hwpod.com
|
||||
auth:
|
||||
sourceRef: /root/.unidesk/.state/secrets/platform-infra/k3s-clusters/nc01-d601-artifact-registry-pull.env
|
||||
sourceRef: /root/.unidesk/.state/secrets/platform-infra/k3s-clusters/d601-artifact-registry-pull.env
|
||||
usernameKey: ARTIFACT_REGISTRY_PULL_USERNAME
|
||||
passwordKey: ARTIFACT_REGISTRY_PULL_PASSWORD
|
||||
passwordHashKey: ARTIFACT_REGISTRY_PULL_PASSWORD_HASH
|
||||
@@ -98,21 +53,3 @@ targets:
|
||||
expectedBytes: 195443636
|
||||
concurrentPulls: 2
|
||||
timeoutSeconds: 900
|
||||
serviceLb:
|
||||
enabledNodeLabel: svccontroller.k3s.cattle.io/enablelb
|
||||
enabledNode: v2202607378382480008
|
||||
workerDns:
|
||||
namespace: kube-system
|
||||
deploymentName: unidesk-worker-coredns
|
||||
serviceName: unidesk-worker-kube-dns
|
||||
clusterIp: 10.43.0.53
|
||||
image: docker.io/coredns/coredns:1.14.4
|
||||
configMapName: coredns
|
||||
customConfigMapName: coredns-custom
|
||||
replicas: 1
|
||||
timeoutSeconds: 120
|
||||
smoke:
|
||||
namespace: platform-infra-smoke
|
||||
jobName: d601-runtime-plane-smoke
|
||||
image: docker.io/library/busybox:1.36.1
|
||||
timeoutSeconds: 180
|
||||
|
||||
@@ -70,7 +70,7 @@ targets:
|
||||
sites:
|
||||
- id: artifact-registry
|
||||
configRef: config/platform-infra/k3s-clusters.yaml
|
||||
path: targets.nc01-d601.k3s.registryMirror
|
||||
path: targets.d601.k3s.registryMirror
|
||||
- id: pikaoa-development
|
||||
configRef: config/pikaoa.yaml
|
||||
path: developmentRuntime.targets.NC01
|
||||
|
||||
@@ -10,125 +10,33 @@
|
||||
- UniDesk 拥有的 image version、namespace、endpoint、FRP port 与 profile 列表不得隐藏在 helper 常量里;外部工具要求的 TOML、JSON 和 env 仅是边缘渲染格式。
|
||||
- 新节点出网 bootstrap 遵循 `docs/reference/yaml-first-ops.md` 的零依赖 host proxy 边界,不得依赖尚未安装的 Docker、k3s、containerd 或包管理器网络能力。
|
||||
|
||||
## k3s 集群拓扑
|
||||
## D601 独立 k3s 集群
|
||||
|
||||
- 权威配置:
|
||||
- 集群边界、拓扑模式、历史 WireGuard 迁移参数、独立 server 角色、
|
||||
registry endpoint、ServiceLB 节点和 smoke 归
|
||||
`config/platform-infra/k3s-clusters.yaml`。
|
||||
- worker host proxy 的服务端来源、客户端工件、Secret 引用、代理环境和 systemd drop-in 归 `config/platform-infra/host-proxy.yaml`。
|
||||
- Hyper-V VM 资源和宿主内网 SSH 归 `config/platform-infra/hyperv-vms.yaml`。
|
||||
- 既有 control-plane 的 k3s systemd drop-in 继续归 `config/hwlab-node-control-plane.yaml`,并通过 `clusterConfigRef` 消费集群网络参数。
|
||||
- 受控入口:
|
||||
- `platform-infra k3s-cluster secret-init --target <id> --confirm` 只在 Secret source 不存在时生成 WireGuard key pair 和 k3s agent token,存在时拒绝覆盖。
|
||||
- `platform-infra k3s-cluster plan|apply|separate|drain|status|smoke
|
||||
--target <id>` 是部署、拓扑拆分、清空不可调度 worker和验收入口。
|
||||
- `topologyMode=separated` 时,`apply --confirm` 与
|
||||
`separate --confirm` 都只执行受控拆分:
|
||||
- 先证明旧 worker 的非 DaemonSet Pod 为零;
|
||||
- 再清理旧 agent datastore、CNI 和跨地域 WireGuard 配置;
|
||||
- 在 D601 安装并等待独立单节点 k3s server Ready;
|
||||
- 最后从 NC01 删除旧 Node 和对端 WireGuard peer。
|
||||
- 拆分不得先删除 NC01 Node 或隧道;D601 独立 server 未 Ready 时必须
|
||||
保留旧控制面侧事实并返回失败。
|
||||
- 无固定公网地址的 worker 由宿主内网 SSH 控制,主动连接 control-plane WireGuard endpoint;worker provider route 只作为部署后补充验收入口。
|
||||
- 集群 apply 先经宿主内网 SSH 分发零依赖 proxy client,确认它连接声明的 `vpn-server` 且外网 probe 成功,再启动或恢复 `k3s-agent`。
|
||||
- 集群 apply 在 proxy 前先收敛 worker WireGuard;内容一致时不强写,只有 unit 未运行或握手过期且隧道探针失败时才恢复 worker 接口。
|
||||
- worker WireGuard 的业务探针固定使用 control-plane API `:6443`,不以可能被策略丢弃的 ICMP 判断隧道健康。
|
||||
- NAT 后 worker 的 WireGuard `PersistentKeepalive` 周期由集群 YAML 声明;跨地域节点优先用较短周期维持映射,但不得据此宣称降低物理 RTT。
|
||||
- 跨地域 TCP 调优由集群 YAML 的 `tcpTuning` 统一声明并受控应用到两端:
|
||||
- 高 RTT、持续丢包链路优先使用内核支持的 BBR、`fq` 与
|
||||
`tcp_mtu_probing=1`,同时保留实际吞吐复测;
|
||||
- sysctl 文件和 live 值均先比较后收敛,不得靠裸 `sysctl`、
|
||||
预拉镜像、导入镜像或反复重启 Pod 掩盖数据面问题;
|
||||
- status 必须披露两端 live tuning 是否与 YAML 一致。
|
||||
- 大型 proxy 工件由 Windows 宿主按 YAML `upstreamUrl` 获取并校验 SHA-256,Provider/trans 只承载控制命令,不作为大二进制数据面。
|
||||
- WireGuard 只承载集群三层网络;D601 worker proxy 使用 `vpn-server` 的
|
||||
NC01 公网 Shadowsocks TCP 入口,不把应用出网 TCP 再套入 WireGuard,
|
||||
也不与集群 WireGuard 竞争 UDP 链路。
|
||||
- proxy client 协议实现版本由 host-proxy YAML 声明,并与所连接的 `vpn-server` 协议实现版本保持一致。
|
||||
- control-plane 节点配置只组合 `hwlab nodes control-plane infra k3s-config plan|status|apply`:
|
||||
- 该窄域只比较并安装 owning YAML 渲染的 k3s systemd drop-in;
|
||||
- 只在 drop-in 或节点 pod 容量需要收敛时重启 k3s;
|
||||
- 确需重启 k3s server 时,受控入口先停止服务并备份
|
||||
`/var/lib/rancher/k3s/server/cred/passwd` 派生文件,再启动服务;
|
||||
- 禁止用 `passwd` 与 `state.db` 的文件 mtime、datastore freshness
|
||||
预判或其他凭据新旧检查阻断 YAML-first 变更;备份动作必须以
|
||||
`blocking=false` warning 披露,只有真实 stop/start、Node Ready
|
||||
或期望状态收敛失败才可使 apply 失败;
|
||||
- 同一入口等待节点 Ready;
|
||||
- 禁止调用 HWLAB 全量 `infra apply`,也不得触碰 Kubernetes manifests、registry、Tekton、Argo、git-mirror 或业务 rollout。
|
||||
- 运行边界:
|
||||
- NC01 与 D601 必须保持两个独立 k3s 集群,禁止把 D601 重新注册为
|
||||
NC01 agent,也禁止把 NC01 注册到 D601。
|
||||
- D601 独立集群只运行一个 server 节点,初始业务 Pod 数必须为零;
|
||||
是否承载新业务必须由新的 workload owning YAML 显式选择,不得继承
|
||||
NC01 的 GitOps、PipelineRun、DaemonSet 或默认 namespace。
|
||||
- NC01 保留自身 control-plane 与运行面;拆分后可保留无 peer 的本地
|
||||
WireGuard 接口以避免重写生产 control-plane 地址,但不得恢复 D601 peer。
|
||||
- 跨地域连接只用于受控 SSH/Provider 控制通道和公网 HTTPS
|
||||
artifact pull,不再承担 Kubernetes API、Pod、Service、DNS 或 VXLAN 数据面。
|
||||
- `topologyMode=separated` 是 owning YAML 的架构事实。CI/CD、GitOps、
|
||||
VM bootstrap 和普通集群 apply 禁止改回 joined、启动 `k3s-agent`、
|
||||
写入 agent token 或渲染跨地域 Flannel/WireGuard peer。
|
||||
- 下列 worker 规则只适用于仍由 owning YAML 明确声明为 joined 的其他
|
||||
集群,不得用于 NC01-D601:
|
||||
- control-plane 保持唯一 k3s server,同时承担 workload;
|
||||
- worker 只运行 `k3s-agent`,加入前卸载旧独立 server;
|
||||
- worker 是否参与业务调度及其 taint 必须由集群 owning YAML 的
|
||||
`workers.<id>.scheduling` 声明。
|
||||
- joined worker 的调度规则:
|
||||
- `schedulable=false` 时必须同时声明 `NoSchedule` 或 `NoExecute` taint;
|
||||
- agent 注册参数和 control-plane 的 `cordon`/taint 收敛必须消费同一声明,
|
||||
防止节点重装、重注册或 CI 自动覆盖后短暂接收业务 Pod;
|
||||
- `drain --confirm` 只允许清空声明为不可调度的 worker,并保留
|
||||
DaemonSet/节点系统 Pod;普通 apply、CI 和业务 renderer 禁止隐式
|
||||
`uncordon` 或移除 owning YAML 声明的 taint;
|
||||
- drain 与 Node Ready 等待必须由本地 CLI 组合短于 `trans` 上限的
|
||||
目标侧操作和短轮询;节点已不可调度且用户要求彻底清空时允许
|
||||
`--disable-eviction` 直接删除非 DaemonSet Pod,但不得扩大到其他节点。
|
||||
- 不可调度且无业务 Pod 的 worker 不要求 PodCIDR 代理 probe 成功;
|
||||
host 本地代理健康与外网 probe 仍须成功,空 DNS Deployment、Service
|
||||
和 Endpoints 必须由 drain 一并清理。
|
||||
- Flannel 使用 WireGuard 接口承载 VXLAN,不能只用 FRP 转发 Kubernetes API。
|
||||
- 参与业务调度的跨地域 worker DNS 使用集群 YAML 声明的独立
|
||||
CoreDNS Deployment 和固定 ClusterIP;不可调度 worker 必须移除该
|
||||
非 DaemonSet DNS Pod:
|
||||
- Deployment 固定调度到 worker,Service 使用 `internalTrafficPolicy: Local`;
|
||||
- worker kubelet 只在本地 DNS Ready 后通过 `cluster-dns` 参数切换;
|
||||
- 禁止 patch k3s Addon 拥有的 `coredns` Deployment 或 `kube-dns` Service;
|
||||
- 受控 apply 先比较声明态哈希,内容一致时跳过 `kubectl apply`,不得触发无意义 rollout。
|
||||
- ServiceLB 只在 owning YAML 指定的公网节点启用,防止无公网地址 worker 接管 hostPort。
|
||||
- worker 的 containerd 外网访问继承 host proxy systemd 环境,直接访问上游 registry;禁止为公共依赖新增 registry mirror。
|
||||
- host proxy 的 Pod 访问监听地址必须由 owning YAML 按目标节点 PodCIDR
|
||||
声明为该节点 `cni0` 地址;禁止复用其他节点的 `10.42.x.1`,
|
||||
否则 sing-box 会因绑定不存在地址持续重启。
|
||||
- `127.0.0.1:5000` 只表示 UniDesk 集群内 artifact registry,worker 的 `registries.yaml` 可把该本地引用映射到集群 endpoint,不得把它扩展为公共镜像代理。
|
||||
- 幂等重复执行不得强写:
|
||||
- WireGuard、systemd drop-in、agent unit、token 文件和节点标签均先比较声明态与运行态;
|
||||
- 内容一致时保持 `mutation=false`,不得无条件重启、重装或覆盖;
|
||||
- WireGuard 只在配置变化或 unit 未运行时重启/启动,避免短暂断网触发 control-plane 级联重启;
|
||||
- worker agent 以 owning YAML、版本、参数、集群内 registry 映射和 token 的脱敏摘要判断是否需要重装;
|
||||
- `Type=notify` agent 在节点 Ready 时可能保持 `activating`;摘要一致、状态为 `active|activating` 且 MainPID 存在即视为运行中,禁止因此重复重装;
|
||||
- worker 安装只做短启动,终态由 control-plane 的 Node Ready 轮询判定,不得让宿主 SSH 会话等待 systemd notify;
|
||||
- k3s API 暂时不可读属于 unknown,不得当作容量漂移触发 restart;
|
||||
- 集群 CLI 只组合各自 owning domain,不借用全量基础设施入口扩大写入范围。
|
||||
- 验收:
|
||||
- `topologyMode=separated` 的 `status` 必须同时证明:
|
||||
- D601 只有一个 Ready 节点;
|
||||
- D601 `k3s` server active 且旧 `k3s-agent` inactive;
|
||||
- D601 非 `kube-system` Pod 为零;
|
||||
- NC01 不再存在 D601 Node。
|
||||
- joined 拓扑的 `status` 才验证 WireGuard handshake、host proxy、
|
||||
server/agent、两个 Ready node、worker InternalIP 和旧 server 退役。
|
||||
- 仅当 worker 的 `scheduling.schedulable=true` 时,
|
||||
`smoke --confirm` 才可把 Job 定向调度到 worker并验证集群 DNS与
|
||||
Kubernetes ClusterIP;不可调度 worker 的 smoke 只读验证 taint、
|
||||
cordon、Node Ready 和非 DaemonSet Pod 为零,不得创建带 toleration
|
||||
的例外 Job。
|
||||
- smoke 等待必须使用短 `kubectl get` 轮询,禁止让单次 `trans` 挂住等待 Job 条件。
|
||||
- 基础 smoke 要求完整服务 FQDN 的解析结果包含 Pod 注入的 Kubernetes Service IP,并通过 API `ClusterIP:443` TCP connect 验证 Service 数据面;不得接受 `nslookup` 的空应答,也不把容器 TLS 客户端能力混入判定。
|
||||
- Secret 输出只显示 sourceRef、presence、key 名和 fingerprint,不显示值。
|
||||
- `config/platform-infra/k3s-clusters.yaml#targets.d601` 是 D601 单节点集群的
|
||||
唯一拓扑真相,只包含节点、server、出网 proxy 引用和只读 artifact
|
||||
registry;不得出现 NC01、agent token、跨地域 CNI 或共享调度参数。
|
||||
- NC01 和 D601 是两个完全独立的 Kubernetes 调度域。禁止以 agent、
|
||||
WireGuard、Flannel/VXLAN、共享 datastore 或隐藏 bootstrap 参数重新连接。
|
||||
- D601 只运行 `k3s` server,不运行 `k3s-agent`。新业务必须由自己的
|
||||
owning YAML 显式选择 D601;NC01 的 PaC、Tekton、Argo、GitOps、
|
||||
DaemonSet 和 namespace 不得默认复制。
|
||||
- 跨地域只保留两类边界:
|
||||
- 宿主 SSH/Provider 控制通道;
|
||||
- 鉴权、只读的公网 HTTPS artifact registry。
|
||||
- D601 的公共依赖出网由 `config/platform-infra/host-proxy.yaml` 声明,
|
||||
proxy drop-in 必须归 `k3s.service`;禁止公共 registry mirror。
|
||||
- 受控入口为
|
||||
`platform-infra k3s-cluster plan|apply|status|smoke|registry-smoke
|
||||
--target d601`。`apply` 先收敛 host proxy,再比较 server 声明摘要;
|
||||
内容一致时不得重装或重启。
|
||||
- `status` 和 `smoke` 必须证明:
|
||||
- 只有一个 Ready 节点;
|
||||
- server active 且 agent inactive;
|
||||
- 全部 `kube-system` Deployment Ready;
|
||||
- 非 `kube-system` Pod 为零。
|
||||
- NC01 节点配置独立归 `config/hwlab-node-control-plane.yaml`,不得引用
|
||||
D601 集群配置。确需重启 NC01 k3s 时,受控入口先备份并移走派生
|
||||
`server/cred/passwd`,该 freshness 事实只能作为非阻塞 warning。
|
||||
## Hyper-V Ubuntu Provider VM
|
||||
|
||||
- `config/platform-infra/hyperv-vms.yaml` 是 Windows Hyper-V Ubuntu Provider VM 的唯一配置真相,完整需求见 `project-management/PJ2026-01/specs/PJ2026-01060314-hyperv-ubuntu-vm.md`。
|
||||
@@ -228,7 +136,7 @@
|
||||
|
||||
- NC01 集群内 registry 的内部 endpoint 是唯一写入 authority:
|
||||
- CI、BuildKit 和 publish 继续使用内部 `127.0.0.1:5000` 语义;
|
||||
- 公网 `registry.hwpod.com` 只用于异地 worker 拉取,不接受 push 或删除。
|
||||
- 公网 `registry.hwpod.com` 只用于 D601 独立集群拉取,不接受 push 或删除。
|
||||
- 公网入口归共享 public-edge 管理:
|
||||
- hostname、DNS、upstream、允许方法和鉴权 SecretRef 来自集群 owning YAML;
|
||||
- Caddy 只允许 `GET`、`HEAD`,其余方法返回拒绝;
|
||||
@@ -239,24 +147,23 @@
|
||||
后仅按 key 继承环境,禁止用语义不同的 `docker run --env-file`,也禁止
|
||||
手工重启容器;
|
||||
- 可信 TLS 由 public-edge 管理,镜像层不得经过 CDN 代理。
|
||||
- worker containerd 归集群 CLI 管理:
|
||||
- D601 containerd 归集群 CLI 管理:
|
||||
- 既有 `127.0.0.1:5000/...` 镜像引用保持不变;
|
||||
- `registries.yaml` 把该引用映射到 YAML 声明的公网 HTTPS endpoint;
|
||||
- 用户名和密码只从同一 SecretRef 渲染;
|
||||
- registry 配置单独变化时只重启 agent 使配置生效,不卸载或重装 k3s。
|
||||
- registry 配置变化时由 server 受控加载,不卸载或重装 k3s。
|
||||
- 受控入口:
|
||||
|
||||
```bash
|
||||
bun scripts/cli.ts platform-infra k3s-cluster registry-credential-init --target nc01-d601 --confirm
|
||||
bun scripts/cli.ts platform-infra k3s-cluster apply --target nc01-d601 --confirm
|
||||
bun scripts/cli.ts platform-infra k3s-cluster registry-smoke --target nc01-d601 --confirm
|
||||
bun scripts/cli.ts platform-infra k3s-cluster registry-credential-init --target d601 --confirm
|
||||
bun scripts/cli.ts platform-infra k3s-cluster apply --target d601 --confirm
|
||||
bun scripts/cli.ts platform-infra k3s-cluster registry-smoke --target d601 --confirm
|
||||
```
|
||||
|
||||
- `registry-smoke` 必须验证:
|
||||
- 未鉴权拒绝、有效凭据、manifest digest、blob range 和写方法拒绝;
|
||||
- YAML 选定的真实镜像冷拉取、并发拉取和重复拉取;
|
||||
- 拉取期间 Kubernetes API 连通,前后 control-plane 与 worker 均为 Ready;
|
||||
- Provider 控制通道能返回结构化终态;
|
||||
- YAML 选定的真实镜像拉取和摘要;
|
||||
- 拉取后 D601 Node 仍为 Ready;
|
||||
- 结束后不保留临时 Kubernetes workload。
|
||||
|
||||
## Gitea 与 Pipelines-as-Code 边界
|
||||
|
||||
@@ -32,7 +32,11 @@
|
||||
|
||||
### 3.1 HYPERV-VM-REQ-001 YAML 真相
|
||||
|
||||
虚拟机名称、宿主 route、CPU、内存、磁盘、镜像 URL 与摘要、网络、Ubuntu 用户、k3s 版本和 Provider ID 必须来自 `config/platform-infra/hyperv-vms.yaml`。跨主机集群拓扑、WireGuard、k3s server/agent 角色、SecretRef、registry mirror 和调度 smoke 必须来自独立的集群 owning YAML。实现只能校验、解析引用和渲染,不能维护隐藏运行参数。
|
||||
虚拟机名称、宿主 route、CPU、内存、磁盘、镜像 URL 与摘要、网络、
|
||||
Ubuntu 用户、k3s 版本和 Provider ID 必须来自
|
||||
`config/platform-infra/hyperv-vms.yaml`。D601 独立 server、host proxy、
|
||||
registry endpoint 和拉取验收必须来自集群 owning YAML。实现只能校验、
|
||||
解析引用和渲染,不能维护隐藏运行参数。
|
||||
|
||||
### 3.2 HYPERV-VM-REQ-002 稳定资源边界
|
||||
|
||||
@@ -79,23 +83,20 @@ Provider TCP 数据池恢复必须满足:
|
||||
|
||||
D601-VM 与 NC01 必须作为两个独立集群运行:
|
||||
|
||||
- `config/platform-infra/k3s-clusters.yaml` 必须显式声明
|
||||
`topologyMode=separated` 和 D601 `k3s.standalone` 参数。
|
||||
- `config/platform-infra/k3s-clusters.yaml#targets.d601` 只声明 D601
|
||||
独立 server,不得出现 NC01、agent token 或跨地域 CNI 参数。
|
||||
- `config/platform-infra/hyperv-vms.yaml` 必须以
|
||||
`management.mode=standalone` 引用独立集群配置。
|
||||
- D601 只运行单节点 k3s server;旧 `k3s-agent`、agent datastore、CNI
|
||||
和跨地域 WireGuard peer 必须由受控拆分入口退役。
|
||||
- 拆分前必须证明 D601 非 DaemonSet Pod 为零;D601 独立 server Ready
|
||||
后,才能从 NC01 删除旧 Node 和 WireGuard peer。
|
||||
- 拆分后 D601 非 `kube-system` Pod 必须为零,NC01 不得存在 D601 Node。
|
||||
- CI/CD、GitOps、VM bootstrap 和业务 renderer 禁止重新安装 agent、
|
||||
恢复 agent token、重建跨地域 CNI 或把 NC01 workload 自动投递到 D601。
|
||||
- D601 只运行单节点 k3s server,`k3s-agent` 必须 inactive。
|
||||
- D601 非 `kube-system` Pod 必须为零,全部系统 Deployment 必须 Ready。
|
||||
- CI/CD、GitOps、VM bootstrap 和业务 renderer 禁止安装 agent、创建
|
||||
跨地域 CNI 或把 NC01 workload 自动投递到 D601。
|
||||
- D601 host proxy 可继续为独立集群提供公共依赖出网,不得依赖公共镜像站
|
||||
或新增公共 registry mirror。
|
||||
- 集群自有 artifact registry 可以通过公网可信 TLS 向异地 worker 提供鉴权只读拉取:
|
||||
- 集群自有 artifact registry 可以通过公网可信 TLS 向 D601 提供鉴权只读拉取:
|
||||
- 内部 CI/publish 继续使用集群内写入 endpoint,公网入口不得成为第二个 push authority。
|
||||
- 公网边缘只允许 Registry V2 拉取所需的 `GET` 与 `HEAD`,其他方法必须在反向代理层拒绝。
|
||||
- endpoint、DNS、上游、允许方法、凭据 `sourceRef`、镜像摘要和拉取验收参数必须来自集群 owning YAML。
|
||||
- D601 containerd 继续消费既有镜像引用,只把该内部引用映射到声明的公网 HTTPS endpoint。
|
||||
- artifact 大文件不得经过 WireGuard、Flannel/VXLAN 或受传输时长和对象大小限制的 CDN。
|
||||
- artifact 大文件不得经过 Kubernetes 数据面或受传输时长和对象大小限制的 CDN。
|
||||
- 宿主内网 SSH 可以作为 D601 的稳定部署控制通道,provider route 只作为部署后验收入口。
|
||||
|
||||
+3
-3
@@ -894,9 +894,9 @@ function platformInfraHelpSummary(): unknown {
|
||||
"bun scripts/cli.ts platform-infra hyperv-vm plan --target D601-VM",
|
||||
"bun scripts/cli.ts platform-infra hyperv-vm apply --target D601-VM --confirm",
|
||||
"bun scripts/cli.ts platform-infra hyperv-vm status --target D601-VM",
|
||||
"bun scripts/cli.ts platform-infra k3s-cluster plan --target nc01-d601",
|
||||
"bun scripts/cli.ts platform-infra k3s-cluster apply --target nc01-d601 --confirm",
|
||||
"bun scripts/cli.ts platform-infra k3s-cluster smoke --target nc01-d601 --confirm",
|
||||
"bun scripts/cli.ts platform-infra k3s-cluster plan --target d601",
|
||||
"bun scripts/cli.ts platform-infra k3s-cluster apply --target d601 --confirm",
|
||||
"bun scripts/cli.ts platform-infra k3s-cluster smoke --target d601",
|
||||
"bun scripts/cli.ts platform-infra hyperv-vm verify --target D601-VM [--repair-provider-channel --confirm]",
|
||||
"bun scripts/cli.ts platform-infra langbot plan",
|
||||
"bun scripts/cli.ts platform-infra langbot apply --confirm",
|
||||
|
||||
@@ -201,12 +201,7 @@ export function prepareHostProxyGuestBundle(targetId: string): HostProxyGuestBun
|
||||
remoteArchivePath: target.source.client.archiveInstallPath,
|
||||
archiveSha256: target.source.client.archiveSha256,
|
||||
prepareScript: remotePrepareScript(target.source.client),
|
||||
applyScript: `${remoteApplyScript(target, material.clientConfigJson)}
|
||||
if [ "$changed" = true ] && [ -s /etc/systemd/system/k3s-agent.service ]; then
|
||||
systemctl daemon-reload
|
||||
systemctl restart --no-block k3s-agent
|
||||
fi
|
||||
`,
|
||||
applyScript: remoteApplyScript(target, material.clientConfigJson),
|
||||
statusScript: remoteStatusScript(target),
|
||||
sourceFingerprint: target.source.fingerprint,
|
||||
materialFingerprint: material.fingerprint,
|
||||
|
||||
@@ -6,69 +6,23 @@ export const K3S_CLUSTER_CONFIG_LABEL = "config/platform-infra/k3s-clusters.yaml
|
||||
export interface K3sClusterTarget {
|
||||
id: string;
|
||||
clusterName: string;
|
||||
topologyMode: "joined" | "separated";
|
||||
controlPlane: {
|
||||
id: string;
|
||||
configRef: string;
|
||||
lane: string;
|
||||
nodeName: string;
|
||||
publicAddress: string;
|
||||
route: string;
|
||||
kubeRoute: string;
|
||||
};
|
||||
worker: {
|
||||
node: {
|
||||
id: string;
|
||||
bootstrapConfigRef: string;
|
||||
hostProxyConfigRef: string;
|
||||
hostProxyTargetId: string;
|
||||
nodeName: string;
|
||||
labels: Record<string, string>;
|
||||
scheduling: {
|
||||
schedulable: boolean;
|
||||
taints: Array<{ key: string; value: string; effect: "NoSchedule" | "PreferNoSchedule" | "NoExecute" }>;
|
||||
};
|
||||
windowsRoute: string;
|
||||
guestAddress: string;
|
||||
guestUser: string;
|
||||
stateDir: string;
|
||||
k3sVersion: string;
|
||||
};
|
||||
wireguard: {
|
||||
interfaceName: string;
|
||||
packageName: string;
|
||||
mtu: number;
|
||||
listenPort: number;
|
||||
controlPlaneAddress: string;
|
||||
workerAddress: string;
|
||||
endpoint: string;
|
||||
persistentKeepaliveSeconds: number;
|
||||
};
|
||||
tcpTuning: {
|
||||
congestionControl: "bbr" | "cubic";
|
||||
defaultQdisc: "fq" | "fq_codel";
|
||||
mtuProbing: 1 | 2;
|
||||
sysctlPath: string;
|
||||
};
|
||||
secrets: {
|
||||
sourceRef: string;
|
||||
controlPlanePrivateKey: string;
|
||||
controlPlanePublicKey: string;
|
||||
workerPrivateKey: string;
|
||||
workerPublicKey: string;
|
||||
agentToken: string;
|
||||
};
|
||||
k3s: {
|
||||
serverUrl: string;
|
||||
agentTokenFile: string;
|
||||
flannelInterface: string;
|
||||
writeKubeconfigMode: "0600" | "0640" | "0644";
|
||||
maxPods: number;
|
||||
standalone: {
|
||||
clusterName: string;
|
||||
nodeName: string;
|
||||
writeKubeconfigMode: string;
|
||||
labels: Record<string, string>;
|
||||
disableComponents: string[];
|
||||
};
|
||||
disableComponents: string[];
|
||||
registryMirror: {
|
||||
source: string;
|
||||
endpoint: string;
|
||||
@@ -86,23 +40,13 @@ export interface K3sClusterTarget {
|
||||
concurrentPulls: number;
|
||||
timeoutSeconds: number;
|
||||
};
|
||||
serviceLb: { enabledNodeLabel: string; enabledNode: string };
|
||||
workerDns: {
|
||||
namespace: string;
|
||||
deploymentName: string;
|
||||
serviceName: string;
|
||||
clusterIp: string;
|
||||
image: string;
|
||||
configMapName: string;
|
||||
customConfigMapName: string;
|
||||
replicas: number;
|
||||
timeoutSeconds: number;
|
||||
};
|
||||
};
|
||||
smoke: { namespace: string; jobName: string; image: string; timeoutSeconds: number };
|
||||
}
|
||||
|
||||
export function readK3sClusterConfig(targetId?: string): { defaultTargetId: string; target: K3sClusterTarget } {
|
||||
export function readK3sClusterConfig(targetId?: string): {
|
||||
defaultTargetId: string;
|
||||
target: K3sClusterTarget;
|
||||
} {
|
||||
const root = record(Bun.YAML.parse(readFileSync(rootPath(K3S_CLUSTER_CONFIG_LABEL), "utf8")), K3S_CLUSTER_CONFIG_LABEL);
|
||||
if (root.kind !== "platform-infra-k3s-clusters") throw new Error(`${K3S_CLUSTER_CONFIG_LABEL}.kind must be platform-infra-k3s-clusters`);
|
||||
const defaults = record(root.defaults, `${K3S_CLUSTER_CONFIG_LABEL}.defaults`);
|
||||
@@ -110,164 +54,63 @@ export function readK3sClusterConfig(targetId?: string): { defaultTargetId: stri
|
||||
const id = targetId ?? defaultTargetId;
|
||||
const raw = record(record(root.targets, `${K3S_CLUSTER_CONFIG_LABEL}.targets`)[id], `${K3S_CLUSTER_CONFIG_LABEL}.targets.${id}`);
|
||||
if (raw.enabled !== true) throw new Error(`${K3S_CLUSTER_CONFIG_LABEL}.targets.${id}.enabled must be true`);
|
||||
const control = record(raw.controlPlane, `targets.${id}.controlPlane`);
|
||||
const workers = record(raw.workers, `targets.${id}.workers`);
|
||||
const workerEntries = Object.entries(workers);
|
||||
if (workerEntries.length !== 1) throw new Error(`targets.${id}.workers must contain exactly one worker`);
|
||||
const [workerId, workerRaw] = workerEntries[0] ?? [];
|
||||
const worker = record(workerRaw, `targets.${id}.workers.${workerId}`);
|
||||
const wireguard = record(raw.wireguard, `targets.${id}.wireguard`);
|
||||
const tcpTuning = record(raw.tcpTuning, `targets.${id}.tcpTuning`);
|
||||
const secrets = record(raw.secrets, `targets.${id}.secrets`);
|
||||
const node = record(raw.node, `targets.${id}.node`);
|
||||
const k3s = record(raw.k3s, `targets.${id}.k3s`);
|
||||
const standalone = record(k3s.standalone, `targets.${id}.k3s.standalone`);
|
||||
const mirror = record(k3s.registryMirror, `targets.${id}.k3s.registryMirror`);
|
||||
const mirrorAuth = record(mirror.auth, `targets.${id}.k3s.registryMirror.auth`);
|
||||
const registryPullSmoke = record(k3s.registryPullSmoke, `targets.${id}.k3s.registryPullSmoke`);
|
||||
const serviceLb = record(k3s.serviceLb, `targets.${id}.k3s.serviceLb`);
|
||||
const workerDns = record(k3s.workerDns, `targets.${id}.k3s.workerDns`);
|
||||
const smoke = record(raw.smoke, `targets.${id}.smoke`);
|
||||
const controlSource = resolveRef(text(control, "configRef", `targets.${id}.controlPlane`));
|
||||
const bootstrapSource = resolveRef(text(worker, "bootstrapConfigRef", `targets.${id}.workers[0]`));
|
||||
const host = record(bootstrapSource.host, "worker bootstrap host");
|
||||
const network = record(bootstrapSource.network, "worker bootstrap network");
|
||||
const ubuntu = record(bootstrapSource.ubuntu, "worker bootstrap ubuntu");
|
||||
const bootstrapK3s = record(bootstrapSource.k3s, "worker bootstrap k3s");
|
||||
const labels = stringMap(record(worker.labels, `targets.${id}.workers.${workerId}.labels`), `targets.${id}.workers.${workerId}.labels`);
|
||||
const scheduling = record(worker.scheduling, `targets.${id}.workers.${workerId}.scheduling`);
|
||||
const taints = objectArray(scheduling.taints, `targets.${id}.workers.${workerId}.scheduling.taints`).map((item, index) => ({
|
||||
key: kubernetesLabelKey(item, "key", `targets.${id}.workers.${workerId}.scheduling.taints[${index}]`),
|
||||
value: kubernetesLabelValue(item, "value", `targets.${id}.workers.${workerId}.scheduling.taints[${index}]`),
|
||||
effect: enumText(item, "effect", `targets.${id}.workers.${workerId}.scheduling.taints[${index}]`, ["NoSchedule", "PreferNoSchedule", "NoExecute"]),
|
||||
}));
|
||||
const target: K3sClusterTarget = {
|
||||
id,
|
||||
clusterName: text(raw, "clusterName", `targets.${id}`),
|
||||
topologyMode: enumText(raw, "topologyMode", `targets.${id}`, ["joined", "separated"]),
|
||||
controlPlane: {
|
||||
id: text(control, "id", `targets.${id}.controlPlane`),
|
||||
configRef: text(control, "configRef", `targets.${id}.controlPlane`),
|
||||
lane: text(control, "lane", `targets.${id}.controlPlane`),
|
||||
nodeName: text(control, "nodeName", `targets.${id}.controlPlane`),
|
||||
publicAddress: text(control, "publicAddress", `targets.${id}.controlPlane`),
|
||||
route: text(controlSource, "route", "control plane configRef"),
|
||||
kubeRoute: text(controlSource, "kubeRoute", "control plane configRef"),
|
||||
},
|
||||
worker: {
|
||||
id: text(worker, "id", `targets.${id}.workers.${workerId}`),
|
||||
bootstrapConfigRef: text(worker, "bootstrapConfigRef", `targets.${id}.workers.${workerId}`),
|
||||
hostProxyConfigRef: text(worker, "hostProxyConfigRef", `targets.${id}.workers.${workerId}`),
|
||||
hostProxyTargetId: hostProxyTargetId(text(worker, "hostProxyConfigRef", `targets.${id}.workers.${workerId}`)),
|
||||
nodeName: text(worker, "nodeName", `targets.${id}.workers.${workerId}`),
|
||||
labels,
|
||||
scheduling: {
|
||||
schedulable: boolean(scheduling, "schedulable", `targets.${id}.workers.${workerId}.scheduling`),
|
||||
taints,
|
||||
const pullSmoke = record(k3s.registryPullSmoke, `targets.${id}.k3s.registryPullSmoke`);
|
||||
const bootstrap = resolveRef(text(node, "bootstrapConfigRef", `targets.${id}.node`));
|
||||
const host = record(bootstrap.host, "node bootstrap host");
|
||||
const network = record(bootstrap.network, "node bootstrap network");
|
||||
const ubuntu = record(bootstrap.ubuntu, "node bootstrap ubuntu");
|
||||
const bootstrapK3s = record(bootstrap.k3s, "node bootstrap k3s");
|
||||
const hostProxyConfigRef = text(node, "hostProxyConfigRef", `targets.${id}.node`);
|
||||
return {
|
||||
defaultTargetId,
|
||||
target: {
|
||||
id,
|
||||
clusterName: kubernetesName(raw, "clusterName", `targets.${id}`),
|
||||
node: {
|
||||
id: text(node, "id", `targets.${id}.node`),
|
||||
bootstrapConfigRef: text(node, "bootstrapConfigRef", `targets.${id}.node`),
|
||||
hostProxyConfigRef,
|
||||
hostProxyTargetId: hostProxyTargetId(hostProxyConfigRef),
|
||||
nodeName: kubernetesName(node, "nodeName", `targets.${id}.node`),
|
||||
labels: stringMap(record(node.labels, `targets.${id}.node.labels`), `targets.${id}.node.labels`),
|
||||
windowsRoute: text(host, "route", "node bootstrap host"),
|
||||
guestAddress: text(network, "address", "node bootstrap network"),
|
||||
guestUser: text(ubuntu, "user", "node bootstrap ubuntu"),
|
||||
stateDir: text(host, "stateDir", "node bootstrap host"),
|
||||
k3sVersion: text(bootstrapK3s, "version", "node bootstrap k3s"),
|
||||
},
|
||||
windowsRoute: text(host, "route", "worker bootstrap host"),
|
||||
guestAddress: text(network, "address", "worker bootstrap network"),
|
||||
guestUser: text(ubuntu, "user", "worker bootstrap ubuntu"),
|
||||
stateDir: text(host, "stateDir", "worker bootstrap host"),
|
||||
k3sVersion: text(bootstrapK3s, "version", "worker bootstrap k3s"),
|
||||
},
|
||||
wireguard: {
|
||||
interfaceName: text(wireguard, "interfaceName", `targets.${id}.wireguard`),
|
||||
packageName: text(wireguard, "packageName", `targets.${id}.wireguard`),
|
||||
mtu: integer(wireguard, "mtu", `targets.${id}.wireguard`),
|
||||
listenPort: integer(wireguard, "listenPort", `targets.${id}.wireguard`),
|
||||
controlPlaneAddress: cidr(wireguard, "controlPlaneAddress", `targets.${id}.wireguard`),
|
||||
workerAddress: cidr(wireguard, "workerAddress", `targets.${id}.wireguard`),
|
||||
endpoint: text(wireguard, "endpoint", `targets.${id}.wireguard`),
|
||||
persistentKeepaliveSeconds: integer(wireguard, "persistentKeepaliveSeconds", `targets.${id}.wireguard`),
|
||||
},
|
||||
tcpTuning: {
|
||||
congestionControl: enumText(tcpTuning, "congestionControl", `targets.${id}.tcpTuning`, ["bbr", "cubic"]),
|
||||
defaultQdisc: enumText(tcpTuning, "defaultQdisc", `targets.${id}.tcpTuning`, ["fq", "fq_codel"]),
|
||||
mtuProbing: oneOfInteger(tcpTuning, "mtuProbing", `targets.${id}.tcpTuning`, [1, 2]),
|
||||
sysctlPath: text(tcpTuning, "sysctlPath", `targets.${id}.tcpTuning`),
|
||||
},
|
||||
secrets: {
|
||||
sourceRef: text(secrets, "sourceRef", `targets.${id}.secrets`),
|
||||
controlPlanePrivateKey: text(secrets, "controlPlanePrivateKey", `targets.${id}.secrets`),
|
||||
controlPlanePublicKey: text(secrets, "controlPlanePublicKey", `targets.${id}.secrets`),
|
||||
workerPrivateKey: text(secrets, "workerPrivateKey", `targets.${id}.secrets`),
|
||||
workerPublicKey: text(secrets, "workerPublicKey", `targets.${id}.secrets`),
|
||||
agentToken: text(secrets, "agentToken", `targets.${id}.secrets`),
|
||||
},
|
||||
k3s: {
|
||||
serverUrl: text(k3s, "serverUrl", `targets.${id}.k3s`),
|
||||
agentTokenFile: text(k3s, "agentTokenFile", `targets.${id}.k3s`),
|
||||
flannelInterface: text(k3s, "flannelInterface", `targets.${id}.k3s`),
|
||||
maxPods: integer(k3s, "maxPods", `targets.${id}.k3s`),
|
||||
standalone: {
|
||||
clusterName: kubernetesName(standalone, "clusterName", "k3s.standalone"),
|
||||
nodeName: kubernetesName(standalone, "nodeName", "k3s.standalone"),
|
||||
writeKubeconfigMode: enumText(standalone, "writeKubeconfigMode", "k3s.standalone", ["0600", "0640", "0644"]),
|
||||
labels: stringMap(record(standalone.labels, "k3s.standalone.labels"), "k3s.standalone.labels"),
|
||||
disableComponents: stringArray(standalone.disableComponents, "k3s.standalone.disableComponents"),
|
||||
},
|
||||
registryMirror: {
|
||||
source: text(mirror, "source", "registryMirror"),
|
||||
endpoint: httpsUrl(mirror, "endpoint", "registryMirror"),
|
||||
auth: {
|
||||
sourceRef: text(mirrorAuth, "sourceRef", "registryMirror.auth"),
|
||||
usernameKey: text(mirrorAuth, "usernameKey", "registryMirror.auth"),
|
||||
passwordKey: text(mirrorAuth, "passwordKey", "registryMirror.auth"),
|
||||
passwordHashKey: text(mirrorAuth, "passwordHashKey", "registryMirror.auth"),
|
||||
k3s: {
|
||||
writeKubeconfigMode: enumText(k3s, "writeKubeconfigMode", `targets.${id}.k3s`, ["0600", "0640", "0644"]),
|
||||
maxPods: integer(k3s, "maxPods", `targets.${id}.k3s`),
|
||||
disableComponents: stringArray(k3s.disableComponents, `targets.${id}.k3s.disableComponents`),
|
||||
registryMirror: {
|
||||
source: text(mirror, "source", "registryMirror"),
|
||||
endpoint: httpsUrl(mirror, "endpoint", "registryMirror"),
|
||||
auth: {
|
||||
sourceRef: text(mirrorAuth, "sourceRef", "registryMirror.auth"),
|
||||
usernameKey: text(mirrorAuth, "usernameKey", "registryMirror.auth"),
|
||||
passwordKey: text(mirrorAuth, "passwordKey", "registryMirror.auth"),
|
||||
passwordHashKey: text(mirrorAuth, "passwordHashKey", "registryMirror.auth"),
|
||||
},
|
||||
},
|
||||
registryPullSmoke: {
|
||||
image: text(pullSmoke, "image", "registryPullSmoke"),
|
||||
expectedDigest: sha256Digest(pullSmoke, "expectedDigest", "registryPullSmoke"),
|
||||
expectedBytes: integer(pullSmoke, "expectedBytes", "registryPullSmoke"),
|
||||
concurrentPulls: integer(pullSmoke, "concurrentPulls", "registryPullSmoke"),
|
||||
timeoutSeconds: integer(pullSmoke, "timeoutSeconds", "registryPullSmoke"),
|
||||
},
|
||||
},
|
||||
registryPullSmoke: {
|
||||
image: text(registryPullSmoke, "image", "registryPullSmoke"),
|
||||
expectedDigest: sha256Digest(registryPullSmoke, "expectedDigest", "registryPullSmoke"),
|
||||
expectedBytes: integer(registryPullSmoke, "expectedBytes", "registryPullSmoke"),
|
||||
concurrentPulls: integer(registryPullSmoke, "concurrentPulls", "registryPullSmoke"),
|
||||
timeoutSeconds: integer(registryPullSmoke, "timeoutSeconds", "registryPullSmoke"),
|
||||
},
|
||||
serviceLb: { enabledNodeLabel: text(serviceLb, "enabledNodeLabel", "serviceLb"), enabledNode: text(serviceLb, "enabledNode", "serviceLb") },
|
||||
workerDns: {
|
||||
namespace: kubernetesName(workerDns, "namespace", "workerDns"),
|
||||
deploymentName: kubernetesName(workerDns, "deploymentName", "workerDns"),
|
||||
serviceName: kubernetesName(workerDns, "serviceName", "workerDns"),
|
||||
clusterIp: ipv4Address(workerDns, "clusterIp", "workerDns"),
|
||||
image: text(workerDns, "image", "workerDns"),
|
||||
configMapName: kubernetesName(workerDns, "configMapName", "workerDns"),
|
||||
customConfigMapName: kubernetesName(workerDns, "customConfigMapName", "workerDns"),
|
||||
replicas: integer(workerDns, "replicas", "workerDns"),
|
||||
timeoutSeconds: integer(workerDns, "timeoutSeconds", "workerDns"),
|
||||
},
|
||||
},
|
||||
smoke: {
|
||||
namespace: text(smoke, "namespace", `targets.${id}.smoke`),
|
||||
jobName: text(smoke, "jobName", `targets.${id}.smoke`),
|
||||
image: text(smoke, "image", `targets.${id}.smoke`),
|
||||
timeoutSeconds: integer(smoke, "timeoutSeconds", `targets.${id}.smoke`),
|
||||
},
|
||||
};
|
||||
if (target.k3s.flannelInterface !== target.wireguard.interfaceName) throw new Error(`targets.${id}.k3s.flannelInterface must match wireguard.interfaceName`);
|
||||
if (target.k3s.serviceLb.enabledNode !== target.controlPlane.nodeName) throw new Error(`targets.${id}.k3s.serviceLb.enabledNode must match controlPlane.nodeName`);
|
||||
if (!target.worker.scheduling.schedulable && !target.worker.scheduling.taints.some((taint) => taint.effect === "NoSchedule" || taint.effect === "NoExecute")) {
|
||||
throw new Error(`targets.${id}.workers.${workerId}.scheduling must declare a NoSchedule or NoExecute taint when schedulable=false`);
|
||||
}
|
||||
if (!target.tcpTuning.sysctlPath.startsWith("/etc/sysctl.d/")) throw new Error(`targets.${id}.tcpTuning.sysctlPath must stay under /etc/sysctl.d`);
|
||||
return { defaultTargetId, target };
|
||||
}
|
||||
|
||||
export function clusterServerArgs(configRef: string): string[] {
|
||||
const match = configRef.match(/^config\/platform-infra\/k3s-clusters\.yaml#targets\.([A-Za-z0-9._-]+)\.controlPlane$/u);
|
||||
if (match === null) throw new Error(`unsupported k3s clusterConfigRef: ${configRef}`);
|
||||
const { target } = readK3sClusterConfig(match[1]);
|
||||
return [
|
||||
"--node-ip", address(target.wireguard.controlPlaneAddress),
|
||||
"--node-external-ip", target.controlPlane.publicAddress,
|
||||
"--flannel-iface", target.k3s.flannelInterface,
|
||||
"--tls-san", address(target.wireguard.controlPlaneAddress),
|
||||
"--tls-san", target.controlPlane.publicAddress,
|
||||
"--agent-token-file", target.k3s.agentTokenFile,
|
||||
];
|
||||
}
|
||||
|
||||
export function address(value: string): string {
|
||||
return value.split("/", 1)[0] ?? value;
|
||||
throw new Error(`external k3s cluster attachment is retired; remove clusterConfigRef: ${configRef}`);
|
||||
}
|
||||
|
||||
function resolveRef(value: string): Record<string, unknown> {
|
||||
@@ -295,17 +138,6 @@ function integer(value: Record<string, unknown>, key: string, label: string): nu
|
||||
return Number(result);
|
||||
}
|
||||
|
||||
function boolean(value: Record<string, unknown>, key: string, label: string): boolean {
|
||||
const result = value[key];
|
||||
if (typeof result !== "boolean") throw new Error(`${label}.${key} must be a boolean`);
|
||||
return result;
|
||||
}
|
||||
|
||||
function objectArray(value: unknown, label: string): Array<Record<string, unknown>> {
|
||||
if (!Array.isArray(value)) throw new Error(`${label} must be an array`);
|
||||
return value.map((item, index) => record(item, `${label}[${index}]`));
|
||||
}
|
||||
|
||||
function httpsUrl(value: Record<string, unknown>, key: string, label: string): string {
|
||||
const result = text(value, key, label);
|
||||
const parsed = new URL(result);
|
||||
@@ -327,26 +159,6 @@ function enumText<const T extends string>(value: Record<string, unknown>, key: s
|
||||
return result as T;
|
||||
}
|
||||
|
||||
function oneOfInteger<const T extends number>(value: Record<string, unknown>, key: string, label: string, allowed: readonly T[]): T {
|
||||
const result = integer(value, key, label);
|
||||
if (!allowed.includes(result as T)) throw new Error(`${label}.${key} must be one of ${allowed.join(", ")}`);
|
||||
return result as T;
|
||||
}
|
||||
|
||||
function cidr(value: Record<string, unknown>, key: string, label: string): string {
|
||||
const result = text(value, key, label);
|
||||
if (!/^\d{1,3}(?:\.\d{1,3}){3}\/\d{1,2}$/u.test(result)) throw new Error(`${label}.${key} must be an IPv4 CIDR`);
|
||||
return result;
|
||||
}
|
||||
|
||||
function ipv4Address(value: Record<string, unknown>, key: string, label: string): string {
|
||||
const result = text(value, key, label);
|
||||
if (!/^\d{1,3}(?:\.\d{1,3}){3}$/u.test(result) || result.split(".").some((part) => Number(part) > 255)) {
|
||||
throw new Error(`${label}.${key} must be an IPv4 address`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function kubernetesName(value: Record<string, unknown>, key: string, label: string): string {
|
||||
const result = text(value, key, label);
|
||||
if (!/^[a-z0-9](?:[-a-z0-9]*[a-z0-9])?$/u.test(result) || result.length > 63) {
|
||||
@@ -355,22 +167,6 @@ function kubernetesName(value: Record<string, unknown>, key: string, label: stri
|
||||
return result;
|
||||
}
|
||||
|
||||
function kubernetesLabelKey(value: Record<string, unknown>, key: string, label: string): string {
|
||||
const result = text(value, key, label);
|
||||
if (!/^(?:[a-z0-9](?:[-a-z0-9.]*[a-z0-9])?\/)?[A-Za-z0-9](?:[-A-Za-z0-9_.]*[A-Za-z0-9])?$/u.test(result) || result.length > 253) {
|
||||
throw new Error(`${label}.${key} must be a Kubernetes label key`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function kubernetesLabelValue(value: Record<string, unknown>, key: string, label: string): string {
|
||||
const result = text(value, key, label);
|
||||
if (!/^[A-Za-z0-9](?:[-A-Za-z0-9_.]*[A-Za-z0-9])?$/u.test(result) || result.length > 63) {
|
||||
throw new Error(`${label}.${key} must be a Kubernetes label value`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function stringMap(value: Record<string, unknown>, label: string): Record<string, string> {
|
||||
return Object.fromEntries(Object.entries(value).map(([key, item]) => {
|
||||
if (typeof item !== "string" || item.length === 0) throw new Error(`${label}.${key} must be a non-empty string`);
|
||||
@@ -387,6 +183,6 @@ function stringArray(value: unknown, label: string): string[] {
|
||||
|
||||
function hostProxyTargetId(ref: string): string {
|
||||
const match = ref.match(/^config\/platform-infra\/host-proxy\.yaml#targets\.([A-Za-z0-9._-]+)$/u);
|
||||
if (match === null) throw new Error(`worker hostProxyConfigRef must point at config/platform-infra/host-proxy.yaml#targets.<id>`);
|
||||
if (match === null) throw new Error(`hostProxyConfigRef must point at config/platform-infra/host-proxy.yaml#targets.<id>`);
|
||||
return match[1] ?? "";
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user