fix: enforce D601 native k3s guards
This commit is contained in:
@@ -2,6 +2,17 @@
|
|||||||
|
|
||||||
HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥官和 runner 进入本仓库时的顶级索引,只放入口、一句话规则和长期参考链接;细则统一维护在 `docs/reference/`。
|
HWLAB 是硬件实验室运行面和控制面项目。本文是 agent、指挥官和 runner 进入本仓库时的顶级索引,只放入口、一句话规则和长期参考链接;细则统一维护在 `docs/reference/`。
|
||||||
|
|
||||||
|
## P0 D601 k3s 控制面归一
|
||||||
|
|
||||||
|
- D601 的 HWLAB DEV 运行态只允许使用自部署原生 k3s;Docker Desktop Kubernetes 已经停用并清理数据,任何人不得重新启用或把它作为 HWLAB DEV、MVP、CI/CD、smoke、部署或验收目标。跨仓库事故跟踪见 [pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138),指挥官热修复背景见 [pikasTech/unidesk#118](https://github.com/pikasTech/unidesk/issues/118)。
|
||||||
|
- D601 上裸 `kubectl` 不可信:默认 kubeconfig 可能仍残留 `docker-desktop` / `127.0.0.1:11700`。所有 D601 `hwlab-dev` 只读观测、DEV CD、rollback、smoke 和排障必须显式使用 `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`,并确认节点名是 `d601`。
|
||||||
|
- 发现 `docker-desktop` context、`desktop-control-plane` 节点、Docker Desktop Kubernetes 中的 `hwlab-dev` 残留,或同一 HWLAB 资源被第二控制面承载时,必须停止部署/验收并按 #138 处理;不要把第二控制面或失效默认 kubeconfig 的输出作为 DEV-LIVE 证据。
|
||||||
|
|
||||||
|
## P0 GitHub Issue 写入纪律
|
||||||
|
|
||||||
|
- HWLAB #7、用户反馈、长期看板和指挥简报的 GitHub issue 正文写入必须走 UniDesk CLI:`cd /root/unidesk && bun scripts/cli.ts gh ...`;禁止直接用原生 `gh issue edit/create/comment` 写这些 issue。事故和工具补强需求见 [pikasTech/unidesk#142](https://github.com/pikasTech/unidesk/issues/142)。
|
||||||
|
- 在 UniDesk CLI 局部替换、写前备份和写后 hash 验证能力完成前,不要对 #7 做无 guard 的整篇 body replace;必须先保留 before body、确认维护纪律 heading 仍存在,再写入。
|
||||||
|
|
||||||
## 工作区
|
## 工作区
|
||||||
|
|
||||||
- Runner 和指挥常用工作区是 `/workspace/hwlab`;进入仓库先检查分支与工作树状态,详见 [docs/reference/commander-collaboration.md](docs/reference/commander-collaboration.md)。
|
- Runner 和指挥常用工作区是 `/workspace/hwlab`;进入仓库先检查分支与工作树状态,详见 [docs/reference/commander-collaboration.md](docs/reference/commander-collaboration.md)。
|
||||||
|
|||||||
+9
-1
@@ -89,12 +89,20 @@ This is source/dry-run support only. It does not prove a registry image exists,
|
|||||||
does not build, pull, push, apply, restart, or touch PROD, and must not be used
|
does not build, pull, push, apply, restart, or touch PROD, and must not be used
|
||||||
as M3 DEV-LIVE evidence.
|
as M3 DEV-LIVE evidence.
|
||||||
|
|
||||||
|
## D601 Native k3s Guard
|
||||||
|
|
||||||
|
DEV deploy and smoke commands must target D601 native k3s with
|
||||||
|
`KUBECONFIG=/etc/rancher/k3s/k3s.yaml`. Do not use the default kubeconfig if it
|
||||||
|
points at `docker-desktop`, `desktop-control-plane`, or `127.0.0.1:11700`.
|
||||||
|
Control-plane confusion is tracked in
|
||||||
|
[pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138).
|
||||||
|
|
||||||
Next DEV deploy smoke commands, for a separately authorized deployment task:
|
Next DEV deploy smoke commands, for a separately authorized deployment task:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
npm run check
|
npm run check
|
||||||
node -e "JSON.parse(require('node:fs').readFileSync('deploy/deploy.json','utf8'))"
|
node -e "JSON.parse(require('node:fs').readFileSync('deploy/deploy.json','utf8'))"
|
||||||
kubectl apply --dry-run=server -k deploy/k8s/dev
|
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl apply --dry-run=server -k deploy/k8s/dev
|
||||||
curl -fsS http://74.48.78.17:16667/health/live
|
curl -fsS http://74.48.78.17:16667/health/live
|
||||||
node scripts/dev-edge-health-smoke.mjs --live --write-report
|
node scripts/dev-edge-health-smoke.mjs --live --write-report
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -93,6 +93,14 @@ The selected path is used for read-only preflight, dry-run planning, and the
|
|||||||
DEV-only apply path. The report records the kubeconfig source and path, but it
|
DEV-only apply path. The report records the kubeconfig source and path, but it
|
||||||
does not print kubeconfig contents or Secret values.
|
does not print kubeconfig contents or Secret values.
|
||||||
|
|
||||||
|
Emergency D601 guard: for normal D601 execution, prefer
|
||||||
|
`/etc/rancher/k3s/k3s.yaml`. Docker Desktop Kubernetes has been disabled and
|
||||||
|
cleaned, but a residual default kubeconfig may still point at
|
||||||
|
`docker-desktop`/`127.0.0.1:11700`; do not pass that path through `KUBECONFIG`
|
||||||
|
or `--kubeconfig`. Any selected kubeconfig whose context/node resolves to
|
||||||
|
`docker-desktop` or `desktop-control-plane` is a blocker, not DEV-LIVE
|
||||||
|
evidence. See [pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138).
|
||||||
|
|
||||||
Runtime DB migration is a separate DEV-only apply boundary. The source-only
|
Runtime DB migration is a separate DEV-only apply boundary. The source-only
|
||||||
check is safe for normal PR validation:
|
check is safe for normal PR validation:
|
||||||
|
|
||||||
|
|||||||
@@ -85,7 +85,9 @@ The preflight checks:
|
|||||||
on public frontend `:16666` and API/edge/live `:16667`. Legacy public
|
on public frontend `:16666` and API/edge/live `:16667`. Legacy public
|
||||||
`:6666/:6667` observations are historical/deprecated only and must not be
|
`:6666/:6667` observations are historical/deprecated only and must not be
|
||||||
used as current green evidence or current active blockers.
|
used as current green evidence or current active blockers.
|
||||||
- The runner can perform read-only `kubectl` probes against `hwlab-dev`.
|
- The runner can perform read-only `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl`
|
||||||
|
probes against `hwlab-dev` and verify node `d601`; default `docker-desktop`
|
||||||
|
kubeconfig output is not DEV evidence.
|
||||||
- `reports/dev-gate/dev-edge-health.json`, when present, records read-only
|
- `reports/dev-gate/dev-edge-health.json`, when present, records read-only
|
||||||
public edge, frps, tunnel-health, and k3s observability evidence.
|
public edge, frps, tunnel-health, and k3s observability evidence.
|
||||||
- `http://74.48.78.17:16667/health/live` responds.
|
- `http://74.48.78.17:16667/health/live` responds.
|
||||||
@@ -106,8 +108,8 @@ publish report. The dimensions are:
|
|||||||
publish.
|
publish.
|
||||||
- `docker-daemon-push-access`: read-only Docker daemon evidence for the
|
- `docker-daemon-push-access`: read-only Docker daemon evidence for the
|
||||||
local/internal registry target. This is the publish-path capability.
|
local/internal registry target. This is the publish-path capability.
|
||||||
- `k3s-pull-access`: read-only `kubectl` evidence for whether the DEV cluster
|
- `k3s-pull-access`: read-only native-k3s `kubectl` evidence for whether the
|
||||||
can inspect image pull state. This is the deploy-path capability.
|
DEV cluster can inspect image pull state. This is the deploy-path capability.
|
||||||
|
|
||||||
## Verdict Rules
|
## Verdict Rules
|
||||||
|
|
||||||
|
|||||||
@@ -120,8 +120,11 @@ Current unblock order:
|
|||||||
1. Provide an approved Node 20 DEV builder base image for artifact publish.
|
1. Provide an approved Node 20 DEV builder base image for artifact publish.
|
||||||
2. Publish every frozen HWLAB service artifact and record immutable registry
|
2. Publish every frozen HWLAB service artifact and record immutable registry
|
||||||
digests for the selected `origin/main` commit.
|
digests for the selected `origin/main` commit.
|
||||||
3. Restore read-only D601 `hwlab-dev` k3s observability with `kubectl` or an
|
3. Restore read-only D601 `hwlab-dev` k3s observability with
|
||||||
approved kubeconfig/maintenance bridge, without reading Secrets.
|
`KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl` or an approved
|
||||||
|
kubeconfig/maintenance bridge, without reading Secrets. Do not use a default
|
||||||
|
`docker-desktop` kubeconfig as DEV evidence; the control-plane incident is
|
||||||
|
tracked in pikasTech/unidesk#138.
|
||||||
4. Prove DEV `hwlab-cloud-api` DB env and health readiness with redacted values
|
4. Prove DEV `hwlab-cloud-api` DB env and health readiness with redacted values
|
||||||
only.
|
only.
|
||||||
5. Repair the `:16667`/`frp`/edge/router path and rerun read-only DEV edge
|
5. Repair the `:16667`/`frp`/edge/router path and rerun read-only DEV edge
|
||||||
|
|||||||
@@ -9,6 +9,22 @@ It is also the long-term reference for
|
|||||||
deployment regularization must progress from stable docs, to controlled
|
deployment regularization must progress from stable docs, to controlled
|
||||||
CLI/script entrypoints, to UniDesk CI/CD plus imageized delivery.
|
CLI/script entrypoints, to UniDesk CI/CD plus imageized delivery.
|
||||||
|
|
||||||
|
## D601 Native k3s Emergency Guard
|
||||||
|
|
||||||
|
HWLAB DEV deploy and publish paths must target D601 native k3s only. Docker
|
||||||
|
Desktop Kubernetes on D601 has been disabled and cleaned; it must not be
|
||||||
|
re-enabled or used as a second orchestrator. The cross-repo incident, evidence,
|
||||||
|
and governance plan are tracked in
|
||||||
|
[pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138), and
|
||||||
|
the 2026-05-23 commander recovery context is in
|
||||||
|
[pikasTech/unidesk#118](https://github.com/pikasTech/unidesk/issues/118).
|
||||||
|
|
||||||
|
Every DEV CD status, apply, rollback, smoke, and manual Kubernetes command must
|
||||||
|
use `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` and verify node `d601` before any
|
||||||
|
mutation or acceptance. Bare `kubectl`, `docker-desktop` context,
|
||||||
|
`desktop-control-plane`, or `127.0.0.1:11700` are wrong-control-plane signals,
|
||||||
|
not HWLAB DEV-LIVE evidence.
|
||||||
|
|
||||||
## Workspaces
|
## Workspaces
|
||||||
|
|
||||||
| Purpose | Path |
|
| Purpose | Path |
|
||||||
@@ -327,7 +343,7 @@ use the repo-owned scripts below instead of ad hoc shell fragments.
|
|||||||
| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. |
|
| Publish DEV images | `node scripts/dev-artifact-publish.mjs --publish` inside `HWLAB_CD_TRANSACTION_ID` | Internal step only; direct side-effect calls are rejected. |
|
||||||
| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Internal transaction step after publish; read/write desired-state only from repo files. |
|
| Refresh catalog/desired state | `node scripts/refresh-artifact-catalog.mjs --target-ref origin/main --publish-report reports/dev-gate/dev-artifacts.json` | Internal transaction step after publish; read/write desired-state only from repo files. |
|
||||||
| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` inside `HWLAB_CD_TRANSACTION_ID` | Internal transaction step; direct side-effect calls are rejected. |
|
| Apply DEV rollout | `node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report` inside `HWLAB_CD_TRANSACTION_ID` | Internal transaction step; direct side-effect calls are rejected. |
|
||||||
| Roll back DEV Deployment | `kubectl -n hwlab-dev rollout undo deployment/<name>` with explicit D601 kubeconfig | Add `hwlab deploy rollback --env dev --service <id> --to-revision <n>`. |
|
| Roll back DEV Deployment | `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout undo deployment/<name>` | Add `hwlab deploy rollback --env dev --service <id> --to-revision <n>`. |
|
||||||
| Verify status | `curl` checks plus `node scripts/d601-k3s-readonly-observability.mjs` | Add `hwlab deploy verify --env dev --write-report`. |
|
| Verify status | `curl` checks plus `node scripts/d601-k3s-readonly-observability.mjs` | Add `hwlab deploy verify --env dev --write-report`. |
|
||||||
|
|
||||||
## Automation Direction
|
## Automation Direction
|
||||||
|
|||||||
@@ -43,7 +43,26 @@ KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy,svc,pod -o
|
|||||||
```
|
```
|
||||||
|
|
||||||
Do not trust the default `kubectl` context as a DEV runtime signal. On D601 it
|
Do not trust the default `kubectl` context as a DEV runtime signal. On D601 it
|
||||||
may point at another cluster such as `docker-desktop`.
|
may point at another cluster such as `docker-desktop`. The 2026-05-23 control
|
||||||
|
plane incident is tracked in
|
||||||
|
[pikasTech/unidesk#138](https://github.com/pikasTech/unidesk/issues/138), with
|
||||||
|
hotfix context in
|
||||||
|
[pikasTech/unidesk#118](https://github.com/pikasTech/unidesk/issues/118):
|
||||||
|
Docker Desktop Kubernetes has been disabled and cleaned, but the default
|
||||||
|
`/home/ubuntu/.kube/config` may still point at the dead
|
||||||
|
`docker-desktop`/`127.0.0.1:11700` entry. Treat that as residual bad config, not
|
||||||
|
as DEV runtime evidence.
|
||||||
|
|
||||||
|
Before any DEV deploy, rollback, smoke, or manual k3s observation, confirm the
|
||||||
|
native node explicitly:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes -o jsonpath='{.items[*].metadata.name}'
|
||||||
|
```
|
||||||
|
|
||||||
|
The result must contain `d601`. Seeing `desktop-control-plane`,
|
||||||
|
`docker-desktop`, or `127.0.0.1:11700` means the command is on the wrong
|
||||||
|
control-plane path and must stop before mutation or acceptance.
|
||||||
|
|
||||||
Use the read-only preflight when a runner needs to report k3s visibility:
|
Use the read-only preflight when a runner needs to report k3s visibility:
|
||||||
|
|
||||||
|
|||||||
@@ -331,11 +331,12 @@ function collectLocalRuntimeObservations(d601Observability) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (commandExists("kubectl")) {
|
if (commandExists("kubectl")) {
|
||||||
const kubectl = commandResult("kubectl", ["get", "deploy,po,svc", "-n", namespace, "-o", "wide"]);
|
const args = ["get", "deploy,po,svc", "-n", namespace, "-o", "wide"];
|
||||||
|
const kubectl = commandResult("env", [`KUBECONFIG=${runnerK3sKubeconfigPath}`, "kubectl", ...args]);
|
||||||
observations.push({
|
observations.push({
|
||||||
id: "kubectl-hwlab-dev",
|
id: "kubectl-hwlab-dev",
|
||||||
status: kubectl.exitCode === 0 ? "observed" : "blocked",
|
status: kubectl.exitCode === 0 ? "observed" : "blocked",
|
||||||
command: kubectl.command,
|
command: runnerKubeconfigCommandText(args),
|
||||||
summary: oneLine(kubectl.exitCode === 0 ? kubectl.stdout : kubectl.stderr || kubectl.stdout)
|
summary: oneLine(kubectl.exitCode === 0 ? kubectl.stdout : kubectl.stderr || kubectl.stdout)
|
||||||
});
|
});
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -1211,15 +1211,37 @@ async function resolveKubectlContext(args, env, runCommand) {
|
|||||||
const selection = resolveDevKubeconfigSelection({ flagValue: args.kubeconfig, env });
|
const selection = resolveDevKubeconfigSelection({ flagValue: args.kubeconfig, env });
|
||||||
const which = await runCommand("which", ["kubectl"], { timeoutMs: 5000 });
|
const which = await runCommand("which", ["kubectl"], { timeoutMs: 5000 });
|
||||||
const executor = which.code === 0 ? which.stdout.trim() : "kubectl";
|
const executor = which.code === 0 ? which.stdout.trim() : "kubectl";
|
||||||
|
const kubectlEnv = {
|
||||||
|
...env,
|
||||||
|
KUBECONFIG: selection.kubeconfig
|
||||||
|
};
|
||||||
|
if (which.code === 0 && args.apply) {
|
||||||
|
const context = await runCommand(executor, ["config", "current-context"], { env: kubectlEnv, timeoutMs: 10000 });
|
||||||
|
const nodes = await runCommand(executor, ["get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], { env: kubectlEnv, timeoutMs: 15000 });
|
||||||
|
const contextName = String(context.stdout ?? "").trim();
|
||||||
|
const nodeNames = String(nodes.stdout ?? "").trim().split(/\s+/u).filter(Boolean);
|
||||||
|
const forbiddenContext = context.code === 0 && /docker-desktop|desktop-control-plane/iu.test(contextName);
|
||||||
|
const forbiddenNode = nodes.code === 0 && (
|
||||||
|
nodeNames.includes("desktop-control-plane") || (nodeNames.length > 0 && !nodeNames.includes("d601"))
|
||||||
|
);
|
||||||
|
if (forbiddenContext || forbiddenNode) {
|
||||||
|
throw new DevCdApplyError("DEV CD kubeconfig does not target D601 native k3s", {
|
||||||
|
code: "wrong-kubernetes-control-plane",
|
||||||
|
blockers: [{
|
||||||
|
type: "environment_blocker",
|
||||||
|
scope: "d601-native-k3s",
|
||||||
|
status: "open",
|
||||||
|
summary: "D601 DEV CD must use native k3s via /etc/rancher/k3s/k3s.yaml; default docker-desktop kubeconfig is forbidden."
|
||||||
|
}]
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
executor,
|
executor,
|
||||||
kubeconfig: selection.kubeconfig,
|
kubeconfig: selection.kubeconfig,
|
||||||
kubeconfigSource: selection.source,
|
kubeconfigSource: selection.source,
|
||||||
commandPrefix: buildKubectlCommandPrefix(selection.kubeconfig),
|
commandPrefix: buildKubectlCommandPrefix(selection.kubeconfig),
|
||||||
env: {
|
env: kubectlEnv
|
||||||
...env,
|
|
||||||
KUBECONFIG: selection.kubeconfig
|
|
||||||
}
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -232,6 +232,32 @@ export function formatKubeconfigAccessFailure(selection, unreadablePath) {
|
|||||||
return `DEV kubeconfig from ${selection.source} is not readable at ${unreadablePath}; set --kubeconfig, ${devKubeconfigEnvName}, or ${kubeconfigEnvName} to a readable DEV kubeconfig path. Kubeconfig contents and Secret values are never printed.`;
|
return `DEV kubeconfig from ${selection.source} is not readable at ${unreadablePath}; set --kubeconfig, ${devKubeconfigEnvName}, or ${kubeconfigEnvName} to a readable DEV kubeconfig path. Kubeconfig contents and Secret values are never printed.`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function validateD601NativeKubeconfig(kubectlPath, env) {
|
||||||
|
const context = await commandResult(kubectlPath, ["config", "current-context"], 10_000, { env });
|
||||||
|
if (!context.ok) {
|
||||||
|
return `DEV kubeconfig could not report a kubectl context: ${redactSensitiveText(context.stderr || context.stdout)}`;
|
||||||
|
}
|
||||||
|
const contextName = context.stdout.trim();
|
||||||
|
if (/docker-desktop|desktop-control-plane/iu.test(contextName)) {
|
||||||
|
return `DEV kubeconfig resolved to forbidden context ${contextName}; D601 DEV must use native k3s at ${defaultD601KubeconfigPath}.`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const server = await commandResult(kubectlPath, ["config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"], 10_000, { env });
|
||||||
|
if (server.ok && /127\.0\.0\.1:11700|docker-desktop/iu.test(server.stdout)) {
|
||||||
|
return "DEV kubeconfig resolved to Docker Desktop Kubernetes server 127.0.0.1:11700; use D601 native k3s kubeconfig instead.";
|
||||||
|
}
|
||||||
|
|
||||||
|
const nodes = await commandResult(kubectlPath, ["get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], 15_000, { env });
|
||||||
|
if (!nodes.ok) {
|
||||||
|
return `DEV kubeconfig could not read native k3s nodes: ${redactSensitiveText(nodes.stderr || nodes.stdout)}`;
|
||||||
|
}
|
||||||
|
const nodeNames = nodes.stdout.trim().split(/\s+/u).filter(Boolean);
|
||||||
|
if (!nodeNames.includes("d601")) {
|
||||||
|
return `DEV kubeconfig did not resolve to D601 native k3s; observed nodes=${nodeNames.join(",") || "none"}.`;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
export function redactSensitiveText(value) {
|
export function redactSensitiveText(value) {
|
||||||
return String(value)
|
return String(value)
|
||||||
.replace(/\b(Bearer\s+)[A-Za-z0-9._~+/=-]+/giu, "$1<redacted>")
|
.replace(/\b(Bearer\s+)[A-Za-z0-9._~+/=-]+/giu, "$1<redacted>")
|
||||||
@@ -300,6 +326,15 @@ async function resolveD601Kubectl(args = {}, env = process.env) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const nativeK3sFailure = await validateD601NativeKubeconfig(kubectlPath, base.env);
|
||||||
|
if (nativeK3sFailure !== null) {
|
||||||
|
return {
|
||||||
|
...base,
|
||||||
|
status: "blocked",
|
||||||
|
reason: nativeK3sFailure
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
...base,
|
...base,
|
||||||
status: "ready",
|
status: "ready",
|
||||||
|
|||||||
@@ -21,6 +21,7 @@ const defaultReportPath = path.join(repoRoot, "reports/dev-gate/dev-edge-health.
|
|||||||
const publicHost = "74.48.78.17";
|
const publicHost = "74.48.78.17";
|
||||||
const publicPort = 16667;
|
const publicPort = 16667;
|
||||||
const namespace = "hwlab-dev";
|
const namespace = "hwlab-dev";
|
||||||
|
const d601KubeconfigPath = "/etc/rancher/k3s/k3s.yaml";
|
||||||
const tcpPorts = [16667, 7000, 7402];
|
const tcpPorts = [16667, 7000, 7402];
|
||||||
const CODE_AGENT_PROVIDER_SECRET_CONTRACT = Object.freeze({
|
const CODE_AGENT_PROVIDER_SECRET_CONTRACT = Object.freeze({
|
||||||
sourceIssue: "pikasTech/HWLAB#143",
|
sourceIssue: "pikasTech/HWLAB#143",
|
||||||
@@ -35,6 +36,18 @@ const runtimeServices = [
|
|||||||
{ serviceId: "hwlab-edge-proxy", port: 6667 }
|
{ serviceId: "hwlab-edge-proxy", port: 6667 }
|
||||||
];
|
];
|
||||||
|
|
||||||
|
function nativeKubectlArgs(args) {
|
||||||
|
return [`KUBECONFIG=${d601KubeconfigPath}`, "kubectl", ...args];
|
||||||
|
}
|
||||||
|
|
||||||
|
function nativeKubectlCommand(args) {
|
||||||
|
return nativeKubectlArgs(args).join(" ");
|
||||||
|
}
|
||||||
|
|
||||||
|
function runKubectl(args, options) {
|
||||||
|
return runCommand("env", nativeKubectlArgs(args), options);
|
||||||
|
}
|
||||||
|
|
||||||
export async function runDevEdgeHealthSmoke(argv) {
|
export async function runDevEdgeHealthSmoke(argv) {
|
||||||
const args = parseArgs(argv);
|
const args = parseArgs(argv);
|
||||||
requireDevEndpoint();
|
requireDevEndpoint();
|
||||||
@@ -636,7 +649,9 @@ async function inspectKubernetes() {
|
|||||||
const result = {
|
const result = {
|
||||||
observable: hasKubectl,
|
observable: hasKubectl,
|
||||||
namespace,
|
namespace,
|
||||||
|
kubeconfig: d601KubeconfigPath,
|
||||||
context: null,
|
context: null,
|
||||||
|
nodeNames: [],
|
||||||
services: [],
|
services: [],
|
||||||
endpoints: [],
|
endpoints: [],
|
||||||
pods: [],
|
pods: [],
|
||||||
@@ -650,6 +665,7 @@ async function inspectKubernetes() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
await collectKubectlContext(result);
|
await collectKubectlContext(result);
|
||||||
|
await collectKubectlNodeIdentity(result);
|
||||||
await collectKubectlJson(result, "services", [
|
await collectKubectlJson(result, "services", [
|
||||||
"-n", namespace, "get", "svc", "hwlab-cloud-api", "hwlab-edge-proxy", "hwlab-tunnel-client", "hwlab-router", "-o", "json"
|
"-n", namespace, "get", "svc", "hwlab-cloud-api", "hwlab-edge-proxy", "hwlab-tunnel-client", "hwlab-router", "-o", "json"
|
||||||
]);
|
]);
|
||||||
@@ -665,16 +681,30 @@ async function inspectKubernetes() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async function collectKubectlContext(result) {
|
async function collectKubectlContext(result) {
|
||||||
const context = await runCommand("kubectl", ["config", "current-context"], { timeoutMs: 5000 });
|
const context = await runKubectl(["config", "current-context"], { timeoutMs: 5000 });
|
||||||
result.context = {
|
result.context = {
|
||||||
exitCode: context.exitCode,
|
exitCode: context.exitCode,
|
||||||
value: context.exitCode === 0 ? context.stdout.trim() : null,
|
value: context.exitCode === 0 ? context.stdout.trim() : null,
|
||||||
|
command: nativeKubectlCommand(["config", "current-context"]),
|
||||||
stderr: context.stderr.trim()
|
stderr: context.stderr.trim()
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function collectKubectlNodeIdentity(result) {
|
||||||
|
const args = ["get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"];
|
||||||
|
const nodes = await runKubectl(args, { timeoutMs: 5000 });
|
||||||
|
if (nodes.exitCode !== 0) {
|
||||||
|
result.notes.push(`kubectl node identity probe failed: ${nodes.stderr.trim()}`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
result.nodeNames = nodes.stdout.trim().split(/\s+/u).filter(Boolean);
|
||||||
|
if (!result.nodeNames.includes("d601")) {
|
||||||
|
result.notes.push(`kubectl node identity did not include d601: ${result.nodeNames.join(",") || "none"}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function collectKubectlJson(result, kind, args) {
|
async function collectKubectlJson(result, kind, args) {
|
||||||
const probe = await runCommand("kubectl", args, { timeoutMs: 10000 });
|
const probe = await runKubectl(args, { timeoutMs: 10000 });
|
||||||
if (probe.exitCode !== 0) {
|
if (probe.exitCode !== 0) {
|
||||||
result.notes.push(`kubectl ${kind} probe failed: ${probe.stderr.trim()}`);
|
result.notes.push(`kubectl ${kind} probe failed: ${probe.stderr.trim()}`);
|
||||||
return;
|
return;
|
||||||
@@ -719,20 +749,20 @@ async function collectDbSecretPresence(result) {
|
|||||||
secretValueRead: false,
|
secretValueRead: false,
|
||||||
redacted: true
|
redacted: true
|
||||||
};
|
};
|
||||||
const exists = await runCommand("kubectl", ["-n", namespace, "get", "secret", ref.secretName, "-o", "name"], {
|
const exists = await runKubectl(["-n", namespace, "get", "secret", ref.secretName, "-o", "name"], {
|
||||||
timeoutMs: 5000
|
timeoutMs: 5000
|
||||||
});
|
});
|
||||||
if (exists.exitCode !== 0) {
|
if (exists.exitCode !== 0) {
|
||||||
result.dbSecret = {
|
result.dbSecret = {
|
||||||
...base,
|
...base,
|
||||||
command: "kubectl get secret -o name",
|
command: nativeKubectlCommand(["-n", namespace, "get", "secret", ref.secretName, "-o", "name"]),
|
||||||
error: exists.stderr.trim() || "secret not observed"
|
error: exists.stderr.trim() || "secret not observed"
|
||||||
};
|
};
|
||||||
result.notes.push(`kubectl DB Secret presence probe failed: ${exists.stderr.trim()}`);
|
result.notes.push(`kubectl DB Secret presence probe failed: ${exists.stderr.trim()}`);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const keyPresence = await runCommand("kubectl", ["-n", namespace, "describe", "secret", ref.secretName], {
|
const keyPresence = await runKubectl(["-n", namespace, "describe", "secret", ref.secretName], {
|
||||||
timeoutMs: 5000
|
timeoutMs: 5000
|
||||||
});
|
});
|
||||||
const keyLinePattern = new RegExp(`^\\s*${escapeRegExp(ref.secretKey)}:\\s+\\d+\\s+bytes\\s*$`, "mu");
|
const keyLinePattern = new RegExp(`^\\s*${escapeRegExp(ref.secretKey)}:\\s+\\d+\\s+bytes\\s*$`, "mu");
|
||||||
@@ -740,7 +770,7 @@ async function collectDbSecretPresence(result) {
|
|||||||
...base,
|
...base,
|
||||||
secretPresent: true,
|
secretPresent: true,
|
||||||
secretKeyPresent: keyPresence.exitCode === 0 && keyLinePattern.test(keyPresence.stdout),
|
secretKeyPresent: keyPresence.exitCode === 0 && keyLinePattern.test(keyPresence.stdout),
|
||||||
command: "kubectl describe secret <name> (key-presence-only)",
|
command: `${nativeKubectlCommand(["-n", namespace, "describe", "secret", ref.secretName])} (key-presence-only)`,
|
||||||
error: keyPresence.exitCode === 0 ? null : keyPresence.stderr.trim()
|
error: keyPresence.exitCode === 0 ? null : keyPresence.stderr.trim()
|
||||||
};
|
};
|
||||||
if (keyPresence.exitCode !== 0) {
|
if (keyPresence.exitCode !== 0) {
|
||||||
@@ -761,20 +791,20 @@ async function collectCodeAgentProviderSecretPresence(result) {
|
|||||||
secretValueRead: false,
|
secretValueRead: false,
|
||||||
redacted: true
|
redacted: true
|
||||||
};
|
};
|
||||||
const exists = await runCommand("kubectl", ["-n", namespace, "get", "secret", ref.secretName, "-o", "name"], {
|
const exists = await runKubectl(["-n", namespace, "get", "secret", ref.secretName, "-o", "name"], {
|
||||||
timeoutMs: 5000
|
timeoutMs: 5000
|
||||||
});
|
});
|
||||||
if (exists.exitCode !== 0) {
|
if (exists.exitCode !== 0) {
|
||||||
result.codeAgentProviderSecret = {
|
result.codeAgentProviderSecret = {
|
||||||
...base,
|
...base,
|
||||||
command: "kubectl get secret -o name",
|
command: nativeKubectlCommand(["-n", namespace, "get", "secret", ref.secretName, "-o", "name"]),
|
||||||
error: exists.stderr.trim() || "secret not observed"
|
error: exists.stderr.trim() || "secret not observed"
|
||||||
};
|
};
|
||||||
result.notes.push(`kubectl Code Agent provider Secret presence probe failed: ${exists.stderr.trim()}`);
|
result.notes.push(`kubectl Code Agent provider Secret presence probe failed: ${exists.stderr.trim()}`);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const keyPresence = await runCommand("kubectl", ["-n", namespace, "describe", "secret", ref.secretName], {
|
const keyPresence = await runKubectl(["-n", namespace, "describe", "secret", ref.secretName], {
|
||||||
timeoutMs: 5000
|
timeoutMs: 5000
|
||||||
});
|
});
|
||||||
const keyLinePattern = new RegExp(`^\\s*${escapeRegExp(ref.secretKey)}:\\s+\\d+\\s+bytes\\s*$`, "mu");
|
const keyLinePattern = new RegExp(`^\\s*${escapeRegExp(ref.secretKey)}:\\s+\\d+\\s+bytes\\s*$`, "mu");
|
||||||
@@ -782,7 +812,7 @@ async function collectCodeAgentProviderSecretPresence(result) {
|
|||||||
...base,
|
...base,
|
||||||
secretPresent: true,
|
secretPresent: true,
|
||||||
secretKeyPresent: keyPresence.exitCode === 0 && keyLinePattern.test(keyPresence.stdout),
|
secretKeyPresent: keyPresence.exitCode === 0 && keyLinePattern.test(keyPresence.stdout),
|
||||||
command: "kubectl describe secret <name> (key-presence-only)",
|
command: `${nativeKubectlCommand(["-n", namespace, "describe", "secret", ref.secretName])} (key-presence-only)`,
|
||||||
error: keyPresence.exitCode === 0 ? null : keyPresence.stderr.trim()
|
error: keyPresence.exitCode === 0 ? null : keyPresence.stderr.trim()
|
||||||
};
|
};
|
||||||
if (keyPresence.exitCode !== 0) {
|
if (keyPresence.exitCode !== 0) {
|
||||||
|
|||||||
@@ -1042,16 +1042,19 @@ async function validateLiveProbes(reporter, catalog, timeoutMs) {
|
|||||||
});
|
});
|
||||||
} else {
|
} else {
|
||||||
const k3sEvidence = [];
|
const k3sEvidence = [];
|
||||||
|
const nativeKubectl = (args) => ["KUBECONFIG=/etc/rancher/k3s/k3s.yaml", "kubectl", ...args];
|
||||||
for (const probe of [
|
for (const probe of [
|
||||||
["kubectl", ["version", "--client=true"]],
|
["env", nativeKubectl(["version", "--client=true"])],
|
||||||
["kubectl", ["config", "current-context"]],
|
["env", nativeKubectl(["config", "current-context"])],
|
||||||
["kubectl", ["auth", "can-i", "get", "pods", "-n", "hwlab-dev"]],
|
["env", nativeKubectl(["get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"])],
|
||||||
["kubectl", ["get", "namespace", "hwlab-dev", "-o", "json"]],
|
["env", nativeKubectl(["auth", "can-i", "get", "pods", "-n", "hwlab-dev"])],
|
||||||
["kubectl", ["-n", "hwlab-dev", "get", "deploy,svc,job,cm", "-o", "name"]]
|
["env", nativeKubectl(["get", "namespace", "hwlab-dev", "-o", "json"])],
|
||||||
|
["env", nativeKubectl(["-n", "hwlab-dev", "get", "deploy,svc,job,cm", "-o", "name"])]
|
||||||
]) {
|
]) {
|
||||||
k3sEvidence.push(await run(probe[0], probe[1], { timeoutMs }));
|
k3sEvidence.push(await run(probe[0], probe[1], { timeoutMs }));
|
||||||
}
|
}
|
||||||
const ok = k3sEvidence.every((probe) => probe.ok);
|
const nodeProbe = k3sEvidence[2];
|
||||||
|
const ok = k3sEvidence.every((probe) => probe.ok) && String(nodeProbe.stdout ?? "").split(/\s+/u).includes("d601");
|
||||||
reporter.check("d601-k3s-read-access", "k3s", ok ? "pass" : "blocked", ok ? "Read-only kubectl probes reached hwlab-dev." : "At least one read-only kubectl probe failed.", k3sEvidence);
|
reporter.check("d601-k3s-read-access", "k3s", ok ? "pass" : "blocked", ok ? "Read-only kubectl probes reached hwlab-dev." : "At least one read-only kubectl probe failed.", k3sEvidence);
|
||||||
if (!ok) {
|
if (!ok) {
|
||||||
reporter.block({
|
reporter.block({
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ import { promisify } from "node:util";
|
|||||||
|
|
||||||
const execFileAsync = promisify(execFile);
|
const execFileAsync = promisify(execFile);
|
||||||
const defaultRegistryPrefix = "127.0.0.1:5000/hwlab";
|
const defaultRegistryPrefix = "127.0.0.1:5000/hwlab";
|
||||||
|
const d601NativeKubeconfigPath = "/etc/rancher/k3s/k3s.yaml";
|
||||||
const imagePullBackoffReasons = new Set(["ErrImagePull", "ImagePullBackOff", "InvalidImageName"]);
|
const imagePullBackoffReasons = new Set(["ErrImagePull", "ImagePullBackOff", "InvalidImageName"]);
|
||||||
|
|
||||||
function commandLine(command, args) {
|
function commandLine(command, args) {
|
||||||
@@ -297,10 +298,11 @@ async function probeK3sPullAccess(registryPrefix, timeoutMs) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const nativeKubectlArgs = (args) => ["KUBECONFIG=" + d601NativeKubeconfigPath, "kubectl", ...args];
|
||||||
const [context, canGetPods, podsResult] = await Promise.all([
|
const [context, canGetPods, podsResult] = await Promise.all([
|
||||||
run("kubectl", ["config", "current-context"], { timeoutMs }),
|
run("env", nativeKubectlArgs(["config", "current-context"]), { timeoutMs }),
|
||||||
run("kubectl", ["auth", "can-i", "get", "pods", "-n", "hwlab-dev"], { timeoutMs }),
|
run("env", nativeKubectlArgs(["auth", "can-i", "get", "pods", "-n", "hwlab-dev"]), { timeoutMs }),
|
||||||
run("kubectl", ["-n", "hwlab-dev", "get", "pods", "-o", "json"], { timeoutMs })
|
run("env", nativeKubectlArgs(["-n", "hwlab-dev", "get", "pods", "-o", "json"]), { timeoutMs })
|
||||||
]);
|
]);
|
||||||
const canReadPods = canGetPods.ok && canGetPods.stdout === "yes" && podsResult.ok;
|
const canReadPods = canGetPods.ok && canGetPods.stdout === "yes" && podsResult.ok;
|
||||||
if (!canReadPods) {
|
if (!canReadPods) {
|
||||||
@@ -314,6 +316,7 @@ async function probeK3sPullAccess(registryPrefix, timeoutMs) {
|
|||||||
summary: "Read-only kubectl probes could not inspect hwlab-dev pods, so k3s pull access is blocked.",
|
summary: "Read-only kubectl probes could not inspect hwlab-dev pods, so k3s pull access is blocked.",
|
||||||
evidence: {
|
evidence: {
|
||||||
kubectlAvailable: true,
|
kubectlAvailable: true,
|
||||||
|
kubeconfig: d601NativeKubeconfigPath,
|
||||||
context: summarizeCommand(context),
|
context: summarizeCommand(context),
|
||||||
canGetPods: summarizeCommand(canGetPods),
|
canGetPods: summarizeCommand(canGetPods),
|
||||||
pods: summarizeCommand(podsResult),
|
pods: summarizeCommand(podsResult),
|
||||||
@@ -341,6 +344,7 @@ async function probeK3sPullAccess(registryPrefix, timeoutMs) {
|
|||||||
: "Read-only kubectl can inspect hwlab-dev, but no current pod proves k3s pulled this registry prefix."),
|
: "Read-only kubectl can inspect hwlab-dev, but no current pod proves k3s pulled this registry prefix."),
|
||||||
evidence: {
|
evidence: {
|
||||||
kubectlAvailable: true,
|
kubectlAvailable: true,
|
||||||
|
kubeconfig: d601NativeKubeconfigPath,
|
||||||
context: summarizeCommand(context),
|
context: summarizeCommand(context),
|
||||||
canGetPods: summarizeCommand(canGetPods),
|
canGetPods: summarizeCommand(canGetPods),
|
||||||
podCount: pods.length,
|
podCount: pods.length,
|
||||||
|
|||||||
Reference in New Issue
Block a user