fix: make root file reclaim exactly selectable

This commit is contained in:
pikastech
2026-07-21 21:20:26 +02:00
parent e7b82a491a
commit 3bc85ca718
9 changed files with 281 additions and 18 deletions
+4
View File
@@ -199,6 +199,8 @@ bun scripts/cli.ts gc remote <providerId> status --job-id <id>
```bash ```bash
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only bun scripts/cli.ts gc remote <node> plan --memory-pressure-only
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> status --job-id <id> bun scripts/cli.ts gc remote <node> status --job-id <id>
bun scripts/cli.ts gc remote <node> memory bun scripts/cli.ts gc remote <node> memory
``` ```
@@ -211,6 +213,8 @@ bun scripts/cli.ts gc remote <node> memory
- job 未终态时不得提前执行 `memory` 或重试 WebProbe - job 未终态时不得提前执行 `memory` 或重试 WebProbe
- 终态后执行 `memory`,只有不再命中 owning YAML 的 `manualComparator` 时才可重试人工启动; - 终态后执行 `memory`,只有不再命中 owning YAML 的 `manualComparator` 时才可重试人工启动;
- 内存处置不得改用未带 `--memory-pressure-only` 的通用 plan/run。 - 内存处置不得改用未带 `--memory-pressure-only` 的通用 plan/run。
- 只执行一个 cgroup 回收目标时,plan/run 必须携带相同的 `--memory-reclaim-config-id <id>`
- `<id>` 只能来自 owning YAML `memoryPressure.cgroupReclaim.targets[].id`,不得按运行面身份或候选排序猜测。
- sentinel cadence - sentinel cadence
- 命中 owning YAML 的 `sentinelComparator` 时返回 `skipped-wait-next-round` - 命中 owning YAML 的 `sentinelComparator` 时返回 `skipped-wait-next-round`
- 不创建 observer、不启动 Chromium、不自动执行 GC - 不创建 observer、不启动 Chromium、不自动执行 GC
+5
View File
@@ -28,6 +28,8 @@
- `--full` 只用于下钻完整候选身份明细,不作为 scoped run 的前置步骤; - `--full` 只用于下钻完整候选身份明细,不作为 scoped run 的前置步骤;
- 不得混入通用磁盘 GC 候选; - 不得混入通用磁盘 GC 候选;
- 执行入口必须显式携带 `--confirm` - 执行入口必须显式携带 `--confirm`
- 需要只执行一个 cgroup 回收策略时,plan 和 run 必须同时携带 `--memory-reclaim-config-id <id>`
- `<id>` 只匹配 owning YAML `memoryPressure.cgroupReclaim.targets[].id`,精确模式不收集或执行其他 cgroup 与进程候选。
- `gc remote <providerId> memory` - `gc remote <providerId> memory`
- 只读读取 `/proc/meminfo``MemAvailable` - 只读读取 `/proc/meminfo``MemAvailable`
- 不执行 cluster preflight、候选扫描或 mutation - 不执行 cluster preflight、候选扫描或 mutation
@@ -232,6 +234,8 @@ JD01 Web observe artifact 是一等 GC 对象。state root 必须来自 YAML
```bash ```bash
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only bun scripts/cli.ts gc remote <node> plan --memory-pressure-only
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> status --job-id <id> bun scripts/cli.ts gc remote <node> status --job-id <id>
bun scripts/cli.ts gc remote <node> memory bun scripts/cli.ts gc remote <node> memory
``` ```
@@ -240,6 +244,7 @@ bun scripts/cli.ts gc remote <node> memory
- 先复核 `plan --memory-pressure-only` 的候选、保护对象、预计回收量和 `runEligibility` - 先复核 `plan --memory-pressure-only` 的候选、保护对象、预计回收量和 `runEligibility`
- 只有 `runEligibility.allowed=true` 时才允许继续; - 只有 `runEligibility.allowed=true` 时才允许继续;
- 只执行参数完全匹配的 `run --confirm --memory-pressure-only` - 只执行参数完全匹配的 `run --confirm --memory-pressure-only`
- 精确 cgroup plan 使用 `--memory-reclaim-config-id` 时,run 必须携带同一个 YAML `id`
- `run` 返回 job id 后,重复执行 `status --job-id <id>` 直到明确终态; - `run` 返回 job id 后,重复执行 `status --job-id <id>` 直到明确终态;
- job 未终态时,禁止提前执行 `memory` 或重试 WebProbe - job 未终态时,禁止提前执行 `memory` 或重试 WebProbe
- job 终态后执行 `memory`,以新的 `/proc/meminfo` 快照重新判定; - job 终态后执行 `memory`,以新的 `/proc/meminfo` 快照重新判定;
@@ -251,8 +251,10 @@ YAML运维状态输出应区分三类依赖:必须保留的 host bootstrap 面
平台 GC 应允许 owning YAML 使用稳定工作负载选择器声明精确 cgroup 回收目标。Kubernetes workload selector 至少包含 namespace、非空 labels 和最大匹配 Pod 数;host service 使用仅包含稳定 unit 名的 `systemdUnitSelector`。root cgroup 文件页回收使用唯一的 `rootCgroupSelector`,不得用任意路径选择器代替。目标可用内存、单目标回收上限、swappiness、inactive anonymous memory 下限、文件页回收下限、swap 保留线和优先级继续由 YAML 声明。YAML 不得保存 Pod UID、容器 ID、MainPID、进程 startTicks、当前 cgroup 路径、cgroup inode 或从运行面反解出的可调数值。 平台 GC 应允许 owning YAML 使用稳定工作负载选择器声明精确 cgroup 回收目标。Kubernetes workload selector 至少包含 namespace、非空 labels 和最大匹配 Pod 数;host service 使用仅包含稳定 unit 名的 `systemdUnitSelector`。root cgroup 文件页回收使用唯一的 `rootCgroupSelector`,不得用任意路径选择器代替。目标可用内存、单目标回收上限、swappiness、inactive anonymous memory 下限、文件页回收下限、swap 保留线和优先级继续由 YAML 声明。YAML 不得保存 Pod UID、容器 ID、MainPID、进程 startTicks、当前 cgroup 路径、cgroup inode 或从运行面反解出的可调数值。
受控 CLI 应允许操作人员按 owning YAML 中唯一且稳定的 cgroup 回收目标 `id` 精确生成 plan 和 run。精确选择不得从当前 Pod、进程、cgroup 路径或候选排序推断目标;同一次 run 只能执行该 `id` 解析出的候选,不得在其之前、之后或并行执行其他 cgroup 回收目标或进程回收候选。
受控 plan 对 Kubernetes selector 只选择 Running Pod,并披露 selector、解析出的 namespace、Pod、UID、cgroup、当前内存、inactive anonymous memory 和保护原因。对 `systemdUnitSelector` 必须通过 `systemctl show` 读取 `LoadState``ActiveState``SubState``ControlGroup``MainPID`,只接受 loaded、active、running 的 serviceControlGroup 必须唯一、绝对、位于 `/system.slice`,映射到 `/sys/fs/cgroup/system.slice` 下的真实非 symlink 目录,且 MainPID 当前 cgroup 必须与 ControlGroup 一致。plan 同时记录 MainPID 的 `/proc/<pid>/stat` startTicks,禁止从 unit 名猜测路径或回退到父 cgroup。 受控 plan 对 Kubernetes selector 只选择 Running Pod,并披露 selector、解析出的 namespace、Pod、UID、cgroup、当前内存、inactive anonymous memory 和保护原因。对 `systemdUnitSelector` 必须通过 `systemctl show` 读取 `LoadState``ActiveState``SubState``ControlGroup``MainPID`,只接受 loaded、active、running 的 serviceControlGroup 必须唯一、绝对、位于 `/system.slice`,映射到 `/sys/fs/cgroup/system.slice` 下的真实非 symlink 目录,且 MainPID 当前 cgroup 必须与 ControlGroup 一致。plan 同时记录 MainPID 的 `/proc/<pid>/stat` startTicks,禁止从 unit 名猜测路径或回退到父 cgroup。
root cgroup plan 必须确认 YAML 声明的 cgroup 根是当前 cgroup v2 mount 的根目录,并记录 mount 身份和目录 inode;任意 symlink、非 cgroup v2、非 mount root 或身份读取失败都必须保护。文件页候选固定使用 `swappiness=0`,并同时读取 `/proc/meminfo` 与 root `memory.stat``file``inactive_file``slab_reclaimable` 证据。候选只有在宿主与 root cgroup 的文件页及可回收 slab 上界都达到 YAML 下限,且每项必要证据完整时才可进入 plan。请求字节必须标记为 `kernel-request-upper-bound`,不得把文件页或 slab 观测量宣称为确定收益。 root cgroup plan 必须确认 YAML 声明的 cgroup 根是当前 cgroup v2 mount 的根目录,并记录 mount 身份和目录 inode;任意 symlink、非 cgroup v2、非 mount root 或身份读取失败都必须保护。文件页候选固定使用 `swappiness=0`,并同时读取 `/proc/meminfo` 与 root `memory.stat``file``inactive_file``slab_reclaimable` 证据。候选只有在宿主与 root cgroup 的文件页及可回收 slab 上界都达到 YAML 下限,且每项必要证据完整时才可进入 plan。请求字节必须标记为 `kernel-request-upper-bound`,不得把文件页或 slab 观测量宣称为确定收益。
执行前必须使用同一 YAML selector 重新解析当前目标。Kubernetes workload 逐项核对 Pod UID 与 cgroup 路径;systemd service 逐项核对 unit、`LoadState``ActiveState``SubState``ControlGroup`、MainPID 与 startTicksroot cgroup 逐项核对 cgroup v2 mount 身份、目录 inode、`/proc/meminfo`、root `memory.stat` 和全部文件页下限。各类目标都必须重新核对全部回收参数。身份变化、目标已满足、证据变化后低于下限或候选不再合格时跳过且结构化披露;只有身份闭合的当前 cgroup 才能写入有界 `memory.reclaim`。root 文件页回收不得使用非零 swappiness、全局 `drop_caches`、workload 停止或重启。终态必须分别披露执行前后宿主 `MemAvailable`、root `memory.stat`、真实增量、剩余目标缺口和内核错误;内核请求上限、文件页上界和 Swap 都不得计入实际收益。 执行前必须使用同一 YAML selector 重新解析当前目标。Kubernetes workload 逐项核对 Pod UID 与 cgroup 路径;systemd service 逐项核对 unit、`LoadState``ActiveState``SubState``ControlGroup`、MainPID 与 startTicksroot cgroup 逐项核对 cgroup v2 mount 身份、目录 inode、`/proc/meminfo`、root `memory.stat` 和全部文件页下限。各类目标都必须重新核对全部回收参数。身份变化、`MemAvailable` 大于等于 YAML 目标、证据变化后低于下限或候选不再合格时跳过且结构化披露;只有身份闭合的当前 cgroup 才能写入有界 `memory.reclaim`。root 文件页回收不得使用非零 swappiness、全局 `drop_caches`、workload 停止或重启。终态必须分别披露执行前后宿主 `MemAvailable`、root `memory.stat`、真实增量、剩余目标缺口和内核错误;compact 与 full 终态都必须保留这些判定证据。内核请求上限、文件页上界和 Swap 都不得计入实际收益。
+2 -1
View File
@@ -465,7 +465,8 @@ def collect_memory_distribution(observed_at):
"swapExcludedFromAvailability": True, "swapExcludedFromAvailability": True,
"availableBytes": available, "availableBytes": available,
"targetAvailableBytes": target, "targetAvailableBytes": target,
"targetGapBytes": max(0, target - available + 1), "targetMet": target > 0 and available >= target,
"targetGapBytes": max(0, target - available),
"anonPagesBytes": meminfo.get("AnonPages"), "anonPagesBytes": meminfo.get("AnonPages"),
"cachedAndBuffersBytes": cached, "cachedAndBuffersBytes": cached,
"slabBytes": meminfo.get("Slab"), "slabBytes": meminfo.get("Slab"),
@@ -13,6 +13,7 @@ function runPythonFixture(body: string): Record<string, unknown> {
const fixture = [ const fixture = [
"import json, os, re, signal, time", "import json, os, re, signal, time",
"PROVIDER_ID = 'NC01'", "PROVIDER_ID = 'NC01'",
"OPTIONS = {}",
"def safe_int(value, fallback=0):", "def safe_int(value, fallback=0):",
" try: return int(value)", " try: return int(value)",
" except (TypeError, ValueError): return fallback", " except (TypeError, ValueError): return fallback",
@@ -34,6 +35,22 @@ function runPythonFixture(body: string): Record<string, unknown> {
return JSON.parse(result.stdout.trim()) as Record<string, unknown>; return JSON.parse(result.stdout.trim()) as Record<string, unknown>;
} }
function runRunnerPythonFixture(functionNames: string[], body: string): Record<string, unknown> {
const source = Buffer.from(runnerSource, "utf8").toString("base64");
const fixture = [
"import ast, base64, json",
`source = base64.b64decode(${JSON.stringify(source)}).decode("utf-8")`,
`wanted = set(${JSON.stringify(functionNames)})`,
"tree = ast.parse(source)",
"selected = [node for node in tree.body if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name in wanted]",
"exec(compile(ast.Module(body=selected, type_ignores=[]), '<gc-remote-runner-fixture>', 'exec'))",
body,
].join("\n");
const result = spawnSync("python3", ["-"], { input: fixture, encoding: "utf8" });
assert.equal(result.status, 0, result.stderr || result.stdout);
return JSON.parse(result.stdout.trim()) as Record<string, unknown>;
}
test("memory-pressure candidate fixtures protect active observers and classify only closed stale/orphan trees", () => { test("memory-pressure candidate fixtures protect active observers and classify only closed stale/orphan trees", () => {
const result = runPythonFixture(String.raw` const result = runPythonFixture(String.raw`
MEMORY_CONFIG = { MEMORY_CONFIG = {
@@ -300,6 +317,160 @@ print(json.dumps({
assert.equal(projected.candidatePreview.candidates[0].resolvedRootCgroup.inode, 1); assert.equal(projected.candidatePreview.candidates[0].resolvedRootCgroup.inode, 1);
}); });
test("root-only selection uses the stable YAML config id, excludes anonymous candidates, and treats equality as met", () => {
const result = runPythonFixture(String.raw`
OPTIONS = {"memoryReclaimConfigId": "root-file-cache"}
MEMORY_CONFIG = {
"attribution": {"cgroupRoot": "/sys/fs/cgroup", "commandTimeoutSeconds": 20},
"cgroupReclaim": {
"enabled": True, "targetMemAvailableBytes": 3221225472,
"minimumSwapFreeBytesAfterReclaim": 1073741824,
"targets": [
{
"id": "root-file-cache", "rootCgroupSelector": {"scope": "root"},
"reclaimBytes": 536870912, "swappiness": 0,
"minimumInactiveFileBytes": 268435456,
"minimumReclaimableFileSlabBytes": 1073741824, "priority": 2,
},
{
"id": "anonymous-pages", "systemdUnitSelector": {"unit": "k3s.service"},
"reclaimBytes": 536870912, "swappiness": 200,
"minimumInactiveAnonBytes": 536870912, "priority": 1,
},
],
},
}
identity = {
"filesystemType": "cgroup2", "mountRoot": "/", "mountPoint": "/sys/fs/cgroup",
"majorMinor": "0:29", "mountId": 31, "parentMountId": 30, "inode": 1,
}
root_stat = {"file": 1610612736, "inactive_file": 805306368, "slab_reclaimable": 536870912}
root_evidence = {
"ok": True,
"host": {"ok": True, "source": "/proc/meminfo", "reclaimableFileSlabUpperBoundBytes": 2147483648},
"rootMemoryStat": root_stat,
"rootReclaimableFileSlabUpperBoundBytes": 2147483648,
"estimateKind": "kernel-reclaimable-upper-bound",
}
available = 2147483648
collect_memory_snapshot = lambda: {"ok": True, "memory": {"availableBytes": available}, "swap": {"freeBytes": 4294967296}}
resolve_root_cgroup_identity = lambda: {"ok": True, "path": "/sys/fs/cgroup", "identity": identity}
collect_root_file_reclaim_evidence = lambda path: root_evidence
read_cgroup_integer = lambda path, name: 8589934592
read_cgroup_memory_stat = lambda path: root_stat
os.path.realpath = lambda path: path
os.path.isdir = lambda path: True
os.path.islink = lambda path: False
os.path.exists = lambda path: True
os.access = lambda path, mode: True
read_proc_process_table = lambda: (_ for _ in ()).throw(RuntimeError("process scan must not run for exact cgroup selection"))
below = collect_memory_reclaim_candidates()
below_diagnostics = memory_reclaim_scan_diagnostics()
available = 3221225472
equal = collect_memory_reclaim_candidates()
equal_diagnostics = memory_reclaim_scan_diagnostics()
equality_execution = execute_cgroup_reclaim_candidate(below[0])
print(json.dumps({
"belowIds": [item.get("configId") for item in below],
"belowSwappiness": [item.get("swappiness") for item in below],
"belowDiagnostics": below_diagnostics,
"equalCount": len(equal),
"equalDiagnostics": equal_diagnostics,
"equalityExecution": equality_execution,
}))
`);
assert.deepEqual(result.belowIds, ["root-file-cache"]);
assert.deepEqual(result.belowSwappiness, [0]);
const belowCgroup = (result.belowDiagnostics as Record<string, any>).cgroupReclaim;
assert.equal(belowCgroup.selectedConfigId, "root-file-cache");
assert.equal(belowCgroup.totalConfiguredCount, 2);
assert.equal(belowCgroup.eligibleCount, 1);
assert.equal(result.equalCount, 0);
const equalProtected = (result.equalDiagnostics as Record<string, any>).cgroupReclaim.protected;
assert.equal(equalProtected[0].id, "root-file-cache");
assert.equal(equalProtected[0].reason, "target-memavailable-already-met");
const equalityExecution = result.equalityExecution as Record<string, any>;
assert.equal(equalityExecution.outcome, "target-memavailable-already-met");
assert.equal(equalityExecution.kernelWriteAttempted, false);
assert.equal(equalityExecution.targetGapBeforeBytes, 0);
assert.equal(equalityExecution.rootFileReclaimEvidenceBefore.ok, true);
});
test("successful memory worker terminal preserves root evidence and gaps in compact and full output", () => {
const result = runRunnerPythonFixture([
"returned_results",
"summarize_memory_candidates",
"memory_pressure_selection",
"finish_memory_pressure_job",
], String.raw`
PROVIDER_ID = "NC01"
OPTIONS = {"full": False, "resultLimit": 50, "memoryReclaimConfigId": "root-file-cache"}
target = 3221225472
MEMORY_CONFIG = {"cgroupReclaim": {"targetMemAvailableBytes": target}}
def safe_int(value, fallback=0):
try: return int(value)
except (TypeError, ValueError): return fallback
def fmt_bytes(value): return str(value)
def job_paths(job_id): return {"state": "/tmp/fixture-state"}
def status_command(job_id): return "gc remote NC01 status --job-id %s" % job_id
def now_iso(): return "2026-07-21T00:00:01Z"
def update_memory_pressure_job_progress(*args): pass
written = []
def write_memory_job_state(paths, payload): written.append(payload)
snapshots = []
def collect_memory_snapshot(): return snapshots.pop(0)
before_evidence = {"ok": True, "rootMemoryStat": {"file": 20, "inactive_file": 10, "slab_reclaimable": 5}}
after_evidence = {"ok": True, "rootMemoryStat": {"file": 15, "inactive_file": 5, "slab_reclaimable": 4}}
executed = []
def execute(candidate):
executed.append(candidate.get("configId"))
return {
"status": "succeeded", "outcome": "partial-reclaim", "reclaimedMemoryBytes": 100,
"hostMemAvailableDeltaBytes": 100, "cgroupMemoryDeltaBytes": 80,
"cgroupMemoryBeforeBytes": 1000, "cgroupMemoryAfterBytes": 920,
"kernelWriteAttempted": True, "kernelWriteError": None,
"memoryBefore": {"memory": {"availableBytes": target - 100}},
"memoryAfter": {"memory": {"availableBytes": target}},
"rootFileReclaimEvidenceBefore": before_evidence,
"rootFileReclaimEvidenceAfter": after_evidence,
"targetGapBeforeBytes": 100, "targetGapAfterBytes": 0,
}
candidate = {
"id": "cgroup-memory-reclaim:root-file-cache", "configId": "root-file-cache",
"kind": "cgroup-memory-reclaim", "reclaimBytes": 100, "swappiness": 0,
"fixtureFullOnly": "kept-only-in-full",
}
snapshots[:] = [
{"ok": True, "memory": {"availableBytes": target - 100}},
{"ok": True, "memory": {"availableBytes": target}},
]
finish_memory_pressure_job("compact", [candidate], "2026-07-21T00:00:00Z", 10, 20)
compact = written[-1]
OPTIONS["full"] = True
snapshots[:] = [
{"ok": True, "memory": {"availableBytes": target - 100}},
{"ok": True, "memory": {"availableBytes": target}},
]
finish_memory_pressure_job("full", [candidate], "2026-07-21T00:00:00Z", 10, 20)
full = written[-1]
print(json.dumps({"compact": compact, "full": full, "executed": executed}))
`);
const compact = result.compact as Record<string, any>;
const full = result.full as Record<string, any>;
assert.deepEqual(result.executed, ["root-file-cache", "root-file-cache"]);
assert.deepEqual(compact.results[0].rootFileReclaimEvidenceBefore.rootMemoryStat, { file: 20, inactive_file: 10, slab_reclaimable: 5 });
assert.deepEqual(compact.results[0].rootFileReclaimEvidenceAfter.rootMemoryStat, { file: 15, inactive_file: 5, slab_reclaimable: 4 });
assert.equal(compact.results[0].targetGapBeforeBytes, 100);
assert.equal(compact.results[0].targetGapAfterBytes, 0);
assert.equal(compact.results[0].fixtureFullOnly, undefined);
assert.equal(full.results[0].fixtureFullOnly, "kept-only-in-full");
assert.equal(full.results[0].targetGapAfterBytes, 0);
assert.equal(compact.summary.targetMet, true);
assert.equal(compact.summary.targetGapBytes, 0);
assert.equal(compact.summary.outcome, "memory-target-met");
assert.equal(compact.selection.configId, "root-file-cache");
});
test("default memory-pressure plan projects an actionable YAML-bounded preview below stdout limit", () => { test("default memory-pressure plan projects an actionable YAML-bounded preview below stdout limit", () => {
const processCandidate = (index: number): Record<string, unknown> => ({ const processCandidate = (index: number): Record<string, unknown> => ({
id: `browser-orphan-process-tree:${1000 + index}:12345`, id: `browser-orphan-process-tree:${1000 + index}:12345`,
+34 -4
View File
@@ -1505,6 +1505,25 @@ def summarize_memory_candidates(candidates, returned):
"byKind": by_kind, "byKind": by_kind,
} }
def memory_pressure_selection():
config_id = OPTIONS.get("memoryReclaimConfigId")
if not isinstance(config_id, str) or not config_id:
return {"mode": "all-eligible", "configId": None}
return {
"mode": "yaml-cgroup-reclaim-config-id",
"configId": config_id,
"configSource": "config/unidesk-cli.yaml#gc.remote.targets.%s.memoryPressure.cgroupReclaim.targets[].id" % PROVIDER_ID,
}
def memory_pressure_option_suffix():
selection = memory_pressure_selection()
suffix = " --memory-pressure-only"
if selection.get("configId"):
suffix += " --memory-reclaim-config-id %s" % selection.get("configId")
return suffix
def assert_tmp_candidate(path): def assert_tmp_candidate(path):
resolved = os.path.abspath(path) resolved = os.path.abspath(path)
if not resolved.startswith("/tmp/"): if not resolved.startswith("/tmp/"):
@@ -1674,6 +1693,10 @@ def returned_results(results):
"hostMemAvailableDeltaBytes", "hostMemAvailableDeltaBytes",
"kernelWriteAttempted", "kernelWriteAttempted",
"kernelWriteError", "kernelWriteError",
"rootFileReclaimEvidenceBefore",
"rootFileReclaimEvidenceAfter",
"targetGapBeforeBytes",
"targetGapAfterBytes",
"processCount", "processCount",
"termSignalCount", "termSignalCount",
"killSignalCount", "killSignalCount",
@@ -1712,6 +1735,7 @@ def plan_payload(observed_at, preflight, protected, candidates, visible):
"options": OPTIONS, "options": OPTIONS,
"memoryBefore": collect_memory_snapshot(), "memoryBefore": collect_memory_snapshot(),
"summary": summary, "summary": summary,
"selection": memory_pressure_selection(),
"candidateScan": scan, "candidateScan": scan,
"candidates": visible, "candidates": visible,
"protected": { "protected": {
@@ -1721,8 +1745,8 @@ def plan_payload(observed_at, preflight, protected, candidates, visible):
}, },
"policy": { "policy": {
"requiresRunConfirm": True, "requiresRunConfirm": True,
"planCommand": "bun scripts/cli.ts gc remote %s plan --memory-pressure-only" % PROVIDER_ID, "planCommand": "bun scripts/cli.ts gc remote %s plan%s" % (PROVIDER_ID, memory_pressure_option_suffix()),
"runCommand": "bun scripts/cli.ts gc remote %s run --confirm --memory-pressure-only" % PROVIDER_ID, "runCommand": "bun scripts/cli.ts gc remote %s run --confirm%s" % (PROVIDER_ID, memory_pressure_option_suffix()),
"candidateKinds": ["cgroup-memory-reclaim", "browser-orphan-process-tree", "stale-web-observer-process-tree"], "candidateKinds": ["cgroup-memory-reclaim", "browser-orphan-process-tree", "stale-web-observer-process-tree"],
"neverTouches": ["disk candidates", "apt cache", "journal", "tmp", "container images", "active observer process trees", "workload lifecycle"], "neverTouches": ["disk candidates", "apt cache", "journal", "tmp", "container images", "active observer process trees", "workload lifecycle"],
"termination": "SIGTERM, bounded wait, identity reclose, SIGKILL only same-tree residual", "termination": "SIGTERM, bounded wait, identity reclose, SIGKILL only same-tree residual",
@@ -2591,6 +2615,10 @@ def finish_memory_pressure_job(job_id, candidates, observed_at, worker_pid, work
"lateAllowlistedIdentityCount": execution.get("lateAllowlistedIdentityCount"), "lateAllowlistedIdentityCount": execution.get("lateAllowlistedIdentityCount"),
"memoryBefore": execution.get("memoryBefore"), "memoryBefore": execution.get("memoryBefore"),
"memoryAfter": execution.get("memoryAfter"), "memoryAfter": execution.get("memoryAfter"),
"rootFileReclaimEvidenceBefore": execution.get("rootFileReclaimEvidenceBefore"),
"rootFileReclaimEvidenceAfter": execution.get("rootFileReclaimEvidenceAfter"),
"targetGapBeforeBytes": execution.get("targetGapBeforeBytes"),
"targetGapAfterBytes": execution.get("targetGapAfterBytes"),
}) })
results.append(item) results.append(item)
except Exception as exc: except Exception as exc:
@@ -2613,11 +2641,11 @@ def finish_memory_pressure_job(job_id, candidates, observed_at, worker_pid, work
}) })
target_available = safe_int((MEMORY_CONFIG.get("cgroupReclaim") or {}).get("targetMemAvailableBytes")) target_available = safe_int((MEMORY_CONFIG.get("cgroupReclaim") or {}).get("targetMemAvailableBytes"))
after_available = safe_int((memory_after.get("memory") or {}).get("availableBytes")) after_available = safe_int((memory_after.get("memory") or {}).get("availableBytes"))
target_met = target_available > 0 and after_available > target_available target_met = target_available > 0 and after_available >= target_available
summary.update({ summary.update({
"targetMemAvailableBytes": target_available, "targetMemAvailableBytes": target_available,
"targetMet": target_met, "targetMet": target_met,
"targetGapBytes": max(0, target_available - after_available + 1) if target_available > 0 else None, "targetGapBytes": max(0, target_available - after_available) if target_available > 0 else None,
"outcome": "memory-target-met" if target_met else "memory-target-not-met", "outcome": "memory-target-met" if target_met else "memory-target-not-met",
}) })
payload = { payload = {
@@ -2643,6 +2671,7 @@ def finish_memory_pressure_job(job_id, candidates, observed_at, worker_pid, work
"memoryBefore": memory_before, "memoryBefore": memory_before,
"memoryAfter": memory_after, "memoryAfter": memory_after,
"summary": summary, "summary": summary,
"selection": memory_pressure_selection(),
"results": returned, "results": returned,
} }
write_memory_job_state(paths, payload) write_memory_job_state(paths, payload)
@@ -2669,6 +2698,7 @@ def start_memory_pressure_job(observed_at, candidates):
"observedAt": observed_at, "observedAt": observed_at,
"startedAt": observed_at, "startedAt": observed_at,
"summary": summary, "summary": summary,
"selection": memory_pressure_selection(),
"candidateScan": scan, "candidateScan": scan,
} }
if scan.get("blocked"): if scan.get("blocked"):
+33 -10
View File
@@ -679,10 +679,27 @@ def resolve_cgroup_reclaim_target(item):
}) })
return {"ok": True, "matches": matches} return {"ok": True, "matches": matches}
def collect_cgroup_reclaim_candidates(): def requested_memory_reclaim_config_id():
options = globals().get("OPTIONS")
value = options.get("memoryReclaimConfigId") if isinstance(options, dict) else None
return value if isinstance(value, str) and re.match(r"^[A-Za-z0-9._-]{1,80}$", value) else None
def collect_cgroup_reclaim_candidates(selected_config_id=None):
global _MEMORY_RECLAIM_CONFIG_ERROR, _CGROUP_RECLAIM_DIAGNOSTICS global _MEMORY_RECLAIM_CONFIG_ERROR, _CGROUP_RECLAIM_DIAGNOSTICS
targets = configured_cgroup_reclaim_targets() all_targets = configured_cgroup_reclaim_targets()
diagnostics = {"configuredCount": len(targets), "eligibleCount": 0, "protected": []} targets = all_targets
if selected_config_id is not None:
targets = [item for item in all_targets if item.get("id") == selected_config_id]
if not targets and _MEMORY_RECLAIM_CONFIG_ERROR is None:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-config-id-not-found:%s" % selected_config_id
diagnostics = {
"configuredCount": len(targets),
"totalConfiguredCount": len(all_targets),
"selectedConfigId": selected_config_id,
"eligibleCount": 0,
"protected": [],
}
_CGROUP_RECLAIM_DIAGNOSTICS = diagnostics _CGROUP_RECLAIM_DIAGNOSTICS = diagnostics
if _MEMORY_RECLAIM_CONFIG_ERROR or not targets: if _MEMORY_RECLAIM_CONFIG_ERROR or not targets:
return [] return []
@@ -716,7 +733,7 @@ def collect_cgroup_reclaim_candidates():
protected_reason = "cgroup-path-unavailable-or-redirected" protected_reason = "cgroup-path-unavailable-or-redirected"
elif not os.path.exists(reclaim_path) or not os.access(reclaim_path, os.W_OK): elif not os.path.exists(reclaim_path) or not os.access(reclaim_path, os.W_OK):
protected_reason = "memory-reclaim-unavailable" protected_reason = "memory-reclaim-unavailable"
elif available > safe_int(item.get("targetMemAvailableBytes")): elif available >= safe_int(item.get("targetMemAvailableBytes")):
protected_reason = "target-memavailable-already-met" protected_reason = "target-memavailable-already-met"
elif safe_int(item.get("swappiness")) > 0 and swap_free - safe_int(item.get("reclaimBytes")) < safe_int(item.get("minimumSwapFreeBytesAfterReclaim")): elif safe_int(item.get("swappiness")) > 0 and swap_free - safe_int(item.get("reclaimBytes")) < safe_int(item.get("minimumSwapFreeBytesAfterReclaim")):
protected_reason = "minimum-swap-free-reserve" protected_reason = "minimum-swap-free-reserve"
@@ -806,6 +823,9 @@ def collect_memory_reclaim_candidates():
_PROCESS_PRESSURE_DIAGNOSTICS = {} _PROCESS_PRESSURE_DIAGNOSTICS = {}
_CGROUP_RECLAIM_DIAGNOSTICS = {"configuredCount": 0, "eligibleCount": 0, "protected": []} _CGROUP_RECLAIM_DIAGNOSTICS = {"configuredCount": 0, "eligibleCount": 0, "protected": []}
_OBSERVE_RUN_CACHE.clear() _OBSERVE_RUN_CACHE.clear()
selected_config_id = requested_memory_reclaim_config_id()
if selected_config_id is not None:
return collect_cgroup_reclaim_candidates(selected_config_id)
required_keys = [ required_keys = [
"observeStateRoots", "minimumReadableObserveRoots", "observeRunDepth", "observeScanLimit", "staleRunMaxAgeHours", "observeStateRoots", "minimumReadableObserveRoots", "observeRunDepth", "observeScanLimit", "staleRunMaxAgeHours",
"processScanLimit", "processStaleMinAgeSeconds", "maxMemoryReclaimCandidates", "processScanLimit", "processStaleMinAgeSeconds", "maxMemoryReclaimCandidates",
@@ -1044,6 +1064,13 @@ def execute_cgroup_reclaim_candidate(candidate):
before_available = safe_int((before.get("memory") or {}).get("availableBytes"), None) before_available = safe_int((before.get("memory") or {}).get("availableBytes"), None)
if before_available is None: if before_available is None:
raise RuntimeError("MemAvailable is unavailable before cgroup reclaim") raise RuntimeError("MemAvailable is unavailable before cgroup reclaim")
root_evidence_before = None
if configured.get("rootCgroupSelector") is not None:
if safe_int(configured.get("swappiness")) != 0 or path != (resolved_root_cgroup or {}).get("mountPoint"):
raise RuntimeError("root cgroup file reclaim identity or swappiness is invalid")
root_evidence_before = collect_root_file_reclaim_evidence(path)
if root_evidence_before.get("ok") is not True:
raise RuntimeError("root cgroup file reclaim evidence is unavailable: %s" % root_evidence_before.get("reason"))
if before_available >= safe_int(configured.get("targetMemAvailableBytes")): if before_available >= safe_int(configured.get("targetMemAvailableBytes")):
return { return {
"status": "succeeded", "status": "succeeded",
@@ -1054,6 +1081,8 @@ def execute_cgroup_reclaim_candidate(candidate):
"cgroupMemoryBeforeBytes": read_cgroup_integer(path, "memory.current"), "cgroupMemoryBeforeBytes": read_cgroup_integer(path, "memory.current"),
"cgroupMemoryAfterBytes": read_cgroup_integer(path, "memory.current"), "cgroupMemoryAfterBytes": read_cgroup_integer(path, "memory.current"),
"kernelWriteAttempted": False, "kernelWriteAttempted": False,
"rootFileReclaimEvidenceBefore": root_evidence_before,
"rootFileReclaimEvidenceAfter": root_evidence_before,
"targetGapBeforeBytes": 0, "targetGapBeforeBytes": 0,
"targetGapAfterBytes": 0, "targetGapAfterBytes": 0,
} }
@@ -1067,13 +1096,7 @@ def execute_cgroup_reclaim_candidate(candidate):
inactive_anon = safe_int(read_cgroup_memory_stat(path).get("inactive_anon")) inactive_anon = safe_int(read_cgroup_memory_stat(path).get("inactive_anon"))
if inactive_anon < safe_int(configured.get("minimumInactiveAnonBytes")): if inactive_anon < safe_int(configured.get("minimumInactiveAnonBytes")):
raise RuntimeError("minimum inactive anonymous memory is no longer available") raise RuntimeError("minimum inactive anonymous memory is no longer available")
root_evidence_before = None
if configured.get("rootCgroupSelector") is not None: if configured.get("rootCgroupSelector") is not None:
if safe_int(configured.get("swappiness")) != 0 or path != (resolved_root_cgroup or {}).get("mountPoint"):
raise RuntimeError("root cgroup file reclaim identity or swappiness is invalid")
root_evidence_before = collect_root_file_reclaim_evidence(path)
if root_evidence_before.get("ok") is not True:
raise RuntimeError("root cgroup file reclaim evidence is unavailable: %s" % root_evidence_before.get("reason"))
if safe_int((root_evidence_before.get("rootMemoryStat") or {}).get("inactive_file")) < safe_int(configured.get("minimumInactiveFileBytes")): if safe_int((root_evidence_before.get("rootMemoryStat") or {}).get("inactive_file")) < safe_int(configured.get("minimumInactiveFileBytes")):
raise RuntimeError("minimum root inactive file memory is no longer available") raise RuntimeError("minimum root inactive file memory is no longer available")
if safe_int(root_evidence_before.get("rootReclaimableFileSlabUpperBoundBytes")) < safe_int(configured.get("minimumReclaimableFileSlabBytes")): if safe_int(root_evidence_before.get("rootReclaimableFileSlabUpperBoundBytes")) < safe_int(configured.get("minimumReclaimableFileSlabBytes")):
+26 -2
View File
@@ -38,6 +38,7 @@ interface RemoteGcOptions {
historyLimit: number; historyLimit: number;
saveSnapshot: boolean; saveSnapshot: boolean;
memoryPressureOnly: boolean; memoryPressureOnly: boolean;
memoryReclaimConfigId?: string;
buildCacheOnly: boolean; buildCacheOnly: boolean;
hostDockerOnly: boolean; hostDockerOnly: boolean;
} }
@@ -71,6 +72,7 @@ const DEFAULT_REMOTE_OPTIONS: RemoteGcOptions = {
historyLimit: 12, historyLimit: 12,
saveSnapshot: true, saveSnapshot: true,
memoryPressureOnly: false, memoryPressureOnly: false,
memoryReclaimConfigId: undefined,
buildCacheOnly: false, buildCacheOnly: false,
hostDockerOnly: false, hostDockerOnly: false,
}; };
@@ -182,6 +184,14 @@ export async function runRemoteGcCommand(config: UniDeskConfig, providerId: stri
}; };
} }
const options = parseRemoteGcOptions(args); const options = parseRemoteGcOptions(args);
if (options.memoryReclaimConfigId !== undefined && !options.memoryPressureOnly) {
return {
ok: false,
error: "gc-remote-memory-reclaim-config-id-requires-memory-pressure-only",
configId: options.memoryReclaimConfigId,
requiredFlag: "--memory-pressure-only",
};
}
if (options.memoryPressureOnly && subaction !== "plan" && subaction !== "run") { if (options.memoryPressureOnly && subaction !== "plan" && subaction !== "run") {
return { return {
ok: false, ok: false,
@@ -209,8 +219,8 @@ export async function runRemoteGcCommand(config: UniDeskConfig, providerId: stri
dryRun: true, dryRun: true,
mutation: false, mutation: false,
requiredFlag: "--confirm", requiredFlag: "--confirm",
planCommand: `bun scripts/cli.ts gc remote ${providerId} plan${options.memoryPressureOnly ? " --memory-pressure-only" : registryFlag}`, planCommand: `bun scripts/cli.ts gc remote ${providerId} plan${options.memoryPressureOnly ? memoryPressureOptionSuffix(options) : registryFlag}`,
runCommand: `bun scripts/cli.ts gc remote ${providerId} run --confirm${options.memoryPressureOnly ? " --memory-pressure-only" : registryFlag}`, runCommand: `bun scripts/cli.ts gc remote ${providerId} run --confirm${options.memoryPressureOnly ? memoryPressureOptionSuffix(options) : registryFlag}`,
}; };
} }
return await runRemoteGc(config, providerId, "run", options); return await runRemoteGc(config, providerId, "run", options);
@@ -317,6 +327,10 @@ function parseRemoteGcOptions(args: string[]): RemoteGcOptions {
options.saveSnapshot = false; options.saveSnapshot = false;
} else if (arg === "--memory-pressure-only") { } else if (arg === "--memory-pressure-only") {
options.memoryPressureOnly = true; options.memoryPressureOnly = true;
} else if (arg === "--memory-reclaim-config-id") {
const value = args[++index];
if (!value || !/^[A-Za-z0-9._-]{1,80}$/u.test(value)) throw new Error("--memory-reclaim-config-id must be a YAML cgroupReclaim target id");
options.memoryReclaimConfigId = value;
} else if (arg === "--build-cache-only") { } else if (arg === "--build-cache-only") {
options.buildCacheOnly = true; options.buildCacheOnly = true;
} else if (arg === "--host-docker-only") { } else if (arg === "--host-docker-only") {
@@ -328,6 +342,13 @@ function parseRemoteGcOptions(args: string[]): RemoteGcOptions {
return options; return options;
} }
function memoryPressureOptionSuffix(options: RemoteGcOptions): string {
const selection = options.memoryReclaimConfigId === undefined
? ""
: ` --memory-reclaim-config-id ${options.memoryReclaimConfigId}`;
return ` --memory-pressure-only${selection}`;
}
function parseNonNegativeNumber(name: string, raw: string | undefined): number { function parseNonNegativeNumber(name: string, raw: string | undefined): number {
const value = Number(raw); const value = Number(raw);
if (!Number.isFinite(value) || value < 0) throw new Error(`${name} must be a non-negative number`); if (!Number.isFinite(value) || value < 0) throw new Error(`${name} must be a non-negative number`);
@@ -647,6 +668,8 @@ function compactMemoryPressureCandidateScan(value: unknown): Record<string, unkn
}, },
cgroupReclaim: { cgroupReclaim: {
configuredCount: cgroupReclaim.configuredCount, configuredCount: cgroupReclaim.configuredCount,
totalConfiguredCount: cgroupReclaim.totalConfiguredCount,
selectedConfigId: cgroupReclaim.selectedConfigId,
eligibleCount: cgroupReclaim.eligibleCount, eligibleCount: cgroupReclaim.eligibleCount,
protectedCount: cgroupProtected.length, protectedCount: cgroupProtected.length,
protected: cgroupProtected.map((item) => { protected: cgroupProtected.map((item) => {
@@ -705,6 +728,7 @@ export function projectRemoteGcResult(
observedAt: payload.observedAt, observedAt: payload.observedAt,
memoryBefore: payload.memoryBefore, memoryBefore: payload.memoryBefore,
summary: payload.summary, summary: payload.summary,
selection: payload.selection,
candidateScan: compactScan, candidateScan: compactScan,
runEligibility: { runEligibility: {
allowed: reasons.length === 0, allowed: reasons.length === 0,
+3
View File
@@ -527,6 +527,8 @@ function gcHelp(): unknown {
"bun scripts/cli.ts gc remote NC01 retention run --confirm --limit 500", "bun scripts/cli.ts gc remote NC01 retention run --confirm --limit 500",
"bun scripts/cli.ts gc remote NC01 plan --memory-pressure-only", "bun scripts/cli.ts gc remote NC01 plan --memory-pressure-only",
"bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only", "bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only",
"bun scripts/cli.ts gc remote NC01 plan --memory-pressure-only --memory-reclaim-config-id <yaml-target-id>",
"bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only --memory-reclaim-config-id <yaml-target-id>",
"bun scripts/cli.ts gc remote NC01 status --job-id <job-id>", "bun scripts/cli.ts gc remote NC01 status --job-id <job-id>",
"bun scripts/cli.ts gc remote G14 plan --target-use-percent 50 --include-hwlab-registry", "bun scripts/cli.ts gc remote G14 plan --target-use-percent 50 --include-hwlab-registry",
"bun scripts/cli.ts gc remote G14 run --confirm", "bun scripts/cli.ts gc remote G14 run --confirm",
@@ -558,6 +560,7 @@ function gcHelp(): unknown {
"--registry-min-age-hours N": "remote registry only: keep all tags newer than N hours; default 48, minimum 0", "--registry-min-age-hours N": "remote registry only: keep all tags newer than N hours; default 48, minimum 0",
"--target-use-percent N": "evaluate whether planned candidates can reduce root filesystem use to N%; reports required reclaim, projected use, shortfall and safe-stop decision", "--target-use-percent N": "evaluate whether planned candidates can reduce root filesystem use to N%; reports required reclaim, projected use, shortfall and safe-stop decision",
"--job-id ID": "remote status only: inspect a long-running remote gc job", "--job-id ID": "remote status only: inspect a long-running remote gc job",
"--memory-reclaim-config-id ID": "remote memory-pressure plan/run only: select exactly one owning YAML cgroupReclaim target id and exclude every other cgroup or process candidate",
"--limit N": "number of candidates returned and executed by run when --full is not set; default 50", "--limit N": "number of candidates returned and executed by run when --full is not set; default 50",
"--result-limit N": "number of per-candidate run results returned when --full is not set; default 50", "--result-limit N": "number of per-candidate run results returned when --full is not set; default 50",
"--full|--raw": "return and run against all candidates rather than the default bounded page", "--full|--raw": "return and run against all candidates rather than the default bounded page",