Merge pull request #2813 from pikasTech/feat/2811-root-cgroup-file-cache-reclaim

为 NC01 增加 root cgroup 文件缓存受控回收
This commit is contained in:
Lyon
2026-07-22 03:32:33 +08:00
committed by GitHub
10 changed files with 763 additions and 31 deletions
+4
View File
@@ -199,6 +199,8 @@ bun scripts/cli.ts gc remote <providerId> status --job-id <id>
```bash ```bash
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only bun scripts/cli.ts gc remote <node> plan --memory-pressure-only
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> status --job-id <id> bun scripts/cli.ts gc remote <node> status --job-id <id>
bun scripts/cli.ts gc remote <node> memory bun scripts/cli.ts gc remote <node> memory
``` ```
@@ -211,6 +213,8 @@ bun scripts/cli.ts gc remote <node> memory
- job 未终态时不得提前执行 `memory` 或重试 WebProbe - job 未终态时不得提前执行 `memory` 或重试 WebProbe
- 终态后执行 `memory`,只有不再命中 owning YAML 的 `manualComparator` 时才可重试人工启动; - 终态后执行 `memory`,只有不再命中 owning YAML 的 `manualComparator` 时才可重试人工启动;
- 内存处置不得改用未带 `--memory-pressure-only` 的通用 plan/run。 - 内存处置不得改用未带 `--memory-pressure-only` 的通用 plan/run。
- 只执行一个 cgroup 回收目标时,plan/run 必须携带相同的 `--memory-reclaim-config-id <id>`
- `<id>` 只能来自 owning YAML `memoryPressure.cgroupReclaim.targets[].id`,不得按运行面身份或候选排序猜测。
- sentinel cadence - sentinel cadence
- 命中 owning YAML 的 `sentinelComparator` 时返回 `skipped-wait-next-round` - 命中 owning YAML 的 `sentinelComparator` 时返回 `skipped-wait-next-round`
- 不创建 observer、不启动 Chromium、不自动执行 GC - 不创建 observer、不启动 Chromium、不自动执行 GC
+9 -1
View File
@@ -223,7 +223,7 @@ gc:
minAgeHours: 336 minAgeHours: 336
maxDeletePerRun: 5 maxDeletePerRun: 5
memoryPressure: memoryPressure:
planPreviewLimit: 4 planPreviewLimit: 2
processPatterns: processPatterns:
- k3s-server - k3s-server
- containerd - containerd
@@ -291,6 +291,14 @@ gc:
targetMemAvailableBytes: 3221225472 targetMemAvailableBytes: 3221225472
minimumSwapFreeBytesAfterReclaim: 34359738368 minimumSwapFreeBytesAfterReclaim: 34359738368
targets: targets:
- id: root-cgroup-file-cache
rootCgroupSelector:
scope: root
reclaimBytes: 805306368
swappiness: 0
minimumInactiveFileBytes: 268435456
minimumReclaimableFileSlabBytes: 1073741824
priority: 2
- id: k3s-service-inactive-anon - id: k3s-service-inactive-anon
systemdUnitSelector: systemdUnitSelector:
unit: k3s.service unit: k3s.service
+5
View File
@@ -28,6 +28,8 @@
- `--full` 只用于下钻完整候选身份明细,不作为 scoped run 的前置步骤; - `--full` 只用于下钻完整候选身份明细,不作为 scoped run 的前置步骤;
- 不得混入通用磁盘 GC 候选; - 不得混入通用磁盘 GC 候选;
- 执行入口必须显式携带 `--confirm` - 执行入口必须显式携带 `--confirm`
- 需要只执行一个 cgroup 回收策略时,plan 和 run 必须同时携带 `--memory-reclaim-config-id <id>`
- `<id>` 只匹配 owning YAML `memoryPressure.cgroupReclaim.targets[].id`,精确模式不收集或执行其他 cgroup 与进程候选。
- `gc remote <providerId> memory` - `gc remote <providerId> memory`
- 只读读取 `/proc/meminfo``MemAvailable` - 只读读取 `/proc/meminfo``MemAvailable`
- 不执行 cluster preflight、候选扫描或 mutation - 不执行 cluster preflight、候选扫描或 mutation
@@ -232,6 +234,8 @@ JD01 Web observe artifact 是一等 GC 对象。state root 必须来自 YAML
```bash ```bash
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only bun scripts/cli.ts gc remote <node> plan --memory-pressure-only
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only
bun scripts/cli.ts gc remote <node> plan --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> run --confirm --memory-pressure-only --memory-reclaim-config-id <id>
bun scripts/cli.ts gc remote <node> status --job-id <id> bun scripts/cli.ts gc remote <node> status --job-id <id>
bun scripts/cli.ts gc remote <node> memory bun scripts/cli.ts gc remote <node> memory
``` ```
@@ -240,6 +244,7 @@ bun scripts/cli.ts gc remote <node> memory
- 先复核 `plan --memory-pressure-only` 的候选、保护对象、预计回收量和 `runEligibility` - 先复核 `plan --memory-pressure-only` 的候选、保护对象、预计回收量和 `runEligibility`
- 只有 `runEligibility.allowed=true` 时才允许继续; - 只有 `runEligibility.allowed=true` 时才允许继续;
- 只执行参数完全匹配的 `run --confirm --memory-pressure-only` - 只执行参数完全匹配的 `run --confirm --memory-pressure-only`
- 精确 cgroup plan 使用 `--memory-reclaim-config-id` 时,run 必须携带同一个 YAML `id`
- `run` 返回 job id 后,重复执行 `status --job-id <id>` 直到明确终态; - `run` 返回 job id 后,重复执行 `status --job-id <id>` 直到明确终态;
- job 未终态时,禁止提前执行 `memory` 或重试 WebProbe - job 未终态时,禁止提前执行 `memory` 或重试 WebProbe
- job 终态后执行 `memory`,以新的 `/proc/meminfo` 快照重新判定; - job 终态后执行 `memory`,以新的 `/proc/meminfo` 快照重新判定;
@@ -72,7 +72,7 @@ YAML运维负责 HWLAB/UniDesk 自有平台配置的真相源、解析、渲染
| host proxy client | 新节点 k3s 安装前由 `trans` 分发的 0 依赖出网代理客户端,用于给 Docker、k3s、containerd、包管理器、Git 和 git-mirror 提供初始出网。 | | host proxy client | 新节点 k3s 安装前由 `trans` 分发的 0 依赖出网代理客户端,用于给 Docker、k3s、containerd、包管理器、Git 和 git-mirror 提供初始出网。 |
| 配置组合 | 通过 baseline、configRef、继承、覆盖和变量渲染生成有效计划的配置方式;组合结果是执行视图,不是第二份可编辑真相源。 | | 配置组合 | 通过 baseline、configRef、继承、覆盖和变量渲染生成有效计划的配置方式;组合结果是执行视图,不是第二份可编辑真相源。 |
| 稳定工作负载选择器 | 由 namespace 和 Kubernetes labels 组成、跨 Pod 重建保持不变的工作负载身份;Pod UID 只能是运行面解析结果。 | | 稳定工作负载选择器 | 由 namespace 和 Kubernetes labels 组成、跨 Pod 重建保持不变的工作负载身份;Pod UID 只能是运行面解析结果。 |
| 精确 cgroup 回收 | 受控 CLI 将稳定工作负载选择器解析为当前 Running Pod cgroup,并在执行前重新解析身份后请求有界 `memory.reclaim`。 | | 精确 cgroup 回收 | 受控 CLI 将稳定工作负载或 root cgroup 选择器解析为当前 cgroup,并在执行前重新解析身份后请求有界 `memory.reclaim`。 |
## 4. 系统边界和接口 ## 4. 系统边界和接口
@@ -249,8 +249,12 @@ YAML运维状态输出应区分三类依赖:必须保留的 host bootstrap 面
| --- | --- | --- | --- | | --- | --- | --- | --- |
| OPS-YAML-REQ-012 | 精确工作负载回收 | PJ2026-01060315 精确工作负载回收 | [平台运维](PJ2026-0106-platform-ops.md)、[NC01 CI资源治理](PJ2026-01060107-nc01-ci-resource-governance.md) | | OPS-YAML-REQ-012 | 精确工作负载回收 | PJ2026-01060315 精确工作负载回收 | [平台运维](PJ2026-0106-platform-ops.md)、[NC01 CI资源治理](PJ2026-01060107-nc01-ci-resource-governance.md) |
平台 GC 应允许 owning YAML 使用稳定工作负载选择器声明精确 cgroup 回收目标。Kubernetes workload selector 至少包含 namespace、非空 labels 和最大匹配 Pod 数;host service 使用仅包含稳定 unit 名的 `systemdUnitSelector`。目标可用内存、单目标回收上限、swappiness、inactive anonymous memory 下限、swap 保留线和优先级继续由 YAML 声明。YAML 不得保存 Pod UID、容器 ID、MainPID、进程 startTicks、当前 cgroup 路径、cgroup inode 或从运行面反解出的可调数值。 平台 GC 应允许 owning YAML 使用稳定工作负载选择器声明精确 cgroup 回收目标。Kubernetes workload selector 至少包含 namespace、非空 labels 和最大匹配 Pod 数;host service 使用仅包含稳定 unit 名的 `systemdUnitSelector`root cgroup 文件页回收使用唯一的 `rootCgroupSelector`,不得用任意路径选择器代替。目标可用内存、单目标回收上限、swappiness、inactive anonymous memory 下限、文件页回收下限、swap 保留线和优先级继续由 YAML 声明。YAML 不得保存 Pod UID、容器 ID、MainPID、进程 startTicks、当前 cgroup 路径、cgroup inode 或从运行面反解出的可调数值。
受控 CLI 应允许操作人员按 owning YAML 中唯一且稳定的 cgroup 回收目标 `id` 精确生成 plan 和 run。精确选择不得从当前 Pod、进程、cgroup 路径或候选排序推断目标;同一次 run 只能执行该 `id` 解析出的候选,不得在其之前、之后或并行执行其他 cgroup 回收目标或进程回收候选。
受控 plan 对 Kubernetes selector 只选择 Running Pod,并披露 selector、解析出的 namespace、Pod、UID、cgroup、当前内存、inactive anonymous memory 和保护原因。对 `systemdUnitSelector` 必须通过 `systemctl show` 读取 `LoadState``ActiveState``SubState``ControlGroup``MainPID`,只接受 loaded、active、running 的 serviceControlGroup 必须唯一、绝对、位于 `/system.slice`,映射到 `/sys/fs/cgroup/system.slice` 下的真实非 symlink 目录,且 MainPID 当前 cgroup 必须与 ControlGroup 一致。plan 同时记录 MainPID 的 `/proc/<pid>/stat` startTicks,禁止从 unit 名猜测路径或回退到父 cgroup。 受控 plan 对 Kubernetes selector 只选择 Running Pod,并披露 selector、解析出的 namespace、Pod、UID、cgroup、当前内存、inactive anonymous memory 和保护原因。对 `systemdUnitSelector` 必须通过 `systemctl show` 读取 `LoadState``ActiveState``SubState``ControlGroup``MainPID`,只接受 loaded、active、running 的 serviceControlGroup 必须唯一、绝对、位于 `/system.slice`,映射到 `/sys/fs/cgroup/system.slice` 下的真实非 symlink 目录,且 MainPID 当前 cgroup 必须与 ControlGroup 一致。plan 同时记录 MainPID 的 `/proc/<pid>/stat` startTicks,禁止从 unit 名猜测路径或回退到父 cgroup。
执行前必须使用同一 YAML selector 重新解析当前目标。Kubernetes workload 逐项核对 Pod UID 与 cgroup 路径;systemd service 逐项核对 unit、`LoadState``ActiveState``SubState``ControlGroup`、MainPID 与 startTicks;两类目标都必须重新核对全部回收参数。身份变化、目标已满足或候选不再合格时跳过且结构化披露;只有身份闭合的当前 cgroup 才能写入有界 `memory.reclaim`。执行不得停止、重启、驱逐或迁移 workload,也不得把内核请求上限宣称为实际收益;终态继续分别披露执行前后 cgroup current、宿主 `MemAvailable` 变化、真实收益、目标缺口和 swap 排除状态 root cgroup plan 必须确认 YAML 声明的 cgroup 根是当前 cgroup v2 mount 的根目录,并记录 mount 身份和目录 inode;任意 symlink、非 cgroup v2、非 mount root 或身份读取失败都必须保护。文件页候选固定使用 `swappiness=0`,并同时读取 `/proc/meminfo` 与 root `memory.stat``file``inactive_file``slab_reclaimable` 证据。候选只有在宿主与 root cgroup 的文件页及可回收 slab 上界都达到 YAML 下限,且每项必要证据完整时才可进入 plan。请求字节必须标记为 `kernel-request-upper-bound`,不得把文件页或 slab 观测量宣称为确定收益
执行前必须使用同一 YAML selector 重新解析当前目标。Kubernetes workload 逐项核对 Pod UID 与 cgroup 路径;systemd service 逐项核对 unit、`LoadState``ActiveState``SubState``ControlGroup`、MainPID 与 startTicksroot cgroup 逐项核对 cgroup v2 mount 身份、目录 inode、`/proc/meminfo`、root `memory.stat` 和全部文件页下限。各类目标都必须重新核对全部回收参数。身份变化、`MemAvailable` 大于等于 YAML 目标、证据变化后低于下限或候选不再合格时跳过且结构化披露;只有身份闭合的当前 cgroup 才能写入有界 `memory.reclaim`。root 文件页回收不得使用非零 swappiness、全局 `drop_caches`、workload 停止或重启。终态必须分别披露执行前后宿主 `MemAvailable`、root `memory.stat`、真实增量、剩余目标缺口和内核错误;compact 与 full 终态都必须保留这些判定证据。内核请求上限、文件页上界和 Swap 都不得计入实际收益。
+2 -1
View File
@@ -465,7 +465,8 @@ def collect_memory_distribution(observed_at):
"swapExcludedFromAvailability": True, "swapExcludedFromAvailability": True,
"availableBytes": available, "availableBytes": available,
"targetAvailableBytes": target, "targetAvailableBytes": target,
"targetGapBytes": max(0, target - available + 1), "targetMet": target > 0 and available >= target,
"targetGapBytes": max(0, target - available),
"anonPagesBytes": meminfo.get("AnonPages"), "anonPagesBytes": meminfo.get("AnonPages"),
"cachedAndBuffersBytes": cached, "cachedAndBuffersBytes": cached,
"slabBytes": meminfo.get("Slab"), "slabBytes": meminfo.get("Slab"),
+367 -2
View File
@@ -13,6 +13,7 @@ function runPythonFixture(body: string): Record<string, unknown> {
const fixture = [ const fixture = [
"import json, os, re, signal, time", "import json, os, re, signal, time",
"PROVIDER_ID = 'NC01'", "PROVIDER_ID = 'NC01'",
"OPTIONS = {}",
"def safe_int(value, fallback=0):", "def safe_int(value, fallback=0):",
" try: return int(value)", " try: return int(value)",
" except (TypeError, ValueError): return fallback", " except (TypeError, ValueError): return fallback",
@@ -34,6 +35,22 @@ function runPythonFixture(body: string): Record<string, unknown> {
return JSON.parse(result.stdout.trim()) as Record<string, unknown>; return JSON.parse(result.stdout.trim()) as Record<string, unknown>;
} }
function runRunnerPythonFixture(functionNames: string[], body: string): Record<string, unknown> {
const source = Buffer.from(runnerSource, "utf8").toString("base64");
const fixture = [
"import ast, base64, json, os, re, time",
`source = base64.b64decode(${JSON.stringify(source)}).decode("utf-8")`,
`wanted = set(${JSON.stringify(functionNames)})`,
"tree = ast.parse(source)",
"selected = [node for node in tree.body if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name in wanted]",
"exec(compile(ast.Module(body=selected, type_ignores=[]), '<gc-remote-runner-fixture>', 'exec'))",
body,
].join("\n");
const result = spawnSync("python3", ["-"], { input: fixture, encoding: "utf8" });
assert.equal(result.status, 0, result.stderr || result.stdout);
return JSON.parse(result.stdout.trim()) as Record<string, unknown>;
}
test("memory-pressure candidate fixtures protect active observers and classify only closed stale/orphan trees", () => { test("memory-pressure candidate fixtures protect active observers and classify only closed stale/orphan trees", () => {
const result = runPythonFixture(String.raw` const result = runPythonFixture(String.raw`
MEMORY_CONFIG = { MEMORY_CONFIG = {
@@ -200,6 +217,354 @@ print(json.dumps({"ids": [item["id"] for item in candidates], "swappiness": [ite
assert.match(webObserveSource, /memory\.reclaim/u); assert.match(webObserveSource, /memory\.reclaim/u);
}); });
test("root cgroup file-cache reclaim requires cgroup2 identity and fresh dual-source lower bounds", () => {
const result = runPythonFixture(String.raw`
MEMORY_CONFIG = {
"observeStateRoots": ["/tmp/observe"], "minimumReadableObserveRoots": 1,
"observeRunDepth": 6, "observeScanLimit": 100, "staleRunMaxAgeHours": 1,
"processScanLimit": 100, "processStaleMinAgeSeconds": 3600,
"maxMemoryReclaimCandidates": 20, "terminationGraceSeconds": 1,
"terminationPollMilliseconds": 50, "observerRootPatterns": ["observer-runner.mjs"],
"browserRootPatterns": ["cliDaemon.mjs"], "browserProcessPatterns": ["chromium"],
"attribution": {"cgroupRoot": "/sys/fs/cgroup", "commandTimeoutSeconds": 20},
"cgroupReclaim": {
"enabled": True, "targetMemAvailableBytes": 4294967296,
"minimumSwapFreeBytesAfterReclaim": 402653184,
"targets": [{
"id": "root-file-cache", "rootCgroupSelector": {"scope": "root"},
"reclaimBytes": 805306368, "swappiness": 0,
"minimumInactiveFileBytes": 268435456,
"minimumReclaimableFileSlabBytes": 1073741824,
"priority": 1,
}],
},
}
identity = {
"filesystemType": "cgroup2", "mountRoot": "/", "mountPoint": "/sys/fs/cgroup",
"majorMinor": "0:29", "mountId": 31, "parentMountId": 30, "inode": 1,
"path": "/sys/fs/cgroup",
}
root_stat = {
"anon": 1024, "file": 1610612736, "inactive_anon": 0,
"inactive_file": 805306368, "slab_reclaimable": 536870912,
}
host_evidence = {
"ok": True, "source": "/proc/meminfo", "cachedBytes": 1610612736,
"buffersBytes": 134217728, "slabReclaimableBytes": 536870912,
"cachedAndBuffersBytes": 1744830464,
"reclaimableFileSlabUpperBoundBytes": 2281701376,
"estimateKind": "kernel-reclaimable-upper-bound",
}
fresh_evidence = lambda: {
"ok": True, "host": host_evidence, "rootMemoryStat": root_stat,
"rootReclaimableFileSlabUpperBoundBytes": 2147483648,
"estimateKind": "kernel-reclaimable-upper-bound",
}
read_proc_process_table = lambda: {1: {"pid": 1, "ppid": 0, "sid": 1, "startTicks": 1, "ageSeconds": 1, "rssBytes": 1, "commandLine": "init", "commandPreview": "init"}}
observer_run_records = lambda: []
collect_memory_snapshot = lambda: {"ok": True, "memory": {"availableBytes": 3 * 1024 * 1024 * 1024}, "swap": {"freeBytes": 2 * 1024 * 1024 * 1024}}
resolve_root_cgroup_identity = lambda: {"ok": True, "path": "/sys/fs/cgroup", "identity": identity}
collect_root_file_reclaim_evidence = lambda path: fresh_evidence()
read_cgroup_memory_stat = lambda path: root_stat
read_cgroup_integer = lambda path, name: 8 * 1024 * 1024 * 1024
os.path.realpath = lambda path: path
os.path.isdir = lambda path: True
os.path.islink = lambda path: False
os.path.exists = lambda path: True
os.access = lambda path, mode: True
candidates = collect_memory_reclaim_candidates()
memory_reclaim_write_attempts = []
original_open = open
def tracked_open(path, *args, **kwargs):
if str(path).endswith("/memory.reclaim"):
memory_reclaim_write_attempts.append(path)
return original_open(path, *args, **kwargs)
open = tracked_open
collect_root_file_reclaim_evidence = lambda path: {
**fresh_evidence(),
"rootMemoryStat": {**root_stat, "inactive_file": 134217728},
}
try:
execute_cgroup_reclaim_candidate(candidates[0])
revalidation_error = None
except RuntimeError as error:
revalidation_error = str(error)
print(json.dumps({
"candidate": candidates[0],
"revalidationError": revalidation_error,
"memoryReclaimWriteAttempts": memory_reclaim_write_attempts,
"diagnostics": memory_reclaim_scan_diagnostics(),
}))
`);
const candidate = result.candidate as Record<string, any>;
assert.equal(candidate.rootCgroupSelector.scope, "root");
assert.equal(candidate.resolvedRootCgroup.filesystemType, "cgroup2");
assert.equal(candidate.resolvedRootCgroup.mountRoot, "/");
assert.equal(candidate.swappiness, 0);
assert.equal(candidate.estimateKind, "kernel-request-upper-bound");
assert.equal(candidate.rootFileReclaimEvidence.host.source, "/proc/meminfo");
assert.equal(result.revalidationError, "minimum root inactive file memory is no longer available");
assert.deepEqual(result.memoryReclaimWriteAttempts, []);
assert.equal(((result.diagnostics as Record<string, any>).cgroupReclaim).eligibleCount, 1);
const projected = projectRemoteGcResult({
ok: true,
summary: { candidateCount: 1 },
candidateScan: result.diagnostics,
candidates: [candidate],
}, { memoryPressure: { planPreviewLimit: 4 } }, { action: "plan", memoryPressureOnly: true, full: false }) as Record<string, any>;
assert.equal(projected.runEligibility.allowed, true);
assert.equal(projected.candidatePreview.candidates[0].rootCgroupSelector, "root");
assert.equal(projected.candidatePreview.candidates[0].resolvedRootCgroup.inode, 1);
});
test("root-only selection uses the stable YAML config id, excludes anonymous candidates, and treats equality as met", () => {
const result = runPythonFixture(String.raw`
OPTIONS = {"memoryReclaimConfigId": "root-file-cache"}
MEMORY_CONFIG = {
"attribution": {"cgroupRoot": "/sys/fs/cgroup", "commandTimeoutSeconds": 20},
"cgroupReclaim": {
"enabled": True, "targetMemAvailableBytes": 3221225472,
"minimumSwapFreeBytesAfterReclaim": 1073741824,
"targets": [
{
"id": "root-file-cache", "rootCgroupSelector": {"scope": "root"},
"reclaimBytes": 536870912, "swappiness": 0,
"minimumInactiveFileBytes": 268435456,
"minimumReclaimableFileSlabBytes": 1073741824, "priority": 2,
},
{
"id": "anonymous-pages", "systemdUnitSelector": {"unit": "k3s.service"},
"reclaimBytes": 536870912, "swappiness": 200,
"minimumInactiveAnonBytes": 536870912, "priority": 1,
},
],
},
}
identity = {
"filesystemType": "cgroup2", "mountRoot": "/", "mountPoint": "/sys/fs/cgroup",
"majorMinor": "0:29", "mountId": 31, "parentMountId": 30, "inode": 1,
}
root_stat = {"file": 1610612736, "inactive_file": 805306368, "slab_reclaimable": 536870912}
root_evidence = {
"ok": True,
"host": {"ok": True, "source": "/proc/meminfo", "reclaimableFileSlabUpperBoundBytes": 2147483648},
"rootMemoryStat": root_stat,
"rootReclaimableFileSlabUpperBoundBytes": 2147483648,
"estimateKind": "kernel-reclaimable-upper-bound",
}
available = 2147483648
collect_memory_snapshot = lambda: {"ok": True, "memory": {"availableBytes": available}, "swap": {"freeBytes": 4294967296}}
resolve_root_cgroup_identity = lambda: {"ok": True, "path": "/sys/fs/cgroup", "identity": identity}
collect_root_file_reclaim_evidence = lambda path: root_evidence
read_cgroup_integer = lambda path, name: 8589934592
read_cgroup_memory_stat = lambda path: root_stat
os.path.realpath = lambda path: path
os.path.isdir = lambda path: True
os.path.islink = lambda path: False
os.path.exists = lambda path: True
os.access = lambda path, mode: True
read_proc_process_table = lambda: (_ for _ in ()).throw(RuntimeError("process scan must not run for exact cgroup selection"))
below = collect_memory_reclaim_candidates()
below_diagnostics = memory_reclaim_scan_diagnostics()
available = 3221225472
equal = collect_memory_reclaim_candidates()
equal_diagnostics = memory_reclaim_scan_diagnostics()
equality_execution = execute_cgroup_reclaim_candidate(below[0])
print(json.dumps({
"belowIds": [item.get("configId") for item in below],
"belowSwappiness": [item.get("swappiness") for item in below],
"belowDiagnostics": below_diagnostics,
"equalCount": len(equal),
"equalDiagnostics": equal_diagnostics,
"equalityExecution": equality_execution,
}))
`);
assert.deepEqual(result.belowIds, ["root-file-cache"]);
assert.deepEqual(result.belowSwappiness, [0]);
const belowCgroup = (result.belowDiagnostics as Record<string, any>).cgroupReclaim;
assert.equal(belowCgroup.selectedConfigId, "root-file-cache");
assert.equal(belowCgroup.totalConfiguredCount, 2);
assert.equal(belowCgroup.eligibleCount, 1);
assert.equal(result.equalCount, 0);
const equalProtected = (result.equalDiagnostics as Record<string, any>).cgroupReclaim.protected;
assert.equal(equalProtected[0].id, "root-file-cache");
assert.equal(equalProtected[0].reason, "target-memavailable-already-met");
const equalityExecution = result.equalityExecution as Record<string, any>;
assert.equal(equalityExecution.outcome, "target-memavailable-already-met");
assert.equal(equalityExecution.kernelWriteAttempted, false);
assert.equal(equalityExecution.targetGapBeforeBytes, 0);
assert.equal(equalityExecution.rootFileReclaimEvidenceBefore.ok, true);
});
test("successful memory worker terminal preserves root evidence and gaps in compact and full output", () => {
const result = runRunnerPythonFixture([
"returned_results",
"summarize_memory_candidates",
"memory_pressure_selection",
"finish_memory_pressure_job",
], String.raw`
PROVIDER_ID = "NC01"
OPTIONS = {"full": False, "resultLimit": 50, "memoryReclaimConfigId": "root-file-cache"}
target = 3221225472
MEMORY_CONFIG = {"cgroupReclaim": {"targetMemAvailableBytes": target}}
def safe_int(value, fallback=0):
try: return int(value)
except (TypeError, ValueError): return fallback
def fmt_bytes(value): return str(value)
def job_paths(job_id): return {"state": "/tmp/fixture-state"}
def status_command(job_id): return "gc remote NC01 status --job-id %s" % job_id
def now_iso(): return "2026-07-21T00:00:01Z"
def update_memory_pressure_job_progress(*args): pass
written = []
def write_memory_job_state(paths, payload): written.append(payload)
snapshots = []
def collect_memory_snapshot(): return snapshots.pop(0)
before_evidence = {"ok": True, "rootMemoryStat": {"file": 20, "inactive_file": 10, "slab_reclaimable": 5}}
after_evidence = {"ok": True, "rootMemoryStat": {"file": 15, "inactive_file": 5, "slab_reclaimable": 4}}
executed = []
def execute(candidate):
executed.append(candidate.get("configId"))
return {
"status": "succeeded", "outcome": "partial-reclaim", "reclaimedMemoryBytes": 100,
"hostMemAvailableDeltaBytes": 100, "cgroupMemoryDeltaBytes": 80,
"cgroupMemoryBeforeBytes": 1000, "cgroupMemoryAfterBytes": 920,
"kernelWriteAttempted": True, "kernelWriteError": None,
"memoryBefore": {"memory": {"availableBytes": target - 100}},
"memoryAfter": {"memory": {"availableBytes": target}},
"rootFileReclaimEvidenceBefore": before_evidence,
"rootFileReclaimEvidenceAfter": after_evidence,
"targetGapBeforeBytes": 100, "targetGapAfterBytes": 0,
}
candidate = {
"id": "cgroup-memory-reclaim:root-file-cache", "configId": "root-file-cache",
"kind": "cgroup-memory-reclaim", "reclaimBytes": 100, "swappiness": 0,
"fixtureFullOnly": "kept-only-in-full",
}
snapshots[:] = [
{"ok": True, "memory": {"availableBytes": target - 100}},
{"ok": True, "memory": {"availableBytes": target}},
]
finish_memory_pressure_job("compact", [candidate], "2026-07-21T00:00:00Z", 10, 20)
compact = written[-1]
OPTIONS["full"] = True
snapshots[:] = [
{"ok": True, "memory": {"availableBytes": target - 100}},
{"ok": True, "memory": {"availableBytes": target}},
]
finish_memory_pressure_job("full", [candidate], "2026-07-21T00:00:00Z", 10, 20)
full = written[-1]
print(json.dumps({"compact": compact, "full": full, "executed": executed}))
`);
const compact = result.compact as Record<string, any>;
const full = result.full as Record<string, any>;
assert.deepEqual(result.executed, ["root-file-cache", "root-file-cache"]);
assert.deepEqual(compact.results[0].rootFileReclaimEvidenceBefore.rootMemoryStat, { file: 20, inactive_file: 10, slab_reclaimable: 5 });
assert.deepEqual(compact.results[0].rootFileReclaimEvidenceAfter.rootMemoryStat, { file: 15, inactive_file: 5, slab_reclaimable: 4 });
assert.equal(compact.results[0].targetGapBeforeBytes, 100);
assert.equal(compact.results[0].targetGapAfterBytes, 0);
assert.equal(compact.results[0].fixtureFullOnly, undefined);
assert.equal(full.results[0].fixtureFullOnly, "kept-only-in-full");
assert.equal(full.results[0].targetGapAfterBytes, 0);
assert.equal(compact.summary.targetMet, true);
assert.equal(compact.summary.targetGapBytes, 0);
assert.equal(compact.summary.outcome, "memory-target-met");
assert.equal(compact.selection.configId, "root-file-cache");
});
test("memory-pressure plan and run treat an exact target with zero candidates as a successful no-op", () => {
const result = runRunnerPythonFixture([
"summarize_memory_candidates",
"apply_memory_target_summary",
"memory_pressure_selection",
"memory_pressure_option_suffix",
"plan_payload",
"start_memory_pressure_job",
], String.raw`
PROVIDER_ID = "NC01"
OPTIONS = {"memoryPressureOnly": True, "memoryReclaimConfigId": "root-file-cache"}
target = 3221225472
MEMORY_CONFIG = {"cgroupReclaim": {"targetMemAvailableBytes": target}}
def safe_int(value, fallback=0):
try: return int(value)
except (TypeError, ValueError): return fallback
def fmt_bytes(value): return str(value)
def now_iso(): return "2026-07-21T00:00:01Z"
def status_command(job_id): return "gc remote NC01 status --job-id %s" % job_id
snapshot = {"ok": True, "memory": {"availableBytes": target}, "swap": {"freeBytes": 1}}
snapshot_count = 0
def collect_memory_snapshot():
global snapshot_count
snapshot_count += 1
return snapshot
def memory_reclaim_scan_diagnostics(): return {"ok": True, "blocked": False}
def job_paths(job_id): return {"state": "/tmp/fixture-state"}
written = []
def write_json_atomic(path, payload): written.append(payload)
def unexpected_fork(): raise RuntimeError("zero-candidate target-met run must not fork")
os.fork = unexpected_fork
plan = plan_payload("2026-07-21T00:00:00Z", {}, [], [], [])
run = start_memory_pressure_job("2026-07-21T00:00:00Z", [])
print(json.dumps({"plan": plan, "run": run, "written": written, "snapshotCount": snapshot_count}))
`);
const plan = result.plan as Record<string, any>;
const run = result.run as Record<string, any>;
assert.equal(plan.summary.targetMet, true);
assert.equal(plan.summary.targetGapBytes, 0);
assert.equal(plan.summary.outcome, "memory-target-met");
assert.equal(plan.summary.candidateCount, 0);
assert.equal(run.ok, true);
assert.equal(run.status, "succeeded");
assert.equal(run.outcome, "memory-target-met");
assert.equal(run.summary.targetMet, true);
assert.equal(run.summary.targetGapBytes, 0);
assert.equal(run.summary.outcome, "memory-target-met");
assert.equal(run.mutation, false);
assert.deepEqual(run.results, []);
assert.equal(result.snapshotCount, 2);
assert.deepEqual(result.written, [run]);
const config = Bun.YAML.parse(readFileSync(rootPath("config/unidesk-cli.yaml"), "utf8")) as Record<string, any>;
const projected = projectRemoteGcResult(plan, config.gc.remote.targets.NC01, {
action: "plan",
memoryPressureOnly: true,
full: false,
}) as Record<string, any>;
assert.equal(projected.runEligibility.allowed, true);
assert.equal(projected.runEligibility.allCandidatesEvaluated, true);
assert.deepEqual(projected.runEligibility.reasons, []);
});
test("memory-pressure zero-candidate run remains blocked below the target", () => {
const result = runRunnerPythonFixture([
"summarize_memory_candidates",
"apply_memory_target_summary",
"memory_pressure_selection",
"start_memory_pressure_job",
], String.raw`
PROVIDER_ID = "NC01"
OPTIONS = {"memoryPressureOnly": True, "memoryReclaimConfigId": "root-file-cache"}
target = 3221225472
MEMORY_CONFIG = {"cgroupReclaim": {"targetMemAvailableBytes": target}}
def safe_int(value, fallback=0):
try: return int(value)
except (TypeError, ValueError): return fallback
def fmt_bytes(value): return str(value)
def now_iso(): return "2026-07-21T00:00:01Z"
def status_command(job_id): return "gc remote NC01 status --job-id %s" % job_id
def collect_memory_snapshot(): return {"ok": True, "memory": {"availableBytes": target - 1}}
def memory_reclaim_scan_diagnostics(): return {"ok": True, "blocked": False}
def job_paths(job_id): return {"state": "/tmp/fixture-state"}
def write_json_atomic(path, payload): pass
run = start_memory_pressure_job("2026-07-21T00:00:00Z", [])
print(json.dumps(run))
`);
assert.equal(result.ok, true);
assert.equal(result.status, "blocked");
assert.equal(result.outcome, "no-memory-reclaim-candidates");
assert.equal((result.summary as Record<string, unknown>).targetMet, false);
assert.equal((result.summary as Record<string, unknown>).targetGapBytes, 1);
});
test("default memory-pressure plan projects an actionable YAML-bounded preview below stdout limit", () => { test("default memory-pressure plan projects an actionable YAML-bounded preview below stdout limit", () => {
const processCandidate = (index: number): Record<string, unknown> => ({ const processCandidate = (index: number): Record<string, unknown> => ({
id: `browser-orphan-process-tree:${1000 + index}:12345`, id: `browser-orphan-process-tree:${1000 + index}:12345`,
@@ -289,8 +654,8 @@ test("default memory-pressure plan projects an actionable YAML-bounded preview b
assert.equal(projected.runEligibility.allCandidatesEvaluated, true); assert.equal(projected.runEligibility.allCandidatesEvaluated, true);
assert.equal(projected.candidatePreview.limit, remoteTarget.memoryPressure.planPreviewLimit); assert.equal(projected.candidatePreview.limit, remoteTarget.memoryPressure.planPreviewLimit);
assert.equal(projected.candidatePreview.candidates.length, Math.min(candidates.length, remoteTarget.memoryPressure.planPreviewLimit)); assert.equal(projected.candidatePreview.candidates.length, Math.min(candidates.length, remoteTarget.memoryPressure.planPreviewLimit));
assert.equal(projected.candidatePreview.candidates[2].identityClosure.identitiesComplete, true); assert.equal(projected.candidatePreview.omittedCount, candidates.length - projected.candidatePreview.candidates.length);
assert.equal(projected.candidatePreview.candidates[2].processIdentities, undefined); assert.ok(projected.candidatePreview.candidates.every((candidate: Record<string, unknown>) => candidate.processIdentities === undefined));
assert.equal(projected.policy.runCommand, "bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only"); assert.equal(projected.policy.runCommand, "bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only");
assert.equal(projected.valuesRedacted, true); assert.equal(projected.valuesRedacted, true);
const renderedEnvelope = `${JSON.stringify({ const renderedEnvelope = `${JSON.stringify({
+68 -7
View File
@@ -1505,6 +1505,40 @@ def summarize_memory_candidates(candidates, returned):
"byKind": by_kind, "byKind": by_kind,
} }
def apply_memory_target_summary(summary, memory_snapshot):
target_available = safe_int((MEMORY_CONFIG.get("cgroupReclaim") or {}).get("targetMemAvailableBytes"))
snapshot_memory = memory_snapshot.get("memory") if isinstance(memory_snapshot, dict) else None
available = safe_int(snapshot_memory.get("availableBytes")) if isinstance(snapshot_memory, dict) else 0
target_met = target_available > 0 and available >= target_available
summary.update({
"targetMemAvailableBytes": target_available,
"targetMet": target_met,
"targetGapBytes": max(0, target_available - available) if target_available > 0 else None,
})
if target_met:
summary["outcome"] = "memory-target-met"
return summary
def memory_pressure_selection():
config_id = OPTIONS.get("memoryReclaimConfigId")
if not isinstance(config_id, str) or not config_id:
return {"mode": "all-eligible", "configId": None}
return {
"mode": "yaml-cgroup-reclaim-config-id",
"configId": config_id,
"configSource": "config/unidesk-cli.yaml#gc.remote.targets.%s.memoryPressure.cgroupReclaim.targets[].id" % PROVIDER_ID,
}
def memory_pressure_option_suffix():
selection = memory_pressure_selection()
suffix = " --memory-pressure-only"
if selection.get("configId"):
suffix += " --memory-reclaim-config-id %s" % selection.get("configId")
return suffix
def assert_tmp_candidate(path): def assert_tmp_candidate(path):
resolved = os.path.abspath(path) resolved = os.path.abspath(path)
if not resolved.startswith("/tmp/"): if not resolved.startswith("/tmp/"):
@@ -1674,6 +1708,10 @@ def returned_results(results):
"hostMemAvailableDeltaBytes", "hostMemAvailableDeltaBytes",
"kernelWriteAttempted", "kernelWriteAttempted",
"kernelWriteError", "kernelWriteError",
"rootFileReclaimEvidenceBefore",
"rootFileReclaimEvidenceAfter",
"targetGapBeforeBytes",
"targetGapAfterBytes",
"processCount", "processCount",
"termSignalCount", "termSignalCount",
"killSignalCount", "killSignalCount",
@@ -1698,7 +1736,8 @@ def returned_results(results):
def plan_payload(observed_at, preflight, protected, candidates, visible): def plan_payload(observed_at, preflight, protected, candidates, visible):
if bool(OPTIONS.get("memoryPressureOnly")): if bool(OPTIONS.get("memoryPressureOnly")):
scan = memory_reclaim_scan_diagnostics() scan = memory_reclaim_scan_diagnostics()
summary = summarize_memory_candidates(candidates, visible) memory_before = collect_memory_snapshot()
summary = apply_memory_target_summary(summarize_memory_candidates(candidates, visible), memory_before)
if scan.get("blocked"): if scan.get("blocked"):
summary["outcome"] = "memory-reclaim-scan-blocked" summary["outcome"] = "memory-reclaim-scan-blocked"
return { return {
@@ -1710,8 +1749,9 @@ def plan_payload(observed_at, preflight, protected, candidates, visible):
"mutation": False, "mutation": False,
"observedAt": observed_at, "observedAt": observed_at,
"options": OPTIONS, "options": OPTIONS,
"memoryBefore": collect_memory_snapshot(), "memoryBefore": memory_before,
"summary": summary, "summary": summary,
"selection": memory_pressure_selection(),
"candidateScan": scan, "candidateScan": scan,
"candidates": visible, "candidates": visible,
"protected": { "protected": {
@@ -1721,8 +1761,8 @@ def plan_payload(observed_at, preflight, protected, candidates, visible):
}, },
"policy": { "policy": {
"requiresRunConfirm": True, "requiresRunConfirm": True,
"planCommand": "bun scripts/cli.ts gc remote %s plan --memory-pressure-only" % PROVIDER_ID, "planCommand": "bun scripts/cli.ts gc remote %s plan%s" % (PROVIDER_ID, memory_pressure_option_suffix()),
"runCommand": "bun scripts/cli.ts gc remote %s run --confirm --memory-pressure-only" % PROVIDER_ID, "runCommand": "bun scripts/cli.ts gc remote %s run --confirm%s" % (PROVIDER_ID, memory_pressure_option_suffix()),
"candidateKinds": ["cgroup-memory-reclaim", "browser-orphan-process-tree", "stale-web-observer-process-tree"], "candidateKinds": ["cgroup-memory-reclaim", "browser-orphan-process-tree", "stale-web-observer-process-tree"],
"neverTouches": ["disk candidates", "apt cache", "journal", "tmp", "container images", "active observer process trees", "workload lifecycle"], "neverTouches": ["disk candidates", "apt cache", "journal", "tmp", "container images", "active observer process trees", "workload lifecycle"],
"termination": "SIGTERM, bounded wait, identity reclose, SIGKILL only same-tree residual", "termination": "SIGTERM, bounded wait, identity reclose, SIGKILL only same-tree residual",
@@ -2591,6 +2631,10 @@ def finish_memory_pressure_job(job_id, candidates, observed_at, worker_pid, work
"lateAllowlistedIdentityCount": execution.get("lateAllowlistedIdentityCount"), "lateAllowlistedIdentityCount": execution.get("lateAllowlistedIdentityCount"),
"memoryBefore": execution.get("memoryBefore"), "memoryBefore": execution.get("memoryBefore"),
"memoryAfter": execution.get("memoryAfter"), "memoryAfter": execution.get("memoryAfter"),
"rootFileReclaimEvidenceBefore": execution.get("rootFileReclaimEvidenceBefore"),
"rootFileReclaimEvidenceAfter": execution.get("rootFileReclaimEvidenceAfter"),
"targetGapBeforeBytes": execution.get("targetGapBeforeBytes"),
"targetGapAfterBytes": execution.get("targetGapAfterBytes"),
}) })
results.append(item) results.append(item)
except Exception as exc: except Exception as exc:
@@ -2613,11 +2657,11 @@ def finish_memory_pressure_job(job_id, candidates, observed_at, worker_pid, work
}) })
target_available = safe_int((MEMORY_CONFIG.get("cgroupReclaim") or {}).get("targetMemAvailableBytes")) target_available = safe_int((MEMORY_CONFIG.get("cgroupReclaim") or {}).get("targetMemAvailableBytes"))
after_available = safe_int((memory_after.get("memory") or {}).get("availableBytes")) after_available = safe_int((memory_after.get("memory") or {}).get("availableBytes"))
target_met = target_available > 0 and after_available > target_available target_met = target_available > 0 and after_available >= target_available
summary.update({ summary.update({
"targetMemAvailableBytes": target_available, "targetMemAvailableBytes": target_available,
"targetMet": target_met, "targetMet": target_met,
"targetGapBytes": max(0, target_available - after_available + 1) if target_available > 0 else None, "targetGapBytes": max(0, target_available - after_available) if target_available > 0 else None,
"outcome": "memory-target-met" if target_met else "memory-target-not-met", "outcome": "memory-target-met" if target_met else "memory-target-not-met",
}) })
payload = { payload = {
@@ -2643,6 +2687,7 @@ def finish_memory_pressure_job(job_id, candidates, observed_at, worker_pid, work
"memoryBefore": memory_before, "memoryBefore": memory_before,
"memoryAfter": memory_after, "memoryAfter": memory_after,
"summary": summary, "summary": summary,
"selection": memory_pressure_selection(),
"results": returned, "results": returned,
} }
write_memory_job_state(paths, payload) write_memory_job_state(paths, payload)
@@ -2654,7 +2699,8 @@ def start_memory_pressure_job(observed_at, candidates):
os.getpid(), os.getpid(),
) )
paths = job_paths(job_id) paths = job_paths(job_id)
summary = summarize_memory_candidates(candidates, candidates) memory_before = collect_memory_snapshot()
summary = apply_memory_target_summary(summarize_memory_candidates(candidates, candidates), memory_before)
scan = memory_reclaim_scan_diagnostics() scan = memory_reclaim_scan_diagnostics()
base = { base = {
"ok": scan.get("ok") is True, "ok": scan.get("ok") is True,
@@ -2669,6 +2715,8 @@ def start_memory_pressure_job(observed_at, candidates):
"observedAt": observed_at, "observedAt": observed_at,
"startedAt": observed_at, "startedAt": observed_at,
"summary": summary, "summary": summary,
"memoryBefore": memory_before,
"selection": memory_pressure_selection(),
"candidateScan": scan, "candidateScan": scan,
} }
if scan.get("blocked"): if scan.get("blocked"):
@@ -2683,6 +2731,19 @@ def start_memory_pressure_job(observed_at, candidates):
write_json_atomic(paths["state"], terminal) write_json_atomic(paths["state"], terminal)
return terminal return terminal
if not candidates: if not candidates:
if summary.get("targetMet") is True:
terminal = dict(base)
terminal.update({
"ok": True,
"status": "succeeded",
"outcome": "memory-target-met",
"mutation": False,
"finishedAt": now_iso(),
"memoryAfter": memory_before,
"results": [],
})
write_json_atomic(paths["state"], terminal)
return terminal
terminal = dict(base) terminal = dict(base)
terminal.update({ terminal.update({
"status": "blocked", "status": "blocked",
+202 -10
View File
@@ -323,12 +323,98 @@ def read_cgroup_memory_stat(path):
with open(os.path.join(path, "memory.stat"), "r", encoding="utf-8") as handle: with open(os.path.join(path, "memory.stat"), "r", encoding="utf-8") as handle:
for line in handle: for line in handle:
parts = line.split() parts = line.split()
if len(parts) == 2 and parts[0] in set(["anon", "file", "inactive_anon", "inactive_file"]): if len(parts) == 2 and parts[0] in set(["anon", "file", "inactive_anon", "inactive_file", "slab_reclaimable"]):
selected[parts[0]] = safe_int(parts[1]) selected[parts[0]] = safe_int(parts[1])
except OSError: except OSError:
return {} return {}
return selected return selected
def read_host_file_reclaim_evidence():
selected = {}
try:
with open("/proc/meminfo", "r", encoding="utf-8") as handle:
for line in handle:
key, separator, raw = line.partition(":")
if separator and key in set(["Cached", "Buffers", "SReclaimable"]):
value = raw.strip().split()[0]
selected[key] = int(value) * 1024
except (OSError, ValueError, IndexError):
return {"ok": False, "reason": "host-file-reclaim-evidence-unavailable"}
if any(key not in selected for key in ["Cached", "Buffers", "SReclaimable"]):
return {"ok": False, "reason": "host-file-reclaim-evidence-incomplete"}
cached_and_buffers = selected["Cached"] + selected["Buffers"]
return {
"ok": True,
"source": "/proc/meminfo",
"cachedBytes": selected["Cached"],
"buffersBytes": selected["Buffers"],
"slabReclaimableBytes": selected["SReclaimable"],
"cachedAndBuffersBytes": cached_and_buffers,
"reclaimableFileSlabUpperBoundBytes": cached_and_buffers + selected["SReclaimable"],
"estimateKind": "kernel-reclaimable-upper-bound",
}
def resolve_root_cgroup_identity():
cgroup_root = (MEMORY_CONFIG.get("attribution") or {}).get("cgroupRoot")
if not isinstance(cgroup_root, str) or not os.path.isabs(cgroup_root) or os.path.normpath(cgroup_root) != cgroup_root:
return {"ok": False, "reason": "root-cgroup-config-path-invalid"}
if os.path.realpath(cgroup_root) != cgroup_root or not os.path.isdir(cgroup_root) or os.path.islink(cgroup_root):
return {"ok": False, "reason": "root-cgroup-path-unavailable-or-redirected"}
matches = []
try:
with open("/proc/self/mountinfo", "r", encoding="utf-8") as handle:
for line in handle:
before, separator, after = line.strip().partition(" - ")
if not separator:
continue
left = before.split()
right = after.split()
if len(left) < 6 or len(right) < 3 or right[0] != "cgroup2" or left[4] != cgroup_root:
continue
matches.append({
"mountId": safe_int(left[0], None),
"parentMountId": safe_int(left[1], None),
"majorMinor": left[2],
"mountRoot": left[3],
"mountPoint": left[4],
"filesystemType": right[0],
})
except OSError:
return {"ok": False, "reason": "root-cgroup-mountinfo-unavailable"}
if len(matches) != 1:
return {"ok": False, "reason": "root-cgroup-v2-mount-not-unique"}
identity = matches[0]
if identity.get("mountRoot") != "/" or identity.get("mountId") is None or identity.get("parentMountId") is None:
return {"ok": False, "reason": "configured-cgroup-path-is-not-mount-root"}
try:
stat = os.stat(cgroup_root, follow_symlinks=False)
except OSError:
return {"ok": False, "reason": "root-cgroup-stat-unavailable"}
stat_major_minor = "%s:%s" % (os.major(stat.st_dev), os.minor(stat.st_dev))
if stat_major_minor != identity.get("majorMinor") or stat.st_ino <= 0:
return {"ok": False, "reason": "root-cgroup-mount-identity-mismatch"}
identity["inode"] = stat.st_ino
identity["path"] = cgroup_root
return {"ok": True, "path": cgroup_root, "identity": identity}
def collect_root_file_reclaim_evidence(path):
host = read_host_file_reclaim_evidence()
stat = read_cgroup_memory_stat(path)
if host.get("ok") is not True:
return {"ok": False, "reason": host.get("reason"), "host": host, "rootMemoryStat": stat}
if any(key not in stat for key in ["file", "inactive_file", "slab_reclaimable"]):
return {"ok": False, "reason": "root-memory-stat-file-evidence-incomplete", "host": host, "rootMemoryStat": stat}
if safe_int(stat.get("inactive_file")) > safe_int(stat.get("file")):
return {"ok": False, "reason": "root-memory-stat-file-evidence-inconsistent", "host": host, "rootMemoryStat": stat}
root_upper_bound = safe_int(stat.get("file")) + safe_int(stat.get("slab_reclaimable"))
return {
"ok": True,
"host": host,
"rootMemoryStat": stat,
"rootReclaimableFileSlabUpperBoundBytes": root_upper_bound,
"estimateKind": "kernel-reclaimable-upper-bound",
}
def configured_cgroup_reclaim_targets(): def configured_cgroup_reclaim_targets():
global _MEMORY_RECLAIM_CONFIG_ERROR global _MEMORY_RECLAIM_CONFIG_ERROR
policy = MEMORY_CONFIG.get("cgroupReclaim") policy = MEMORY_CONFIG.get("cgroupReclaim")
@@ -361,18 +447,25 @@ def configured_cgroup_reclaim_targets():
path = item.get("path") path = item.get("path")
workload_selector = item.get("workloadSelector") workload_selector = item.get("workloadSelector")
systemd_unit_selector = item.get("systemdUnitSelector") systemd_unit_selector = item.get("systemdUnitSelector")
root_cgroup_selector = item.get("rootCgroupSelector")
reclaim_bytes = item.get("reclaimBytes") reclaim_bytes = item.get("reclaimBytes")
swappiness = item.get("swappiness") swappiness = item.get("swappiness")
minimum_inactive_anon = item.get("minimumInactiveAnonBytes", 0) minimum_inactive_anon = item.get("minimumInactiveAnonBytes", 0)
minimum_inactive_file = item.get("minimumInactiveFileBytes", 0)
minimum_file_slab = item.get("minimumReclaimableFileSlabBytes", 0)
if not isinstance(target_id, str) or not re.match(r"^[A-Za-z0-9._-]{1,80}$", target_id) or target_id in seen_ids: if not isinstance(target_id, str) or not re.match(r"^[A-Za-z0-9._-]{1,80}$", target_id) or target_id in seen_ids:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-target-id-invalid:%s" % index _MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-target-id-invalid:%s" % index
return [] return []
if path is not None: if path is not None:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-path-selector-unsupported:%s" % target_id _MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-path-selector-unsupported:%s" % target_id
return [] return []
if (workload_selector is None) == (systemd_unit_selector is None): if sum(selector is not None for selector in [workload_selector, systemd_unit_selector, root_cgroup_selector]) != 1:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-target-identity-invalid:%s" % target_id _MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-target-identity-invalid:%s" % target_id
return [] return []
if root_cgroup_selector is not None:
if not isinstance(root_cgroup_selector, dict) or root_cgroup_selector != {"scope": "root"}:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-root-selector-invalid:%s" % target_id
return []
if systemd_unit_selector is not None: if systemd_unit_selector is not None:
if not isinstance(systemd_unit_selector, dict) or set(systemd_unit_selector.keys()) != set(["unit"]): if not isinstance(systemd_unit_selector, dict) or set(systemd_unit_selector.keys()) != set(["unit"]):
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-systemd-selector-invalid:%s" % target_id _MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-systemd-selector-invalid:%s" % target_id
@@ -413,6 +506,20 @@ def configured_cgroup_reclaim_targets():
if swappiness > 0 and minimum_inactive_anon < reclaim_bytes: if swappiness > 0 and minimum_inactive_anon < reclaim_bytes:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-minimum-inactive-anon-too-small:%s" % target_id _MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-minimum-inactive-anon-too-small:%s" % target_id
return [] return []
if any(isinstance(value, bool) or not isinstance(value, int) or value < 0 for value in [minimum_inactive_file, minimum_file_slab]):
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-file-lower-bound-invalid:%s" % target_id
return []
if root_cgroup_selector is not None and (
swappiness != 0
or minimum_inactive_anon != 0
or minimum_inactive_file <= 0
or minimum_file_slab < reclaim_bytes
):
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-root-cgroup-file-reclaim-policy-invalid:%s" % target_id
return []
if root_cgroup_selector is None and (minimum_inactive_file != 0 or minimum_file_slab != 0):
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-file-lower-bound-requires-root-selector:%s" % target_id
return []
priority = item.get("priority", index + 1) priority = item.get("priority", index + 1)
if isinstance(priority, bool) or not isinstance(priority, int) or priority < 1 or priority > 1000: if isinstance(priority, bool) or not isinstance(priority, int) or priority < 1 or priority > 1000:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-priority-invalid:%s" % target_id _MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-priority-invalid:%s" % target_id
@@ -422,9 +529,12 @@ def configured_cgroup_reclaim_targets():
"id": target_id, "id": target_id,
"workloadSelector": workload_selector, "workloadSelector": workload_selector,
"systemdUnitSelector": systemd_unit_selector, "systemdUnitSelector": systemd_unit_selector,
"rootCgroupSelector": root_cgroup_selector,
"reclaimBytes": reclaim_bytes, "reclaimBytes": reclaim_bytes,
"swappiness": swappiness, "swappiness": swappiness,
"minimumInactiveAnonBytes": minimum_inactive_anon, "minimumInactiveAnonBytes": minimum_inactive_anon,
"minimumInactiveFileBytes": minimum_inactive_file,
"minimumReclaimableFileSlabBytes": minimum_file_slab,
"priority": priority, "priority": priority,
"targetMemAvailableBytes": target_available, "targetMemAvailableBytes": target_available,
"minimumSwapFreeBytesAfterReclaim": swap_reserve, "minimumSwapFreeBytesAfterReclaim": swap_reserve,
@@ -487,6 +597,17 @@ def cgroup_reclaim_runtime_inventory():
return _CGROUP_RECLAIM_RUNTIME_INVENTORY return _CGROUP_RECLAIM_RUNTIME_INVENTORY
def resolve_cgroup_reclaim_target(item): def resolve_cgroup_reclaim_target(item):
root_selector = item.get("rootCgroupSelector")
if isinstance(root_selector, dict):
resolved = resolve_root_cgroup_identity()
if resolved.get("ok") is not True:
return {"ok": False, "reason": resolved.get("reason"), "matches": []}
return {"ok": True, "matches": [{
"path": resolved.get("path"),
"pod": None,
"systemdUnit": None,
"rootCgroup": resolved.get("identity"),
}]}
systemd_selector = item.get("systemdUnitSelector") systemd_selector = item.get("systemdUnitSelector")
if isinstance(systemd_selector, dict): if isinstance(systemd_selector, dict):
unit = systemd_selector.get("unit") unit = systemd_selector.get("unit")
@@ -528,7 +649,7 @@ def resolve_cgroup_reclaim_target(item):
"mainPid": main_pid, "mainPid": main_pid,
"startTicks": safe_int(start_ticks), "startTicks": safe_int(start_ticks),
} }
return {"ok": True, "matches": [{"path": path, "pod": None, "systemdUnit": identity}]} return {"ok": True, "matches": [{"path": path, "pod": None, "systemdUnit": identity, "rootCgroup": None}]}
selector = item.get("workloadSelector") or {} selector = item.get("workloadSelector") or {}
labels = selector.get("matchLabels") or {} labels = selector.get("matchLabels") or {}
label_selector = ",".join("%s=%s" % pair for pair in sorted(labels.items())) label_selector = ",".join("%s=%s" % pair for pair in sorted(labels.items()))
@@ -554,13 +675,31 @@ def resolve_cgroup_reclaim_target(item):
"path": resolved_path, "path": resolved_path,
"pod": {"namespace": selector.get("namespace"), "name": name, "uid": uid, "labelSelector": label_selector}, "pod": {"namespace": selector.get("namespace"), "name": name, "uid": uid, "labelSelector": label_selector},
"systemdUnit": None, "systemdUnit": None,
"rootCgroup": None,
}) })
return {"ok": True, "matches": matches} return {"ok": True, "matches": matches}
def collect_cgroup_reclaim_candidates(): def requested_memory_reclaim_config_id():
options = globals().get("OPTIONS")
value = options.get("memoryReclaimConfigId") if isinstance(options, dict) else None
return value if isinstance(value, str) and re.match(r"^[A-Za-z0-9._-]{1,80}$", value) else None
def collect_cgroup_reclaim_candidates(selected_config_id=None):
global _MEMORY_RECLAIM_CONFIG_ERROR, _CGROUP_RECLAIM_DIAGNOSTICS global _MEMORY_RECLAIM_CONFIG_ERROR, _CGROUP_RECLAIM_DIAGNOSTICS
targets = configured_cgroup_reclaim_targets() all_targets = configured_cgroup_reclaim_targets()
diagnostics = {"configuredCount": len(targets), "eligibleCount": 0, "protected": []} targets = all_targets
if selected_config_id is not None:
targets = [item for item in all_targets if item.get("id") == selected_config_id]
if not targets and _MEMORY_RECLAIM_CONFIG_ERROR is None:
_MEMORY_RECLAIM_CONFIG_ERROR = "yaml-cgroup-reclaim-config-id-not-found:%s" % selected_config_id
diagnostics = {
"configuredCount": len(targets),
"totalConfiguredCount": len(all_targets),
"selectedConfigId": selected_config_id,
"eligibleCount": 0,
"protected": [],
}
_CGROUP_RECLAIM_DIAGNOSTICS = diagnostics _CGROUP_RECLAIM_DIAGNOSTICS = diagnostics
if _MEMORY_RECLAIM_CONFIG_ERROR or not targets: if _MEMORY_RECLAIM_CONFIG_ERROR or not targets:
return [] return []
@@ -578,6 +717,7 @@ def collect_cgroup_reclaim_candidates():
"id": item.get("id"), "id": item.get("id"),
"workloadSelector": item.get("workloadSelector"), "workloadSelector": item.get("workloadSelector"),
"systemdUnitSelector": item.get("systemdUnitSelector"), "systemdUnitSelector": item.get("systemdUnitSelector"),
"rootCgroupSelector": item.get("rootCgroupSelector"),
"reason": resolution.get("reason"), "reason": resolution.get("reason"),
"matchedPodCount": resolution.get("matchedPodCount"), "matchedPodCount": resolution.get("matchedPodCount"),
}) })
@@ -586,13 +726,14 @@ def collect_cgroup_reclaim_candidates():
path = resolved.get("path") path = resolved.get("path")
pod = resolved.get("pod") pod = resolved.get("pod")
systemd_unit = resolved.get("systemdUnit") systemd_unit = resolved.get("systemdUnit")
root_cgroup = resolved.get("rootCgroup")
reclaim_path = os.path.join(path, "memory.reclaim") reclaim_path = os.path.join(path, "memory.reclaim")
protected_reason = None protected_reason = None
if os.path.realpath(path) != path or not os.path.isdir(path) or os.path.islink(path): if os.path.realpath(path) != path or not os.path.isdir(path) or os.path.islink(path):
protected_reason = "cgroup-path-unavailable-or-redirected" protected_reason = "cgroup-path-unavailable-or-redirected"
elif not os.path.exists(reclaim_path) or not os.access(reclaim_path, os.W_OK): elif not os.path.exists(reclaim_path) or not os.access(reclaim_path, os.W_OK):
protected_reason = "memory-reclaim-unavailable" protected_reason = "memory-reclaim-unavailable"
elif available > safe_int(item.get("targetMemAvailableBytes")): elif available >= safe_int(item.get("targetMemAvailableBytes")):
protected_reason = "target-memavailable-already-met" protected_reason = "target-memavailable-already-met"
elif safe_int(item.get("swappiness")) > 0 and swap_free - safe_int(item.get("reclaimBytes")) < safe_int(item.get("minimumSwapFreeBytesAfterReclaim")): elif safe_int(item.get("swappiness")) > 0 and swap_free - safe_int(item.get("reclaimBytes")) < safe_int(item.get("minimumSwapFreeBytesAfterReclaim")):
protected_reason = "minimum-swap-free-reserve" protected_reason = "minimum-swap-free-reserve"
@@ -601,17 +742,31 @@ def collect_cgroup_reclaim_candidates():
inactive_anon = safe_int(memory_stat.get("inactive_anon")) inactive_anon = safe_int(memory_stat.get("inactive_anon"))
if protected_reason is None and safe_int(item.get("swappiness")) > 0 and inactive_anon < safe_int(item.get("minimumInactiveAnonBytes")): if protected_reason is None and safe_int(item.get("swappiness")) > 0 and inactive_anon < safe_int(item.get("minimumInactiveAnonBytes")):
protected_reason = "minimum-inactive-anon-not-met" protected_reason = "minimum-inactive-anon-not-met"
root_file_evidence = None
if protected_reason is None and item.get("rootCgroupSelector") is not None:
root_file_evidence = collect_root_file_reclaim_evidence(path)
if root_file_evidence.get("ok") is not True:
protected_reason = root_file_evidence.get("reason")
elif safe_int((root_file_evidence.get("rootMemoryStat") or {}).get("inactive_file")) < safe_int(item.get("minimumInactiveFileBytes")):
protected_reason = "minimum-root-inactive-file-not-met"
elif safe_int(root_file_evidence.get("rootReclaimableFileSlabUpperBoundBytes")) < safe_int(item.get("minimumReclaimableFileSlabBytes")):
protected_reason = "minimum-root-file-slab-upper-bound-not-met"
elif safe_int((root_file_evidence.get("host") or {}).get("reclaimableFileSlabUpperBoundBytes")) < safe_int(item.get("minimumReclaimableFileSlabBytes")):
protected_reason = "minimum-host-file-slab-upper-bound-not-met"
if protected_reason is not None: if protected_reason is not None:
diagnostics["protected"].append({ diagnostics["protected"].append({
"id": item.get("id"), "id": item.get("id"),
"path": path, "path": path,
"workloadSelector": item.get("workloadSelector"), "workloadSelector": item.get("workloadSelector"),
"systemdUnitSelector": item.get("systemdUnitSelector"), "systemdUnitSelector": item.get("systemdUnitSelector"),
"rootCgroupSelector": item.get("rootCgroupSelector"),
"resolvedPod": pod, "resolvedPod": pod,
"resolvedSystemdUnit": systemd_unit, "resolvedSystemdUnit": systemd_unit,
"resolvedRootCgroup": root_cgroup,
"reason": protected_reason, "reason": protected_reason,
"memoryCurrentBytes": current, "memoryCurrentBytes": current,
"inactiveAnonBytes": inactive_anon, "inactiveAnonBytes": inactive_anon,
"rootFileReclaimEvidence": root_file_evidence,
}) })
continue continue
reclaim_bytes = safe_int(item.get("reclaimBytes")) reclaim_bytes = safe_int(item.get("reclaimBytes"))
@@ -629,17 +784,22 @@ def collect_cgroup_reclaim_candidates():
"path": path, "path": path,
"workloadSelector": item.get("workloadSelector"), "workloadSelector": item.get("workloadSelector"),
"systemdUnitSelector": item.get("systemdUnitSelector"), "systemdUnitSelector": item.get("systemdUnitSelector"),
"rootCgroupSelector": item.get("rootCgroupSelector"),
"resolvedPod": pod, "resolvedPod": pod,
"resolvedSystemdUnit": systemd_unit, "resolvedSystemdUnit": systemd_unit,
"resolvedRootCgroup": root_cgroup,
"priority": safe_int(item.get("priority")), "priority": safe_int(item.get("priority")),
"reclaimBytes": reclaim_bytes, "reclaimBytes": reclaim_bytes,
"swappiness": safe_int(item.get("swappiness")), "swappiness": safe_int(item.get("swappiness")),
"minimumInactiveAnonBytes": safe_int(item.get("minimumInactiveAnonBytes")), "minimumInactiveAnonBytes": safe_int(item.get("minimumInactiveAnonBytes")),
"minimumInactiveFileBytes": safe_int(item.get("minimumInactiveFileBytes")),
"minimumReclaimableFileSlabBytes": safe_int(item.get("minimumReclaimableFileSlabBytes")),
"targetMemAvailableBytes": safe_int(item.get("targetMemAvailableBytes")), "targetMemAvailableBytes": safe_int(item.get("targetMemAvailableBytes")),
"minimumSwapFreeBytesAfterReclaim": safe_int(item.get("minimumSwapFreeBytesAfterReclaim")), "minimumSwapFreeBytesAfterReclaim": safe_int(item.get("minimumSwapFreeBytesAfterReclaim")),
"memoryCurrentBytes": current, "memoryCurrentBytes": current,
"memorySwapCurrentBytes": read_cgroup_integer(path, "memory.swap.current"), "memorySwapCurrentBytes": read_cgroup_integer(path, "memory.swap.current"),
"memoryStat": memory_stat, "memoryStat": memory_stat,
"rootFileReclaimEvidence": root_file_evidence,
"requestedMemoryReclaimBytes": reclaim_bytes, "requestedMemoryReclaimBytes": reclaim_bytes,
"requestedMemoryReclaim": fmt_bytes(reclaim_bytes), "requestedMemoryReclaim": fmt_bytes(reclaim_bytes),
"estimateKind": "kernel-request-upper-bound", "estimateKind": "kernel-request-upper-bound",
@@ -647,7 +807,7 @@ def collect_cgroup_reclaim_candidates():
"expectedMemoryRss": fmt_bytes(reclaim_bytes), "expectedMemoryRss": fmt_bytes(reclaim_bytes),
"estimatedDiskBytes": 0, "estimatedDiskBytes": 0,
"estimatedReclaimBytes": 0, "estimatedReclaimBytes": 0,
"action": {"op": "cgroup-v2-memory-reclaim", "allowlist": "yaml-stable-cgroup-selector"}, "action": {"op": "cgroup-v2-memory-reclaim", "allowlist": "yaml-root-cgroup-file-cache" if root_cgroup else "yaml-stable-cgroup-selector"},
}) })
diagnostics["eligibleCount"] = len(candidates) diagnostics["eligibleCount"] = len(candidates)
return candidates return candidates
@@ -663,6 +823,9 @@ def collect_memory_reclaim_candidates():
_PROCESS_PRESSURE_DIAGNOSTICS = {} _PROCESS_PRESSURE_DIAGNOSTICS = {}
_CGROUP_RECLAIM_DIAGNOSTICS = {"configuredCount": 0, "eligibleCount": 0, "protected": []} _CGROUP_RECLAIM_DIAGNOSTICS = {"configuredCount": 0, "eligibleCount": 0, "protected": []}
_OBSERVE_RUN_CACHE.clear() _OBSERVE_RUN_CACHE.clear()
selected_config_id = requested_memory_reclaim_config_id()
if selected_config_id is not None:
return collect_cgroup_reclaim_candidates(selected_config_id)
required_keys = [ required_keys = [
"observeStateRoots", "minimumReadableObserveRoots", "observeRunDepth", "observeScanLimit", "staleRunMaxAgeHours", "observeStateRoots", "minimumReadableObserveRoots", "observeRunDepth", "observeScanLimit", "staleRunMaxAgeHours",
"processScanLimit", "processStaleMinAgeSeconds", "maxMemoryReclaimCandidates", "processScanLimit", "processStaleMinAgeSeconds", "maxMemoryReclaimCandidates",
@@ -869,7 +1032,11 @@ def execute_cgroup_reclaim_candidate(candidate):
configured = next((item for item in configured_cgroup_reclaim_targets() if item.get("id") == config_id), None) configured = next((item for item in configured_cgroup_reclaim_targets() if item.get("id") == config_id), None)
if configured is None: if configured is None:
raise RuntimeError("cgroup reclaim candidate is no longer present in YAML allowlist") raise RuntimeError("cgroup reclaim candidate is no longer present in YAML allowlist")
for key in ["workloadSelector", "systemdUnitSelector", "reclaimBytes", "swappiness", "minimumInactiveAnonBytes", "targetMemAvailableBytes", "minimumSwapFreeBytesAfterReclaim", "priority"]: for key in [
"workloadSelector", "systemdUnitSelector", "rootCgroupSelector", "reclaimBytes", "swappiness",
"minimumInactiveAnonBytes", "minimumInactiveFileBytes", "minimumReclaimableFileSlabBytes",
"targetMemAvailableBytes", "minimumSwapFreeBytesAfterReclaim", "priority",
]:
if configured.get(key) != candidate.get(key): if configured.get(key) != candidate.get(key):
raise RuntimeError("cgroup reclaim candidate no longer matches YAML: %s" % key) raise RuntimeError("cgroup reclaim candidate no longer matches YAML: %s" % key)
_CGROUP_RECLAIM_RUNTIME_INVENTORY = None _CGROUP_RECLAIM_RUNTIME_INVENTORY = None
@@ -879,10 +1046,12 @@ def execute_cgroup_reclaim_candidate(candidate):
path = candidate.get("path") path = candidate.get("path")
resolved_uid = ((candidate.get("resolvedPod") or {}).get("uid")) resolved_uid = ((candidate.get("resolvedPod") or {}).get("uid"))
resolved_systemd_unit = candidate.get("resolvedSystemdUnit") resolved_systemd_unit = candidate.get("resolvedSystemdUnit")
resolved_root_cgroup = candidate.get("resolvedRootCgroup")
current_match = next((item for item in resolution.get("matches") or [] current_match = next((item for item in resolution.get("matches") or []
if item.get("path") == path if item.get("path") == path
and ((item.get("pod") or {}).get("uid")) == resolved_uid and ((item.get("pod") or {}).get("uid")) == resolved_uid
and item.get("systemdUnit") == resolved_systemd_unit and item.get("systemdUnit") == resolved_systemd_unit
and item.get("rootCgroup") == resolved_root_cgroup
), None) ), None)
if current_match is None: if current_match is None:
raise RuntimeError("cgroup reclaim candidate identity changed before execution") raise RuntimeError("cgroup reclaim candidate identity changed before execution")
@@ -895,7 +1064,14 @@ def execute_cgroup_reclaim_candidate(candidate):
before_available = safe_int((before.get("memory") or {}).get("availableBytes"), None) before_available = safe_int((before.get("memory") or {}).get("availableBytes"), None)
if before_available is None: if before_available is None:
raise RuntimeError("MemAvailable is unavailable before cgroup reclaim") raise RuntimeError("MemAvailable is unavailable before cgroup reclaim")
if before_available > safe_int(configured.get("targetMemAvailableBytes")): root_evidence_before = None
if configured.get("rootCgroupSelector") is not None:
if safe_int(configured.get("swappiness")) != 0 or path != (resolved_root_cgroup or {}).get("mountPoint"):
raise RuntimeError("root cgroup file reclaim identity or swappiness is invalid")
root_evidence_before = collect_root_file_reclaim_evidence(path)
if root_evidence_before.get("ok") is not True:
raise RuntimeError("root cgroup file reclaim evidence is unavailable: %s" % root_evidence_before.get("reason"))
if before_available >= safe_int(configured.get("targetMemAvailableBytes")):
return { return {
"status": "succeeded", "status": "succeeded",
"outcome": "target-memavailable-already-met", "outcome": "target-memavailable-already-met",
@@ -905,6 +1081,10 @@ def execute_cgroup_reclaim_candidate(candidate):
"cgroupMemoryBeforeBytes": read_cgroup_integer(path, "memory.current"), "cgroupMemoryBeforeBytes": read_cgroup_integer(path, "memory.current"),
"cgroupMemoryAfterBytes": read_cgroup_integer(path, "memory.current"), "cgroupMemoryAfterBytes": read_cgroup_integer(path, "memory.current"),
"kernelWriteAttempted": False, "kernelWriteAttempted": False,
"rootFileReclaimEvidenceBefore": root_evidence_before,
"rootFileReclaimEvidenceAfter": root_evidence_before,
"targetGapBeforeBytes": 0,
"targetGapAfterBytes": 0,
} }
swap_free = safe_int((before.get("swap") or {}).get("freeBytes"), None) swap_free = safe_int((before.get("swap") or {}).get("freeBytes"), None)
if safe_int(configured.get("swappiness")) > 0: if safe_int(configured.get("swappiness")) > 0:
@@ -916,6 +1096,13 @@ def execute_cgroup_reclaim_candidate(candidate):
inactive_anon = safe_int(read_cgroup_memory_stat(path).get("inactive_anon")) inactive_anon = safe_int(read_cgroup_memory_stat(path).get("inactive_anon"))
if inactive_anon < safe_int(configured.get("minimumInactiveAnonBytes")): if inactive_anon < safe_int(configured.get("minimumInactiveAnonBytes")):
raise RuntimeError("minimum inactive anonymous memory is no longer available") raise RuntimeError("minimum inactive anonymous memory is no longer available")
if configured.get("rootCgroupSelector") is not None:
if safe_int((root_evidence_before.get("rootMemoryStat") or {}).get("inactive_file")) < safe_int(configured.get("minimumInactiveFileBytes")):
raise RuntimeError("minimum root inactive file memory is no longer available")
if safe_int(root_evidence_before.get("rootReclaimableFileSlabUpperBoundBytes")) < safe_int(configured.get("minimumReclaimableFileSlabBytes")):
raise RuntimeError("minimum root file/slab reclaim upper bound is no longer available")
if safe_int((root_evidence_before.get("host") or {}).get("reclaimableFileSlabUpperBoundBytes")) < safe_int(configured.get("minimumReclaimableFileSlabBytes")):
raise RuntimeError("minimum host file/slab reclaim upper bound is no longer available")
cgroup_before = read_cgroup_integer(path, "memory.current") cgroup_before = read_cgroup_integer(path, "memory.current")
write_error = None write_error = None
try: try:
@@ -927,6 +1114,7 @@ def execute_cgroup_reclaim_candidate(candidate):
after = collect_memory_snapshot() after = collect_memory_snapshot()
after_available = safe_int((after.get("memory") or {}).get("availableBytes"), None) after_available = safe_int((after.get("memory") or {}).get("availableBytes"), None)
cgroup_after = read_cgroup_integer(path, "memory.current") cgroup_after = read_cgroup_integer(path, "memory.current")
root_evidence_after = collect_root_file_reclaim_evidence(path) if configured.get("rootCgroupSelector") is not None else None
cgroup_delta = max(0, cgroup_before - cgroup_after) if cgroup_before is not None and cgroup_after is not None else 0 cgroup_delta = max(0, cgroup_before - cgroup_after) if cgroup_before is not None and cgroup_after is not None else 0
host_delta = max(0, after_available - before_available) if after_available is not None else 0 host_delta = max(0, after_available - before_available) if after_available is not None else 0
succeeded = write_error is None or cgroup_delta > 0 or host_delta > 0 succeeded = write_error is None or cgroup_delta > 0 or host_delta > 0
@@ -949,6 +1137,10 @@ def execute_cgroup_reclaim_candidate(candidate):
"kernelWriteError": write_error, "kernelWriteError": write_error,
"memoryBefore": before, "memoryBefore": before,
"memoryAfter": after, "memoryAfter": after,
"rootFileReclaimEvidenceBefore": root_evidence_before,
"rootFileReclaimEvidenceAfter": root_evidence_after,
"targetGapBeforeBytes": max(0, safe_int(configured.get("targetMemAvailableBytes")) - before_available),
"targetGapAfterBytes": max(0, safe_int(configured.get("targetMemAvailableBytes")) - after_available) if after_available is not None else None,
} }
def execute_memory_process_candidate(candidate): def execute_memory_process_candidate(candidate):
+96 -7
View File
@@ -38,6 +38,7 @@ interface RemoteGcOptions {
historyLimit: number; historyLimit: number;
saveSnapshot: boolean; saveSnapshot: boolean;
memoryPressureOnly: boolean; memoryPressureOnly: boolean;
memoryReclaimConfigId?: string;
buildCacheOnly: boolean; buildCacheOnly: boolean;
hostDockerOnly: boolean; hostDockerOnly: boolean;
} }
@@ -71,6 +72,7 @@ const DEFAULT_REMOTE_OPTIONS: RemoteGcOptions = {
historyLimit: 12, historyLimit: 12,
saveSnapshot: true, saveSnapshot: true,
memoryPressureOnly: false, memoryPressureOnly: false,
memoryReclaimConfigId: undefined,
buildCacheOnly: false, buildCacheOnly: false,
hostDockerOnly: false, hostDockerOnly: false,
}; };
@@ -182,6 +184,14 @@ export async function runRemoteGcCommand(config: UniDeskConfig, providerId: stri
}; };
} }
const options = parseRemoteGcOptions(args); const options = parseRemoteGcOptions(args);
if (options.memoryReclaimConfigId !== undefined && !options.memoryPressureOnly) {
return {
ok: false,
error: "gc-remote-memory-reclaim-config-id-requires-memory-pressure-only",
configId: options.memoryReclaimConfigId,
requiredFlag: "--memory-pressure-only",
};
}
if (options.memoryPressureOnly && subaction !== "plan" && subaction !== "run") { if (options.memoryPressureOnly && subaction !== "plan" && subaction !== "run") {
return { return {
ok: false, ok: false,
@@ -209,8 +219,8 @@ export async function runRemoteGcCommand(config: UniDeskConfig, providerId: stri
dryRun: true, dryRun: true,
mutation: false, mutation: false,
requiredFlag: "--confirm", requiredFlag: "--confirm",
planCommand: `bun scripts/cli.ts gc remote ${providerId} plan${options.memoryPressureOnly ? " --memory-pressure-only" : registryFlag}`, planCommand: `bun scripts/cli.ts gc remote ${providerId} plan${options.memoryPressureOnly ? memoryPressureOptionSuffix(options) : registryFlag}`,
runCommand: `bun scripts/cli.ts gc remote ${providerId} run --confirm${options.memoryPressureOnly ? " --memory-pressure-only" : registryFlag}`, runCommand: `bun scripts/cli.ts gc remote ${providerId} run --confirm${options.memoryPressureOnly ? memoryPressureOptionSuffix(options) : registryFlag}`,
}; };
} }
return await runRemoteGc(config, providerId, "run", options); return await runRemoteGc(config, providerId, "run", options);
@@ -317,6 +327,10 @@ function parseRemoteGcOptions(args: string[]): RemoteGcOptions {
options.saveSnapshot = false; options.saveSnapshot = false;
} else if (arg === "--memory-pressure-only") { } else if (arg === "--memory-pressure-only") {
options.memoryPressureOnly = true; options.memoryPressureOnly = true;
} else if (arg === "--memory-reclaim-config-id") {
const value = args[++index];
if (!value || !/^[A-Za-z0-9._-]{1,80}$/u.test(value)) throw new Error("--memory-reclaim-config-id must be a YAML cgroupReclaim target id");
options.memoryReclaimConfigId = value;
} else if (arg === "--build-cache-only") { } else if (arg === "--build-cache-only") {
options.buildCacheOnly = true; options.buildCacheOnly = true;
} else if (arg === "--host-docker-only") { } else if (arg === "--host-docker-only") {
@@ -328,6 +342,13 @@ function parseRemoteGcOptions(args: string[]): RemoteGcOptions {
return options; return options;
} }
function memoryPressureOptionSuffix(options: RemoteGcOptions): string {
const selection = options.memoryReclaimConfigId === undefined
? ""
: ` --memory-reclaim-config-id ${options.memoryReclaimConfigId}`;
return ` --memory-pressure-only${selection}`;
}
function parseNonNegativeNumber(name: string, raw: string | undefined): number { function parseNonNegativeNumber(name: string, raw: string | undefined): number {
const value = Number(raw); const value = Number(raw);
if (!Number.isFinite(value) || value < 0) throw new Error(`${name} must be a non-negative number`); if (!Number.isFinite(value) || value < 0) throw new Error(`${name} must be a non-negative number`);
@@ -453,6 +474,45 @@ function compactResolvedSystemdUnit(value: unknown): Record<string, unknown> | u
}; };
} }
function compactRootCgroupSelector(value: unknown): string | undefined {
const selector = recordOrEmpty(value);
return selector.scope === "root" && Object.keys(selector).length === 1 ? "root" : undefined;
}
function compactResolvedRootCgroup(value: unknown): Record<string, unknown> | undefined {
const identity = recordOrEmpty(value);
if (identity.filesystemType !== "cgroup2"
|| identity.mountRoot !== "/"
|| typeof identity.mountPoint !== "string"
|| typeof identity.majorMinor !== "string"
|| positiveInteger(identity.mountId) === null
|| positiveInteger(identity.parentMountId) === null
|| positiveInteger(identity.inode) === null) return undefined;
return {
filesystemType: identity.filesystemType,
mountRoot: identity.mountRoot,
mountPoint: identity.mountPoint,
majorMinor: identity.majorMinor,
mountId: identity.mountId,
parentMountId: identity.parentMountId,
inode: identity.inode,
};
}
function compactRootFileReclaimEvidence(value: unknown): Record<string, unknown> | undefined {
const evidence = recordOrEmpty(value);
const host = recordOrEmpty(evidence.host);
const rootMemoryStat = recordOrEmpty(evidence.rootMemoryStat);
if (evidence.ok !== true || host.source !== "/proc/meminfo") return undefined;
return {
source: host.source,
estimateKind: evidence.estimateKind,
hostFileSlabUpperBoundBytes: host.reclaimableFileSlabUpperBoundBytes,
rootFileSlabUpperBoundBytes: evidence.rootReclaimableFileSlabUpperBoundBytes,
rootInactiveFileBytes: rootMemoryStat.inactive_file,
};
}
function compactMemoryPressureCandidate(value: unknown): Record<string, unknown> { function compactMemoryPressureCandidate(value: unknown): Record<string, unknown> {
const candidate = recordOrEmpty(value); const candidate = recordOrEmpty(value);
const action = recordOrEmpty(candidate.action); const action = recordOrEmpty(candidate.action);
@@ -476,14 +536,22 @@ function compactMemoryPressureCandidate(value: unknown): Record<string, unknown>
resolvedPod: compactResolvedPod(candidate.resolvedPod), resolvedPod: compactResolvedPod(candidate.resolvedPod),
systemdUnitSelector: compactSystemdUnitSelector(candidate.systemdUnitSelector), systemdUnitSelector: compactSystemdUnitSelector(candidate.systemdUnitSelector),
resolvedSystemdUnit: compactResolvedSystemdUnit(candidate.resolvedSystemdUnit), resolvedSystemdUnit: compactResolvedSystemdUnit(candidate.resolvedSystemdUnit),
rootCgroupSelector: compactRootCgroupSelector(candidate.rootCgroupSelector),
resolvedRootCgroup: compactResolvedRootCgroup(candidate.resolvedRootCgroup),
requestedMemoryReclaimBytes: candidate.requestedMemoryReclaimBytes, requestedMemoryReclaimBytes: candidate.requestedMemoryReclaimBytes,
requestedMemoryReclaim: candidate.requestedMemoryReclaim, requestedMemoryReclaim: candidate.requestedMemoryReclaim,
estimateKind: candidate.estimateKind, estimateKind: candidate.estimateKind,
swappiness: candidate.swappiness, swappiness: candidate.swappiness,
minimumInactiveAnonBytes: candidate.minimumInactiveAnonBytes, minimumInactiveAnonBytes: candidate.minimumInactiveAnonBytes,
minimumInactiveFileBytes: candidate.minimumInactiveFileBytes,
minimumReclaimableFileSlabBytes: candidate.minimumReclaimableFileSlabBytes,
memoryCurrentBytes: candidate.memoryCurrentBytes, memoryCurrentBytes: candidate.memoryCurrentBytes,
memorySwapCurrentBytes: candidate.memorySwapCurrentBytes, memorySwapCurrentBytes: candidate.memorySwapCurrentBytes,
inactiveAnonBytes: memoryStat.inactive_anon, inactiveAnonBytes: memoryStat.inactive_anon,
inactiveFileBytes: memoryStat.inactive_file,
fileBytes: memoryStat.file,
slabReclaimableBytes: memoryStat.slab_reclaimable,
rootFileReclaimEvidence: compactRootFileReclaimEvidence(candidate.rootFileReclaimEvidence),
}; };
} }
const identities = Array.isArray(candidate.processIdentities) ? candidate.processIdentities.map(recordOrEmpty) : []; const identities = Array.isArray(candidate.processIdentities) ? candidate.processIdentities.map(recordOrEmpty) : [];
@@ -524,6 +592,11 @@ function memoryPressureCandidateAllowed(value: unknown): boolean {
const resolvedPod = recordOrEmpty(candidate.resolvedPod); const resolvedPod = recordOrEmpty(candidate.resolvedPod);
const systemdUnitSelector = recordOrEmpty(candidate.systemdUnitSelector); const systemdUnitSelector = recordOrEmpty(candidate.systemdUnitSelector);
const resolvedSystemdUnit = recordOrEmpty(candidate.resolvedSystemdUnit); const resolvedSystemdUnit = recordOrEmpty(candidate.resolvedSystemdUnit);
const rootCgroupSelector = compactRootCgroupSelector(candidate.rootCgroupSelector);
const resolvedRootCgroup = compactResolvedRootCgroup(candidate.resolvedRootCgroup);
const reclaimBytes = positiveInteger(candidate.reclaimBytes);
const minimumInactiveFileBytes = positiveInteger(candidate.minimumInactiveFileBytes);
const minimumReclaimableFileSlabBytes = positiveInteger(candidate.minimumReclaimableFileSlabBytes);
const workloadIdentityClosed = typeof workloadSelector.namespace === "string" && typeof resolvedPod.uid === "string"; const workloadIdentityClosed = typeof workloadSelector.namespace === "string" && typeof resolvedPod.uid === "string";
const systemdIdentityClosed = typeof systemdUnitSelector.unit === "string" const systemdIdentityClosed = typeof systemdUnitSelector.unit === "string"
&& resolvedSystemdUnit.unit === systemdUnitSelector.unit && resolvedSystemdUnit.unit === systemdUnitSelector.unit
@@ -535,14 +608,23 @@ function memoryPressureCandidateAllowed(value: unknown): boolean {
&& candidate.path === `/sys/fs/cgroup${resolvedSystemdUnit.controlGroup}` && candidate.path === `/sys/fs/cgroup${resolvedSystemdUnit.controlGroup}`
&& positiveInteger(resolvedSystemdUnit.mainPid) !== null && positiveInteger(resolvedSystemdUnit.mainPid) !== null
&& positiveInteger(resolvedSystemdUnit.startTicks) !== null; && positiveInteger(resolvedSystemdUnit.startTicks) !== null;
const rootIdentityClosed = rootCgroupSelector === "root"
&& resolvedRootCgroup !== undefined
&& candidate.path === resolvedRootCgroup.mountPoint
&& swappiness === 0
&& reclaimBytes !== null
&& minimumInactiveFileBytes !== null
&& minimumReclaimableFileSlabBytes !== null
&& minimumReclaimableFileSlabBytes >= reclaimBytes;
return typeof candidate.configId === "string" && candidate.configId.length > 0 return typeof candidate.configId === "string" && candidate.configId.length > 0
&& typeof candidate.path === "string" && candidate.path.startsWith("/") && typeof candidate.path === "string" && candidate.path.startsWith("/")
&& positiveInteger(candidate.reclaimBytes) !== null && reclaimBytes !== null
&& swappiness !== null && swappiness >= 0 && swappiness <= 200 && swappiness !== null && swappiness >= 0 && swappiness <= 200
&& (swappiness === 0 || positiveInteger(candidate.minimumInactiveAnonBytes) !== null) && (swappiness === 0 || positiveInteger(candidate.minimumInactiveAnonBytes) !== null)
&& action.op === "cgroup-v2-memory-reclaim" && action.op === "cgroup-v2-memory-reclaim"
&& action.allowlist === "yaml-stable-cgroup-selector" && (action.allowlist === "yaml-stable-cgroup-selector" || action.allowlist === "yaml-root-cgroup-file-cache")
&& workloadIdentityClosed !== systemdIdentityClosed; && [workloadIdentityClosed, systemdIdentityClosed, rootIdentityClosed].filter(Boolean).length === 1
&& (rootIdentityClosed === (action.allowlist === "yaml-root-cgroup-file-cache"));
} }
if (candidate.kind !== "browser-orphan-process-tree" && candidate.kind !== "stale-web-observer-process-tree") return false; if (candidate.kind !== "browser-orphan-process-tree" && candidate.kind !== "stale-web-observer-process-tree") return false;
const compact = compactMemoryPressureCandidate(candidate); const compact = compactMemoryPressureCandidate(candidate);
@@ -586,6 +668,8 @@ function compactMemoryPressureCandidateScan(value: unknown): Record<string, unkn
}, },
cgroupReclaim: { cgroupReclaim: {
configuredCount: cgroupReclaim.configuredCount, configuredCount: cgroupReclaim.configuredCount,
totalConfiguredCount: cgroupReclaim.totalConfiguredCount,
selectedConfigId: cgroupReclaim.selectedConfigId,
eligibleCount: cgroupReclaim.eligibleCount, eligibleCount: cgroupReclaim.eligibleCount,
protectedCount: cgroupProtected.length, protectedCount: cgroupProtected.length,
protected: cgroupProtected.map((item) => { protected: cgroupProtected.map((item) => {
@@ -596,6 +680,9 @@ function compactMemoryPressureCandidateScan(value: unknown): Record<string, unkn
matchedPodCount: protectedItem.matchedPodCount, matchedPodCount: protectedItem.matchedPodCount,
memoryCurrentBytes: protectedItem.memoryCurrentBytes, memoryCurrentBytes: protectedItem.memoryCurrentBytes,
inactiveAnonBytes: protectedItem.inactiveAnonBytes, inactiveAnonBytes: protectedItem.inactiveAnonBytes,
rootCgroupSelector: compactRootCgroupSelector(protectedItem.rootCgroupSelector),
resolvedRootCgroup: compactResolvedRootCgroup(protectedItem.resolvedRootCgroup),
rootFileReclaimEvidence: compactRootFileReclaimEvidence(protectedItem.rootFileReclaimEvidence),
workloadSelector: compactWorkloadSelector(protectedItem.workloadSelector), workloadSelector: compactWorkloadSelector(protectedItem.workloadSelector),
resolvedPod: compactResolvedPod(protectedItem.resolvedPod), resolvedPod: compactResolvedPod(protectedItem.resolvedPod),
systemdUnitSelector: compactSystemdUnitSelector(protectedItem.systemdUnitSelector), systemdUnitSelector: compactSystemdUnitSelector(protectedItem.systemdUnitSelector),
@@ -620,11 +707,12 @@ export function projectRemoteGcResult(
const memoryPressure = recordOrEmpty(remoteTarget.memoryPressure); const memoryPressure = recordOrEmpty(remoteTarget.memoryPressure);
const previewLimit = positiveInteger(memoryPressure.planPreviewLimit); const previewLimit = positiveInteger(memoryPressure.planPreviewLimit);
const totalCandidateCount = positiveInteger(summary.candidateCount) ?? 0; const totalCandidateCount = positiveInteger(summary.candidateCount) ?? 0;
const allCandidatesEvaluated = totalCandidateCount > 0 && totalCandidateCount === candidates.length; const targetMet = summary.targetMet === true;
const allCandidatesEvaluated = totalCandidateCount === candidates.length;
const reasons: string[] = []; const reasons: string[] = [];
if (payload.ok !== true) reasons.push("plan-not-ok"); if (payload.ok !== true) reasons.push("plan-not-ok");
if (scan.ok !== true || scan.blocked === true) reasons.push("candidate-scan-blocked"); if (scan.ok !== true || scan.blocked === true) reasons.push("candidate-scan-blocked");
if (totalCandidateCount === 0) reasons.push("no-candidates"); if (totalCandidateCount === 0 && !targetMet) reasons.push("no-candidates");
if (!allCandidatesEvaluated) reasons.push("candidate-page-incomplete"); if (!allCandidatesEvaluated) reasons.push("candidate-page-incomplete");
if (!candidates.every(memoryPressureCandidateAllowed)) reasons.push("unsupported-or-incomplete-candidate"); if (!candidates.every(memoryPressureCandidateAllowed)) reasons.push("unsupported-or-incomplete-candidate");
if (previewLimit === null) reasons.push("yaml-plan-preview-limit-invalid"); if (previewLimit === null) reasons.push("yaml-plan-preview-limit-invalid");
@@ -641,6 +729,7 @@ export function projectRemoteGcResult(
observedAt: payload.observedAt, observedAt: payload.observedAt,
memoryBefore: payload.memoryBefore, memoryBefore: payload.memoryBefore,
summary: payload.summary, summary: payload.summary,
selection: payload.selection,
candidateScan: compactScan, candidateScan: compactScan,
runEligibility: { runEligibility: {
allowed: reasons.length === 0, allowed: reasons.length === 0,
+3
View File
@@ -527,6 +527,8 @@ function gcHelp(): unknown {
"bun scripts/cli.ts gc remote NC01 retention run --confirm --limit 500", "bun scripts/cli.ts gc remote NC01 retention run --confirm --limit 500",
"bun scripts/cli.ts gc remote NC01 plan --memory-pressure-only", "bun scripts/cli.ts gc remote NC01 plan --memory-pressure-only",
"bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only", "bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only",
"bun scripts/cli.ts gc remote NC01 plan --memory-pressure-only --memory-reclaim-config-id <yaml-target-id>",
"bun scripts/cli.ts gc remote NC01 run --confirm --memory-pressure-only --memory-reclaim-config-id <yaml-target-id>",
"bun scripts/cli.ts gc remote NC01 status --job-id <job-id>", "bun scripts/cli.ts gc remote NC01 status --job-id <job-id>",
"bun scripts/cli.ts gc remote G14 plan --target-use-percent 50 --include-hwlab-registry", "bun scripts/cli.ts gc remote G14 plan --target-use-percent 50 --include-hwlab-registry",
"bun scripts/cli.ts gc remote G14 run --confirm", "bun scripts/cli.ts gc remote G14 run --confirm",
@@ -558,6 +560,7 @@ function gcHelp(): unknown {
"--registry-min-age-hours N": "remote registry only: keep all tags newer than N hours; default 48, minimum 0", "--registry-min-age-hours N": "remote registry only: keep all tags newer than N hours; default 48, minimum 0",
"--target-use-percent N": "evaluate whether planned candidates can reduce root filesystem use to N%; reports required reclaim, projected use, shortfall and safe-stop decision", "--target-use-percent N": "evaluate whether planned candidates can reduce root filesystem use to N%; reports required reclaim, projected use, shortfall and safe-stop decision",
"--job-id ID": "remote status only: inspect a long-running remote gc job", "--job-id ID": "remote status only: inspect a long-running remote gc job",
"--memory-reclaim-config-id ID": "remote memory-pressure plan/run only: select exactly one owning YAML cgroupReclaim target id and exclude every other cgroup or process candidate",
"--limit N": "number of candidates returned and executed by run when --full is not set; default 50", "--limit N": "number of candidates returned and executed by run when --full is not set; default 50",
"--result-limit N": "number of per-candidate run results returned when --full is not set; default 50", "--result-limit N": "number of per-candidate run results returned when --full is not set; default 50",
"--full|--raw": "return and run against all candidates rather than the default bounded page", "--full|--raw": "return and run against all candidates rather than the default bounded page",