feat: 收敛 Sub2API 账号调优诊断

This commit is contained in:
Codex
2026-07-16 06:39:11 +02:00
parent 1744f30765
commit d42cf7bae9
7 changed files with 197 additions and 12 deletions
+9
View File
@@ -24,10 +24,19 @@ bun scripts/cli.ts platform-infra sub2api ops diagnosis --target PK01
bun scripts/cli.ts platform-infra sub2api ops channels --target PK01 --window 7d
bun scripts/cli.ts platform-infra sub2api plan --target PK01
bun scripts/cli.ts platform-infra sub2api image-prepull --target PK01 --confirm
bun scripts/cli.ts platform-infra sub2api codex-pool runtime get --target PK01 --account <id-or-exact-name>
```
先看报表和状态,再做计划或变更。详细规则按职责拆在 `references/` 下;不要新增 `full.md``all.md``guide.md` 这类变相超级文件。
- `priority` 数值越小优先级越高,`0` 合法且最高:
- 这是 Sub2API 调度的稳定合同,后续调优不得为确认方向重复检索源码;
- 只有真实调度结果与该合同冲突时,才下钻官方 scheduler 实现。
- 查询账号周限保护统一使用 `runtime get --account <id-or-exact-name>`
- 单次展示原生 5h/7d 用量、重置时间、有效自动保护阈值、阈值来源和触发状态;
- 账号级 `auto_pause_*_threshold` 优先于全局 Ops 默认值;
- 原生 quota probe 失败时明确标记不可用,并保留账号缓存字段,禁止把用户计费额度当成上游账号周限。
## 边界
- PK01 默认只允许只读诊断:
@@ -198,10 +198,10 @@ bun scripts/cli.ts platform-infra sub2api codex-pool cleanup-probes --target D60
- 单账号成本标注与优先级调优使用以下最短路径:
- 已知账号 selector 时,首个诊断入口固定为:
`runtime errors --target <target> --since <window> --group <group> --account <account>`
- 同一次账号级查询直接读取评分、可归因上游失败率、TTFT、用量、Token、标准 API 计费基数当前名称成本标注;
- 只有需要确认 priority、capacity、schedulable、proxy临时不可调度规则时,才补一次 `runtime get --account <account>`
- 同一次账号级查询直接读取评分、可归因上游失败率、TTFT、用量、Token、标准 API 计费基数当前名称成本标注和 priority
- 只有需要确认 capacity、schedulable、proxy临时不可调度规则或上游 5h/7d 周窗口保护时,才补一次 `runtime get --account <account>`
- 只有需要跨账号比较时才使用全组 `runtime errors --group <group>`;单账号任务不得先跑全组大报表,再重复账号级查询;
- priority 数值越小越优先。成本只参与人工经济性判断,必须同时考虑可靠性、TTFT、容量、可调度状态和样本置信度,不得按成本自动排序;
- priority 数值越小越优先`0` 合法且最高;这是稳定合同,除非真实调度结果冲突,不得重复检索源码确认方向。成本只参与人工经济性判断,必须同时考虑可靠性、TTFT、容量、可调度状态和样本置信度,不得按成本自动排序;
- 名称成本标注和 priority 是两个独立 runtime 字段:分别先 dry-run,再顺序 confirm;禁止并发更新同一账号,避免整对象更新互相覆盖;
- 两项 confirm 已各自回读一致后,最后只执行一次 `runtime get` 验收完整状态;不重复 confirm,不用全组查询代替账号回读。
- 模型映射先用 `runtime models --account ...` 获取原生配置与实时上游能力:
@@ -0,0 +1,20 @@
# R2.19 任务报告
## 结果
- 在 Sub2API skill 高频入口固化稳定合同:`priority` 数值越小越高,`0` 合法且最高;只有真实调度结果冲突时才下钻官方 scheduler。
- `runtime get --account <id-or-exact-name>` 现在单次展示原生 5h/7d 上游窗口、重置时间、数据来源、账号级或全局自动保护阈值、触发状态及原生 quota probe 状态。
- 保护阈值按官方调度语义解析:账号 `auto_pause_*_threshold` 优先,全局 Ops 默认次之;旧版本接口不可用时显示 `global-settings-unavailable`,不会误报为禁用。
- `runtime errors --account` 的账号质量表直接包含当前 priority,并标注 `lower-is-higher / highest=0`;账号下钻只显示该账号客户错误,不再混入同组其他账号。
- runtime 文本失败现在直接显示错误原因,不再要求补跑 JSON 才能看到 `account-not-found-or-ambiguous`
## 只读验收
- `runtime get --target PK01 --account 15` 一次返回账号 `lyon9801 0.0`、priority 0、原生窗口和自动保护状态。
- 实际运行面显示 7 天原生用量 0%、账号级保护阈值 97%、未触发;这与用户提出的 95% 预期不一致,本任务未修改 runtime。
- `runtime errors --target PK01 --since 30m --group 2 --account 15` 一次返回 priority、评分、用量、Token、成本和置信度,且不再展示其他账号客户错误。
- 修改文件通过 Bun 语法检查、`git diff --check` 和 Sub2API skill `quick_validate.py`
## 边界
未修改 Sub2API 源码、版本、账号 runtime、外部哨兵或生产配置;所有生产调用均为官方原生后端支持的受控只读查询。
@@ -234,6 +234,9 @@
### R2.18 [completed]
基于 R2.16 优先级和 R2.17 通用 failover 模板下发后约 3 小时的新窗口,再按 PK01 Sub2API 所有分组的用户可见错误与人民币边际成本调优全局 priority:使用原生 runtime errors 聚合完整账号名称、评分、请求、Token、API USD、成本、可归因失败、TTFT、规则命中、切号和置信度,区分盈利池与自用池;只对新窗口证据充分的账号做精准 dry-run、确认写入和回读,不重复计入旧错误,不修改版本、源码、容量、代理、凭据、分组、可调度、临时规则或外部哨兵,完成任务后将详细报告写入[任务报告](./details/sub2api-upstream-reliability/R2.18_Task_Report.md)。
### R2.19 [completed]
关联 [UniDesk #2270](https://github.com/pikasTech/unidesk/issues/2270),固化 priority 数值越小越高且 0 最高的稳定合同,并增强 Sub2API 受控 CLI 单次展示账号原生周窗口状态、YAML 周限保护阈值/来源及紧凑评分摘要,减少重复源码检索、逐账号查询和超量输出;只读验证,不修改 Sub2API 源码、版本、runtime、外部哨兵或生产运行面,完成任务后将详细报告写入[任务报告](./details/sub2api-upstream-reliability/R2.19_Task_Report.md)。
## R3 [completed]
增加 Sub2API 客户余额精准批量充值 CLI:支持从原生最近24小时用量提取活跃用户并生成每人增加20美元的补偿计划,也支持显式用户ID或账号 selector;默认 dry-run 输出完整账号、用户ID、当前余额、增额和目标余额,写入必须显式 --confirm 并自动回读对账。本轮只实现并验证 dry-run,禁止实际充值,必须等待用户再次明确授权,完成任务后将详细报告写入[任务报告](./details/sub2api-upstream-reliability/R3_Task_Report.md)。
@@ -211,7 +211,110 @@ def matching_template(credentials):
return template.get("id")
return None
def account_summary(detail, include_rules=False):
def quota_percent(value):
if not isinstance(value, (int, float)):
return None
return round(value * 100, 4) if 0 <= value <= 1 else round(value, 4)
def cached_codex_window(extra, label):
return {
"window": label,
"usedPercent": extra.get(f"codex_{label}_used_percent"),
"resetAfterSeconds": extra.get(f"codex_{label}_reset_after_seconds"),
"windowMinutes": extra.get(f"codex_{label}_window_minutes"),
"resetAt": extra.get(f"codex_{label}_reset_at"),
"source": "sub2api-account-extra",
}
def unix_time_text(value):
if not isinstance(value, (int, float)) or value <= 0:
return value
try:
return datetime.fromtimestamp(value, tz=timezone.utc).isoformat()
except (ValueError, OSError, OverflowError):
return value
def openai_quota_auto_pause_settings(token):
try:
advanced = data_of(curl_api("GET", "/api/v1/admin/ops/advanced-settings", bearer=token), "get advanced settings")
settings = advanced.get("openai_account_quota_auto_pause") if isinstance(advanced, dict) else None
return {"status": "available", "settings": settings if isinstance(settings, dict) else {}}
except Exception as exc:
return {"status": "unavailable", "settings": {}, "reason": compact_error(str(exc))}
def openai_quota_summary(token, detail):
extra = detail.get("extra") if isinstance(detail.get("extra"), dict) else {}
global_result = openai_quota_auto_pause_settings(token)
global_settings = global_result.get("settings") or {}
cached_windows = [cached_codex_window(extra, label) for label in ("5h", "7d")]
probe = {"status": "unsupported", "source": "sub2api-native-openai-quota", "windows": []}
if detail.get("platform") == "openai" or detail.get("type") in ("oauth", "setup-token"):
try:
usage = data_of(curl_api("GET", f"/api/v1/admin/openai/accounts/{detail.get('id')}/quota", bearer=token), "query OpenAI quota")
rate_limit = usage.get("rate_limit") if isinstance(usage, dict) else None
rows = []
if isinstance(rate_limit, dict):
for name in ("primary_window", "secondary_window"):
item = rate_limit.get(name)
if not isinstance(item, dict):
continue
seconds = item.get("limit_window_seconds")
label = "5h" if seconds == 18000 else "7d" if seconds == 604800 else name.replace("_window", "")
rows.append({
"window": label,
"usedPercent": item.get("used_percent"),
"resetAfterSeconds": item.get("reset_after_seconds"),
"resetAt": unix_time_text(item.get("reset_at")),
"source": "sub2api-native-openai-quota",
})
probe = {
"status": "available",
"source": "sub2api-native-openai-quota",
"allowed": rate_limit.get("allowed") if isinstance(rate_limit, dict) else None,
"limitReached": rate_limit.get("limit_reached") if isinstance(rate_limit, dict) else None,
"fetchedAt": unix_time_text(usage.get("fetched_at")) if isinstance(usage, dict) else None,
"windows": rows,
}
except Exception as exc:
probe = {"status": "unavailable", "source": "sub2api-native-openai-quota", "reason": compact_error(str(exc)), "windows": []}
probed_by_window = {item.get("window"): item for item in probe.get("windows", []) if isinstance(item, dict)}
windows = [probed_by_window.get(item.get("window"), item) for item in cached_windows]
protections = []
for window in windows:
label = window.get("window")
account_threshold = extra.get(f"auto_pause_{label}_threshold")
global_threshold = global_settings.get(f"default_threshold_{label}")
if isinstance(account_threshold, (int, float)) and account_threshold > 0:
threshold = account_threshold
source = "account-extra"
elif isinstance(global_threshold, (int, float)) and global_threshold > 0:
threshold = global_threshold
source = "global-ops-default"
elif global_result.get("status") == "available":
threshold = None
source = "disabled"
else:
threshold = None
source = "global-settings-unavailable"
used = window.get("usedPercent")
threshold_percent = quota_percent(threshold)
protections.append({
"window": label,
"thresholdPercent": threshold_percent,
"source": source,
"triggered": isinstance(used, (int, float)) and isinstance(threshold_percent, (int, float)) and used >= threshold_percent,
})
return {
"windows": windows,
"cachedWindows": cached_windows,
"protections": protections,
"nativeProbe": probe,
"globalSettings": {"status": global_result.get("status"), "reason": global_result.get("reason")},
"updatedAt": extra.get("codex_usage_updated_at"),
"attribution": "Window values prefer the native read-only quota probe and fall back to account cache. Protection uses account threshold first, then the global Ops default.",
}
def account_summary(detail, include_rules=False, token=None):
credentials = detail.get("credentials") if isinstance(detail.get("credentials"), dict) else {}
policy = normalize_policy(credentials)
name = detail.get("name")
@@ -228,6 +331,7 @@ def account_summary(detail, include_rules=False):
"currentConcurrency": detail.get("current_concurrency"),
"loadFactor": detail.get("load_factor"),
"priority": detail.get("priority"),
"prioritySemantics": {"order": "lower-is-higher", "highest": 0},
"tempUnschedulable": {key: policy[key] for key in ("enabled", "ruleCount", "statusCodes")},
"matchingTemplate": matching_template(credentials),
"poolModeEnabled": credentials.get("pool_mode") is True,
@@ -235,6 +339,7 @@ def account_summary(detail, include_rules=False):
}
if include_rules:
result["tempUnschedulable"]["rules"] = policy["rules"]
result["upstreamLimits"] = openai_quota_summary(token, detail) if token else None
return result
def list_proxies(token):
@@ -1,5 +1,5 @@
export function runtimeRemoteScriptTail(): string {
return `def native_customer_error_profile(token, group_id, platform):
return `def native_customer_error_profile(token, group_id, platform, account_ids=None):
start_time = quote(since_start_time(PAYLOAD.get("since")), safe="")
platform_query = "&platform=" + quote(str(platform), safe="") if platform else ""
page = 1
@@ -11,7 +11,8 @@ export function runtimeRemoteScriptTail(): string {
response = curl_api("GET", f"/api/v1/admin/ops/request-errors?group_id={group_id}&start_time={start_time}&view=all&page={page}&page_size={page_size}{platform_query}", bearer=token)
data = data_of(response, "get customer-visible request errors")
page_rows = items_of(data)
rows.extend(page_rows)
selected_rows = [row for row in page_rows if not account_ids or str(row.get("account_id")) in account_ids]
rows.extend(selected_rows)
if total is None and isinstance(data, dict) and isinstance(data.get("total"), int):
total = data.get("total")
if not page_rows or len(page_rows) < page_size or (isinstance(total, int) and len(rows) >= total):
@@ -19,7 +20,9 @@ export function runtimeRemoteScriptTail(): string {
page += 1
except Exception as exc:
return {"status": "unavailable", "source": "sub2api-native-admin-ops-request-errors", "reason": compact_error(str(exc))}
if total is None:
if account_ids:
total = len(rows)
elif total is None:
total = len(rows)
def buckets(key_fn):
counts = {}
@@ -273,6 +276,7 @@ def account_quality_scores(token, accounts, group_id, native_ops, effectiveness)
rows.append({
"accountId": account_id, "accountName": account.get("name"), "status": account.get("status"),
"schedulable": account.get("schedulable"), "currentlyAvailable": current_available,
"priority": account.get("priority"), "priorityOrder": "lower-is-higher",
"score": score, "grade": grade, "assessment": assessment, "confidence": confidence,
"scoreComparable": score_comparable,
"observedAttempts": observed_attempts, "successRequests": success_requests,
@@ -816,7 +820,7 @@ def observed_runtime_errors(token, accounts, group_id, platform=None):
overview_summary = ((native_ops.get("overview") or {}).get("summary") or {})
native_ops["responseHeaderTimeout"] = runtime_response_header_timeout()
native_ops["requestTiming"] = native_request_timing(token, group_id, platform, overview_summary)
native_ops["customerErrors"] = native_customer_error_profile(token, group_id, platform)
native_ops["customerErrors"] = native_customer_error_profile(token, group_id, platform, selected_ids if selector is not None else None)
native_ops["accountQuality"] = account_quality_scores(token, selected, group_id, native_ops, effectiveness)
request_total = overview_summary.get("request_count_total")
error_total = overview_summary.get("error_count_total")
@@ -1204,7 +1208,7 @@ def runtime_result():
return {**base, "ok": True, "operation": "errors", "mutation": False, "errors": errors}
detail = find_account(token, accounts, str(PAYLOAD["account"])) if action in ("get", "infrastructure") else None
if action == "get":
return {**base, "ok": True, "operation": "get", "mutation": False, "account": account_summary(detail, True)}
return {**base, "ok": True, "operation": "get", "mutation": False, "account": account_summary(detail, True, token)}
if action == "infrastructure":
return {
**base,
@@ -227,6 +227,8 @@ function compactAccountQuality(value: unknown, detailed: boolean): unknown {
accountId: account.accountId,
accountName: account.accountName,
status: account.status,
priority: account.priority,
priorityOrder: account.priorityOrder,
currentlyAvailable: account.currentlyAvailable,
score: account.score,
grade: account.grade,
@@ -254,6 +256,8 @@ function compactAccountQuality(value: unknown, detailed: boolean): unknown {
} : {
accountId: account.accountId,
accountName: account.accountName,
priority: account.priority,
priorityOrder: account.priorityOrder,
currentlyAvailable: account.currentlyAvailable,
score: account.score,
grade: account.grade,
@@ -311,6 +315,11 @@ export function renderRuntimeResult(response: Record<string, unknown>, options:
const runtime = runtimeRecord(response.runtime) ?? {};
const ok = response.ok === true;
const lines: string[] = [];
if (runtime.ok !== true && typeof runtime.error === "string") {
lines.push(`SUB2API RUNTIME ${options.action.toUpperCase()} ok=false`, `ERROR: ${runtime.error}`);
lines.push("", `JSON: bun scripts/cli.ts platform-infra sub2api codex-pool runtime ${options.action} <same-options> --json`);
return renderedCliResult(ok, `platform-infra sub2api codex-pool runtime ${options.action}`, lines.join("\n"));
}
if (options.action === "errors") renderRuntimeErrors(lines, runtime, options);
else if (options.action === "infrastructure") renderRuntimeInfrastructure(lines, runtime);
else if (options.action === "list") renderRuntimeAccountList(lines, runtime);
@@ -406,9 +415,9 @@ function renderRuntimeErrors(lines: string[], runtime: Record<string, unknown>,
const qualityAccounts = runtimeRecords(accountQuality.accounts);
if (qualityAccounts.length > 0) {
lines.push("", "ACCOUNT QUALITY", renderTable([
["ACCOUNT", "ID", "SCORE", "GRADE", "QUALITY", "CONF", "ATTEMPTS", "SUCCESS", "FAIL", "UP_CUST", "EXCLUDED", "FAIL_RATE", "TTFT_P95_MS", "TTFT_N", "AVAILABLE", "REASONS"],
["ACCOUNT", "ID", "PRIORITY", "SCORE", "GRADE", "QUALITY", "CONF", "ATTEMPTS", "SUCCESS", "FAIL", "UP_CUST", "EXCLUDED", "FAIL_RATE", "TTFT_P95_MS", "TTFT_N", "AVAILABLE", "REASONS"],
...qualityAccounts.map((item) => [
runtimeText(item.accountName), runtimeText(item.accountId), runtimeText(item.score), runtimeText(item.grade), runtimeText(item.assessment), runtimeText(item.confidence),
runtimeText(item.accountName), runtimeText(item.accountId), runtimeText(item.priority), runtimeText(item.score), runtimeText(item.grade), runtimeText(item.assessment), runtimeText(item.confidence),
runtimeText(item.observedAttempts), runtimeText(item.successRequests), runtimeText(item.failureRequests), runtimeText(item.scoreableUpstreamErrorRequests),
runtimeText(item.excludedNonUpstreamErrorRequests), runtimeRate(item.failureRate),
runtimeText(item.ttftP95Ms), runtimeText(item.firstTokenSamples), runtimeText(item.currentlyAvailable),
@@ -439,7 +448,7 @@ function renderRuntimeErrors(lines: string[], runtime: Record<string, unknown>,
]),
]));
const scoreRule = runtimeRecord(accountQuality.scoreRule) ?? {};
lines.push(`SCORE RULE reliability=${runtimeText(scoreRule.reliabilityWeight)} latency=${runtimeText(scoreRule.latencyWeight)} availability=${runtimeText(scoreRule.availabilityWeight)} confidence=separate`);
lines.push(`SCORE RULE reliability=${runtimeText(scoreRule.reliabilityWeight)} latency=${runtimeText(scoreRule.latencyWeight)} availability=${runtimeText(scoreRule.availabilityWeight)} confidence=separate priority=lower-is-higher highest=0`);
if (typeof accountQuality.attribution === "string") lines.push(`NOTE: ${accountQuality.attribution}`);
}
const errorModels = runtimeRecords(customerErrors.modelBuckets);
@@ -535,6 +544,12 @@ function renderRuntimeAccountList(lines: string[], runtime: Record<string, unkno
function renderRuntimeAccountDetail(lines: string[], runtime: Record<string, unknown>): void {
const account = runtimeRecord(runtime.account) ?? {};
const temp = runtimeRecord(account.tempUnschedulable) ?? {};
const priority = runtimeRecord(account.prioritySemantics) ?? {};
const limits = runtimeRecord(account.upstreamLimits) ?? {};
const probe = runtimeRecord(limits.nativeProbe) ?? {};
const globalSettings = runtimeRecord(limits.globalSettings) ?? {};
const windows = runtimeRecords(limits.windows);
const protections = runtimeRecords(limits.protections);
lines.push(`SUB2API RUNTIME ACCOUNT ok=${runtime.ok === true ? "true" : "false"}`);
lines.push(renderTable([
["NAME", "ID", "MANAGEMENT", "STATUS", "SCHED", "PROXY", "CONCURRENCY", "LOAD", "PRIORITY", "TEMPLATE"],
@@ -544,10 +559,39 @@ function renderRuntimeAccountDetail(lines: string[], runtime: Record<string, unk
runtimeText(account.loadFactor), runtimeText(account.priority), runtimeText(account.matchingTemplate),
],
]));
lines.push(renderTable([
["PRIORITY_ORDER", "HIGHEST_PRIORITY"],
[runtimeText(priority.order), runtimeText(priority.highest)],
]));
lines.push(renderTable([
["TEMP_ENABLED", "RULES", "STATUS_CODES", "POOL_MODE"],
[runtimeText(temp.enabled), runtimeText(temp.ruleCount), runtimeText(temp.statusCodes), runtimeText(account.poolModeEnabled)],
]));
lines.push("", "UPSTREAM LIMITS");
lines.push(renderTable([
["WINDOW", "USED", "RESET_AFTER", "RESET_AT", "DATA_SOURCE", "PROTECT_AT", "PROTECT_SOURCE", "TRIGGERED"],
...windows.map((window) => {
const protection = protections.find((item) => item.window === window.window) ?? {};
return [
runtimeText(window.window), runtimePercent(window.usedPercent), runtimeSeconds(window.resetAfterSeconds),
runtimeText(window.resetAt), runtimeText(window.source), runtimePercent(protection.thresholdPercent),
runtimeText(protection.source), runtimeText(protection.triggered),
];
}),
]));
lines.push(`NATIVE_QUOTA status=${runtimeText(probe.status)} allowed=${runtimeText(probe.allowed)} limitReached=${runtimeText(probe.limitReached)} fetchedAt=${runtimeText(probe.fetchedAt)}`);
if (typeof probe.reason === "string") lines.push(`NATIVE_QUOTA_REASON: ${probe.reason}`);
lines.push(`GLOBAL_AUTO_PAUSE status=${runtimeText(globalSettings.status)}`);
if (typeof globalSettings.reason === "string") lines.push(`GLOBAL_AUTO_PAUSE_REASON: ${globalSettings.reason}`);
if (typeof limits.attribution === "string") lines.push(`NOTE: ${limits.attribution}`);
}
function runtimePercent(value: unknown): string {
return typeof value === "number" && Number.isFinite(value) ? `${value.toFixed(2)}%` : "-";
}
function runtimeSeconds(value: unknown): string {
return typeof value === "number" && Number.isFinite(value) ? `${Math.round(value)}s` : "-";
}
function renderRuntimeInfrastructure(lines: string[], runtime: Record<string, unknown>): void {