Merge pull request #637 from pikasTech/fix/v02-runner-output-transport-hardening

fix: harden v0.2 code agent transport
This commit is contained in:
Lyon
2026-05-31 22:56:09 +08:00
committed by GitHub
8 changed files with 553 additions and 10 deletions
@@ -206,6 +206,21 @@ Workbench trace 对已知 JSON-RPC gateway 响应应按普通 tool call 展示
## 短连接 result 轮询
## Codex app-server stdio 请求处理与 trace 可见性
repo-owned Codex app-server stdio client 必须把 app-server 发来的 JSON-RPC client request 当作一等协议处理,不能只等待 notification。请求 id 可能是数字也可能是字符串;只支持数字 id 会导致 approval/requestUserInput 等请求永远得不到响应,最终表现为已有 assistant partial output 后卡在 `waitingFor=app-server-notification`,直到 idle timeout。
非交互 HWLAB Code Agent 的处理规则如下:
- `item/commandExecution/requestApproval``item/fileChange/requestApproval``item/permissions/requestApproval` 必须自动返回可用的 approved 决策,优先使用 `availableDecisions` 中的 `approved``approved_for_session` 或其他 `approved*`,不能因为旧 approval gate 把真实工具调用拒绝掉。
- `item/tool/requestUserInput``mcpServer/elicitation/request` 必须返回结构化 denied/canceled;聊天运行中不能弹人工输入,也不能静默等待用户。
- 未知 client request 必须返回 JSON-RPC unsupported error,并在 trace 中记录 `client/request:unsupported`;禁止忽略请求。
- 每个已处理 client request 都必须追加 `client/request/handled` trace 事件,包含 method、decision、itemId/targetItemId,并把 `waitingFor` 推进到 `turn/completed`,避免前端只显示未知 `providerTrace 缺失``app-server-notification`
commandExecution trace 必须 bounded:命令文本和 stdout/stderr summary 默认只展示有界摘要,但必须保留 `commandBytes``commandTruncated``outputBytes` 和 stderr 摘要,复制/下载 trace 仍可按后端保留策略展开原始事件。前端和 CLI 不能为了避免输出爆炸而要求 Code Agent 默认加 `| head``grep` 或 shell pipe;输出裁剪应在 trace/result 层完成,不能把 stdout pipe 行为变成工具调用是否完成的隐式前提。
在 Linux container 内启动 Codex app-server 时,应优先直接执行 `@openai/codex-linux-*` native binary,并把同包 `path` 目录加入 `PATH`;只有找不到 native binary 时才回退到 npm wrapper。直接启动 native binary 可以减少 Node wrapper 双进程、孤儿进程和 close 后未清理进程组导致的 session 卡死风险。关闭 stdio client 时必须终止整个子进程组,不只关闭父进程 stdin。
Workbench 与 Code Agent 的用户请求必须是短连接 submit + 短连接 result/trace 轮询;浏览器或 cloud-web 不应持有一次长 HTTP 请求等待整个 Codex turn 结束。`POST /v1/agent/chat` 返回 `202` 后,前端轮询 `/v1/agent/chat/result/<traceId>` 获取终态,并用 `/v1/agent/chat/trace/<traceId>` 刷新可视 trace。
Code Agent backend 的 completed 语义只能来自真实 Codex app-server `turn/completed` 成功事件。`item/agentMessage/delta``item/completed`、已有 assistant 文本、transport close 或 activity idle timeout 都不能单独升级成 `status: "completed"`。如果已经收到部分 assistant 文本但没有收到 `turn/completed`,终态必须是 timeout/partial blocker,并保留 trace、session、thread、partial output 摘要和可重试提示;Workbench 只能显示“部分回复/超时”,不能标 DEV-LIVE reply pass。
+10 -1
View File
@@ -8,11 +8,12 @@
- 提供 WEB 等价的非视觉业务入口:登录鉴权、session 恢复、Device Pod 看板、Code Agent 对话、trace/result 轮询、logout 和工作台 live summary。
- 只走 Cloud Web 同源 API surface;默认 base URL 是 `http://74.48.78.17:19666`,也可通过 `--base-url``HWLAB_CLIENT_BASE_URL` 指向其他 Cloud Web 入口。
- 不直连 Postgres、Kubernetes Service、Secret、device-pod 内部 Service、gateway RPC 或本地 fixture;需要鉴权的请求使用 `/auth/*` 返回的 cookie 或显式 `--cookie`
- 默认业务子命令不直连 Postgres、Kubernetes Service、Secret、device-pod 内部 Service、gateway RPC 或本地 fixture;需要鉴权的请求使用 `/auth/*` 返回的 cookie 或显式 `--cookie`唯一例外是 `client gateway` 诊断族:它允许显式 `--api-base-url` 指向 Cloud API,用于短连接观测 gateway session、单次 shell invoke 和 transport 压测;该入口只验证底层传输稳定性,不替代 Web 用户流程授权,也不发布镜像或常驻服务。
- Pod 内透传执行不放进 `hwlab-cli`;需要进入正在工作的 Code Agent/Cloud API pod 时,`hwlab-cli` 只查询并输出 UniDesk 标准 route,实际透传由 UniDesk `bun scripts/cli.ts ssh 'G14:k3s:hwlab-v02:pod:<pod>:<container>' ...` 完成。`pod:` 是 route 语法,`/` 只用于 pod 内文件系统路径。
- `client runtime routes` 必须按当前运行 profile/lane 的数据生成 UniDesk `pod:` route;实现不得硬编码 `dev``v0.2``v0.3`、namespace 或 catalog path。新增版本只允许通过 `deploy.json.lanes[profile]` 声明 namespace、artifact catalog 和 service overrides,不为每个版本新增代码分支。
- 运行时不做内部证明型校验、旧健康诊断或重断言;CI/CD 只保留能证明代码可构建、语法正确和最小冒烟可用的校验。功能正确性通过 `hwlab-cli client` 短连接真实业务 E2E 暴露和修复。
- 专用子命令覆盖高频用户工作台;`client request METHOD /path` 覆盖 WEB 同源代理允许的其他非视觉 API。`client request` 只接受以 `/` 开头的 Cloud Web 相对路径,禁止绝对 URL,避免绕过 Cloud Web 直接打内部服务。
- `client gateway pressure` 是 device-pod/gateway 高频故障的真实业务传输压测入口;必须覆盖 small stdout、大 stdout、长单行 stdout、stderr flood、结构化 timeout 和超出 gateway inflight 上限的并发请求。所有场景必须返回 JSON、HTTP/route/traceId/requestId、字节数、truncated 标记、sha256 和 bounded preview;失败必须明确是 `http_*``stdout_not_truncated``stderr_not_truncated``timeout_not_observed``structured_gateway_busy` 等可定位原因,禁止无输出、长时间黑洞或只靠 shell pipe 截断。
- 输出默认是 JSON;任何失败都要有 `ok:false``action``status`、HTTP 状态、route 和可定位错误,不允许无 stdout 成功。可能返回大对象的 `client` 子命令默认返回紧凑摘要,避免高频排障输出爆炸;需要完整响应体时显式加 `--full`
- `device-pod-cli`/`hwpod``job output` 默认也必须返回紧凑 JSON:保留 job/status/blocker/freshness/text/evidence 摘要,省略嵌套 gateway dispatch 和长命令;需要完整 payload 时显式加 `--full`。Code Agent 和人工不得用 `| head``grep` 或 shell 管道作为默认输出压缩方式,避免 stdout pipe、子进程信号转发或长输出造成 commandExecution 黑洞。
- Code Agent 交互必须默认暴露 `traceId``resultUrl`、终态和 assistant 回复文本摘要;不能要求用户先拉全量 trace 再手工查找回复。
@@ -40,6 +41,9 @@
| `hwlab-cli client device-pods events POD` | `GET /v1/device-pods/{pod}/events` | 对应纯文本事件流。 |
| `hwlab-cli client device-pods probe POD` | `/debug-probe/chip-id``/io-probe/uart/1``/tail` | 对应 Target/Debug/IO 摘要。 |
| `hwlab-cli client runtime routes` | `GET /v1/live-builds` | 查询当前工作面 pod,并输出 UniDesk 标准 `pod:` route;不执行透传、不调用 kubectl、不内嵌 UniDesk。 |
| `hwlab-cli client gateway sessions` | `GET Cloud API /v1/gateway/sessions` | 显式 Cloud API 诊断入口,观察 gateway online/stale、inflight 和 capability;默认不带 Web cookie。 |
| `hwlab-cli client gateway invoke` | `POST Cloud API /v1/rpc/hardware.invoke.shell` | 显式 Cloud API 诊断入口,执行一次 bounded shell dispatch 并返回结构化 dispatch 摘要。 |
| `hwlab-cli client gateway pressure` | `POST Cloud API /v1/rpc/hardware.invoke.shell` | 显式 Cloud API 压测入口,真实验证大输出、长单行、stderr、timeout 和并发超容量不会造成黑洞。 |
| `hwlab-cli client agent send` | `POST /v1/agent/chat` + `GET /result/{trace}` | 以 short connection 提交 Code Agent 消息并轮询结果,默认输出 assistant 回复文本摘要。 |
| `hwlab-cli client agent trace TRACE` | `GET /v1/agent/chat/trace/{trace}` | 回放 trace,默认输出状态、事件摘要和 assistant stream 文本。 |
| `hwlab-cli client agent cancel TRACE` | `POST /v1/agent/chat/cancel` | 取消当前 Code Agent 请求。 |
@@ -78,6 +82,10 @@
阅读 docs/reference/spec-v02-hwlab-cli.md,然后在 `G14:/root/hwlab-v02` 用 cli 手动测试以下内容:运行 `hwpod job output --pod-id D601-F103-V2 <jobId> --api-base-url http://74.48.78.17:19667`,确认默认输出包含 `body.compacted=true`、状态、job 摘要和 bounded text,且不包含嵌套 `dispatch.command`;再加 `--full` 确认完整 payload 可按需展开。通过 `client harness submit` 让 Code Agent 执行同一 `hwpod job output`,确认 trace 中 commandExecution 可以完成,不需要 `| head`
## T7
阅读 docs/reference/spec-v02-hwlab-cli.md,然后在 `G14:/root/hwlab-v02` 用 cli 手动测试以下内容:运行 `node scripts/run-bun.mjs tools/hwlab-cli/bin/hwlab-cli.ts client gateway pressure --api-base-url http://74.48.78.17:19667 --gateway-session-id gws_D601_F103 --large-bytes 131072 --parallel 8 --request-timeout-ms 60000 --timeout-scenario-ms 1000 --no-auth --full`。确认 small stdout、大 stdout、长单行 stdout、stderr flood、timeout 和并发超容量全部返回结构化 JSON;大输出应显示 `stdoutBytes``stderrBytes``stdoutTruncated``stderrTruncated`、sha256 和 preview,超出 `maxInflightRequests` 的请求必须显示 `structured_gateway_busy`,不能出现无输出或卡死。
## 规格的实现情况
| 规格项 | 状态 | 说明 |
@@ -87,6 +95,7 @@
| JSON-RPC 同源 API | 目标状态 | `client rpc` 自动补齐 Web JSON-RPC envelope 的 `meta` 字段。 |
| 通用同源 API request | 目标状态 | `client request` 用于追平低频和新增 WEB API,禁止绝对 URL。 |
| G14 harness-ops 短连接能力 | 目标状态 | `client harness` / `client harness-ops` / `client harness-opt` 覆盖 submit/result/trace/wait/audit,只作为业务 API client。 |
| Gateway transport 压测 | 已实现 | `client gateway pressure` 只作为显式短连接诊断入口,覆盖大输出、timeout 和并发超容量的结构化返回。 |
| Device Pod job output 紧凑输出 | 已实现 | `hwpod job output` 默认省略嵌套 dispatch`--full` 才展开完整 payload,防止 Code Agent 通过 shell pipe 压输出。 |
| 本地 cookie session | 目标状态 | `.state/hwlab-cli/session.json` 只保存 cookie/session 摘要。 |
| 镜像/Service/Job template | 已废弃 | 相关 deploy、GitOps、artifact 和 Tekton 口径必须删除。 |
+8
View File
@@ -7,12 +7,15 @@
-`hwlab-cloud-api` 注册 gateway session、resource 和 capability。
- 通过主动轮询 `/v1/gateway/poll` 获取 cloud-api 分发的 `hardware.invoke.shell` 请求,并把结果回传 `/v1/gateway/result`
- 只做 transport 和 bounded command execution,不理解用户权限、device-pod grant 或业务授权。
- transport 稳定性是 P0 基础能力:大 stdout、长单行 stdout、stderr flood、命令 timeout、gateway 超容量并发和结果回传失败都必须返回结构化 JSON 状态;禁止让 poll/result、Cloud API request、Code Agent commandExecution 或 `hwlab-cli` 调用进入无 stdout、无 trace、无 terminal status 的黑洞。
## 内部架构
- `cmd/hwlab-gateway/main.ts` 维护 gateway state、outbound poll loop、inflight request map 和 command execution,运行入口必须使用 Bun。
- `internal/cloud/gateway-demo-registry.ts` 在 cloud-api 内保存 gateway session、队列和 pending result。
- command execution 默认关闭;只有显式 `HWLAB_GATEWAY_CMD_EXEC_ENABLED=1` 或 demo open 时才执行 shell。
- command execution 必须同时读取 stdout/stderr,并对每路输出做有界收集和 `stdoutTruncated`/`stderrTruncated` 标记,避免任一路 pipe 填满导致子进程或 gateway result 卡死。默认 bounded body 可以截断,但必须保留字节数、truncated 标记和 enough preview;需要完整超大输出时应另设计 evidence/spool,不得把完整无限输出塞进同步 JSON 响应。
- gateway inflight 上限是背压机制,不是黑洞机制。超过 `maxInflightRequests` 时,Cloud API/gateway 必须返回包含 `reason=gateway_busy``inflightCount``maxInflightRequests` 的结构化结果,调用方可重试或排队。
## API 接口说明
@@ -34,6 +37,10 @@
阅读 docs/reference/spec-v02-hwlab-gateway.md,然后用 cli 手动测试以下内容:通过 cloud-api `/v1/gateway/sessions` 观察 gateway online/stale 状态;不要直接从前端或普通用户请求 gateway shell。
## T3
阅读 docs/reference/spec-v02-hwlab-gateway.md,然后在 `G14:/root/hwlab-v02``hwlab-cli client gateway pressure` 手动测试以下内容:对目标 gateway 运行 small stdout、至少 128KiB stdout、至少 128KiB 单行 stdout、至少 128KiB stderr、短 timeout 和超过 gateway `maxInflightRequests` 的并发请求。验收条件是全部场景在 CLI timeout 内返回结构化 JSON;大输出显示 truncated 标记,timeout 显示 `timed_out`,超容量显示 `gateway_busy`,不能出现无输出、HTTP transport timeout 或 Code Agent trace 黑洞。
## 规格的实现情况
| 规格项 | 状态 | 说明 |
@@ -41,6 +48,7 @@
| health/status/capabilities | 已实现 | gateway service 提供只读观测。 |
| outbound poll/result | 已实现 | 与 cloud-api registry 对接。 |
| bounded shell execution | 已实现 | 受 env 开关、timeout 和 output limit 约束。 |
| gateway 压测闭环 | 已实现 | `hwlab-cli client gateway pressure` 覆盖大输出、timeout 和并发背压,不依赖 shell pipe 裁剪。 |
| device-pod grant/lease | 不在本服务 | 由 cloud-api/device-pod 负责。 |
| 生产级 gateway 多租户隔离 | 未完全实现 | 当前是 demo/transport skeleton。 |
@@ -13,6 +13,7 @@ import {
firstNonEmpty,
optionalId,
redactText,
boundToolOutput,
tailText
} from "./codex-stdio-session-helpers.ts";
@@ -178,6 +179,25 @@ export function createAppServerTurnState({ traceRecorder, session } = {}) {
});
return;
}
if (method === "client/request/handled") {
const decision = extractAppServerString(params, "decision") ?? "handled";
const handledMethod = extractAppServerString(params, "method") ?? "client/request";
appendTrace({
type: "client_request",
status: decision === "unsupported" || decision === "denied" ? "degraded" : "completed",
label: `client/request:${decision}`,
toolName: handledMethod,
itemId: optionalId(params?.itemId ?? params?.targetItemId),
message: `Handled app-server client request ${handledMethod} with decision=${decision}.`,
sessionId: session?.sessionId,
sessionStatus: session?.status,
turn: session?.turn,
threadId,
turnId,
waitingFor: "turn/completed"
});
return;
}
if (method === "item/started" && item?.type === "commandExecution") {
appendCommandExecutionTrace(item, {
status: "started",
@@ -253,6 +273,9 @@ export function createAppServerTurnState({ traceRecorder, session } = {}) {
function appendCommandExecutionTrace(item, { status, label }) {
const output = commandExecutionOutputText(item);
const command = commandExecutionCommandText(item);
const boundedCommand = boundToolOutput(redactText(command), 1600);
const boundedOutput = output ? boundToolOutput(redactText(tailText(output, 2000)), 2000) : { text: "", truncated: false };
appendTrace({
type: "tool_call",
stage: "tool_call",
@@ -260,11 +283,13 @@ export function createAppServerTurnState({ traceRecorder, session } = {}) {
label,
itemId: optionalId(item.id),
toolName: "commandExecution",
command: redactText(commandExecutionCommandText(item)),
command: boundedCommand.text,
commandBytes: Buffer.byteLength(command, "utf8"),
commandTruncated: boundedCommand.truncated,
exitCode: Number.isInteger(item.exitCode) ? item.exitCode : undefined,
durationMs: typeof item.durationMs === "number" ? item.durationMs : undefined,
outputBytes: output.length,
stdoutSummary: output ? tailText(redactText(output), 600) : undefined,
outputBytes: Buffer.byteLength(output, "utf8"),
stdoutSummary: boundedOutput.text || undefined,
stderrSummary: commandExecutionErrorText(item),
sessionId: session?.sessionId,
sessionStatus: session?.status,
@@ -410,6 +435,7 @@ export function createAppServerTurnState({ traceRecorder, session } = {}) {
}
function appServerWaitingForMethod(method) {
if (method === "client/request/handled") return "turn/completed";
if (method === "thread/started") return "turn/start";
if (method === "turn/started") return "assistant-message";
if (method === "item/agentMessage/delta") return "turn/completed";
+91 -6
View File
@@ -1420,10 +1420,60 @@ function tomlString(value) {
return JSON.stringify(String(value ?? ""));
}
function spawnCodexAppServerProcess(command, args, options = {}) {
const native = resolveCodexNativeSpawn(command, options.env ?? process.env);
return spawn(native?.command ?? command, args, {
...options,
env: native?.env ?? options.env
});
}
function resolveCodexNativeSpawn(command, env = process.env) {
const resolved = commandPathSync(command, env);
if (!resolved) return null;
let real = resolved;
try {
real = realpathSync(resolved);
} catch {
// Keep the commandPathSync result; spawn will surface any executable error.
}
const packageRoot = findPackageRoot(real);
if (!packageRoot) return null;
const candidates = codexNativeBinaryCandidates(packageRoot);
const nativeCommand = candidates.find((candidate) => existsSync(candidate) && accessSyncBoolean(candidate, fsConstants.X_OK));
if (!nativeCommand) return null;
const pathDir = path.resolve(nativeCommand, "..", "..", "path");
const envPath = String(env.PATH ?? process.env.PATH ?? "");
return {
command: nativeCommand,
env: {
...env,
PATH: existsSync(pathDir) ? [pathDir, envPath].filter(Boolean).join(path.delimiter) : envPath,
CODEX_MANAGED_BY_NPM: env.CODEX_MANAGED_BY_NPM ?? "1"
}
};
}
function codexNativeBinaryCandidates(packageRoot) {
if (process.platform === "linux" && process.arch === "x64") {
return [
path.join(packageRoot, "..", "codex-linux-x64", "vendor", "x86_64-unknown-linux-musl", "codex", "codex"),
path.join(packageRoot, "vendor", "x86_64-unknown-linux-musl", "codex", "codex")
];
}
if (process.platform === "linux" && process.arch === "arm64") {
return [
path.join(packageRoot, "..", "codex-linux-arm64", "vendor", "aarch64-unknown-linux-musl", "codex", "codex"),
path.join(packageRoot, "vendor", "aarch64-unknown-linux-musl", "codex", "codex")
];
}
return [];
}
export function createCodexAppServerJsonLineClient({ command = DEFAULT_CODEX_STDIO_COMMAND, args = null, env = process.env, cwd = repoRoot, onNotification = null } = {}) {
const spawnArgs = Array.isArray(args) ? args : codexAppServerArgs(env);
const childDetached = process.platform !== "win32";
const child = spawn(command, spawnArgs, {
const child = spawnCodexAppServerProcess(command, spawnArgs, {
cwd,
env,
detached: childDetached,
@@ -1464,7 +1514,7 @@ export function createCodexAppServerJsonLineClient({ command = DEFAULT_CODEX_STD
} catch {
return;
}
const id = typeof payload.id === "number" ? payload.id : null;
const id = typeof payload.id === "number" || typeof payload.id === "string" ? payload.id : null;
const method = typeof payload.method === "string" ? payload.method : null;
if (id !== null && method === null) {
const request = pending.get(id);
@@ -1479,7 +1529,7 @@ export function createCodexAppServerJsonLineClient({ command = DEFAULT_CODEX_STD
return;
}
if (id !== null && method !== null) {
handleServerRequest(id, method);
handleServerRequest(id, method, extractAppServerRecord(payload.params));
return;
}
if (method !== null && notificationHandler) {
@@ -1490,6 +1540,7 @@ export function createCodexAppServerJsonLineClient({ command = DEFAULT_CODEX_STD
function closeWith(message, code, signal) {
if (closed) return;
closed = true;
terminateChildProcessGroup(child, { detached: childDetached });
for (const [id, request] of pending.entries()) {
pending.delete(id);
clearTimeout(request.timer);
@@ -1547,14 +1598,44 @@ export function createCodexAppServerJsonLineClient({ command = DEFAULT_CODEX_STD
child.stdin.write(`${JSON.stringify({ method, params })}\n`);
}
function handleServerRequest(id, method) {
if (method === "item/commandExecution/requestApproval" || method === "item/fileChange/requestApproval") {
child.stdin.write(`${JSON.stringify({ id, result: { decision: "decline" } })}\n`);
function handleServerRequest(id, method, params = {}) {
if (method === "item/commandExecution/requestApproval" || method === "item/fileChange/requestApproval" || method === "item/permissions/requestApproval") {
const approval = approvalResponseForServerRequest(params);
notifyClientRequest(method, params, approval.decision);
child.stdin.write(`${JSON.stringify({ id, result: approval })}\n`);
return;
}
if (method === "item/tool/requestUserInput" || method === "mcpServer/elicitation/request") {
notifyClientRequest(method, params, "denied");
child.stdin.write(`${JSON.stringify({ id, result: { canceled: true, decision: "denied", message: "Non-interactive HWLAB Code Agent cannot request user input during a running turn." } })}\n`);
return;
}
notifyClientRequest(method, params, "unsupported");
child.stdin.write(`${JSON.stringify({ id, error: { code: -32601, message: `Unsupported client-side request: ${method}` } })}\n`);
}
function approvalResponseForServerRequest(params = {}) {
const available = Array.isArray(params?.availableDecisions) ? params.availableDecisions.map(String) : [];
const decision = available.find((item) => item === "approved")
?? available.find((item) => item === "approved_for_session")
?? available.find((item) => item.startsWith("approved"))
?? "approved";
return { decision };
}
function notifyClientRequest(method, params, decision) {
notificationHandler?.({
method: "client/request/handled",
params: {
method,
decision,
itemId: optionalId(params?.itemId ?? params?.item?.id),
approvalId: optionalId(params?.approvalId ?? params?.id),
targetItemId: optionalId(params?.targetItemId)
}
});
}
async function startThread({ model, cwd: threadCwd, sandbox, approvalPolicy = "never", serviceName = "hwlab-cloud-api" } = {}, timeoutMs = 30000) {
const result = await request("thread/start", dropEmpty({
model,
@@ -1616,6 +1697,10 @@ export function createCodexAppServerJsonLineClient({ command = DEFAULT_CODEX_STD
}
function terminateChildProcess(child, { detached = false } = {}) {
return terminateChildProcessGroup(child, { detached });
}
function terminateChildProcessGroup(child, { detached = false } = {}) {
if (!child || child.killed) return;
try {
if (detached && Number.isInteger(child.pid)) {
+113
View File
@@ -13,8 +13,10 @@ import {
import {
codexAppServerArgs,
codexAppServerProviderBaseUrl,
createCodexAppServerJsonLineClient,
createCodexStdioSessionManager
} from "../internal/cloud/codex-stdio-session.ts";
import { createAppServerTurnState } from "../internal/cloud/codex-stdio-session-turn-state.ts";
import {
classifyCodexRunnerCapability,
classifyCodeAgentChatReadiness,
@@ -48,6 +50,67 @@ function runnerTraceLabels(payload) {
: [];
}
async function runJsonLineClientApprovalSmoke() {
const fakeCodex = await createFakeJsonLineCodexCommand();
const events = [];
const client = createCodexAppServerJsonLineClient({
command: fakeCodex.command,
args: ["app-server", "--listen", "stdio://"],
cwd: fakeCodex.root,
env: { ...process.env, PATH: `${path.dirname(fakeCodex.command)}:${process.env.PATH ?? ""}` },
onNotification: (message) => events.push(message)
});
try {
await client.initialize(2000);
const thread = await client.startThread({ model: "gpt-test", cwd: fakeCodex.root, sandbox: "workspace-write" }, 2000);
assert.equal(thread.threadId, "thread_jsonline_test");
const turn = await client.startTurn({ threadId: thread.threadId, prompt: "trigger approval", model: "gpt-test", cwd: fakeCodex.root }, 2000);
assert.equal(turn.turnId, "turn_jsonline_test");
assert.equal(turn.result.approvalDecision, "approved_for_session");
assert.equal(turn.result.spawnedFrom, "native");
assert.ok(events.some((event) => event.method === "client/request/handled" && event.params?.decision === "approved_for_session"));
assert.ok(events.some((event) => event.method === "item/completed" && event.params?.item?.type === "commandExecution"));
} finally {
client.close();
await rm(fakeCodex.root, { recursive: true, force: true });
}
logOk("codex app-server JSON-line client approves command requests and uses native binary directly");
}
function runTurnStateVisibilitySmoke() {
const events = [];
const state = createAppServerTurnState({
traceRecorder: {
append: (event) => {
events.push(event);
return event;
},
appendAssistantDelta: (event) => {
events.push({ ...event, label: "assistant:stream" });
return event;
}
},
session: { sessionId: "ses_visibility_smoke", status: "busy", turn: 1 }
});
const command = "printf ".padEnd(3000, "c");
const stdout = "x".repeat(5000);
state.handle({ method: "thread/started", params: { thread: { id: "thread_visibility_smoke" } } });
state.handle({ method: "turn/started", params: { turn: { id: "turn_visibility_smoke" } } });
state.handle({ method: "client/request/handled", params: { method: "item/commandExecution/requestApproval", decision: "approved", itemId: "cmd_visibility" } });
state.handle({
method: "item/completed",
params: { item: { id: "cmd_visibility", type: "commandExecution", status: "completed", exitCode: 0, command, stdout } }
});
const clientEvent = events.find((event) => event.label === "client/request:approved");
const commandEvent = events.find((event) => event.label === "item/commandExecution:completed");
assert.equal(clientEvent?.status, "completed");
assert.equal(commandEvent?.commandBytes, Buffer.byteLength(command, "utf8"));
assert.equal(commandEvent?.outputBytes, Buffer.byteLength(stdout, "utf8"));
assert.equal(commandEvent?.commandTruncated, true);
assert.ok(commandEvent?.stdoutSummary?.length <= 2100);
logOk("codex app-server turn trace bounds large command/output while preserving byte counts");
}
async function runLocalContractSmoke() {
const echoReadiness = classifyCodeAgentChatReadiness({
conversationId: "cnv_code-agent-chat-smoke-echo",
@@ -117,6 +180,9 @@ async function runLocalContractSmoke() {
}
logOk("skills/pwd/ls/rg/cat prompts fail with Codex stdio blocker instead of local fallback");
await runJsonLineClientApprovalSmoke();
await runTurnStateVisibilitySmoke();
const stdioWorkspaceRoot = await mkdtemp(path.join(os.tmpdir(), `hwlab-code-agent-stdio-smoke-${process.pid}-`));
const stdioWorkspace = path.join(stdioWorkspaceRoot, "workspace");
const stdioCodexHome = path.join(stdioWorkspaceRoot, "codex-home");
@@ -487,6 +553,53 @@ async function createFakeCodexCommand() {
return { root, command };
}
async function createFakeJsonLineCodexCommand() {
const root = await mkdtemp(path.join(os.tmpdir(), "hwlab-fake-jsonline-codex-"));
const packageRoot = path.join(root, "node_modules", "@openai", "codex");
const command = path.join(packageRoot, "bin", "codex.js");
const native = path.join(root, "node_modules", "@openai", "codex-linux-x64", "vendor", "x86_64-unknown-linux-musl", "codex", "codex");
await mkdir(path.dirname(command), { recursive: true });
await mkdir(path.dirname(native), { recursive: true });
await writeFile(path.join(packageRoot, "package.json"), JSON.stringify({ name: "@openai/codex", version: "0.128.0" }), "utf8");
await writeFile(command, "#!/usr/bin/env node\nconsole.log('node-wrapper-should-not-run');\n", "utf8");
await writeFile(native, [
"#!/usr/bin/env node",
"const readline = require('node:readline');",
"let turnRequestId = null;",
"let approvalDecision = null;",
"const rl = readline.createInterface({ input: process.stdin });",
"function send(payload) { process.stdout.write(JSON.stringify(payload) + '\\n'); }",
"rl.on('line', (line) => {",
" if (!line.trim()) return;",
" const payload = JSON.parse(line);",
" if (payload.id === 'approval-test' && !payload.method) {",
" approvalDecision = payload.result && payload.result.decision;",
" send({ method: 'item/completed', params: { item: { id: 'cmd_jsonline_test', type: 'commandExecution', status: 'completed', exitCode: 0, command: 'echo approval', stdout: 'approval ok' } } });",
" send({ id: turnRequestId, result: { turn: { id: 'turn_jsonline_test' }, approvalDecision, spawnedFrom: 'native' } });",
" return;",
" }",
" if (payload.method === 'initialize') { send({ id: payload.id, result: { initialized: true } }); return; }",
" if (payload.method === 'initialized') return;",
" if (payload.method === 'thread/start') {",
" send({ method: 'thread/started', params: { thread: { id: 'thread_jsonline_test' } } });",
" send({ id: payload.id, result: { thread: { id: 'thread_jsonline_test' } } });",
" return;",
" }",
" if (payload.method === 'turn/start') {",
" turnRequestId = payload.id;",
" send({ method: 'turn/started', params: { turn: { id: 'turn_jsonline_test' } } });",
" send({ id: 'approval-test', method: 'item/commandExecution/requestApproval', params: { itemId: 'cmd_jsonline_test', availableDecisions: ['approved_for_session', 'decline'] } });",
" return;",
" }",
" send({ id: payload.id, error: { code: -32601, message: 'unknown method' } });",
"});",
"setTimeout(() => {}, 30000);"
].join("\n"), "utf8");
await chmod(command, 0o755);
await chmod(native, 0o755);
return { root, command };
}
function createFakeAppServerClient({ rpcCalls }) {
let notificationHandler = null;
let turn = 0;
+73
View File
@@ -127,6 +127,79 @@ test("hwlab-cli client runtime routes reports UniDesk pod passthrough route", as
assert.equal(result.payload.discovery.readyRouteCount, 1);
});
test("hwlab-cli client gateway sessions uses direct Cloud API short connection", async () => {
const calls: any[] = [];
const result = await runHwlabCli([
"client",
"gateway",
"sessions",
"--api-base-url",
"http://api.test",
"--no-auth",
"--full"
], {
fetchImpl: async (url, init) => {
calls.push({ url: String(url), init });
return new Response(JSON.stringify({ ok: true, sessions: [{ gatewaySessionId: "gws_D601_F103", inflightCount: 0 }] }), { status: 200 });
}
});
assert.equal(result.exitCode, 0);
assert.equal(calls[0].url, "http://api.test/v1/gateway/sessions");
assert.equal(calls[0].init.headers.cookie, undefined);
assert.equal(result.payload.action, "client.gateway.sessions");
assert.equal(result.payload.baseUrl, "http://api.test");
assert.equal(result.payload.body.sessions[0].gatewaySessionId, "gws_D601_F103");
});
test("hwlab-cli client gateway pressure reports byte/truncation visibility", async () => {
const calls: any[] = [];
const result = await runHwlabCli([
"client",
"gateway",
"pressure",
"--api-base-url",
"http://api.test",
"--gateway-session-id",
"gws_test",
"--large-bytes",
"8192",
"--parallel",
"2",
"--request-timeout-ms",
"10000",
"--timeout-scenario-ms",
"500"
], {
fetchImpl: async (url, init) => {
const callIndex = calls.length;
const body = JSON.parse(String(init?.body ?? "{}"));
calls.push({ url: String(url), init, body });
const dispatch: any = { dispatchStatus: "succeeded", shellExecuted: true, exitCode: 0, stdout: "", stderr: "" };
if (callIndex === 0) dispatch.stdout = "hwlab-gateway-pressure-small-ok\r\n";
else if (callIndex === 1 || callIndex === 2) Object.assign(dispatch, { stdout: "O".repeat(65536), stdoutTruncated: true });
else if (callIndex === 3) Object.assign(dispatch, { stderr: "E".repeat(65536), stderrTruncated: true });
else if (callIndex === 4) Object.assign(dispatch, { dispatchStatus: "timed_out", timedOut: true, exitCode: null });
else dispatch.stdout = `parallel-${callIndex - 4}-ok\r\n`;
return new Response(JSON.stringify({ ok: true, result: { status: "succeeded", dispatch } }), { status: 200 });
}
});
assert.equal(result.exitCode, 0);
assert.equal(result.payload.action, "client.gateway.pressure");
assert.equal(result.payload.status, "succeeded");
assert.equal(result.payload.scenarioCount, 7);
assert.equal(result.payload.failedCount, 0);
assert.equal(calls[0].url, "http://api.test/v1/rpc/hardware.invoke.shell");
assert.equal(calls[0].init.headers.cookie, undefined);
assert.equal(calls[0].body.gatewaySessionId, "gws_test");
assert.match(calls[1].body.input.command, /-EncodedCommand/u);
assert.equal(result.payload.results.find((item: any) => item.name === "large-stdout").stdoutTruncated, true);
assert.equal(result.payload.results.find((item: any) => item.name === "stderr-flood").stderrTruncated, true);
assert.equal(result.payload.results.find((item: any) => item.name === "timeout-kill").reason, "timeout_structured");
assert.match(result.payload.results.find((item: any) => item.name === "large-stdout").stdoutSha256, /^[a-f0-9]{64}$/u);
});
test("hwlab-cli client agent send submits async and polls result", async () => {
const calls: any[] = [];
const result = await runHwlabCli([
+214
View File
@@ -1,3 +1,4 @@
import { createHash } from "node:crypto";
import { chmod, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import path from "node:path";
@@ -9,6 +10,10 @@ const DEFAULT_AGENT_TIMEOUT_MS = 120000;
const DEFAULT_POLL_INTERVAL_MS = 1000;
const HARNESS_WAIT_DEFAULT_TIMEOUT_MS = 50000;
const HARNESS_WAIT_MAX_TIMEOUT_MS = 50000;
const DEFAULT_GATEWAY_SESSION_ID = "gws_D601_F103";
const DEFAULT_GATEWAY_RESOURCE_ID = "res_windows_host";
const DEFAULT_GATEWAY_CAPABILITY_ID = "cap_windows_cmd_exec";
const DEFAULT_GATEWAY_PROJECT_ID = "prj_mvp_topology";
type EnvLike = Record<string, string | undefined>;
type ParsedArgs = Record<string, unknown> & { _: string[] };
@@ -58,6 +63,7 @@ async function clientCommand(context: any) {
if (group === "auth") return authCommand(next);
if (group === "device-pods" || group === "device-pod") return devicePodsCommand(next);
if (group === "runtime") return runtimeCommand(next);
if (group === "gateway") return gatewayCommand(next);
if (group === "agent") return agentCommand(next);
if (["harness", "harness-ops", "harness-opt"].includes(group)) return harnessCommand(next);
if (group === "workbench") return workbenchCommand(next);
@@ -83,6 +89,8 @@ function help() {
"hwlab-cli client device-pods list",
"hwlab-cli client device-pods status device-pod-71-freq",
"hwlab-cli client runtime routes",
"hwlab-cli client gateway sessions",
"hwlab-cli client gateway pressure --gateway-session-id gws_D601_F103 --api-base-url http://API:19667",
"hwlab-cli client workbench summary --pod-id device-pod-71-freq",
"hwlab-cli client request GET /v1/access/status [--full]",
"hwlab-cli client rpc system.health [--full]",
@@ -281,6 +289,207 @@ async function devicePodProbe(context: any, podId: string) {
return ok("client.device-pods.probe", { status: failed ? "degraded" : "succeeded", devicePodId: podId, probes }, failed ? "degraded" : "succeeded");
}
async function gatewayCommand(context: any) {
const subcommand = context.rest[0] || "sessions";
if (["help", "--help", "-h"].includes(subcommand)) return gatewayHelp();
const apiContext = gatewayApiContext(context);
if (subcommand === "sessions") {
const response = await requestJson({
...apiContext,
method: "GET",
path: "/v1/gateway/sessions",
auth: context.parsed.noAuth === true ? false : context.parsed.auth === true
});
return responsePayload("client.gateway.sessions", response, apiContext, {
route: route("GET", "/v1/gateway/sessions"),
body: responseBodyForCli(response.body, context.parsed)
});
}
if (subcommand === "invoke") {
const command = requiredText(context.parsed.command ?? context.rest.slice(1).join(" "), "command");
const result = await invokeGatewayShell(apiContext, {
name: "invoke",
command,
timeoutMs: numberOption(context.parsed.commandTimeoutMs) ?? numberOption(context.parsed.timeoutMs) ?? 30000
});
return ok("client.gateway.invoke", { status: result.ok ? "succeeded" : "failed", result }, result.ok ? "succeeded" : "failed");
}
if (subcommand === "pressure") return gatewayPressure(apiContext);
throw cliError("unsupported_gateway_command", `unsupported gateway command: ${subcommand}`, { subcommand });
}
function gatewayHelp() {
return ok("client.gateway.help", {
serviceRuntime: false,
imagePublished: false,
commands: [
"sessions [--api-base-url URL|--base-url URL]",
"invoke --command CMD [--gateway-session-id ID] [--api-base-url URL]",
"pressure [--gateway-session-id ID] [--api-base-url URL] [--large-bytes N] [--parallel N]"
]
});
}
function gatewayApiContext(context: any) {
const apiBaseUrl = text(
context.parsed.apiBaseUrl ??
context.parsed.cloudApiUrl ??
context.env.HWLAB_CLOUD_API_URL ??
context.env.HWLAB_CLOUD_API_BASE_URL
);
if (!apiBaseUrl) return context;
return { ...context, parsed: { ...context.parsed, baseUrl: apiBaseUrl } };
}
async function gatewayPressure(context: any) {
const largeBytes = boundedInteger(context.parsed.largeBytes, 131072, 4096, 1048576);
const parallel = boundedInteger(context.parsed.parallel, 8, 0, 32);
const requestTimeoutMs = boundedInteger(context.parsed.requestTimeoutMs, 45000, 1000, 180000);
const timeoutScenarioMs = boundedInteger(context.parsed.timeoutScenarioMs, 1000, 250, 30000);
const startedAt = Date.now();
const scenarios = gatewayPressureScenarios({ largeBytes, timeoutScenarioMs });
const results = [];
for (const scenario of scenarios) {
results.push(await invokeGatewayShell(context, { ...scenario, requestTimeoutMs }));
}
if (parallel > 0) {
const parallelResults = await Promise.all(Array.from({ length: parallel }, (_, index) => invokeGatewayShell(context, {
name: `parallel-${index + 1}`,
kind: "parallel",
command: powershellEncodedCommand(`Start-Sleep -Milliseconds 3500; Write-Output 'parallel-${index + 1}-ok'`),
timeoutMs: 12000,
requestTimeoutMs,
allowGatewayBusy: true,
marker: `parallel-${index + 1}-ok`
})));
results.push(...parallelResults);
}
const failures = results.filter((item) => item.ok !== true);
return ok("client.gateway.pressure", {
status: failures.length > 0 ? "failed" : "succeeded",
baseUrl: baseUrl(context.parsed, context.env),
gatewaySessionId: gatewaySelector(context).gatewaySessionId,
scenarioCount: results.length,
failedCount: failures.length,
elapsedMs: Date.now() - startedAt,
config: { largeBytes, parallel, requestTimeoutMs, timeoutScenarioMs },
results,
recommendation: failures.length > 0
? "Gateway transport pressure found failures; inspect failed result.request/result.rpcSummary before continuing device-pod work."
: "Gateway transport pressure passed: large stdout/stderr and over-capacity parallel requests returned bounded structured results without request blackholes."
}, failures.length > 0 ? "failed" : "succeeded");
}
function gatewayPressureScenarios({ largeBytes, timeoutScenarioMs }: { largeBytes: number; timeoutScenarioMs: number }) {
return [
{ name: "small-stdout", kind: "small", command: "echo hwlab-gateway-pressure-small-ok", timeoutMs: 10000, marker: "hwlab-gateway-pressure-small-ok" },
{ name: "large-stdout", kind: "large-stdout", command: powershellEncodedCommand(`[Console]::Out.Write(('O' * ${largeBytes}))`), timeoutMs: 20000, expectStdoutTruncated: true },
{ name: "long-single-line", kind: "long-line", command: powershellEncodedCommand(`[Console]::Out.Write(('L' * ${largeBytes}))`), timeoutMs: 20000, expectStdoutTruncated: true },
{ name: "stderr-flood", kind: "stderr", command: powershellEncodedCommand(`[Console]::Error.Write(('E' * ${largeBytes}))`), timeoutMs: 20000, expectStderrTruncated: true },
{ name: "timeout-kill", kind: "timeout", command: powershellEncodedCommand("Start-Sleep -Seconds 5; Write-Output 'timeout-missed'"), timeoutMs: timeoutScenarioMs, expectTimedOut: true }
];
}
function powershellEncodedCommand(script: string) {
const encoded = Buffer.from(script, "utf16le").toString("base64");
return `powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -EncodedCommand ${encoded}`;
}
async function invokeGatewayShell(context: any, scenario: any) {
const selector = gatewaySelector(context);
const traceId = text(context.parsed.traceId) || makeId(`trc_gateway_${scenario.name.replace(/[^a-z0-9]+/giu, "_")}`);
const requestId = text(context.parsed.id) || makeId(`req_gateway_${scenario.name.replace(/[^a-z0-9]+/giu, "_")}`);
const body = {
gatewaySessionId: selector.gatewaySessionId,
resourceId: selector.resourceId,
capabilityId: selector.capabilityId,
projectId: selector.projectId,
input: {
command: scenario.command,
timeoutMs: scenario.timeoutMs
}
};
const startedAt = Date.now();
const response = await requestJson({
...context,
method: "POST",
path: "/v1/rpc/hardware.invoke.shell",
body,
auth: context.parsed.auth === true,
timeoutMs: scenario.requestTimeoutMs ?? numberOption(context.parsed.requestTimeoutMs) ?? DEFAULT_TIMEOUT_MS,
extraHeaders: {
"x-trace-id": traceId,
"x-request-id": requestId
}
});
return summarizeGatewayScenario({ scenario, response, traceId, requestId, startedAt });
}
function gatewaySelector(context: any) {
return {
gatewaySessionId: text(context.parsed.gatewaySessionId) || DEFAULT_GATEWAY_SESSION_ID,
resourceId: text(context.parsed.resourceId) || DEFAULT_GATEWAY_RESOURCE_ID,
capabilityId: text(context.parsed.capabilityId) || DEFAULT_GATEWAY_CAPABILITY_ID,
projectId: text(context.parsed.projectId) || DEFAULT_GATEWAY_PROJECT_ID
};
}
function summarizeGatewayScenario({ scenario, response, traceId, requestId, startedAt }: any) {
const rpc = response.body && typeof response.body === "object" ? response.body : {};
const result = rpc.result && typeof rpc.result === "object" ? rpc.result : {};
const dispatch = result.dispatch && typeof result.dispatch === "object" ? result.dispatch : {};
const stdout = String(dispatch.stdout ?? "");
const stderr = String(dispatch.stderr ?? "");
const gatewayBusy = dispatch.error?.data?.reason === "gateway_busy" || /gateway is busy/iu.test(String(dispatch.error?.message ?? dispatch.message ?? ""));
const okByKind = gatewayScenarioPass({ scenario, response, dispatch, stdout, stderr, gatewayBusy });
return pruneUndefined({
name: scenario.name,
kind: scenario.kind,
ok: okByKind.ok,
reason: okByKind.reason,
httpStatus: response.status,
elapsedMs: Date.now() - startedAt,
requestElapsedMs: response.elapsedMs,
traceId,
requestId,
operationId: result.operationId ?? dispatch.operationId,
rpcStatus: result.status,
dispatchStatus: dispatch.dispatchStatus,
shellExecuted: dispatch.shellExecuted,
exitCode: dispatch.exitCode,
timedOut: dispatch.timedOut,
gatewayBusy,
stdoutBytes: Buffer.byteLength(stdout, "utf8"),
stderrBytes: Buffer.byteLength(stderr, "utf8"),
stdoutTruncated: dispatch.stdoutTruncated,
stderrTruncated: dispatch.stderrTruncated,
stdoutSha256: stdout ? sha256(stdout) : undefined,
stderrSha256: stderr ? sha256(stderr) : undefined,
stdoutPreview: stdout ? preview(stdout, 160) : undefined,
stderrPreview: stderr ? preview(stderr, 160) : undefined,
error: rpc.error ?? dispatch.error,
message: dispatch.message,
commandPreview: preview(scenario.command, 220)
});
}
function gatewayScenarioPass({ scenario, response, dispatch, stdout, stderr, gatewayBusy }: any) {
if (!isHttpSuccess(response)) return { ok: false, reason: `http_${response.status}` };
if (scenario.allowGatewayBusy && gatewayBusy) return { ok: true, reason: "structured_gateway_busy" };
if (scenario.expectTimedOut) {
return dispatch.timedOut === true || dispatch.dispatchStatus === "timed_out"
? { ok: true, reason: "timeout_structured" }
: { ok: false, reason: "timeout_not_observed" };
}
if (scenario.expectStdoutTruncated && dispatch.stdoutTruncated !== true) return { ok: false, reason: "stdout_not_truncated" };
if (scenario.expectStderrTruncated && dispatch.stderrTruncated !== true) return { ok: false, reason: "stderr_not_truncated" };
if (scenario.marker && !stdout.includes(scenario.marker)) return { ok: false, reason: "marker_missing" };
if (dispatch.exitCode !== 0) return { ok: false, reason: `exit_${dispatch.exitCode}` };
if (scenario.expectStdoutTruncated && Buffer.byteLength(stdout, "utf8") === 0) return { ok: false, reason: "stdout_empty" };
if (scenario.expectStderrTruncated && Buffer.byteLength(stderr, "utf8") === 0) return { ok: false, reason: "stderr_empty" };
return { ok: true, reason: "completed" };
}
async function agentCommand(context: any) {
const subcommand = context.rest[0] || "send";
if (subcommand === "send") return agentSend(context);
@@ -1314,9 +1523,14 @@ function text(value: unknown) { return String(value ?? "").trim(); }
function textOrNull(value: unknown) { const result = text(value); return result || null; }
function requiredText(value: unknown, field: string) { const result = text(value); if (!result) throw cliError("missing_required_value", `${field} is required`, { field }); return result; }
function numberOption(value: unknown) { const parsed = Number.parseInt(String(value ?? ""), 10); return Number.isFinite(parsed) ? parsed : undefined; }
function boundedInteger(value: unknown, fallback: number, min: number, max: number) {
const parsed = numberOption(value) ?? fallback;
return Math.min(max, Math.max(min, parsed));
}
function clean<T extends Record<string, any>>(value: T): T { return Object.fromEntries(Object.entries(value).filter(([, item]) => item !== undefined && item !== "" && item !== false && item !== null)) as T; }
function pruneUndefined<T extends Record<string, any>>(value: T): T { return Object.fromEntries(Object.entries(value).filter(([, item]) => item !== undefined)) as T; }
function parseJson(value: string) { if (!value) return null; try { return JSON.parse(value); } catch { return { rawText: value.slice(0, 2000), parseError: true }; } }
function sha256(value: string) { return createHash("sha256").update(value).digest("hex"); }
function makeId(prefix: string) { return `${prefix}_${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 10)}`; }
function wait(ms: number) { return new Promise<void>((resolve) => setTimeout(resolve, ms)); }
function requiredTraceId(value: unknown) {