feat: publish AgentRun Kafka event streams

This commit is contained in:
root
2026-07-09 19:25:31 +02:00
parent 4466ba6eb6
commit 120360090c
8 changed files with 560 additions and 65 deletions
+2
View File
@@ -11,6 +11,7 @@
- 校验并持久化 `tenantId``projectId``workspaceRef``providerId``backendProfile``executionPolicy``traceSink`
- 执行最小 tenant policy boundary:只做 schema、allowlist、idempotency、secret scope 和 executionPolicy 范围检查;不内建 UniDesk/HWLAB 的业务授权。
- 使用 Postgres 保存 runs、commands、events、runners、backends、leases 和 migration ledger。
- 以 Kafka topic `agentrun.event.v1` 发布 manager durable event 流,作为 HWLAB 转换前的 AgentRun event 权威源;Postgres 仍是 manager 在线读写的 durable store。
- 输出结构化 health/readiness、failureKind、redacted SecretRef 和 trace correlation。
- 可观测性只能定位和验证状态,不能替代缺失能力实现;如果 HWLAB canary 需要的 final reply、command result、runner 多 turn、SessionRef 或 cancel 能力缺失,manager 必须补 durable API/状态机,而不是只补 trace 文案。
@@ -23,6 +24,7 @@
- Migration 必须在 readiness 前完成或显式 fail fast,不能以空 schema 静默启动。
- Provider credential、Codex auth/config、Postgres DSN 明文不进数据库、event、trace、日志或 CLI 输出。
- Manager 可以保存 SecretRef 和 credential source reference,但不得读取 provider Secret 值后存库。
- Kafka 发布在本阶段是 produce-only 可见性链路:manager 将 `createRun``createCommand``appendEvent``finishCommand``finishRun` 等 durable store 写入镜像到 `agentrun.event.v1`schema 为 `agentrun.event.v1`。不得让 Kafka consumer 反向驱动 manager 状态,也不得把 HWLAB event 或 Codex stdio raw 混入该 topic。
## API 接口说明
+6
View File
@@ -20,6 +20,12 @@ codex app-server --listen stdio://
Adapter 通过 stdin 写入换行分隔 JSON-RPC 请求,通过 stdout 逐行读取 JSON-RPC response 和 notificationstderr 只作为有界诊断日志。最小请求序列是 `initialize``thread/start``thread/resume``turn/start`response 中必须提取 thread/turn identitynotification 和后续输出必须归一化为 `backend_status``assistant_message``tool_call``command_output``error``terminal_status` events。运行中 steer 使用同一 app-server 进程的 `turn/steer` JSON-RPC 方法,参数为 `threadId``expectedTurnId` 和文本 `input` 数组;取消/中断使用 `turn/interrupt`,参数为 `threadId``turnId`。已有 `SessionRef.threadId` 时只能执行 Codex stdio 原生 `thread/resume` 后接 `turn/start`;当 `thread/resume` 返回 `no rollout found for thread id` 或任何其他协议错误时,adapter 必须输出 `thread-resume-failed` 并终止当前 turn。adapter 不得启动替代 `thread/start`、拼接历史 prompt、回写新 threadId 或用其他上下文模拟继续会话。
### Codex stdio Kafka 原始流
AgentRun v0.2 的 Codex stdio 原始流权威源是 Kafka topic `codex-stdio.raw.v1`。该流由 runner 内的 `CodexStdioClient` 在真实 stdio 边界发布,覆盖写入 child stdin 的 JSON-RPC line、从 child stdout 读取的每一行、stderr chunk 和 process close lifecycle。消息 schema 固定为 `codex-stdio.raw.v1`,携带 `runId``commandId``sessionId``threadId``runnerJobId``podName`、direction、method、phase、raw bytes/hash 和原始文本/JSON。Kafka value 是分析用的裸 stdio 事实,生产前不得做 redaction;CLI、日志和 UI 默认只显示 hash/bytes,只有显式 raw/value 查询才展示 value。
启用条件来自 runner env`AGENTRUN_KAFKA_ENABLED=true``AGENTRUN_KAFKA_STDIO_PRODUCE_ENABLED=true`,并声明 `AGENTRUN_KAFKA_BOOTSTRAP_SERVERS``AGENTRUN_KAFKA_STDIO_TOPIC``AGENTRUN_KAFKA_STDIO_CLIENT_ID`。这些 env 由 manager Deployment 通过 runner Job manifest 透传,配置 authority 在 UniDesk `config/agentrun.yaml`,不由 HWLAB workbench 或 AgentRun service repo 的 legacy `deploy.json` 决定。
若 run 的 `ResourceBundleRef` 包含 `promptRefs` 或 gitbundle `.agents/skills`Codex adapter 只能消费 runner 已装配好的有界 `initialPrompt`、skill summary 和 skill registry path。对新 threadadapter 在首个 `turn/start` 中把 `initialPrompt` 和 skill facts 放在用户 message 之前;对 `thread/resume`adapter 不重复注入 `initialPrompt`,只发送当前 command 的用户 message。当前 Codex app-server 若只有 `input: [{ type: "text", text }]`,允许使用结构化文本前缀承载 initial prompt;若后续 app-server 支持 developer/runtime instruction item,优先映射到该标准 item。无论哪种 wire shapeevents 只记录 prompt/skill 的 path/hash/bytes/injected 状态,不输出全文。
不得把以下路径作为 `v0.1` Codex stdio backend 的正式实现或综合联调通过证据:直接 Responses HTTP 代理、OpenAI SDK wrapper、`codex exec` 一次性命令输出、fake provider、固定文本回复、只读 shortcut 或本地 shell 模拟。裸 HTTP 或 `codex exec --json` 可以作为 provider/upstream 诊断,但最终通过必须来自 app-server stdio turn。
+3
View File
@@ -39,6 +39,8 @@ CLI 官方 TypeScript 入口固定为 `scripts/agentrun-cli.ts`。在 G14 非交
./scripts/agentrun runs events <runId> --after-seq <n> --limit <n> [--summary|--tail-summary] [--tail <n>] [--summary-chars <n>] [--format json|tsv]
./scripts/agentrun runs result <runId> [--command-id <commandId>]
./scripts/agentrun runs cancel <runId> [--reason <text>]
./scripts/agentrun kafka tail [agentrun|stdio] [--topic <topic>] [--brokers <host:port,...>] [--limit <n>] [--timeout-ms <ms>] [--run-id <id>] [--session-id <id>] [--command-id <id>] [--trace-id <id>] [--values]
./scripts/agentrun kafka events [agentrun|stdio] [--topic <topic>] [--brokers <host:port,...>] [--limit <n>] [--timeout-ms <ms>]
./scripts/agentrun commands create <runId> --type turn|steer|interrupt --json-stdin|--json-file <payload.json>
./scripts/agentrun commands show <commandId> --run-id <runId>
./scripts/agentrun commands result <commandId> --run-id <runId>
@@ -100,6 +102,7 @@ CLI 官方 TypeScript 入口固定为 `scripts/agentrun-cli.ts`。在 G14 非交
- `events --summary` 返回低噪声 JSON summary,单条至少包含 `seq``type``method``status``command``text``exitCode``durationMs``outputTruncated``outputBytes``outputSummary``summary`;summary 文本必须压缩换行并继续沿用 redaction,不能泄漏 Secret/env value。
- `events --tail <n>``--after-seq`/`--limit` 组合时,只显示本次分页结果最后 `n` 条 summary`--tail-summary` 是默认取最多 20 条 tail 的低噪声快捷入口。
- `events --format tsv` 只在显式请求时输出 TSV 文本,用于人工现场跟踪和上层 trace 压缩;默认 JSON envelope 行为不得改变。
- `kafka tail/events` 是 AgentRun Kafka 事件流的受控查询入口,默认读取 `agentrun.event.v1`,传 `stdio` 时读取 `codex-stdio.raw.v1`;默认只输出 topic/offset/key hash/value hash/bytes/schema/eventType/runId/sessionId/commandId/direction/method,不输出 Kafka value。只有显式 `--values``--raw-values` 才返回有界 value 预览,用于针对性调试裸 stdio 或 manager event。
- `server start` 默认以本地后台进程启动 manager,立刻返回 pid、pidFile、logPath、baseUrl 和后续 `status/stop` 命令;只有显式 `--foreground` 才允许占用当前终端。启动前必须检查 pidFile 与端口占用,避免同一端口上堆叠临时 manager。
- `server status` 必须同时返回本地 pid/port/logPath 状态和 `/health/readiness` 结果;即使 readiness 失败,也要输出结构化 JSON 和 failure details。
- `server logs` 必须返回有界日志尾部、bytes、truncated 和 logPath;找不到日志文件时也必须返回非空 JSON。
+35
View File
@@ -16,6 +16,7 @@ import type { RunnerOnceOptions } from "../../src/runner/run-once.js";
import { backendProfileSpec, isBackendProfile } from "../../src/common/backend-profiles.js";
import { outputBytesFromPayload, outputTruncatedFromPayload } from "../../src/common/output.js";
import { redactJson, redactText } from "../../src/common/redaction.js";
import { tailAgentRunKafka } from "../../src/common/kafka-events.js";
interface ParsedArgs {
positional: string[];
@@ -52,6 +53,7 @@ async function dispatch(args: ParsedArgs): Promise<CliResult> {
if (args.flags.get("help") === true) return help(args, group);
if (command === "help" || command === "--help") return help(args, group);
if (group === "manager" && (command === "url" || command === "resolve-url" || command === "status")) return managerEndpoint(args);
if (group === "kafka" && (command === "tail" || command === "events")) return kafkaTail(args);
if (group === "server" && command === "start") return startServer(args);
if (group === "server" && command === "status") return serverStatus(args);
if (group === "server" && command === "logs") return serverLogs(args);
@@ -159,6 +161,29 @@ async function listRunnerJobs(args: ParsedArgs): Promise<JsonValue> {
return client(args).get(`/api/v1/runs/${encodeURIComponent(runId)}/runner-jobs${commandId ? `?commandId=${encodeURIComponent(commandId)}` : ""}`);
}
async function kafkaTail(args: ParsedArgs): Promise<JsonValue> {
const stream = optionalFlag(args, "stream") ?? args.positional[2] ?? null;
const topic = optionalFlag(args, "topic");
const limit = optionalNumberFlag(args, "limit");
const timeoutMs = optionalNumberFlag(args, "timeout-ms");
return await tailAgentRunKafka({
brokers: optionalFlag(args, "brokers") ?? undefined,
stream: stream ?? undefined,
topic: topic ?? undefined,
groupId: optionalFlag(args, "group-id") ?? undefined,
limit: limit ?? undefined,
timeoutMs: timeoutMs ?? undefined,
fromBeginning: args.flags.get("latest") === true ? false : true,
values: args.flags.get("values") === true || args.flags.get("raw-values") === true,
filter: {
traceId: optionalFlag(args, "trace-id"),
sessionId: optionalFlag(args, "session-id"),
runId: optionalFlag(args, "run-id"),
commandId: optionalFlag(args, "command-id"),
},
}) as JsonValue;
}
async function showRun(args: ParsedArgs, runId: string): Promise<JsonValue> {
const run = await client(args).get(`/api/v1/runs/${encodeURIComponent(runId)}`);
if (wantsExpandedOutput(args)) return run;
@@ -2039,6 +2064,14 @@ function optionalFlag(args: ParsedArgs, name: string): string | null {
return typeof value === "string" && value.length > 0 ? value : null;
}
function optionalNumberFlag(args: ParsedArgs, name: string): number | null {
const value = optionalFlag(args, name);
if (value === null) return null;
const parsed = Number(value);
if (!Number.isFinite(parsed)) throw new AgentRunError("schema-invalid", `--${name} must be a finite number`, { httpStatus: 2 });
return parsed;
}
function cancelBody(args: ParsedArgs): JsonRecord {
const reason = optionalFlag(args, "reason");
return reason ? { reason } : {};
@@ -2051,6 +2084,8 @@ function help(args: ParsedArgs, group?: string): JsonRecord {
"runs events <runId> --after-seq <n> --limit <n> [--summary|--tail-summary] [--tail <n>] [--summary-chars <n>] [--format json|tsv]",
"runs result <runId> [--command-id <commandId>]",
"runs cancel <runId> [--reason <text>]",
"kafka tail [agentrun|stdio] [--topic <topic>] [--brokers <host:port,...>] [--limit <n>] [--timeout-ms <ms>] [--run-id <id>] [--session-id <id>] [--command-id <id>] [--trace-id <id>] [--values]",
"kafka events [agentrun|stdio] [--topic <topic>] [--brokers <host:port,...>] [--limit <n>] [--timeout-ms <ms>]",
"sessions ps [--state default|running|unread|terminal|idle|all] [--profile codex|deepseek|minimax-m3|dsflash-go|<dynamic-profile>|M3] [--reader-id <reader>]",
"sessions create [sessionId] [--profile codex|deepseek|minimax-m3|dsflash-go|<dynamic-profile>|M3] [--expires-in-days <n>]",
"sessions storage <sessionId>",
+160 -6
View File
@@ -9,6 +9,7 @@ import { redactJson, redactText } from "../common/redaction.js";
import { backendProfileSpec } from "../common/backend-profiles.js";
import { boundedTextSummary, commandOutputPayload } from "../common/output.js";
import { emitAgentRunOtelSpan } from "../common/otel-trace.js";
import { agentRunKafkaConfig, publishAgentRunKafkaMessageBestEffort, rawFrameValue, safeText, sha256Hex, type AgentRunKafkaConfig } from "../common/kafka-events.js";
const codexProtocol = "codex-app-server-jsonrpc-stdio";
const defaultCodexArgs = ["app-server", "--listen", "stdio://"];
@@ -81,6 +82,28 @@ interface CodexLifecycleOtelContext {
attributes: JsonRecord;
}
interface CodexStdioKafkaContext extends JsonRecord {
runId: string | null;
commandId: string | null;
attemptId: string | null;
runnerId: string | null;
runnerJobId: string | null;
jobName: string | null;
podName: string | null;
sessionId: string | null;
conversationId: string | null;
threadId: string | null;
backendProfile: string | null;
sourceCommit: string | null;
valuesPrinted: false;
}
interface CodexStdioKafkaPublisher {
config: AgentRunKafkaConfig;
context: CodexStdioKafkaContext;
source: string;
}
export interface CodexActiveTurnControl {
threadId: string;
turnId: string;
@@ -149,6 +172,7 @@ class CodexStdioFailure extends Error {
export class CodexStdioClient {
private readonly child: ChildProcessWithoutNullStreams;
private stdioKafka: CodexStdioKafkaPublisher | null;
private readonly pending = new Map<number, PendingRequest>();
private stderrTailBuffer = Buffer.alloc(0);
private stderrBytes = 0;
@@ -158,8 +182,9 @@ export class CodexStdioClient {
readonly closedPromise: Promise<CodexStdioCloseInfo>;
private closeResolve!: (value: CodexStdioCloseInfo) => void;
constructor(options: { command?: string; args?: string[]; cwd: string; env?: NodeJS.ProcessEnv; onNotification: (message: JsonRecord) => void }) {
constructor(options: { command?: string; args?: string[]; cwd: string; env?: NodeJS.ProcessEnv; onNotification: (message: JsonRecord) => void; stdioKafka?: CodexStdioKafkaPublisher | null }) {
this.closedPromise = new Promise((resolve) => { this.closeResolve = resolve; });
this.stdioKafka = options.stdioKafka ?? null;
const command = options.command ?? "codex";
const args = options.args ?? defaultCodexArgs;
try {
@@ -172,7 +197,14 @@ export class CodexStdioClient {
} catch (error) {
throw spawnFailure(command, error);
}
this.child.stderr.on("data", (chunk: Buffer) => this.appendStderr(chunk));
this.child.stderr.on("data", (chunk: Buffer) => {
this.appendStderr(chunk);
publishCodexStdioFrame(this.stdioKafka, "stderr", {
rawText: chunk.toString("utf8"),
method: "process.stderr",
phase: "stderr:data",
});
});
const rl = readline.createInterface({ input: this.child.stdout, crlfDelay: Infinity });
void this.readLines(rl, options.onNotification);
this.child.on("close", (code, signal) => this.handleClose(code, signal));
@@ -183,6 +215,10 @@ export class CodexStdioClient {
return this.closed;
}
setStdioKafka(publisher: CodexStdioKafkaPublisher | null): void {
this.stdioKafka = publisher;
}
request(method: string, params: JsonRecord, timeoutMs = requestTimeoutCapMs): Promise<unknown> {
if (this.closed) return Promise.reject(this.closeFailure ?? new CodexStdioFailure("backend-failed", "codex app-server is closed", `request:${method}`));
const id = this.nextId++;
@@ -193,7 +229,15 @@ export class CodexStdioClient {
this.rejectRequest(id, new CodexStdioFailure("backend-timeout", `Codex stdio request ${method} timed out after ${effectiveTimeoutMs}ms`, `request:${method}`, { method, timeoutMs: effectiveTimeoutMs }));
}, effectiveTimeoutMs);
this.pending.set(id, { method, timer, resolve, reject });
this.child.stdin.write(`${JSON.stringify(message)}\n`, "utf8", (error: Error | null | undefined) => {
const rawLine = `${JSON.stringify(message)}\n`;
publishCodexStdioFrame(this.stdioKafka, "stdin", {
rawText: rawLine,
rawJson: message,
method,
rpcId: id,
phase: "client.request",
});
this.child.stdin.write(rawLine, "utf8", (error: Error | null | undefined) => {
if (!error) return;
this.rejectRequest(id, new CodexStdioFailure("backend-failed", `failed to write Codex stdio request ${method}: ${error.message}`, `request:${method}`, { method }));
});
@@ -202,7 +246,15 @@ export class CodexStdioClient {
notify(method: string, params: JsonRecord = {}): void {
if (this.closed) return;
this.child.stdin.write(`${JSON.stringify({ method, params })}\n`, "utf8", () => undefined);
const message = { method, params };
const rawLine = `${JSON.stringify(message)}\n`;
publishCodexStdioFrame(this.stdioKafka, "stdin", {
rawText: rawLine,
rawJson: message,
method,
phase: "client.notify",
});
this.child.stdin.write(rawLine, "utf8", () => undefined);
}
stop(): void {
@@ -241,9 +293,21 @@ export class CodexStdioClient {
try {
message = JSON.parse(trimmed) as JsonRecord;
} catch {
publishCodexStdioFrame(this.stdioKafka, "stdout", {
rawText: line,
method: "stdout.parse-failed",
phase: "stdout:parse",
});
this.handleProtocolFailure(new CodexStdioFailure("backend-json-parse-error", "codex app-server emitted invalid JSON on stdout", "stdout:parse", { linePreview: redactText(trimmed.slice(0, 800)), lineChars: trimmed.length }));
break;
}
publishCodexStdioFrame(this.stdioKafka, "stdout", {
rawText: line,
rawJson: message,
method: typeof message.method === "string" ? message.method : "response",
rpcId: typeof message.id === "number" ? message.id : null,
phase: "stdout:message",
});
this.handleMessage(message, onNotification);
}
} catch (error) {
@@ -287,10 +351,28 @@ export class CodexStdioClient {
private handleServerRequest(id: number, method: string): void {
if (method === "item/commandExecution/requestApproval" || method === "item/fileChange/requestApproval") {
this.child.stdin.write(`${JSON.stringify({ id, result: { decision: "decline" } })}\n`);
const message = { id, result: { decision: "decline" } };
const rawLine = `${JSON.stringify(message)}\n`;
publishCodexStdioFrame(this.stdioKafka, "stdin", {
rawText: rawLine,
rawJson: message,
method,
rpcId: id,
phase: "client.response",
});
this.child.stdin.write(rawLine);
return;
}
this.child.stdin.write(`${JSON.stringify({ id, error: { code: -32601, message: `Unsupported client-side request: ${method}` } })}\n`);
const message = { id, error: { code: -32601, message: `Unsupported client-side request: ${method}` } };
const rawLine = `${JSON.stringify(message)}\n`;
publishCodexStdioFrame(this.stdioKafka, "stdin", {
rawText: rawLine,
rawJson: message,
method,
rpcId: id,
phase: "client.response",
});
this.child.stdin.write(rawLine);
}
private rejectRequest(id: number, error: CodexStdioFailure): void {
@@ -330,6 +412,11 @@ export class CodexStdioClient {
failureKind: this.closeFailure?.failureKind ?? null,
message: this.closeFailure?.message ?? null,
};
publishCodexStdioFrame(this.stdioKafka, "lifecycle", {
rawJson: closeInfo,
method: "process.close",
phase: "process:close",
});
this.rejectAll(this.closeFailure ?? new CodexStdioFailure("backend-failed", `codex app-server closed code=${code} signal=${signal}`, "process:close", closeInfo));
this.closeResolve(closeInfo);
}
@@ -351,6 +438,71 @@ export async function runCodexStdioTurn(options: CodexStdioTurnOptions): Promise
return { ...result, events: [...result.events, ...closeEvents] };
}
function codexStdioKafkaPublisher(options: CodexStdioTurnOptions, env: NodeJS.ProcessEnv): CodexStdioKafkaPublisher | null {
if (!truthy(env.AGENTRUN_KAFKA_ENABLED) && !truthy(env.AGENTRUN_KAFKA_STDIO_PRODUCE_ENABLED)) return null;
const config = agentRunKafkaConfig(env, { clientId: env.AGENTRUN_KAFKA_STDIO_CLIENT_ID ?? env.AGENTRUN_KAFKA_CLIENT_ID ?? "agentrun-v02-runner", stream: "stdio" });
if (config.brokers.length === 0) return null;
return {
config,
source: config.clientId,
context: codexStdioKafkaContext(options),
};
}
function truthy(value: string | undefined): boolean {
return new Set(["1", "true", "yes", "on"]).has(String(value ?? "").trim().toLowerCase());
}
function codexStdioKafkaContext(options: CodexStdioTurnOptions): CodexStdioKafkaContext {
const run = options.otelContext?.run;
const command = options.otelContext?.command;
return {
runId: run?.id ?? null,
commandId: command?.id ?? null,
attemptId: safeText(options.otelContext?.attemptId) ?? null,
runnerId: safeText(options.otelContext?.runnerId) ?? null,
runnerJobId: safeText(options.otelContext?.runnerJobId) ?? null,
jobName: safeText(options.otelContext?.jobName) ?? null,
podName: safeText(options.otelContext?.podName) ?? null,
sessionId: run?.sessionRef?.sessionId ?? null,
conversationId: run?.sessionRef?.conversationId ?? null,
threadId: run?.sessionRef?.threadId ?? options.threadId ?? null,
backendProfile: run?.backendProfile ?? options.backendProfile ?? null,
sourceCommit: safeText(options.otelContext?.sourceCommit) ?? null,
valuesPrinted: false,
};
}
function publishCodexStdioFrame(publisher: CodexStdioKafkaPublisher | null | undefined, direction: "stdin" | "stdout" | "stderr" | "lifecycle", input: { rawText?: string; rawJson?: JsonRecord; method: string; phase: string; rpcId?: number | null }): void {
if (!publisher) return;
const rawText = input.rawText ?? (input.rawJson ? JSON.stringify(input.rawJson) : "");
const context = publisher.context;
const keySource = `${context.runId ?? "run-unknown"}|${context.commandId ?? "command-unknown"}|${direction}|${input.method}|${Date.now()}|${sha256Hex(rawText).slice(0, 12)}`;
publishAgentRunKafkaMessageBestEffort(publisher.config, {
topic: publisher.config.codexStdioTopic,
key: keySource,
value: rawFrameValue({
schema: "codex-stdio.raw.v1",
eventType: `codex.stdio.${direction}`,
source: publisher.source,
context,
stdio: {
direction,
method: input.method,
phase: input.phase,
rpcId: input.rpcId ?? null,
},
rawText,
rawJson: input.rawJson,
}),
headers: {
"x-event-type": `codex.stdio.${direction}`,
"x-stdio-direction": direction,
"x-values-printed": "true",
},
}, { eventType: `codex.stdio.${direction}`, direction, method: input.method });
}
export class CodexStdioBackendSession {
private client: CodexStdioClient | null = null;
private clientKey: string | null = null;
@@ -376,6 +528,7 @@ export class CodexStdioBackendSession {
const key = codexClientKey(options, env);
this.lifecycleOtel = codexLifecycleOtelContext(options, env);
if (this.client && !this.client.isClosed && this.clientKey === key) {
this.client.setStdioKafka(codexStdioKafkaPublisher(options, env));
emitEvent({ type: "backend_status", payload: { phase: "codex-app-server:reused", ...backendMetadata(options), protocol: codexProtocol } });
emitCodexOtelSpan("codex_app_server.reused", options, env, { command: options.command ?? "codex", argsFingerprint: shortHash(JSON.stringify(options.args ?? defaultCodexArgs)) });
return this.client;
@@ -398,6 +551,7 @@ export class CodexStdioBackendSession {
cwd: options.cwd,
env,
onNotification: (message) => this.onNotification(message),
stdioKafka: codexStdioKafkaPublisher(options, env),
};
if (options.command) clientOptions.command = options.command;
if (options.args) clientOptions.args = options.args;
+325
View File
@@ -0,0 +1,325 @@
import { Kafka, logLevel, type Consumer, type EachMessagePayload, type Producer } from "kafkajs";
import { createHash } from "node:crypto";
import type { JsonRecord, JsonValue } from "./types.js";
import { redactJson, redactText } from "./redaction.js";
const TRUE_VALUES = new Set(["1", "true", "yes", "on"]);
const DEFAULT_BROKERS = "platform-infra-kafka-kafka-bootstrap.platform-infra.svc.cluster.local:9092";
const DEFAULT_AGENTS_TOPIC = "agentrun.event.v1";
const DEFAULT_STDIO_TOPIC = "codex-stdio.raw.v1";
const MAX_VALUE_PREVIEW_CHARS = 4_000;
let producerState: { key: string; producer: Producer; connecting: Promise<Producer> | null } | null = null;
export interface AgentRunKafkaConfig {
brokers: string[];
clientId: string;
agentrunEventTopic: string;
codexStdioTopic: string;
enabled: boolean;
valuesPrinted: false;
}
export interface AgentRunKafkaMessage {
topic: string;
key: string;
value: JsonRecord;
headers?: Record<string, string>;
}
export interface KafkaTailOptions {
env?: NodeJS.ProcessEnv;
brokers?: string;
topic?: string;
stream?: string;
groupId?: string;
limit?: number;
timeoutMs?: number;
fromBeginning?: boolean;
values?: boolean;
filter?: {
traceId?: string | null;
sessionId?: string | null;
runId?: string | null;
commandId?: string | null;
};
}
export function agentRunKafkaConfig(env: NodeJS.ProcessEnv = process.env, overrides: { brokers?: string; clientId?: string; topic?: string; stream?: string } = {}): AgentRunKafkaConfig {
const brokers = csv(overrides.brokers ?? env.AGENTRUN_KAFKA_BOOTSTRAP_SERVERS ?? DEFAULT_BROKERS);
const clientId = safeText(overrides.clientId ?? env.AGENTRUN_KAFKA_CLIENT_ID) ?? "agentrun-v02";
const agentrunEventTopic = safeText(env.AGENTRUN_KAFKA_EVENT_TOPIC) ?? DEFAULT_AGENTS_TOPIC;
const codexStdioTopic = safeText(env.AGENTRUN_KAFKA_STDIO_TOPIC) ?? DEFAULT_STDIO_TOPIC;
const topicOverride = safeText(overrides.topic);
return {
brokers,
clientId,
agentrunEventTopic: topicOverride && overrides.stream !== "stdio" ? topicOverride : agentrunEventTopic,
codexStdioTopic: topicOverride && overrides.stream === "stdio" ? topicOverride : codexStdioTopic,
enabled: truthy(env.AGENTRUN_KAFKA_ENABLED) || truthy(env.AGENTRUN_KAFKA_SHADOW_PRODUCE_ENABLED) || truthy(env.AGENTRUN_KAFKA_STDIO_PRODUCE_ENABLED),
valuesPrinted: false,
};
}
export function kafkaTopicForStream(config: AgentRunKafkaConfig, streamOrTopic: string | null | undefined): string {
const value = safeText(streamOrTopic);
if (!value || value === "agentrun" || value === "event" || value === "events") return config.agentrunEventTopic;
if (value === "stdio" || value === "codex-stdio" || value === "raw") return config.codexStdioTopic;
return value;
}
export function kafkaStreamForTopic(config: AgentRunKafkaConfig, topic: string): string {
if (topic === config.codexStdioTopic) return "stdio";
if (topic === config.agentrunEventTopic) return "agentrun";
return "custom";
}
export async function publishAgentRunKafkaMessage(config: AgentRunKafkaConfig, message: AgentRunKafkaMessage): Promise<void> {
if (!config.enabled) return;
if (config.brokers.length === 0) throw new Error("AGENTRUN_KAFKA_BOOTSTRAP_SERVERS is empty");
const producer = await producerForConfig(config);
await producer.send({
topic: message.topic,
messages: [{
key: message.key,
value: JSON.stringify(message.value),
headers: {
"x-values-printed": "false",
...(message.headers ?? {}),
},
}],
});
}
export function publishAgentRunKafkaMessageBestEffort(config: AgentRunKafkaConfig, message: AgentRunKafkaMessage, context: JsonRecord = {}): void {
void publishAgentRunKafkaMessage(config, message)
.catch((error) => {
console.warn(JSON.stringify({
code: "agentrun-kafka-produce-failed",
component: "agentrun-kafka",
message: error instanceof Error ? error.message : String(error),
topic: message.topic,
keySha256: sha256Hex(message.key),
...context,
valuesPrinted: false,
}));
});
}
export async function tailAgentRunKafka(options: KafkaTailOptions): Promise<JsonRecord> {
const env = options.env ?? process.env;
const config = agentRunKafkaConfig(env, { brokers: options.brokers, stream: options.stream, topic: options.topic });
const topic = kafkaTopicForStream(config, options.topic ?? options.stream);
if (config.brokers.length === 0) throw new Error("AGENTRUN_KAFKA_BOOTSTRAP_SERVERS is empty; pass --brokers or set env");
const limit = boundedInteger(options.limit ?? 20, 1, 200, "limit");
const timeoutMs = boundedInteger(options.timeoutMs ?? 5_000, 500, 60_000, "timeout-ms");
const groupId = safeText(options.groupId) ?? `agentrun-cli-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`;
const kafka = new Kafka({ clientId: `${config.clientId}-cli`, brokers: config.brokers, logLevel: logLevel.NOTHING });
const consumer = kafka.consumer({ groupId });
const messages: JsonRecord[] = [];
const startedAt = Date.now();
let timedOut = false;
await consumer.connect();
try {
await consumer.subscribe({ topic, fromBeginning: options.fromBeginning !== false });
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
timedOut = true;
resolve();
}, timeoutMs);
timer.unref?.();
consumer.run({
eachMessage: async (payload) => {
const item = kafkaMessageSummary(topic, payload, Boolean(options.values));
if (matchesKafkaFilter(item, options.filter)) messages.push(item);
if (messages.length >= limit) {
clearTimeout(timer);
resolve();
}
},
}).catch((error) => {
clearTimeout(timer);
reject(error);
});
});
} finally {
await disconnectQuietly(consumer);
}
return {
action: "agentrun-kafka-tail",
topic,
stream: kafkaStreamForTopic(config, topic),
brokers: config.brokers,
groupId,
fromBeginning: options.fromBeginning !== false,
limit,
timeoutMs,
elapsedMs: Date.now() - startedAt,
timedOut,
matched: messages.length,
messages,
valuesPrinted: Boolean(options.values),
};
}
export function sha256Hex(value: string | Buffer): string {
return createHash("sha256").update(value).digest("hex");
}
export function stableKafkaJson(value: JsonValue): string {
return JSON.stringify(sortJson(value));
}
export function jsonBytes(value: JsonValue): number {
return Buffer.byteLength(JSON.stringify(value), "utf8");
}
export function safeText(value: unknown, max = 240): string | null {
const text = typeof value === "string" ? value.trim() : value === undefined || value === null ? "" : String(value).trim();
if (text.length === 0) return null;
return text.length > max ? `${text.slice(0, max)}...` : text;
}
function kafkaMessageSummary(topic: string, payload: EachMessagePayload, includeValue: boolean): JsonRecord {
const raw = payload.message.value?.toString("utf8") ?? "";
const parsed = parseJsonRecord(raw);
const key = payload.message.key?.toString("utf8") ?? "";
const base: JsonRecord = {
topic,
partition: payload.partition,
offset: payload.message.offset,
timestamp: payload.message.timestamp ?? null,
keySha256: key ? sha256Hex(key) : null,
valueSha256: sha256Hex(raw),
valueBytes: Buffer.byteLength(raw, "utf8"),
schema: typeof parsed?.schema === "string" ? parsed.schema : null,
eventType: typeof parsed?.eventType === "string" ? parsed.eventType : null,
producedAt: typeof parsed?.producedAt === "string" ? parsed.producedAt : null,
runId: nestedString(parsed, ["run", "runId"]) ?? nestedString(parsed, ["context", "runId"]),
commandId: nestedString(parsed, ["command", "commandId"]) ?? nestedString(parsed, ["context", "commandId"]) ?? nestedString(parsed, ["event", "commandId"]),
sessionId: nestedString(parsed, ["run", "sessionId"]) ?? nestedString(parsed, ["context", "sessionId"]),
traceId: nestedString(parsed, ["context", "traceId"]) ?? nestedString(parsed, ["traceId"]),
method: nestedString(parsed, ["stdio", "method"]),
direction: nestedString(parsed, ["stdio", "direction"]),
valuesPrinted: includeValue,
};
if (includeValue) {
base.value = raw.length > MAX_VALUE_PREVIEW_CHARS ? `${raw.slice(0, MAX_VALUE_PREVIEW_CHARS)}...` : raw;
base.valueTruncated = raw.length > MAX_VALUE_PREVIEW_CHARS;
}
return base;
}
function matchesKafkaFilter(message: JsonRecord, filter: KafkaTailOptions["filter"]): boolean {
if (!filter) return true;
if (filter.traceId && message.traceId !== filter.traceId) return false;
if (filter.sessionId && message.sessionId !== filter.sessionId) return false;
if (filter.runId && message.runId !== filter.runId) return false;
if (filter.commandId && message.commandId !== filter.commandId) return false;
return true;
}
async function producerForConfig(config: AgentRunKafkaConfig): Promise<Producer> {
const key = `${config.clientId}|${config.brokers.join(",")}`;
if (producerState?.key === key && producerState.producer) return producerState.producer;
if (producerState?.key === key && producerState.connecting) return producerState.connecting;
const kafka = new Kafka({ clientId: config.clientId, brokers: config.brokers, logLevel: logLevel.NOTHING });
const producer = kafka.producer({ allowAutoTopicCreation: false });
const connecting = producer.connect().then(() => {
if (producerState?.producer === producer) producerState.connecting = null;
return producer;
}).catch((error) => {
if (producerState?.producer === producer) producerState = null;
throw error;
});
producerState = { key, producer, connecting };
return await connecting;
}
async function disconnectQuietly(consumer: Consumer): Promise<void> {
try {
await consumer.disconnect();
} catch {
// CLI tail should report already captured messages even if disconnect races with timeout.
}
}
function parseJsonRecord(value: string): JsonRecord | null {
try {
const parsed = JSON.parse(value) as unknown;
return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? parsed as JsonRecord : null;
} catch {
return null;
}
}
function nestedString(record: JsonRecord | null, path: string[]): string | null {
let current: unknown = record;
for (const key of path) {
if (!current || typeof current !== "object" || Array.isArray(current)) return null;
current = (current as Record<string, unknown>)[key];
}
return typeof current === "string" && current.length > 0 ? current : null;
}
function sortJson(value: JsonValue): JsonValue {
if (Array.isArray(value)) return value.map(sortJson);
if (!value || typeof value !== "object") return value;
const sorted: JsonRecord = {};
for (const key of Object.keys(value).sort()) sorted[key] = sortJson((value as JsonRecord)[key] as JsonValue);
return sorted;
}
function boundedInteger(value: number, min: number, max: number, label: string): number {
if (!Number.isFinite(value)) throw new Error(`${label} must be a finite number`);
const next = Math.floor(value);
if (next < min || next > max) throw new Error(`${label} must be between ${min} and ${max}`);
return next;
}
function truthy(value: string | undefined): boolean {
return TRUE_VALUES.has(String(value ?? "").trim().toLowerCase());
}
function csv(value: string | undefined): string[] {
return String(value ?? "").split(",").map((entry) => entry.trim()).filter(Boolean);
}
export function rawFrameValue(input: {
schema: string;
eventType: string;
source: string;
context: JsonRecord;
stdio: JsonRecord;
rawText?: string;
rawJson?: JsonRecord;
}): JsonRecord {
const raw = input.rawText ?? (input.rawJson ? stableKafkaJson(input.rawJson) : "");
return {
schema: input.schema,
eventType: input.eventType,
source: input.source,
producedAt: new Date().toISOString(),
context: input.context,
stdio: {
...input.stdio,
rawSha256: sha256Hex(raw),
rawBytes: Buffer.byteLength(raw, "utf8"),
rawText: input.rawText ?? null,
rawJson: input.rawJson ?? null,
valuesPrinted: true,
},
diagnostics: {
valuesPrinted: true,
},
valuesPrinted: true,
};
}
export function redactedKafkaValue(value: JsonRecord): JsonRecord {
return redactJson(value) as JsonRecord;
}
export function kafkaErrorMessage(error: unknown): string {
return error instanceof Error ? redactText(error.message) : redactText(String(error));
}
+18 -59
View File
@@ -1,18 +1,9 @@
import { Kafka, logLevel, type Producer } from "kafkajs";
import type { AgentRunStore } from "./store.js";
import type { CommandRecord, JsonRecord, JsonValue, RunEvent, RunRecord } from "../common/types.js";
import { stableHash } from "../common/validation.js";
import { agentRunKafkaConfig, jsonBytes, publishAgentRunKafkaMessageBestEffort, safeText, sha256Hex, stableKafkaJson, type AgentRunKafkaConfig } from "../common/kafka-events.js";
const TRUE_VALUES = new Set(["1", "true", "yes", "on"]);
const warnedCodes = new Set<string>();
let producerState: { key: string; producer: Producer; connecting: Promise<Producer> | null } | null = null;
interface KafkaShadowConfig {
brokers: string[];
topic: string;
clientId: string;
key: string;
}
interface ShadowEventInput {
env: NodeJS.ProcessEnv;
@@ -28,7 +19,7 @@ interface ShadowEventInput {
type StoreFunction = (...args: never[]) => unknown;
export function withKafkaShadowProducer(store: AgentRunStore, env: NodeJS.ProcessEnv = process.env): AgentRunStore {
if (!kafkaShadowConfig(env)) return store;
if (!kafkaEventConfig(env)) return store;
return new Proxy(store, {
get(target, prop, receiver) {
if (prop === "createRun") {
@@ -74,17 +65,17 @@ export function withKafkaShadowProducer(store: AgentRunStore, env: NodeJS.Proces
}
function publishAgentRunShadow(input: ShadowEventInput): void {
const config = kafkaShadowConfig(input.env);
const config = kafkaEventConfig(input.env);
if (!config) return;
void Promise.resolve(resolveRun(input))
.then((run) => {
const message = buildAgentRunShadowMessage({ ...input, run, config });
return producerForConfig(config).then((producer) => producer.send({ topic: config.topic, messages: [message] }));
publishAgentRunKafkaMessageBestEffort(config, message, { eventType: input.eventType });
})
.catch((error) => warnShadowProducer("agentrun-kafka-shadow-produce-failed", {
message: error instanceof Error ? error.message : String(error),
eventType: input.eventType,
topic: config.topic,
topic: config.agentrunEventTopic,
clientId: config.clientId,
}));
}
@@ -95,10 +86,10 @@ async function resolveRun(input: ShadowEventInput): Promise<RunRecord | null> {
return await input.store.getRun(input.runId);
}
function buildAgentRunShadowMessage(input: ShadowEventInput & { run: RunRecord | null; config: KafkaShadowConfig }): { key: string; value: string; headers: Record<string, string> } {
function buildAgentRunShadowMessage(input: ShadowEventInput & { run: RunRecord | null; config: AgentRunKafkaConfig }): { topic: string; key: string; value: JsonRecord; headers: Record<string, string> } {
const payload = input.eventPayload ?? eventPayload(input.event);
const event = {
schema: "agentrun.hwlab.event.shadow.v1",
schema: "agentrun.event.v1",
eventType: input.eventType,
source: input.config.clientId,
producedAt: new Date().toISOString(),
@@ -107,7 +98,7 @@ function buildAgentRunShadowMessage(input: ShadowEventInput & { run: RunRecord |
command: commandSummary(input.command),
event: runEventSummary(input.event, payload),
diagnostics: {
payloadSha256: stableHash((payload ?? {}) as JsonValue),
payloadSha256: sha256Hex(stableKafkaJson((payload ?? {}) as JsonValue)),
payloadBytes: jsonBytes(payload ?? {}),
valuesPrinted: false,
shadowConsumeEnabled: false,
@@ -115,8 +106,9 @@ function buildAgentRunShadowMessage(input: ShadowEventInput & { run: RunRecord |
valuesPrinted: false,
};
return {
topic: input.config.agentrunEventTopic,
key: safeText(input.command?.id ?? input.eventPayload?.commandId ?? input.run?.id ?? input.runId) ?? input.config.clientId,
value: JSON.stringify(event),
value: event as JsonRecord,
headers: {
"x-shadow-mode": "produce-only",
"x-values-printed": "false",
@@ -125,47 +117,24 @@ function buildAgentRunShadowMessage(input: ShadowEventInput & { run: RunRecord |
};
}
function kafkaShadowConfig(env: NodeJS.ProcessEnv): KafkaShadowConfig | null {
if (!truthy(env.AGENTRUN_KAFKA_SHADOW_PRODUCE_ENABLED)) return null;
function kafkaEventConfig(env: NodeJS.ProcessEnv): AgentRunKafkaConfig | null {
if (!truthy(env.AGENTRUN_KAFKA_ENABLED) && !truthy(env.AGENTRUN_KAFKA_SHADOW_PRODUCE_ENABLED)) return null;
if (truthy(env.AGENTRUN_KAFKA_SHADOW_CONSUME_ENABLED)) {
warnOnce("agentrun-kafka-shadow-consume-ignored", {
message: "AgentRun Kafka consumer cutover is disabled in this stage; producer continues in shadow mode.",
valuesPrinted: false,
});
}
const brokers = csv(env.AGENTRUN_KAFKA_BOOTSTRAP_SERVERS);
const topic = safeText(env.AGENTRUN_KAFKA_EVENT_TOPIC);
const clientId = safeText(env.AGENTRUN_KAFKA_CLIENT_ID);
const config = agentRunKafkaConfig(env, { clientId: env.AGENTRUN_KAFKA_CLIENT_ID ?? "agentrun-v02-manager" });
const missing: string[] = [];
if (brokers.length === 0) missing.push("AGENTRUN_KAFKA_BOOTSTRAP_SERVERS");
if (!topic) missing.push("AGENTRUN_KAFKA_EVENT_TOPIC");
if (!clientId) missing.push("AGENTRUN_KAFKA_CLIENT_ID");
if (missing.length > 0 || !topic || !clientId) {
if (config.brokers.length === 0) missing.push("AGENTRUN_KAFKA_BOOTSTRAP_SERVERS");
if (!config.agentrunEventTopic) missing.push("AGENTRUN_KAFKA_EVENT_TOPIC");
if (!config.clientId) missing.push("AGENTRUN_KAFKA_CLIENT_ID");
if (missing.length > 0) {
warnOnce("agentrun-kafka-shadow-config-missing", { missing, valuesPrinted: false });
return null;
}
return { brokers, topic, clientId, key: `${clientId}|${topic}|${brokers.join(",")}` };
}
async function producerForConfig(config: KafkaShadowConfig): Promise<Producer> {
if (producerState?.key === config.key && producerState.producer) return producerState.producer;
if (producerState?.key === config.key && producerState.connecting) return producerState.connecting;
const kafka = new Kafka({ clientId: config.clientId, brokers: config.brokers, logLevel: logLevel.NOTHING });
const producer = kafka.producer({ allowAutoTopicCreation: false });
const connecting = producer.connect().then(() => {
if (producerState?.producer === producer) producerState.connecting = null;
return producer;
}).catch((error) => {
if (producerState?.producer === producer) producerState = null;
throw error;
});
const state: { key: string; producer: Producer; connecting: Promise<Producer> | null } = {
key: config.key,
producer,
connecting,
};
producerState = state;
return await connecting;
return config;
}
function runSummary(run: RunRecord | null): JsonRecord | null {
@@ -244,16 +213,6 @@ function csv(value: string | undefined): string[] {
return String(value ?? "").split(",").map((entry) => entry.trim()).filter(Boolean);
}
function safeText(value: unknown, max = 240): string | null {
const text = typeof value === "string" ? value.trim() : value === undefined || value === null ? "" : String(value).trim();
if (text.length === 0) return null;
return text.length > max ? `${text.slice(0, max)}...` : text;
}
function jsonBytes(value: JsonRecord): number {
return Buffer.byteLength(JSON.stringify(value), "utf8");
}
function warnOnce(code: string, details: JsonRecord): void {
if (warnedCodes.has(code)) return;
warnedCodes.add(code);
+11
View File
@@ -284,6 +284,7 @@ function runnerEnv(options: RunnerJobRenderOptions, context: { namespace: string
{ name: "CODEX_HOME", value: codexHome },
{ name: "AGENTRUN_CODEX_HOME_STORAGE_KIND", value: context.sessionPvc ? "session-pvc" : "runner-home" },
...runnerOtelEnvVars(process.env),
...runnerKafkaEnvVars(process.env),
...(selectedSecret ? [{ name: "AGENTRUN_CODEX_SECRET_HOME", value: selectedSecret.projectionMountPath }] : []),
...(context.sessionPvc ? [
{ name: "AGENTRUN_SESSION_PVC_NAME", value: context.sessionPvc.pvcName },
@@ -435,6 +436,16 @@ function runnerOtelEnvVars(env: NodeJS.ProcessEnv): JsonRecord[] {
];
}
function runnerKafkaEnvVars(env: NodeJS.ProcessEnv): JsonRecord[] {
return [
...optionalEnvVar("AGENTRUN_KAFKA_ENABLED", env.AGENTRUN_KAFKA_ENABLED),
...optionalEnvVar("AGENTRUN_KAFKA_BOOTSTRAP_SERVERS", env.AGENTRUN_KAFKA_BOOTSTRAP_SERVERS),
...optionalEnvVar("AGENTRUN_KAFKA_STDIO_TOPIC", env.AGENTRUN_KAFKA_STDIO_TOPIC),
...optionalEnvVar("AGENTRUN_KAFKA_STDIO_CLIENT_ID", env.AGENTRUN_KAFKA_STDIO_CLIENT_ID),
...optionalEnvVar("AGENTRUN_KAFKA_STDIO_PRODUCE_ENABLED", env.AGENTRUN_KAFKA_STDIO_PRODUCE_ENABLED),
];
}
function optionalEnvVar(name: string, value: string | undefined): JsonRecord[] {
const normalized = value?.trim();
return normalized ? [{ name, value: normalized }] : [];