fix: 增加隔离 Kafka 重放分层诊断
This commit is contained in:
@@ -0,0 +1,72 @@
|
||||
# R2.3 隔离 Kafka 重放分层诊断任务报告
|
||||
|
||||
## 上下文
|
||||
|
||||
- 主问题:
|
||||
- [HWLAB #2474](https://github.com/pikasTech/HWLAB/issues/2474)
|
||||
- [主问题调查结论](https://github.com/pikasTech/HWLAB/issues/2474#issuecomment-4935408473)
|
||||
- 隔离重放专项:
|
||||
- [HWLAB #2476](https://github.com/pikasTech/HWLAB/issues/2476)
|
||||
- [隔离重放根因](https://github.com/pikasTech/HWLAB/issues/2476#issuecomment-4935398125)
|
||||
|
||||
## 完成结果
|
||||
|
||||
- AgentRun 到 HWLAB 的 debug producer 增加批次关联合同:
|
||||
- 每批生成或接受一个 `replayId`;
|
||||
- 每个 `hwlab.event.debug.v1` envelope 保留 `replayId`、批次序号和原始 AgentRun lineage;
|
||||
- 正常 Kafka 调试默认读取 canonical `agentrun.event.v1`;
|
||||
- `agentrun.event.debug.v1` 仅由调用方显式选择,用于 stdio partial reconstruction;
|
||||
- Kafka 读取只做映射预检,发布必须显式指定 `--publish`。
|
||||
- debug producer 形成确定的输出 barrier:
|
||||
- 只向 YAML 指定的隔离 debug topic 发布;
|
||||
- 所有批次记录固定写入 debug topic 的 partition 0;
|
||||
- 由 Kafka publish metadata 返回 `firstOffset`、`lastOffset` 和 `count`;
|
||||
- 发布后无法形成唯一 offset range 时,以 `debug_publish_barrier_missing` 失败关闭。
|
||||
- debug SSE 在 JSON parse 和 trace/replay filter 前采集 broker 事实:
|
||||
- 报告 `scanned`、`barrierScanned`、`parsed`、`traceMatched`、`replayMatched`、`matched` 和 `delivered`;
|
||||
- 分别保留 requested、scanned、matched 和 delivered offset range;
|
||||
- completion 保留第一条匹配事件的脱敏 source lineage;
|
||||
- `clientReceived`、`decoded` 和 `applied` 在客户端尚未回报时保持 `null`,不伪造成功计数。
|
||||
- 新增分层结果合同:
|
||||
- producer/source:`producer_not_invoked`、`source_trace_missing`;
|
||||
- Kafka/server:`topic_no_append_for_replay`、`record_parse_rejected`、`records_scanned_but_filter_mismatch`、`sse_write_failed`;
|
||||
- client:`transport_failed`、`decoder_rejected`、`reducer_rejected`;
|
||||
- terminal:`terminal_missing`、`terminal_complete`。
|
||||
- 输入合同失败关闭:
|
||||
- 不完整或逆序 offset range 返回 `workbench_kafka_debug_offset_range_invalid`;
|
||||
- 非 `rpl_` 安全标识返回 `workbench_kafka_debug_replay_id_invalid`;
|
||||
- product topic 和 product group 继续禁止作为隔离 debug 资源。
|
||||
|
||||
## 验证
|
||||
|
||||
- 定向命令:
|
||||
|
||||
```bash
|
||||
bun test \
|
||||
internal/cloud/workbench-kafka-debug-replay-contract.test.ts \
|
||||
internal/cloud/workbench-kafka-sse-debug.test.ts \
|
||||
internal/cloud/kafka-event-bridge.test.ts \
|
||||
tools/hwlab-cli/kafka-regenerate.test.ts
|
||||
```
|
||||
|
||||
- 结果:
|
||||
- `45 pass`;
|
||||
- `0 fail`。
|
||||
- 已覆盖四类 MDTODO 验收结果:
|
||||
- empty:`producer_not_invoked`;
|
||||
- matched-no-terminal:`terminal_missing`;
|
||||
- reducer-rejected:`reducer_rejected`;
|
||||
- terminal-complete:`terminal_complete`。
|
||||
- 产品路径回归:
|
||||
- `hwlab` 与 `stdio` diagnostic stream 既有测试通过;
|
||||
- live Kafka direct publish/fanout 测试通过;
|
||||
- 未启用 projector、read model、snapshot、sync、recovery replay 或 polling。
|
||||
|
||||
## 边界与后续集成
|
||||
|
||||
- 本子任务未修改 `WorkbenchKafkaDebugPanel.vue` 或其他 UI。
|
||||
- 本子任务未部署或修改 NC01/v03 运行面。
|
||||
- UI 集成时应:
|
||||
- 把 producer 返回的 `replayId` 和 `outputBarrier` 原样传入 debug SSE;
|
||||
- 把客户端实际 `received/decoded/applied` 计数交给同一分类合同;
|
||||
- 不把 debug topic、debug group 或结果合同接入产品实时事实源。
|
||||
@@ -9,15 +9,17 @@
|
||||
|
||||
- Kafka 输入适合验证真实 topic 中的 session 事件:
|
||||
- 命令为 `hwlab-cli kafka regenerate hwlab --from kafka --session-id <sessionId>`;
|
||||
- 默认输入 topic 为 `agentrun.event.debug.v1`;
|
||||
- 产品 canonical 输入可显式指定 `--input-topic agentrun.event.v1`;
|
||||
- 默认输入 topic 优先读取 owning YAML 注入的 `HWLAB_KAFKA_AGENTRUN_EVENT_TOPIC`;
|
||||
- 未注入时默认读取 canonical `agentrun.event.v1`;
|
||||
- stdio partial reconstruction 必须显式指定 `--input-topic agentrun.event.debug.v1`;
|
||||
- output topic 优先读取 owning YAML 注入的 `HWLAB_KAFKA_HWLAB_DEBUG_EVENT_TOPIC`;
|
||||
- consumer group prefix 优先读取 owning YAML 注入的 `HWLAB_KAFKA_HWLAB_DEBUG_GROUP_PREFIX`;
|
||||
- 未注入时使用离线合同值 `hwlab.event.debug.v1` 与 `hwlab-v03-workbench-isolated-debug`;
|
||||
- output topic 未注入时使用隔离合同值 `hwlab.event.debug.v1`;
|
||||
- Kafka 读取或发布缺少 group prefix 时失败关闭,不生成代码内运维默认值;
|
||||
- Workbench 消费 debug topic 由 `HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_ENABLED` 独立控制;
|
||||
- CLI 的离线映射不依赖该运行面开关;
|
||||
- Kafka 输入默认发布到独立 HWLAB debug topic;
|
||||
- `--no-publish` 可只生成本地证据。
|
||||
- Kafka 输入默认只做映射预检和本地证据固化;
|
||||
- 只有显式 `--publish` 才发布到独立 HWLAB debug topic。
|
||||
|
||||
- JSONL 输入适合完全离线的单步测试:
|
||||
- 命令为 `hwlab-cli kafka regenerate hwlab --from jsonl --session-id <sessionId> --jsonl-file <path> --no-publish`;
|
||||
@@ -35,6 +37,11 @@
|
||||
|
||||
- 输出验证采用一对一和原序映射:
|
||||
- `--expect-count <n>` 在映射和发布前锁定输入数量;
|
||||
- 每次预检生成 `replayId`,也可用 `--replay-id rpl_...` 显式复用同一批次标识;
|
||||
- `next.command` 保留同一个 `replayId`,用于先预检、再显式发布;
|
||||
- 发布固定使用隔离 debug topic 的 partition 0;
|
||||
- 成功发布必须从 Kafka metadata 形成 `firstOffset`、`lastOffset` 和 `count` barrier;
|
||||
- 无法形成唯一 barrier 时以 `debug_publish_barrier_missing` 失败关闭;
|
||||
- 命令验证 source sequence、trace、raw AgentRun session、派生 HWLAB session 和源消息哈希;
|
||||
- 默认输出为 compact text;
|
||||
- 只有显式 `--json` 才输出结构化 JSON;
|
||||
|
||||
@@ -12,7 +12,7 @@ import {
|
||||
} from "./workbench-realtime-capabilities.ts";
|
||||
|
||||
const TRUE_VALUES = new Set(["1", "true", "yes", "on"]);
|
||||
const DEFAULT_AGENTRUN_EVENT_TOPIC = "agentrun.event.v1";
|
||||
export const DEFAULT_AGENTRUN_EVENT_TOPIC = "agentrun.event.v1";
|
||||
const DEFAULT_HWLAB_EVENT_TOPIC = "hwlab.event.v1";
|
||||
const DEFAULT_STDIO_TOPIC = "codex-stdio.raw.v1";
|
||||
export const DEFAULT_AGENTRUN_DEBUG_EVENT_TOPIC = "agentrun.event.debug.v1";
|
||||
@@ -733,6 +733,9 @@ export function projectAgentRunKafkaMessageToHwlabDebugEvent(kafkaMessage = {},
|
||||
};
|
||||
}
|
||||
const debugSessionId = decoded.reconstruction ? stringValue(decoded.event.hwlabSessionId) : stringValue(projected.sessionId);
|
||||
const replayId = stringValue(options.replayId);
|
||||
const replayIndex = integerValue(options.replayIndex);
|
||||
const replayCount = integerValue(options.replayCount);
|
||||
const debugEnvelope = {
|
||||
...projected,
|
||||
schema: DEFAULT_HWLAB_DEBUG_EVENT_TOPIC,
|
||||
@@ -742,6 +745,15 @@ export function projectAgentRunKafkaMessageToHwlabDebugEvent(kafkaMessage = {},
|
||||
event: decoded.reconstruction ? { ...projected.event, sessionId: debugSessionId } : projected.event,
|
||||
outboxSeq: null,
|
||||
...(decoded.reconstruction ? { eventId: null, sourceEventId: null } : {}),
|
||||
...(replayId ? {
|
||||
replayId,
|
||||
debugReplay: {
|
||||
replayId,
|
||||
index: replayIndex,
|
||||
count: replayCount,
|
||||
valuesPrinted: false
|
||||
}
|
||||
} : {}),
|
||||
diagnostics: {
|
||||
projectedBy: "hwlab-kafka-event-bridge",
|
||||
projectionMode: "source-direct-debug",
|
||||
@@ -759,6 +771,7 @@ export function projectAgentRunKafkaMessageToHwlabDebugEvent(kafkaMessage = {},
|
||||
inputRunId: firstText(decoded.event.runId, decoded.event.run?.runId),
|
||||
inputCommandId: firstText(decoded.event.commandId, decoded.event.command?.commandId),
|
||||
sourceSeq: integerValue(decoded.event.sourceSeq ?? decoded.event.event?.seq),
|
||||
replayId,
|
||||
valuesPrinted: false
|
||||
},
|
||||
...(decoded.reconstruction ? {
|
||||
@@ -977,13 +990,14 @@ export async function queryKafkaEventStream({ env = process.env, stream = "hwlab
|
||||
};
|
||||
}
|
||||
|
||||
export async function openKafkaEventStream({ env = process.env, stream = "hwlab", topic = null, groupIdPrefix = null, traceId = null, sessionId = null, runId = null, commandId = null, fromBeginning = false, onEvent, onError = null, kafkaFactory = defaultKafkaFactory } = {}) {
|
||||
export async function openKafkaEventStream({ env = process.env, stream = "hwlab", topic = null, groupIdPrefix = null, traceId = null, sessionId = null, runId = null, commandId = null, replayId = null, offsetRange = null, fromBeginning = false, onEvent, onRecord = null, onError = null, kafkaFactory = defaultKafkaFactory } = {}) {
|
||||
if (typeof onEvent !== "function") throw new Error("onEvent callback is required for Kafka event streaming.");
|
||||
const brokers = csv(env.HWLAB_KAFKA_BOOTSTRAP_SERVERS);
|
||||
if (brokers.length === 0) throw new Error("HWLAB_KAFKA_BOOTSTRAP_SERVERS is required for Kafka event streams.");
|
||||
const clientId = stringValue(env.HWLAB_KAFKA_CLIENT_ID) || DEFAULT_CLIENT_ID;
|
||||
const resolvedTopic = stringValue(topic) || kafkaTopicForStream(stream, env);
|
||||
const filters = compactObject({ traceId, sessionId, runId, commandId });
|
||||
const filters = compactObject({ traceId, sessionId, runId, commandId, replayId });
|
||||
const resolvedOffsetRange = normalizeKafkaOffsetRange(offsetRange);
|
||||
const kafka = kafkaFactory({ brokers, clientId: `${clientId}-debug-sse` });
|
||||
const resolvedGroupIdPrefix = stringValue(groupIdPrefix) || `${clientId}-debug-sse`;
|
||||
const groupId = `${resolvedGroupIdPrefix}-${Date.now()}-${randomUUID().slice(0, 8)}`;
|
||||
@@ -999,7 +1013,23 @@ export async function openKafkaEventStream({ env = process.env, stream = "hwlab"
|
||||
if (stopped) return;
|
||||
const valueText = message.value ? Buffer.from(message.value).toString("utf8") : "";
|
||||
const value = parseJson(valueText);
|
||||
if (!value || !eventMatchesFilters(value, filters)) return;
|
||||
const parsed = Boolean(value && typeof value === "object" && !Array.isArray(value));
|
||||
const inOffsetRange = kafkaRecordInOffsetRange(partition, message.offset, resolvedOffsetRange);
|
||||
const filterMatches = parsed ? eventFilterResults(value, filters) : {};
|
||||
const matched = parsed && inOffsetRange && Object.values(filterMatches).every(Boolean);
|
||||
if (typeof onRecord === "function") {
|
||||
await onRecord({
|
||||
topic: messageTopic,
|
||||
partition,
|
||||
offset: message.offset,
|
||||
parsed,
|
||||
inOffsetRange,
|
||||
filterMatches,
|
||||
matched,
|
||||
valuesPrinted: false
|
||||
});
|
||||
}
|
||||
if (!matched) return;
|
||||
await onEvent({
|
||||
topic: messageTopic,
|
||||
partition,
|
||||
@@ -1023,6 +1053,7 @@ export async function openKafkaEventStream({ env = process.env, stream = "hwlab"
|
||||
topic: resolvedTopic,
|
||||
groupId,
|
||||
filters,
|
||||
offsetRange: resolvedOffsetRange,
|
||||
async stop() {
|
||||
stopped = true;
|
||||
if (running) await consumer.stop().catch(() => undefined);
|
||||
@@ -1105,12 +1136,17 @@ function mapAgentRunSourceEventToHwlabEvent({ input, sourceEvent, payload, run,
|
||||
}
|
||||
|
||||
function eventMatchesFilters(value, filters) {
|
||||
return Object.values(eventFilterResults(value, filters)).every(Boolean);
|
||||
}
|
||||
|
||||
function eventFilterResults(value, filters) {
|
||||
const results = {};
|
||||
for (const [name, expected] of Object.entries(filters)) {
|
||||
const needle = stringValue(expected);
|
||||
if (!needle) continue;
|
||||
if (!eventFieldCandidates(value, name).includes(needle)) return false;
|
||||
results[name] = eventFieldCandidates(value, name).includes(needle);
|
||||
}
|
||||
return true;
|
||||
return results;
|
||||
}
|
||||
|
||||
function eventFieldCandidates(value, name) {
|
||||
@@ -1124,15 +1160,40 @@ function eventFieldCandidates(value, name) {
|
||||
const ids = objectValue(value.ids);
|
||||
const stdio = objectValue(value.stdio ?? value.frame ?? value.codexStdio);
|
||||
const sourceEvent = objectValue(value.sourceEvent);
|
||||
const debugReplay = objectValue(value.debugReplay);
|
||||
const debugLineage = objectValue(value.debugLineage);
|
||||
const nestedEvent = objectValue(value.event?.sourceEvent ?? value.agentRunEvent ?? payload.event);
|
||||
const nestedPayload = objectValue(nestedEvent.payload);
|
||||
if (name === "traceId") return textCandidates(value.traceId, value.trace_id, event.traceId, event.trace_id, context.traceId, context.trace_id, sourceEvent.traceId, sourceEvent.trace_id, nestedEvent.traceId, nestedEvent.trace_id, nestedPayload.traceId, nestedPayload.trace_id, payload.traceId, payload.trace_id, metadata.traceId, metadata.trace_id, trace.traceId, trace.trace_id, ids.traceId, ids.trace_id, stdio.traceId, stdio.trace_id);
|
||||
if (name === "sessionId") return textCandidates(value.sessionId, value.session_id, event.sessionId, event.session_id, context.sessionId, context.session_id, run.sessionId, run.session_id, nestedPayload.sessionId, nestedPayload.session_id, payload.sessionId, payload.session_id, metadata.sessionId, metadata.session_id, ids.sessionId, ids.session_id, stdio.sessionId, stdio.session_id);
|
||||
if (name === "runId") return textCandidates(value.runId, value.run_id, event.runId, event.run_id, context.runId, context.run_id, run.runId, run.run_id, nestedEvent.runId, nestedEvent.run_id, nestedPayload.runId, nestedPayload.run_id, payload.runId, payload.run_id, metadata.runId, metadata.run_id, ids.runId, ids.run_id, stdio.runId, stdio.run_id);
|
||||
if (name === "commandId") return textCandidates(value.commandId, value.command_id, event.commandId, event.command_id, context.commandId, context.command_id, command.commandId, command.command_id, nestedEvent.commandId, nestedEvent.command_id, nestedPayload.commandId, nestedPayload.command_id, payload.commandId, payload.command_id, metadata.commandId, metadata.command_id, ids.commandId, ids.command_id, stdio.commandId, stdio.command_id);
|
||||
if (name === "replayId") return textCandidates(value.replayId, value.replay_id, debugReplay.replayId, debugReplay.replay_id, debugLineage.replayId, debugLineage.replay_id);
|
||||
return [];
|
||||
}
|
||||
|
||||
function normalizeKafkaOffsetRange(value) {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
||||
const partition = integerValue(value.partition);
|
||||
const firstOffset = kafkaOffsetValue(value.firstOffset ?? value.startOffset);
|
||||
const lastOffset = kafkaOffsetValue(value.lastOffset ?? value.endOffset);
|
||||
if (partition === null || firstOffset === null || lastOffset === null || BigInt(firstOffset) > BigInt(lastOffset)) return null;
|
||||
return { partition, firstOffset, lastOffset };
|
||||
}
|
||||
|
||||
function kafkaRecordInOffsetRange(partitionValue, offsetValue, range) {
|
||||
if (!range) return true;
|
||||
const partition = integerValue(partitionValue);
|
||||
const offset = kafkaOffsetValue(offsetValue);
|
||||
return partition === range.partition && offset !== null && BigInt(offset) >= BigInt(range.firstOffset) && BigInt(offset) <= BigInt(range.lastOffset);
|
||||
}
|
||||
|
||||
function kafkaOffsetValue(value) {
|
||||
const text = stringValue(value);
|
||||
if (!text || !/^\d+$/u.test(text)) return null;
|
||||
try { return BigInt(text).toString(); } catch { return null; }
|
||||
}
|
||||
|
||||
function kafkaTopicForStream(stream, env) {
|
||||
const normalized = stringValue(stream) || "hwlab";
|
||||
if (normalized === "stdio" || normalized === "codex-stdio") return stringValue(env.HWLAB_KAFKA_STDIO_TOPIC ?? env.AGENTRUN_KAFKA_STDIO_TOPIC) || DEFAULT_STDIO_TOPIC;
|
||||
@@ -1178,7 +1239,8 @@ function kafkaHeaders(projected) {
|
||||
"x-session-id": projected.sessionId ?? "",
|
||||
"x-run-id": projected.context?.runId ?? "",
|
||||
"x-command-id": projected.context?.commandId ?? "",
|
||||
"x-values-printed": "false"
|
||||
"x-values-printed": "false",
|
||||
...(projected.replayId ? { "x-debug-replay-id": projected.replayId } : {})
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "bun:test";
|
||||
|
||||
import { classifyWorkbenchKafkaDebugReplayResult } from "./workbench-kafka-debug-replay-contract.ts";
|
||||
|
||||
const readyConsumer = { ready: true, groupId: "hwlab-debug-test" };
|
||||
const deliveredCounts = {
|
||||
scanned: 1,
|
||||
barrierScanned: 1,
|
||||
parsed: 1,
|
||||
traceMatched: 1,
|
||||
replayMatched: 1,
|
||||
matched: 1,
|
||||
delivered: 1,
|
||||
clientReceived: 1,
|
||||
decoded: 1,
|
||||
applied: 1
|
||||
};
|
||||
|
||||
test("isolated replay classifies an empty producer result", () => {
|
||||
const result = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: "rpl_empty",
|
||||
traceId: "trc_empty",
|
||||
producer: { invoked: false, sourceMatched: null },
|
||||
consumer: readyConsumer,
|
||||
counts: {},
|
||||
terminalObserved: false
|
||||
});
|
||||
assert.deepEqual(result, {
|
||||
ok: false,
|
||||
phase: "producer",
|
||||
code: "producer_not_invoked",
|
||||
message: "No isolated debug producer was invoked for this replay."
|
||||
});
|
||||
});
|
||||
|
||||
test("isolated replay classifies matching events without terminal", () => {
|
||||
const result = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: "rpl_no_terminal",
|
||||
traceId: "trc_no_terminal",
|
||||
producer: { invoked: true, sourceMatched: true },
|
||||
consumer: readyConsumer,
|
||||
counts: deliveredCounts,
|
||||
terminalObserved: false
|
||||
});
|
||||
assert.equal(result.code, "terminal_missing");
|
||||
assert.equal(result.phase, "terminal");
|
||||
});
|
||||
|
||||
test("isolated replay classifies production reducer rejection", () => {
|
||||
const result = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: "rpl_reducer_rejected",
|
||||
traceId: "trc_reducer_rejected",
|
||||
producer: { invoked: true, sourceMatched: true },
|
||||
consumer: readyConsumer,
|
||||
counts: { ...deliveredCounts, applied: 0 },
|
||||
terminalObserved: true
|
||||
});
|
||||
assert.equal(result.code, "reducer_rejected");
|
||||
assert.equal(result.phase, "client");
|
||||
});
|
||||
|
||||
test("isolated replay classifies terminal completion", () => {
|
||||
const result = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: "rpl_complete",
|
||||
traceId: "trc_complete",
|
||||
producer: { invoked: true, sourceMatched: true },
|
||||
consumer: readyConsumer,
|
||||
counts: deliveredCounts,
|
||||
terminalObserved: true
|
||||
});
|
||||
assert.deepEqual(result, {
|
||||
ok: true,
|
||||
phase: "terminal",
|
||||
code: "terminal_complete",
|
||||
message: "The isolated replay reached a terminal event."
|
||||
});
|
||||
});
|
||||
|
||||
test("isolated replay keeps source, filter, and decoder failures distinct", () => {
|
||||
const sourceMissing = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: "rpl_source_missing",
|
||||
traceId: "trc_source_missing",
|
||||
producer: { invoked: true, sourceMatched: false },
|
||||
consumer: readyConsumer,
|
||||
counts: { scanned: 2, barrierScanned: 2, parsed: 2, traceMatched: 0 },
|
||||
terminalObserved: false
|
||||
});
|
||||
const filterMismatch = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: "rpl_filter_mismatch",
|
||||
traceId: "trc_filter_mismatch",
|
||||
producer: { invoked: true, sourceMatched: true },
|
||||
consumer: readyConsumer,
|
||||
counts: { scanned: 2, barrierScanned: 2, parsed: 2, traceMatched: 2, replayMatched: 0 },
|
||||
terminalObserved: false
|
||||
});
|
||||
const decoderRejected = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: "rpl_decoder_rejected",
|
||||
traceId: "trc_decoder_rejected",
|
||||
producer: { invoked: true, sourceMatched: true },
|
||||
consumer: readyConsumer,
|
||||
counts: { ...deliveredCounts, decoded: 0, applied: 0 },
|
||||
terminalObserved: true
|
||||
});
|
||||
assert.equal(sourceMissing.code, "source_trace_missing");
|
||||
assert.equal(filterMismatch.code, "records_scanned_but_filter_mismatch");
|
||||
assert.equal(decoderRejected.code, "decoder_rejected");
|
||||
});
|
||||
@@ -0,0 +1,260 @@
|
||||
// Responsibility: classify isolated Kafka debug replay progress without turning debug state into a product authority.
|
||||
|
||||
export const WORKBENCH_KAFKA_DEBUG_REPLAY_CONTRACT_VERSION = "workbench-kafka-debug-replay-v2";
|
||||
|
||||
export function createWorkbenchKafkaDebugReplayTracker(input = {}) {
|
||||
return {
|
||||
replayId: textValue(input.replayId),
|
||||
traceId: textValue(input.traceId),
|
||||
topic: textValue(input.topic),
|
||||
producer: {
|
||||
invoked: booleanOrNull(input.producer?.invoked),
|
||||
sourceMatched: booleanOrNull(input.producer?.sourceMatched),
|
||||
publishedCount: integerOrNull(input.producer?.publishedCount)
|
||||
},
|
||||
consumer: { ready: false, groupId: null },
|
||||
requestedOffsetRange: normalizeOffsetRange(input.offsetRange),
|
||||
counts: {
|
||||
scanned: 0,
|
||||
barrierScanned: 0,
|
||||
parsed: 0,
|
||||
traceMatched: 0,
|
||||
replayMatched: 0,
|
||||
matched: 0,
|
||||
delivered: 0,
|
||||
clientReceived: null,
|
||||
decoded: null,
|
||||
applied: null
|
||||
},
|
||||
rejectedByReason: {},
|
||||
offsets: {
|
||||
scanned: new Map(),
|
||||
matched: new Map(),
|
||||
delivered: new Map()
|
||||
},
|
||||
sourceLineage: null
|
||||
};
|
||||
}
|
||||
|
||||
export function markWorkbenchKafkaDebugReplayConsumerReady(tracker, input = {}) {
|
||||
tracker.consumer.ready = true;
|
||||
tracker.consumer.groupId = textValue(input.groupId);
|
||||
return tracker;
|
||||
}
|
||||
|
||||
export function observeWorkbenchKafkaDebugReplayRecord(tracker, observation = {}) {
|
||||
tracker.counts.scanned += 1;
|
||||
recordOffset(tracker.offsets.scanned, observation.partition, observation.offset);
|
||||
if (observation.inOffsetRange === false) {
|
||||
reject(tracker, "offset_range_mismatch");
|
||||
return tracker;
|
||||
}
|
||||
tracker.counts.barrierScanned += 1;
|
||||
if (observation.parsed !== true) {
|
||||
reject(tracker, "record_parse_rejected");
|
||||
return tracker;
|
||||
}
|
||||
tracker.counts.parsed += 1;
|
||||
if (observation.filterMatches?.traceId === true) tracker.counts.traceMatched += 1;
|
||||
if (observation.filterMatches?.replayId === true) tracker.counts.replayMatched += 1;
|
||||
if (observation.matched !== true) {
|
||||
reject(tracker, "filter_mismatch");
|
||||
return tracker;
|
||||
}
|
||||
tracker.counts.matched += 1;
|
||||
recordOffset(tracker.offsets.matched, observation.partition, observation.offset);
|
||||
return tracker;
|
||||
}
|
||||
|
||||
export function recordWorkbenchKafkaDebugReplayDelivery(tracker, record = {}, delivered = true) {
|
||||
if (!delivered) {
|
||||
reject(tracker, "sse_write_failed");
|
||||
return tracker;
|
||||
}
|
||||
tracker.counts.delivered += 1;
|
||||
recordOffset(tracker.offsets.delivered, record.partition, record.offset);
|
||||
if (!tracker.sourceLineage) tracker.sourceLineage = safeSourceLineage(record.value);
|
||||
return tracker;
|
||||
}
|
||||
|
||||
export function completeWorkbenchKafkaDebugReplay(tracker, input = {}) {
|
||||
const counts = {
|
||||
...tracker.counts,
|
||||
clientReceived: nullableCount(input.clientReceived, tracker.counts.clientReceived),
|
||||
decoded: nullableCount(input.decoded, tracker.counts.decoded),
|
||||
applied: nullableCount(input.applied, tracker.counts.applied)
|
||||
};
|
||||
const terminalObserved = input.terminalObserved === true;
|
||||
const result = classifyWorkbenchKafkaDebugReplayResult({
|
||||
replayId: tracker.replayId,
|
||||
traceId: tracker.traceId,
|
||||
producer: tracker.producer,
|
||||
consumer: tracker.consumer,
|
||||
requestedOffsetRange: tracker.requestedOffsetRange,
|
||||
counts,
|
||||
completionReason: textValue(input.reason),
|
||||
terminalObserved
|
||||
});
|
||||
return {
|
||||
contractVersion: WORKBENCH_KAFKA_DEBUG_REPLAY_CONTRACT_VERSION,
|
||||
replayId: tracker.replayId,
|
||||
traceId: tracker.traceId,
|
||||
topic: tracker.topic,
|
||||
producer: tracker.producer,
|
||||
consumer: tracker.consumer,
|
||||
phase: result.phase,
|
||||
code: result.code,
|
||||
ok: result.ok,
|
||||
message: result.message,
|
||||
completionReason: textValue(input.reason),
|
||||
terminalObserved,
|
||||
counts,
|
||||
rejectedByReason: { ...tracker.rejectedByReason },
|
||||
offsetRange: {
|
||||
requested: tracker.requestedOffsetRange,
|
||||
scanned: summarizeOffsets(tracker.offsets.scanned),
|
||||
matched: summarizeOffsets(tracker.offsets.matched),
|
||||
delivered: summarizeOffsets(tracker.offsets.delivered)
|
||||
},
|
||||
sourceLineage: tracker.sourceLineage,
|
||||
valuesPrinted: false
|
||||
};
|
||||
}
|
||||
|
||||
export function classifyWorkbenchKafkaDebugReplayResult(input = {}) {
|
||||
const counts = normalizeCounts(input.counts);
|
||||
const producer = input.producer ?? {};
|
||||
const consumer = input.consumer ?? {};
|
||||
const hasTraceFilter = Boolean(textValue(input.traceId));
|
||||
const hasReplayFilter = Boolean(textValue(input.replayId));
|
||||
const hasOffsetBarrier = Boolean(normalizeOffsetRange(input.requestedOffsetRange));
|
||||
|
||||
if (consumer.ready === false) return failure("consumer", "consumer_not_ready", "The isolated Kafka consumer did not reach its ready barrier.");
|
||||
if (producer.sourceMatched === false) return failure("producer", "source_trace_missing", "The source AgentRun topic did not contain the requested trace lineage.");
|
||||
if (producer.invoked === false && counts.matched === 0) return failure("producer", "producer_not_invoked", "No isolated debug producer was invoked for this replay.");
|
||||
if (hasOffsetBarrier && counts.barrierScanned === 0) return failure("kafka", "topic_no_append_for_replay", "The requested replay offset range was not appended or observed.");
|
||||
if (counts.scanned === 0) return failure("producer", "producer_not_invoked", "The isolated debug topic contained no records for this replay.");
|
||||
if (counts.barrierScanned > 0 && counts.parsed === 0) return failure("server", "record_parse_rejected", "Kafka records were scanned but none decoded as JSON objects.");
|
||||
if (hasTraceFilter && counts.traceMatched === 0) {
|
||||
if (producer.sourceMatched === true && producer.invoked === true) return failure("kafka", "topic_no_append_for_replay", "The producer reported success but the debug topic did not expose the requested trace append.");
|
||||
if (producer.sourceMatched === true) return failure("producer", "producer_not_invoked", "The source trace exists but no isolated debug producer append was observed.");
|
||||
return failure("producer", "source_trace_missing", "Kafka records were scanned but none carried the requested trace lineage.");
|
||||
}
|
||||
if (hasReplayFilter && counts.replayMatched === 0) return failure("server", "records_scanned_but_filter_mismatch", "Kafka records were scanned but none matched the requested replayId.");
|
||||
if (counts.matched === 0) return failure("server", "records_scanned_but_filter_mismatch", "Kafka records were scanned but none matched the replay filters.");
|
||||
if (counts.delivered < counts.matched && input.completionReason !== "limit") return failure("server", "sse_write_failed", "Matched Kafka records were not all delivered over SSE.");
|
||||
if (counts.clientReceived !== null && counts.clientReceived < counts.delivered) return failure("client", "transport_failed", "The browser received fewer records than the server delivered.");
|
||||
if (counts.decoded !== null && counts.clientReceived !== null && counts.decoded < counts.clientReceived) return failure("client", "decoder_rejected", "The browser decoder rejected one or more delivered records.");
|
||||
if (counts.applied !== null && counts.decoded !== null && counts.applied < counts.decoded) return failure("client", "reducer_rejected", "The production reducer rejected one or more decoded records.");
|
||||
if (input.terminalObserved !== true) return failure("terminal", "terminal_missing", "Matching events were delivered but no terminal event was observed.");
|
||||
return { ok: true, phase: "terminal", code: "terminal_complete", message: "The isolated replay reached a terminal event." };
|
||||
}
|
||||
|
||||
export function normalizeWorkbenchKafkaDebugOffsetRange(value) {
|
||||
return normalizeOffsetRange(value);
|
||||
}
|
||||
|
||||
function failure(phase, code, message) {
|
||||
return { ok: false, phase, code, message };
|
||||
}
|
||||
|
||||
function reject(tracker, reason) {
|
||||
tracker.rejectedByReason[reason] = (tracker.rejectedByReason[reason] ?? 0) + 1;
|
||||
}
|
||||
|
||||
function safeSourceLineage(value) {
|
||||
const lineage = objectValue(value?.debugLineage);
|
||||
const sourceEvent = objectValue(value?.sourceEvent);
|
||||
if (Object.keys(lineage).length === 0) return null;
|
||||
return {
|
||||
inputSchema: textValue(lineage.inputSchema),
|
||||
inputEventType: textValue(lineage.inputEventType),
|
||||
inputTraceId: textValue(lineage.inputTraceId),
|
||||
inputSessionId: textValue(lineage.inputSessionId),
|
||||
inputHwlabSessionId: textValue(lineage.inputHwlabSessionId),
|
||||
inputRunId: textValue(lineage.inputRunId),
|
||||
inputCommandId: textValue(lineage.inputCommandId),
|
||||
sourceSeq: integerOrNull(lineage.sourceSeq),
|
||||
sourceTopic: textValue(sourceEvent.topic),
|
||||
sourcePartition: integerOrNull(sourceEvent.partition),
|
||||
sourceOffset: kafkaOffset(sourceEvent.offset),
|
||||
valuesPrinted: false
|
||||
};
|
||||
}
|
||||
|
||||
function recordOffset(target, partitionValue, offsetValue) {
|
||||
const partition = integerOrNull(partitionValue);
|
||||
const offset = kafkaOffset(offsetValue);
|
||||
if (partition === null || offset === null) return;
|
||||
const current = target.get(partition);
|
||||
if (!current) {
|
||||
target.set(partition, { partition, firstOffset: offset, lastOffset: offset, count: 1 });
|
||||
return;
|
||||
}
|
||||
if (BigInt(offset) < BigInt(current.firstOffset)) current.firstOffset = offset;
|
||||
if (BigInt(offset) > BigInt(current.lastOffset)) current.lastOffset = offset;
|
||||
current.count += 1;
|
||||
}
|
||||
|
||||
function summarizeOffsets(target) {
|
||||
return [...target.values()].sort((left, right) => left.partition - right.partition).map((entry) => ({ ...entry }));
|
||||
}
|
||||
|
||||
function normalizeOffsetRange(value) {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
||||
const partition = integerOrNull(value.partition);
|
||||
const firstOffset = kafkaOffset(value.firstOffset ?? value.startOffset);
|
||||
const lastOffset = kafkaOffset(value.lastOffset ?? value.endOffset);
|
||||
if (partition === null || firstOffset === null || lastOffset === null || BigInt(firstOffset) > BigInt(lastOffset)) return null;
|
||||
return { partition, firstOffset, lastOffset };
|
||||
}
|
||||
|
||||
function normalizeCounts(value = {}) {
|
||||
const nullable = (name) => nullableCount(value[name], null);
|
||||
return {
|
||||
scanned: count(value.scanned),
|
||||
barrierScanned: count(value.barrierScanned),
|
||||
parsed: count(value.parsed),
|
||||
traceMatched: count(value.traceMatched),
|
||||
replayMatched: count(value.replayMatched),
|
||||
matched: count(value.matched),
|
||||
delivered: count(value.delivered),
|
||||
clientReceived: nullable("clientReceived"),
|
||||
decoded: nullable("decoded"),
|
||||
applied: nullable("applied")
|
||||
};
|
||||
}
|
||||
|
||||
function nullableCount(value, fallback) {
|
||||
const parsed = integerOrNull(value);
|
||||
return parsed === null || parsed < 0 ? fallback : parsed;
|
||||
}
|
||||
|
||||
function count(value) {
|
||||
return nullableCount(value, 0);
|
||||
}
|
||||
|
||||
function kafkaOffset(value) {
|
||||
const text = textValue(value);
|
||||
if (!text || !/^\d+$/u.test(text)) return null;
|
||||
try { return BigInt(text).toString(); } catch { return null; }
|
||||
}
|
||||
|
||||
function integerOrNull(value) {
|
||||
if (value === null || value === undefined || (typeof value === "string" && value.trim() === "")) return null;
|
||||
const parsed = Number(value);
|
||||
return Number.isSafeInteger(parsed) && parsed >= 0 ? parsed : null;
|
||||
}
|
||||
|
||||
function booleanOrNull(value) {
|
||||
return value === true ? true : value === false ? false : null;
|
||||
}
|
||||
|
||||
function objectValue(value) {
|
||||
return value && typeof value === "object" && !Array.isArray(value) ? value : {};
|
||||
}
|
||||
|
||||
function textValue(value) {
|
||||
const text = typeof value === "string" ? value.trim() : value === null || value === undefined ? "" : String(value).trim();
|
||||
return text.length > 0 ? text : null;
|
||||
}
|
||||
@@ -88,6 +88,49 @@ test("isolated Workbench Kafka replay uses its YAML-owned topic, unique group, a
|
||||
}
|
||||
});
|
||||
|
||||
test("isolated Workbench Kafka replay applies replayId and offset barriers with bounded counts", async () => {
|
||||
const fakeKafka = createFakeKafkaFactory();
|
||||
const env = {
|
||||
HWLAB_KAFKA_BOOTSTRAP_SERVERS: "127.0.0.1:9092",
|
||||
HWLAB_KAFKA_CLIENT_ID: "test-hwlab-cloud-api",
|
||||
HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_ENABLED: "true",
|
||||
HWLAB_KAFKA_HWLAB_DEBUG_EVENT_TOPIC: "hwlab.event.debug.v1",
|
||||
HWLAB_KAFKA_HWLAB_DEBUG_GROUP_PREFIX: "hwlab-test-workbench-isolated-debug",
|
||||
HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_LIMIT: "10",
|
||||
HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_TIMEOUT_MS: "3000"
|
||||
};
|
||||
const server = createServer((request, response) => {
|
||||
const url = new URL(request.url ?? "/", "http://127.0.0.1");
|
||||
void handleWorkbenchKafkaSseDebugHttp(request, response, url, { env, kafkaFactory: fakeKafka.factory, accessController: fakeAccessController(), logger: null });
|
||||
});
|
||||
await listen(server);
|
||||
try {
|
||||
const query = "stream=hwlab-debug&traceId=trc_barrier&replayId=rpl_barrier&fromBeginning=true&partition=0&firstOffset=10&lastOffset=11&producerInvoked=true&sourceMatched=true&publishedCount=2";
|
||||
const response = await fetch(`${serverUrl(server)}/v1/workbench/debug/kafka-sse/events?${query}`);
|
||||
const reader = response.body?.getReader();
|
||||
assert.ok(reader);
|
||||
const connected = await readUntil(reader, "hwlab.kafka.connected");
|
||||
assert.match(connected, /"replayId":"rpl_barrier"/u);
|
||||
assert.match(connected, /"firstOffset":"10"/u);
|
||||
|
||||
await fakeKafka.emit({ schema: "hwlab.event.debug.v1", traceId: "trc_barrier", replayId: "rpl_barrier", event: { type: "assistant" } }, { offset: "9" });
|
||||
await fakeKafka.emit({ schema: "hwlab.event.debug.v1", traceId: "trc_barrier", replayId: "rpl_barrier", debugLineage: { inputTraceId: "trc_barrier", inputSessionId: "ses_agentrun_barrier", sourceSeq: 1 }, event: { type: "assistant" } }, { offset: "10" });
|
||||
await fakeKafka.emit({ schema: "hwlab.event.debug.v1", traceId: "trc_barrier", replayId: "rpl_barrier", debugLineage: { inputTraceId: "trc_barrier", inputSessionId: "ses_agentrun_barrier", sourceSeq: 2 }, eventType: "terminal", event: { type: "result", terminal: true } }, { offset: "11" });
|
||||
const replay = await readUntil(reader, "hwlab.kafka.replay-complete");
|
||||
assert.match(replay, /"code":"terminal_complete"/u);
|
||||
assert.match(replay, /"scanned":3/u);
|
||||
assert.match(replay, /"barrierScanned":2/u);
|
||||
assert.match(replay, /"parsed":2/u);
|
||||
assert.match(replay, /"matched":2/u);
|
||||
assert.match(replay, /"delivered":2/u);
|
||||
assert.match(replay, /"offset_range_mismatch":1/u);
|
||||
assert.match(replay, /"firstOffset":"10","lastOffset":"11","count":2/u);
|
||||
assert.match(replay, /"inputSessionId":"ses_agentrun_barrier"/u);
|
||||
} finally {
|
||||
await close(server);
|
||||
}
|
||||
});
|
||||
|
||||
test("isolated Workbench Kafka capability rejects product topic and group identities", () => {
|
||||
const env = {
|
||||
HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_ENABLED: "true",
|
||||
@@ -137,12 +180,22 @@ test("isolated Workbench Kafka debug fails closed when disabled and rejects repl
|
||||
assert.equal(replay.status, 400);
|
||||
assert.match(await replay.text(), /workbench_kafka_debug_replay_required/u);
|
||||
assert.equal(fakeKafka.consumerCreated, false);
|
||||
|
||||
const invalidRange = await fetch(`${serverUrl(server)}/v1/workbench/debug/kafka-sse/events?stream=hwlab-debug&traceId=trc_replay&fromBeginning=true&partition=0&firstOffset=20`);
|
||||
assert.equal(invalidRange.status, 400);
|
||||
assert.match(await invalidRange.text(), /workbench_kafka_debug_offset_range_invalid/u);
|
||||
assert.equal(fakeKafka.consumerCreated, false);
|
||||
|
||||
const invalidReplayId = await fetch(`${serverUrl(server)}/v1/workbench/debug/kafka-sse/events?stream=hwlab-debug&traceId=trc_replay&fromBeginning=true&replayId=not-a-replay`);
|
||||
assert.equal(invalidReplayId.status, 400);
|
||||
assert.match(await invalidReplayId.text(), /workbench_kafka_debug_replay_id_invalid/u);
|
||||
assert.equal(fakeKafka.consumerCreated, false);
|
||||
} finally {
|
||||
await close(server);
|
||||
}
|
||||
});
|
||||
|
||||
test("isolated Workbench Kafka replay closes as incomplete when terminal is absent", async () => {
|
||||
test("isolated Workbench Kafka replay reports matched events without terminal", async () => {
|
||||
const fakeKafka = createFakeKafkaFactory();
|
||||
const env = {
|
||||
HWLAB_KAFKA_BOOTSTRAP_SERVERS: "127.0.0.1:9092",
|
||||
@@ -150,7 +203,7 @@ test("isolated Workbench Kafka replay closes as incomplete when terminal is abse
|
||||
HWLAB_KAFKA_HWLAB_DEBUG_EVENT_TOPIC: "hwlab.event.debug.v1",
|
||||
HWLAB_KAFKA_HWLAB_DEBUG_GROUP_PREFIX: "hwlab-test-workbench-isolated-debug",
|
||||
HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_LIMIT: "10",
|
||||
HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_TIMEOUT_MS: "30"
|
||||
HWLAB_WORKBENCH_KAFKA_DEBUG_REPLAY_TIMEOUT_MS: "100"
|
||||
};
|
||||
const server = createServer((request, response) => {
|
||||
const url = new URL(request.url ?? "/", "http://127.0.0.1");
|
||||
@@ -162,9 +215,12 @@ test("isolated Workbench Kafka replay closes as incomplete when terminal is abse
|
||||
assert.equal(response.status, 200);
|
||||
const reader = response.body?.getReader();
|
||||
assert.ok(reader);
|
||||
await readUntil(reader, "hwlab.kafka.connected");
|
||||
await fakeKafka.emit({ schema: "hwlab.event.debug.v1", traceId: "trc_incomplete", eventType: "assistant", event: { traceId: "trc_incomplete", type: "assistant" } }, { offset: "5" });
|
||||
const replay = await readUntil(reader, "hwlab.kafka.replay-complete");
|
||||
assert.match(replay, /"reason":"timeout"/u);
|
||||
assert.match(replay, /"count":0/u);
|
||||
assert.match(replay, /"count":1/u);
|
||||
assert.match(replay, /"code":"terminal_missing"/u);
|
||||
assert.match(replay, /"terminalObserved":false/u);
|
||||
await waitUntil(() => fakeKafka.stopCalls > 0);
|
||||
} finally {
|
||||
@@ -380,13 +436,13 @@ function createFakeKafkaFactory(options: { deferRun?: boolean } = {}) {
|
||||
get disconnectCalls() { return disconnectCalls; },
|
||||
get consumerCreated() { return consumerCreated; },
|
||||
releaseRun: () => releaseRun?.(),
|
||||
emit: async (value: Record<string, unknown>) => {
|
||||
emit: async (value: Record<string, unknown>, transport: { offset?: string; partition?: number } = {}) => {
|
||||
assert.ok(eachMessage, "consumer.run must be called before emitting fake Kafka events");
|
||||
await eachMessage({
|
||||
topic: subscribedTopic,
|
||||
partition: 0,
|
||||
partition: transport.partition ?? 0,
|
||||
message: {
|
||||
offset: String(value.sessionId === "ses_other" ? 1 : 2),
|
||||
offset: transport.offset ?? String(value.sessionId === "ses_other" ? 1 : 2),
|
||||
key: Buffer.from(String(value.sessionId ?? "unknown")),
|
||||
timestamp: "2026-07-09T18:30:00.000Z",
|
||||
value: Buffer.from(JSON.stringify(value))
|
||||
|
||||
@@ -7,6 +7,14 @@ import { openKafkaEventStream } from "./kafka-event-bridge.ts";
|
||||
import { defaultCodeAgentTraceStore } from "./code-agent-trace-store.ts";
|
||||
import { authenticateWorkbenchRead } from "./server-workbench-read-http.ts";
|
||||
import { workbenchKafkaDebugCapability } from "./workbench-kafka-debug-capability.ts";
|
||||
import {
|
||||
completeWorkbenchKafkaDebugReplay,
|
||||
createWorkbenchKafkaDebugReplayTracker,
|
||||
markWorkbenchKafkaDebugReplayConsumerReady,
|
||||
normalizeWorkbenchKafkaDebugOffsetRange,
|
||||
observeWorkbenchKafkaDebugReplayRecord,
|
||||
recordWorkbenchKafkaDebugReplayDelivery
|
||||
} from "./workbench-kafka-debug-replay-contract.ts";
|
||||
|
||||
const CONTRACT_VERSION = "workbench-debug-kafka-sse-v1";
|
||||
const DEFAULT_STREAM = "hwlab";
|
||||
@@ -48,7 +56,7 @@ export async function handleWorkbenchKafkaSseDebugHttp(request, response, url, o
|
||||
function describeKafkaSseDebug(url, env, isolatedDebug = null) {
|
||||
const stream = streamFromUrl(url);
|
||||
const requestedFilters = filtersFromUrl(url);
|
||||
const resolvedFilters = stream === "hwlab-debug" ? compactObject({ traceId: requestedFilters.traceId }) : resolveKafkaDebugFilters(requestedFilters, { traceStore: defaultCodeAgentTraceStore });
|
||||
const resolvedFilters = stream === "hwlab-debug" ? compactObject({ traceId: requestedFilters.traceId, replayId: requestedFilters.replayId }) : resolveKafkaDebugFilters(requestedFilters, { traceStore: defaultCodeAgentTraceStore });
|
||||
return {
|
||||
ok: true,
|
||||
contractVersion: CONTRACT_VERSION,
|
||||
@@ -60,6 +68,7 @@ function describeKafkaSseDebug(url, env, isolatedDebug = null) {
|
||||
replayLimit: isolatedDebug?.replayLimit ?? null,
|
||||
replayTimeoutMs: isolatedDebug?.replayTimeoutMs ?? null,
|
||||
groupIdPrefix: isolatedDebug?.groupIdPrefix ?? null,
|
||||
replayRequest: isolatedDebug?.replayRequest ?? null,
|
||||
filters: requestedFilters,
|
||||
resolvedFilters,
|
||||
eventsRoute: `/v1/workbench/debug/kafka-sse/events${url.search || ""}`,
|
||||
@@ -72,7 +81,15 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
const stream = streamFromUrl(url);
|
||||
const filters = filtersFromUrl(url);
|
||||
const isolatedReplay = stream === "hwlab-debug";
|
||||
const resolvedFilters = isolatedReplay ? compactObject({ traceId: filters.traceId }) : resolveKafkaDebugFilters(filters, options);
|
||||
const resolvedFilters = isolatedReplay ? compactObject({ traceId: filters.traceId, replayId: filters.replayId }) : resolveKafkaDebugFilters(filters, options);
|
||||
const replayRequest = isolatedDebug?.replayRequest ?? replayRequestFromUrl(url);
|
||||
const replayTracker = isolatedReplay ? createWorkbenchKafkaDebugReplayTracker({
|
||||
replayId: resolvedFilters.replayId,
|
||||
traceId: resolvedFilters.traceId,
|
||||
topic: isolatedDebug.topic,
|
||||
offsetRange: replayRequest.offsetRange,
|
||||
producer: replayRequest.producer
|
||||
}) : null;
|
||||
response.writeHead(200, {
|
||||
"content-type": "text/event-stream; charset=utf-8",
|
||||
"cache-control": "no-store, no-transform",
|
||||
@@ -102,6 +119,10 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
if (!isolatedReplay || replayFinished || closed) return;
|
||||
replayFinished = true;
|
||||
if (replayTimer) clearTimeout(replayTimer);
|
||||
const replayResult = completeWorkbenchKafkaDebugReplay(replayTracker, {
|
||||
reason,
|
||||
terminalObserved: replayTerminalObserved
|
||||
});
|
||||
writeSse(response, "hwlab.kafka.replay-complete", {
|
||||
ok: true,
|
||||
contractVersion: CONTRACT_VERSION,
|
||||
@@ -109,8 +130,16 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
topic: isolatedDebug.topic,
|
||||
traceId: resolvedFilters.traceId,
|
||||
replayComplete: true,
|
||||
replayId: replayResult.replayId,
|
||||
phase: replayResult.phase,
|
||||
code: replayResult.code,
|
||||
result: replayResult,
|
||||
reason,
|
||||
count: replayCount,
|
||||
counts: replayResult.counts,
|
||||
rejectedByReason: replayResult.rejectedByReason,
|
||||
offsetRange: replayResult.offsetRange,
|
||||
sourceLineage: replayResult.sourceLineage,
|
||||
limit: isolatedDebug.replayLimit,
|
||||
timeoutMs: isolatedDebug.replayTimeoutMs,
|
||||
terminalObserved: replayTerminalObserved,
|
||||
@@ -120,7 +149,7 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
};
|
||||
const writeRecord = (record) => {
|
||||
if (closed || replayFinished) return;
|
||||
writeSse(response, "hwlab.kafka.event", {
|
||||
const delivered = writeSse(response, "hwlab.kafka.event", {
|
||||
ok: true,
|
||||
contractVersion: CONTRACT_VERSION,
|
||||
stream,
|
||||
@@ -134,8 +163,9 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
valuesPrinted: false
|
||||
});
|
||||
if (!isolatedReplay) return;
|
||||
replayCount += 1;
|
||||
replayTerminalObserved = debugRecordIsTerminal(record);
|
||||
recordWorkbenchKafkaDebugReplayDelivery(replayTracker, record, delivered);
|
||||
replayCount = replayTracker.counts.delivered;
|
||||
replayTerminalObserved ||= debugRecordIsTerminal(record);
|
||||
if (replayTerminalObserved) finishReplay("terminal");
|
||||
else if (replayCount >= isolatedDebug.replayLimit) finishReplay("limit");
|
||||
};
|
||||
@@ -151,7 +181,11 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
groupIdPrefix: isolatedDebug?.groupIdPrefix ?? null,
|
||||
fromBeginning: isolatedReplay || url.searchParams.get("fromBeginning") === "1" || url.searchParams.get("fromBeginning") === "true",
|
||||
...resolvedFilters,
|
||||
offsetRange: replayRequest.offsetRange,
|
||||
kafkaFactory: options.kafkaFactory,
|
||||
onRecord: isolatedReplay ? async (observation) => {
|
||||
observeWorkbenchKafkaDebugReplayRecord(replayTracker, observation);
|
||||
} : null,
|
||||
onEvent: async (record) => {
|
||||
if (isolatedReplay && !consumerReady) {
|
||||
if (bufferedRecords.length < isolatedDebug.replayLimit) bufferedRecords.push(record);
|
||||
@@ -164,6 +198,7 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
await kafkaStream.stop?.();
|
||||
return;
|
||||
}
|
||||
if (isolatedReplay) markWorkbenchKafkaDebugReplayConsumerReady(replayTracker, { groupId: kafkaStream.groupId });
|
||||
writeSse(response, "hwlab.kafka.connected", {
|
||||
ok: true,
|
||||
contractVersion: CONTRACT_VERSION,
|
||||
@@ -177,6 +212,9 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
replay: isolatedReplay,
|
||||
replayLimit: isolatedDebug?.replayLimit ?? null,
|
||||
replayTimeoutMs: isolatedDebug?.replayTimeoutMs ?? null,
|
||||
replayId: replayTracker?.replayId ?? null,
|
||||
offsetRange: replayTracker?.requestedOffsetRange ?? null,
|
||||
replayContractVersion: replayTracker ? "workbench-kafka-debug-replay-v2" : null,
|
||||
filters,
|
||||
resolvedFilters,
|
||||
serverSentAt: new Date().toISOString(),
|
||||
@@ -190,7 +228,7 @@ async function openKafkaDebugSse(request, response, url, options, isolatedDebug
|
||||
} catch (error) {
|
||||
if (replayTimer) clearTimeout(replayTimer);
|
||||
writeSse(response, "hwlab.kafka.error", { ok: false, error: errorMessagePayload(error), valuesPrinted: false });
|
||||
if (isolatedReplay && !response.writableEnded) response.end();
|
||||
if (isolatedReplay) finishReplay("error");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -254,7 +292,8 @@ function filtersFromUrl(url) {
|
||||
traceId: safeId(url.searchParams.get("traceId") || url.searchParams.get("trace-id")),
|
||||
sessionId: safeId(url.searchParams.get("sessionId") || url.searchParams.get("session-id")),
|
||||
runId: safeId(url.searchParams.get("runId") || url.searchParams.get("run-id")),
|
||||
commandId: safeId(url.searchParams.get("commandId") || url.searchParams.get("command-id"))
|
||||
commandId: safeId(url.searchParams.get("commandId") || url.searchParams.get("command-id")),
|
||||
replayId: safeReplayId(url.searchParams.get("replayId") || url.searchParams.get("replay-id"))
|
||||
});
|
||||
}
|
||||
|
||||
@@ -285,7 +324,38 @@ function isolatedDebugConfig(response, url, env, options = {}) {
|
||||
sendJson(response, 400, debugError("workbench_kafka_debug_replay_required", "Isolated Workbench Kafka debug requires explicit fromBeginning=true."));
|
||||
return false;
|
||||
}
|
||||
return config;
|
||||
const rawReplayId = url.searchParams.get("replayId") || url.searchParams.get("replay-id");
|
||||
if (textValue(rawReplayId) && !safeReplayId(rawReplayId)) {
|
||||
sendJson(response, 400, debugError("workbench_kafka_debug_replay_id_invalid", "Replay correlation requires replayId to start with rpl_ and contain only safe identifier characters."));
|
||||
return false;
|
||||
}
|
||||
const replayRequest = replayRequestFromUrl(url);
|
||||
if (options.requireReplay && replayRequest.offsetRangeInvalid) {
|
||||
sendJson(response, 400, debugError("workbench_kafka_debug_offset_range_invalid", "Replay offset barrier requires partition, firstOffset, and lastOffset with firstOffset <= lastOffset."));
|
||||
return false;
|
||||
}
|
||||
return { ...config, replayRequest };
|
||||
}
|
||||
|
||||
function replayRequestFromUrl(url) {
|
||||
const rawRange = {
|
||||
partition: url.searchParams.get("partition"),
|
||||
firstOffset: url.searchParams.get("firstOffset") || url.searchParams.get("first-offset") || url.searchParams.get("startOffset") || url.searchParams.get("start-offset"),
|
||||
lastOffset: url.searchParams.get("lastOffset") || url.searchParams.get("last-offset") || url.searchParams.get("endOffset") || url.searchParams.get("end-offset")
|
||||
};
|
||||
const offsetRangeProvided = Object.values(rawRange).some((entry) => textValue(entry));
|
||||
const offsetRange = normalizeWorkbenchKafkaDebugOffsetRange(rawRange);
|
||||
return {
|
||||
replayId: safeReplayId(url.searchParams.get("replayId") || url.searchParams.get("replay-id")),
|
||||
offsetRange,
|
||||
offsetRangeInvalid: offsetRangeProvided && !offsetRange,
|
||||
producer: {
|
||||
invoked: optionalBoolean(url.searchParams.get("producerInvoked") || url.searchParams.get("producer-invoked")),
|
||||
sourceMatched: optionalBoolean(url.searchParams.get("sourceMatched") || url.searchParams.get("source-matched")),
|
||||
publishedCount: optionalNonNegativeInteger(url.searchParams.get("publishedCount") || url.searchParams.get("published-count"))
|
||||
},
|
||||
valuesPrinted: false
|
||||
};
|
||||
}
|
||||
|
||||
function methodNotAllowed(response, allow) {
|
||||
@@ -313,6 +383,24 @@ function safeId(value) {
|
||||
return text && /^[A-Za-z0-9_.:-]{3,220}$/u.test(text) ? text : null;
|
||||
}
|
||||
|
||||
function safeReplayId(value) {
|
||||
const text = safeId(value);
|
||||
return text && /^rpl_[A-Za-z0-9_.:-]+$/u.test(text) ? text : null;
|
||||
}
|
||||
|
||||
function optionalBoolean(value) {
|
||||
const text = textValue(value)?.toLowerCase();
|
||||
if (text === "true" || text === "1") return true;
|
||||
if (text === "false" || text === "0") return false;
|
||||
return null;
|
||||
}
|
||||
|
||||
function optionalNonNegativeInteger(value) {
|
||||
if (!textValue(value)) return null;
|
||||
const parsed = Number(value);
|
||||
return Number.isSafeInteger(parsed) && parsed >= 0 ? parsed : null;
|
||||
}
|
||||
|
||||
function textValue(value) {
|
||||
const text = typeof value === "string" ? value.trim() : value === null || value === undefined ? "" : String(value).trim();
|
||||
return text.length > 0 ? text : null;
|
||||
|
||||
@@ -25,6 +25,7 @@ test("offline JSONL maps 35 stdio reconstructions to 35 ordered HWLAB debug even
|
||||
"--from", "jsonl",
|
||||
"--session-id", SESSION_ID,
|
||||
"--trace-id", TRACE_ID,
|
||||
"--replay-id", "rpl_offline_35",
|
||||
"--jsonl-file", inputFile,
|
||||
"--output-jsonl", outputFile,
|
||||
"--expect-count", "35",
|
||||
@@ -62,6 +63,9 @@ test("offline JSONL maps 35 stdio reconstructions to 35 ordered HWLAB debug even
|
||||
assert.deepEqual(output.map((row) => row.value.debugLineage.sourceSeq), Array.from({ length: 35 }, (_, index) => index + 1));
|
||||
assert.ok(output.every((row) => row.topic === "hwlab.event.debug.v1"));
|
||||
assert.ok(output.every((row) => row.value.schema === "hwlab.event.debug.v1"));
|
||||
assert.ok(output.every((row) => row.partition === 0));
|
||||
assert.ok(output.every((row) => row.value.replayId === "rpl_offline_35"));
|
||||
assert.deepEqual(output.map((row) => row.value.debugReplay.index), Array.from({ length: 35 }, (_, index) => index + 1));
|
||||
assert.ok(output.every((row) => row.value.eventId === null && row.value.sourceEventId === null && row.value.outboxSeq === null));
|
||||
assert.ok(output.every((row) => row.value.debugLineage.inputEventId === null && row.value.debugLineage.inputOutboxSeq === null));
|
||||
assert.ok(output.every((row) => row.value.sourceEvent.partition === null && row.value.sourceEvent.offset === null));
|
||||
@@ -86,7 +90,9 @@ test("Kafka reader and producer are injected while canonical AgentRun events use
|
||||
"--from", "kafka",
|
||||
"--session-id", SESSION_ID,
|
||||
"--trace-id", TRACE_ID,
|
||||
"--replay-id", "rpl_canonical_35",
|
||||
"--input-topic", "agentrun.event.v1",
|
||||
"--publish",
|
||||
"--expect-count", "35",
|
||||
"--output-jsonl", path.join(cwd, "canonical-output.jsonl"),
|
||||
"--json"
|
||||
@@ -121,7 +127,10 @@ test("Kafka reader and producer are injected while canonical AgentRun events use
|
||||
assert.equal(producerCalls.length, 1);
|
||||
assert.equal(producerCalls[0].topic, "hwlab.event.debug.v1");
|
||||
assert.equal(producerCalls[0].messages.length, 35);
|
||||
assert.ok(producerCalls[0].messages.every((message: any) => message.partition === 0));
|
||||
const envelopes = producerCalls[0].messages.map((message: any) => JSON.parse(message.value));
|
||||
assert.ok(envelopes.every((event: any) => event.replayId === "rpl_canonical_35"));
|
||||
assert.ok(producerCalls[0].messages.every((message: any) => message.headers["x-debug-replay-id"] === "rpl_canonical_35"));
|
||||
assert.deepEqual(envelopes.map((event: any) => event.debugLineage.sourceSeq), Array.from({ length: 35 }, (_, index) => index + 1));
|
||||
assert.deepEqual(envelopes.map((event: any) => event.eventId), Array.from({ length: 35 }, (_, index) => `hwlab:evt_${index + 1}`));
|
||||
assert.deepEqual(envelopes.map((event: any) => event.debugLineage.inputOutboxSeq), Array.from({ length: 35 }, (_, index) => index + 1));
|
||||
@@ -132,6 +141,100 @@ test("Kafka reader and producer are injected while canonical AgentRun events use
|
||||
assert.deepEqual(result.payload.config.groupPrefix, { value: "hwlab-v03-workbench-isolated-debug", source: "env:HWLAB_KAFKA_HWLAB_DEBUG_GROUP_PREFIX" });
|
||||
assert.equal(result.payload.validation.orderPreserved, true);
|
||||
assert.equal(result.payload.validation.lineagePreserved, true);
|
||||
assert.deepEqual(result.payload.output.offsetRanges, [{
|
||||
topic: "hwlab.event.debug.v1",
|
||||
partition: 0,
|
||||
firstOffset: "100",
|
||||
lastOffset: "134",
|
||||
count: 35,
|
||||
valuesPrinted: false
|
||||
}]);
|
||||
assert.equal(result.payload.replay.replayId, "rpl_canonical_35");
|
||||
assert.equal(result.payload.replay.producerInvoked, true);
|
||||
assert.equal(result.payload.replay.sourceMatched, true);
|
||||
assert.deepEqual(result.payload.replay.outputBarrier, result.payload.output.offsetRanges[0]);
|
||||
});
|
||||
|
||||
test("Kafka regeneration reports source trace missing before creating a producer", async () => {
|
||||
let producerCreated = false;
|
||||
let readerInput: any = null;
|
||||
const result = await runKafkaCli([
|
||||
"regenerate", "hwlab",
|
||||
"--from", "kafka",
|
||||
"--session-id", SESSION_ID,
|
||||
"--trace-id", "trc_missing_debug_source",
|
||||
"--replay-id", "rpl_missing_debug_source",
|
||||
"--group-prefix", "hwlab-v03-workbench-isolated-debug"
|
||||
], {
|
||||
env: {},
|
||||
now: () => "2026-07-10T12:00:00.000Z",
|
||||
async readKafka(input) { readerInput = input; return { events: [canonicalRecord(1)] }; },
|
||||
async createProducer() { producerCreated = true; throw new Error("must not create producer"); }
|
||||
});
|
||||
|
||||
assert.equal(result.exitCode, 1);
|
||||
assert.equal(result.payload.error.code, "source_trace_missing");
|
||||
assert.equal(readerInput.topic, "agentrun.event.v1");
|
||||
assert.equal(producerCreated, false);
|
||||
});
|
||||
|
||||
test("canonical Kafka regeneration preflights without publishing and preserves replayId in the publish hint", async () => {
|
||||
const cwd = await mkdtemp(path.join(os.tmpdir(), "hwlab-kafka-debug-preflight-"));
|
||||
let producerCreated = false;
|
||||
const result = await runKafkaCli([
|
||||
"regenerate", "hwlab",
|
||||
"--from", "kafka",
|
||||
"--session-id", SESSION_ID,
|
||||
"--trace-id", TRACE_ID,
|
||||
"--replay-id", "rpl_canonical_preflight",
|
||||
"--group-prefix", "hwlab-v03-workbench-isolated-debug",
|
||||
"--output-jsonl", path.join(cwd, "output.jsonl")
|
||||
], {
|
||||
cwd,
|
||||
env: {},
|
||||
now: () => "2026-07-10T12:00:00.000Z",
|
||||
async readKafka() { return { events: [canonicalRecord(1)] }; },
|
||||
async createProducer() { producerCreated = true; throw new Error("preflight must not create a producer"); }
|
||||
});
|
||||
|
||||
assert.equal(result.exitCode, 0);
|
||||
assert.equal(result.payload.config.inputTopic.value, "agentrun.event.v1");
|
||||
assert.equal(result.payload.output.published, false);
|
||||
assert.equal(result.payload.output.publishedCount, 0);
|
||||
assert.equal(result.payload.replay.producerInvoked, false);
|
||||
assert.equal(result.payload.replay.outputBarrier, null);
|
||||
assert.match(result.payload.next.command, /--replay-id rpl_canonical_preflight/u);
|
||||
assert.match(result.payload.next.command, /--publish/u);
|
||||
assert.equal(producerCreated, false);
|
||||
});
|
||||
|
||||
test("Kafka regeneration fails closed when publish metadata cannot form an offset barrier", async () => {
|
||||
const cwd = await mkdtemp(path.join(os.tmpdir(), "hwlab-kafka-debug-barrier-"));
|
||||
const result = await runKafkaCli([
|
||||
"regenerate", "hwlab",
|
||||
"--from", "kafka",
|
||||
"--session-id", SESSION_ID,
|
||||
"--trace-id", TRACE_ID,
|
||||
"--replay-id", "rpl_missing_offset_barrier",
|
||||
"--group-prefix", "hwlab-v03-workbench-isolated-debug",
|
||||
"--publish",
|
||||
"--output-jsonl", path.join(cwd, "output.jsonl")
|
||||
], {
|
||||
cwd,
|
||||
env: {},
|
||||
now: () => "2026-07-10T12:00:00.000Z",
|
||||
async readKafka() { return { events: [canonicalRecord(1)] }; },
|
||||
async createProducer() {
|
||||
return {
|
||||
async connect() {},
|
||||
async send() { return []; },
|
||||
async disconnect() {}
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
assert.equal(result.exitCode, 1);
|
||||
assert.equal(result.payload.error.code, "debug_publish_barrier_missing");
|
||||
});
|
||||
|
||||
test("reconstruction identity fabrication and product output topic fail closed", async () => {
|
||||
@@ -161,6 +264,13 @@ test("reconstruction identity fabrication and product output topic fail closed",
|
||||
], { cwd, env: {}, now: () => "2026-07-10T12:00:00.000Z" });
|
||||
assert.equal(productGroupResult.exitCode, 1);
|
||||
assert.equal(productGroupResult.payload.error.code, "debug_group_required");
|
||||
|
||||
const replayIdResult = await runKafkaCli([
|
||||
"regenerate", "hwlab", "--from", "jsonl", "--session-id", SESSION_ID,
|
||||
"--replay-id", "not-a-replay", "--jsonl-file", inputFile, "--no-publish"
|
||||
], { cwd, env: {}, now: () => "2026-07-10T12:00:00.000Z" });
|
||||
assert.equal(replayIdResult.exitCode, 1);
|
||||
assert.equal(replayIdResult.payload.error.code, "invalid_replay_id");
|
||||
});
|
||||
|
||||
test("Kafka mode requires a CLI or YAML-owned isolated group prefix", async () => {
|
||||
|
||||
@@ -9,15 +9,17 @@ import { Kafka, logLevel } from "kafkajs";
|
||||
import {
|
||||
buildHwlabKafkaProducerMessage,
|
||||
DEFAULT_AGENTRUN_DEBUG_EVENT_TOPIC,
|
||||
DEFAULT_AGENTRUN_EVENT_TOPIC,
|
||||
DEFAULT_HWLAB_DEBUG_EVENT_TOPIC,
|
||||
projectAgentRunKafkaMessageToHwlabDebugEvent,
|
||||
queryKafkaEventStream
|
||||
} from "../../../internal/cloud/kafka-event-bridge.ts";
|
||||
|
||||
const CLI_NAME = "hwlab-cli";
|
||||
const VERSION = "0.3.0-kafka-debug";
|
||||
const VERSION = "0.3.1-kafka-debug-replay";
|
||||
const DEFAULT_LIMIT = 500;
|
||||
const DEFAULT_TIMEOUT_MS = 5000;
|
||||
const DEBUG_OUTPUT_PARTITION = 0;
|
||||
|
||||
type JsonRecord = Record<string, any>;
|
||||
type EnvLike = Record<string, string | undefined>;
|
||||
@@ -66,8 +68,11 @@ export async function runKafkaCli(argv: string[], options: KafkaCliOptions = {})
|
||||
export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencies: Required<KafkaCliOptions>) {
|
||||
const sessionId = requiredSessionId(parsed.sessionId);
|
||||
const traceId = optionalTraceId(parsed.traceId);
|
||||
const replayId = optionalReplayId(parsed.replayId) || `rpl_${randomUUID().replaceAll("-", "")}`;
|
||||
const sourceMode = enumValue(parsed.from ?? "kafka", "from", ["kafka", "jsonl"]);
|
||||
const inputTopicResolution = resolveConfig(parsed.inputTopic, undefined, DEFAULT_AGENTRUN_DEBUG_EVENT_TOPIC, "--input-topic", "contract-default");
|
||||
const inputTopicResolution = sourceMode === "kafka"
|
||||
? resolveConfig(parsed.inputTopic, dependencies.env.HWLAB_KAFKA_AGENTRUN_EVENT_TOPIC, DEFAULT_AGENTRUN_EVENT_TOPIC, "--input-topic", "env:HWLAB_KAFKA_AGENTRUN_EVENT_TOPIC")
|
||||
: resolveConfig(parsed.inputTopic, undefined, DEFAULT_AGENTRUN_DEBUG_EVENT_TOPIC, "--input-topic", "contract-default");
|
||||
const outputTopicResolution = resolveConfig(parsed.outputTopic, dependencies.env.HWLAB_KAFKA_HWLAB_DEBUG_EVENT_TOPIC, DEFAULT_HWLAB_DEBUG_EVENT_TOPIC, "--output-topic", "env:HWLAB_KAFKA_HWLAB_DEBUG_EVENT_TOPIC");
|
||||
const inputTopic = inputTopicResolution.value;
|
||||
const outputTopic = outputTopicResolution.value;
|
||||
@@ -76,7 +81,7 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
const timeoutMs = boundedInteger(parsed.timeoutMs, DEFAULT_TIMEOUT_MS, 250, 60000, "timeoutMs");
|
||||
const expectedCount = optionalPositiveInteger(parsed.expectCount, "expectCount");
|
||||
const readLimit = expectedCount === undefined ? limit : Math.min(5000, expectedCount + 1);
|
||||
const shouldPublish = parsed.noPublish === true ? false : parsed.publish === true ? true : sourceMode === "kafka";
|
||||
const shouldPublish = parsed.publish === true && parsed.noPublish !== true;
|
||||
const groupPrefixResolution = resolveKafkaGroupConfig(
|
||||
parsed.groupPrefix,
|
||||
dependencies.env.HWLAB_KAFKA_HWLAB_DEBUG_GROUP_PREFIX,
|
||||
@@ -103,7 +108,8 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
sessionId,
|
||||
traceId,
|
||||
expectedCount,
|
||||
producedAt: dependencies.now()
|
||||
producedAt: dependencies.now(),
|
||||
replayId
|
||||
});
|
||||
|
||||
const stateId = `hwlab-debug-${Date.now().toString(36)}-${randomUUID().slice(0, 8)}`;
|
||||
@@ -115,7 +121,7 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
const message = buildHwlabKafkaProducerMessage(envelope);
|
||||
return {
|
||||
topic: outputTopic,
|
||||
partition: null,
|
||||
partition: DEBUG_OUTPUT_PARTITION,
|
||||
offset: null,
|
||||
key: message.key,
|
||||
value: envelope,
|
||||
@@ -134,7 +140,7 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
await producer.connect?.();
|
||||
publishMetadata = await producer.send({
|
||||
topic: outputTopic,
|
||||
messages: mapped.events.map(buildHwlabKafkaProducerMessage)
|
||||
messages: mapped.events.map((event) => ({ ...buildHwlabKafkaProducerMessage(event), partition: DEBUG_OUTPUT_PARTITION }))
|
||||
});
|
||||
publishedCount = mapped.events.length;
|
||||
} catch (error) {
|
||||
@@ -152,11 +158,21 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
|
||||
const first = mapped.events[0];
|
||||
const last = mapped.events.at(-1);
|
||||
const offsetRanges = publishOffsetRanges(publishMetadata, publishedCount, DEBUG_OUTPUT_PARTITION);
|
||||
if (shouldPublish && offsetRanges.length !== 1) {
|
||||
throw cliError("debug_publish_barrier_missing", "Kafka publish succeeded without a usable isolated replay offset barrier", {
|
||||
replayId,
|
||||
topic: outputTopic,
|
||||
partition: DEBUG_OUTPUT_PARTITION,
|
||||
publishedCount,
|
||||
metadataCount: Array.isArray(publishMetadata) ? publishMetadata.length : 0
|
||||
});
|
||||
}
|
||||
const sourceFile = text(readResult.sourceFile);
|
||||
const traceArgument = traceId ? ` --trace-id ${traceId}` : "";
|
||||
const baseCommand = sourceMode === "jsonl"
|
||||
? `hwlab-cli kafka regenerate hwlab --from jsonl --session-id ${sessionId}${traceArgument} --jsonl-file ${shellArg(sourceFile || String(parsed.jsonlFile))}`
|
||||
: `hwlab-cli kafka regenerate hwlab --from kafka --session-id ${sessionId}${traceArgument} --input-topic ${inputTopic}`;
|
||||
? `hwlab-cli kafka regenerate hwlab --from jsonl --session-id ${sessionId}${traceArgument} --replay-id ${replayId} --jsonl-file ${shellArg(sourceFile || String(parsed.jsonlFile))}`
|
||||
: `hwlab-cli kafka regenerate hwlab --from kafka --session-id ${sessionId}${traceArgument} --replay-id ${replayId} --input-topic ${inputTopic}`;
|
||||
const nextCommand = shouldPublish
|
||||
? `${baseCommand} --no-publish --output-topic ${outputTopic} --expect-count ${mapped.events.length} --json`
|
||||
: `${baseCommand} --publish --output-topic ${outputTopic} --expect-count ${mapped.events.length} --json`;
|
||||
@@ -164,6 +180,7 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
ok: true,
|
||||
action: "kafka.regenerate.hwlab",
|
||||
status: "succeeded",
|
||||
replayId,
|
||||
sourceMode,
|
||||
config: {
|
||||
inputTopic: inputTopicResolution,
|
||||
@@ -185,6 +202,7 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
published: shouldPublish,
|
||||
publishedCount,
|
||||
metadataCount: Array.isArray(publishMetadata) ? publishMetadata.length : 0,
|
||||
offsetRanges,
|
||||
artifact: {
|
||||
path: artifactPath,
|
||||
bytes: Buffer.byteLength(artifactText),
|
||||
@@ -200,6 +218,27 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
lastSourceSeq: last?.debugLineage?.sourceSeq ?? null,
|
||||
inputKinds: [...new Set(mapped.inputKinds)]
|
||||
},
|
||||
replay: {
|
||||
replayId,
|
||||
producerInvoked: shouldPublish,
|
||||
sourceMatched: true,
|
||||
sourceLineage: {
|
||||
traceId: first?.debugLineage?.inputTraceId ?? traceId ?? null,
|
||||
sessionId: first?.debugLineage?.inputSessionId ?? sessionId,
|
||||
hwlabSessionId: first?.debugLineage?.inputHwlabSessionId ?? null,
|
||||
runId: first?.debugLineage?.inputRunId ?? null,
|
||||
commandId: first?.debugLineage?.inputCommandId ?? null,
|
||||
inputTopic,
|
||||
sourcePartition: first?.sourceEvent?.partition ?? null,
|
||||
firstSourceOffset: first?.sourceEvent?.offset ?? null,
|
||||
lastSourceOffset: last?.sourceEvent?.offset ?? null,
|
||||
firstSourceSeq: first?.debugLineage?.sourceSeq ?? null,
|
||||
lastSourceSeq: last?.debugLineage?.sourceSeq ?? null,
|
||||
valuesPrinted: false
|
||||
},
|
||||
outputBarrier: offsetRanges.length === 1 ? offsetRanges[0] : null,
|
||||
valuesPrinted: false
|
||||
},
|
||||
runtimeDependencies: {
|
||||
cloudApi: false,
|
||||
database: false,
|
||||
@@ -216,10 +255,10 @@ export async function regenerateHwlabDebugEvents(parsed: ParsedArgs, dependencie
|
||||
};
|
||||
}
|
||||
|
||||
export function mapAgentRunRecordsToHwlabDebugEvents(records: JsonRecord[], options: { sessionId: string; traceId?: string; expectedCount?: number; producedAt?: string }) {
|
||||
export function mapAgentRunRecordsToHwlabDebugEvents(records: JsonRecord[], options: { sessionId: string; traceId?: string; expectedCount?: number; producedAt?: string; replayId?: string }) {
|
||||
const matchingRecords = records.filter((record) => recordMatches(record, options.sessionId, options.traceId));
|
||||
if (matchingRecords.length === 0) {
|
||||
throw cliError("agentrun_debug_events_missing", "no AgentRun events matched the requested sessionId/traceId", {
|
||||
throw cliError("source_trace_missing", "no AgentRun events matched the requested sessionId/traceId", {
|
||||
sessionId: options.sessionId,
|
||||
traceId: options.traceId || null,
|
||||
readCount: records.length
|
||||
@@ -237,7 +276,12 @@ export function mapAgentRunRecordsToHwlabDebugEvents(records: JsonRecord[], opti
|
||||
let lineagePreserved = true;
|
||||
for (const [index, record] of matchingRecords.entries()) {
|
||||
const kafkaMessage = kafkaMessageFromRecord(record, index);
|
||||
const projected = projectAgentRunKafkaMessageToHwlabDebugEvent(kafkaMessage, { producedAt: options.producedAt });
|
||||
const projected = projectAgentRunKafkaMessageToHwlabDebugEvent(kafkaMessage, {
|
||||
producedAt: options.producedAt,
|
||||
replayId: options.replayId,
|
||||
replayIndex: index + 1,
|
||||
replayCount: matchingRecords.length
|
||||
});
|
||||
if (!projected.ok) {
|
||||
throw cliError(projected.error?.code || "hwlab_debug_projection_failed", projected.error?.message || "AgentRun debug projection failed", {
|
||||
index,
|
||||
@@ -251,7 +295,8 @@ export function mapAgentRunRecordsToHwlabDebugEvents(records: JsonRecord[], opti
|
||||
orderPreserved &&= projected.envelope.debugLineage?.sourceSeq === sourceSeq;
|
||||
lineagePreserved &&= projected.envelope.debugLineage?.inputTraceId === text(input.traceId ?? input.event?.payload?.traceId)
|
||||
&& sessionCandidates(input).includes(options.sessionId)
|
||||
&& projected.envelope.sourceEvent?.sha256 === kafkaMessage.inputSha256;
|
||||
&& projected.envelope.sourceEvent?.sha256 === kafkaMessage.inputSha256
|
||||
&& (!options.replayId || projected.envelope.replayId === options.replayId);
|
||||
events.push(projected.envelope);
|
||||
inputKinds.push(projected.inputKind);
|
||||
}
|
||||
@@ -272,19 +317,21 @@ function kafkaHelp() {
|
||||
action: "kafka.help",
|
||||
status: "succeeded",
|
||||
commands: [
|
||||
"hwlab-cli kafka regenerate hwlab --from kafka --session-id ses_... [--trace-id trc_...] [--input-topic agentrun.event.debug.v1] [--expect-count 35] [--json]",
|
||||
"hwlab-cli kafka regenerate hwlab --from jsonl --session-id ses_... --jsonl-file agentrun-events.jsonl --no-publish [--output-jsonl hwlab-events.jsonl] [--json]"
|
||||
"hwlab-cli kafka regenerate hwlab --from kafka --session-id ses_... [--trace-id trc_...] [--replay-id rpl_...] [--input-topic agentrun.event.debug.v1] [--expect-count 35] [--json]",
|
||||
"hwlab-cli kafka regenerate hwlab --from jsonl --session-id ses_... [--replay-id rpl_...] --jsonl-file agentrun-events.jsonl --no-publish [--output-jsonl hwlab-events.jsonl] [--json]"
|
||||
],
|
||||
defaults: {
|
||||
inputTopic: DEFAULT_AGENTRUN_DEBUG_EVENT_TOPIC,
|
||||
kafkaInputTopic: DEFAULT_AGENTRUN_EVENT_TOPIC,
|
||||
jsonlInputTopic: DEFAULT_AGENTRUN_DEBUG_EVENT_TOPIC,
|
||||
outputTopic: DEFAULT_HWLAB_DEBUG_EVENT_TOPIC,
|
||||
groupPrefix: null,
|
||||
groupPrefixAuthority: "--group-prefix or env:HWLAB_KAFKA_HWLAB_DEBUG_GROUP_PREFIX when Kafka is used",
|
||||
kafkaInputPublishes: true,
|
||||
kafkaInputPublishes: false,
|
||||
jsonlInputPublishes: false
|
||||
},
|
||||
boundaries: {
|
||||
productTopicAllowed: false,
|
||||
explicitPublishRequired: true,
|
||||
cloudApi: false,
|
||||
database: false,
|
||||
transactionalProjector: false,
|
||||
@@ -462,6 +509,33 @@ function optionalTraceId(value: unknown) {
|
||||
return result;
|
||||
}
|
||||
|
||||
function optionalReplayId(value: unknown) {
|
||||
const result = text(value);
|
||||
if (result && !/^rpl_[A-Za-z0-9_.:-]+$/u.test(result)) throw cliError("invalid_replay_id", "replayId must start with rpl_", { replayId: result });
|
||||
return result;
|
||||
}
|
||||
|
||||
function publishOffsetRanges(metadata: unknown, publishedCount: number, partition: number) {
|
||||
if (!Array.isArray(metadata) || publishedCount <= 0) return [];
|
||||
const item = metadata.find((entry: any) => Number(entry?.partition) === partition);
|
||||
const firstOffset = kafkaOffset(item?.baseOffset);
|
||||
if (!item || firstOffset === null) return [];
|
||||
return [{
|
||||
topic: text(item.topicName) || null,
|
||||
partition,
|
||||
firstOffset,
|
||||
lastOffset: (BigInt(firstOffset) + BigInt(publishedCount - 1)).toString(),
|
||||
count: publishedCount,
|
||||
valuesPrinted: false
|
||||
}];
|
||||
}
|
||||
|
||||
function kafkaOffset(value: unknown) {
|
||||
const result = text(value);
|
||||
if (!result || !/^\d+$/u.test(result)) return null;
|
||||
try { return BigInt(result).toString(); } catch { return null; }
|
||||
}
|
||||
|
||||
function enumValue(value: unknown, field: string, allowed: string[]) {
|
||||
const result = text(value);
|
||||
if (!allowed.includes(result)) throw cliError("invalid_option", `${field} must be one of: ${allowed.join(", ")}`, { field, value: result });
|
||||
|
||||
Reference in New Issue
Block a user