Commit Graph

496 Commits

Author SHA1 Message Date
lyon feddcfdd6c fix(workbench): force AgentRun result authority on projection resume 2026-06-22 08:22:25 +08:00
lyon 4abc44bede fix(workbench): normalize legacy agentrun unreachable projection 2026-06-22 06:59:48 +08:00
lyon b3f7175a3c fix: keep recoverable dispatch failures nonterminal 2026-06-22 06:13:29 +08:00
lyon ba49294ea7 fix: require explicit agentrun terminal events 2026-06-22 06:05:23 +08:00
lyon 437ea223bb fix: stop stale agentrun unreachable terminal projection 2026-06-22 05:56:24 +08:00
lyon 32df45349b fix: retry AgentRun transport failures 2026-06-22 04:31:42 +08:00
lyon 13d74c3c06 fix: trace turn status read disconnects 2026-06-22 01:23:48 +08:00
lyon f930b21bb6 fix: avoid blocking projection resume on running AgentRun result 2026-06-22 01:06:31 +08:00
lyon ea0354aca2 instrument workbench runtime otel spans 2026-06-21 21:55:52 +08:00
lyon 2bafed5b85 fix workbench runtime nullable json reads 2026-06-21 21:38:30 +08:00
lyon 0fe111747e move workbench session list to go runtime 2026-06-21 21:25:14 +08:00
lyon cf9ac0b08e feat: split workbench runtime read service 2026-06-21 19:42:26 +08:00
Lyon e23e9522d1 fix: keep workbench alive on postgres timeout (#1831) 2026-06-21 19:20:34 +08:00
Lyon 0833e115bf fix: reduce workbench read model load (#1829) 2026-06-21 18:41:07 +08:00
Lyon 507850579e fix: stabilize workbench turn diagnostics (#1828) 2026-06-21 18:23:13 +08:00
lyon 84180e3ae2 fix: do not seal running agentrun turns on caught-up projection 2026-06-21 14:47:06 +08:00
Lyon a5b47d5d8d fix: expose workbench projection failures in otel (#1824) 2026-06-21 13:44:22 +08:00
Lyon c3abe1a51a fix: seal AgentRun terminal projection (#1820) 2026-06-21 13:17:37 +08:00
Lyon 70d90e85eb fix: emit workbench sse accepted span (#1816) 2026-06-21 12:20:25 +08:00
Lyon 0f3bc3f99a fix workbench otel http span timing (#1815) 2026-06-21 12:07:37 +08:00
Lyon 6cafd495de fix(cloud-api): keep health live nonblocking (#1808)
Route /health/live through cached runtime summary so DB durable retry cannot block liveness probes. Keep /health as the full durable readiness endpoint.\n\nRefs #1807.
2026-06-21 10:28:02 +08:00
Lyon a0c54934b2 fix(workbench): lengthen projection resume retry delay (#1804) 2026-06-21 09:57:26 +08:00
Lyon 0c25d44eac fix(workbench): back off non-terminal projection resumes (#1803) 2026-06-21 09:51:15 +08:00
Lyon bc8805baaf fix(workbench): back off projection resume pass failures (#1802) 2026-06-21 09:45:11 +08:00
Lyon 5b66e9db10 fix(workbench): back off projection resume candidates (#1801) 2026-06-21 09:39:10 +08:00
lyon 6e05c05732 fix(workbench): seal message status from turn projection 2026-06-21 07:52:59 +08:00
lyon b6dc2b4126 fix(workbench): seal final response lifecycle 2026-06-21 07:26:22 +08:00
lyon eedd8b6bfb fix(workbench): keep provider retry diagnostics non-terminal 2026-06-21 07:12:33 +08:00
lyon e9d19c8617 fix: retry transient workbench facts reads 2026-06-21 05:08:08 +08:00
lyon 7ac200cd7e fix: expose retryable projection gaps 2026-06-21 04:14:58 +08:00
lyon adba4a4354 fix: contain postgres unhandled connection timeouts 2026-06-21 04:00:50 +08:00
lyon cf13bf80a9 fix: contain postgres projection timeouts 2026-06-21 03:45:23 +08:00
lyon 1f59fbd33e fix: keep running turn timing authoritative 2026-06-21 03:29:29 +08:00
lyon 0d861d03ec fix: prioritize terminal checkpoint status 2026-06-21 03:21:09 +08:00
lyon e30e9e84aa fix: keep turn timing independent from sessions 2026-06-21 03:08:14 +08:00
lyon bcc8aa189b fix: stabilize workbench timing diagnostics 2026-06-21 02:56:19 +08:00
lyon a6b5e9a975 fix: harden workbench agent retry projection 2026-06-21 02:43:33 +08:00
lyon 49e42be440 fix(workbench): derive session terminal status from durable trace 2026-06-21 01:56:09 +08:00
Lyon cd981355c6 fix(workbench): keep per-turn message timing isolated (#1768) 2026-06-21 00:56:12 +08:00
Lyon fb1fe8d94a feat(otel): 服务端 HTTP request span 追穿 hwlab-cloud-api 503 (#1764)
问题:hwlab-cloud-api 服务端 503(如 workbench 投影存储不可用)在 Tempo
里只有 workbench-ui 浏览器侧 span,没有服务端 request handler span,导致
后端根因(error.code=projection_store_unavailable 等)只能在应用日志里看,
OTel 链路追不到服务端处理层。

修复:
- otel-trace.ts 新增 emitHttpServerRequestSpan(scope hwlab.http.server,
  SPAN_KIND_SERVER),捕获 http.route/method/status_code/status_class、
  request.id,5xx 时落 STATUS_CODE_ERROR 并把 error.code/category/layer
  写入 span 属性与 status.message。
- error-envelope.ts 的 decorateErrorPayloadForHttp 把诊断 error code/
  category/layer 回写到 hwlabHttpRequestContext.lastHttpError,供 finish
  时复用,避免重复解析。
- server.ts 在 response finish 时按 http context + backend performance
  时序发服务端 span;health/metrics 探针路由跳过,避免 OTel 噪声洪泛。

验收:canary 调用 emitHttpServerRequestSpan 发 503 projection_store_unavailable
span 到 D601 platform-infra OTel collector,Tempo 可查到 service=hwlab-cloud-api、
scope=hwlab.http.server、http.response.status_code=503、error.code=
projection_store_unavailable 的 ERROR span。
2026-06-20 20:31:36 +08:00
lyon bad5a478bd fix: recover Workbench terminal outbox projection 2026-06-20 20:12:01 +08:00
lyon 800f674d5c fix: recover AgentRun terminal projection 2026-06-20 20:05:13 +08:00
lyon 6ab94f23c8 fix: clarify workbench trace event diagnostics 2026-06-20 19:43:54 +08:00
Lyon 4f7e55e2d8 fix: PJ2026-0104010803 P5 残余 events.at(-1) 读侧推理清理 + code-agent metric status (#1759)
useTraceSubscription.ts:
- mergeTraceResults: lastEventLabel 不 fallback 到 events.at(-1)
- mergeTraceSnapshots: lastEventLabel 不 fallback 到 events.at(-1)
- traceSnapshotSignalKey: 不从 events.at(-1) 推导 signal key

server-code-agent-http.ts:
- recordCodeAgentProjectionMetric: status 只读 projection.status
- source 固定为 workbench_read_model

负向源码扫描确认:
- result?.status ?? trace?.status: 消除
- events.at(-1) lifecycle 推断: 消除
- .catch(() => {}) 静默吞错: 消除
- traceStore.subscribe: 仅作为 outbox 不可用时的兼容降级

Closes #1751
Refs #1742
2026-06-20 19:42:19 +08:00
Lyon 9574e2b514 feat: PJ2026-0104010803 P3 后端读侧推理清零 - session list + metric status 只读 projection (#1757)
server-workbench-http.ts:
- sessionListProjectionOptions: 当 session snapshot 已有 durable projection 字段
  (projectionStatus/terminal/sealedAt) 时,直接使用 projection 字段构建 turn projection,
  不再从 compactSessionTraceResult/compactSessionTraceSnapshot 多来源混合
- recordWorkbenchProjectionMetric: status 只读 projection.status,不再 fallback 到
  result?.status ?? trace?.status;source 固定为 workbench_read_model
- SPEC header 更新到 draft-2026-06-20-p3-read-side-zero

workbench-turn-projection.ts:
- SPEC header 更新到 draft-2026-06-20-p1-zero-split-durable-realtime

Closes #1749
Refs #1742
2026-06-20 19:34:52 +08:00
Lyon 7c5a1f8c5b feat: PJ2026-0104010803 P2 SSE/realtime 无状态化 - durable outbox cursor replay (#1756)
server-workbench-http.ts:
- handleWorkbenchRealtimeHttp 优先从 durable outbox 读取 (readWorkbenchProjectionOutbox)
- 当 runtimeStore 支持 outbox 时,SSE 用 polling cursor replay 代替 traceStore.subscribe
- outbox 不支持时保留内存 traceStore.subscribe 作为兼容降级
- terminal commit 从 outbox terminal 行触发 snapshot + trace realtime
- 新增 HWLAB_WORKBENCH_SSE_OUTBOX_POLL_MS 环境变量控制 poll 间隔

workbench-facts-store.ts:
- 新增 subscribeDurableProjection 方法,从 durable outbox polling 读取
- subscribeTrace 保留兼容(内存 trace),新增 durable projection subscription 入口
- SPEC header 更新到 draft-2026-06-20-p1-zero-split-durable-realtime

cloud-api 重启后 SSE 从 DB outbox cursor 继续输出,不依赖内存 listener

Closes #1748
Refs #1742
2026-06-20 19:31:49 +08:00
Lyon b254c1875e feat: PJ2026-0104010803 P1 durable projection outbox + DB constraints + error visibility (#1754)
DB migration (0001_cloud_core_skeleton.sql):
- Add (trace_id, source_event_id) unique index on workbench_trace_events
- Add (trace_id, projected_seq) unique index on workbench_trace_events
- Add workbench_projection_outbox table with BIGSERIAL outbox_seq

Schema (schema.ts):
- Bump CLOUD_RUNTIME_DURABLE_ADAPTER_SCHEMA_VERSION to v5
- Add workbench_projection_outbox to CLOUD_RUNTIME_DURABLE_TABLE_COLUMNS

Runtime store (runtime-store.ts):
- Add persistWorkbenchProjectionOutbox method
- Add readWorkbenchProjectionOutbox method for SSE cursor replay
- Write outbox rows in writeWorkbenchFacts for trace events and terminal turns
- Outbox write errors are logged, not silently swallowed

Trace store (code-agent-trace-store.ts):
- persistTraceEvent no longer uses .catch(() => {}) to swallow errors
- Projection write errors are logged with traceId/sourceEventId for diagnostics
- Sync errors also logged instead of empty catch block

SPEC headers updated to draft-2026-06-20-p1-zero-split-durable-realtime

Closes #1747
Refs #1742
2026-06-20 19:29:44 +08:00
lyon 062df3bf49 test: align user billing async agentrun coverage 2026-06-20 18:17:30 +08:00
lyon 3718b5d77d fix: persist manual workbench sessions 2026-06-20 16:58:31 +08:00
lyon 11e2f08e70 fix: order workbench sessions by updated desc 2026-06-20 15:36:41 +08:00