Commit Graph

578 Commits

Author SHA1 Message Date
lyon 94e77dab1f fix(workbench-runtime): ensure session read indexes 2026-06-22 08:34:40 +08:00
lyon feddcfdd6c fix(workbench): force AgentRun result authority on projection resume 2026-06-22 08:22:25 +08:00
lyon 4abc44bede fix(workbench): normalize legacy agentrun unreachable projection 2026-06-22 06:59:48 +08:00
lyon b3f7175a3c fix: keep recoverable dispatch failures nonterminal 2026-06-22 06:13:29 +08:00
lyon ba49294ea7 fix: require explicit agentrun terminal events 2026-06-22 06:05:23 +08:00
lyon 437ea223bb fix: stop stale agentrun unreachable terminal projection 2026-06-22 05:56:24 +08:00
lyon 32df45349b fix: retry AgentRun transport failures 2026-06-22 04:31:42 +08:00
lyon 13d74c3c06 fix: trace turn status read disconnects 2026-06-22 01:23:48 +08:00
lyon f930b21bb6 fix: avoid blocking projection resume on running AgentRun result 2026-06-22 01:06:31 +08:00
lyon d4929bcbb1 fix: handle text session summary timestamps 2026-06-21 23:07:19 +08:00
lyon 4279543d0b fix: scope workbench runtime session list by actor 2026-06-21 22:45:38 +08:00
Lyon 4b7a06e299 fix: harden workbench runtime session summary query (#1843) 2026-06-21 22:08:41 +08:00
lyon ea0354aca2 instrument workbench runtime otel spans 2026-06-21 21:55:52 +08:00
lyon 2bafed5b85 fix workbench runtime nullable json reads 2026-06-21 21:38:30 +08:00
lyon 0fe111747e move workbench session list to go runtime 2026-06-21 21:25:14 +08:00
lyon cf9ac0b08e feat: split workbench runtime read service 2026-06-21 19:42:26 +08:00
Lyon e23e9522d1 fix: keep workbench alive on postgres timeout (#1831) 2026-06-21 19:20:34 +08:00
Lyon 0833e115bf fix: reduce workbench read model load (#1829) 2026-06-21 18:41:07 +08:00
Lyon 507850579e fix: stabilize workbench turn diagnostics (#1828) 2026-06-21 18:23:13 +08:00
lyon 84180e3ae2 fix: do not seal running agentrun turns on caught-up projection 2026-06-21 14:47:06 +08:00
Lyon a5b47d5d8d fix: expose workbench projection failures in otel (#1824) 2026-06-21 13:44:22 +08:00
Lyon c3abe1a51a fix: seal AgentRun terminal projection (#1820) 2026-06-21 13:17:37 +08:00
Lyon 70d90e85eb fix: emit workbench sse accepted span (#1816) 2026-06-21 12:20:25 +08:00
Lyon 0f3bc3f99a fix workbench otel http span timing (#1815) 2026-06-21 12:07:37 +08:00
lyon b776a9631a fix(auth): reset postgres pool after transient timeout 2026-06-21 11:43:31 +08:00
Lyon 6cafd495de fix(cloud-api): keep health live nonblocking (#1808)
Route /health/live through cached runtime summary so DB durable retry cannot block liveness probes. Keep /health as the full durable readiness endpoint.\n\nRefs #1807.
2026-06-21 10:28:02 +08:00
Lyon a0c54934b2 fix(workbench): lengthen projection resume retry delay (#1804) 2026-06-21 09:57:26 +08:00
Lyon 0c25d44eac fix(workbench): back off non-terminal projection resumes (#1803) 2026-06-21 09:51:15 +08:00
Lyon bc8805baaf fix(workbench): back off projection resume pass failures (#1802) 2026-06-21 09:45:11 +08:00
Lyon 5b66e9db10 fix(workbench): back off projection resume candidates (#1801) 2026-06-21 09:39:10 +08:00
Lyon 8b529c39a0 fix(workbench): retry transient runtime store reads (#1797) 2026-06-21 09:16:23 +08:00
lyon 6e05c05732 fix(workbench): seal message status from turn projection 2026-06-21 07:52:59 +08:00
lyon b6dc2b4126 fix(workbench): seal final response lifecycle 2026-06-21 07:26:22 +08:00
lyon eedd8b6bfb fix(workbench): keep provider retry diagnostics non-terminal 2026-06-21 07:12:33 +08:00
lyon e9d19c8617 fix: retry transient workbench facts reads 2026-06-21 05:08:08 +08:00
lyon 7ac200cd7e fix: expose retryable projection gaps 2026-06-21 04:14:58 +08:00
lyon adba4a4354 fix: contain postgres unhandled connection timeouts 2026-06-21 04:00:50 +08:00
lyon cf13bf80a9 fix: contain postgres projection timeouts 2026-06-21 03:45:23 +08:00
lyon 1f59fbd33e fix: keep running turn timing authoritative 2026-06-21 03:29:29 +08:00
lyon 0d861d03ec fix: prioritize terminal checkpoint status 2026-06-21 03:21:09 +08:00
lyon e30e9e84aa fix: keep turn timing independent from sessions 2026-06-21 03:08:14 +08:00
lyon bcc8aa189b fix: stabilize workbench timing diagnostics 2026-06-21 02:56:19 +08:00
lyon a6b5e9a975 fix: harden workbench agent retry projection 2026-06-21 02:43:33 +08:00
lyon 49e42be440 fix(workbench): derive session terminal status from durable trace 2026-06-21 01:56:09 +08:00
Lyon cd981355c6 fix(workbench): keep per-turn message timing isolated (#1768) 2026-06-21 00:56:12 +08:00
lyon 183cf653f4 fix: resequence existing workbench trace migration 2026-06-20 20:43:37 +08:00
lyon 482218f2cc fix: resequence workbench trace migration 2026-06-20 20:34:15 +08:00
Lyon fb1fe8d94a feat(otel): 服务端 HTTP request span 追穿 hwlab-cloud-api 503 (#1764)
问题:hwlab-cloud-api 服务端 503(如 workbench 投影存储不可用)在 Tempo
里只有 workbench-ui 浏览器侧 span,没有服务端 request handler span,导致
后端根因(error.code=projection_store_unavailable 等)只能在应用日志里看,
OTel 链路追不到服务端处理层。

修复:
- otel-trace.ts 新增 emitHttpServerRequestSpan(scope hwlab.http.server,
  SPAN_KIND_SERVER),捕获 http.route/method/status_code/status_class、
  request.id,5xx 时落 STATUS_CODE_ERROR 并把 error.code/category/layer
  写入 span 属性与 status.message。
- error-envelope.ts 的 decorateErrorPayloadForHttp 把诊断 error code/
  category/layer 回写到 hwlabHttpRequestContext.lastHttpError,供 finish
  时复用,避免重复解析。
- server.ts 在 response finish 时按 http context + backend performance
  时序发服务端 span;health/metrics 探针路由跳过,避免 OTel 噪声洪泛。

验收:canary 调用 emitHttpServerRequestSpan 发 503 projection_store_unavailable
span 到 D601 platform-infra OTel collector,Tempo 可查到 service=hwlab-cloud-api、
scope=hwlab.http.server、http.response.status_code=503、error.code=
projection_store_unavailable 的 ERROR span。
2026-06-20 20:31:36 +08:00
lyon 69d5264360 fix: make workbench trace migration idempotent 2026-06-20 20:24:48 +08:00
lyon bad5a478bd fix: recover Workbench terminal outbox projection 2026-06-20 20:12:01 +08:00