diff --git a/docs/operator-runbook.md b/docs/operator-runbook.md new file mode 100644 index 00000000..00259a0b --- /dev/null +++ b/docs/operator-runbook.md @@ -0,0 +1,238 @@ +# HWLAB Operator Runbook + +This runbook is the manual acceptance guide for commanders and operators during +the first HWLAB DEV-only MVP. It defines when a phase may start, when it must +stop, how failures are classified, and which actions are prohibited. + +The runbook supports M0-M5 and the DEV-only MVP path. It does not authorize a +real DEV deployment, PROD deployment, secret access, service restart, or +heavyweight e2e run by itself. + +## Fixed Boundary + +- Repository: `pikasTech/HWLAB`. +- MVP environment: `dev`. +- DEV endpoint: `http://74.48.78.17:6667`. +- PROD endpoint `:6666` is reserved and is not an MVP acceptance target. +- Runtime route: client, master `hwlab-edge-proxy`, `frp`, D601 + `hwlab-dev/hwlab-router`, then HWLAB cloud/runtime services. +- HWLAB runtime services must keep the frozen service IDs in `README.md`. +- UniDesk services may support scheduling, CI, or CD only. They are not accepted + as substitutes for HWLAB runtime services. + +## Hard Stop Actions + +Stop the run immediately and classify the event as a safety blocker if any +operator, script, or agent attempts one of these actions: + +- Perform a real DEV deployment from this runbook. +- Deploy, smoke, or mutate PROD in any form. +- Read, print, copy, or infer secrets, tokens, kubeconfig, tunnel credentials, + API keys, or private environment values. +- Restart UniDesk, Code Queue, or `backend-core`. +- Run heavyweight e2e, destructive hardware tests, load tests, or unbounded + browser automation. +- Use UniDesk backend, provider-gateway, or microservice proxy as the HWLAB + runtime instead of `hwlab-cloud-api`, `hwlab-router`, `hwlab-gateway`, + `hwlab-patch-panel`, agent services, and the other frozen HWLAB services. +- Promote a source-only, local smoke, or dry-run result as real DEV evidence. + +## Evidence Log + +Every phase result must be recorded in the operator log before the next phase +starts. The log may be a ticket comment, incident note, or release checklist, +but it must include: + +- phase name and milestone IDs; +- command or observation used; +- commit SHA; +- environment, which must be `dev` for MVP evidence; +- endpoint, when a network observation is made; +- pass, fail, blocked, or not-applicable status; +- failure class, when failed or blocked; +- artifact identity for deployable runtime observations: `serviceId`, + `commitId`, `image`, `tag`, `digest` or `not_applicable` reason, + `buildSource`, `deployEnv`, and `healthTimestamp`; +- evidence, audit, trace, operation, agent session, worker session, gateway + session, and project IDs when the phase touches an MVP loop. + +## Phase Gates + +Run phases in order. A later phase may start only after every earlier required +exit condition is met or explicitly marked not applicable with a DEV-only +reason. + +| Phase | Covers | Entry Conditions | Exit Conditions | +| --- | --- | --- | --- | +| Source-only | M0 contract, M2/M4/M5 source readiness | Worktree is based on latest `origin/main`; no secret or deployment access is needed; target files are docs, protocol, schema, fixtures, scripts, or manifests only. | Contract files parse; frozen endpoint and service IDs are unchanged; `npm run check` or the narrow validation command passes; no DEV/PROD mutation occurred. | +| Local smoke | M1 plus local M3/M4 contract smoke | Source-only has passed; commands use localhost or fixture data only; ports are ephemeral; no DEV endpoint call is needed. | Smoke commands pass; generated output states local or dry-run boundary; no real hardware, tunnel, kube, or deploy action occurred. | +| Dry-run | M2/M4/M5 readiness without runtime mutation | Source-only and local smoke pass; CLI/runtime fixtures identify `dev` and `http://74.48.78.17:6667`; operator confirms command includes `--dry-run` where applicable. | Dry-run output names MVP route, hardware trusted closed loop, agent automation closed loop, evidence records, and cleanup plan; output states no DEV/PROD changes were made. | +| DEV runtime smoke | M2 DEV runtime observation | M0-M1 and required dry-runs pass; commander explicitly authorizes a read-only DEV smoke window; expected HWLAB artifacts and rollback owner are known; no secret reads are required. | Read-only health/status observations match `docs/dev-acceptance-matrix.md`; every observed service is HWLAB, in `dev`, and traceable to commit/image/tag/digest or a documented `not_applicable` reason. | +| MVP e2e | M3/M4/M5 DEV-only MVP acceptance | DEV runtime smoke has passed; commander explicitly authorizes a narrow DEV-only MVP smoke; project/gateway/agent/hardware scope is named; cleanup owner and stop criteria are recorded. | One bounded DEV-only flow completes: project, gateway session, resources, patch-panel wiring, agent and worker sessions, hardware operation, trace events, audit events, evidence record, and cleanup. No PROD or heavyweight e2e action occurred. | + +## Milestone Coverage + +| Milestone | Operator Gate | Required Acceptance | +| --- | --- | --- | +| M0 contract audit | Source-only | `docs/m0-contract-audit.md`, protocol docs, schemas, deploy schema, and M0 examples agree on service IDs, JSON-RPC envelope, errors, audit fields, topology, and DEV-only deploy shape. | +| M1 local contract smoke | Local smoke | `npm run m1:smoke` or `node scripts/m1-contract-smoke.mjs` passes on localhost and confirms skeleton services, simulator contracts, patch-panel routing, CLI dry-run boundary, and no DEV/PROD changes. | +| M2 DEV deploy/runtime smoke | Dry-run, then authorized DEV runtime smoke | Source/manifests and `docs/dev-acceptance-matrix.md` identify the DEV route through edge, `frp`, and D601 router. Real DEV observation is read-only and must not be performed without commander authorization. | +| M3 hardware trusted loop | Local smoke, then MVP e2e | `node scripts/m3-hardware-loop-smoke.mjs` proves the local `box-simu-1 DO1 -> box-simu-2 DI1` patch-panel contract. Real DEV acceptance must preserve patch-panel ownership and produce trace, audit, and evidence records. | +| M4 agent automation loop | Source-only or dry-run, then MVP e2e | Agent manager, worker, skills, session lifecycle, scoped workspace, trace events, cleanup, and evidence chain are observable in fixtures or dry-run before any DEV runtime action. | +| M5 MVP e2e dry-run and acceptance | Dry-run, then MVP e2e | `npm run cli:dry-run` or equivalent names the MVP route, closed loops, evidence, and cleanup without mutation. Real acceptance is DEV-only, bounded, and requires prior DEV runtime smoke pass. | + +## Phase Procedures + +### Source-Only Procedure + +Use this gate for contract review, static checks, fixture checks, and manifest +review. It is the default state before any runtime action is authorized. + +Entry: + +- `git fetch origin main` has completed. +- The working tree is based on latest `origin/main`. +- The change or review does not require secrets, kube access, tunnel access, or + a deployment command. +- The operator has identified the relevant contract source: + `README.md`, `protocol/`, `deploy/deploy.schema.json`, + `docs/dev-acceptance-matrix.md`, M0/M1/M3 docs, or MVP fixtures. + +Exit: + +- Frozen DEV endpoint remains `http://74.48.78.17:6667`. +- PROD is not introduced as an acceptance target. +- Frozen service IDs remain HWLAB service IDs. +- JSON, schema, script syntax, and documented contract checks pass. +- Any unresolved issue is logged under one of the failure classes below. + +### Local Smoke Procedure + +Use this gate for localhost skeleton checks and in-process model checks. + +Entry: + +- Source-only gate passed. +- Commands are known local checks such as `npm run m1:smoke`, + `node scripts/m3-hardware-loop-smoke.mjs`, or targeted `node --test`. +- No command calls `http://74.48.78.17:6667`, `:6666`, kube, `frp`, edge + runtime, or real hardware. + +Exit: + +- Local services start on loopback or ephemeral ports and shut down cleanly. +- Patch-panel-only propagation is preserved. +- CLI output states dry-run or local boundary when applicable. +- Audit and evidence expectations are shape-checked but not claimed as + persisted DEV records. + +### Dry-Run Procedure + +Use this gate for CLI, web, runtime plan, agent loop, and MVP e2e readiness +without mutation. + +Entry: + +- Source-only and required local smoke gates passed. +- The dry-run command has an explicit dry-run flag or documented fixture input. +- The operator has confirmed the command cannot deploy, restart services, read + secrets, write to real hardware, or mutate DEV/PROD state. + +Exit: + +- Output identifies the DEV endpoint and MVP route. +- Output includes both required loops: hardware trusted closed loop and agent + automation closed loop. +- Output includes expected trace, audit, evidence, and cleanup records. +- Output explicitly states that no DEV/PROD changes were made. + +### DEV Runtime Smoke Procedure + +Use this gate only after commander authorization for read-only DEV observation. +This runbook does not grant that authorization. + +Entry: + +- M0, M1, and required dry-runs passed. +- Commander recorded an authorized DEV-only, read-only smoke window. +- The operator has the expected artifact list and service IDs. +- The planned observations require no secret reads and no restart. +- Stop criteria and rollback owner are recorded. + +Exit: + +- DEV endpoint responds through the HWLAB route, not a UniDesk substitute. +- Edge, `frp`, D601 router, cloud API/web, gateway or simulator, + patch-panel, agent manager, agent worker, agent skills, and CLI observations + match `docs/dev-acceptance-matrix.md` where applicable. +- Every observed runtime component identifies `dev` and a HWLAB service ID. +- Artifact identity is traceable or a `not_applicable` reason is recorded. +- Any timeout, stale health, wrong namespace, wrong endpoint, or missing + identity is classified before further action. + +### MVP E2E Procedure + +Use this gate only for a bounded DEV-only MVP acceptance run after DEV runtime +smoke has passed. It must remain much narrower than a heavyweight e2e suite. + +Entry: + +- DEV runtime smoke passed in the same acceptance window or was explicitly + accepted by the commander as still current. +- The MVP scenario is named and scoped to one project, one gateway path, one + patch-panel wiring config, one agent session, and one worker session unless + the commander records a narrower or equivalent scope. +- Expected hardware operation, cleanup action, evidence target, and stop + criteria are recorded. +- No PROD, secret read, restart, or UniDesk runtime substitute is involved. + +Exit: + +- Project is created or selected. +- Gateway session starts for `hwlab-gateway` or `hwlab-gateway-simu`. +- Box resources and capabilities are registered. +- Patch-panel wiring config passes topology validation and owns routing. +- Agent and worker sessions start with scoped identity. +- Hardware operation is requested through the HWLAB JSON-RPC path. +- Trace events cover accepted, dispatched, executed, completed, evidence, and + cleanup milestones. +- Audit events cover mutating steps and final outcome. +- Evidence record is persisted or, for a dry-run, fixture-bound and clearly + marked as not persisted. +- Cleanup completes and no worker, gateway, patch, project, or hardware state + leak remains. + +## Failure Classes + +Classify failures before retrying. If more than one class applies, choose the +earliest blocker in the run path and mention secondary symptoms in the log. + +| Failure Class | Typical Symptoms | Immediate Operator Action | +| --- | --- | --- | +| Contract failure | Schema parse failure, JSON-RPC envelope mismatch, service ID drift, endpoint drift, topology rule mismatch, audit field drift, unsupported PROD target in an MVP artifact. | Stop phase; link the failing file or command; keep investigation source-only until contract drift is fixed. | +| Image/version failure | Missing image, mutable tag, missing digest, health reports wrong commit, skill bundle has no commit ID, web or CLI artifact cannot be tied to source. | Stop runtime promotion; require artifact rebuild or provenance fix; do not substitute another service. | +| k3s/frp/edge failure | DEV ingress timeout, wrong port, edge route missing, `frp` tunnel down, D601 router namespace mismatch, stale route, non-HWLAB response. | Stop DEV runtime smoke; record route segment and timestamp; do not restart UniDesk, Code Queue, `backend-core`, or unrelated infrastructure. | +| Agent failure | Agent manager unavailable, worker session unscoped, missing agent/worker IDs, unsafe mutation, skill mismatch, worker cleanup leak, missing trace event. | Stop agent loop or MVP e2e; preserve logs and IDs; do not rerun with broader permissions or secret access. | +| Hardware loop failure | Gateway unavailable, box resource missing, capability mismatch, direct box mutation, patch-panel bypass, topology conflict, DO/DI observation mismatch. | Stop hardware loop; leave hardware state unchanged when possible; require patch-panel/topology review before retry. | +| Evidence/audit failure | Missing audit event, missing evidence record, trace/audit/evidence IDs not linked, wrong environment, missing operation ID, artifact hash mismatch, cleanup not evidenced. | Stop acceptance; do not claim MVP pass; repair evidence chain or rerun only the smallest authorized DEV-only scope. | + +## DEV-Only MVP Acceptance Path + +The commander may call MVP accepted only when all items below are true: + +1. M0 source-only contract gate passed. +2. M1 local smoke gate passed. +3. M2 dry-run readiness passed and any real DEV runtime smoke was explicitly + authorized, read-only, and passed. +4. M3 hardware trusted loop passed locally and, when authorized, through the + DEV HWLAB route with patch-panel ownership. +5. M4 agent automation loop passed dry-run or source contract checks and, when + authorized, through scoped DEV agent and worker sessions. +6. M5 dry-run passed, then the bounded DEV-only MVP e2e completed with trace, + audit, evidence, and cleanup records. +7. The evidence log contains no unclassified failure, safety blocker, PROD + action, secret read, unauthorized restart, heavyweight e2e, or UniDesk + runtime substitution. + +If any item is missing, the result is blocked or partially passed, not MVP +accepted.