# HWLAB Operator Runbook This runbook is the manual acceptance guide for commanders and operators during the first HWLAB DEV-only MVP. It defines when a phase may start, when it must stop, how failures are classified, and which actions are prohibited. The runbook supports M0-M5 and the DEV-only MVP path. It does not authorize a real DEV deployment, PROD deployment, secret access, service restart, or heavyweight e2e run by itself. ## Fixed Boundary - Repository: `pikasTech/HWLAB`. - MVP environment: `dev`. - DEV endpoint: `http://74.48.78.17:6667`. - PROD endpoint `:6666` is reserved and is not an MVP acceptance target. - Runtime route: client, master `hwlab-edge-proxy`, `frp`, D601 `hwlab-dev/hwlab-router`, then HWLAB cloud/runtime services. - HWLAB runtime services must keep the frozen service IDs in `README.md`. - UniDesk services may support scheduling, CI, or CD only. They are not accepted as substitutes for HWLAB runtime services. ## Runner Issue Visibility Handoff GitHub issue and PR threads are auxiliary context only. The runner prompt must be self-contained and must include the full task objective, background, constraints, acceptance, and validation commands. Do not assume issue comments are readable in the runner. If a task depends on GitHub visibility, use `docs/reference/runner-issue-visibility-handoff.md` as the stable prompt contract and run `node scripts/runner-issue-visibility-preflight.mjs` before queueing the work. The current runner class may have git transport access while GitHub REST issue and PR reads are blocked. Treat that as a prompt-handoff requirement, not a reason to omit context from the prompt. ## Hard Stop Actions Stop the run immediately and classify the event as a safety blocker if any operator, script, or agent attempts one of these actions: - Perform a real DEV deployment from this runbook. - Deploy, smoke, or mutate PROD in any form. - Read, print, copy, or infer secrets, tokens, kubeconfig, tunnel credentials, API keys, or private environment values. - Restart UniDesk, Code Queue, or `backend-core`. - Run heavyweight e2e, destructive hardware tests, load tests, or unbounded browser automation. - Use UniDesk backend, provider-gateway, or microservice proxy as the HWLAB runtime instead of `hwlab-cloud-api`, `hwlab-router`, `hwlab-gateway`, `hwlab-patch-panel`, agent services, and the other frozen HWLAB services. - Promote a source-only, local smoke, or dry-run result as real DEV evidence. ## Evidence Log Every phase result must be recorded in the operator log before the next phase starts. The log may be a ticket comment, incident note, or release checklist, but it must include: - phase name and milestone IDs; - command or observation used; - commit SHA; - environment, which must be `dev` for MVP evidence; - endpoint, when a network observation is made; - pass, fail, blocked, or not-applicable status; - failure class, when failed or blocked; - artifact identity for deployable runtime observations: `serviceId`, `commitId`, `image`, `tag`, `digest` or `not_applicable` reason, `buildSource`, `deployEnv`, and `healthTimestamp`; - evidence, audit, trace, operation, agent session, worker session, gateway session, and project IDs when the phase touches an MVP loop. ## Phase Gates Run phases in order. A later phase may start only after every earlier required exit condition is met or explicitly marked not applicable with a DEV-only reason. | Phase | Covers | Entry Conditions | Exit Conditions | | --- | --- | --- | --- | | Source-only | M0 contract, M2/M4/M5 source readiness | Worktree is based on latest `origin/main`; no secret or deployment access is needed; target files are docs, protocol, schema, fixtures, scripts, or manifests only. | Contract files parse; frozen endpoint and service IDs are unchanged; `npm run check` or the narrow validation command passes; no DEV/PROD mutation occurred. | | Local smoke | M1 plus local M3/M4 contract smoke | Source-only has passed; commands use localhost or fixture data only; ports are ephemeral; no DEV endpoint call is needed. | Smoke commands pass; generated output states local or dry-run boundary; no real hardware, tunnel, kube, or deploy action occurred. | | Dry-run | M2/M4/M5 readiness without runtime mutation | Source-only and local smoke pass; CLI/runtime fixtures identify `dev` and `http://74.48.78.17:6667`; operator confirms command includes `--dry-run` where applicable. | Dry-run output names MVP route, hardware trusted closed loop, agent automation closed loop, evidence records, and cleanup plan; output states no DEV/PROD changes were made. | | DEV runtime smoke | M2 DEV runtime observation | M0-M1 and required dry-runs pass; commander explicitly authorizes a read-only DEV smoke window; expected HWLAB artifacts and rollback owner are known; no secret reads are required. | Read-only health/status observations match `docs/dev-acceptance-matrix.md`; every observed service is HWLAB, in `dev`, and traceable to commit/image/tag/digest or a documented `not_applicable` reason. | | MVP e2e | M3/M4/M5 DEV-only MVP acceptance | DEV runtime smoke has passed; commander explicitly authorizes a narrow DEV-only MVP smoke; project/gateway/agent/hardware scope is named; cleanup owner and stop criteria are recorded. | One bounded DEV-only flow completes: project, gateway session, resources, patch-panel wiring, agent and worker sessions, hardware operation, trace events, audit events, evidence record, and cleanup. No PROD or heavyweight e2e action occurred. | ## Milestone Coverage | Milestone | Operator Gate | Required Acceptance | | --- | --- | --- | | M0 contract audit | Source-only | `docs/m0-contract-audit.md`, protocol docs, schemas, deploy schema, and M0 examples agree on service IDs, JSON-RPC envelope, errors, audit fields, topology, and DEV-only deploy shape. | | M1 local contract smoke | Local smoke | `npm run m1:smoke` or `node scripts/m1-contract-smoke.mjs` passes on localhost and confirms skeleton services, simulator contracts, patch-panel routing, CLI dry-run boundary, and no DEV/PROD changes. | | M2 DEV deploy/runtime smoke | Dry-run, then authorized DEV runtime smoke | Source/manifests and `docs/dev-acceptance-matrix.md` identify the DEV route through edge, `frp`, and D601 router. Real DEV observation is read-only and must not be performed without commander authorization. | | M3 hardware trusted loop | Local smoke, then MVP e2e | `node scripts/m3-hardware-loop-smoke.mjs` proves the local `box-simu-1 DO1 -> box-simu-2 DI1` patch-panel contract. Real DEV acceptance must preserve patch-panel ownership and produce trace, audit, and evidence records. | | M4 agent automation loop | Source-only or dry-run, then MVP e2e | Agent manager, worker, skills, session lifecycle, scoped workspace, trace events, cleanup, and evidence chain are observable in fixtures or dry-run before any DEV runtime action. | | M5 MVP e2e dry-run and acceptance | Dry-run, then MVP e2e | `npm run cli:dry-run` or equivalent names the MVP route, closed loops, evidence, and cleanup without mutation. Real acceptance is DEV-only, bounded, and requires prior DEV runtime smoke pass. | ## Phase Procedures ### Source-Only Procedure Use this gate for contract review, static checks, fixture checks, and manifest review. It is the default state before any runtime action is authorized. Entry: - `git fetch origin main` has completed. - The working tree is based on latest `origin/main`. - The change or review does not require secrets, kube access, tunnel access, or a deployment command. - The operator has identified the relevant contract source: `README.md`, `protocol/`, `deploy/deploy.schema.json`, `docs/dev-acceptance-matrix.md`, M0/M1/M3 docs, or MVP fixtures. Exit: - Frozen DEV endpoint remains `http://74.48.78.17:6667`. - PROD is not introduced as an acceptance target. - Frozen service IDs remain HWLAB service IDs. - JSON, schema, script syntax, and documented contract checks pass. - Any unresolved issue is logged under one of the failure classes below. ### Local Smoke Procedure Use this gate for localhost skeleton checks and in-process model checks. Entry: - Source-only gate passed. - Commands are known local checks such as `npm run m1:smoke`, `node scripts/m3-hardware-loop-smoke.mjs`, or targeted `node --test`. - No command calls `http://74.48.78.17:6667`, `:6666`, kube, `frp`, edge runtime, or real hardware. Exit: - Local services start on loopback or ephemeral ports and shut down cleanly. - Patch-panel-only propagation is preserved. - CLI output states dry-run or local boundary when applicable. - Audit and evidence expectations are shape-checked but not claimed as persisted DEV records. ### Dry-Run Procedure Use this gate for CLI, web, runtime plan, agent loop, and MVP e2e readiness without mutation. Entry: - Source-only and required local smoke gates passed. - The dry-run command has an explicit dry-run flag or documented fixture input. - The operator has confirmed the command cannot deploy, restart services, read secrets, write to real hardware, or mutate DEV/PROD state. Exit: - Output identifies the DEV endpoint and MVP route. - Output includes both required loops: hardware trusted closed loop and agent automation closed loop. - Output includes expected trace, audit, evidence, and cleanup records. - Output explicitly states that no DEV/PROD changes were made. ### DEV Runtime Smoke Procedure Use this gate only after commander authorization for read-only DEV observation. This runbook does not grant that authorization. Entry: - M0, M1, and required dry-runs passed. - Commander recorded an authorized DEV-only, read-only smoke window. - The operator has the expected artifact list and service IDs. - The planned observations require no secret reads and no restart. - Stop criteria and rollback owner are recorded. Exit: - DEV endpoint responds through the HWLAB route, not a UniDesk substitute. - Edge, `frp`, D601 router, cloud API/web, gateway or simulator, patch-panel, agent manager, agent worker, agent skills, and CLI observations match `docs/dev-acceptance-matrix.md` where applicable. - Every observed runtime component identifies `dev` and a HWLAB service ID. - Artifact identity is traceable or a `not_applicable` reason is recorded. - Any timeout, stale health, wrong namespace, wrong endpoint, or missing identity is classified before further action. ### MVP E2E Procedure Use this gate only for a bounded DEV-only MVP acceptance run after DEV runtime smoke has passed. It must remain much narrower than a heavyweight e2e suite. Entry: - DEV runtime smoke passed in the same acceptance window or was explicitly accepted by the commander as still current. - The MVP scenario is named and scoped to one project, one gateway path, one patch-panel wiring config, one agent session, and one worker session unless the commander records a narrower or equivalent scope. - Expected hardware operation, cleanup action, evidence target, and stop criteria are recorded. - No PROD, secret read, restart, or UniDesk runtime substitute is involved. Exit: - Project is created or selected. - Gateway session starts for `hwlab-gateway` or `hwlab-gateway-simu`. - Box resources and capabilities are registered. - Patch-panel wiring config passes topology validation and owns routing. - Agent and worker sessions start with scoped identity. - Hardware operation is requested through the HWLAB JSON-RPC path. - Trace events cover accepted, dispatched, executed, completed, evidence, and cleanup milestones. - Audit events cover mutating steps and final outcome. - Evidence record is persisted or, for a dry-run, fixture-bound and clearly marked as not persisted. - Cleanup completes and no worker, gateway, patch, project, or hardware state leak remains. ## Failure Classes Classify failures before retrying. If more than one class applies, choose the earliest blocker in the run path and mention secondary symptoms in the log. | Failure Class | Typical Symptoms | Immediate Operator Action | | --- | --- | --- | | Contract failure | Schema parse failure, JSON-RPC envelope mismatch, service ID drift, endpoint drift, topology rule mismatch, audit field drift, unsupported PROD target in an MVP artifact. | Stop phase; link the failing file or command; keep investigation source-only until contract drift is fixed. | | Image/version failure | Missing image, mutable tag, missing digest, health reports wrong commit, skill bundle has no commit ID, web or CLI artifact cannot be tied to source. | Stop runtime promotion; require artifact rebuild or provenance fix; do not substitute another service. | | k3s/frp/edge failure | DEV ingress timeout, wrong port, edge route missing, `frp` tunnel down, D601 router namespace mismatch, stale route, non-HWLAB response. | Stop DEV runtime smoke; record route segment and timestamp; do not restart UniDesk, Code Queue, `backend-core`, or unrelated infrastructure. | | Agent failure | Agent manager unavailable, worker session unscoped, missing agent/worker IDs, unsafe mutation, skill mismatch, worker cleanup leak, missing trace event. | Stop agent loop or MVP e2e; preserve logs and IDs; do not rerun with broader permissions or secret access. | | Hardware loop failure | Gateway unavailable, box resource missing, capability mismatch, direct box mutation, patch-panel bypass, topology conflict, DO/DI observation mismatch. | Stop hardware loop; leave hardware state unchanged when possible; require patch-panel/topology review before retry. | | Evidence/audit failure | Missing audit event, missing evidence record, trace/audit/evidence IDs not linked, wrong environment, missing operation ID, artifact hash mismatch, cleanup not evidenced. | Stop acceptance; do not claim MVP pass; repair evidence chain or rerun only the smallest authorized DEV-only scope. | ## DEV-Only MVP Acceptance Path The commander may call MVP accepted only when all items below are true: 1. M0 source-only contract gate passed. 2. M1 local smoke gate passed. 3. M2 dry-run readiness passed and any real DEV runtime smoke was explicitly authorized, read-only, and passed. 4. M3 hardware trusted loop passed locally and, when authorized, through the DEV HWLAB route with patch-panel ownership. 5. M4 agent automation loop passed dry-run or source contract checks and, when authorized, through scoped DEV agent and worker sessions. 6. M5 dry-run passed, then the bounded DEV-only MVP e2e completed with trace, audit, evidence, and cleanup records. 7. The evidence log contains no unclassified failure, safety blocker, PROD action, secret read, unauthorized restart, heavyweight e2e, or UniDesk runtime substitution. If any item is missing, the result is blocked or partially passed, not MVP accepted.