pikasTech-HWLAB/docs/operator-runbook.md

HWLAB Operator Runbook

This runbook is the manual acceptance guide for commanders and operators during the first HWLAB DEV-only MVP. It defines when a phase may start, when it must stop, how failures are classified, and which actions are prohibited.

The runbook supports M0-M5 and the DEV-only MVP path. It does not authorize a real DEV deployment, PROD deployment, secret access, service restart, or heavyweight e2e run by itself.

Fixed Boundary

• Repository: pikasTech/HWLAB.
• MVP environment: dev.
• DEV frontend endpoint: http://74.48.78.17:16666.
• DEV API/edge endpoint: http://74.48.78.17:16667.
• Legacy public endpoints :6666 and :6667 are deprecated and are not MVP acceptance targets.
• Internal k3s service ports may still use 6667; internal 6667 is not a public acceptance endpoint.
• Runtime route: client, master hwlab-edge-proxy, frp, D601 hwlab-dev/hwlab-router, then HWLAB cloud/runtime services.
• HWLAB runtime services must keep the frozen service IDs in README.md.
• UniDesk services may support scheduling, CI, or CD only. They are not accepted as substitutes for HWLAB runtime services.
• Code Agent provider credentials are DEV-only runtime inputs: OPENAI_API_KEY must come from Secret hwlab-code-agent-provider/openai-api-key, and HWLAB_CODE_AGENT_OPENAI_BASE_URL must route through the DEV egress/proxy path, not direct public api.openai.com.
• Cloud API DB readiness is layered as DNS, TCP, auth, and schema. .invalid hosts are never desired DEV runtime targets; they are negative fixtures only.

M3 MVP Acceptance

M3 MVP PASS is the smallest real DEV evidence chain that proves the hardware trusted loop. It requires:

• two box simulators with different observable identities;
• two gateway simulators with different observable identities;
• one active patch-panel participant;
• the traced res_boxsimu_1:DO1 -> hwlab-patch-panel -> res_boxsimu_2:DI1 operation with trace, audit, operation, and evidence IDs;
• no box loopback substitution, no front-end direct state edit, and no UniDesk runtime or proxy substitution.

Cloud Workbench, generic console work, artifact report generation, and desired-state planning are support surfaces only. They may help operators, but they do not count as M3 PASS on their own and cannot replace the loop evidence above.

The detailed step order and misclassification bans live in docs/reference/m3-loop-rollout-runbook.md; this runbook and the acceptance matrix mirror that contract.

Runner Issue Visibility Handoff

GitHub issue and PR threads are auxiliary context only. The runner prompt must be self-contained and must include the full task objective, background, constraints, acceptance, and validation commands.

Do not assume issue comments are readable in the runner. If a task depends on GitHub visibility, use docs/reference/runner-issue-visibility-handoff.md as the stable prompt contract and run node scripts/runner-issue-visibility-preflight.mjs before queueing the work.

The current runner class may have git transport access while GitHub REST issue and PR reads are blocked. Treat that as a prompt-handoff requirement, not a reason to omit context from the prompt.

Hard Stop Actions

Stop the run immediately and classify the event as a safety blocker if any operator, script, or agent attempts one of these actions:

• Perform a real DEV deployment from this runbook.
• Deploy, smoke, or mutate PROD in any form.
• Read, print, copy, or infer secrets, tokens, kubeconfig, tunnel credentials, API keys, or private environment values.
• Restart UniDesk, Code Queue, or backend-core.
• Run heavyweight e2e, destructive hardware tests, load tests, or unbounded browser automation.
• Use UniDesk backend, provider-gateway, or microservice proxy as the HWLAB runtime instead of hwlab-cloud-api, hwlab-router, hwlab-gateway, hwlab-patch-panel, agent services, and the other frozen HWLAB services.
• Promote SOURCE, LOCAL, DRY-RUN, or edge-only result as real DEV evidence.
• Treat Cloud Workbench polish, generic console work, artifact reports, or desired state plans as M3 PASS evidence.
• Leave an emergency DEV runtime hotfix as the only source of truth after it unblocks a P0 path. The follow-up must codify source manifests, contracts, docs, and validation without rerunning the live patch.

Evidence Log

Every phase result must be recorded in the operator log before the next phase starts. The log may be a ticket comment, incident note, or release checklist, but it must include:

• phase name and milestone IDs;
• command or observation used;
• commit SHA;
• environment, which must be dev for MVP evidence;
• endpoint, when a network observation is made;
• pass, fail, blocked, or not-applicable status;
• failure class, when failed or blocked;
• artifact identity for deployable runtime observations: serviceId, commitId, image, tag, digest or not_applicable reason, buildSource, deployEnv, and healthTimestamp;
• evidence, audit, trace, operation, agent session, worker session, gateway session, and project IDs when the phase touches an MVP loop.

DEV Gate Aggregator v2

Use /tmp/hwlab-dev-gate/dev-m5-gate-aggregator-v2.json and /tmp/hwlab-dev-gate/dev-m5-gate-aggregator-v2.md as the current report-only M5 gate summary for pikasTech/HWLAB#58. Refresh them only from committed fixtures:

node scripts/dev-evidence-blocker-aggregator.mjs --report /tmp/hwlab-dev-gate/report.json
node scripts/validate-dev-gate-report.mjs

The aggregator separates SOURCE, LOCAL, DRY-RUN, DEV-LIVE, and BLOCKED. It must never promote source, local, dry-run, contract-only, not-run, or failed-live evidence into DEV-LIVE.

Current unblock order:

1. Provide an approved Node 20 DEV builder base image for artifact publish.
2. Publish every frozen HWLAB service artifact and record immutable registry digests for the selected origin/main commit.
3. Restore read-only D601 hwlab-dev k3s observability with KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl or an approved kubeconfig/maintenance bridge, without reading Secrets. Do not use a default docker-desktop kubeconfig as DEV evidence; the control-plane incident is tracked in pikasTech/unidesk#138.
4. Prove DEV hwlab-cloud-api DB env and health readiness with redacted values only.
5. Repair the :16667/frp/edge/router path and rerun read-only DEV edge health before any M3, M4, or M5 live loop.

For #164-style hotfix follow-up, treat provider and DB fixes as independent contracts:

• Code Agent provider: source and k8s workload env must declare OPENAI_API_KEY from hwlab-code-agent-provider/openai-api-key and HWLAB_CODE_AGENT_OPENAI_BASE_URL through DEV egress/proxy. Validation may prove Secret ref shape and key-presence evidence only; it must not print the API key.
• DB live: source and runtime health must reject .invalid as a DEV target and report DNS/TCP/auth/schema readiness separately. liveDbEvidence=true is allowed only after live runtime connectivity evidence, never from fixtures or env presence alone.
• Follow-up automation: update deploy/deploy.json, deploy/k8s/base/workloads.yaml, source contract code, docs, and tests; do not apply manifests, restart services, mutate PROD, or repeat the manual hotfix from the source PR.

Phase Gates

Run phases in order. A later phase may start only after every earlier required exit condition is met or explicitly marked not applicable with a DEV-only reason.

Phase	Covers	Entry Conditions	Exit Conditions
Source-only	M0 contract, M2/M4/M5 source readiness	Worktree is based on latest origin/main; no secret or deployment access is needed; target files are docs, protocol, schema, fixtures, scripts, or manifests only.	Contract files parse; frozen endpoint and service IDs are unchanged; npm run check or the narrow validation command passes; no DEV/PROD mutation occurred.
Local smoke	M1 plus local M3/M4 contract smoke	Source-only has passed; commands use localhost or fixture data only; ports are ephemeral; no DEV endpoint call is needed.	Smoke commands pass; generated output states local or dry-run boundary; no real hardware, tunnel, kube, or deploy action occurred.
Dry-run	M2/M4/M5 readiness without runtime mutation	Source-only and local smoke pass; CLI/runtime fixtures identify dev and http://74.48.78.17:16667; operator confirms command includes --dry-run where applicable.	Dry-run output names MVP route, hardware trusted closed loop, agent automation closed loop, evidence records, and cleanup plan; output states no DEV/PROD changes were made.
DEV runtime smoke	M2 DEV runtime observation	M0-M1 and required dry-runs pass; commander explicitly authorizes a read-only DEV smoke window; expected HWLAB artifacts and rollback owner are known; no secret reads are required.	Read-only health/status observations match docs/dev-acceptance-matrix.md; every observed service is HWLAB, in dev, and traceable to commit/image/tag/digest or a documented not_applicable reason.
MVP e2e	M3/M4/M5 DEV-only MVP acceptance	DEV runtime smoke has passed; commander explicitly authorizes a narrow DEV-only MVP smoke; project/gateway/agent/hardware scope is named; cleanup owner and stop criteria are recorded.	One bounded DEV-only flow completes: project, gateway session, resources, patch-panel wiring, agent and worker sessions, hardware operation, trace events, audit events, evidence record, and cleanup. No PROD or heavyweight e2e action occurred.

Milestone Coverage

Milestone	Operator Gate	Required Acceptance
M0 contract audit	Source-only	docs/m0-contract-audit.md, protocol docs, schemas, deploy schema, and M0 examples agree on service IDs, JSON-RPC envelope, errors, audit fields, topology, and DEV-only deploy shape.
M1 local contract smoke	Local smoke	npm run m1:smoke or node scripts/m1-contract-smoke.mjs passes on localhost and confirms the local cloud-api runtime, simulator contracts, patch-panel routing, CLI dry-run boundary, and no DEV/PROD changes.
M2 DEV deploy/runtime smoke	Dry-run, then authorized DEV runtime smoke	Source/manifests and docs/dev-acceptance-matrix.md identify the DEV route through edge, frp, and D601 router. Real DEV observation is read-only and must not be performed without commander authorization.
M3 hardware trusted loop	Local smoke, then MVP e2e	node scripts/m3-hardware-loop-smoke.mjs proves the local contract, but M3 MVP PASS in DEV requires the live res_boxsimu_1:DO1 -> hwlab-patch-panel -> res_boxsimu_2:DI1 evidence chain with distinct box/gateway identities and operation, trace, audit, and evidence IDs.
M4 agent automation loop	Source-only or dry-run, then MVP e2e	Agent manager, worker, skills, session lifecycle, scoped workspace, trace events, cleanup, and evidence chain are observable in fixtures or dry-run before any DEV runtime action.
M5 MVP e2e dry-run and acceptance	Dry-run, then MVP e2e	npm run cli:dry-run or equivalent names the MVP route, closed loops, evidence, and cleanup without mutation. Real acceptance is DEV-only, bounded, and requires prior DEV runtime smoke pass.

L6 CLI/Web Acceptance Entry

Use the L6 entry points for external manual acceptance triage. They read the checked-in GitHub issue/PR comments and fixtures/mvp/m5-e2e/dry-run-plan.json; they do not call UniDesk runtime and do not claim live DEV evidence from fixtures.

Safe source and dry-run checks:

npm run l6:smoke
npm run web:m3-readonly
npm run web:check
npm run cli:dry-run
node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp

For Cloud Web #99, docs/cloud-web-workbench.md is the frontend design and acceptance contract. The browser endpoint remains public :16666, while API/edge status remains public :16667. The default page must be the Cloud Workbench: left activity rail/resource tree, center Agent conversation/trace, right hardware/evidence side panel, and bottom command/input area.

Cloud Web #59 diagnostics move under the secondary diagnostics route. Keep diagnostics/status and diagnostics/gate available for read-only topology/route/health/evidence review, but do not use Gate or M3 Diagnostics as the default homepage.

npm run web:m3-readonly remains the current safety guard until replaced by a workbench-aware equivalent. It must keep same-origin /v1, read-only /json-rpc, source/gate report, and blocked report boundaries; it must fail on hardware write APIs, endpoint drift, missing SOURCE/DRY-RUN/DEV-LIVE or BLOCKED source labels, and edge-only observations presented as M3 DEV-LIVE.

The default CLI MVP command is a JSON dry-run plan even when --dry-run is not provided. It must include the DEV endpoint, artifact/health counts, two gateway sessions, two box resources, the patch-panel status, agent and worker session IDs, evidence records, and a statement that no DEV/PROD changes were made.

The dry-run shape is support evidence only. It cannot be promoted to M3 PASS unless the real DEV loop above is present and traceable.

The live DEV entry point is intentionally separate:

node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --live --confirm-dev --confirmed-non-production

When the M0-M5 gate report still has open blockers, the live command must stop with code: "BLOCKED" and include the blocker evidence. It must not use fixture data, UniDesk services, PROD, heavy e2e, restarts, or secret reads to produce a green result.

Phase Procedures

Source-Only Procedure

Use this gate for contract review, static checks, fixture checks, and manifest review. It is the default state before any runtime action is authorized.

Entry:

• git fetch origin main has completed.
• The working tree is based on latest origin/main.
• The change or review does not require secrets, kube access, tunnel access, or a deployment command.
• The operator has identified the relevant contract source: README.md, protocol/, deploy/deploy.schema.json, docs/dev-acceptance-matrix.md, M0/M1/M3 docs, or MVP fixtures.

Exit:

• Frozen DEV endpoint remains http://74.48.78.17:16667.
• PROD is not introduced as an acceptance target.
• Frozen service IDs remain HWLAB service IDs.
• JSON, schema, script syntax, and documented contract checks pass.
• Any unresolved issue is logged under one of the failure classes below.

Local Smoke Procedure

Use this gate for localhost skeleton checks and in-process model checks.

Entry:

• Source-only gate passed.
• Commands are known local checks such as npm run m1:smoke, node scripts/m3-hardware-loop-smoke.mjs, or targeted node --test.
• No command calls http://74.48.78.17:16667, :16666, kube, frp, edge runtime, or real hardware.

Exit:

• Local services start on loopback or ephemeral ports and shut down cleanly.
• Patch-panel-only propagation is preserved.
• CLI output states dry-run or local boundary when applicable.
• Audit and evidence expectations are shape-checked but not claimed as persisted DEV records.

Dry-Run Procedure

Use this gate for CLI, web, runtime plan, agent loop, and MVP e2e readiness without mutation.

Entry:

• Source-only and required local smoke gates passed.
• The dry-run command has an explicit dry-run flag or documented fixture input.
• The operator has confirmed the command cannot deploy, restart services, read secrets, write to real hardware, or mutate DEV/PROD state.

Exit:

• Output identifies the DEV endpoint and MVP route.
• Output includes both required loops: hardware trusted closed loop and agent automation closed loop.
• Output includes expected trace, audit, evidence, and cleanup records.
• Output explicitly states that no DEV/PROD changes were made.

DEV Runtime Smoke Procedure

Use this gate only after commander authorization for read-only DEV observation. This runbook does not grant that authorization.

Entry:

• M0, M1, and required dry-runs passed.
• Commander recorded an authorized DEV-only, read-only smoke window.
• The operator has the expected artifact list and service IDs.
• The planned observations require no secret reads and no restart.
• Stop criteria and rollback owner are recorded.

Exit:

• DEV endpoint responds through the HWLAB route, not a UniDesk substitute.
• Edge, frp, D601 router, cloud API/web, gateway or simulator, patch-panel, agent manager, agent worker, agent skills, and CLI observations match docs/dev-acceptance-matrix.md where applicable.
• Every observed runtime component identifies dev and a HWLAB service ID.
• Artifact identity is traceable or a not_applicable reason is recorded.
• Any timeout, stale health, wrong namespace, wrong endpoint, or missing identity is classified before further action.

MVP E2E Procedure

Use this gate only for a bounded DEV-only MVP acceptance run after DEV runtime smoke has passed. It must remain much narrower than a heavyweight e2e suite.

Entry:

• DEV runtime smoke passed in the same acceptance window or was explicitly accepted by the commander as still current.
• The MVP scenario is named and scoped to one project, one gateway path, one patch-panel wiring config, one agent session, and one worker session unless the commander records a narrower or equivalent scope.
• Expected hardware operation, cleanup action, evidence target, and stop criteria are recorded.
• No PROD, secret read, restart, or UniDesk runtime substitute is involved.

Exit:

• Project is created or selected.
• Gateway session starts for hwlab-gateway or hwlab-gateway-simu.
• Box resources and capabilities are registered.
• Patch-panel wiring config passes topology validation and owns routing.
• Agent and worker sessions start with scoped identity.
• Hardware operation is requested through the HWLAB JSON-RPC path.
• Trace events cover accepted, dispatched, executed, completed, evidence, and cleanup milestones.
• Audit events cover mutating steps and final outcome.
• Evidence record is persisted or, for a dry-run, fixture-bound and clearly marked as not persisted.
• Cleanup completes and no worker, gateway, patch, project, or hardware state leak remains.

Failure Classes

Classify failures before retrying. If more than one class applies, choose the earliest blocker in the run path and mention secondary symptoms in the log.

Failure Class	Typical Symptoms	Immediate Operator Action
Contract failure	Schema parse failure, JSON-RPC envelope mismatch, service ID drift, endpoint drift, topology rule mismatch, audit field drift, unsupported PROD target in an MVP artifact.	Stop phase; link the failing file or command; keep investigation source-only until contract drift is fixed.
Image/version failure	Missing image, mutable tag, missing digest, health reports wrong commit, skill bundle has no commit ID, web or CLI artifact cannot be tied to source.	Stop runtime promotion; require artifact rebuild or provenance fix; do not substitute another service.
k3s/frp/edge failure	DEV ingress timeout, wrong port, edge route missing, frp tunnel down, D601 router namespace mismatch, stale route, non-HWLAB response.	Stop DEV runtime smoke; record route segment and timestamp; do not restart UniDesk, Code Queue, backend-core, or unrelated infrastructure.
Agent failure	Agent manager unavailable, worker session unscoped, missing agent/worker IDs, unsafe mutation, skill mismatch, worker cleanup leak, missing trace event.	Stop agent loop or MVP e2e; preserve logs and IDs; do not rerun with broader permissions or secret access.
Hardware loop failure	Gateway unavailable, box resource missing, capability mismatch, direct box mutation, patch-panel bypass, topology conflict, DO/DI observation mismatch.	Stop hardware loop; leave hardware state unchanged when possible; require patch-panel/topology review before retry.
Evidence/audit failure	Missing audit event, missing evidence record, trace/audit/evidence IDs not linked, wrong environment, missing operation ID, artifact hash mismatch, cleanup not evidenced.	Stop acceptance; do not claim MVP pass; repair evidence chain or rerun only the smallest authorized DEV-only scope.

DEV-Only MVP Acceptance Path

The commander may call MVP accepted only when all items below are true:

1. M0 source-only contract gate passed.
2. M1 local smoke gate passed.
3. M2 dry-run readiness passed and any real DEV runtime smoke was explicitly authorized, read-only, and passed.
4. M3 hardware trusted loop passed locally and, when authorized, through the DEV HWLAB route with patch-panel ownership.
5. M4 agent automation loop passed dry-run or source contract checks and, when authorized, through scoped DEV agent and worker sessions.
6. M5 dry-run passed, then the bounded DEV-only MVP e2e completed with trace, audit, evidence, and cleanup records.
7. The evidence log contains no unclassified failure, safety blocker, PROD action, secret read, unauthorized restart, heavyweight e2e, or UniDesk runtime substitution.

If any item is missing, the result is blocked or partially passed, not MVP accepted.