pikasTech-HWLAB/docs/operator-runbook.md

# HWLAB Operator Runbook

This runbook is the manual acceptance guide for commanders and operators during
the first HWLAB DEV-only MVP. It defines when a phase may start, when it must
stop, how failures are classified, and which actions are prohibited.

The runbook supports M0-M5 and the DEV-only MVP path. It does not authorize a
real DEV deployment, PROD deployment, secret access, service restart, or
heavyweight e2e run by itself.

## Fixed Boundary

- Repository: `pikasTech/HWLAB`.
- MVP environment: `dev`.
- DEV frontend endpoint: `http://74.48.78.17:16666`.
- DEV API/edge endpoint: `http://74.48.78.17:16667`.
- Legacy public endpoints `:6666` and `:6667` are deprecated and are not MVP acceptance targets.
- Internal k3s service ports may still use `6667`; internal `6667` is not a public acceptance endpoint.
- Runtime route: client, master `hwlab-edge-proxy`, `frp`, D601
  `hwlab-dev/hwlab-router`, then HWLAB cloud/runtime services.
- HWLAB runtime services must keep the frozen service IDs in `README.md`.
- UniDesk services may support scheduling, CI, or CD only. They are not accepted
  as substitutes for HWLAB runtime services.
- Code Agent provider credentials are DEV-only runtime inputs: `OPENAI_API_KEY`
  must come from Secret `hwlab-code-agent-provider/openai-api-key`, and
  `HWLAB_CODE_AGENT_OPENAI_BASE_URL` must route through the DEV egress/proxy
  path, not direct public `api.openai.com`.
- Cloud API DB readiness is layered as DNS, TCP, auth, and schema. `.invalid`
  hosts are never desired DEV runtime targets; they are negative fixtures only.

## M3 MVP Acceptance

M3 MVP PASS is the smallest real DEV evidence chain that proves the hardware
trusted loop. It requires:

- two box simulators with different observable identities;
- two gateway simulators with different observable identities;
- one active patch-panel participant;
- the traced `res_boxsimu_1:DO1 -> hwlab-patch-panel -> res_boxsimu_2:DI1`
  operation with trace, audit, operation, and evidence IDs;
- no box loopback substitution, no front-end direct state edit, and no
  UniDesk runtime or proxy substitution.

Cloud Workbench, generic console work, artifact report generation, and
desired-state planning are support surfaces only. They may help operators, but
they do not count as M3 PASS on their own and cannot replace the loop evidence
above.

The detailed step order and misclassification bans live in
`docs/reference/m3-loop-rollout-runbook.md`; this runbook and the acceptance
matrix mirror that contract.

## Runner Issue Visibility Handoff

GitHub issue and PR threads are auxiliary context only. The runner prompt must be self-contained and must include the full task objective, background, constraints, acceptance, and validation commands.

Do not assume issue comments are readable in the runner. If a task depends on GitHub visibility, use `docs/reference/runner-issue-visibility-handoff.md` as the stable prompt contract and run `node scripts/runner-issue-visibility-preflight.mjs` before queueing the work.

The current runner class may have git transport access while GitHub REST issue and PR reads are blocked. Treat that as a prompt-handoff requirement, not a reason to omit context from the prompt.

## Hard Stop Actions

Stop the run immediately and classify the event as a safety blocker if any
operator, script, or agent attempts one of these actions:

- Perform a real DEV deployment from this runbook.
- Deploy, smoke, or mutate PROD in any form.
- Read, print, copy, or infer secrets, tokens, kubeconfig, tunnel credentials,
  API keys, or private environment values.
- Restart UniDesk, Code Queue, or `backend-core`.
- Run heavyweight e2e, destructive hardware tests, load tests, or unbounded
  browser automation.
- Use UniDesk backend, provider-gateway, or microservice proxy as the HWLAB
  runtime instead of `hwlab-cloud-api`, `hwlab-router`, `hwlab-gateway`,
  `hwlab-patch-panel`, agent services, and the other frozen HWLAB services.
- Promote SOURCE, LOCAL, DRY-RUN, or edge-only result as real DEV evidence.
- Treat Cloud Workbench polish, generic console work, artifact reports, or
  desired state plans as M3 PASS evidence.
- Leave an emergency DEV runtime hotfix as the only source of truth after it
  unblocks a P0 path. The follow-up must codify source manifests, contracts,
  docs, and validation without rerunning the live patch.

## Evidence Log

Every phase result must be recorded in the operator log before the next phase
starts. The log may be a ticket comment, incident note, or release checklist,
but it must include:

- phase name and milestone IDs;
- command or observation used;
- commit SHA;
- environment, which must be `dev` for MVP evidence;
- endpoint, when a network observation is made;
- pass, fail, blocked, or not-applicable status;
- failure class, when failed or blocked;
- artifact identity for deployable runtime observations: `serviceId`,
  `commitId`, `image`, `tag`, `digest` or `not_applicable` reason,
  `buildSource`, `deployEnv`, and `healthTimestamp`;
- evidence, audit, trace, operation, agent session, worker session, gateway
  session, and project IDs when the phase touches an MVP loop.

## DEV Gate Aggregator v2

Use `/tmp/hwlab-dev-gate/dev-m5-gate-aggregator-v2.json` and
`/tmp/hwlab-dev-gate/dev-m5-gate-aggregator-v2.md` as the current report-only M5
gate summary for `pikasTech/HWLAB#58`. Refresh them only from committed
fixtures:

```sh
node scripts/dev-evidence-blocker-aggregator.mjs --report /tmp/hwlab-dev-gate/report.json
node scripts/validate-dev-gate-report.mjs
```

The aggregator separates `SOURCE`, `LOCAL`, `DRY-RUN`, `DEV-LIVE`, and
`BLOCKED`. It must never promote source, local, dry-run, contract-only,
not-run, or failed-live evidence into `DEV-LIVE`.

Current unblock order:

1. Provide an approved Node 20 DEV builder base image for artifact publish.
2. Publish every frozen HWLAB service artifact and record immutable registry
   digests for the selected `origin/main` commit.
3. Restore read-only D601 `hwlab-dev` k3s observability with
   `KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl` or an approved
   kubeconfig/maintenance bridge, without reading Secrets. Do not use a default
   `docker-desktop` kubeconfig as DEV evidence; the control-plane incident is
   tracked in pikasTech/unidesk#138.
4. Prove DEV `hwlab-cloud-api` DB env and health readiness with redacted values
   only.
5. Repair the `:16667`/`frp`/edge/router path and rerun read-only DEV edge
   health before any M3, M4, or M5 live loop.

For #164-style hotfix follow-up, treat provider and DB fixes as independent
contracts:

- Code Agent provider: source and k8s workload env must declare
  `OPENAI_API_KEY` from `hwlab-code-agent-provider/openai-api-key` and
  `HWLAB_CODE_AGENT_OPENAI_BASE_URL` through DEV egress/proxy. Validation may
  prove Secret ref shape and key-presence evidence only; it must not print the
  API key.
- DB live: source and runtime health must reject `.invalid` as a DEV target and
  report DNS/TCP/auth/schema readiness separately. `liveDbEvidence=true` is
  allowed only after live runtime connectivity evidence, never from fixtures or
  env presence alone.
- Follow-up automation: update `deploy/deploy.json`,
  `deploy/k8s/base/workloads.yaml`, source contract code, docs, and tests; do
  not apply manifests, restart services, mutate PROD, or repeat the manual
  hotfix from the source PR.

## Phase Gates

Run phases in order. A later phase may start only after every earlier required
exit condition is met or explicitly marked not applicable with a DEV-only
reason.

| Phase | Covers | Entry Conditions | Exit Conditions |
| --- | --- | --- | --- |
| Source-only | M0 contract, M2/M4/M5 source readiness | Worktree is based on latest `origin/main`; no secret or deployment access is needed; target files are docs, protocol, schema, fixtures, scripts, or manifests only. | Contract files parse; frozen endpoint and service IDs are unchanged; `npm run check` or the narrow validation command passes; no DEV/PROD mutation occurred. |
| Local smoke | M1 plus local M3/M4 contract smoke | Source-only has passed; commands use localhost or fixture data only; ports are ephemeral; no DEV endpoint call is needed. | Smoke commands pass; generated output states local or dry-run boundary; no real hardware, tunnel, kube, or deploy action occurred. |
| Dry-run | M2/M4/M5 readiness without runtime mutation | Source-only and local smoke pass; CLI/runtime fixtures identify `dev` and `http://74.48.78.17:16667`; operator confirms command includes `--dry-run` where applicable. | Dry-run output names MVP route, hardware trusted closed loop, agent automation closed loop, evidence records, and cleanup plan; output states no DEV/PROD changes were made. |
| DEV runtime smoke | M2 DEV runtime observation | M0-M1 and required dry-runs pass; commander explicitly authorizes a read-only DEV smoke window; expected HWLAB artifacts and rollback owner are known; no secret reads are required. | Read-only health/status observations match `docs/dev-acceptance-matrix.md`; every observed service is HWLAB, in `dev`, and traceable to commit/image/tag/digest or a documented `not_applicable` reason. |
| MVP e2e | M3/M4/M5 DEV-only MVP acceptance | DEV runtime smoke has passed; commander explicitly authorizes a narrow DEV-only MVP smoke; project/gateway/agent/hardware scope is named; cleanup owner and stop criteria are recorded. | One bounded DEV-only flow completes: project, gateway session, resources, patch-panel wiring, agent and worker sessions, hardware operation, trace events, audit events, evidence record, and cleanup. No PROD or heavyweight e2e action occurred. |

## Milestone Coverage

| Milestone | Operator Gate | Required Acceptance |
| --- | --- | --- |
| M0 contract audit | Source-only | `docs/m0-contract-audit.md`, protocol docs, schemas, deploy schema, and M0 examples agree on service IDs, JSON-RPC envelope, errors, audit fields, topology, and DEV-only deploy shape. |
| M1 local contract smoke | Local smoke | `npm run m1:smoke` or `node scripts/m1-contract-smoke.mjs` passes on localhost and confirms the local cloud-api runtime, simulator contracts, patch-panel routing, CLI dry-run boundary, and no DEV/PROD changes. |
| M2 DEV deploy/runtime smoke | Dry-run, then authorized DEV runtime smoke | Source/manifests and `docs/dev-acceptance-matrix.md` identify the DEV route through edge, `frp`, and D601 router. Real DEV observation is read-only and must not be performed without commander authorization. |
| M3 hardware trusted loop | Local smoke, then MVP e2e | `node scripts/m3-hardware-loop-smoke.mjs` proves the local contract, but M3 MVP PASS in DEV requires the live `res_boxsimu_1:DO1 -> hwlab-patch-panel -> res_boxsimu_2:DI1` evidence chain with distinct box/gateway identities and operation, trace, audit, and evidence IDs. |
| M4 agent automation loop | Source-only or dry-run, then MVP e2e | Agent manager, worker, skills, session lifecycle, scoped workspace, trace events, cleanup, and evidence chain are observable in fixtures or dry-run before any DEV runtime action. |
| M5 MVP e2e dry-run and acceptance | Dry-run, then MVP e2e | `npm run cli:dry-run` or equivalent names the MVP route, closed loops, evidence, and cleanup without mutation. Real acceptance is DEV-only, bounded, and requires prior DEV runtime smoke pass. |

## L6 CLI/Web Acceptance Entry

Use the L6 entry points for external manual acceptance triage. They read the
checked-in `GitHub issue/PR comments` and
`fixtures/mvp/m5-e2e/dry-run-plan.json`; they do not call UniDesk runtime and
do not claim live DEV evidence from fixtures.

Safe source and dry-run checks:

```sh
npm run l6:smoke
npm run web:m3-readonly
npm run web:check
npm run cli:dry-run
node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp
```

For Cloud Web #99, `docs/cloud-web-workbench.md` is the frontend design and
acceptance contract. The browser endpoint remains public `:16666`, while
API/edge status remains public `:16667`. The default page must be the Cloud
Workbench: left activity rail/resource tree, center Agent conversation/trace,
right hardware/evidence side panel, and bottom command/input area.

Cloud Web #59 diagnostics move under the secondary diagnostics route. Keep
`diagnostics/status` and `diagnostics/gate` available for read-only
topology/route/health/evidence review, but do not use Gate or M3 Diagnostics
as the default homepage.

`npm run web:m3-readonly` remains the current safety guard until replaced by a
workbench-aware equivalent. It must keep same-origin `/v1`, read-only
`/json-rpc`, source/gate report, and blocked report boundaries; it must fail
on hardware write APIs, endpoint drift, missing `SOURCE`/`DRY-RUN`/`DEV-LIVE`
or `BLOCKED` source labels, and edge-only observations presented as M3
`DEV-LIVE`.

The default CLI MVP command is a JSON dry-run plan even when `--dry-run` is not
provided. It must include the DEV endpoint, artifact/health counts, two
gateway sessions, two box resources, the patch-panel status, agent and worker
session IDs, evidence records, and a statement that no DEV/PROD changes were
made.

The dry-run shape is support evidence only. It cannot be promoted to M3 PASS
unless the real DEV loop above is present and traceable.

The live DEV entry point is intentionally separate:

```sh
node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --live --confirm-dev --confirmed-non-production
```

When the M0-M5 gate report still has open blockers, the live command must stop
with `code: "BLOCKED"` and include the blocker evidence. It must not use
fixture data, UniDesk services, PROD, heavy e2e, restarts, or secret reads to
produce a green result.

## Phase Procedures

### Source-Only Procedure

Use this gate for contract review, static checks, fixture checks, and manifest
review. It is the default state before any runtime action is authorized.

Entry:

- `git fetch origin main` has completed.
- The working tree is based on latest `origin/main`.
- The change or review does not require secrets, kube access, tunnel access, or
  a deployment command.
- The operator has identified the relevant contract source:
  `README.md`, `protocol/`, `deploy/deploy.schema.json`,
  `docs/dev-acceptance-matrix.md`, M0/M1/M3 docs, or MVP fixtures.

Exit:

- Frozen DEV endpoint remains `http://74.48.78.17:16667`.
- PROD is not introduced as an acceptance target.
- Frozen service IDs remain HWLAB service IDs.
- JSON, schema, script syntax, and documented contract checks pass.
- Any unresolved issue is logged under one of the failure classes below.

### Local Smoke Procedure

Use this gate for localhost skeleton checks and in-process model checks.

Entry:

- Source-only gate passed.
- Commands are known local checks such as `npm run m1:smoke`,
  `node scripts/m3-hardware-loop-smoke.mjs`, or targeted `node --test`.
- No command calls `http://74.48.78.17:16667`, `:16666`, kube, `frp`, edge
  runtime, or real hardware.

Exit:

- Local services start on loopback or ephemeral ports and shut down cleanly.
- Patch-panel-only propagation is preserved.
- CLI output states dry-run or local boundary when applicable.
- Audit and evidence expectations are shape-checked but not claimed as
  persisted DEV records.

### Dry-Run Procedure

Use this gate for CLI, web, runtime plan, agent loop, and MVP e2e readiness
without mutation.

Entry:

- Source-only and required local smoke gates passed.
- The dry-run command has an explicit dry-run flag or documented fixture input.
- The operator has confirmed the command cannot deploy, restart services, read
  secrets, write to real hardware, or mutate DEV/PROD state.

Exit:

- Output identifies the DEV endpoint and MVP route.
- Output includes both required loops: hardware trusted closed loop and agent
  automation closed loop.
- Output includes expected trace, audit, evidence, and cleanup records.
- Output explicitly states that no DEV/PROD changes were made.

### DEV Runtime Smoke Procedure

Use this gate only after commander authorization for read-only DEV observation.
This runbook does not grant that authorization.

Entry:

- M0, M1, and required dry-runs passed.
- Commander recorded an authorized DEV-only, read-only smoke window.
- The operator has the expected artifact list and service IDs.
- The planned observations require no secret reads and no restart.
- Stop criteria and rollback owner are recorded.

Exit:

- DEV endpoint responds through the HWLAB route, not a UniDesk substitute.
- Edge, `frp`, D601 router, cloud API/web, gateway or simulator,
  patch-panel, agent manager, agent worker, agent skills, and CLI observations
  match `docs/dev-acceptance-matrix.md` where applicable.
- Every observed runtime component identifies `dev` and a HWLAB service ID.
- Artifact identity is traceable or a `not_applicable` reason is recorded.
- Any timeout, stale health, wrong namespace, wrong endpoint, or missing
  identity is classified before further action.

### MVP E2E Procedure

Use this gate only for a bounded DEV-only MVP acceptance run after DEV runtime
smoke has passed. It must remain much narrower than a heavyweight e2e suite.

Entry:

- DEV runtime smoke passed in the same acceptance window or was explicitly
  accepted by the commander as still current.
- The MVP scenario is named and scoped to one project, one gateway path, one
  patch-panel wiring config, one agent session, and one worker session unless
  the commander records a narrower or equivalent scope.
- Expected hardware operation, cleanup action, evidence target, and stop
  criteria are recorded.
- No PROD, secret read, restart, or UniDesk runtime substitute is involved.

Exit:

- Project is created or selected.
- Gateway session starts for `hwlab-gateway` or `hwlab-gateway-simu`.
- Box resources and capabilities are registered.
- Patch-panel wiring config passes topology validation and owns routing.
- Agent and worker sessions start with scoped identity.
- Hardware operation is requested through the HWLAB JSON-RPC path.
- Trace events cover accepted, dispatched, executed, completed, evidence, and
  cleanup milestones.
- Audit events cover mutating steps and final outcome.
- Evidence record is persisted or, for a dry-run, fixture-bound and clearly
  marked as not persisted.
- Cleanup completes and no worker, gateway, patch, project, or hardware state
  leak remains.

## Failure Classes

Classify failures before retrying. If more than one class applies, choose the
earliest blocker in the run path and mention secondary symptoms in the log.

| Failure Class | Typical Symptoms | Immediate Operator Action |
| --- | --- | --- |
| Contract failure | Schema parse failure, JSON-RPC envelope mismatch, service ID drift, endpoint drift, topology rule mismatch, audit field drift, unsupported PROD target in an MVP artifact. | Stop phase; link the failing file or command; keep investigation source-only until contract drift is fixed. |
| Image/version failure | Missing image, mutable tag, missing digest, health reports wrong commit, skill bundle has no commit ID, web or CLI artifact cannot be tied to source. | Stop runtime promotion; require artifact rebuild or provenance fix; do not substitute another service. |
| k3s/frp/edge failure | DEV ingress timeout, wrong port, edge route missing, `frp` tunnel down, D601 router namespace mismatch, stale route, non-HWLAB response. | Stop DEV runtime smoke; record route segment and timestamp; do not restart UniDesk, Code Queue, `backend-core`, or unrelated infrastructure. |
| Agent failure | Agent manager unavailable, worker session unscoped, missing agent/worker IDs, unsafe mutation, skill mismatch, worker cleanup leak, missing trace event. | Stop agent loop or MVP e2e; preserve logs and IDs; do not rerun with broader permissions or secret access. |
| Hardware loop failure | Gateway unavailable, box resource missing, capability mismatch, direct box mutation, patch-panel bypass, topology conflict, DO/DI observation mismatch. | Stop hardware loop; leave hardware state unchanged when possible; require patch-panel/topology review before retry. |
| Evidence/audit failure | Missing audit event, missing evidence record, trace/audit/evidence IDs not linked, wrong environment, missing operation ID, artifact hash mismatch, cleanup not evidenced. | Stop acceptance; do not claim MVP pass; repair evidence chain or rerun only the smallest authorized DEV-only scope. |

## DEV-Only MVP Acceptance Path

The commander may call MVP accepted only when all items below are true:

1. M0 source-only contract gate passed.
2. M1 local smoke gate passed.
3. M2 dry-run readiness passed and any real DEV runtime smoke was explicitly
   authorized, read-only, and passed.
4. M3 hardware trusted loop passed locally and, when authorized, through the
   DEV HWLAB route with patch-panel ownership.
5. M4 agent automation loop passed dry-run or source contract checks and, when
   authorized, through scoped DEV agent and worker sessions.
6. M5 dry-run passed, then the bounded DEV-only MVP e2e completed with trace,
   audit, evidence, and cleanup records.
7. The evidence log contains no unclassified failure, safety blocker, PROD
   action, secret read, unauthorized restart, heavyweight e2e, or UniDesk
   runtime substitution.

If any item is missing, the result is blocked or partially passed, not MVP
accepted.