HWLAB Code Queue
247 lines
14 KiB
Markdown
247 lines
14 KiB
Markdown
# HWLAB Operator Runbook
|
|
|
|
This runbook is the manual acceptance guide for commanders and operators during
|
|
the first HWLAB DEV-only MVP. It defines when a phase may start, when it must
|
|
stop, how failures are classified, and which actions are prohibited.
|
|
|
|
The runbook supports M0-M5 and the DEV-only MVP path. It does not authorize a
|
|
real DEV deployment, PROD deployment, secret access, service restart, or
|
|
heavyweight e2e run by itself.
|
|
|
|
## Fixed Boundary
|
|
|
|
- Repository: `pikasTech/HWLAB`.
|
|
- MVP environment: `dev`.
|
|
- DEV endpoint: `http://74.48.78.17:6667`.
|
|
- PROD endpoint `:6666` is reserved and is not an MVP acceptance target.
|
|
- Runtime route: client, master `hwlab-edge-proxy`, `frp`, D601
|
|
`hwlab-dev/hwlab-router`, then HWLAB cloud/runtime services.
|
|
- HWLAB runtime services must keep the frozen service IDs in `README.md`.
|
|
- UniDesk services may support scheduling, CI, or CD only. They are not accepted
|
|
as substitutes for HWLAB runtime services.
|
|
|
|
## Runner Issue Visibility Handoff
|
|
|
|
GitHub issue and PR threads are auxiliary context only. The runner prompt must be self-contained and must include the full task objective, background, constraints, acceptance, and validation commands.
|
|
|
|
Do not assume issue comments are readable in the runner. If a task depends on GitHub visibility, use `docs/reference/runner-issue-visibility-handoff.md` as the stable prompt contract and run `node scripts/runner-issue-visibility-preflight.mjs` before queueing the work.
|
|
|
|
The current runner class may have git transport access while GitHub REST issue and PR reads are blocked. Treat that as a prompt-handoff requirement, not a reason to omit context from the prompt.
|
|
|
|
## Hard Stop Actions
|
|
|
|
Stop the run immediately and classify the event as a safety blocker if any
|
|
operator, script, or agent attempts one of these actions:
|
|
|
|
- Perform a real DEV deployment from this runbook.
|
|
- Deploy, smoke, or mutate PROD in any form.
|
|
- Read, print, copy, or infer secrets, tokens, kubeconfig, tunnel credentials,
|
|
API keys, or private environment values.
|
|
- Restart UniDesk, Code Queue, or `backend-core`.
|
|
- Run heavyweight e2e, destructive hardware tests, load tests, or unbounded
|
|
browser automation.
|
|
- Use UniDesk backend, provider-gateway, or microservice proxy as the HWLAB
|
|
runtime instead of `hwlab-cloud-api`, `hwlab-router`, `hwlab-gateway`,
|
|
`hwlab-patch-panel`, agent services, and the other frozen HWLAB services.
|
|
- Promote a source-only, local smoke, or dry-run result as real DEV evidence.
|
|
|
|
## Evidence Log
|
|
|
|
Every phase result must be recorded in the operator log before the next phase
|
|
starts. The log may be a ticket comment, incident note, or release checklist,
|
|
but it must include:
|
|
|
|
- phase name and milestone IDs;
|
|
- command or observation used;
|
|
- commit SHA;
|
|
- environment, which must be `dev` for MVP evidence;
|
|
- endpoint, when a network observation is made;
|
|
- pass, fail, blocked, or not-applicable status;
|
|
- failure class, when failed or blocked;
|
|
- artifact identity for deployable runtime observations: `serviceId`,
|
|
`commitId`, `image`, `tag`, `digest` or `not_applicable` reason,
|
|
`buildSource`, `deployEnv`, and `healthTimestamp`;
|
|
- evidence, audit, trace, operation, agent session, worker session, gateway
|
|
session, and project IDs when the phase touches an MVP loop.
|
|
|
|
## Phase Gates
|
|
|
|
Run phases in order. A later phase may start only after every earlier required
|
|
exit condition is met or explicitly marked not applicable with a DEV-only
|
|
reason.
|
|
|
|
| Phase | Covers | Entry Conditions | Exit Conditions |
|
|
| --- | --- | --- | --- |
|
|
| Source-only | M0 contract, M2/M4/M5 source readiness | Worktree is based on latest `origin/main`; no secret or deployment access is needed; target files are docs, protocol, schema, fixtures, scripts, or manifests only. | Contract files parse; frozen endpoint and service IDs are unchanged; `npm run check` or the narrow validation command passes; no DEV/PROD mutation occurred. |
|
|
| Local smoke | M1 plus local M3/M4 contract smoke | Source-only has passed; commands use localhost or fixture data only; ports are ephemeral; no DEV endpoint call is needed. | Smoke commands pass; generated output states local or dry-run boundary; no real hardware, tunnel, kube, or deploy action occurred. |
|
|
| Dry-run | M2/M4/M5 readiness without runtime mutation | Source-only and local smoke pass; CLI/runtime fixtures identify `dev` and `http://74.48.78.17:6667`; operator confirms command includes `--dry-run` where applicable. | Dry-run output names MVP route, hardware trusted closed loop, agent automation closed loop, evidence records, and cleanup plan; output states no DEV/PROD changes were made. |
|
|
| DEV runtime smoke | M2 DEV runtime observation | M0-M1 and required dry-runs pass; commander explicitly authorizes a read-only DEV smoke window; expected HWLAB artifacts and rollback owner are known; no secret reads are required. | Read-only health/status observations match `docs/dev-acceptance-matrix.md`; every observed service is HWLAB, in `dev`, and traceable to commit/image/tag/digest or a documented `not_applicable` reason. |
|
|
| MVP e2e | M3/M4/M5 DEV-only MVP acceptance | DEV runtime smoke has passed; commander explicitly authorizes a narrow DEV-only MVP smoke; project/gateway/agent/hardware scope is named; cleanup owner and stop criteria are recorded. | One bounded DEV-only flow completes: project, gateway session, resources, patch-panel wiring, agent and worker sessions, hardware operation, trace events, audit events, evidence record, and cleanup. No PROD or heavyweight e2e action occurred. |
|
|
|
|
## Milestone Coverage
|
|
|
|
| Milestone | Operator Gate | Required Acceptance |
|
|
| --- | --- | --- |
|
|
| M0 contract audit | Source-only | `docs/m0-contract-audit.md`, protocol docs, schemas, deploy schema, and M0 examples agree on service IDs, JSON-RPC envelope, errors, audit fields, topology, and DEV-only deploy shape. |
|
|
| M1 local contract smoke | Local smoke | `npm run m1:smoke` or `node scripts/m1-contract-smoke.mjs` passes on localhost and confirms skeleton services, simulator contracts, patch-panel routing, CLI dry-run boundary, and no DEV/PROD changes. |
|
|
| M2 DEV deploy/runtime smoke | Dry-run, then authorized DEV runtime smoke | Source/manifests and `docs/dev-acceptance-matrix.md` identify the DEV route through edge, `frp`, and D601 router. Real DEV observation is read-only and must not be performed without commander authorization. |
|
|
| M3 hardware trusted loop | Local smoke, then MVP e2e | `node scripts/m3-hardware-loop-smoke.mjs` proves the local `box-simu-1 DO1 -> box-simu-2 DI1` patch-panel contract. Real DEV acceptance must preserve patch-panel ownership and produce trace, audit, and evidence records. |
|
|
| M4 agent automation loop | Source-only or dry-run, then MVP e2e | Agent manager, worker, skills, session lifecycle, scoped workspace, trace events, cleanup, and evidence chain are observable in fixtures or dry-run before any DEV runtime action. |
|
|
| M5 MVP e2e dry-run and acceptance | Dry-run, then MVP e2e | `npm run cli:dry-run` or equivalent names the MVP route, closed loops, evidence, and cleanup without mutation. Real acceptance is DEV-only, bounded, and requires prior DEV runtime smoke pass. |
|
|
|
|
## Phase Procedures
|
|
|
|
### Source-Only Procedure
|
|
|
|
Use this gate for contract review, static checks, fixture checks, and manifest
|
|
review. It is the default state before any runtime action is authorized.
|
|
|
|
Entry:
|
|
|
|
- `git fetch origin main` has completed.
|
|
- The working tree is based on latest `origin/main`.
|
|
- The change or review does not require secrets, kube access, tunnel access, or
|
|
a deployment command.
|
|
- The operator has identified the relevant contract source:
|
|
`README.md`, `protocol/`, `deploy/deploy.schema.json`,
|
|
`docs/dev-acceptance-matrix.md`, M0/M1/M3 docs, or MVP fixtures.
|
|
|
|
Exit:
|
|
|
|
- Frozen DEV endpoint remains `http://74.48.78.17:6667`.
|
|
- PROD is not introduced as an acceptance target.
|
|
- Frozen service IDs remain HWLAB service IDs.
|
|
- JSON, schema, script syntax, and documented contract checks pass.
|
|
- Any unresolved issue is logged under one of the failure classes below.
|
|
|
|
### Local Smoke Procedure
|
|
|
|
Use this gate for localhost skeleton checks and in-process model checks.
|
|
|
|
Entry:
|
|
|
|
- Source-only gate passed.
|
|
- Commands are known local checks such as `npm run m1:smoke`,
|
|
`node scripts/m3-hardware-loop-smoke.mjs`, or targeted `node --test`.
|
|
- No command calls `http://74.48.78.17:6667`, `:6666`, kube, `frp`, edge
|
|
runtime, or real hardware.
|
|
|
|
Exit:
|
|
|
|
- Local services start on loopback or ephemeral ports and shut down cleanly.
|
|
- Patch-panel-only propagation is preserved.
|
|
- CLI output states dry-run or local boundary when applicable.
|
|
- Audit and evidence expectations are shape-checked but not claimed as
|
|
persisted DEV records.
|
|
|
|
### Dry-Run Procedure
|
|
|
|
Use this gate for CLI, web, runtime plan, agent loop, and MVP e2e readiness
|
|
without mutation.
|
|
|
|
Entry:
|
|
|
|
- Source-only and required local smoke gates passed.
|
|
- The dry-run command has an explicit dry-run flag or documented fixture input.
|
|
- The operator has confirmed the command cannot deploy, restart services, read
|
|
secrets, write to real hardware, or mutate DEV/PROD state.
|
|
|
|
Exit:
|
|
|
|
- Output identifies the DEV endpoint and MVP route.
|
|
- Output includes both required loops: hardware trusted closed loop and agent
|
|
automation closed loop.
|
|
- Output includes expected trace, audit, evidence, and cleanup records.
|
|
- Output explicitly states that no DEV/PROD changes were made.
|
|
|
|
### DEV Runtime Smoke Procedure
|
|
|
|
Use this gate only after commander authorization for read-only DEV observation.
|
|
This runbook does not grant that authorization.
|
|
|
|
Entry:
|
|
|
|
- M0, M1, and required dry-runs passed.
|
|
- Commander recorded an authorized DEV-only, read-only smoke window.
|
|
- The operator has the expected artifact list and service IDs.
|
|
- The planned observations require no secret reads and no restart.
|
|
- Stop criteria and rollback owner are recorded.
|
|
|
|
Exit:
|
|
|
|
- DEV endpoint responds through the HWLAB route, not a UniDesk substitute.
|
|
- Edge, `frp`, D601 router, cloud API/web, gateway or simulator,
|
|
patch-panel, agent manager, agent worker, agent skills, and CLI observations
|
|
match `docs/dev-acceptance-matrix.md` where applicable.
|
|
- Every observed runtime component identifies `dev` and a HWLAB service ID.
|
|
- Artifact identity is traceable or a `not_applicable` reason is recorded.
|
|
- Any timeout, stale health, wrong namespace, wrong endpoint, or missing
|
|
identity is classified before further action.
|
|
|
|
### MVP E2E Procedure
|
|
|
|
Use this gate only for a bounded DEV-only MVP acceptance run after DEV runtime
|
|
smoke has passed. It must remain much narrower than a heavyweight e2e suite.
|
|
|
|
Entry:
|
|
|
|
- DEV runtime smoke passed in the same acceptance window or was explicitly
|
|
accepted by the commander as still current.
|
|
- The MVP scenario is named and scoped to one project, one gateway path, one
|
|
patch-panel wiring config, one agent session, and one worker session unless
|
|
the commander records a narrower or equivalent scope.
|
|
- Expected hardware operation, cleanup action, evidence target, and stop
|
|
criteria are recorded.
|
|
- No PROD, secret read, restart, or UniDesk runtime substitute is involved.
|
|
|
|
Exit:
|
|
|
|
- Project is created or selected.
|
|
- Gateway session starts for `hwlab-gateway` or `hwlab-gateway-simu`.
|
|
- Box resources and capabilities are registered.
|
|
- Patch-panel wiring config passes topology validation and owns routing.
|
|
- Agent and worker sessions start with scoped identity.
|
|
- Hardware operation is requested through the HWLAB JSON-RPC path.
|
|
- Trace events cover accepted, dispatched, executed, completed, evidence, and
|
|
cleanup milestones.
|
|
- Audit events cover mutating steps and final outcome.
|
|
- Evidence record is persisted or, for a dry-run, fixture-bound and clearly
|
|
marked as not persisted.
|
|
- Cleanup completes and no worker, gateway, patch, project, or hardware state
|
|
leak remains.
|
|
|
|
## Failure Classes
|
|
|
|
Classify failures before retrying. If more than one class applies, choose the
|
|
earliest blocker in the run path and mention secondary symptoms in the log.
|
|
|
|
| Failure Class | Typical Symptoms | Immediate Operator Action |
|
|
| --- | --- | --- |
|
|
| Contract failure | Schema parse failure, JSON-RPC envelope mismatch, service ID drift, endpoint drift, topology rule mismatch, audit field drift, unsupported PROD target in an MVP artifact. | Stop phase; link the failing file or command; keep investigation source-only until contract drift is fixed. |
|
|
| Image/version failure | Missing image, mutable tag, missing digest, health reports wrong commit, skill bundle has no commit ID, web or CLI artifact cannot be tied to source. | Stop runtime promotion; require artifact rebuild or provenance fix; do not substitute another service. |
|
|
| k3s/frp/edge failure | DEV ingress timeout, wrong port, edge route missing, `frp` tunnel down, D601 router namespace mismatch, stale route, non-HWLAB response. | Stop DEV runtime smoke; record route segment and timestamp; do not restart UniDesk, Code Queue, `backend-core`, or unrelated infrastructure. |
|
|
| Agent failure | Agent manager unavailable, worker session unscoped, missing agent/worker IDs, unsafe mutation, skill mismatch, worker cleanup leak, missing trace event. | Stop agent loop or MVP e2e; preserve logs and IDs; do not rerun with broader permissions or secret access. |
|
|
| Hardware loop failure | Gateway unavailable, box resource missing, capability mismatch, direct box mutation, patch-panel bypass, topology conflict, DO/DI observation mismatch. | Stop hardware loop; leave hardware state unchanged when possible; require patch-panel/topology review before retry. |
|
|
| Evidence/audit failure | Missing audit event, missing evidence record, trace/audit/evidence IDs not linked, wrong environment, missing operation ID, artifact hash mismatch, cleanup not evidenced. | Stop acceptance; do not claim MVP pass; repair evidence chain or rerun only the smallest authorized DEV-only scope. |
|
|
|
|
## DEV-Only MVP Acceptance Path
|
|
|
|
The commander may call MVP accepted only when all items below are true:
|
|
|
|
1. M0 source-only contract gate passed.
|
|
2. M1 local smoke gate passed.
|
|
3. M2 dry-run readiness passed and any real DEV runtime smoke was explicitly
|
|
authorized, read-only, and passed.
|
|
4. M3 hardware trusted loop passed locally and, when authorized, through the
|
|
DEV HWLAB route with patch-panel ownership.
|
|
5. M4 agent automation loop passed dry-run or source contract checks and, when
|
|
authorized, through scoped DEV agent and worker sessions.
|
|
6. M5 dry-run passed, then the bounded DEV-only MVP e2e completed with trace,
|
|
audit, evidence, and cleanup records.
|
|
7. The evidence log contains no unclassified failure, safety blocker, PROD
|
|
action, secret read, unauthorized restart, heavyweight e2e, or UniDesk
|
|
runtime substitution.
|
|
|
|
If any item is missing, the result is blocked or partially passed, not MVP
|
|
accepted.
|