# HWLAB DEV Deploy Apply

This document records the DEV-only apply/preflight path for
`pikasTech/HWLAB#33` and the apply-plan/report path for
`pikasTech/HWLAB#50`. It targets only the D601 `hwlab-dev` namespace and must
not be used for PROD.

## Scope

- Work from latest `origin/main`.
- Read only HWLAB repository deploy inputs: `deploy/deploy.json`,
  `deploy/artifact-catalog.dev.json`, and `deploy/k8s/dev`.
- Validate artifact identity, image commit tags, Kubernetes workload/service
  shape, DEV health, and cloud-api DB connectivity.
- Emit an operator-readable plan in `devDeployApply`: plan/apply boundary,
  expected artifact commit, target namespace, workloads, services, remaining
  blockers, minimal manual commands, and rollback hints.
- Plan and, in apply mode, execute a scoped replacement for DEV suspended
  template Jobs whose image changed but whose Kubernetes `spec.template` is
  immutable.
- Stop before mutation when artifact publish evidence, registry digests,
  `kubectl`, DEV health, or cloud-api DB connectivity are missing.

## Commands

Run a preflight that is expected to report blockers on the current skeleton:

```sh
node --check scripts/dev-deploy-apply.mjs
node --check scripts/src/dev-deploy-apply.mjs
node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report
node scripts/validate-dev-gate-report.mjs
```

Run the existing CLI dry-run surface:

```sh
node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run
```

Only after CI publish evidence and executor prerequisites exist, a real DEV
apply requires explicit non-production confirmation:

```sh
node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report
```

The script refuses PROD flags, secret-read flags, heavyweight e2e flags, and
force-push flags. It also refuses to apply if the DEV artifact catalog still
contains skeleton-only unpublished images.

## Report Contract

The machine-readable result is written to
`reports/dev-gate/dev-deploy-report.json` when `--write-report` is used. The
report remains compatible with the #33 DEV gate validator and adds these
operator fields:

- `devDeployApply.conclusion`: `blocked` while blockers are open, otherwise
  `ready` for human approval.
- `devDeployApply.applyBoundary`: default no-write behavior, required apply
  flags, DEV-only write scope, no-write scope, and forbidden actions.
- `devDeployApply.artifactPlan`: expected artifact commit, source commit,
  deploy/catalog commit IDs, publish/registry status, and unpublished services.
- `devDeployApply.target`: DEV endpoint and `hwlab-dev` namespace.
- `devDeployApply.workloadPlan` and `servicePlan`: planned Kubernetes
  workloads, containers, images, service IDs, ports, and replica intent.
- `devDeployApply.templateJobReplacementPolicy`: the DEV-only allowlist for
  suspended template Job replacement. The only allowlisted Jobs are
  `hwlab-agent-worker-template` and `hwlab-cli-template` in `hwlab-dev`.
- `devDeployApply.templateJobReplacements`: per-Job replacement decisions with
  namespace, Job name, old image, new image, and result (`planned`, `replaced`,
  `not_needed`, `not_found`, or blocked/failure states).
- `devDeployApply.manualCommands`: dry-run/report commands before human
  approval; the apply command is present only when the plan is ready.
- `devDeployApply.rollbackHint`: pre-apply capture commands, deployment
  rollbacks, suspended job cleanup, and post-rollback health checks.
- `devDeployApply.remainingBlockers`: open blockers with concrete unblock
  hints. These do not claim live apply happened.

When `conclusion.status` is `blocked`, operators should clear the listed
`remainingBlockers` and rerun:

```sh
node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report
node scripts/validate-dev-gate-report.mjs
```

When `conclusion.status` is `ready`, run the dry-run plan and validator
immediately before human approval:

```sh
node scripts/dev-deploy-apply.mjs --dry-run --write-report
node scripts/validate-dev-gate-report.mjs
```

Only after that approval may the DEV-only apply command be used.

## Suspended Template Job Replacement

Kubernetes Jobs cannot update `spec.template` in place. For DEV template Jobs
that are intentionally suspended and used as templates, an image tag change
would otherwise make a normal `kubectl apply -k deploy/k8s/dev` non-idempotent.

The apply script handles only the explicit DEV allowlist:

- `hwlab-agent-worker-template`
- `hwlab-cli-template`

For each allowlisted Job, the script reads the live Job in `hwlab-dev` and
compares its container image with the desired manifest image. If the live Job
exists, is suspended, and the image differs, apply mode deletes that Job before
running the normal kustomize apply so Kubernetes recreates it from the desired
manifest. Dry-run mode records the same decision as `planned` and runs
server-side kustomize dry-run without mutating the cluster. If that dry-run
reports only the expected immutable `spec.template` errors for the planned
allowlisted Jobs, the report records those errors as expected replacement
evidence instead of treating them as blockers.

The policy refuses any other Job name, any non-`hwlab-dev` namespace, and any
live Job that is not suspended. It does not change Deployment, Service,
ConfigMap, Secret, or PROD behavior.

## Current Result

The current DEV apply contract is published for artifact commit `cb35ada`.
`deploy/artifact-catalog.dev.json`, `deploy/deploy.json`, and the DEV
workloads point at `127.0.0.1:5000/hwlab/*:cb35ada` images with sha256 digest
evidence.

The D601 runner can read and apply the `hwlab-dev` namespace. A real DEV apply
may replace only the allowlisted suspended template Jobs above; the normal
Deployment, Service, ConfigMap, and health-contract apply path remains
unchanged.

`hwlab-cloud-api` has a DEV DB env contract placeholder:
`HWLAB_CLOUD_DB_URL` must come from Secret reference
`hwlab-cloud-api-dev-db/database-url`, and
`HWLAB_CLOUD_DB_SSL_MODE=require`. The local health gate reports DB runtime
config without printing the connection string.

No PROD resource, secret value, UniDesk runtime substitute, browser e2e,
heavyweight e2e, or force push is part of this flow.