mirrors/pikasTech-HWLAB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e3c2537091e3878e2bbef62953b7ea905d1042e4
pikasTech-HWLAB/docs/dev-deploy-apply.md
T
Code Queue Review e3c2537091 test: guard artifact runtime readiness
2026-05-23 02:31:56 +00:00

14 KiB
Raw Blame History

HWLAB DEV Deploy Apply

This document records the DEV-only apply/preflight path for pikasTech/HWLAB#33 and the apply-plan/report path for pikasTech/HWLAB#50. It targets only the D601 hwlab-dev namespace and must not be used for PROD.

Scope

  • Work from latest origin/main.
  • Read only HWLAB repository deploy inputs: deploy/deploy.json, deploy/artifact-catalog.dev.json, and deploy/k8s/dev.
  • Use scripts/deploy-desired-state-plan.mjs as source/dry-run support to review commit, image, tag, workload image, and env mirror convergence before any apply review.
  • Validate artifact identity, image commit tags, Kubernetes workload/service shape, DEV health, and cloud-api DB connectivity.
  • Validate that hwlab-cloud-api desired state declares the Code Agent provider env contract: OPENAI_API_KEY from hwlab-code-agent-provider/openai-api-key and HWLAB_CODE_AGENT_OPENAI_BASE_URL through the DEV egress/proxy path.
  • Emit an operator-readable plan in devDeployApply: plan/apply boundary, expected artifact commit, target namespace, workloads, services, remaining blockers, minimal manual commands, and rollback hints.
  • Plan and, in apply mode, execute a scoped replacement for DEV suspended template Jobs whose image changed but whose Kubernetes spec.template is immutable.
  • Stop before mutation when artifact publish evidence, registry digests, kubectl, DEV health, cloud-api DB connectivity, or Code Agent provider env preservation are missing.

Commands

Run a preflight that is expected to report blockers on the current skeleton:

node --check scripts/dev-deploy-apply.mjs
node --check scripts/src/dev-deploy-apply.mjs
node --check scripts/dev-runtime-migration.mjs
node --check scripts/src/dev-runtime-migration.mjs
node scripts/deploy-desired-state-plan.mjs --check
node scripts/dev-runtime-migration.mjs --check
node scripts/artifact-runtime-readiness-guard.mjs --target-ref origin/main --check --no-report --expect-blocked
node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report
node scripts/validate-dev-gate-report.mjs

Run the existing CLI dry-run surface:

node tools/hwlab-cli/bin/hwlab-cli.mjs test e2e --env dev --mvp --dry-run

Only after CI publish evidence and executor prerequisites exist, a real DEV apply requires explicit non-production confirmation:

node scripts/dev-deploy-apply.mjs --apply --confirm-dev --confirmed-non-production --write-report

The script refuses PROD flags, secret-read flags, heavyweight e2e flags, and force-push flags. It also refuses to apply if the DEV artifact catalog still contains skeleton-only unpublished images.

deploy-desired-state-plan.mjs is not an apply preflight substitute. Pair it with artifact-runtime-readiness-guard.mjs before apply review so stale artifact catalogs, stale desired-state, stale API runtime identity, or a stale Cloud Web :16666 build stay explicit blockers.

Kubeconfig selection is explicit and safe. The script uses this precedence:

  1. --kubeconfig PATH
  2. HWLAB_DEV_KUBECONFIG
  3. KUBECONFIG
  4. /etc/rancher/k3s/k3s.yaml

The selected path is used for read-only preflight, dry-run planning, and the DEV-only apply path. The report records the kubeconfig source and path, but it does not print kubeconfig contents or Secret values.

Runtime DB migration is a separate DEV-only apply boundary. The source-only check is safe for normal PR validation:

node scripts/dev-runtime-migration.mjs --check
node scripts/dev-runtime-migration.mjs --dry-run --write-report

Read-only live DB verification requires explicit DEV confirmation and an already injected HWLAB_CLOUD_DB_URL; the report redacts the value and keeps auth, schema, migration ledger, and readiness blockers separate:

node scripts/dev-runtime-migration.mjs --dry-run --allow-live-db-read --confirm-dev --write-report

A real migration write is not part of default deploy validation. It requires a separately authorized DEV operation:

node scripts/dev-runtime-migration.mjs --apply --confirm-dev --confirmed-non-production --write-report

That command may only apply internal/db/migrations/0001_cloud_core_skeleton.sql to the DEV Postgres target provided by HWLAB_CLOUD_DB_URL. It must not read Kubernetes Secret resources, print the DB URL/password/token, mutate PROD, restart services, or claim M3/M4/M5 acceptance.

deploy-desired-state-plan.mjs is not an apply preflight substitute. It is a read-only source planner and does not prove registry image existence, DEV cluster pullability, DEV apply, service restart, or M3 DEV-LIVE evidence.

Report Contract

The machine-readable result is written to reports/dev-gate/dev-deploy-report.json when --write-report is used. The report remains compatible with the #33 DEV gate validator and adds these operator fields:

  • devDeployApply.conclusion: blocked while blockers are open, otherwise ready for human approval.
  • devDeployApply.applyBoundary: default no-write behavior, required apply flags, DEV-only write scope, no-write scope, and forbidden actions.
  • devDeployApply.artifactPlan: expected artifact commit, source commit, deploy/catalog commit IDs, publish/registry status, and unpublished services.
  • devDeployApply.target: DEV endpoint and hwlab-dev namespace.
  • devDeployApply.workloadPlan and servicePlan: planned Kubernetes workloads, containers, images, service IDs, ports, and replica intent.
  • devDeployApply.templateJobReplacementPolicy: the DEV-only allowlist for suspended template Job replacement. The only allowlisted Jobs are hwlab-agent-worker-template and hwlab-cli-template in hwlab-dev.
  • devDeployApply.templateJobReplacements: per-Job replacement decisions with namespace, Job name, old image, new image, and result (planned, replaced, not_needed, not_found, or blocked/failure states).
  • devDeployApply.cloudWebRollout: Cloud Web source commit, image tag, image reference, digest, live image, and Kubernetes rollout revision when read-only Deployment access is available. If access is missing, this field records the read/rollout commands and the blocker instead of claiming a live rollout.
  • devDeployApply.codeAgentProvider: no-secret provider validation gate. It records desired-state env/ref readiness plus live hwlab-cloud-api Deployment env preservation before apply and, in apply mode, after apply. This gate reads Deployment env metadata only; it must not read Kubernetes Secret data or attempt a provider call.
  • devDeployApply.manualCommands: dry-run/report commands before human approval; the apply command is present only when the plan is ready.
  • devDeployApply.rollbackHint: pre-apply capture commands, deployment rollbacks, suspended job cleanup, and post-rollback health checks.
  • devDeployApply.remainingBlockers: open blockers with concrete unblock hints. These do not claim live apply happened.

When conclusion.status is blocked, operators should clear the listed remainingBlockers and rerun:

node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report
node scripts/validate-dev-gate-report.mjs

When conclusion.status is ready, run the dry-run plan and validator immediately before human approval:

node scripts/dev-deploy-apply.mjs --dry-run --write-report
node scripts/validate-dev-gate-report.mjs

Only after that approval may the DEV-only apply command be used.

Cloud Web Manual DEV Rollout

This is the currently verified manual CD path for hwlab-cloud-web on D601. It is the operational baseline that future CLI automation should reproduce from deploy/deploy.json; do not rediscover ports or kube contexts during an incident.

Start from the D601 deployment copy, not a runner worktree:

cd /home/ubuntu/workspace/hwlab
git pull --ff-only origin main

Validate source in a clean checkout, then refresh the static bundle before artifact publish. Cloud Web images serve web/hwlab-cloud-web/dist first, so stale ignored build output can keep an old English or old layout bundle alive even after the source commit is correct. check.mjs does not require ignored dist/; build.mjs and dev-artifact-publish.mjs are the build-first freshness gates.

node web/hwlab-cloud-web/scripts/check.mjs
node web/hwlab-cloud-web/scripts/build.mjs
node scripts/dev-artifact-publish.mjs --publish --services hwlab-cloud-web --report reports/dev-gate/dev-artifacts-hwlab-cloud-web-<tag>.json

Probe the image locally before touching k3s:

docker run --rm -d --name hwlab-cloud-web-probe -p 127.0.0.1:18088:8080 127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
sleep 1
curl -fsS http://127.0.0.1:18088/health/live
curl -fsS http://127.0.0.1:18088/ | sed -n '1,40p'
docker rm -f hwlab-cloud-web-probe

Use the D601 native k3s kubeconfig explicitly:

KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev set image deployment/hwlab-cloud-web hwlab-cloud-web=127.0.0.1:5000/hwlab/hwlab-cloud-web:<tag>
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev rollout status deployment/hwlab-cloud-web --timeout=180s
KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-dev get deploy hwlab-cloud-web -o wide

Do not use the default kube context as the source of truth. On D601 it may be docker-desktop, which is not the public DEV runtime behind 16666/16667.

Public acceptance must verify all of these:

curl -fsS http://74.48.78.17:16666/health/live
curl -fsS http://74.48.78.17:16666/ | sed -n '1,80p'
curl -fsS http://74.48.78.17:16666/styles.css | rg 'overflow: hidden|100dvh'
curl -fsS http://74.48.78.17:16666/help.md | sed -n '1,40p'

For the Cloud Workbench Chinese/no-outer-scroll rollout, the accepted target state was:

  • GET /health/live reports revision 1e8805664970839b72be40c34636b08f6d18b131;
  • GET / returns html lang="zh-CN" and title HWLAB 云工作台;
  • CSS locks the outer shell with html, body { overflow: hidden; } and .workbench-shell { height: 100dvh; overflow: hidden; };
  • GET /help.md returns the Chinese Cloud Workbench help document.

The public path is:

  1. master hwlab-frps-dev listens on 16666/16667;
  2. D601 hwlab-frpc runs in hwlab-dev from the hwlab-frpc-config ConfigMap;
  3. hwlab-dev-cloud-web maps to hwlab-cloud-web.hwlab-dev.svc.cluster.local:8080;
  4. hwlab-dev-edge-proxy maps to hwlab-edge-proxy.hwlab-dev.svc.cluster.local:6667.

Master hwlab-browser-proxy-dev is only an auxiliary local proxy on 18087; it is not the public browser endpoint.

Suspended Template Job Replacement

Kubernetes Jobs cannot update spec.template in place. For DEV template Jobs that are intentionally suspended and used as templates, an image tag change would otherwise make a normal kubectl apply -k deploy/k8s/dev non-idempotent.

The apply script handles only the explicit DEV allowlist:

  • hwlab-agent-worker-template
  • hwlab-cli-template

For each allowlisted Job, the script reads the live Job in hwlab-dev and compares its container image with the desired manifest image. If the live Job exists, is suspended, and the image differs, apply mode deletes that Job before running the normal kustomize apply so Kubernetes recreates it from the desired manifest. Dry-run mode records the same decision as planned and runs server-side kustomize dry-run without mutating the cluster. If that dry-run reports only the expected immutable spec.template errors for the planned allowlisted Jobs, the report records those errors as expected replacement evidence instead of treating them as blockers.

The policy refuses any other Job name, any non-hwlab-dev namespace, and any live Job that is not suspended. It does not change Deployment, Service, ConfigMap, Secret, or PROD behavior.

Current Result

The current DEV apply contract is published for artifact commit cb35ada. deploy/artifact-catalog.dev.json, deploy/deploy.json, and the DEV workloads point at 127.0.0.1:5000/hwlab/*:cb35ada images with sha256 digest evidence.

On Code Queue runners, the approved mounted kubeconfig path is typically /var/lib/unidesk/code-queue/kubeconfig. Export HWLAB_DEV_KUBECONFIG or pass --kubeconfig explicitly when the default D601 path is not available:

HWLAB_DEV_KUBECONFIG=/var/lib/unidesk/code-queue/kubeconfig node scripts/dev-deploy-apply.mjs --dry-run --expect-blocked --write-report

The D601 runner can read and apply the hwlab-dev namespace. A real DEV apply may replace only the allowlisted suspended template Jobs above; the normal Deployment, Service, ConfigMap, and health-contract apply path remains unchanged.

hwlab-cloud-api has a DEV DB env contract placeholder: HWLAB_CLOUD_DB_URL must come from Secret reference hwlab-cloud-api-dev-db/database-url, and HWLAB_CLOUD_DB_SSL_MODE=require. Runtime readiness dials the redacted host parsed from that Secret URL. cloud-api-db remains an optional desired alias, not a readiness gate or TCP probe target, until this repo owns its Service/Endpoint rollout contract. The local health gate reports DB runtime config without printing the connection string.

Durable runtime migration now has a repo-owned path: scripts/dev-runtime-migration.mjs --check proves the source migration and ledger contract without DB access; --dry-run --allow-live-db-read --confirm-dev verifies live auth/schema/migration/readiness without writes; --apply --confirm-dev --confirmed-non-production is the only DEV DB write mode and remains outside default PR validation.

No PROD resource, secret value, UniDesk runtime substitute, browser e2e, heavyweight e2e, or force push is part of this flow.

Source plans, dry-run reports, and desired-state convergence checks are M3 support only. They must not be labelled DEV-LIVE without a real DEV observation of the DO1 -> patch-panel -> DI1 loop plus operation, trace, audit, and evidence records.

Reference in New Issue View Git Blame Copy Permalink