feat: expose G14 prod through FRP

2026-05-26 03:56:53 +08:00
parent f557f1cd20
commit aabff4059b
6 changed files with 78 additions and 30 deletions
@@ -7,7 +7,7 @@ The DEV reverse link is D601 to master edge through frp.
 - `hwlab-tunnel-client` owns the frp client health contract.
 - `hwlab-cloud-web` owns the public browser entry on port `16666`.
 - `hwlab-edge-proxy` owns public API/health ingress on port `16667`.
- `frps.dev.toml` reserves D601 `16666` / `16667`, G14 `17666` / `17667`, and tunnel health `7402`
+- `frps.dev.toml` reserves D601 `16666` / `16667`, G14 DEV `17666` / `17667`, G14 PROD `18666` / `18667`, and tunnel health `7402`
  as TCP `remotePort` values; do not also bind them as `vhostHTTPPort`, because that collides with
  the DEV TCP proxy.
 - `deploy/deploy.json` is the source for the public endpoints and FRP proxy
@@ -20,17 +20,22 @@ The DEV reverse link is D601 to master edge through frp.

 ## G14 public preview

-G14 uses a separate preview port range and must not reuse D601's `16666` / `16667`:
+G14 uses separate preview port ranges and must not reuse D601's `16666` / `16667`:

- G14 browser endpoint: `http://74.48.78.17:17666`
- G14 API/edge endpoint: `http://74.48.78.17:17667`
- G14 frpc proxy names: `hwlab-g14-cloud-web` and `hwlab-g14-edge-proxy`
- G14 k3s Deployment: `hwlab-dev/hwlab-g14-frpc`, generated by `scripts/g14-gitops-render.mjs`
+- G14 DEV browser endpoint: `http://74.48.78.17:17666`
+- G14 DEV API/edge endpoint: `http://74.48.78.17:17667`
+- G14 DEV frpc proxy names: `hwlab-g14-cloud-web` and `hwlab-g14-edge-proxy`
+- G14 DEV k3s Deployment: `hwlab-dev/hwlab-g14-frpc`, generated by `scripts/g14-gitops-render.mjs`
+- G14 PROD browser endpoint: `http://74.48.78.17:18666`
+- G14 PROD API/edge endpoint: `http://74.48.78.17:18667`
+- G14 PROD frpc proxy names: `hwlab-g14-prod-cloud-web` and `hwlab-g14-prod-edge-proxy`
+- G14 PROD k3s Deployment: `hwlab-prod/hwlab-g14-prod-frpc`, generated by `scripts/g14-gitops-render.mjs`
 - Master frps allow-list template: `deploy/frp/frps.dev.toml`
- G14 frpc local template: `deploy/frp/frpc.g14.toml`
+- G14 DEV frpc local template: `deploy/frp/frpc.g14.toml`
+- G14 PROD frpc local template: `deploy/frp/frpc.g14-prod.toml`

 Only `hwlab-cloud-web:8080` and `hwlab-edge-proxy:6667` are exposed through FRP.
 Internal services, Tekton, Argo CD, registry, database, and simulator services stay ClusterIP-only.
-The master-side `/opt/hwlab-frp/frps.dev.toml` may need a manual allow-list update matching `deploy/frp/frps.dev.toml` before
-the G14 frpc Deployment can register `17666` and `17667`; this is an edge operation, not a D601
+The master-side `/opt/hwlab-frp/frps.dev.toml` and active `/etc/frp/frps.toml` may need a manual allow-list update matching `deploy/frp/frps.dev.toml` before
+the G14 frpc Deployments can register `17666` / `17667` and `18666` / `18667`; this is an edge operation, not a D601
 client change.
@@ -0,0 +1,17 @@
+serverAddr = "74.48.78.17"
+serverPort = 7000
+loginFailExit = true
+
+[[proxies]]
+name = "hwlab-g14-prod-cloud-web"
+type = "tcp"
+localIP = "hwlab-cloud-web.hwlab-prod.svc.cluster.local"
+localPort = 8080
+remotePort = 18666
+
+[[proxies]]
+name = "hwlab-g14-prod-edge-proxy"
+type = "tcp"
+localIP = "hwlab-edge-proxy.hwlab-prod.svc.cluster.local"
+localPort = 6667
+remotePort = 18667
@@ -16,6 +16,14 @@ end = 17666
 start = 17667
 end = 17667

+[[allowPorts]]
+start = 18666
+end = 18666
+
+[[allowPorts]]
+start = 18667
+end = 18667
+
 [[allowPorts]]
 start = 7402
 end = 7402
@@ -22,6 +22,11 @@ master hwlab-frps-dev :17666/:17667
 -> G14 hwlab-g14-frpc in namespace hwlab-dev
 -> hwlab-cloud-web on internal :8080 for browser traffic
 -> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health
+
+master hwlab-frps-dev :18666/:18667
+-> G14 hwlab-g14-prod-frpc in namespace hwlab-prod
+-> hwlab-cloud-web on internal :8080 for browser traffic
+-> hwlab-edge-proxy / hwlab-cloud-api on internal :6667 for API and health
 ```

 G14 PROD is rendered by GitOps as namespace `hwlab-prod` and Argo CD application `hwlab-g14-prod`; it uses the `:18666/:18667` public endpoints when enabled. G14 desired state is generated into `deploy/gitops/g14/runtime-dev` and `deploy/gitops/g14/runtime-prod` and promoted through the `G14-gitops` branch.
@@ -9,6 +9,7 @@ G14 是 HWLAB 当前 DEV/PROD 原生 k8s 与 GitOps 运行面目标。G14 CI/CD
 - Artifact：镜像使用 commit tag，例如 `127.0.0.1:5000/hwlab/hwlab-cloud-api:<shortCommit>`；digest 由 registry 返回，CI report 只作为审计证据，不作为 CD 真相。
 - Branch split：`G14` 是源码监控分支；`G14-gitops` 是 Tekton promotion 写入的生成分支，保存 `deploy/gitops/g14/**` desired state。
 - CD：Argo CD 只消费 `G14-gitops:deploy/gitops/g14/runtime-dev` 与 `deploy/gitops/g14/runtime-prod` 的 Git desired state，不重新构建镜像，不读取 D601 状态，不获取 legacy DEV CD Lease。
+- FRP：G14 DEV 通过 `hwlab-dev/hwlab-g14-frpc` 暴露 `17666/17667`，G14 PROD 通过 `hwlab-prod/hwlab-g14-prod-frpc` 暴露 `18666/18667`；master frps 的 `deploy/frp/frps.dev.toml` 与实际 `/etc/frp/frps.toml` 必须同时放行这四个 G14 端口。
 - 并行性：不同 source commit 的 CI build 不共享发布锁；并行安全由 immutable commit tag/digest 和 Git desired state 保证。最终运行版本由 Argo CD 当前同步的 Git revision 决定。

 ## 生成入口
@@ -79,7 +80,7 @@ KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n hwlab-ci logs pod/<pipelinerun-t
 - `hwlab-ci/hwlab-g14-control-plane-reconciler`：自动 render/apply G14 CI 控制面 manifest 的 CronJob。
 - `argocd`：Argo CD 控制面和 `hwlab-g14-dev` / `hwlab-g14-prod` Application 所在 namespace。
 - `hwlab-dev`：HWLAB DEV runtime namespace，由 Argo CD 应用 `deploy/gitops/g14/runtime-dev`。
- `hwlab-prod`：HWLAB PROD 预留 runtime namespace，由 Argo CD 应用 `deploy/gitops/g14/runtime-prod`；生产 Secret、入口和健康门禁未全部满足前，只能作为预留同步对象，不得用它替代 DEV 验收。
+- `hwlab-prod`：HWLAB PROD runtime namespace，由 Argo CD 应用 `deploy/gitops/g14/runtime-prod`；通过 `hwlab-g14-prod-frpc` 映射到 `18666/18667`，正式验收必须同时检查 Argo sync、Deployment ready、FRP public health 与 source commit。

 G14 registry 由 Kubernetes Deployment `hwlab-ci/hwlab-registry` 承载，使用 host network 暴露 `127.0.0.1:5000` 给 k3s/containerd 和 host-network Tekton build pod。旧 host Docker registry 不能与该 Deployment 同时占用 5000 端口。

@@ -1287,12 +1287,20 @@ function argoApplication(args, profile = "dev") {
  };
 }

-function g14FrpcManifest() {
+function g14FrpcManifest({ profile = "dev" } = {}) {
+  const namespace = namespaceNameForProfile(profile);
+  const profileLabel = runtimeLabelForProfile(profile);
+  const deploymentName = profile === "prod" ? "hwlab-g14-prod-frpc" : "hwlab-g14-frpc";
+  const configName = `${deploymentName}-config`;
+  const proxyPrefix = profile === "prod" ? "hwlab-g14-prod" : "hwlab-g14";
+  const webRemotePort = profile === "prod" ? 18666 : 17666;
+  const edgeRemotePort = profile === "prod" ? 18667 : 17667;
  const labels = {
-    "app.kubernetes.io/name": "hwlab-g14-frpc",
+    "app.kubernetes.io/name": deploymentName,
    "app.kubernetes.io/part-of": "hwlab",
-    "hwlab.pikastech.local/environment": "dev",
-    "hwlab.pikastech.local/gitops-target": "g14"
+    "hwlab.pikastech.local/environment": profileLabel,
+    "hwlab.pikastech.local/gitops-target": "g14",
+    "hwlab.pikastech.local/profile": profileLabel
  };
  return {
    apiVersion: "v1",
@@ -1302,8 +1310,8 @@ function g14FrpcManifest() {
        apiVersion: "v1",
        kind: "ConfigMap",
        metadata: {
-          name: "hwlab-g14-frpc-config",
-          namespace: "hwlab-dev",
+          name: configName,
+          namespace,
          labels
        },
        data: {
@@ -1312,18 +1320,18 @@ serverPort = 7000
 loginFailExit = true

 [[proxies]]
-name = "hwlab-g14-cloud-web"
+name = "${proxyPrefix}-cloud-web"
 type = "tcp"
-localIP = "hwlab-cloud-web.hwlab-dev.svc.cluster.local"
+localIP = "hwlab-cloud-web.${namespace}.svc.cluster.local"
 localPort = 8080
-remotePort = 17666
+remotePort = ${webRemotePort}

 [[proxies]]
-name = "hwlab-g14-edge-proxy"
+name = "${proxyPrefix}-edge-proxy"
 type = "tcp"
-localIP = "hwlab-edge-proxy.hwlab-dev.svc.cluster.local"
+localIP = "hwlab-edge-proxy.${namespace}.svc.cluster.local"
 localPort = 6667
-remotePort = 17667
+remotePort = ${edgeRemotePort}
 `
        }
      },
@@ -1331,13 +1339,13 @@ remotePort = 17667
        apiVersion: "apps/v1",
        kind: "Deployment",
        metadata: {
-          name: "hwlab-g14-frpc",
-          namespace: "hwlab-dev",
+          name: deploymentName,
+          namespace,
          labels
        },
        spec: {
          replicas: 1,
-          selector: { matchLabels: { "app.kubernetes.io/name": "hwlab-g14-frpc" } },
+          selector: { matchLabels: { "app.kubernetes.io/name": deploymentName } },
          template: {
            metadata: { labels },
            spec: {
@@ -1348,7 +1356,7 @@ remotePort = 17667
                args: ["-c", "/etc/frp/frpc.toml"],
                volumeMounts: [{ name: "config", mountPath: "/etc/frp", readOnly: true }]
              }],
-              volumes: [{ name: "config", configMap: { name: "hwlab-g14-frpc-config" } }]
+              volumes: [{ name: "config", configMap: { name: configName } }]
            }
          }
        }
@@ -1416,7 +1424,7 @@ function deepSeekProxyManifest({ profile = "dev" } = {}) {

 function runtimeKustomization({ profile = "dev" } = {}) {
  const resources = ["namespace.yaml", "code-agent-codex-config.yaml", "services.yaml", "health-contract.yaml", "workloads.yaml", "deepseek-proxy.yaml"];
-  if (profile === "dev") resources.push("g14-frpc.yaml");
+  resources.push("g14-frpc.yaml");
  return {
    apiVersion: "kustomize.config.k8s.io/v1beta1",
    kind: "Kustomization",
@@ -1434,8 +1442,8 @@ This directory is generated from \`CI.json\`, \`deploy/deploy.json\`, and \`depl
 - CI: Tekton Pipeline \`hwlab-ci/hwlab-g14-ci-image-publish\`.
 - Artifact contract: images are tagged by source git commit \`${source.short}\` and pushed to \`${args.registryPrefix}\`.
 - Branch split: source branch \`${args.sourceBranch}\` remains the watched code branch; generated desired state is promoted to \`${args.gitopsBranch}\`.
- CD: Argo CD Application \`argocd/hwlab-g14-dev\` consumes \`${args.gitopsBranch}:deploy/gitops/g14/runtime\`.
- Public preview: FRP exposes only G14 web \`${args.webEndpoint}\` and edge/API \`${args.runtimeEndpoint}\` through \`hwlab-g14-frpc\`; D601 keeps \`:16666/:16667\`.
+- CD: Argo CD Applications \`argocd/hwlab-g14-dev\` and \`argocd/hwlab-g14-prod\` consume \`${args.gitopsBranch}:deploy/gitops/g14/runtime-dev\` and \`runtime-prod\`.
+- Public preview: FRP exposes G14 DEV web \`${args.webEndpoint}\` / edge \`${args.runtimeEndpoint}\` through \`hwlab-dev/hwlab-g14-frpc\`, and G14 PROD web \`${args.prodWebEndpoint}\` / edge \`${args.prodRuntimeEndpoint}\` through \`hwlab-prod/hwlab-g14-prod-frpc\`; D601 keeps \`:16666/:16667\`.
 - Manual polling: run \`kubectl -n hwlab-ci create job --from=cronjob/hwlab-g14-branch-poller hwlab-g14-branch-poller-manual-$(date -u +%Y%m%d%H%M%S)\`; this reuses the same poller path and creates a commit-pinned Tekton PipelineRun only when needed.
 - Control-plane reconcile: CronJob \`hwlab-ci/hwlab-g14-control-plane-reconciler\` polls \`${args.sourceBranch}\`, runs \`scripts/g14-gitops-render.mjs\`, and server-side-applies the generated Tekton RBAC/Pipeline/Poller/Reconciler manifests so CI control-plane changes do not require a manual render/apply step.
 - D601 boundary: this path does not target D601, does not use UniDesk Code Queue, and does not change D601 production traffic.
@@ -1474,6 +1482,10 @@ async function plannedFiles(args) {
      prod: { web: args.prodWebEndpoint, runtime: args.prodRuntimeEndpoint }
    },
    frpDeployment: "hwlab-dev/hwlab-g14-frpc",
+    frpDeployments: {
+      dev: "hwlab-dev/hwlab-g14-frpc",
+      prod: "hwlab-prod/hwlab-g14-prod-frpc"
+    },
    tektonPipeline: "hwlab-ci/hwlab-g14-ci-image-publish",
    argoApplications: ["argocd/hwlab-g14-dev", "argocd/hwlab-g14-prod"]
  });
@@ -1504,7 +1516,7 @@ async function plannedFiles(args) {
    putJson(`${runtimePath}/health-contract.yaml`, transformHealthContract(health, namespace, profileLabels, annotations, endpoints.runtimeEndpoint, endpoints.webEndpoint, profile));
    putJson(`${runtimePath}/workloads.yaml`, transformWorkloads({ workloads, deploy, source, registryPrefix: args.registryPrefix, runtimeEndpoint: endpoints.runtimeEndpoint, webEndpoint: endpoints.webEndpoint, profile }));
    putJson(`${runtimePath}/deepseek-proxy.yaml`, deepSeekProxyManifest({ profile }));
-    if (profile === "dev") putJson(`${runtimePath}/g14-frpc.yaml`, g14FrpcManifest());
+    putJson(`${runtimePath}/g14-frpc.yaml`, g14FrpcManifest({ profile }));
  }
  return { files, source };
 }