ADR-032: PR Preview Environments

Date: 2026-05-28 (amended 2026-07-01, verified live 2026-07-01)

Status: Accepted — built + live, proven end-to-end (asanexample/alpha-shop#14); see Implementation Status

Implementation Status

Live and proven. A real PR against alpha-shop (with spec.preview: true on its dev claim) produced an isolated, auto-cleaned-up preview: image built + signed, ArgoCD generated the Application, it synced Healthy, was reachable over HTTPS at the designed hostname, was provably isolated from the stable deployment, and was pruned on PR close. platform/argocd/github-app’s installation now carries pull_requests: read alongside its existing repo-creds scope.

Before this, most of this pipeline was already built, piece by piece, without anyone tracking that the last piece — actual delivery — was still missing. This ADR originally described the v2 design: preview was opt-in per app via preview = true in teams.hcl (ADR-031), and the argocd-apps module built a per-app PR-generator ApplicationSet from tenants / github_org inputs. That entire surface (tenants, github_org, preview_appset, github_token_secret_name, teams.hcl) was removed at the v3 cutover (ADR-067/069), which replaced per-tenant delivery with registry-derived, Release-keyed per-Product ApplicationSets (gitops/releases/<team>/<product>/*.yaml).

Issue #721 caught the resulting drift (docs describing a shipped feature that didn’t exist) and was closed after fixing the docs — but the actual re-implementation was never done until now. Issue #111 (migrate ArgoCD’s GitHub auth off a PAT) is also fully resolved by this work — both consumers it described (repo-creds and a PR-generator token) now run on the same GitHub App.

Already live on the v3 model, independent of this ADR:

• preview.yml (generated by the New Product scaffolder for every product) builds, cosign-signs, and SLSA-attests an image on every pull_request, tagged by github.event.pull_request.head.sha.
• OIDC trust for PR-triggered builds — infra/modules/aws/github_oidc already supports event-shaped subjects (repo:<org>/<repo>:pull_request) via roles.<name>.events, already set on every per-Product ECR-push role. (This ADR originally asked for a github_events variable; that request was folded into the roles.*.events field during an unrelated per-team-role refactor, and already does what was asked.)
• name-reference.yaml (also scaffolder-generated) — the kustomize nameReference config that rewrites HTTPRoute backendRefs[].name under a pr-<N>- namePrefix (issue #155, resolved).
• The XEnvironment claim already has a preview: boolean field (default false) — the intended per-environment activation switch.
• Kyverno’s verify-images-product / verify-attestations-product already carry a preview_subject_regexp slot for admitting PR-preview-built image signatures.
• The Crossplane Composition’s restrict-route-hostnames-<ns> policy already unconditionally allow-lists a wildcard <product>-<team>-<stage>-pr-*.<baseDomain> hostname pattern — PR-preview routes are already pre-approved by admission, no per-PR reconciliation needed.
• ArgoCD already authenticates to GitHub via a GitHub App (platform/argocd/github-app), not a PAT (TD2-02b) — used today for repo-clone credentials.

What closed the gap: the ArgoCD delivery mechanism itself — a pullRequest-generator ApplicationSet in argocd-apps (infra/modules/argocd-apps/pr-preview.tf) that turns an open PR into an ephemeral, auto-cleaned-up Application, gated per-product on spec.preview: true on that product’s dev Environment claim, reusing the already-existing GitHub App credential (its installation now carries Pull requests: Read-only alongside its existing repo-creds scope) rather than a separate token.

One real bug found and fixed during live verification: kustomize’s commonLabels transformer only auto-patches selector + pod-template paths for well-known built-in types (Deployment, StatefulSet, etc.) — Rollout (an Argo Rollouts CRD, what every workload here actually is, per ADR-056) only got its top-level metadata.labels patched. Since preview isolation depends on commonLabels propagating app.kubernetes.io/instance: pr-<N> into both the Service selector and the Rollout’s pod template, this left the Service with zero endpoints (“no healthy upstream” at the Gateway) for any product’s first preview. Fixed in each app’s k8s/base/kustomizeconfig.yaml (and the New Product scaffolder template, so new products don’t hit it) with a commonLabels FieldSpec extension for Rollout, mirroring the pattern that file already uses for the replicas: transformer on the same CRD. See the “Kustomization Requirement” section below.

Context

Development teams need a way to preview changes from pull requests before merging. A common pattern is to deploy each open PR as an ephemeral environment with its own URL, allowing reviewers to test the change in a running cluster without affecting the stable deployment.

The platform uses ArgoCD for GitOps (ADR-021) with the ApplicationSet controller, which supports a Pull Request generator that polls GitHub for open PRs and creates ephemeral Applications from a template.

The key challenges:

1. Label selector collision. A preview Deployment and the stable Deployment must not share selector labels, or their Services/ReplicaSets cross-select each other’s pods.
2. HTTPRoute hostname collision. Each preview needs a unique hostname; Gateway API HTTPRoute backendRefs are not automatically updated by kustomize namePrefix.
3. OIDC trust for PR workflows. PR-triggered GitHub Actions use a different subject claim shape than branch-push workflows. — Already solved, see above.
4. Image tagging. ECR tag immutability (ADR-028) requires a unique, non-:latest tag per image; PR previews use the PR’s head commit SHA. — Already solved, see above.

Alternatives Considered

1. Manual preview deployments. Error-prone, namespace sprawl, easily forgotten. Rejected.

2. Argo Rollouts traffic splitting. Designed for progressive delivery of one revision, not concurrent multi-PR previews. Rejected.

3. External preview orchestrator (PullApprove, Uffizzi, etc.). Adds an operational dependency that doesn’t integrate with the existing ArgoCD + Gateway API stack for no clear benefit. Rejected.

4. ArgoCD ApplicationSet with PR generator (chosen). Native ArgoCD capability, no new infrastructure, and (on v3) almost every other piece of the pipeline needed for it already exists.

Decision

Enable PR preview environments using an ArgoCD ApplicationSet with the Pull Request generator. Preview is opt-in per Environment, via spec.preview: true on a product’s dev XEnvironment claim (gitops/environments/<team>/<product>/dev.yaml) — not a new claim kind, and not a re-provisioned environment per PR. A preview deploys into the existing dev namespace (<team>-<product>-dev), isolated by kustomize namePrefix/commonLabels, because provisioning a full Crossplane Environment (ResourceQuota, per-product Kyverno policies, Pod Identity, IAM) per PR would be far too slow and heavy for something meant to appear on PR opened and vanish on PR closed.

Architecture

Developer opens PR
        |
        v
GitHub Actions (preview.yml — already live)
  - OIDC auth (pull_request event, already trusted)
  - Build + cosign-sign + SLSA-attest, push to ECR as
    team-<team>/<product>-<svc>:<head-sha>
        |
        v
ArgoCD ApplicationSet (pullRequest generator — THIS PR)
  - Auths via the existing platform/argocd/github-app GitHub App
  - Polls for open PRs on the product's repo
  - Filtered to Products whose dev Environment has spec.preview: true
  - Creates an ephemeral Application from a template
        |
        v
Kustomize inline overrides (rendered by the ApplicationSet template)
  - namePrefix: pr-<N>-
  - commonLabels: app.kubernetes.io/instance = pr-<N>
  - images: head-SHA-tagged image (no Release record, no digest promotion)
  - patches: HTTPRoute hostname -> <product>-<team>-dev-pr-<N>.<baseDomain>
             (already pre-approved by Kyverno's restrict-route-hostnames)
        |
        v
Kubernetes resources in the EXISTING <team>-<product>-dev namespace
  - pr-<N>-<deployment>, pr-<N>-<service>, pr-<N>-<httproute>
  - Isolated by label selectors from the stable dev deployment

Label Selector Isolation

commonLabels on the ApplicationSet template (app.kubernetes.io/instance: pr-<N>) injects the label into Deployment spec.selector.matchLabels, spec.template.metadata.labels, and Service spec.selector — guaranteeing the stable dev Service only selects stable pods, and each preview Service only selects its own preview pods. Without this, namePrefix renames resources but does not touch label selectors, and Services cross-select pods from different deployments.

HTTPRoute Patching

Preview kustomize transforms are injected by the platform’s ApplicationSet template (infra/modules/argocd-apps/pr-preview.tf), not by files in the app repo. The template sets namePrefix = "pr-{{.number}}-", commonLabels, an images override to the PR head SHA, and a hostname patch to the already-Kyverno-allowed pattern:

patches:
  - target: { kind: HTTPRoute }
    patch: |
      - op: replace
        path: /spec/hostnames/0
        value: <product>-<team>-dev-pr-<N>.<baseDomain>

HTTPRoute backendRef rewriting (issue #155, already resolved). Kustomize’s built-in nameReference transformer doesn’t know about Gateway API HTTPRoute, so namePrefix alone would rename the Service but leave HTTPRoute.spec.rules[].backendRefs[].name pointing at the un-prefixed (stable) Service — silently routing preview traffic to stable pods. The fix, already generated for every product by the New Product scaffolder (scaffolder/templates/new-product/skeleton/k8s/base/name-reference.yaml), teaches kustomize that an HTTPRoute backendRef references a Service, so the same rename rewrites the backendRef too. This lives in the app repo (not the ApplicationSet template) because ArgoCD’s source.kustomize exposes no field to inject transformer configurations.

OIDC Trust Policy — already done

infra/modules/aws/github_oidc supports event-shaped subject claims via roles.<name>.events; events = ["pull_request"] is already set on every per-Product ECR-push role in the live platform unit. No change needed here.

Preview Lifecycle

Event	Action
PR opened	ApplicationSet creates Application, ArgoCD syncs resources
PR updated (push)	preview.yml rebuilds image with new SHA, ArgoCD syncs the updated tag
PR closed/merged	ApplicationSet deletes Application, ArgoCD prunes all resources

Cleanup is automatic — prune = true in the sync policy.

GitHub Actions Workflows — already scaffolded

Every product gets both deploy.yml (push to main → stable delivery via the Release/promote flow) and preview.yml (pull_request → build+sign+push a head-SHA-tagged image, no manifest/Release write) from the New Product scaffolder today.

GitHub Credential — reuse the existing App, don’t mint a new one

ArgoCD already authenticates to GitHub as a GitHub App (platform/argocd/github-app; App ID + installation ID + private key in Secrets Manager, projected via External Secrets into github-asanexample-app-creds), used today for repo-clone credentials (credential_templates/repo-creds). The pullRequest generator needs the same kind of credential — GitHub API read access to list open PRs — so it reuses this same App rather than a separate token, via ArgoCD’s App-based SCM/generator auth (appSecretName, not tokenRef). This requires the App’s installation to carry Pull requests: Read-only in addition to its existing Contents/Metadata read scope — a manual, one-time permission addition on the existing App (not a new App), verified/applied outside this PR.

Kustomization Requirement — already scaffolded

App repos need a kustomization.yaml (already scaffolded) listing all resources, and two configurations: entries (both already scaffolded):

• name-reference.yaml — rewrites HTTPRoute backendRefs[].name under the pr-<N>- namePrefix (issue #155).
• kustomizeconfig.yaml’s commonLabels FieldSpecs for Rollout — extends the namePrefix ApplicationSet template’s commonLabels (app.kubernetes.io/instance: pr-<N>) into spec.selector.matchLabels and spec.template.metadata.labels, which kustomize doesn’t do automatically for a CRD. Without it, the preview Service has zero endpoints — found live verifying the first real preview (asanexample/alpha-shop#14), fixed in asanexample/alpha-shop#15 and the scaffolder template.

Consequences

Positive

• PR previews are fully automated — developers only need to open a PR.
• Cleanup is automatic — no orphaned preview resources (native prune).
• Previews share the dev Environment’s existing namespace and resource quota — no namespace sprawl, no per-PR Crossplane provisioning latency.
• The stable dev deployment is protected via label selector isolation.
• No new infrastructure, no new GitHub App, no new credential shape — every other piece of this pipeline (image build/sign, OIDC trust, hostname allow-listing, backendRef rewriting) was already built independently before this ADR was revisited.

Negative

• commonLabels on the stable dev Application also gets an app.kubernetes.io/instance: stable label — semantically correct, but a visible change from unlabeled raw manifests.
• The PR generator polls the GitHub API on an interval; API-side latency, not a blocker.

Risks

• Preview routing correctness depends on the per-app name-reference.yaml. Already scaffolded and CI-guarded (issue #155) for new products; a hand-rolled app repo that skips scaffolding and forgets it would silently route previews to the stable Service.
• Fork PRs cannot push images. GitHub blocks id-token: write on fork PRs by default, so preview.yml can’t authenticate to ECR from a fork. The ApplicationSet will still create an Application, which fails to sync (no image tag exists) — safe (no resources created) but may confuse external contributors.
• PR image accumulation. Each PR push creates a new ECR tag; the ECR lifecycle policy (50 images, ADR-028) could evict older tags under high PR velocity.
• Resource exhaustion from concurrent PRs. Each preview adds pods against the dev Environment’s existing ResourceQuota — high PR volume on one product may need a quota bump.
• Per-team ECR push scope is unaffected. Push access is still one OIDC role per team/product (github-actions-ecr-push-<team>-<product>), scoped to that product’s team-<team>/<product>-* repositories — a compromised PR workflow for one product still can’t push to another’s images.