Learn: Developer Experience — orientation

A developer states intent and gets governed infrastructure back — without ever learning Crossplane, ArgoCD, Kyverno, or IAM. Two halves make that possible: a single pane of glass to see everything (Backstage), and golden paths to create things (the scaffolder). Every change is a pull request the platform reconciles.

This is written for developers shipping on the platform and for the platform engineers who build the paved road. The domain model — Team / Product / Service / Environment — is the vocabulary the portal renders, so it’s worth reading first. The Environment API and Delivery modules are helpful but optional; this one is the front door to both.

The question

The platform underneath is deliberately deep. Crossplane provisions environments, ArgoCD delivers apps, Kyverno enforces admission, Keycloak does identity, the LGTM+P stack does observability. Each is the right tool, and each has its own CLI, CRD, and mental model. A developer who has to learn all of them to ship one service has, in effect, no platform. So the question is this:

How do you give a developer a Vercel-like experience — push code, get a URL — over a governed, compliance-aware, multi-tenant substrate, without exposing them to (or trusting them with) the machinery?

The paved road: intent in, a PR out, reconciled infra

The developer states intent — a form in a portal, or a git push — and the platform turns it into a pull request to the git registries. The normal GitOps path (Gate → ArgoCD → Crossplane) reconciles that PR into real infrastructure. The developer names what: team, product, stage, a bucket. The platform derives everything security-sensitive: IAM, ECR, namespace, Kyverno scope, routing.

Two surfaces make it work — Backstage, a single pane of glass to see everything, and the scaffolder, golden-path templates to create things. And one rule ties them together: the portal never writes to anything directly. Every change is a PR.

This is the BACK stack: Backstage (the form) → ArgoCD (delivery) → Crossplane (the control plane) → Kubernetes. The order is the whole philosophy. The control plane was built first, and Backstage is a thin portal over real, reconciled APIs — never a button that fires an imperative pipeline. The portal makes the platform visible and orderable; it doesn’t become the platform.

Think of it as a storefront over a warehouse. The warehouse — the git registries plus the cluster — holds the real inventory; Backstage is the storefront that makes it browsable and orderable. Every order is a mail-order form with guardrails (a scaffolder template): you tick boxes on a catalog card — team, product, stage, language — drop it in the outbox (the PR), and a clerk (the Gate) checks it’s a legal order before the warehouse (Crossplane) ships it. Unlike a real storefront, this one never touches the warehouse directly. It can’t hand you goods off the shelf, only place an order that git and the Gate fulfil. That constraint is the governance.

The rest of this page walks the two surfaces, then the guardrails that make self-service safe.

---

Stop 1 — the single pane of glass: Backstage

A developer opens backstage.aws.refplat.org (internal, via Tailscale) and sees their own Products and Environments, who owns what, and each one’s health and cost — without touching terragrunt or a CRD. That’s the Software Catalog, and one design fact keeps it honest: the catalog is a projection of the git registries, not a scrape of the cluster. A backend provider reads this repo’s git-native registries through a read-only GitHub App and projects them:

• gitops/teams/<team>.yaml (a Team) → a catalog Group
• gitops/products/<team>/<product>.yaml (a Product) → a System
• gitops/environments/<team>/<product>/<stage>.yaml (an XEnvironment claim) → an Environment

So the browse experience is a direct rendering of the domain model (Team → Product → Service → Environment). Because git is the source of truth, the catalog stays correct with no privileged cluster access and can’t drift from what’s declared. On top of the catalog sit read-only, scoped plugins that show the platform’s own view: the Kubernetes plugin (live workload health, read via a Pod-Identity viewer role that can’t see Secrets), the ArgoCD plugin (deploy health), and the Cost tab (per-team spend against the team’s declared budget). Never cluster-admin — the portal shows, it doesn’t wield.

One thing trips everyone up. The Backstage app — the image, its custom plugins, the Cost tab itself — is a separate repo; this infra repo only controls the deployment. So “I grepped the backstage module and the Cost tab isn’t there” is the classic wrong-place error: the plugin ships in the app image, and the infra side is often a single line elsewhere. Two repos, one portal.

The Backstage deep dive covers the projection, the direct-Keycloak-OIDC auth, the plugins, and the app-vs-infra split in full.

---

Stop 2 — the golden paths: the scaffolder

Seeing is half of it; the other half is creating. Instead of hand-writing a Product file plus an XEnvironment claim plus wiring IAM, a developer fills in a form and the scaffolder opens a PR to the git registries. The template is the paved road, encoded. There are ten of them, covering the whole lifecycle. The ones that matter most:

• new-product — creates the app repo from a golden starter and opens a platform PR adding the Product registry entry plus a first dev XEnvironment.
• new-environment — a new Environment (Product × Stage): PRs an XEnvironment claim.
• new-resource — self-service S3 / SQS / SNS / DynamoDB on a Service: patches the claim; the platform derives least-privilege IAM.
• request-promotion — the dev → test → staging → prod ladder: resolves the digest already running at the source stage and PRs a Release for the target. The same signed artifact moves up, no rebuild.
• onboard/offboard-person, new-team, deprovision-*, hello-world — round out identity, envelopes, and reversible wind-down.

The mechanism is always the same shape: a form (a JSON-schema the portal renders), steps (render a YAML skeleton, then publish:github:pull-request), and a PR link. Trace new-environment: you pick team, product (a picker that only shows Systems your team owns), and stage; the template renders a kind: XEnvironment claim to gitops/environments/<team>/<product>/<stage>.yaml and opens a PR. On merge, ArgoCD’s registry-sync projects it and Crossplane provisions the namespace, quota, network policy, and Pod-Identity — and the Environment API takes it from there. You named a stage; you got a governed environment.

The scaffolder deep dive traces new-product, new-environment, and request-promotion end to end, and covers the custom actions like platform:verify-team-membership.

---

Stop 3 — self-service with guardrails: why every change is a PR

Vercel-like DX with none of Vercel’s blast radius. The trick is that self-service here is self-service with guardrails, and there are three of them:

1. The schema constrains the ask. The form is a JSON-schema: you can only pick a stage inside your team’s envelope, an engine in your allowedEngines, a product your team owns. You can’t even express an illegal request.

2. The PR keeps a human (or a gate) in the loop, calibrated to blast radius. Low-risk, already-proven changes auto-merge — a ≤-staging promotion of an already-signed digest. Privilege grants and drains are reviewer-merged, never auto — a new team grants an envelope; a decommission drains workloads. Irreversible, high-stakes actions need named approval — a prod promotion goes to a release-approver with author ≠ approver; a product purge goes to an admin. The Gate is the toll booth: most traffic waved through, the on-ramps to prod and to deletion staffed.

   

3. The platform derives the dangerous parts, so you never touch them. You name intent — team, product, stage, an access level like read/readwrite. The platform derives the ECR repo, the namespace, the Kyverno image-scope, and, critically, the IAM: least-privilege, deny-set-validated. This is registries-as-single-source: argocd-apps, policy, and github-oidc all read the same Product entry, so there’s one truth and no drift between what you asked for and what got wired.

Behind those, the same request is checked by four independent layers: the portal’s permission policy, a server-side verify-team-membership step that binds the run to your own team, the gitops Gate (schema plus envelope, shift-left), and Kyverno re-enforcing the envelope at admission. No single bypass compromises it. That’s why a developer can be handed real power safely — the guardrails aren’t in the developer’s hands.

---

The honest status — live vs designed

• Live: Backstage Phase 2 at backstage.aws.refplat.org (verified running) — direct Keycloak OIDC, catalog projection (v3), and the Kubernetes / ArgoCD / Cost plugins. The scaffolder is enabled — team members self-serve the non-privileged paths (new-environment / new-resource / request-promotion / new-product); privileged templates (new-team, offboarding) are gated server-side. Those paths have been proven end to end — the alpha-shop product went the whole way.
• Known live gaps (from the templates’ own headers, not speculation): new-product’s repo-creation step 403s until the scaffolder GitHub App is broadened org-wide (the registry-PR half works); the gitops Gate’s auto-merge isn’t armed yet, so a reviewer merges even the low-risk cases today.
• Designed, not wired: TechDocs — serving this learning corpus inside Backstage — exists in the image but isn’t wired. It’s why these docs use absolute source links: they’ll survive the move into TechDocs. Also an RDS-backed prod database mode; dev runs in-cluster Postgres today.

The pane of glass and the golden paths are built and exercised. What’s maturing is the friction — arming auto-merge, broadening the GitHub App, moving the docs in — not the model.

When it breaks — the ones you’ll actually hit

• Backstage sign-in 503s. The single in-cluster auth Postgres blipped and the OIDC client cached a failed discovery — kubectl rollout restart deploy/backstage.
• You added a config key in the module and the feature silently does nothing. A config key with no matching schema in the app image is silently ignored. You substitute against the image’s existing shape; you can’t invent a new config section from the infra side.
• The Cost tab, or some plugin, isn’t in the backstage module. Right — it ships in the app repo’s image; the infra side is usually one line in a different unit. Check the app repo.
• new-product failed creating my repo. Known gap — the scaffolder GitHub App isn’t org-wide yet, so repo creation 403s; the registry PR still opens.
• My new environment or product isn’t in the catalog yet. The catalog projects from merged git — it appears after the PR merges and projection catches up, not when the PR opens.

Go deeper

• The Backstage portal — the catalog projection, direct Keycloak OIDC, the read-only plugins (Kubernetes / ArgoCD / Cost), and the app-vs-infra split.
• The scaffolder golden paths — the ten templates, three traced form → PR → registry → provision, the scaffolder engine, and self-service-with-guardrails.
• The lookup: the Reference. Related: domain model, Environment API, Delivery, Onboarding a Product.