Skip to content
refplat

Learn: Secrets & Config — reference

A lookup for what lives where. Verified against code and the live cluster — ESO is up: the operator, two ClusterSecretStores, and 16 ExternalSecrets, all synced. No secret values, account IDs, or emails appear here, only non-sensitive paths and mechanism.

The model

Section titled “The model”

The best secret is no secret. Three moves, in order: eliminate (federate — Pod Identity/OIDC/keyless cosign) → seal in git (SOPS, for config identifiers) → vault + sync (Secrets Manager + ESO, for runtime credentials). Two planes, and they never cross:

Config-in-git (SOPS) Runtime secrets (ESO)
What infra config identifiers credentials a workload uses
Examples account IDs, emails, state bucket/role, SSO endpoints OAuth secrets, DB passwords, Slack/PagerDuty/GitHub-App tokens
Home committed infra/live/aws/secrets.enc.yaml (KMS-sealed) AWS Secrets Manager → synced k8s Secret
Read when Terragrunt config-load (before providers) runtime, in-cluster
Credentials? no (identifiers only) yes

Config-in-git — SOPS (built+live)

Section titled “Config-in-git — SOPS (built+live)”
  • File: infra/live/aws/secrets.enc.yaml — SOPS-encrypted, committed to the public repo. Keys: account_ids, admin_email/account_emails, state_bucket, state_role_arn, cloudflare_zone_id, argocd_sso_*, optional keycloak_sso_*. No credentials.
  • Decrypt: root.hcl and common.hcl set _secrets = yamldecode(sops_decrypt_file(...)) inline, in memory — no CI fetch, no plaintext on disk. Exposed to units via _base.hcl → include.base.locals.account_ids["platform"] / account_id / admin_email. _base.hcl also cross-checks env.hcl’s account_id against the map and aborts on mismatch.
  • The KMS key is the ACL: platform-sops (mgmt account, module aws/sops-kms, alias alias/platform-sops, pinned by .sops.yaml). The key policy grants operators (SSO admin) Encrypt+Decrypt to edit, and the ARC runner Decrypt-only. Decrypt principals are named by account-root plus an aws:PrincipalArn ArnLike condition, which survives role recreation across a rebuild where a direct role-ARN principal wouldn’t. Every decrypt is a CloudTrail event. prevent_destroy guards it — destroying it bricks a rebuild.
  • Bootstrap escape: TG_SOPS_BOOTSTRAP=1 reads a local plaintext infra/live/aws/secrets.hcl (gitignored; structure in secrets.hcl.example) — only for the true from-zero moment before the KMS key exists.
  • Edit: sops infra/live/aws/secrets.enc.yaml decrypts to $EDITOR and re-encrypts on save. Needs an SSO-admin identity, not the decrypt-only runner.
  • Belongs here: ✅ stable config identifiers. ❌ app secrets/credentials (those go to ESO). ❌ customer PII — never in git, encrypted or not.

Runtime secrets — ESO + Secrets Manager (built+live)

Section titled “Runtime secrets — ESO + Secrets Manager (built+live)”
  • Source of truth: AWS Secrets Manager. The bridge is the External Secrets Operator (chart 0.14.3, module external-secrets, ns external-secrets). Alternatives that don’t fit: secrets in TF state (couples rotation to apply, and state readers see all), Sealed Secrets (per-cluster key, no cloud integration), Vault (a Tier-0 system to run).
  • ClusterSecretStore (module secret-stores): aws-secrets-manager (SecretsManager) and aws-secrets-manager-ssm (Parameter Store, cheaper). No auth block — ESO authenticates as its own Pod Identity, so the bridge holds no secret. Both are live and Valid/ReadWrite/Ready.
  • ExternalSecret (per workload): secretStoreRef {name, kind: ClusterSecretStore} · refreshInterval (1h everywhere today) · target.name + creationPolicy: Owner · data[] (secretKey ← remoteRef.key + property) or dataFrom. ESO reads the remote, materializes a k8s Secret, and the pod consumes it via envFrom. If ESO is down, existing Secrets keep working; only new syncs and rotations pause.
  • IAM (read-only): the ESO role gets secretsmanager:GetSecretValue/DescribeSecret/ListSecretVersionIds scoped to secret:platform/* (the live unit sets secret_path_prefix = "platform"). The module default is a wide *; the unit narrows it. No write or delete.
  • Worked example (live, SecretSynced): keycloak-config writes ArgoCD’s OIDC client secret to platform/keycloak/argocd-oidc; an ExternalSecret (store aws-secrets-manager, remoteRef that key, refresh 1h, target argocd-keycloak-oidc) becomes the k8s Secret ArgoCD mounts. Same shape for grafana-oidc and the two-key keycloak-admin blob.
  • Naming: platform/<subsystem>/<name> (e.g. platform/keycloak/admin, platform/backstage/github-app). 16 ExternalSecrets live across 8 namespaces, all SecretSynced.

The tenant paved road (designed, NOT built)

Section titled “The tenant paved road (designed, NOT built)”

The designed model: config (non-secret) lives in git on the claim’s services.<svc>.config and becomes a ConfigMap; secrets live in the store, the claim holds only key names, and the Composition mints a per-environment ExternalSecret. The write path makes Backstage the sole broker (platctl secret set calls the same API), Pod-Identity-scoped to …/tenants/<team>/<product>/<stage>, prod writes gated on release-approver, with a Kyverno backstop that denies a team’s ExternalSecret targeting another team’s path. What’s landed is only the XRD schema reservation (services.<svc>.config/.secrets, flagged “inert until the paved road ships”). No tenant ExternalSecret exists in gitops/ today — a tenant gets nothing for runtime secrets yet, though the platform’s own services use ESO extensively.

Keyless-first (the goal)

Section titled “Keyless-first (the goal)”
Concern Keyless (preferred) Stored secret (only if unavoidable)
Pod → AWS Pod Identity —
CI → AWS GitHub OIDC —
Image signing keyless cosign —
3rd-party tokens — Slack/PagerDuty/GitHub-App/DB → Secrets Manager

ESO itself proves the point: it reads the store via Pod Identity, so the secrets bridge has no secret of its own.

Rotation (mostly design)

Section titled “Rotation (mostly design)”

The hard half is solved — identity creds are keyless. The static residue (~26 secrets at Rotation: null) sorts into four classes:

Class Owns both sides? Mechanism Examples
A Terraform two-sided yes (TF/CNPG) time_rotating keeper on random_password + scheduled apply — rotates both sides in lockstep Keycloak OIDC (archetype), Grafana admin, Backstage session, CNPG passwords
B external provider no (3rd party) native SM rotation where supported, else scheduled-manual + expiry alert Tailscale, PagerDuty, GitHub App keys, Cloudflare
C keyless n/a nothing to rotate — the goal; keep growing it Pod Identity, OIDC, cosign, SSO
D tenant future design rotation in day one (don’t retrofit) (rebuild-gated)
  • Three shared primitives: Reloader (restart a workload on secret change — the keystone that turns a cached-old-value outage into a hands-off event) · rotation-age alerts (secret_age_days, alert past a per-class max) · refresh-interval tiers (24h/1h/15m — defined, never wired).
  • One-owner guardrail: exactly one owner per secret — TF or a rotation Lambda, never both. That’s the top risk.
  • Not Vault, deliberately: ESO, Secrets Manager, and Pod Identity cover it. Vault would be a new Tier-0 system, and keyless-first shrinks the problem instead of adding one.
  • Status: Reloader is not deployed (the only reloader in-tree is Alloy’s disabled configReloader); there are zero aws_secretsmanager_secret_rotation resources; there’s no age metric (the only secrets alerts are ExternalSecret/ClusterSecretStore Ready=False — sync-failures, not age); refresh tiers are unwired (flat 1h). A manual runbook covers ~4 secrets. Phase 1 (deploy the primitives) is authored but unbuilt; verify preprod first on unpark.

Status ledger

Section titled “Status ledger”
  • LIVE: keyless everywhere (Pod Identity/OIDC/cosign); SOPS config-in-git; ESO plus ClusterSecretStores plus 16 platform ExternalSecrets; IAM scoped to platform/*.
  • Designed, not built: the tenant config/secrets paved road (schema reserved, inert), and automated rotation (classification decided, primitives not deployed).

Gotchas

Section titled “Gotchas”
  • git vs store: config identifier the IaC reads → SOPS; credential a workload uses → Secrets Manager + ExternalSecret. Never cross them.
  • Rotated secret didn’t take: ESO re-synced the Secret, but the pod cached the env var — restart it (Reloader will automate; today manual).
  • SecretSyncedError: the remoteRef.key doesn’t exist, or isn’t under the IAM-scoped platform/*.
  • config-load fails to decrypt: the running identity needs kms:Decrypt on platform-sops — Terragrunt decrypts in-process; the sops CLI is only for editing.
  • module default * vs unit platform/*: the external-secrets module defaults to a wide * scope; the live unit narrows it — always set the prefix.

Go deeper

Section titled “Go deeper”

Deep dives: config-in-git (SOPS) · runtime secrets (ESO) · rotation & lifecycle. Runbook: docs/runbooks/secret-rotation.md. Related: Identity, Foundations → IaC. External: SOPS · External Secrets Operator · AWS Secrets Manager · EKS Pod Identity.