Secret Sync Failures

Before You Change Anything

Start with inspection and narrowing steps first. Some fixes in debugging pages mutate shared resources, so separate observation from recovery.

Use this page when Kubernetes secrets are missing, stale, or incomplete because the sync from AWS did not happen correctly.

External Secrets Operator (ESO) syncs secrets from AWS Secrets Manager into Kubernetes Secrets.

When syncing fails, pods that depend on those secrets usually fail to start or run with missing configuration.

Symptoms

Pods stuck in CrashLoopBackOff with "secret not found" or "key not found" in logs.
ExternalSecret resources show SecretSyncedError status.
New secrets added to AWS Secrets Manager do not appear in the cluster.

If you are new, the most common root cause on fresh clusters is missing aws-credentials in the external-secrets namespace.

Debug: Secret Sync Failures

What is the ExternalSecret status?

The tree above is the fastest path. If you prefer the manual checklist, use this flow:

Step 1

Check ExternalSecret status

Start by checking whether the ExternalSecret is healthy.

kubectl get externalsecret -n backend
kubectl describe externalsecret orchestrator-secrets -n backend

A healthy status is usually SecretSynced. If you see SecretSyncedError, read the Conditions section for the exact failure.

Step 2

Check the ESO controller

If the controller itself is down, nothing will sync.

kubectl get pods -n external-secrets
kubectl logs -n external-secrets deployment/external-secrets

Step 3

Fix missing aws-credentials

On fresh clusters, the most common root cause is a missing aws-credentials secret in the external-secrets namespace.

kubectl get secret aws-credentials -n external-secrets
kubectl create secret generic aws-credentials \
  -n external-secrets \
  --from-literal=access-key-id=$AWS_ACCESS_KEY_ID \
  --from-literal=secret-access-key=$AWS_SECRET_ACCESS_KEY
kubectl rollout restart deployment/external-secrets -n external-secrets

Step 4

Verify the AWS source secret

If ESO has credentials but still fails, make sure the source secret exists in AWS Secrets Manager.

aws secretsmanager get-secret-value \
  --secret-id crawbl/dev/backend/orchestrator \
  --region eu-central-1

Step 5

Force a re-sync

Bump the force-sync annotation so ESO fetches the latest value immediately.

kubectl annotate externalsecret orchestrator-secrets \
  -n backend \
  force-sync=$(date +%s) --overwrite

Step 6

Verify the Kubernetes secret

Confirm the materialized Kubernetes secret exists and has the expected keys.

kubectl get secret orchestrator-vault-secrets -n backend
kubectl get secret orchestrator-vault-secrets -n backend -o jsonpath='{.data}' | python3 -m json.tool

Step 7

Restart dependent pods

If workloads started before the secret was fixed, restart them so they read the updated values.

kubectl rollout restart deployment/orchestrator -n backend

Quick Reference

Problem	Command
Check ExternalSecret status	`kubectl get externalsecret -n backend`
Check ESO controller	`kubectl get pods -n external-secrets`
Create aws-credentials	`kubectl create secret generic aws-credentials -n external-secrets --from-literal=...`
Force re-sync	`kubectl annotate externalsecret <name> -n <ns> force-sync=$(date +%s) --overwrite`
Restart ESO	`kubectl rollout restart deployment/external-secrets -n external-secrets`

What's next: Image Pull Errors

🔗 Terms On This Page

If a term below is unfamiliar, open its glossary entry. For the full list, go to Internal Glossary.

AWS Secrets Manager: The AWS service that stores shared secret values before they are copied into Kubernetes.
External Secrets Operator: The controller that copies secrets from AWS Secrets Manager into Kubernetes Secrets.
Pulumi: The infrastructure-as-code tool that creates the cluster and foundational platform resources.

Symptoms​