Secrets Management

Before You Change Anything

Security pages describe boundaries that protect shared systems and user data. Validate the boundary you are changing before you test or modify anything.

Crawbl never stores secrets in Git. Application secrets such as database passwords, provider API keys, and signing keys live in AWS Secrets Manager and are synced into Kubernetes by External Secrets Operator (ESO).

The Secrets Flow

Click diagram to zoom

How It Works

1

Step 1

Store secrets in AWS

All secrets follow the naming convention crawbl/{environment}/{scope}/{name}.

AWS Secrets Manager Path	What It Contains
crawbl/dev/backend/orchestrator	Orchestrator runtime secrets such as DB credentials, MCP signing key, and Firebase config
crawbl/dev/backend/postgresql	Database credentials
crawbl/dev/backend/redis	Redis auth password
crawbl/dev/infra/cloudflare	Cloudflare API token for DNS
crawbl/dev/runtime/openai	OpenAI API key for agent pods
crawbl/dev/edge/hmac	HMAC secret for the Envoy auth filter

2

Step 2

Sync them into Kubernetes

External Secrets Operator watches ExternalSecret resources. Each one points at an AWS Secrets Manager path and materializes a Kubernetes Secret in the target namespace.

A cluster-scoped ClusterSecretStore named aws-secrets-manager connects ESO to AWS. It authenticates with the bootstrap aws-credentials secret in the external-secrets namespace.

3

Step 3

Let workloads consume them

Pods read the synced Kubernetes secrets through envFrom, env.valueFrom, or mounted files.

At runtime, the orchestrator reads orchestrator-vault-secrets, the Envoy WASM filter reads hmac-wasm-config, and ZeroClaw pods read runtime-openai-secrets.

Bootstrap: The Chicken-And-Egg Problem

ESO needs AWS credentials to pull the rest of the secrets. Pulumi solves this by creating the initial aws-credentials Secret during cluster bootstrap. That bootstrap secret is the only credential Pulumi manages directly.

Rotation

Rotating a secret follows this flow:

1

Step 1

Update the source secret

Change the value in AWS Secrets Manager first.

2

Step 2

Re-sync Kubernetes

Force or wait for the matching ExternalSecret to sync the updated value into the cluster.

3

Step 3

Restart affected pods

Restart workloads that only read the value during startup.

For shared secrets, restart every component that consumes them.

What's Next

See the HMAC Auth Filter to understand how incoming requests are authenticated at the edge.