Auth Failures

Before You Change Anything

Start with inspection and narrowing steps first. Some fixes in debugging pages mutate shared resources, so separate observation from recovery.

Use this page when a request is being rejected before your application code seems to run.

Authentication problems usually fail before your handler runs. In Crawbl, that usually means one of three places:

• Envoy Gateway rejected the request
• the HMAC WASM filter rejected an X-Token request
• the backend middleware did not see trusted identity headers

There is no dev-token parser in the current backend. Use the right auth path for the environment you are testing.

If you are new, the most important first step is simple: decide whether you are testing local dev, E2E bypass, mobile X-Token, or bearer-token auth. Most confusion comes from mixing those paths.

Which Auth Path Are You Using?

Case	What to use
Local backend (./crawbl dev start)	No token needed. Local auth is skipped and a default principal is injected.
Shared dev cluster, automated tests, manual smoke checks	X-E2E-Token plus X-E2E-UID, X-E2E-Email, and X-E2E-Name
Mobile or real user traffic	Firebase JWT via X-Token or Authorization: Bearer

E2E Bypass Debugging

In non-production, the backend accepts a matching X-E2E-Token when CRAWBL_E2E_TOKEN is configured.

Example request:

curl -i -X POST https://dev.api.crawbl.com/v1/auth/sign-in \
  -H "X-E2E-Token: $CRAWBL_E2E_TOKEN" \
  -H "X-E2E-UID: test-user" \
  -H "X-E2E-Email: test@crawbl.test" \
  -H "X-E2E-Name: Test User" \
  -H "X-Device-Info: my-laptop" \
  -H "X-Device-ID: dev-001" \
  -H "X-Version: 0.1.0+dev" \
  -H "X-Timezone: UTC"

If you get a 401 or 400, verify this flow:

1

Step 1

Check the backend token

Confirm CRAWBL_E2E_TOKEN is configured on the backend.

2

Step 2

Check the E2E identity headers

Make sure the request includes X-E2E-Token, X-E2E-UID, X-E2E-Email, and X-E2E-Name.

3

Step 3

Check the device headers

Confirm the standard device headers are present as well.

Required Headers Checklist

For the mobile X-Token path, these headers matter:

Header	Example Value	Purpose
X-Token	<firebase-jwt>	Mobile auth credential
X-Device-Info	iPhone 15 Pro	Client device description
X-Device-ID	abc-123	Unique device identifier
X-Version	0.1.0+dev	Client app version
X-Timezone	UTC	User timezone

If you are using Authorization: Bearer, Envoy can still validate the JWT, but the HMAC WASM filter only cares about the X-Token path.

HMAC Debugging

The Envoy WASM filter validates HMAC only for requests that include X-Token.

If you get 403 Forbidden with no clear backend error, start here, because the request may have been rejected at the edge before it ever reached the orchestrator.

Check the HMAC Secret

Check the filter config secret:

kubectl get secret hmac-wasm-config -n envoy-gateway-system -o jsonpath='{.data.hmac_secret}' | base64 -d

If that secret is missing or stale, the filter rejects the request before it reaches the backend.

Check the Auth Filter Logs

kubectl logs -n envoy-gateway-system deployment/envoy-gateway -c envoy

Look for hmac mismatch or invalid signature messages.

Fix a Mismatched Secret

Update the secret in AWS Secrets Manager, then force ESO to re-sync:

kubectl annotate externalsecret hmac-config \
  -n envoy-gateway-system \
  force-sync=$(date +%s) --overwrite

kubectl annotate externalsecret orchestrator-secrets \
  -n backend \
  force-sync=$(date +%s) --overwrite

Then restart affected workloads:

kubectl rollout restart deployment/orchestrator -n backend
kubectl rollout restart deployment/userswarm-webhook -n backend

JWT Debugging

Envoy SecurityPolicy validates Firebase JWTs and forwards trusted X-Firebase-* headers to the backend.

The orchestrator does not re-validate the same mobile JWT itself.

Token Expired

Firebase JWTs expire after 1 hour. If you get 401 with token expired:

• Refresh the token from the mobile app.
• For API testing, generate a fresh token using the Firebase Admin SDK or the mobile app's auth flow.

Invalid Audience

The JWT aud claim must match the Firebase project ID.

If it does not match, the token was minted for the wrong project and the request is rejected.

Clock Skew

If the cluster node clock is badly off, JWT validation can fail. Check the node heartbeat times:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].lastHeartbeatTime}{"\n"}{end}'

Quick Debugging Flow

1

Step 1

Decide the auth path

First decide whether you are testing local auth bypass, E2E bypass, mobile X-Token, or standard bearer-token auth.

2

Step 2

Stop early for local dev

If you are running locally, stop looking for tokens. Local auth is bypassed.

3

Step 3

Verify E2E headers if needed

For the E2E path, confirm the X-E2E-* headers and CRAWBL_E2E_TOKEN.

4

Step 4

Check the HMAC path

For X-Token requests, verify HMAC filter config and device headers.

5

Step 5

Check the JWT path

For bearer tokens, verify Envoy auth behavior and token freshness.

6

Step 6

Check backend logs last

Then inspect backend logs to confirm whether the request reached the orchestrator at all.

kubectl logs -n backend deployment/orchestrator --tail=20

---

What's next: Namespace Stuck Terminating

🔗 Terms On This Page

If a term below is unfamiliar, open its glossary entry. For the full list, go to Internal Glossary.

• Envoy Gateway: The public gateway that receives incoming traffic and routes it to internal services.
• Firebase JWT: The signed Firebase identity token used to authenticate a user request.
• HMAC: A shared-secret signature scheme used to prove a request came from a trusted client.
• SecurityPolicy: The gateway rule that validates bearer tokens before traffic reaches the backend.