Test Architecture

This guide explains the design decisions behind Crawbl's E2E test suite — the 3-user model, auth bypass, feature file layout, and cleanup strategy.

3-User Model

Each test run creates exactly 3 users at suite level. These users are reused across all 50 scenarios:

User	Purpose
primary	Most scenarios — auth, profile, legal, workspaces, chat, agents
frank	Multi-user isolation and account deletion tests
grace	Multi-user isolation tests

Why 3 Users Instead of Per-Scenario

Each user triggers a UserSwarm custom resource, which provisions a StatefulSet, PVC, Services, and pods. Creating a user per scenario would spawn 50+ UserSwarm CRs, overwhelming the cluster with hundreds of Kubernetes resources.

The 3-user model caps cluster load at 3 UserSwarms regardless of how many scenarios run.

Auth Bypass (X-E2E-Token)

Tests skip Firebase JWT authentication using the X-E2E-Token header. The flow looks like this:

1

Step 1

Send the E2E headers

The test request includes the shared X-E2E-Token value plus the X-E2E-* identity headers.

2

Step 2

Route through the E2E path

Envoy sends the request through orchestrator-e2e, which has no JWT SecurityPolicy.

3

Step 3

Validate inside the backend

The orchestrator middleware checks CRAWBL_E2E_TOKEN and reads identity from X-E2E-UID, X-E2E-Email, and X-E2E-Name.

4

Step 4

Handle the request normally

Once the principal is attached, the rest of the handler logic runs as usual.

This bypass is disabled in production. The orchestrator checks env != "production" && env != "prod" before accepting E2E tokens. The token itself is stored in AWS Secrets Manager and synced to the cluster via External Secrets Operator.

Feature Files

All 13 feature files live in crawbl-backend/internal/testsuite/e2e/features/:

File	Coverage
01_health.feature	Health check, public endpoints
02_auth.feature	Sign-up, sign-in, idempotency
03_profile.feature	Profile CRUD, FCM token registration
04_legal.feature	Terms/privacy acceptance and status
05_workspaces.feature	Workspace listing, default agents
06_chat.feature	Conversations, messages, pagination, validation
07_multi_user.feature	Workspace/agent/conversation isolation between users
08_cleanup.feature	Account deletion, soft-delete verification
09_agent_communication.feature	ZeroClaw identity, web search, attribution
10_mentions.feature	@Research/@Writer routing in swarm chat
11_agent_conversations.feature	Per-agent conversation isolation
12_mobile_first_launch.feature	Complete onboarding + retry flow
13_edge_cases.feature	404s, 400s, 401s, runtime status

Step Definition Files

Step definitions are split by responsibility:

File	Responsibility
steps_http.go	HTTP request construction and execution
steps_assert.go	Response status, body, and JSON assertions
steps_db.go	Direct database state verification
steps_user.go	User creation, authentication, lifecycle
steps_state.go	Value storage and {{...}} interpolation

3-Layer Cleanup Strategy

Test resources must never accumulate in the cluster. Crawbl uses three independent cleanup layers:

1

Step 1

Suite cleanup

After all scenarios finish, the suite calls DELETE /v1/auth/delete for each of the 3 test users.

2

Step 2

Signal handler cleanup

If the test process gets SIGINT or SIGTERM, RunUntilSignal performs graceful cleanup before exit.

3

Step 3

Reaper CronJob cleanup

A cluster-side reaper runs every 2 minutes and removes stale e2e-* users older than 2 hours.

crawbl platform operator reaper --max-age=2h

Summary

Layer	Trigger	Action
Suite cleanup	All scenarios complete	DELETE /v1/auth/delete for all 3 users
Signal handler	SIGINT/SIGTERM	Graceful cleanup before exit
Reaper CronJob	Every 2 minutes	Finds and removes stale e2e-* users older than 2h

The DELETE /v1/auth/delete handler looks up the user by subject, lists their workspaces, calls runtimeClient.DeleteRuntime(workspaceID) for each, and soft-deletes the user record. The operator finalizer then cleans up all child Kubernetes resources (namespace, StatefulSet, PVC, Services).

---

What's next: CI/CD Pipeline