Test Architecture
This guide explains the design decisions behind Crawbl's E2E test suite — the 3-user model, auth bypass, feature file layout, and cleanup strategy.
3-User Model
Each test run creates exactly 3 users at suite level. These users are reused across all 73 scenarios:
| User | Purpose |
|---|---|
primary | Most scenarios — auth, profile, legal, workspaces, chat, agents |
frank | Multi-user isolation and account deletion tests |
grace | Multi-user isolation tests |
Why 3 Users Instead of Per-Scenario
Each user triggers a UserSwarm custom resource, which provisions a workspace runtime Deployment, Services, and pods. Creating a user per scenario would spawn 73+ Deployments, overwhelming the cluster with hundreds of Kubernetes resources.
The 3-user model caps cluster load at 3 UserSwarms regardless of how many scenarios run.
Auth Bypass (X-E2E-Token)
Tests skip Firebase JWT authentication using the X-E2E-Token header. The flow looks like this:
Send the E2E headers
The test request includes the shared X-E2E-Token value plus the X-E2E-* identity headers.
Route through the E2E path
Envoy sends the request through orchestrator-e2e, which has no JWT SecurityPolicy.
Validate inside the backend
The orchestrator middleware checks CRAWBL_E2E_TOKEN and reads identity from X-E2E-UID, X-E2E-Email, and X-E2E-Name.
Handle the request normally
Once the principal is attached, the rest of the handler logic runs as usual.
This bypass is disabled in production. The orchestrator checks env != "production" && env != "prod" before accepting E2E tokens. The token itself is stored in AWS Secrets Manager and synced to the cluster via External Secrets Operator.
Feature Files
Feature files live in crawbl-backend/internal/testsuite/e2e/features/ organized into 8 category folders — 30 files, 73 scenarios total:
| Category | Files | Scenarios | Coverage |
|---|---|---|---|
auth/ | 4 | 9 | Sign-up, sign-in, logout, deletion, error cases |
chat/ | 7 | 18 | Conversations, messages, mentions, agent details, memories |
core/ | 5 | 10 | Health, legal, models, uploads, edge cases |
integrations/ | 1 | 3 | Tool catalog, agent tools, integration apps |
multi_user/ | 2 | 6 | Workspace isolation, cross-user access prevention |
profile/ | 2 | 5 | Profile CRUD, FCM tokens, error cases |
tools/ | 2 | 4 | Memory (store/recall/forget), web search |
workspaces/ | 3 | 9 | Default workspace, first launch, runtime readiness |
Step Definition Files
Step definitions are split by responsibility across 14 files:
| File | Responsibility |
|---|---|
steps_http.go | HTTP request construction and execution |
steps_assert.go | Response status, body, and JSON assertions |
steps_db.go | Direct database state verification |
steps_user.go | User creation, authentication, lifecycle |
steps_state.go | Value storage and {{...}} interpolation |
steps_auth.go | Auth flow steps (sign-up, sign-in, token handling) |
steps_chat.go | Conversation and message steps |
steps_workspaces.go | Workspace listing and runtime status steps |
steps_agents.go | Agent detail, history, and settings steps |
steps_profile.go | Profile and FCM token steps |
steps_legal.go | Terms/privacy acceptance steps |
steps_tools.go | Memory and web search tool steps |
steps_integrations.go | Integration catalog and app steps |
steps_multi_user.go | Cross-user isolation and access prevention steps |
Async Polling
Agent-side effects (memory writes, tool results, swarm state) are not synchronous. Tests use pollUntil(ctx, fn) to wait for these effects to settle:
- Polls every 1 second for up to 30 seconds
- Fails the step if the condition is not met within the timeout
- Used wherever a test depends on the agent runtime completing work before asserting
3-Layer Cleanup Strategy
Test resources must never accumulate in the cluster. Crawbl uses three independent cleanup layers:
Suite cleanup
After all scenarios finish, the suite calls DELETE /v1/auth/delete for each of the 3 test users.
Signal handler cleanup
If the test process gets SIGINT or SIGTERM, RunUntilSignal performs graceful cleanup before exit.
Reaper CronJob cleanup
A cluster-side reaper runs every 2 minutes and removes stale e2e-* users older than 2 hours along with their workspace runtime resources.
crawbl platform operator reaper --max-age=2h
Summary
| Layer | Trigger | Action |
|---|---|---|
| Suite cleanup | All scenarios complete | DELETE /v1/auth/delete for all 3 users |
| Signal handler | SIGINT/SIGTERM | Graceful cleanup before exit |
| Reaper CronJob | Every 2 minutes | Finds and removes stale e2e-* users older than 2h |
The DELETE /v1/auth/delete handler looks up the user by subject, lists their workspaces, calls runtimeClient.DeleteRuntime(workspaceID) for each, and soft-deletes the user record. The operator finalizer then cleans up all child Kubernetes resources (Deployment, Services).
What's next: CI/CD Pipeline