Architecture
Crawbl uses a three-layer architecture so each layer can scale and change independently.
How Crawbl Works
A person asks
A user sends a request through the mobile app, REST API, or WebSocket.
Crawbl coordinates
The orchestrator routes the request, checks approvals, and manages provider access.
Agent work happens in isolation
AI work runs in isolated containers. Each user gets their own runtime, not a shared process.
Results come back clearly
Results return as answers, suggestions, or approval requests.
- A chatbot answers once. Crawbl keeps working after the first answer.
- One request can trigger follow-up actions, approval checks, and background work.
Three-Layer Overview
Layer 1: Client Interface
| Interface | Purpose |
|---|---|
| Mobile App | Primary user interface (Flutter) |
| REST API | Synchronous operations |
| WebSocket | Real-time events and streaming |
All clients share one backend and one security boundary. Runtimes are never directly accessible from outside.
Layer 2: Orchestration Layer
The orchestrator is a Go service that coordinates the entire system.
- Auth -- JWT for mobile users, API keys for programmatic access, workspace-scoped permissions
- Routing -- Directs requests to the right agent runtime with load balancing and failover
- LLM Mediation -- Provider selection, cost controls, usage tracking, rate limiting
- Integration Hub -- OAuth token management, secrets storage, webhook handling
- Audit -- Request logging, access control verification, compliance reporting
The orchestrator talks to runtimes over internal networking and never exposes them to the public internet.
Layer 3: Runtime Layer
Each paid user gets a dedicated Agent Runtime pod. Pods are stateless and replaceable.
| Property | Description |
|---|---|
| Isolation | Each paid user gets a dedicated runtime |
| Security | No direct internet access |
| State | Stored in shared databases, not local volumes |
| Scaling | Automatic based on demand |
Platform data is stored in managed databases with encryption at rest.
The runtime layer scales independently of the orchestrator.
Communication Patterns
Request Flow
Tool Calling
- Agent runtime sends a tool request to the orchestrator via MCP
- Orchestrator validates permissions and calls the external service
- Result returns to the agent runtime
All external calls are logged, permissions are enforced centrally, and secrets never leave the control plane.
Why This Separation Matters
Security isolation
Agent pods never talk to the internet directly. The orchestrator mediates everything.
Independent scaling
The control plane and runtime layer scale separately.
Client flexibility
New interfaces (Slack bot, VS Code extension, CLI) only need the orchestrator API.
Cloud portability
The runtime layer is pure Kubernetes. Swapping clouds means changing cluster config, not rewriting agents.
Crawbl is built on modern cloud-native technologies optimized for security, density, and portability.