Crawbl Internal Docs
Private developer documentation for the Crawbl team. Start here on your first day, come back when you need to look something up.
Learn the platform shape first
Crawbl has one public API, one orchestrator, and one isolated runtime per user. If you understand that shape first, the rest of the docs stop feeling random.
The 60-second tour is the front door to this library. It gives you the mental model first, then hands you off to the right starting pages.
Your first day. Get your machine set up, connect to the dev environment, and run the backend locally.
1. Install Tools
Install Go, Docker, kubectl, and the crawbl CLI. Takes about 15 minutes.
2. Connect to Dev
Get cluster access, verify the API health endpoint, and find the shared service URLs.
3. Run Locally
Connect to the dev cluster, hit the health endpoint, and run the tests.
Your second day. Understand how the system works before writing code.
Codebase Layout
Where handlers, services, repos, and infrastructure code live in the repository.
Layered Design
The handler, service, repo layering rules and how data flows through each layer.
Runtime Contract
The gRPC bidirectional streaming protocol between the orchestrator and agent pods.
Your first week. Start building features, writing tests, and following the team's patterns.
Build Features
Add endpoints, write migrations, and follow the coding patterns the backend uses.
Ship Changes
Deploy locally via the crawbl CLI. CI validates on push to main.
Fix Problems Fast
Runbooks for crash loops, secret sync, auth failures, image pulls, and stuck namespaces.
Observability
Search logs across all services, query metrics, and track token usage and billing.
Security
HMAC edge filter, secrets management, and how credentials flow from AWS to pods.
Look up exact paths, endpoints, table schemas, and CLI commands.