Codebase Layout

The Crawbl backend lives in a single Go repository (crawbl-backend).

In plain language, this page is the map you use when you know what kind of change you need to make, but you do not yet know where in the repo that change belongs.

Understanding its directory structure is the fastest way to orient yourself when you need to find or change something.

Repository Tree

crawbl-backend/
├── cmd/
│   ├── crawbl/                   # Unified CLI + runtime entrypoints
│   │   ├── app/                  # `crawbl app build`
│   │   ├── dev/                  # local dev commands (`start`, `lint`, `verify`, ...)
│   │   ├── infra/                # `crawbl infra plan/update/destroy/bootstrap`
│   │   ├── platform/             # platform subcommands (orchestrator, userswarm)
│   │   ├── setup/                # developer machine bootstrap
│   │   └── test/                 # `crawbl test unit|e2e`
│   └── envoy-auth-filter/        # Envoy ext_authz WASM filter binary
│
├── internal/                     # All business logic (unexported)
│   ├── orchestrator/             # API domain core
│   │   ├── mcp/                  # Embedded MCP server
│   │   ├── repo/                 # Repository layer (persistence)
│   │   ├── service/              # Service layer (business logic)
│   │   ├── server/               # HTTP handlers + Socket.IO
│   │   └── types.go              # Domain types, interfaces, constants
│   ├── userswarm/                # Runtime client, Metacontroller webhook, reaper
│   ├── zeroclaw/                 # ZeroClaw runtime configuration + bootstrap
│   ├── infra/                    # Pulumi IaC (cluster + ArgoCD only)
│   │   ├── cluster/              # DOKS, VPC, DOCR
│   │   └── platform/             # ArgoCD bootstrap
│   ├── testsuite/                # Shared Godog e2e helpers and step bindings
│   └── pkg/                      # Shared internal packages
│       ├── configenv/            # Secret/env loading
│       ├── database/             # Postgres connection + transactions
│       ├── errors/               # Structured error types
│       ├── firebase/             # FCM push client
│       ├── hmac/                 # Reusable HMAC helpers
│       ├── httpserver/           # Auth middleware + response writers
│       ├── kube/                 # Kubernetes helpers
│       ├── realtime/             # Socket.IO broadcasting
│       ├── redisclient/          # Redis client
│       └── yamlvalues/           # YAML stack config loading
│
├── api/v1alpha1/                 # Kubernetes CRD types for UserSwarm
├── migrations/orchestrator/      # SQL migration files
├── dockerfiles/                  # Dockerfiles for all images
├── test-features/                # Cucumber/Gherkin feature files
└── .github/workflows/            # CI workflows

Key Directories Explained

cmd/ — Entry Points

This is where Go binaries start. The crawbl binary is both the developer CLI and the deployed platform binary. The envoy-auth-filter is a separate binary compiled to WASM for the Envoy Gateway edge filter.

internal/orchestrator/ — The Heart of the Backend

This is where you will spend most of your time. It contains the API's domain core, organized into three sub-layers (server, service, repo) that follow a strict top-down dependency rule. The types.go file at the root is the central contract shared by the rest of the package.

internal/userswarm/ and internal/zeroclaw/

These packages manage the runtime side. userswarm contains the Kubernetes-backed runtime client, the Metacontroller sync/finalize webhook, and the cleanup reaper job. zeroclaw handles runtime configuration and bootstrap file generation for ZeroClaw agent containers.

internal/infra/

Pulumi infrastructure code for bootstrapping the Kubernetes cluster and installing ArgoCD. This is intentionally minimal. The application workloads themselves are deployed from the separate crawbl-argocd-apps repository.

internal/pkg/ — Shared Utilities

Reusable packages used across the codebase. Notable ones include database (Postgres connections and transactions), errors (structured business vs. server errors), httpserver (middleware and auth), and realtime (Socket.IO broadcasting).

api/v1alpha1/

Kubernetes Custom Resource Definition (CRD) types for the cluster-scoped UserSwarm resource. These Go types define the schema that the runtime client and webhook work with.

migrations/orchestrator/

SQL migration files for the PostgreSQL database. Migrations run automatically on startup, so adding a new table or column means adding a new migration file here.

internal/testsuite/e2e/ and test-features/

End-to-end coverage uses Godog/Cucumber. The Go test harness and step definitions live under internal/testsuite/e2e/, while the Gherkin feature files live under test-features/.

What's Next

Now that you know where code lives, see the Layered Design to understand the rules governing how these packages interact.