Dev Environment & Services

Everything you need to work with the shared dev environment — URLs, credentials, fast checks, and cluster access.

Before You Change Anything

These pages often point at shared systems. Confirm the cluster, namespace, and ownership boundary before running mutating commands.

Getting Started

Prerequisites

# Install mise (version manager) and all required tools
curl https://mise.run | sh
eval "$(~/.local/bin/mise activate zsh)"
cd crawbl-backend && mise install

This installs the main developer tools at the versions the repo expects. See .mise.toml for the exact list.

Sourcing Secrets

All secrets are in crawbl-backend/.env and AWS Secrets Manager. Source them before testing or running infrastructure commands:

cd crawbl-backend
set -a && source .env && set +a

Any ./crawbl command that requires environment variables must be run with .env sourced:

set -a && source .env && set +a && ./crawbl <command>

kubectl Access

doctl kubernetes cluster kubeconfig save crawbl-dev
kubectl get pods -n backend
kubectl get pods -n userswarm-controller
kubectl get pods -n userswarms

What lives where:

• backend: the main backend services and shared support services
• userswarm-controller: the controller that turns UserSwarm records into actual runtime resources
• userswarms: the per-user runtime workloads and their storage

Fast Checks

1

Step 1

Check the backend health

Use the /v1/health endpoint first. If that is down, there is no point debugging higher-level behavior yet.

2

Step 2

Smoke-test auth

If CRAWBL_E2E_TOKEN is set, make one authenticated request without the mobile app. That tells you whether the non-production auth path is wired correctly.

3

Step 3

Run the live test suite

Use ./crawbl test e2e --base-url https://api-dev.crawbl.com -v when you want the suite to exercise the deployed cluster end to end.

# Health check
curl https://api-dev.crawbl.com/v1/health

# Authenticated request (non-production E2E bypass)
curl -i -X POST https://api-dev.crawbl.com/v1/auth/sign-in \
  -H "X-E2E-Token: $CRAWBL_E2E_TOKEN" \
  -H "X-E2E-UID: test-user" \
  -H "X-E2E-Email: test@crawbl.test" \
  -H "X-E2E-Name: Test User" \
  -H "X-Device-Info: CLI/test/test" \
  -H "X-Device-ID: cli" \
  -H "X-Version: 1.0.0 (1)" \
  -H "X-Timezone: UTC"

# Run E2E tests against dev
./crawbl test e2e --base-url https://api-dev.crawbl.com -v

Expected results:

• /v1/health returns HTTP 200
• POST /v1/auth/sign-in returns HTTP 204 when CRAWBL_E2E_TOKEN is set correctly
• ./crawbl test e2e starts the end-to-end test suite against the live cluster

---

Services

URL	Service	Auth	Purpose
api-dev.crawbl.com	Orchestrator API	Firebase token or E2E token	Main backend API for the mobile app, agents, and tests
postgres-dev.crawbl.com	pgweb	Basic Auth	Browser UI for checking database tables and running queries
dev.docs.crawbl.com	Documentation	Basic Auth	This internal docs site
argocd-dev.crawbl.com	ArgoCD	Username/password	Deployment dashboard that shows whether the cluster matches Git
(cluster-internal)	ClickHouse	Secret	LLM usage analytics database
(cluster-internal)	NATS JetStream	Secret	Usage event streaming between orchestrator and ClickHouse

How Traffic Reaches These Services

All shared dev services ultimately run inside the crawbl-dev Kubernetes cluster.

Public URLs first hit the gateway, then TLS is handled, and finally traffic is routed to the right internal service.

Implementation details:

• DOKS is the DigitalOcean Kubernetes cluster that hosts the dev environment
• Envoy Gateway is the public entry point
• cert-manager keeps TLS certificates valid
• external-dns updates Cloudflare DNS records automatically

---

Orchestrator API

URL: https://api-dev.crawbl.com

This is the main backend API.

It is used by the mobile app, by agents through MCP-backed platform calls, and by E2E tests.

If you are unsure where to start, this is usually the first service to check because most platform work eventually flows through it.

Authentication

The API accepts three auth paths, depending on who is calling it.

If the phrase Firebase JWT is unfamiliar, read it as "the signed login token produced after a user authenticates with Firebase."

Path	Header	Used by
Firebase JWT	X-Token: <firebase-jwt>	Mobile app
Firebase JWT	Authorization: Bearer <firebase-jwt>	Dev tooling (Postman, curl)
E2E bypass	X-E2E-Token: <token>	CI pipeline only

Public endpoints (no auth required):

• GET /v1/health — liveness probe
• GET /v1/legal — terms and privacy policy

Credential	Location
Firebase JWT	Generated by Firebase SDK in the mobile app
E2E token	crawbl-backend/.env → CRAWBL_E2E_TOKEN
E2E token (cluster)	AWS SM → crawbl/dev/backend/orchestrator → CRAWBL_E2E_TOKEN

Quick test

# Health check (no auth)
curl https://api-dev.crawbl.com/v1/health

# With E2E token (from .env)
source crawbl-backend/.env
curl -H "X-E2E-Token: $CRAWBL_E2E_TOKEN" https://api-dev.crawbl.com/v1/health

---

pgweb (Database UI)

URL: https://postgres-dev.crawbl.com

pgweb is the quickest way to inspect the dev PostgreSQL database from a browser.

It supports table browsing, schema inspection, and ad hoc SQL queries.

Use this when you need to inspect data quickly without opening psql.

Credentials

Field	Value
Username	admin
Password	crawbl2026

Your browser will show a Basic Auth prompt when you visit the URL.

Connected database

Parameter	Value
Host	backend-postgresql (cluster-internal)
Port	5432
User	crawbl
Database	crawbl
Schemas	orchestrator, public

Updating the password

The Basic Auth file is stored in AWS Secrets Manager.

The gateway expects the password in {SHA} format:

cd crawbl-backend && source .env

# Generate new htpasswd
SHA_HASH=$(printf "new-password" | openssl dgst -sha1 -binary | openssl base64)
HTPASSWD="admin:{SHA}${SHA_HASH}"

# Update AWS SM
aws secretsmanager update-secret \
  --secret-id crawbl/dev/pgweb/auth \
  --secret-string "{\".htpasswd\":\"${HTPASSWD}\"}" \
  --region eu-central-1

# Force ExternalSecret re-sync (or wait ~1 min)
kubectl annotate externalsecret pgweb-secrets -n backend \
  force-sync=$(date +%s) --overwrite

---

Documentation Site

URL: https://dev.docs.crawbl.com

This is the docs site you are reading now. It is hosted on Cloudflare Pages and deployed locally via crawbl app deploy docs.

Access Control

Internal docs at /internal are protected by Cloudflare Access (email OTP). Allowed identities:

• Any @crawbl.com email
• Specific team member emails configured in the Cloudflare Access policy

Public docs at / are accessible to everyone without authentication.

See the Cloudflare Pages guide for full details.

---

ArgoCD

URL: https://argocd-dev.crawbl.com

ArgoCD is the deployment dashboard for the dev cluster.

It shows whether the live cluster matches what is committed in the deployment repo.

If you are new to ArgoCD, the useful mental model is: it is the system that makes the cluster match what is committed in the deployment repo.

Credentials

Field	Value
Username	admin
Password	Retrieve with the command below

# Get the admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d && echo

Alternative: kubectl port-forward

If you want local-only access instead of going through the public gateway:

kubectl port-forward -n argocd svc/argocd-server 8080:80
# Open http://localhost:8080 — same username/password

---

Observability

The monitoring stack (VictoriaMetrics, VictoriaLogs, Fluent Bit) is prod-only and is not deployed in the dev environment.

For logs in dev, use kubectl logs directly:

kubectl logs -n backend deployment/orchestrator -f
kubectl logs -n backend deployment/orchestrator --previous

For resource usage in dev, use kubectl top or the DigitalOcean cluster dashboard:

kubectl top pods -n backend
kubectl top nodes

---

Where Shared Secrets Live

AWS Secrets Manager is the source of truth for shared cluster secrets.

Kubernetes receives copies of those secrets through the External Secrets Operator.

If that name is unfamiliar, it is the controller that reads secrets from AWS and writes matching Kubernetes Secret objects for workloads to use.

AWS SM Path	K8s Secret	Namespace	Contents
crawbl/dev/backend/orchestrator	orchestrator-vault-secrets	backend	DB password, Redis password, JWT secret, E2E token, MCP signing key
crawbl/dev/backend/postgresql	backend-postgresql-auth	backend	PostgreSQL postgres and crawbl user passwords
crawbl/dev/backend/firebase	firebase-service-account	backend	Firebase Admin SDK service account JSON (FCM push)
crawbl/dev/infra/cloudflare	cloudflare-api-token	cert-manager, external-dns	Cloudflare API token for DNS-01 and DNS records
crawbl/dev/runtime/openai	runtime-openai-secrets	userswarms	OpenAI API key for Agent Runtime pods
crawbl/dev/edge/hmac	hmac-wasm-config	envoy-gateway-system	HMAC signing key for edge auth filter
crawbl/dev/backup/aws	backup-aws-credentials	backend, userswarms	AWS credentials for PVC backup jobs
crawbl/dev/pgweb/auth	pgweb-htpasswd	backend	pgweb Basic Auth htpasswd
crawbl/dev/backend/clickhouse	backend-clickhouse-auth	backend	ClickHouse password
crawbl/dev/backend/nats	backend-nats-auth	backend	NATS credentials

Viewing a secret value

cd crawbl-backend && source .env

# List all secrets
aws secretsmanager list-secrets --region eu-central-1 \
  --query 'SecretList[].Name' --output table

# View a specific secret
aws secretsmanager get-secret-value \
  --secret-id crawbl/dev/backend/orchestrator \
  --region eu-central-1 \
  --query SecretString --output text | jq .

---

How TLS Is Handled

Dev subdomains that route through the K8s gateway share one certificate managed by cert-manager. Domains hosted on Cloudflare Pages (dev.docs.crawbl.com, crawbl.com) manage their own TLS automatically.

• Issuer: Let's Encrypt (letsencrypt-cloudflare ClusterIssuer)
• Challenge: DNS-01 via Cloudflare
• Secret: public-edge-tls in envoy-gateway-system
• Domains: api-dev.crawbl.com, argocd-dev.crawbl.com, postgres-dev.crawbl.com, river-dev.crawbl.com

To add a new subdomain, update the dnsNames list in crawbl-argocd-apps/components/cert-manager/resources/certificate.yaml.

🔗 Terms On This Page

If a term below is unfamiliar, open its glossary entry. For the full list, go to Internal Glossary.

• DOKS: DigitalOcean Kubernetes, the managed Kubernetes service used for the Crawbl cluster.
• Envoy Gateway: The public gateway that receives incoming traffic and routes it to internal services.
• cert-manager: The controller that requests, renews, and stores TLS certificates inside Kubernetes.
• external-dns: The controller that creates and updates DNS records from Kubernetes or gateway resources.
• Firebase JWT: The signed Firebase identity token used to authenticate a user request.
• ArgoCD: The GitOps deployment system that keeps the cluster aligned with what is committed in Git.
• AWS Secrets Manager: The AWS service that stores shared secret values before they are copied into Kubernetes.
• External Secrets Operator: The controller that copies secrets from AWS Secrets Manager into Kubernetes Secrets.