CI/CD Pipeline

Before You Change Anything

These steps can affect shared dev workloads or the GitOps control layer. Confirm the target repo, environment, and intended owner before mutating anything.

Builds and deployments happen locally via crawbl app deploy. CI is a validation gate only — it does not build or push images.

Pipeline Overview

Click diagram to zoom

What Triggers What

Trigger	Action	Workflow
Push to main in crawbl-backend	Infra drift check → E2E tests against live cluster + release tagging	deploy-dev.yml
crawbl app deploy agent-runtime (local)	Agent Runtime image build + deploy + release	local CLI

CI Jobs (deploy-dev.yml)

CI runs jobs on every push to main. The infra drift check runs first as a gate; E2E and release run after it passes.

infra-drift-check (runs first)

Builds its own crawbl CLI binary from the current commit, then runs crawbl infra plan to detect any uncommitted infrastructure drift. This is the first step in the pipeline — if there is uncommitted Pulumi drift, the build fails before any tests or tags are created.

e2e-test

Builds the crawbl CLI binary, configures kubectl against the dev cluster, and runs the full E2E suite against https://dev.api.crawbl.com. This validates that the code already deployed to the cluster is working correctly.

release

Calculates the next semver tag from conventional commit messages (feat: → minor, breaking !: → major, everything else → patch). If the tag already exists on remote, patch is bumped until a free tag is found. Creates a Git tag and publishes a GitHub release with notes enriched by the local claude CLI (sonnet model) from commit messages, plus a full changelog link (compare/prev...new).

CI Secrets

Secret	Purpose
DIGITALOCEAN_ACCESS_TOKEN	DOCR login, kubectl cluster access
CRAWBL_E2E_TOKEN	E2E auth bypass (must match orchestrator Helm values)

Local Deploy Workflow

Deployments are driven locally using crawbl app deploy. Each call does these steps atomically:

1

Step 1

Verify working tree

Checks that the working tree is clean and all commits are pushed. Skipped for docs, website, and agent-runtime.

2

Step 2

Build the image

The component image is built locally. Platform and agent-runtime use ko (Go binary compiled on your machine, no Dockerfile). The auth-filter uses Docker (TinyGo/WASM).

3

Step 3

Push to DOCR

The image is pushed to registry.digitalocean.com/crawbl/.

4

Step 4

Update crawbl-argocd-apps

The image tag is patched in the crawbl-argocd-apps repo and committed.

5

Step 5

Create Git tag and GitHub release

Creates a Git tag (auto-calculated; bumps patch if tag exists on remote). Publishes a GitHub release with notes enriched by the local claude CLI (sonnet model) and a full changelog link.

6

Step 6

Push and let ArgoCD sync

The apps repo commit is pushed. ArgoCD detects the Git change, renders Helm charts, and rolls the new image into the cluster.

Deploy commands

The tag is auto-calculated from conventional commits — --tag is optional. The working tree must be clean and fully pushed before deploying (skipped for docs, website, agent-runtime).

# Deploy a single component (tag auto-calculated via semver)
crawbl app deploy platform
crawbl app deploy auth-filter
crawbl app deploy docs
crawbl app deploy website
crawbl app deploy agent-runtime

# Deploy platform + auth-filter only
crawbl app deploy all

# Override with an explicit tag
crawbl app deploy platform --tag v1.2.3

Semver logic: scans commits since the last v* tag — feat: → minor bump, !: (breaking) → major bump, everything else → patch bump.

CLI shortcuts

Auto-semver applies — no manual tag needed:

crawbl app deploy platform        # deploy platform
crawbl app deploy auth-filter
crawbl app deploy docs
crawbl app deploy website
crawbl app deploy agent-runtime

Prerequisites

Log in to the DigitalOcean Container Registry before deploying:

doctl registry login

The token expires periodically — re-run if you get authentication errors.

Agent Runtime Image Builds

The Agent Runtime is deployed locally via crawbl app deploy agent-runtime from crawbl-backend. CI in crawbl-agent-runtime is disabled.

Tags follow the fork convention v<upstream>-crawbl.<N> (e.g. v0.6.5-crawbl.3). The N suffix auto-increments — no manual tag needed.

cd crawbl-backend
crawbl app deploy agent-runtime

Each call: builds the image → pushes to DOCR → updates crawbl-argocd-apps → creates the Git tag → publishes a GitHub release with Claude-enriched notes. ArgoCD detects the apps repo change and rolls the new runtime into the cluster.

Reusable Workflows

The following workflow files are kept for reference and for use with deploy-prod.yml, but are no longer called by deploy-dev.yml:

• reusable-build.yml
• reusable-deploy.yml
• reusable-update-argocd.yml
• reusable-rollback-argocd.yml
• reusable-infra-drift-check.yml
• reusable-e2e-test.yml

---

What's next: Local Deploy