Skip to main content

Cloudflare Pages

The documentation site (crawbl-docs) and marketing website (crawbl-website) are hosted on Cloudflare Pages — a static site hosting platform with a global CDN, automatic HTTPS, and built-in access control.

Why Cloudflare Pages​

Previously, both sites were served by nginx pods inside the Kubernetes cluster, managed by ArgoCD. This worked but had drawbacks:

  • Static file serving consumed cluster resources meant for compute workloads
  • No edge caching — every visitor hit the DigitalOcean datacenter directly
  • Scaling required adding K8s replicas
  • No built-in authentication for private docs

Cloudflare Pages solves all of these:

  • Zero K8s resources — no deployments, services, or HTTPRoutes to manage
  • Global CDN — 300+ edge locations, sub-50ms TTFB worldwide
  • Built-in auth — Cloudflare Access protects /internal paths
  • Free tier — unlimited bandwidth, 500 deploys/month
  • DDoS protection — enterprise-grade, included

Architecture​

┌──────────────┐     CNAME      ┌─────────────────────┐
│ Browser      │ ──────────────▶│ Cloudflare CDN      │
│              │                │ (300+ edge PoPs)    │
└──────────────┘                └────────┬────────────┘
                                         │
                                         ▼
                                ┌─────────────────────┐
                                │ Cloudflare Pages     │
                                │ (static files)       │
                                └─────────────────────┘

  dev.docs.crawbl.com  →  crawbl-docs.pages.dev
  crawbl.com           →  crawbl-website.pages.dev

Projects​

ProjectRepoBuild OutputCustom DomainPages.dev URL
crawbl-docscrawbl-docsbuild/ (Docusaurus)dev.docs.crawbl.comcrawbl-docs.pages.dev
crawbl-websitecrawbl-websiteout/ (Next.js)crawbl.comcrawbl-website.pages.dev

Deploying​

# Deploy docs
crawbl app deploy docs

# Deploy website
crawbl app deploy website

Each command:

  1. Resolves the sibling repo directory (e.g., ../crawbl-docs)
  2. Runs npm run build to generate static files
  3. Runs wrangler pages deploy <output-dir> --project-name <project> to upload to Cloudflare
  4. Creates a Git tag (semver auto-calculated from conventional commits)
  5. Publishes a GitHub release with Claude-enriched notes

Manual deploy with wrangler​

If you need to deploy without the CLI:

# Docs
cd crawbl-docs
npm run build
wrangler pages deploy build --project-name crawbl-docs

# Website
cd crawbl-website
npm run build
wrangler pages deploy out --project-name crawbl-website

Required environment variables​

The wrangler CLI needs these env vars (already in crawbl-backend/.env):

VariableDescription
CLOUDFLARE_API_TOKENCloudflare API token with Pages permissions
CLOUDFLARE_ACCOUNT_IDCloudflare account ID: 27414702a30e3e8ce6d2054aeecef638

Required tools​

  • npm — builds the static site
  • wrangler — Cloudflare's CLI for Pages deployments (npm install -g wrangler)
  • gh — GitHub CLI for creating releases
  • claude — Claude CLI for enriched release notes

API Token Permissions​

The Cloudflare API token needs these permission scopes:

ScopePermissionPurpose
Account > Cloudflare PagesEditCreate projects, deploy, manage domains
Account > Access: Apps and PoliciesEditManage Cloudflare Access (auth for /internal)
Zone > DNSEditUpdate DNS records

To create or update the token:

  1. Go to https://dash.cloudflare.com/27414702a30e3e8ce6d2054aeecef638/api-tokens
  2. Click "Create Token" or edit the existing token
  3. Add the three permission scopes above
  4. Set Zone to crawbl.com and Account to your account

DNS Configuration​

Both domains use CNAME records pointing to Cloudflare Pages, proxied through Cloudflare:

RecordTypeNameContentProxied
docsCNAMEdev.docscrawbl-docs.pages.devYes
websiteCNAME@ (root)crawbl-website.pages.devYes

Cloudflare handles CNAME flattening for the root domain automatically.

info

The old A records pointing to the DigitalOcean load balancer (68.183.241.139) have been removed. Other services (API, ArgoCD, pgweb, etc.) still use A records to the LB.

Access Control (Internal Docs)​

Internal documentation at /internal/* is protected by Cloudflare Access:

  • Team members authenticate via email OTP (one-time password)
  • No passwords to manage — Cloudflare sends a code to your email
  • Public docs at / remain accessible to everyone
  • The *.pages.dev URL bypasses Access (use custom domain for production)

How it works​

  1. User visits dev.docs.crawbl.com/internal/
  2. Cloudflare Access intercepts the request
  3. User enters their email → receives a one-time code
  4. After authentication, a session cookie is set (configurable expiry)
  5. Subsequent requests pass through until the session expires

Managing Access​

Access policies are managed in the Cloudflare Zero Trust dashboard: https://one.dash.cloudflare.com/

To add a new team member: add their email domain or specific email to the Access policy.

Troubleshooting​

Deploy fails with "Authentication error"​

The API token lacks Cloudflare Pages permissions. Update the token at: https://dash.cloudflare.com/27414702a30e3e8ce6d2054aeecef638/api-tokens

Custom domain shows "pending"​

SSL certificate provisioning takes 1-5 minutes after adding a custom domain. Check status:

curl -s "https://api.cloudflare.com/client/v4/accounts/27414702a30e3e8ce6d2054aeecef638/pages/projects/crawbl-docs/domains" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" | python3 -m json.tool

Site shows old content after deploy​

Cloudflare caches aggressively. Purge the cache:

curl -X POST "https://api.cloudflare.com/client/v4/zones/a6e53522f634b470a7d5ca3fb0928934/purge_cache" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"purge_everything":true}'

wrangler not found​

Install it globally:

npm install -g wrangler