Cloudflare Pages

The documentation site (crawbl-docs) and marketing website (crawbl-website) are hosted on Cloudflare Pages — a static site hosting platform with a global CDN, automatic HTTPS, and built-in access control.

Why Cloudflare Pages

Previously, both sites were served by nginx pods inside the Kubernetes cluster, managed by ArgoCD. This worked but had drawbacks:

• Static file serving consumed cluster resources meant for compute workloads
• No edge caching — every visitor hit the DigitalOcean datacenter directly
• Scaling required adding K8s replicas
• No built-in authentication for private docs

Cloudflare Pages solves all of these:

• Zero K8s resources — no deployments, services, or HTTPRoutes to manage
• Global CDN — 300+ edge locations, sub-50ms TTFB worldwide
• Built-in auth — Cloudflare Access protects /internal paths
• Free tier — unlimited bandwidth, 500 deploys/month
• DDoS protection — enterprise-grade, included

Architecture

┌──────────────┐     CNAME      ┌─────────────────────┐
│ Browser      │ ──────────────▶│ Cloudflare CDN      │
│              │                │ (300+ edge PoPs)    │
└──────────────┘                └────────┬────────────┘
                                         │
                                         ▼
                                ┌─────────────────────┐
                                │ Cloudflare Pages     │
                                │ (static files)       │
                                └─────────────────────┘

  dev.docs.crawbl.com  →  crawbl-docs.pages.dev
  crawbl.com           →  crawbl-website.pages.dev

Projects

Project	Repo	Build Output	Custom Domain	Pages.dev URL
crawbl-docs	crawbl-docs	build/ (Docusaurus)	dev.docs.crawbl.com	crawbl-docs.pages.dev
crawbl-website	crawbl-website	out/ (Next.js)	crawbl.com	crawbl-website.pages.dev

Deploying

Using the crawbl CLI (recommended)

# Deploy docs
crawbl app deploy docs

# Deploy website
crawbl app deploy website

Each command:

1. Resolves the sibling repo directory (e.g., ../crawbl-docs)
2. Runs npm run build to generate static files
3. Runs wrangler pages deploy <output-dir> --project-name <project> to upload to Cloudflare
4. Creates a Git tag (semver auto-calculated from conventional commits)
5. Publishes a GitHub release with Claude-enriched notes

Manual deploy with wrangler

If you need to deploy without the CLI:

# Docs
cd crawbl-docs
npm run build
wrangler pages deploy build --project-name crawbl-docs

# Website
cd crawbl-website
npm run build
wrangler pages deploy out --project-name crawbl-website

Required environment variables

The wrangler CLI needs these env vars (already in crawbl-backend/.env):

Variable	Description
CLOUDFLARE_API_TOKEN	Cloudflare API token with Pages permissions
CLOUDFLARE_ACCOUNT_ID	Cloudflare account ID: 27414702a30e3e8ce6d2054aeecef638

Required tools

• npm — builds the static site
• wrangler — Cloudflare's CLI for Pages deployments (npm install -g wrangler)
• gh — GitHub CLI for creating releases
• claude — Claude CLI for enriched release notes

API Token Permissions

The Cloudflare API token needs these permission scopes:

Scope	Permission	Purpose
Account > Cloudflare Pages	Edit	Create projects, deploy, manage domains
Account > Access: Apps and Policies	Edit	Manage Cloudflare Access (auth for /internal)
Zone > DNS	Edit	Update DNS records

To create or update the token:

1. Go to https://dash.cloudflare.com/27414702a30e3e8ce6d2054aeecef638/api-tokens
2. Click "Create Token" or edit the existing token
3. Add the three permission scopes above
4. Set Zone to crawbl.com and Account to your account

DNS Configuration

Both domains use CNAME records pointing to Cloudflare Pages, proxied through Cloudflare:

Record	Type	Name	Content	Proxied
docs	CNAME	dev.docs	crawbl-docs.pages.dev	Yes
website	CNAME	@ (root)	crawbl-website.pages.dev	Yes

Cloudflare handles CNAME flattening for the root domain automatically.

к сведению

The old A records pointing to the DigitalOcean load balancer (68.183.241.139) have been removed. Other services (API, ArgoCD, pgweb, etc.) still use A records to the LB.

Access Control (Internal Docs)

Internal documentation at /internal/* is protected by Cloudflare Access:

• Team members authenticate via email OTP (one-time password)
• No passwords to manage — Cloudflare sends a code to your email
• Public docs at / remain accessible to everyone
• The *.pages.dev URL bypasses Access (use custom domain for production)

How it works

1. User visits dev.docs.crawbl.com/internal/
2. Cloudflare Access intercepts the request
3. User enters their email → receives a one-time code
4. After authentication, a session cookie is set (configurable expiry)
5. Subsequent requests pass through until the session expires

Managing Access

Access policies are managed in the Cloudflare Zero Trust dashboard: https://one.dash.cloudflare.com/

To add a new team member: add their email domain or specific email to the Access policy.

Troubleshooting

Deploy fails with "Authentication error"

The API token lacks Cloudflare Pages permissions. Update the token at: https://dash.cloudflare.com/27414702a30e3e8ce6d2054aeecef638/api-tokens

Custom domain shows "pending"

SSL certificate provisioning takes 1-5 minutes after adding a custom domain. Check status:

curl -s "https://api.cloudflare.com/client/v4/accounts/27414702a30e3e8ce6d2054aeecef638/pages/projects/crawbl-docs/domains" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" | python3 -m json.tool

Site shows old content after deploy

Cloudflare caches aggressively. Purge the cache:

curl -X POST "https://api.cloudflare.com/client/v4/zones/a6e53522f634b470a7d5ca3fb0928934/purge_cache" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"purge_everything":true}'

wrangler not found

Install it globally:

npm install -g wrangler