Skip to main content

Migrations

Database migrations for the orchestrator schema.

Current Migration Files​

Postgres (migrations/orchestrator/)​

000001_create_all.up.sql          — Core tables: users, preferences, push_tokens, workspaces,
                                    agents, tools, agent_settings, agent_prompts, conversations,
                                    agent_history, messages, mcp_audit_logs, integration_connections
000002_multi_agent.up.sql         — Multi-agent: agent_delegations, agent_messages, artifacts,
                                    artifact_versions, artifact_reviews, workflow_definitions,
                                    workflow_executions, workflow_step_executions, agent_triggers,
                                    agent_trigger_executions, memory_triggers
000003_seed_catalogs.up.sql       — Catalog tables: models, tool_categories,
                                    integration_categories, integration_providers
000004_agent_memories.up.sql      — [retired] Agent runtime durable memory: agent_memories
                                    (superseded by the memory palace; dropped in 000012)
000005_memory_palace.up.sql       — Memory Palace: memory_drawers, memory_entities,
                                    memory_triples, memory_identities
000006_token_usage.up.sql         — Billing/usage: model_pricing, usage_plans, usage_quotas,
                                    usage_counters, agent_usage_counters
000007_memory_drawer_pipeline.up.sql
                                  — MemPalace pipeline columns: state, summary, added_by_agent,
                                    last_accessed_at, access_count, superseded_by, cluster_id,
                                    retry_count. Adds embedding column to memory_entities.
000008_ivfflat_index.up.sql       — IVFFlat vector index on memory_drawers.embedding
                                    (HNSW alternative for non-AVX2 CPUs)

ClickHouse (migrations/clickhouse/)​

001_llm_usage.sql                 — llm_usage table, llm_usage_daily rollup,
                                    llm_usage_daily_mv materialized view

Seed Data (migrations/orchestrator/seed/)​

tools.json                        — Built-in tool definitions
models.json                       — Available LLM model catalog
usage_plans.json                  — Plan definitions (free: 500K, pro: 10M tokens/month)

Running Migrations​

Direct (Repo-Local Binary)​

./crawbl platform orchestrator migrate

Against Remote Database​

kubectl port-forward -n backend svc/backend-postgresql 5432:5432
# In another terminal:
./crawbl platform orchestrator migrate

Creating a New Migration​

1
Step 1

Create the file

Add the next sequence number under migrations/orchestrator/.

touch migrations/orchestrator/000007_<description>.up.sql
2
Step 2

Write idempotent SQL

Use IF NOT EXISTS and IF EXISTS where the database allows it.

3
Step 3

Test locally

Port-forward Postgres from the dev cluster, run migrations, then run the unit tests.

kubectl port-forward -n backend svc/backend-postgresql 5432:5432
./crawbl platform orchestrator migrate
./crawbl test unit          # Verify nothing breaks

Conventions​

ConventionDetail
NamingNNNNNN_<description>.up.sql (zero-padded 6-digit sequence)
DirectionUp-only migrations (no .down.sql files)
SchemaAll DDL targets the orchestrator schema
IdempotencyUse IF NOT EXISTS / IF EXISTS guards
IDsUUIDs generated in application code, not DEFAULT gen_random_uuid()
TimestampsAlways TIMESTAMPTZ, never TIMESTAMP