Add an API Endpoint

Follow these 9 steps in order to add a new API endpoint to the Crawbl orchestrator. Each step builds on the previous one.

Prerequisites

Local dev environment running (./crawbl dev start, or ./crawbl dev start --database-only plus ./crawbl platform orchestrator)
Familiarity with the layered architecture (server -> service -> repo -> database)

Step 1

Define Domain Types

Open internal/orchestrator/types.go and add your new structs, interfaces, or constants.

// internal/orchestrator/types.go

type Integration struct {
    ID          string    `json:"id"`
    WorkspaceID string    `json:"workspace_id"`
    Provider    string    `json:"provider"`
    Status      string    `json:"status"`
    CreatedAt   time.Time `json:"created_at"`
}

This file is the central contract. Every layer imports from here.

Step 2

Write a Migration

Create a new migration file in migrations/orchestrator/. See Add a Database Migration for naming conventions and details.

-- migrations/orchestrator/000004_add_integrations.up.sql

CREATE TABLE integrations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id UUID NOT NULL REFERENCES workspaces(id),
    provider TEXT NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Run the migration locally:

./crawbl dev migrate

Step 3

Add a Repository

Create a new sub-package under internal/orchestrator/repo/:

repo/integrationrepo/
├── integrationrepo.go    # Implementation
└── types.go              # Repo-specific interfaces (optional)

The repo receives a database.SessionRunner so it works inside or outside transactions:

// internal/orchestrator/repo/integrationrepo/integrationrepo.go

type repo struct{}

func New() *repo { return &repo{} }

func (r *repo) Create(ctx context.Context, runner database.SessionRunner, integration *orchestrator.Integration) error {
    _, err := runner.InsertInto("integrations").
        Columns("workspace_id", "provider", "status").
        Record(integration).
        ExecContext(ctx)
    return err
}

Add the interface to repo/types.go:

type IntegrationRepo interface {
    Create(ctx context.Context, runner database.SessionRunner, integration *orchestrator.Integration) error
}

Step 4

Add Service Logic

Create a service under internal/orchestrator/service/ with typed option structs:

// internal/orchestrator/service/integrationservice/integrationservice.go

type CreateIntegrationOpts struct {
    Session     *dbr.Session
    WorkspaceID string
    Provider    string
}

func (s *service) CreateIntegration(ctx context.Context, opts *CreateIntegrationOpts) (*orchestrator.Integration, *merrors.Error) {
    // Business validation
    if opts.Provider == "" {
        return nil, errors.NewBusinessError("provider is required", errors.ErrCodeINT0001)
    }
    // Call repo
    integration := &orchestrator.Integration{
        WorkspaceID: opts.WorkspaceID,
        Provider:    opts.Provider,
        Status:      "pending",
    }
    if err := s.integrationRepo.Create(ctx, opts.Session, integration); err != nil {
        return nil, errors.NewServerError(err)
    }
    return integration, nil
}

Add the service interface to service/types.go.

Step 5

Add an HTTP Handler

Create a handler file in internal/orchestrator/server/. Handlers parse requests, call services, and write responses — no business logic:

// internal/orchestrator/server/integrations.go

func (s *Server) handleCreateIntegration(w http.ResponseWriter, r *http.Request) {
    session := s.db.NewSession(nil)
    principal := httpserver.PrincipalFromContext(r.Context())

    var req CreateIntegrationRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        httpserver.WriteErrorResponse(w, errors.NewBusinessError("invalid request body", errors.ErrCodeBAD0001))
        return
    }

    result, merr := s.integrationService.CreateIntegration(r.Context(), &integrationservice.CreateIntegrationOpts{
        Session:     session,
        WorkspaceID: req.WorkspaceID,
        Provider:    req.Provider,
    })
    if merr != nil {
        httpserver.WriteErrorResponse(w, merr)
        return
    }

    httpserver.WriteSuccessResponse(w, http.StatusCreated, result)
}

Step 6

Wire the Route

r.Route("/v1", func(r chi.Router) {
    r.Group(func(r chi.Router) {
        r.Use(httpserver.AuthMiddleware(s.httpMiddleware, s.logger))
        r.Post("/integrations/connect", s.handleCreateIntegration)
    })
})

Step 7

Update the API Contract

Add the new endpoint to API Endpoints with request/response schemas, status codes, and example curl commands.

Step 8

Write Tests

Write unit tests for the service logic with mocked repos, then add an e2e scenario:

# Run unit tests
./crawbl test unit

# Run a specific e2e test
./crawbl test e2e --base-url http://localhost:7171 -v

See Writing E2E Scenarios for details on Gherkin feature files.

Step 9

Deploy

Push to main. GitHub Actions CI handles the rest:

git add .
git commit -m "feat: add integrations endpoint"
git push origin main

The pipeline builds the image, pushes to DOCR, updates crawbl-argocd-apps, and ArgoCD syncs automatically. Total time: approximately 5 minutes. See CI/CD Pipeline for details.

Checklist Summary

Step	File/Location	Action
1	`internal/orchestrator/types.go`	Define domain types
2	`migrations/orchestrator/`	Write SQL migration
3	`internal/orchestrator/repo/{entity}repo/`	Implement persistence
4	`internal/orchestrator/service/{domain}service/`	Business logic + opts struct
5	`internal/orchestrator/server/`	HTTP handler
6	Server router setup	Wire chi route
7	`docs/reference/api/endpoints.md`	Document the endpoint
8	`internal/testsuite/e2e/` + `test-features/` + unit tests	Test coverage
9	Push to `main`	CI/CD deploys

What's next: Add a Database Migration

Prerequisites​