Shared Packages

The internal/pkg/ directory contains reusable packages shared across the orchestrator, agent runtime, CLI, and webhook processes. Each package has a single responsibility.

---

Package Reference

Core Infrastructure

Package	Purpose	Key Exports
database	PostgreSQL connection and query building	New(), SessionRunner interface, dbr.Session wrappers
redisclient	Redis client with typed operations	New(), Get/Set, counters, hashes, sets
crawblnats	NATS JetStream client for usage events	New(), Publish(), stream lifecycle management
configenv	Env var loading with _FILE secret fallback	SecretString(), String(), Int(), Bool()

Authentication & Security

Package	Purpose	Key Exports
hmac	HMAC-SHA256 token generation and validation	GenerateToken(), ValidateToken()
firebase	Firebase Cloud Messaging (FCM) push notifications	Client, SendNotification()
grpc	gRPC server utilities and auth interceptors	NewServer(), HMAC bearer token interceptor
httpserver	HTTP header constants and auth metadata	TokenSourceFromHeaders(), header name constants

Domain Helpers

Package	Purpose	Key Exports
errors	Structured error types (business vs server)	BusinessError, ServerError, Is(), As()
pricing	LLM model pricing cache with DB refresh	Cache, CostForTokens(), periodic refresh goroutine
embed	Vector embedding interface for LLM providers	Embedder interface, Embed(text) -> []float32
realtime	Broadcaster interface for workspace events	Broadcaster, event types (message, agent, artifact, workflow, usage)
kube	Kubernetes naming constants	MaxNameLength (63), MaxWorkloadNameLength (52)

Build & Deploy Tooling

Package	Purpose	Key Exports
argocd	Image tag updater for crawbl-argocd-apps repo	UpdateImageTag(), DOCR registry helpers
release	Git tag and GitHub release creation	CreateTag(), CreateRelease(), auto-generated notes
versioning	Semver calculation from conventional commits	NextVersion(), major/minor/patch bump logic
gitutil	Git helpers (root dir, sibling repos, verification)	RootDir(), SiblingRepo(), IsClean()
cli	CLI output formatting and styling	out.Success(), out.Error(), style.Bold()

Utilities

Package	Purpose	Key Exports
fileutil	Atomic file I/O (temp + rename)	WriteAtomic()
runtime	OS signal handling for graceful shutdown	Run() — blocks until SIGINT/SIGTERM
telemetry	OpenTelemetry metrics export to VictoriaMetrics	Setup(), OTLP HTTP exporter, disabled by default
yamlvalues	Pulumi stack config file parsing	StackConfig struct for Pulumi.<env>.yaml

---

Usage Patterns

Database sessions

import "github.com/Crawbl-AI/crawbl-backend/internal/pkg/database"

// One connection per process
db, err := database.New(cfg.Postgres)

// One session per request (pass through service/repo layers)
sess := db.NewSession(nil)

Config with secret fallback

import "github.com/Crawbl-AI/crawbl-backend/internal/pkg/configenv"

// Reads CRAWBL_DB_PASSWORD or CRAWBL_DB_PASSWORD_FILE (for volume-mounted secrets)
password := configenv.SecretString("CRAWBL_DB_PASSWORD", "")

HMAC tokens

import crawblhmac "github.com/Crawbl-AI/crawbl-backend/internal/pkg/hmac"

// Generate
token := crawblhmac.GenerateToken(signingKey, userID, workspaceID)

// Validate
userID, workspaceID, err := crawblhmac.ValidateToken(signingKey, token)

Structured errors

import "github.com/Crawbl-AI/crawbl-backend/internal/pkg/errors"

// Business error (client sees it)
return errors.NewBusinessError("workspace not found", 404)

// Server error (logged, client gets generic 500)
return errors.NewServerError("redis connection failed", err)

---

What's next: See Codebase Layout for the full directory structure, or Coding Patterns for how these packages are used in practice.