Skip to main content

Why ko for Container Builds

Crawbl uses ko to build container images for the platform and agent-runtime binaries. The auth-filter still uses Docker because it compiles Go to WASM via TinyGo, which ko does not support.

This page explains what ko is, why we switched from Docker, and what the real-world impact is.

What is ko?​

ko is an open-source container image builder designed specifically for Go applications. It is a CNCF Sandbox project trusted by major projects including Knative, Tekton, Karpenter, and Sigstore.

Instead of writing a Dockerfile, you point ko at a Go main package:

ko build ./cmd/crawbl --bare --push --tags v1.0.0

ko does the following:

  1. Runs go build on your machine (using your local Go toolchain and module cache)
  2. Layers the resulting binary onto a minimal base image (distroless/static)
  3. Pushes the OCI image directly to your container registry

No Docker daemon required. No Dockerfile to maintain.

Why we switched​

1. Build speed​

Docker multi-stage builds download the Go toolchain image, copy your entire source tree into the build context, and compile from scratch inside the container. ko skips all of that — it runs go build locally and reuses your Go module and build caches.

MetricDocker (multi-stage)ko
Cold build (no cache)60-90 seconds15-25 seconds
Warm build (cached modules)30-45 seconds5-10 seconds
What gets sent to registryFull build context + layersBinary layer only

These numbers come from our own platform image builds. The Docker build included golang:1.25 as the builder stage, copied the full vendor directory, and compiled inside the container. ko uses the local Go cache and only pushes the final binary layer.

Why this matters: Every crawbl app deploy platform and crawbl app deploy agent-runtime is faster. In CI, the Go build cache is restored from GitHub Actions cache, making warm builds the common case.

2. No Docker daemon dependency​

Docker builds require a running Docker daemon. In CI, this means either Docker-in-Docker (DinD) or a privileged runner. Both add complexity, security surface, and startup time.

ko needs only the ko binary and a Go toolchain. CI runners do not need Docker installed for Go image builds (we still need Docker for the auth-filter WASM build, but that is one image out of three).

Real impact: Simpler CI runners, no DinD security concerns for Go builds, faster job startup.

3. No Dockerfile maintenance​

Before ko, we maintained three Dockerfiles for Go binaries:

  • platform-full.dockerfile (multi-stage, full build)
  • platform.dockerfile (CI mode, pre-built binary)
  • agent-runtime.dockerfile (multi-stage, distroless)

Each Dockerfile had to:

  • Pin the Go version (and keep it in sync with go.mod)
  • Copy vendor patches and apply them at build time
  • Set the correct build flags (CGO_ENABLED=0, -trimpath, -ldflags)
  • Match the base image with the correct user ID and permissions

With ko, all of this is a single .ko.yaml file (25 lines) at the repo root. Build flags, base image, and platform targeting are configured once.

Real impact: Three Dockerfiles deleted. One .ko.yaml file added. Build configuration is centralized and version-controlled.

4. Reproducible images​

ko produces images where the binary layer is deterministic given the same source code and Go version. Docker layer caching can produce different layers depending on the build host's cache state, timestamps, and file ordering.

5. Automatic SBOM generation​

ko generates a Software Bill of Materials (SBOM) for every image automatically. This is useful for security audits and supply chain compliance without any additional tooling.

6. Multi-platform builds are trivial​

Adding linux/arm64 support is one line in .ko.yaml:

defaultPlatforms:
  - linux/amd64
  - linux/arm64

No docker buildx cross-compilation setup required.

What ko does NOT do​

  • Non-Go builds: ko only works with go build. The auth-filter compiles Go to WASM via TinyGo — ko cannot do this. That image still uses a Dockerfile.
  • System dependencies: If your binary needs C libraries (CGO), OS packages, or runtime files beyond the binary itself, ko is not the right tool. Our Go binaries are all CGO_ENABLED=0 with go:embed for assets, so this is not an issue.
  • General-purpose container builds: ko is not a Docker replacement for non-Go projects. It solves one problem (Go container images) extremely well.

How it works in practice​

Local builds​

# Build and push the platform image
crawbl app build platform --tag v1.0.0

# Build and push the agent-runtime image
crawbl app build agent-runtime --tag v1.0.0

# Build the auth-filter image (still Docker)
crawbl app build auth-filter --tag v1.0.0

Deploy (build + push + ArgoCD update + release)​

crawbl app deploy platform           # auto-semver tag
crawbl app deploy agent-runtime      # auto-semver tag
crawbl app deploy auth-filter        # auto-semver tag (Docker)

CI​

The GitHub Actions workflow (reusable-build.yml) uses ko-build/setup-ko@v0.8 to install ko, then builds platform and agent-runtime images. Auth-filter still uses docker/build-push-action.

Configuration​

All ko settings live in .ko.yaml at the repo root:

defaultBaseImage: gcr.io/distroless/static-debian12:nonroot
defaultPlatforms:
  - linux/amd64
builds:
  - id: platform
    main: ./cmd/crawbl
    env:
      - CGO_ENABLED=0
    flags:
      - -trimpath
      - -mod=vendor
    ldflags:
      - -s -w
      - -X main.version={{ .Env.KO_BUILD_VERSION }}
  - id: agent-runtime
    main: ./cmd/crawbl-agent-runtime
    env:
      - CGO_ENABLED=0
    flags:
      - -trimpath
      - -mod=vendor
    ldflags:
      - -s -w

Key points:

  • Base image: distroless/static-debian12:nonroot — same base we used in Dockerfiles. Runs as uid 65532, no shell, no package manager.
  • -mod=vendor: Uses vendored dependencies, same as before.
  • -trimpath and -s -w: Stripped symbols and reproducible paths, same as before.
  • KO_BUILD_VERSION: Template variable injected at build time for the platform version string.

Further reading​