Optimize Docker Build Cache

Cut Docker build times through effective layer caching and build optimization.

When Use

• Docker builds slow due to repeated package installations
• Rebuilds reinstall all deps on every code change
• Image sizes unnecessarily large
• CI/CD pipeline builds are bottleneck

Inputs

• Required: Existing Dockerfile to optimize
• Optional: Target build time improvement
• Optional: Target image size reduction

Steps

Step 1: Order Layers by Change Frequency

Place least-changing layers first:

# 1. Base image (rarely changes)
FROM rocker/r-ver:4.5.0

# 2. System dependencies (change occasionally)
RUN apt-get update && apt-get install -y \
    libcurl4-openssl-dev \
    libssl-dev \
    && rm -rf /var/lib/apt/lists/*

# 3. Dependency files only (change when deps change)
COPY renv.lock renv.lock
COPY renv/activate.R renv/activate.R
RUN R -e "renv::restore()"

# 4. Source code (changes frequently)
COPY . .

Key principle: Docker caches each layer. When layer changes, all subsequent layers rebuilt. Dependency installation should come before source code copy.

Got: Dockerfile layers ordered from least-changing (base image, system deps) to most-changing (source code), with dependency lockfiles copied before full source.

If fail: Builds still reinstall deps on every code change? Verify COPY . . comes after dependency installation RUN command, not before.

Step 2: Separate Dependency Installation from Code

Bad (rebuilds packages on every code change):

COPY . .
RUN R -e "renv::restore()"

Good (only rebuilds packages when lockfile changes):

COPY renv.lock renv.lock
RUN R -e "renv::restore()"
COPY . .

Same pattern for Node.js:

COPY package.json package-lock.json ./
RUN npm ci
COPY . .

Got: Dependency lockfile (renv.lock, package-lock.json, requirements.txt) copied and installed in separate layer before full source COPY . ..

If fail: Lockfile copy fails? Ensure file exists in build context, not excluded by .dockerignore.

Step 3: Use Multi-Stage Builds

Separate build dependencies from runtime:

# Build stage - includes dev tools
FROM rocker/r-ver:4.5.0 AS builder
RUN apt-get update && apt-get install -y \
    libcurl4-openssl-dev libssl-dev build-essential
COPY renv.lock .
RUN R -e "install.packages('renv'); renv::restore()"

# Runtime stage - minimal image
FROM rocker/r-ver:4.5.0
RUN apt-get update && apt-get install -y \
    libcurl4 libssl3 \
    && rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/local/lib/R/site-library /usr/local/lib/R/site-library
COPY . /app
WORKDIR /app
CMD ["Rscript", "main.R"]

Got: Dockerfile has builder stage with dev tools and runtime stage with only production deps. Final image significantly smaller than single-stage build.

If fail: COPY --from=builder fails to find libraries? Verify install path matches between stages. Use docker build --target builder . to debug build stage independently.

Step 4: Combine RUN Commands

Each RUN creates layer. Combine related commands:

Bad (3 layers, apt cache persists):

RUN apt-get update
RUN apt-get install -y curl git
RUN rm -rf /var/lib/apt/lists/*

Good (1 layer, clean cache):

RUN apt-get update && apt-get install -y \
    curl \
    git \
    && rm -rf /var/lib/apt/lists/*

Got: Related apt-get or package install commands combined into single RUN instructions, each ending with cache cleanup (rm -rf /var/lib/apt/lists/*).

If fail: Combined RUN fails midway? Temporarily split to identify failing command, recombine after fixing.

Step 5: Use .dockerignore

Prevent unnecessary files from entering build context:

.git
.Rproj.user
.Rhistory
.RData
renv/library
renv/cache
node_modules
docs/
*.tar.gz
.env

Got: .dockerignore exists in project root excluding .git, node_modules, renv/library, build artifacts, environment files. Build context size noticeably smaller.

If fail: Needed files missing in container? Check .dockerignore for overly broad patterns. Use docker build verbose output to verify which files sent to daemon.

Step 6: Enable BuildKit

DOCKER_BUILDKIT=1 docker build -t myimage .

Or in docker-compose.yml:

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile

With COMPOSE_DOCKER_CLI_BUILD=1 and DOCKER_BUILDKIT=1 environment variables.

BuildKit enables:

• Parallel stage builds
• Better cache management
• --mount=type=cache for persistent package caches

Got: Builds run with BuildKit enabled (indicated by #1 [internal] load build definition style output). Multi-stage builds execute stages in parallel where possible.

If fail: BuildKit not active? Verify env vars exported before build command. On older Docker versions, upgrade Docker Engine to 18.09+ for BuildKit support.

Step 7: Use Cache Mounts for Package Managers

# R packages with persistent cache
RUN --mount=type=cache,target=/usr/local/lib/R/site-library \
    R -e "install.packages('dplyr')"

# npm with persistent cache
RUN --mount=type=cache,target=/root/.npm \
    npm ci

Got: Subsequent builds reuse cached packages from mount, dramatically reducing install times even when layer invalidated. Cache persists across builds.

If fail: --mount=type=cache not recognized? Ensure BuildKit enabled (DOCKER_BUILDKIT=1). Syntax requires BuildKit, not supported by legacy builder.

Checks

• Rebuilds after code-only changes significantly faster
• Dependency installation layer cached when lockfile unchanged
• .dockerignore excludes unnecessary files
• Image size reduced compared to unoptimized build
• Multi-stage build (if used) separates build and runtime deps

Pitfalls

• Copying all files before installing deps: Invalidates dependency cache on every code change
• Forgetting .dockerignore: Large build contexts slow every build
• Too many layers: Each RUN, COPY, ADD creates layer. Combine where logical.
• Not cleaning apt cache: Always end apt-get installs with && rm -rf /var/lib/apt/lists/*
• Platform-specific caches: Cache layers platform-specific. CI runners may not benefit from local caches.

See Also

• create-r-dockerfile - initial Dockerfile creation
• setup-docker-compose - compose build configuration
• containerize-mcp-server - apply optimizations to MCP server builds