Build CI/CD Pipeline

Design and implement production-grade continuous integration and deployment pipelines with GitHub Actions.

When Use

• Setting up automated testing and deployment for new project
• Migrating from Jenkins, Travis CI, or CircleCI to GitHub Actions
• Implementing matrix builds across multiple platforms or language versions
• Adding build caching to speed up CI/CD execution time
• Creating multi-stage pipelines with environment-specific deployments
• Implementing security scanning and code quality gates

Inputs

• Required: Repository with code to test/build/deploy
• Required: GitHub Actions workflow directory (.github/workflows/)
• Optional: Secrets for deployment targets (AWS, Azure, Docker registries)
• Optional: Self-hosted runner config for specialized builds
• Optional: Branch protection rules and required status checks

Steps

Step 1: Create Base Workflow Structure

Create .github/workflows/ci.yml with trigger config and basic job structure.

name: CI Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]
  workflow_dispatch:  # Manual trigger

env:
  NODE_VERSION: '18'
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  lint:
    name: Lint Code
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run ESLint
        run: npm run lint

      - name: Check formatting
        run: npm run format:check

Got: Workflow file created, proper YAML syntax, triggers configured, basic lint job defined.

If fail: Validate YAML with yamllint .github/workflows/ci.yml. Check indentation (spaces, not tabs). Verify action versions current via GitHub Marketplace.

Step 2: Implement Matrix Build Strategy

Add matrix builds to test across multiple platforms, language versions, configurations.

  test:
    name: Test (${{ matrix.os }}, Node ${{ matrix.node }})
    runs-on: ${{ matrix.os }}
    needs: lint
    strategy:
      fail-fast: false  # Continue testing other matrix combinations on failure
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node: ['16', '18', '20']
        exclude:
          - os: macos-latest
            node: '16'  # Skip old Node on macOS

    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js ${{ matrix.node }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run tests with coverage
        run: npm run test:coverage

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        if: matrix.os == 'ubuntu-latest' && matrix.node == '18'
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: ./coverage/lcov.info
          fail_ci_if_error: true

Got: Matrix generates 8 parallel jobs (3 OS × 3 Node versions - 1 exclusion). All tests pass across platforms. Coverage uploads from single canonical job.

If fail: Matrix syntax errors? Verify indentation and array notation. For flaky tests, add retry logic with uses: nick-invision/retry@v2. For platform-specific failures, add OS conditionals or expand exclusions.

Step 3: Configure Dependency Caching and Artifact Management

Optimize build speed with intelligent caching, preserve build artifacts.

  build:
    name: Build Application
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Cache build output
        uses: actions/cache@v3
        with:
          path: |
            .next/cache
            dist/
            build/
          key: ${{ runner.os }}-build-${{ hashFiles('**/package-lock.json') }}-${{ hashFiles('**/*.ts', '**/*.tsx') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ hashFiles('**/package-lock.json') }}-
            ${{ runner.os }}-build-

      - name: Install dependencies
        run: npm ci

      - name: Build application
        run: npm run build
        env:
          NODE_ENV: production

      - name: Upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: dist-${{ github.sha }}
          path: |
            dist/
            build/
          retention-days: 7
          if-no-files-found: error

Got: First run downloads dependencies (slow), subsequent runs restore from cache (fast). Build artifacts upload with unique SHA-based naming.

If fail: Cache misses frequently? Verify cache key includes all relevant file hashes. Upload failures? Check path exists, glob patterns match actual build output. Verify retention-days meets organizational policies.

Step 4: Implement Security Scanning and Quality Gates

Add security vulnerability scanning and code quality enforcement.

  security:
    name: Security Scan
    runs-on: ubuntu-latest
    needs: lint
    permissions:
      security-events: write  # Required for uploading SARIF results
    steps:
      - uses: actions/checkout@v4

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        if: always()  # Upload even if scan finds vulnerabilities
        with:
          sarif_file: 'trivy-results.sarif'

      - name: Dependency audit
        run: npm audit --audit-level=high
        continue-on-error: true  # Don't fail build, but show warnings

      - name: Check for leaked secrets
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD

Got: Security scans complete, results upload to GitHub Security tab. Critical vulnerabilities block merge if branch protection configured. No secrets detected in commits.

If fail: False positives? Create .trivyignore with CVE IDs and justifications. Audit failures? Review npm audit fix suggestions. Secret detection false positives? Add patterns to .trufflehog.yml exclude list.

Step 5: Configure Environment-Specific Deployments

Set up deployment stages with environment protection rules and approval gates.

  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: [build, security]
    if: github.ref == 'refs/heads/develop'
    environment:
      name: staging
      url: https://staging.example.com
    steps:
      - name: Download build artifacts
        uses: actions/download-artifact@v3
        with:
          name: dist-${{ github.sha }}
          path: ./dist

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_STAGING }}
          aws-region: us-east-1

      - name: Deploy to S3
        run: |
          aws s3 sync ./dist s3://${{ secrets.S3_BUCKET_STAGING }} --delete
          aws cloudfront create-invalidation --distribution-id ${{ secrets.CF_DIST_STAGING }} --paths "/*"

  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: [build, security]
    if: github.ref == 'refs/heads/main'
    environment:
      name: production
      url: https://example.com
    steps:
      - name: Download build artifacts
        uses: actions/download-artifact@v3
        with:
          name: dist-${{ github.sha }}
          path: ./dist

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_PRODUCTION }}
          aws-region: us-east-1

      - name: Deploy to S3 with blue-green
        run: |
          # Deploy to new version
          aws s3 sync ./dist s3://${{ secrets.S3_BUCKET_PRODUCTION }}/releases/${{ github.sha }} --delete

          # Update symlink to new version
          aws s3 cp s3://${{ secrets.S3_BUCKET_PRODUCTION }}/releases/${{ github.sha }} s3://${{ secrets.S3_BUCKET_PRODUCTION }}/current --recursive

          # Invalidate CloudFront
          aws cloudfront create-invalidation --distribution-id ${{ secrets.CF_DIST_PRODUCTION }} --paths "/*"

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v1
        if: startsWith(github.ref, 'refs/tags/')
        with:
          files: ./dist/**/*
          generate_release_notes: true

Got: Staging deploys automatically on develop branch. Production requires manual approval (from GitHub Environment settings). CloudFront invalidation clears CDN cache. Release created for tagged commits.

If fail: AWS credential errors? Verify OIDC trust relationship allows role-to-assume. S3 sync failures? Check bucket policies and IAM permissions. Environment approval issues? Verify protection rules in Settings > Environments.

Step 6: Add Notification and Monitoring Integration

Integrate Slack notifications, deployment tracking, performance monitoring.

  notify:
    name: Notify Results
    runs-on: ubuntu-latest
    needs: [deploy-staging, deploy-production]
    if: always()  # Run even if previous jobs fail
    steps:
      - name: Check job status
        id: status
        run: |
          if [ "${{ needs.deploy-production.result }}" == "success" ]; then
            echo "status=success" >> $GITHUB_OUTPUT
            echo "color=#00FF00" >> $GITHUB_OUTPUT
          else
            echo "status=failure" >> $GITHUB_OUTPUT
            echo "color=#FF0000" >> $GITHUB_OUTPUT
          fi

      - name: Send Slack notification
        uses: slackapi/[email protected]
        with:
          payload: |
            {
              "text": "Deployment ${{ steps.status.outputs.status }}",
              "blocks": [
                {
                  "type": "header",
                  "text": {
                    "type": "plain_text",
                    "text": "🚀 Deployment Status: ${{ steps.status.outputs.status }}"
                  }
                },
                {
                  "type": "section",
                  "fields": [
                    {"type": "mrkdwn", "text": "*Repository:*\n${{ github.repository }}"},
                    {"type": "mrkdwn", "text": "*Branch:*\n${{ github.ref_name }}"},
                    {"type": "mrkdwn", "text": "*Commit:*\n${{ github.sha }}"},
                    {"type": "mrkdwn", "text": "*Actor:*\n${{ github.actor }}"}
                  ]
                },
                {
                  "type": "actions",
                  "elements": [
                    {
                      "type": "button",
                      "text": {"type": "plain_text", "text": "View Workflow"},
                      "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
                    }
                  ]
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
          SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

      - name: Record deployment in Datadog
        if: steps.status.outputs.status == 'success'
        run: |
          curl -X POST "https://api.datadoghq.com/api/v1/events" \
            -H "Content-Type: application/json" \
            -H "DD-API-KEY: ${{ secrets.DD_API_KEY }}" \
            -d @- <<EOF
          {
            "title": "Deployment: ${{ github.repository }}",
            "text": "Deployed commit ${{ github.sha }} to production",
            "tags": ["env:production", "service:${{ github.event.repository.name }}"],
            "alert_type": "info"
          }
          EOF

Got: Slack receives formatted notification with deployment status, repository details, clickable workflow link. Datadog event logged for successful production deployments with appropriate tags.

If fail: Slack failures? Verify webhook URL valid, workspace allows incoming webhooks. Test with curl -X POST $SLACK_WEBHOOK_URL -d '{"text":"test"}'. Datadog failures? Verify API key has event submission permissions.

Checks

• Workflow syntax validates with yamllint or GitHub's workflow editor
• All jobs have explicit dependencies (needs:) to control execution order
• Matrix builds cover all target platforms and versions
• Caching reduces build time by >50% on subsequent runs
• Secrets stored in GitHub Secrets, never hardcoded in workflow files
• Security scans upload results to GitHub Security tab
• Environment protection rules require approval for production deployments
• Failed deployments don't leave system in inconsistent state
• Notifications reach appropriate channels (Slack, email, monitoring tools)
• Workflow completes in <10 minutes for typical changes

Pitfalls

• Cache key too broad: Using ${{ runner.os }}-build- as cache key causes false hits when dependencies change. Include hashFiles('**/package-lock.json') in key.

• Artifact name collisions: Using static artifact names like dist causes overwrites in concurrent builds. Include ${{ github.sha }} or ${{ matrix.os }}-${{ matrix.node }} in names.

• Secrets in logs: Avoid echo $SECRET or similar commands. GitHub masks registered secrets, but derived values may leak. Use ::add-mask:: for dynamic secrets.

• Insufficient permissions: Default GITHUB_TOKEN has limited permissions. Add explicit permissions: block for security events, packages, issues, etc.

• Missing if conditionals: Jobs run on all triggers unless guarded with if: github.ref == 'refs/heads/main'. Prevent accidental production deploys from PRs.

• No rollback strategy: Deployment failures leave system in broken state. Implement blue-green or canary deployments with automatic rollback on health check failures.

• Hardcoded values: Workflow contains environment-specific URLs, bucket names, API endpoints. Use environment variables and GitHub Secrets.

• No timeout limits: Jobs hang indefinitely on network issues or infinite loops. Add timeout-minutes: 15 to all jobs.

See Also

• setup-github-actions-ci - Initial GitHub Actions config for R packages and basic projects
• commit-changes - Proper Git workflow integration with CI/CD triggers
• configure-git-repository - Repository settings and branch protection rules
• setup-container-registry - Docker image builds in CI/CD pipelines
• implement-gitops-workflow - ArgoCD/Flux integration with CI/CD