Skip to content

Chapter 4 of 7

Pipeline Internals, Caching, and Configuration

Build artifacts are the output files produced by a build stage — compiled binaries, Docker images, test reports, coverage reports, or any other deliverable. They flow between stages and jobs, and the same artifact should be promoted across environments unchanged. In GitHub Actions `actions/upload-artifact` and `actions/download-artifact` move files between jobs in a run; in Jenkins `archiveArtifacts` makes a file persistent across the controller, and `stash` provides a transient, same-node handoff between stages. A build number — exposed as `github.run_number` in Actions and `BUILD_NUMBER` in Jenkins — gives every run a monotonically increasing identifier for traceability, and artifact retention policies govern how long built binaries remain downloadable. The default GitHub Actions artifact retention is 90 days, configurable per artifact. An artifact itself is distinct from a release: a release is a named, versioned collection of artifacts plus release notes, typically marked with a Git tag.

Caching is one of the highest-leverage optimizations in any pipeline. Dependency caching stores downloaded packages such as `node_modules`, `~/.m2`, or pip's cache so that they need not be re-fetched on every run. The cache key typically encodes a hash of the lockfile so that the cache invalidates automatically when dependencies change; in GitHub Actions, both `actions/cache@v4` and `actions/setup-node` with `cache: 'npm'` implement this pattern. Build caching, by contrast, stores intermediate compilation outputs — Gradle's build cache, Next.js's `.next/cache`, ccache for C/C++, or Docker layer caching — to skip work done in previous runs. Remote build caching, where the cache is shared across CI jobs and developer machines via systems like Bazel's remote cache, Nx cloud, or Turborepo's remote cache, can cut monorepo build times dramatically. The distinction between `actions/cache` and `actions/upload-artifact` is important: the former is an opaque, restored-on-hit accelerator keyed on inputs, while the latter is a named bundle that downstream consumers can explicitly download.

Containers make every step of a build reproducible. Running pipeline steps inside Docker eliminates environment drift between machines and between local development and CI. In GitHub Actions the `container:` key launches all steps inside a specified image, while Jenkins can use `agent { docker { image 'maven:3.9' } }` to achieve the same effect. For multi-platform support — shipping one image that runs on both Intel and ARM nodes — `docker buildx build --platform linux/amd64,linux/arm64 --push` produces a manifest list that the registry serves correctly per architecture. Daemonless builders such as Kaniko and Buildah are useful in locked-down environments where a Docker daemon is unavailable. Best practices for containerized builds include tagging images with the Git commit SHA (never just `latest`), using multi-stage builds for smaller final images, and pinning deployments to the image's content-addressable digest for true reproducibility. Docker layer caching in particular uses `cache-from: type=gha` to pull previous layers from the GitHub Actions cache and typically yields five-to-ten-times speedups when only the application layer changes.

Secrets need careful handling because pipelines touch dozens of credentials. GitHub Secrets encrypt values at the repository, environment, or organization level; they are automatically masked in logs and are not passed to workflows triggered from forks by default. Environments can require approvers before secrets are exposed, making them an effective approval gate. Jenkins stores secrets through the Credentials plugin across scopes of system, global, or folder; `credentials()` and `withCredentials` bind them to environment variables within a block, with masking applied. For more demanding scenarios, HashiCorp Vault offers dynamic, short-lived secrets, automatic rotation, encryption as a service, and a complete audit log of every access. AWS Secrets Manager provides equivalent functionality tightly integrated with AWS IAM, which is convenient when your infrastructure already lives in AWS. Modern pipelines increasingly use OIDC: GitHub Actions can mint a short-lived cloud IAM token for each workflow run, eliminating the need to store long-lived cloud keys at all.

Several configuration primitives apply across platforms. Environment variables pass values to pipeline steps without hardcoding, with the `env:` key in GitHub Actions and the `environment { }` block in Jenkins, often scoped to the workflow, job, or step. Notifications alert teams to pipeline results through Slack or Microsoft Teams integrations, email, or PagerDuty for production incidents — best practice is to alert on every failure and on successes only when relevant. Approval gates insert a human checkpoint before sensitive stages, implemented in GitHub Actions through environment protection rules (required reviewers, a wait timer, deployment branch restrictions, or external policy checks) and in Jenkins through the `input` step. Infrastructure as Code — Terraform, Pulumi, Ansible, CloudFormation — follows a deliberate pipeline rhythm of `fmt -check`, `validate`, `plan`, manual approval, and `apply`, with state files stored remotely and locked to prevent concurrent modifications. Shell steps benefit from `set -euo pipefail` to fail on the first error, undefined variables, or any pipeline-stage failure rather than only the last.

All chapters
  1. 1Foundations of CI/CD
  2. 2GitHub Actions in Depth
  3. 3Jenkins and Alternative CI/CD Platforms
  4. 4Pipeline Internals, Caching, and Configuration
  5. 5Testing, Code Quality, and Security Scanning
  6. 6Deployment Strategies and Progressive Delivery
  7. 7Supply Chain Security and Engineering Excellence

Drill it

Reading is not remembering. These come from the Cicd Pipelines deck:

Q

What is Continuous Integration (CI)?

Continuous Integration is a development practice where developers frequently merge code changes into a shared repository, often multiple times a day. Each merge...

Q

What is the difference between Continuous Delivery and Continuous Deployment?

Continuous Delivery ensures code is always in a deployable state but requires a manual approval step before production release.Continuous Deployment goes furthe...

Q

What are the typical stages of a CI/CD pipeline?

A standard CI/CD pipeline includes these stages:Source – code checkout from version controlBuild – compile code and resolve dependenciesTest – run unit, integra...

Q

What is a GitHub Actions workflow?

A GitHub Actions workflow is an automated process defined in a YAML file under .github/workflows/. It is triggered by events such as push, pull_request, or sche...