Binary Substitutes and Caches

Binary Substitutes and Caches are mechanisms used in the Nix package management system to optimize and streamline the process of building and distributing software packages. Instead of always building packages from source, which can be time-consuming and resource-intensive, Nix can use pre-built binary substitutes obtained from caches. These substitutes are binary artifacts that precisely correspond to the build outputs of given package derivations, allowing users to quickly download and deploy ready-to-use software.

Binary Substitutes

Definition and Purpose

Binary substitutes are exact pre-built binaries of package derivations created by the Nix build system. Each substitute is an artifact representing the output of a package build step, identified by a cryptographic hash of all inputs involved in the derivation. This ensures that the binary is reproducible and corresponds exactly to the requested build.

The main purpose of binary substitutes is to avoid redundant compilation or building, enabling immediate installation of software without the overhead of building from source. This accelerates deployment, saves computational resources, and improves user experience.

How Binary Substitutes Work

When a user requests a package installation, Nix computes the derivation’s unique hash. Instead of building the package locally, Nix queries configured binary caches to check if a substitute matching that hash is available. If found, the binary substitute is downloaded and stored in the Nix store, bypassing the build.

If no substitute is found, Nix falls back to building the package locally and can optionally upload the resulting binary to a cache for future reuse.

Security and Integrity

Because binary substitutes are identified by cryptographic hashes of all inputs (including source code, dependencies, and build instructions), their integrity is guaranteed. Users can trust that a downloaded substitute corresponds exactly to the derivation they requested, preventing tampering or mismatches.

Caches

Types of Caches

Nix supports several types of caches:

• Public Caches: Hosted by third parties, such as the official Nix binary cache (cache.nixos.org), providing pre-built binaries for a wide range of packages and platforms.
• Private Caches: Organizations or users can set up their own binary caches to share custom-built packages internally or to reduce build times on their infrastructure.
• Local Caches: Nix stores downloaded substitutes locally in the Nix store directory, avoiding repeated downloads.

Cache Structure and Protocols

Binary caches are HTTP(S) servers serving substitutes indexed by their cryptographic hashes. They respond to queries for specific derivations and provide the corresponding binary outputs as compressed archives.

Nix uses a well-defined protocol to communicate with caches, which includes:

• Fetching substitute files via GET requests.
• Authenticating and signing cache content optionally for security.
• Indexing substitutes by their content-addressed hashes.

Cache Configuration

Users configure caches via the Nix configuration files, specifying URLs and priority orders. Multiple caches can be listed, and Nix attempts to fetch substitutes in the order configured until a match is found.

Example configuration snippet:

substituters = [
  "https://cache.nixos.org/"
  "https://my-private-cache.example.org/"
];

trusted-public-keys = [
  "cache.nixos.org-1:abcdef1234567890..."
  "my-private-cache.example.org-1:123456abcdef7890..."
];

Benefits of Using Caches

• Speed: Avoid expensive builds by downloading ready-to-use binaries.
• Efficiency: Reduce CPU and energy usage on local machines or build servers.
• Reproducibility: Guarantees that binaries match the exact build inputs.
• Collaboration: Share builds across teams or organizations via private caches.

Integration in Nix Package Management Workflow

Build vs. Download Decision

When installing or rebuilding packages, Nix first tries to download substitutes from configured caches, falling back to local build only if necessary. This approach maximizes efficiency and user convenience.

Uploading Substitutes

Build machines or CI systems can upload built substitutes to caches, increasing cache coverage and availability for downstream users. This upload process includes signing the artifacts to maintain trust.

Cache Expiration and Garbage Collection

Nix manages local cache storage carefully. It performs garbage collection to remove unused substitutes and keeps the cache size manageable without compromising availability of frequently used binaries.

Summary of Key Concepts

Binary substitutes and caches are fundamental to Nix’s ability to provide fast, reliable, and reproducible package management by leveraging pre-built outputs, reducing build times, and facilitating distribution of software artifacts across diverse systems.