Optimise your Rust CI pipeline "blazingly fast"!

The presentation slide can be downloaded in here ¹

Premise

The Rust programming language has become mainstream and is getting a relatively decent adoption. For us, DevOps Engineers, we need to support Software Engineers and our organisation if they want/need to build Rust software. So, what can we do?

The Good and Bad

Let's get to know Rust a bit. It's a compiled language (like C++ and Golang) with emphasis on "Performance", "Reliability", and "Productivity". The language itself has gained quite a traction over the years, and it keeps rolling. If you're not sure, the Rust section on readytotouch.com ² lists 250+ companies that are looking for Rust experience. Is it really that good?

For Performance and Reliability, I think it's debatable and will depend on your use case of each language. If you are working in an industry where performance must be top-notch (Oil and Gas, HPC, etc) or the system must be reliable (Bank, Power Plant, Embedding, etc), then you can try to use Rust. For backend development, check this video from Anton Putra ³ (He's a great DevOps guy). However, the "Productivity" claim is the main point of this article.

If you want to believe me, I can tell that Rust have the best tooling of other programming languages. The documentation is easy to find and read, compiler errors are 90% super helpful, and the tooling they provided (cargo, clippy, etc) is really good. However, the language itself has a steep learning curve. When I started to pick up on Rust, I needed to learn (and still learning) new concepts like memory management (duh), how computer works, borrow-checker, lifetime, primitives, functional programming, and so on. If you can pass and enjoy the learning curve, the other problem is the slow compile time in Rust that impacts the Continuous Integration (CI) pipelines. Let's see how we can make our compiler work better with CI pipelines.

This blog is heavily inspired by corode.dev articles about Improving Rust CI ⁴ and Faster Rust Compile Time ⁵. Go check them out!

First Step: Optimise the Rust itself

The easiest way to speed up CI time is to tinker with Rust itself. Of course, you must optimise your codebase, but that's a different topic. If you are not familiar with Rust tooling, cargo is the Rust package manager, similar to npm in JavaScript and uv in Python. The other common tools are clippy for linting and rustfmt for formatting.

The first optimisation we can do is to use --locked when we run cargo cli. The locked flag tells cargo to skip the dependencies update phase and run the real thing (build with cargo build, test with cargo test, bench with cargo bench, etc). This flag is useful since you won't need to update the dependencies that much in the CI pipeline and instead use the lock file ⁶, Cargo.lock, as your dependencies source. It's an easy win!

Next is to disable the cargo incremental feature. Why? Incremental is useful in your local development because the Rust compiler doesn't have to compile all of the code from scratch and instead uses the saved compiled software that by default is located in the target folder incrementally ⁷. In the CI, however, we don't need this feature because we build from scratch anyway, and the incremental make target folder is bloated. So better to turn this off in the CI with CARGO_INCREMENTAL: 0 environment variable.

name: Rust Continuous Integration

on:
  ...

env:
  CARGO_INCREMENTAL: 0
  CARGO_PROFILE_DEV_DEBUG: 0
  CARGO_PROFILE_DEV_STRIP: "debuginfo"

jobs:
  unit_test:
    ...

The last one is to tune the cargo profiles. By default, there are four cargo profiles: dev, release, test, bench ⁸. The profile is automatically chosen based on which command is being run if a profile is not specified on the command line, and you can create your own profiles. By default, the dev profile enables debug and disables strip. In CI pipelines, we can turn off the debug and strip the debuginfo to reduce the target size. You can set CARGO_PROFILE_DEV_DEBUG: 0 and CARGO_PROFILE_DEV_STRIP: "debuginfo" to do that. If you are not using a debugger often (like me), you can set those profile configurations in the Cargo.toml file.

[package]
# ...

[profile.release]
strip = "debuginfo"
debug = 0

Second Step: Parallelise the test

Specifically for the testing phase, we can use cargo-nextest instead of cargo test to improve test speed 1 - 3x ⁹. The improvement comes from the design behind Nextest that enables the "parallelism" by running one process per test, while in the cargo test, it uses a shared process. The process-per-test model forms a universal protocol that everyone in the ecosystem can rely on without explicit coordination. If you are interested to learn more, check their article here ¹⁰ (Game Theory included!).

To install cargo-nextest with cargo, use the commands below

# Install cargo-nextest
cargo install cargo-nextest --locked

# Run cargo-nextest
cargo nextest run

Third Step: Cache (most of) the dependencies

Cargo downloads your dependencies locally by default in the ~/.cargo folder. However, there is such a cache in CI pipelines, right? Fortunately, we can use sccache, developed by Mozilla as a compiler cache tool. It not only supports Rust but also codebases written in C/C++ and NVIDIA CUDA.

To install sccache with cargo, use the commands below

# Install sccache
cargo install sccache --locked

# Set Rust Wrapper environment variable and build
export RUSTC_WRAPPER=$(which sccache) && cargo build --locked

# Use --timings flag to check the build time
export RUSTC_WRAPPER=$(which sccache) && cargo build --locked --timings

Or if you prefer to use Cargo.toml

[package]
# ...

[build]
rustc-wrapper = "/path/to/sccache/binary"

Disabling incremental, turning off Cargo Debug, and stripping debuginfo works well with sccache since we limited the artefacts we want to cache from the target folder.

And of course, if you store cache in disk (by default), you'll not gain any time skip. sccache provides robust storage options like S3, Redis, Github Action (Cache), and more, but be wary about the network overhead. Using distributed caching storage options requires additional latency to upload and download the cache artefacts. Do diligence by picking the right option for your use case. For me, it's sufficient to use Github Action Cache since my code is in Github ¹¹. Check their repo for more info ¹².

If you're curious about my implementation, you can check this PR ¹³ for one of my Rust projects. The performance gain is decent in my opinion. Overall CI times change per job:

Linting: 91s to 57s (37% time reduction)
Unit Test (Cargo Test): 110s to 58s (47% time reduction)
Code Coverage: 321s to 114s (64% time reduction)
Cargo Test vs Nextest in the last CI: 58s vs 55s (5% time reduction)

Well, the time improvements are not as bad as I thought, but not so "blazingly fast" either. Do I think it's a good thing to do? Yeah, especially if it's scalable. If your codebases already have a long build time (not like my project), you'll get more benefit from these optimisations.

Another thing you can do

release-plz

There is much you can do to improve your pipelines. Here is a non-exhaustive list to give you some ideas:

Use release-plz to release your Rust application ¹⁴
Tune your Rust profile ⁸
Speed up Docker build time with Cargo Chef ¹⁵
Trim your code dependencies with Cargo Machete ¹⁶
Split big codebase into smaller workspaces ¹⁷
Template for Azure pipeline ¹⁸
Template for Github Workflow ¹⁹
...and much more. Check corode.dev articles! ⁴ ⁵

Conclusion?

In the end, there is much we can do to optimise our Rust CI pipelines. The easiest step, in my opinion, is to set the --locked flag, then move to another optimisation like caching, testing, and easier release. Hopefully, after you finish this article, you'll be ready when your software engineer team asks for your help to set up their Rust CI pipeline. Happy weekend!

Tags: