How do you speed up slow CI builds?

A lot of contributing factors make up a CI build and any of them can cause a slow down. Here are a few of the suspects that should be checked.

1. CPU usage: is the build agent maxing out the CPU? (cpu load)
2. Memory usage: Excessive swapping of disks or a large amount of memory usage can slow down builds. (free memory)
3. Build Agent availability: In a large estate, maybe the build is waiting a while (job queue time) for a free build agent to become available.
4. Lack of caching: some operations do take time, but we can cache the result. Examples are python venv folders, NPM and node_modules, go-lang src folders or perhaps docker image layers. Each can be stored after the resources are downloaded. Re-hydrating the resources from a known hash/config is faster than downloading all the dependencies.

By checking timestamps of the logging during the build, you can build a profile of where the time is spent and focus your attention on that step.

Make it faster, one step at a time.

The single highest-leverage fix is usually dependency or build caching (for example GitHub Actions' actions/cache, or a Docker layer cache) — teams that add it typically cut build time substantially simply by not re-downloading or re-compiling unchanged dependencies on every run.