How Bazel built its CI system on top of Buildkite

Philipp Wollermann, a Staff Software Engineer at Google, shared his insights on Bazel CI and Buildkite during UnblockConf '21. Bazel is an open-source build system created by Google to solve specific issues the company faced. It can provide considerably faster build times by recompiling only the files that need to be recompiled and skipping re-running tests that haven't changed. Bazel CI, built on top of Buildkite, is used for testing and releasing Bazel and its ecosystem. Philipp discussed common problems faced while implementing Bazel CI and suggested solutions such as parallelisation and target sharding to reduce build times, using path-based triggers to identify affected targets in large monorepositories, splitting pipelines into multiple ones for better maintainability, and utilizing remote caching for sharing cache with multiple workers or stateless workers. He also highlighted the benefits of Bazel's remote execution feature, which allows distributing build and test actions across multiple machines, increasing parallelism and providing an automatic, persistent, shared cache. Philipp introduced the bazelci-agent tool that listens to a Build Event Protocol (BEP) stream and identifies failing test logs in real time, uploading them continuously for faster feedback. Lastly, he touched upon Bazel's built-in support for detecting flaky tests and automatic retries, which can help improve the reliability of test results.