Experimenting with Google Cloud Build and Bazel

I’m Christian Roggia, a Backend Engineer at Engel & Völkers Technology. At our company choosing the best tools and platforms is an ever-changing challenge and we are encouraged to experiment and share with other engineers the results we have been able to achieve.In this article we will analyze two tools I’ve been experimenting with for months and I will share my experience, and why we decided to drop one of them in favor of something else.

Google Cloud Build (GCB for short)

The first tool we will analyze is Google Cloud Build, formerly known as Google Cloud Container Builder, which is a CI platform provided by the Google Cloud Platform.

GCB is a platform with a lot of potential. It is similar to other Docker-based CI platforms like Travis, Circle or Drone CI, but unfortunately is lacking community support completely and even from the Google side it looks like there is not enough manpower to answer to customer requests.

GCB uses Docker images, called Cloud Builders, to provide tools and environments required for running specific actions, such as kubectl for deploying to Kubernetes, or go for building Go binaries.

GCB is bad for cache

There is currently a large amount of missing features in GCB and the worst one is that it does not allow us to configure zones and regions for the GCE instance where the Cloud Build is running: using remote cache (for example GCS) is nearly impossible when the location is randomly selected, latency will skyrocket and the benefits of using cache is nullified by the overhead of transferring the data across the globe. Another related issue is that we cannot mount Persistent Volumes and therefore make use of local cache. Platforms like Bazel are heavily affected by this.

GCB has a bad IAM permissions management

IAM permissions are a powerful tool provided by the Google Cloud Platform, unfortunately GCB does not make good use of them (yet). You can only set the IAM permissions per-build and not per-step, this makes it impossible to use generic service accounts and instead forces you to create a single, large service account that needs to be activated before launching the build. There is no granularity and no reuse of the same permissions. If different projects need different level of access you will end up creating dozens similar service accounts.

GCB doesn’t have enough support

From both the community and Google point of view, GCB has very little support: Github issues will stay open for months before been closed for inactivity with no response from the Google team, PRs will sometimes share the same fate. The Github projects are evolving very slowly and the little support external engineers try to provide is many times discarded without providing any feedback, this discourages developers from contributing further.

Bazel

Bazel is a build system that enables incremental builds and makes heavy use of cache. The compilation time can be dramatically reduced if Bazel is configured properly. One of our internal project has seen a decrease of compilation time from 11 minutes to roughly 1 minute, this was an impressive achievement and we are currently working on bringing Bazel to more projects. On the other side, the biggest downside of Bazel is that it is currently in Beta version and is therefore not completely stable: sometimes documentation is missing, sometimes core features are not available.

Bazel with Jenkins CI

Our company is currently using Jenkins as CI solution for all our projects and products, therefore integrating Bazel with Jenkins was a mandatory requirement. The final Jenkins pipeline is divided in 3 steps: 2 optional steps for creating the GCS bucket and building the Bazel Docker image that will be cached in the registry afterwards, and one mandatory step that executes Bazel which in turn tests, builds and deploys the binaries to GKE. The first run of the full pipeline takes around an hour, the second run won’t take more than 5 minutes.

An optional persistent volume is mounted in the machine where the Jenkins agent is running, this will enable local cache for Bazel: the building time will be reduced to less than 2 minutes for building, testing and deploying the service.

This is so far the fastest pipeline we’ve been able to achieve.

Conclusion

Google Cloud Build has a lot of potential but unfortunately the project is not mature enough and we decided to drop its use in our teams for now. We hope that Google will put more effort and manpower in improving this software and community to become professionally usable.

Bazel on the other hand looks mature enough and to be used at production level in our teams. We have already been able to achieve amazing results and we hope that the documentation will improve and be more complete over time.