-
Notifications
You must be signed in to change notification settings - Fork 15.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bazel: provide pre-built binary toolchain for protoc #19558
Comments
Yes, we have changes pending to provide prebuilt protocs in the open source realm. I don't have the exact timeline tho; currently expecting for H1 2025 |
Having to constantly rebuild It also makes builds more "heavy" than they could otherwise be, which is a problem in certain more "light" (lite?) build environments; note e.g. jitpack/jitpack.io#3129. It would be really cool if using Protocol Buffers with Bazel would no longer require building |
In setups with RBE available, this is not a problem since it is mostly just downloaded from the remote cache anyways. In fact, in such setups it is usually preferred to build from source instead of downloading random binaries from the internet for compliance reasons. So building from source should definitely be kept as an option and downloading pre built binaries should be optional. |
Sure, but not every user of Protobuf of Bazel has RBE set-up for every project.
This can of course easily be solved with some sort of checksum / hash that's verified on the download; à la
An "opt in" flag (?) would already be a huge progress over the current situation! |
This is only a small part of the story. How do you verify that the binary doesn't contain malicious code? You need reproducible builds first. Then you need to build from source and verify that the binary you download is actually the artifact built from the source you are expecting. You need to do this for every version as inspecting binary diffs is not handy. When you build from source, you only need to check the source diff which is much easier to review. |
@mering yes, supply chain security is an important consideration here, both under Bazel and any other build system. I think you're pointing out #16165 again - since Then yes, it would be nice to have a proof of provenance, via some attestation published on protoc releases. We are adding these right now to modules on the BCR. It will be trivial for tools like
This isn't true for most Bazel users, since they ship Macs to their developers and the remote cache contains Linux binaries only. |
This requires trusting whoever is specifying the checksums. If you don't want to (or are not allowed to) blindly trust someone providing correct checksums but reviewing the code yourself, this is much easier when you build source code instead of comparing binaries (which usually also requires building the code in the first place).
Why are they not just using Linux for development in the first place if this is what they test and ship with their CI/CD? It's usually a bad idea to test something different from what you ship... |
Most of the time before bazel is even introduced to the environment you have existing machines and at least for the last 10 years of my career that has been macs. I personally would like a linux machine the same as my deploy environment however it is just not the reality for most workplaces. |
Interesting. In the various companies I worked across different industries, I have seen more Linux machines than Macs in the past 15 years (mostly without using Bazel). |
From https://protobuf.dev/news/2024-10-01/#end-goal:
This is that request.
What language does this apply to?
All
Describe the problem you are trying to solve.
All Bazel users are expected to build
protoc
from source as acc_binary
. This leads to problems which are often reported on the Bazel Slack:protoc
.protoc
frequently gets recompiled rather than being a cache hit - example report, issue. This makes Bazel builds slow.Describe the solution you'd like
Bazel's toolchain feature allows it to download the pre-built binaries from protobuf releases
bazelbuild/rules_proto#205 was part of my earlier work to provide this capability. The ruleset mirrors its own integrity hashes as part of each release.
https://github.com/aspect-build/toolchains_protoc/ is a user-land implementation of this proposal, however it was broken by changes in Bazel 8 and rules_proto described by https://protobuf.dev/news/2024-10-01/.
Additional context
Some user reports:
I describe some GitHub Actions workflows for automating this pattern on https://blog.aspect.build/releasing-bazel-rulesets-rust (and the earlier https://blog.aspect.build/releasing-bazel-rulesets)
The text was updated successfully, but these errors were encountered: