slsa-github-generator: [bug] ”Generate Builder“ step fails

Hi! Thanks for the report: we’ve seen this and are trying to fix this all the way through.

This is the same issue that @stephenfuqua reported: https://github.com/slsa-framework/slsa-github-generator/issues/876#issuecomment-1261469816

In the meantime, enable compile-builder: true to succeed.

What we know has happened is that Rekor (the transparency log that the builders provenance is recorded on) had breaking changes that broke our verification.

Here’s what happened in this particular case when we were verifying the builder provenance’s inclusion in the log: we checked verified inclusion proof against the root hash, and then verified the root hash against a current signed tree head. After Rekor sharded (rotated trees) in production, the tree head no longer corresponds to the same tree that the entry was in, which caused a verification error. In particular, the tree size given in the entry’s inclusion proof is LARGER than the current tree size after rotation.

This will fix this https://github.com/slsa-framework/slsa-verifier/pull/277, and we’ll backport the changes to the release/v1.1 branch of the verifier. I think we’ll need to update all builder workflows to stop using verifier v1.1.1. @laurentsimon right?

I think this conversation illustrates the importance and difficulty of deep supply chain integrity. My workflow failed because the SLSA provenance generator failed because the Rekor service failed…

And the above discussions raise a few more points that should be considered:

• Does Rekor version its own API, so that tools (like this SLSA provenance generator) can build against a stable service even if/when Rekor changes?
• Does this SLSA provenance generator bump its own version according to Rekor’s changes, if they impact end-users? What other external services does SLSA itself depend on?

Unfortunately Rekor is making changes that break clients and will likely continue to do so until it’s GA.

@ianlewis I think GA is no safeguard against breakage. I’d argue that this SLSA provenance generator should be tolerant to Rekor failures beyond GA. For example @asraa’s proposed fix above uses an undocumented feature, which should probably be documented for users because it decouples from yet another level of dependencies.

Further still, it may even make sense to make compile-generator: true the default because it avoids that extra dependency on Rekor. Personally, I prefer a slower self-contained action over a faster one with dependencies on a potentially failing external service (thus jeopardizing stability). It’s kind-of ironic that a tool checking software supply chain itself depends on a more complex supply chain.

I’d argue that this SLSA provenance generator should be tolerant to Rekor failures beyond GA.

Two things we’ve prioritized on this front for the next quarter:

1. Supporting offline validation; so in case there’s a Rekor failure or breakage, we’re robust to verification and will continue doing so. https://github.com/slsa-framework/slsa-github-generator/issues/716
2. Supporting testing against staging to block Rekor releases in case they break us. https://github.com/slsa-framework/slsa-github-generator/issues/372

While providing a security tool though, we also did require the most robust security verification (with an online verification from Rekor directly) like Ian mentioned. I can see the argument both ways though, and reducing its reliance on an online service or transparency log is helpful for the future.

This is because it allows building from unreleased/untested versions of the workflow (e.g. from HEAD). We only have the input there at all in order to support our pre-submit tests.

@ianlewis @laurentsimon can we support a build from the tag checkout using a secure-checkout that ensures that tag resolves to a safe/attested SHA? This is similar to @behnazh-w suggestion.

Edit: Is it https://github.com/slsa-framework/slsa-github-generator/issues/968?

Yes, I’ll add a note on release of v1.3.1 in that title.

@jenstroeger We finally made a release with the fix for this. Please use v1.2.1.

This is because it allows building from unreleased/untested versions of the workflow (e.g. from HEAD)

As a new feature, how about allowing building from source with the correct version? Would there still be any security concerns?