jib: problem achieving reproducibility
Environment: local docker build
- Jib version: 3.4.0
- Build tool: gradle 8.3
- OS: mac m2
Description of the issue:
I’m trying to produce reproducible builds within gradle projects that include internal dependencies so I’m not able to share the project.
I’m using jib on a gradle project that contains only a single application so there’s no dependencies pulled in with project references. I can include the build block below. Nothing is placed in src/main/jib.
// Create JARs in a reproducible build fashion.
tasks.withType<AbstractArchiveTask>().configureEach {
isPreserveFileTimestamps = false
isReproducibleFileOrder = true
}
With the above, I’m able to clean build and verify that the jar is reproducible (jar checksum is the same).
Although, when I clean build jibDockerBuild the build/jib-image.digest contents are not reproducible.
From what I can tell jib does not place any files besides under /app so I did the following and was able to confirm that no files there differ between clean build jibDockerBuild calls. find app/. -type f -exec sha1sum {} + was used within separate image builds and I can diff those files to verify there’s no diff. I also spot checked that the date on all of those files is the same (epoch time).
I can also run this find app/. -type f -newermt +3 -print and verify that the old files that show up are /proc/., /sys/., and /.dockerenv. This tells me that the jib build isn’t building any files and placing them on the image with the current time.
I’m at a loss at what else to check for to see what differs between jib builds that’s keeping us from having reproducibility. Is there any suggestions at all that might help with my debugging effort?
Expected behavior:
Reproducible digest
Steps to reproduce:
Unfortunately, I’m not able to reproduce with a hello world project and our build environment contains internal dependencies.
jib-gradle-plugin Configuration:
In an effort to debug this problem, I’ve been using the below configuration so it’s easy to docker run and get dropped into a shell.
jib {
from {
image = "bash:alpine3.18"
}
container {
setEntrypoint("INHERIT")
}
}
Here’s an example of the layers produced across builds. The first four layers are always reproducible.
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438",
"sha256:623c87dfb411ee966cdb233833033a338e48cd6eaef3466b16f8f700a63ad564",
"sha256:37d625053e3d7cf997a808f8b4bbf64e218a726fc0a4ee11526a81b32e544f54",
"sha256:78c166a57f0d187d5e6530e653e9dd5550d0a34d05b841d8d97fb9d94223859a",
"sha256:ac519d1616320e98bee34efe63bec0d921ff4a7e5217ad68b25c416ed54e3914",
"sha256:7cb6b88dcd323cf433ef8e0682bba6135052dc7b7234cda26b59f5fc57386eb5",
"sha256:5cdee48ac2bc11383889fe79a98b97dd45e2c2f728ff33b562c7ca6a9c7a9a1d"
]
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438",
"sha256:623c87dfb411ee966cdb233833033a338e48cd6eaef3466b16f8f700a63ad564",
"sha256:37d625053e3d7cf997a808f8b4bbf64e218a726fc0a4ee11526a81b32e544f54",
"sha256:78c166a57f0d187d5e6530e653e9dd5550d0a34d05b841d8d97fb9d94223859a",
"sha256:c0d051434e6a3ba9863de261ad7d8e08ff8ee6086e66d62af20219d2572f91bf",
"sha256:5f396e1c50fb0d234e9b230fd9504e3db74b53a5d1fb77a6ce5bec97fa7c8d8e",
"sha256:b3b9c146e328ef8b7d0be37c7f0e751ec909713bb4ff786316ce9c2acbaa30f8"
]
},
About this issue
- Original URL
- State: open
- Created 8 months ago
- Comments: 23 (13 by maintainers)
This proves that there are cases where the last three layers can be different. A layer is actually a
.tarfile (could have been gzipped as.tar.gz, but that’s just a matter of representation), so you can diff those files directly. It doesn’t even need the Docker runtime to run the image. Just save the image usingdocker save, unzip it, and look into the file contents.Note it’s also possible that all the file contents inside a layer
.tarfile are the same (including timestamps and various file attributes) but the original gzipped.tar.gzfiles are different. I’ve seen some rare cases where a new gzip library version generates different bytes.