jib: Images created by Jib randomly causes 500 Internal server error in Red Hat Quay

Environment:

Jib version: 1.5.1
Build tool: Maven
OS: Linux and Windows

Description of the issue: Images created by Jib randomly causes 500 Internal server error in Red Hat Quay. This has been reproduced with the HelloWorld example from Jib, but does not happen with images created in any other way than with Jib.

Expected behavior: Images created by Jib should consistently be accepted by Quay.

Steps to reproduce:

Download https://github.com/GoogleContainerTools/jib/tree/master/examples/helloworld
Change <image>gcr.io/REPLACE-WITH-YOUR-GCP-PROJECT/image-built-with-jib</image> to <image>QUAY-REGISTRY/image-built-with-jib</image>
Run mvn compile jib:build

jib-maven-plugin Configuration:

<configuration>
    <to>
        <image>QUAY-REGISTRY/image-built-with-jib</image>
    </to>
</configuration>

Log output:

> [2019-09-13T07:50:47.986Z] [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.5.1:build (build_docker_image) on project xx: 500 Internal Server Error
> [2019-09-13T07:50:47.986Z] [ERROR] <html>
> [2019-09-13T07:50:47.986Z] [ERROR]   <head>
> [2019-09-13T07:50:47.986Z] [ERROR]     <title>Internal Server Error</title>
> [2019-09-13T07:50:47.986Z] [ERROR]   </head>
> [2019-09-13T07:50:47.986Z] [ERROR]   <body>
> [2019-09-13T07:50:47.986Z] [ERROR]     <h1>Internal Server Error</h1>
> [2019-09-13T07:50:47.986Z] [ERROR]
> [2019-09-13T07:50:47.986Z] [ERROR]   </body>
> [2019-09-13T07:50:47.986Z] [ERROR] </html>

Additional Information: Red Hat’s response is that this only happens with Jib-created images and therefore points the finger at Jib.

Is it possible to get Google and Red Hat to cooperate in finding a solution to this? Red Hat Case #: 02456747

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 42 (25 by maintainers)

Most upvoted comments

Quay developer here; I just wanted to take a moment to chime in.

@jonjohnsonjr

Probably because jib creates schema 2 images? Has quay rolled out schema 2 support fully yet?

Great catch! The https://quay.io service does not (yet) support Schema 2 images, yet. To the best of my knowledge, some on-premise installations may have this enabled depending on their configuration.

Our initial struggle was being able to reproduce this issue locally. There’s a ton of really great information in here that will get us one step closer. I, personally, have never heard of jib until this issue popped up so it’s all new ground for me.

The HTTP 500 response sounds like an issue on our end. I can’t promise when it will be fixed, but regardless of the payload received by Quay, it should respond with an appropriate status. I will do my best to provide updates if anyone is interested.

Please don’t hesitate to @ me if anyone has questions I might be able to answer.

kurtismullins on Sep 19, 2019

Update:

I’ve been able to create the “Hello World” example using jib and push it to my local (development) Quay environment. Thanks to all for the reproduction steps!

Here’s some initial observations:

If Schema 2_2 is not enabled, I always get the (expected) “Manifest Invalid” error
On a new/fresh image build+push using jib, I am seeing an exception related to a chunk missing a newline terminator.

It’s still pretty premature but I believe that exception is not caught and handled correctly (hence the 5xx) and I’m not sure which chunk (if any?) is malformed. On my local system and example, the request which generates this error is:

PATCH /v2/devtable/image-built-with-jib-test/blobs/uploads/bcab81cf-b0d9-4972-b4a6-63ba87d13da5 HTTP/1.1

Moving forward, I’m digging through the raw requests using Wireshark to see if I can find a malformed chunk. My goal is to produce a testable scenario to trigger this exception so that we can at least get it handled properly on Quay’s end. If I find a malformed request being generated by jib and am able to reproduce it, I’ll share that information as well.

For reference purposes, here’s essentially my current test-case:

Clone jib (working directly out of master)
Point the helloWorld example to my Quay instance
Execute mvn -Djib.console=plain -Djava.util.logging.config.file=./logging.properties -DsendCredentialsOverHttp=true compile jib:build
Modify HelloWorld.java and the repository’s name in pom.xml for each fresh run.

Note: After encountering the HTTP 500 error initially for a given image, a subsequent retry succeeds. Hence the purpose of my changes to pom.xml and HelloWorld.java in my test-case.

I am far from a Java, maven, or jib expert so please don’t hesitate to share suggestions if you have them. I had hoped I could use mitmproxy but neither the command-line arguments (e.g. -Dhttp.proxyHost [...]) or the maven settings file seemed to have worked. It is probably related to my local configuration and not using TLS/HTTPs.

kurtismullins on Sep 23, 2019

I’m surprised it’s causing a 500 instead of the 415 like you’re seeing.

jonjohnsonjr on Sep 13, 2019

@jonjohnsonjr Quay still seems to default to Schema 1, as @briandealwis and I observed today. People have been individually requesting whitelisting their namespaces, even until recently.

chanseokoh on Sep 13, 2019