jib: Unable to push to harbor image with http
Environment:
- Jib version: 1.5.0
- Build tool: Gradle
- OS: Mac OSX
- Repository: Harbor
Description of the issue:
I have insecure harbor registry for now and trying to push it and pushing it with property sendCredentialsOverHttp=true but push always times out
Expected behavior:
Should be able to push
Steps to reproduce:
/gradlew jib -DsendCredentialsOverHttp=true -Djib.to.image=dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample -Djib.console=plain
jib-gradle-plugin Configuration:
jib {
to {
allowInsecureRegistries = true
tags = ['latest', '1.0.0-SNAPSHOT']
auth {
username = 'xyz'
password = 'xyz'
}
}
container {
ports = ['8080']
mainClass = "$mainClassName"
args = ["run", mainVerticleName, "-conf staging.json"]
jvmFlags = ['-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory']
}
}
Log output:
➜ ./gradlew jib -DsendCredentialsOverHttp=true -Djib.to.image=dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample -Djib.console=plain
Starting a Gradle Daemon, 1 busy and 1 incompatible Daemons could not be reused, use --status for details
> Task :jib
Authentication over HTTP is enabled. It is strongly recommended that you do not enable this on a public network!
Containerizing application to dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample, dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample, dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample:1.0.0-SNAPSHOT...
Retrieving registry credentials for dockerhub-1003850024.ap-south-1.elb.amazonaws.com...
Getting base image gcr.io/distroless/java:8...
Building dependencies layer...
Building project dependencies layer...
Building resources layer...
Building classes layer...
> Task :jib FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Connect to dockerhub-1003850024.ap-south-1.elb.amazonaws.com:443 [dockerhub-1003850024.ap-south-1.elb.amazonaws.com/35.154.123.120, dockerhub-1003850024.ap-south-1.elb.amazonaws.com/13.234.102.169] failed: Operation timed out (Connection timed out)
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 2m 40s
8 actionable tasks: 1 executed, 7 up-to-date
Additional Information:
Push works fine with docker command-line
docker tag xyz:1.0.0-SNAPSHOT dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample/xyz:1.0.0-SNAPSHOT
docker push dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample/xyz:1.0.0-SNAPSHOT
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 26 (15 by maintainers)
I did an investigation, and I think it is the bug in recent Google HTTP Client versions that does not honor the given connection timeout. (Haven’t tested read timeout yet.) Jib 1.5.0 is affected, and I am positive that the timeout will work correctly on 1.4.0.
@riyaz looked at your last log. No wonder it takes > 20 minutes (with 1.5.0). At least it is indeed working but very inefficient because Jib first tries HTTPS (port 443) before trying HTTP (port 80) for every access to your registry. And this HTTPS-to-HTTP failover can happen only when the HTTPS attempt times out. The log shows that the timeout is about 75 seconds. That is, for every HTTP request Jib makes to your registry, it first has to wait for ~75 seconds. But I do see that the plain
http://is very fast.That said, if you specify the port
:80in your image reference (e.g.,...elb.amazonaws.com:80/sample/...), I bet it will be very fast, as it will attempt the plain HTTP directly. Later you could add an additional tag using whatever means to have the correct name without the port part (:80). Another dirty hack would be to somehow configure your firewall or network so that any connection attempt toamazonaws.com:443drops immediately as @briandealwis said.But, I strongly recommend you make the port 443 work as a registry and close the insecure registry at port 80. I’m emphasizing again that by sending credentials over HTTP, anyone in the wild can capture and steal your username and password easily.
Given that you don’t have a secure registry, ECR seems rather the right choice. I would continue using it, at least until you set up SSL.
Thanks for the info. It’s good to know. We are baffled too.
My test above used Jib 1.2.0 and I had the same result on 1.4.0. With Jib 1.5.0 I do see Jib cascading from HTTPS to HTTP. I see lengthy timeouts though (1m15s or so) on any request though.
@raizoor if you look at the
jib-cred-helper.logandjib.logfrom https://github.com/GoogleContainerTools/jib/issues/1917#issuecomment-523134890, we can check the HTTPS-to-HTTP fallback did happen. I suspect this happens only with 1.5.0 but not with 1.4.0.Found the issue with network logs, gradle generates log only if i add
--no-daemonAttaching the logs, from the logs found 401 errors, so tried removing auth block and used helper from docker even that’s giving 401, however push in docker works fine.
jib-1.4.log jib-cred-helper.log jib.log