spring-boot: Maven plugin build-image creator step fails due to “Invalid content length provided” in Bitbucket pipelines

Running mvn spring-boot:build-image in bitbucket pipeline (maven:3.6.3-openjdk-11 image with default docker service) fails in the “Running creator” step.

From the docker log it appears that “authorization (was) denied by plugin pipelines: Invalid content length provided”. This could be a red herring as bitbucket documentation indicates the docker commands are restricted https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/

Unfortunately, there isn’t enough debug logging to troubleshoot the createContainer method further.

Can you please indicate whether this is a known limitation or if there is some configuration that can be included to work around the bitbucket restrictions?

Bitbucket docker log:

time="2020-08-09T11:28:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2020-08-09T11:28:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/v1.39/version
time="2020-08-09T11:29:47Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=gcr.io%2Fpaketo-buildpacks%2Fbuilder%3Alatest"
time="2020-08-09T11:30:11Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/gcr.io/paketo-buildpacks/builder@sha256:c4feeff9bb38c56cc615129c65162e5e8f6bfd3cd7ffae6e2e673c57ac6b9d5b/json"
time="2020-08-09T11:30:11Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=docker.io%2Fpaketobuildpacks%2Frun%3Afull-cnb-cf"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/docker.io/paketobuildpacks/run@sha256:66128f7a3675b8fd8cd454ed2b266910a04f6473983801bb0250e1acd80f3560/json"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.24/images/load
**time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/v1.24/containers/create
time="2020-08-09T11:30:12.607341026Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: Invalid content length provided"**
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-layers-qktednfhbn?force=1"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-app-olvjnashcf?force=1"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/images/pack.local/builder/ssmtjetwcz:latest?force=1"

Maven stack trace:

[INFO] --- spring-boot-maven-plugin:2.3.2.RELEASE:build-image (default-cli) @ xyz-services ---
[INFO] Building image 'docker.io/library/xyz-services:2.1.0-SNAPSHOT'
[INFO] 
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 3%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 10%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 10%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 12%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 14%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 33%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 48%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 100%
[INFO]  > Pulled builder image 'gcr.io/paketo-buildpacks/builder@sha256:c4feeff9bb38c56cc615129c65162e5e8f6bfd3cd7ffae6e2e673c57ac6b9d5b'
[INFO]  > Pulling run image 'docker.io/paketobuildpacks/run:full-cnb-cf' 100%
[INFO]  > Pulled run image 'paketobuildpacks/run@sha256:66128f7a3675b8fd8cd454ed2b266910a04f6473983801bb0250e1acd80f3560'
[INFO]  > Executing lifecycle version v0.8.1
[INFO]  > Using build cache volume 'pack-cache-900830d40c15.build'
[INFO] 
[INFO]  > Running creator
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  52.776 s
[INFO] Finished at: 2020-08-09T11:30:12Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image (default-cli) on project xyz-services: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden" -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image (default-cli) on project xyz-services: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.springframework.boot.buildpack.platform.docker.transport.DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
    at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute (HttpClientTransport.java:136)
    at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute (HttpClientTransport.java:123)
    at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.post (HttpClientTransport.java:94)
    at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.createContainer (DockerApi.java:234)
    at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.create (DockerApi.java:225)
    at org.springframework.boot.buildpack.platform.build.Lifecycle.createContainer (Lifecycle.java:166)
    at org.springframework.boot.buildpack.platform.build.Lifecycle.run (Lifecycle.java:146)
    at org.springframework.boot.buildpack.platform.build.Lifecycle.execute (Lifecycle.java:113)
    at org.springframework.boot.buildpack.platform.build.Builder.executeLifecycle (Builder.java:122)
    at org.springframework.boot.buildpack.platform.build.Builder.build (Builder.java:71)
    at org.springframework.boot.maven.BuildImageMojo.buildImage (BuildImageMojo.java:144)
    at org.springframework.boot.maven.BuildImageMojo.execute (BuildImageMojo.java:136)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 25 (8 by maintainers)

Commits related to this issue

Provide content-length header to Docker API calls Docker daemon authorization plugins reject POST or PUT requests that have a content type `application/json` header but no content length header. This... — committed to spring-projects/spring-boot by scottfrederick 4 years ago

Most upvoted comments

Yes @iquequerio. Scott (above) was working on making the buildpack process more configurable so from Spring Boot 3.2 we can use buidpacks in BB pipelines. I’ve explained all the details here.

dbaltor on Nov 19, 2023

Hi @scottfrederick , thanks for the swift response. Yeah, you’re right. I run it in debug mode and it’s the same error reported by @corneil above. It seems related to how the buildpack runs not being compatible with the BitBucket env.

[DEBUG] http-outgoing-0 >> POST /v1.24/containers/create HTTP/1.1
[DEBUG] http-outgoing-0 >> Accept-Encoding: gzip, x-gzip, deflate
[DEBUG] http-outgoing-0 >> Content-Length: 742
[DEBUG] http-outgoing-0 >> Content-Type: application/json
[DEBUG] http-outgoing-0 >> Content-Encoding: UTF-8
[DEBUG] http-outgoing-0 >> Host: localhost:2375
[DEBUG] http-outgoing-0 >> Connection: keep-alive
[DEBUG] http-outgoing-0 >> User-Agent: Apache-HttpClient/5.2.1 (Java/21.0.1)
[DEBUG] http-outgoing-0 >> "POST /v1.24/containers/create HTTP/1.1[\r][\n]"
[DEBUG] http-outgoing-0 >> "Accept-Encoding: gzip, x-gzip, deflate[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Length: 742[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Type: application/json[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Encoding: UTF-8[\r][\n]"
[DEBUG] http-outgoing-0 >> "Host: localhost:2375[\r][\n]"
[DEBUG] http-outgoing-0 >> "Connection: keep-alive[\r][\n]"
[DEBUG] http-outgoing-0 >> "User-Agent: Apache-HttpClient/5.2.1 (Java/21.0.1)[\r][\n]"
[DEBUG] http-outgoing-0 >> "[\r][\n]"
[DEBUG] http-outgoing-0 >> "{[\n]"
[DEBUG] http-outgoing-0 >> "  "User" : "root",[\n]"
[DEBUG] http-outgoing-0 >> "  "Image" : "pack.local/builder/wrficodrgr:latest",[\n]"
[DEBUG] http-outgoing-0 >> "  "Cmd" : [ "/cnb/lifecycle/creator", "-app", "/workspace", "-platform", "/platform", "-run-image", "docker.io/paketobuildpacks/run-jammy-base:latest", "-layers", "/layers", "-cache-dir", "/cache", "-launch-cache", "/launch-cache", "-daemon", "docker.io/library/example:0.0.1-SNAPSHOT" ],[\n]"
[DEBUG] http-outgoing-0 >> "  "Env" : [ "CNB_PLATFORM_API=0.11" ],[\n]"
[DEBUG] http-outgoing-0 >> "  "Labels" : {[\n]"
[DEBUG] http-outgoing-0 >> "    "author" : "spring-boot"[\n]"
[DEBUG] http-outgoing-0 >> "  },[\n]"
[DEBUG] http-outgoing-0 >> "  "HostConfig" : {[\n]"
[DEBUG] http-outgoing-0 >> "    "Binds" : [ "/var/run/docker.sock:/var/run/docker.sock", "pack-layers-tjjgagqsgk:/layers", "pack-app-stzbsvwuxn:/workspace", "pack-cache-8847f704ad41.build:/cache", "pack-cache-8847f704ad41.launch:/launch-cache" ],[\n]"
[DEBUG] http-outgoing-0 >> "    "SecurityOpt" : [ "label=disable" ][\n]"
[DEBUG] http-outgoing-0 >> "  }[\n]"
[DEBUG] http-outgoing-0 >> "}"
[DEBUG] http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
[DEBUG] http-outgoing-0 << "Content-Type: application/json[\r][\n]"
[DEBUG] http-outgoing-0 << "Date: Fri, 03 Nov 2023 17:28:44 GMT[\r][\n]"
[DEBUG] http-outgoing-0 << "Content-Length: 117[\r][\n]"
[DEBUG] http-outgoing-0 << "[\r][\n]"
[DEBUG] http-outgoing-0 << "{"message":"authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"}[\n]"
[DEBUG] http-outgoing-0 << HTTP/1.1 403 Forbidden
[DEBUG] http-outgoing-0 << Content-Type: application/json
[DEBUG] http-outgoing-0 << Date: Fri, 03 Nov 2023 17:28:44 GMT
[DEBUG] http-outgoing-0 << Content-Length: 117

I’ll follow your suggestion and look for help with the buildpack folks. Have a great weekend 🙂

dbaltor on Nov 3, 2023

BitBucket/Atlassian have not solved this issue: jira.atlassian.com/browse/BCLOUD-17592s

fastzhong on Nov 1, 2022

Today the bitbucket team made an update explaining why they can’t support Transfer-Encoding Chunked see here

In short, this is not an issue with bitbucket in particular, but with how dockers authorization plugins work and they can’t do much about it currently, although they would appreciate an update by docker, since other projects are experience similar problems. Other environments with dockers authZ plugin configured are affected as well.

In this specific case, we could supply a Content-Length with the container creation POST request. However, it would then fail for the subsequent PUT that uploads the container’s content.

The plugin only applies to content-type application/json. The subsequent PUT should be content-type application/x-tar, so the plugin wouldn’t intercept and reject a chunked transfer there.

Could you reopen this issue and update the POST request, since a change to dockers authZ plugin seems unlikely currently?

From the linked duplicate

It’s not clear to me if this is an RFC-style should where it’s something that’s recommended but optional or if it’s a requirement a must in RFC terms). It would appear that the local Docker daemon treats it as optional (but perhaps recommended) whereas BitBucket’s pipelines treat it as a requirement. Either way, it’s probably worth us complying with the API’s recommendations if we can.

I guess the answer is, it is a requirement that applies as soon as docker plugin infrastructure is involved 😄

goatfryed on Oct 14, 2020

Thanks for the report.

The problem appears to be that BitBucket considers the POST request that is creating a container to have an invalid Content-Length:

time="2020-08-09T11:30:12.607341026Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: Invalid content length provided"

As far as I can tell, BitBucket’s problem is that the POST request is sent using Transfer-Encoding: chunked as the content length isn’t known. There is, therefore, no Content-Length header in the request. This is what I assume causes BitBucket to consider the Content-Length to be invalid but we can’t be certain as, AFAIK, BitBucket Pipelines isn’t open source.

In this specific case, we could supply a Content-Length with the container creation POST request. However, it would then fail for the subsequent PUT that uploads the container’s content.

I think this will have to be raised with Atlassian. Assuming my analysis above is correct, ideally they would support chunked transfer encoding as Docker usually does. Failing that, if my analysis is incorrect or they’re unable to support chunked transfer encoding and they can provide detailed information about the Docker-related restrictions, we can re-open this and take another look.

wilkinsona on Aug 10, 2020