jib: 'MANIFEST_INVALID' when uploading to Artifactory

Description of the issue: When running jib as part of a maven goal, I get the following:

[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:0.9.2:build (build-docker-image) on project jengel-service: Build image failed: Tried to push image manifest for $OUR_REGISTRY/sre/jengel-test:jib but failed because: manifest invalid (something went wrong) | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new: 400 Bad Request
[ERROR] {"errors":[{"code":"MANIFEST_INVALID","message":"manifest invalid","detail":{"description":"null"}}]}

When looking at the artifactory logs, it seems like the manifest being uploaded is null:

2018-07-10 19:55:25,650 [http-nio-8081-exec-55] [INFO ] (o.a.e.UploadServiceImpl:360) - Deploy to 'et-docker-local:sre/jengel-test/_uploads/49928209-6f48-451f-8bd3-57a329be6880' Content-Length: unspecified
2018-07-10 19:55:25,748 [http-nio-8081-exec-66] [INFO ] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:257) - Deploying docker manifest for repo 'sre/jengel-test' and tag 'jib' into repo 'et-docker-local'
2018-07-10 19:55:25,761 [http-nio-8081-exec-66] [ERROR] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:298) - Error uploading manifest: 'null'

Expected behavior: The image should be uploaded to artifactory 😄

Steps to reproduce:

  • Add an artifactory repo as your <to><image> block
  • Execute mvn to publish image
  • Receive error ):

Environment:

  • Current trying on macOS 10.13.5
  • mvn 3.5.3
  • jib 0.9.2
  • Artifactory 6.0.1

jib-maven-plugin Configuration:

                    <plugin>
                        <groupId>com.google.cloud.tools</groupId>
                        <artifactId>jib-maven-plugin</artifactId>
                        <version>0.9.2</version>
                        <configuration>
                            <from>
                                <image>$OUR_REGISTRY/frolvlad/alpine-oraclejdk8:slim</image>
                            </from>
                            <to>
                                <image> $OUR_REGISTRY/sre/jengel-test:jib</image>
                            </to>
                            <container>
                                <jvmFlags>
                                    ...
                                </jvmFlags>
                                <format>OCI</format>
                            </container>
                        </configuration>
                        <executions>
                            <execution>
                                <id>build-docker-image</id>
                                <goals>
                                    <goal>build</goal>
                                </goals>
                                <phase>package</phase>
                            </execution>
                        </executions>
                    </plugin>

Log output: See description above for relevant logs

Additional Information:

Tried running as well with the -X flag in maven with nothing else extreme helpful. I do actually see part of the image layers in an _uploads folder in artifactory, but no manifest.json like I’d expect from an image. Happy to provide additional information if needed!

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 1
  • Comments: 39 (16 by maintainers)

Most upvoted comments

Confirmed: Adding as many history elements as there’s layers make container push to Artifactory. Red Hat Container Registry still throws error with this quick try, so it might still require something else.

But Artifactory works when there’s equal number of (non-empty-layer) history elements than layers.

For test I only added static ones to jib-core/src/main/java/com/google/cloud/tools/jib/image/json/ContainerConfigurationTemplate.java:

private final List history = new ArrayList();

  private static class History implements JsonTemplate {
        private String author = "Bazel";
        private String created = "1970-01-01T00:00:00:Z";
        private String created_by = "bazel_build ...";
  }

public void addLayerDiffId(DescriptorDigest diffId) {
    history.add(new History());
    rootfs.diff_ids.add(diffId);
  }
+6
Hi-Fi on Aug 24, 2018

Tested with Artifactory 6.0.2, works well. Big Thanks!

+3
zhzhm on Aug 30, 2018

Version 0.9.10 has been released!

+2
coollog on Aug 31, 2018

@jainishshah17 @jbaruch

I used a method decompiler and walked through Artifactory’s ManifestSchema2Deserializer class while attempting to push up an image from Jib. The problem occurs in ManifestSchema2Deserializer#applyAttributesFromContent().

Basically this code walks through the layers and creates blob infos for each layer. Although it attempts to support containers with no history (which is allowed by the spec, and which Jib does not produce), it does not increment layerIndex when there is no history. The code looks something like:

private static ManifestMetadata applyAttributesFromContent(...) {
    ...
    final JsonNode history = config.get("history");
    final JsonNode layers = manifest.get("layers");
    final boolean foreignHasHistory = layers.size() == historyCounter;

    int historyIndex = 0;
    int layersIndex = 0;
    while (historyIndex < historySize || layersIndex < layers.size()) {
        final JsonNode historyLayer = (history == null) ? null : history.get(historyIndex);
        final JsonNode layer = layers.get(layersIndex);
        long size = 0L;
        String digest = null;
        // This condition appears to be problematic:
        //  - Jib images have no history, thus `historyLayer` is always `null`,
        //     and so `notEmptyHistoryLayer(null)` is always `false`
        //  - None of our layers are foreign
        // and so layersIndex is never incremented
        if (notEmptyHistoryLayer(historyLayer) || (!foreignHasHistory && isForeignLayer(layer))) { // !!!
            size = layer.get("size").asLong();
            totalSize += size;
            digest = layer.get("digest").asText();
            ++layersIndex; // XXX never incremented
        }
        // build a BlobInfo from digest, size, and created
        // add the blobInfo to the manifest metadata
        // circuit breaker checks
    }
    // populate remaining more manifest metadata
}

...

private static boolean notEmptyHistoryLayer(final JsonNode historyLayer) {
    return historyLayer != null && historyLayer.get("empty_layer") == null;
}

It seems to me that either the conditional marked with !!! should be expanded to include historyLayer == null?

+2
briandealwis on Aug 17, 2018

For the Artifactory issue: from the Artifactory server log in the above Artifactory service ticket (https://www.jfrog.com/jira/browse/RTFACT-17134) created by @mzagar, I am suspecting it’s the Artifactory JSON manifest parser that is breaking. Also from the log, Artifactory does seem like it’s capable of handling manifest schema 2.

2018-07-18 08:09:06,855 [http-nio-8081-exec-200] [ERROR] (o.j.r.d.m.ManifestSchema2Deserializer:133) - ManifestSchema2Deserializer CIRCUIT BREAKER: 5000 Iterations ware performed breaking operation.
+2
chanseokoh on Jul 25, 2018

Made some tcpdumping to see what’s different. Results below (haven’t yet figured out what is the thing causing the issue). Also not sure if (without clearing up those messages) the put should work twice or long after the other things are uploaded. But at least some differences can be seen from those.

Jib:

PUT /v2/test_project/test_application/manifests/latest HTTP/1.1
Accept:
Accept-Encoding: gzip
Authorization: Bearer.removeValidToken
User-Agent: jib.0.9.9-SNAPSHOT.jib-maven-plugin.Google-HTTP-Java-Client/1.23.0.(gzip)
Transfer-Encoding: chunked
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Host: test_artifactory_server:15000
Connection: Keep-Alive

{
	"schemaVersion":2,
	"mediaType":"application/vnd.docker.distribution.manifest.v2+json",
	"config":
		{
			"mediaType":"application/vnd.docker.container.image.v1+json",
			"digest":"sha256:b2a10b2a921637bbee627553275e6f7a849fa511b2bc125167a00494da6e090e",
			"size":963
		},
	"layers": [
		{
			"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip",
			"digest":"sha256:fc7b4c7963f54d0db0563d722be2969ca2b2962bb2cd18ea92e1ff50da34b052",
			"size":7869666
		},
		{
			"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip",
			"digest":"sha256:5ffca8b298d5abfc906ae13e87ec2627d482ae8cb8184ca031accb963243fc6a",
			"size":643663
		},
		{
			"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip",
			"digest":"sha256:a1aef001875f598a7051ad8d2abbb89be66a29d13c608d65dba7c1c906c668a9",
			"size":38870533
		},
		{
			"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip",
			"digest":"sha256:7c46de080f2d1c5fb8c4bfbb3cca885c7570299597673f4d985549f850875828",
			"size":58172233
		},
		{
			"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip",
			"digest":"sha256:363596415e324a41b3cb785da315800d728221092766287806f50f950b5f5366",
			"size":315660
		},
		{
			"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip",
			"digest":"sha256:f73f9ca393435c6e2f011779ee355e622205ac06d88ab956e480001764b4b1ad",
			"size":3112
		},
		{
			"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip",
			"digest":"sha256:ceea43ecb1e567009daa48f0ff3f987a49c01c4b0e20fded2e7dc13d26f7632e",
			"size":19381
		}
	]
}


HTTP/1.1.400.Bad.Request
Date: Thu,.09.Aug.2018.11:31:17.GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Server: Artifactory/6.0.1
X-Artifactory-Id: artifactory_identifier
Docker-Distribution-Api-Version: registry/2.0

{
	"errors":
	[
		{
			"code":"MANIFEST_INVALID",
			"message":"manifest.invalid",
			"detail":
			{
				"description":"Circuit.Breaker.Threshold.Reached,.Breaking.Operation see.log.output.for.manifest.detail"
			}
		}
	]
}


Docker-client

PUT /v2/test_project/test_application/manifests/latest HTTP/1.1
Host: test_artifactory_server:15000
User-Agent: docker/1.12.6.go/go1.7.4.kernel/3.10.0-514.10.2.el7.x86_64.os/linux.arch/amd64.UpstreamClient(Docker-Client/1.12.6.\(linux\))
Content-Length: 1789
Authorization: Bearer.removeValidToken
Content-Type: application/vnd.docker.distribution.manifest.v2
Accept-Encoding: gzip
Connection: keep-alive

{
	"schemaVersion": 2,
	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
	"config": 
		{
			"mediaType": "application/vnd.docker.container.image.v1+json",
			"size": 3473,
			"digest": "sha256:088d204168a91fa85bae7d7208de148c8733243a56f5f3410cb3435be858e499"
		},
	"layers": [
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 7869666,
			"digest": "sha256:fc7b4c7963f54d0db0563d722be2969ca2b2962bb2cd18ea92e1ff50da34b052"
		},
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 643663,
			"digest": "sha256:5ffca8b298d5abfc906ae13e87ec2627d482ae8cb8184ca031accb963243fc6a"
		},
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 38870533,
			"digest": "sha256:a1aef001875f598a7051ad8d2abbb89be66a29d13c608d65dba7c1c906c668a9"
		},
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 58191675,
			"digest": "sha256:a1d09f4bd8711a78772117d4adc9f0b420a3773cbfe15846f4c65b88b6255a31"
		},
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 315927,
			"digest": "sha256:7c4d4cbb2a07735cba5bd8a9c7624ecd5170951ec59ac5b96cb5c10e9b51eaad"
		},
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 3231,
			"digest": "sha256:747822417e29d78d147ceabef840d1b771a7e0d23378bfe3c81cf98e43d9b7b1"
		},
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 19333,
			"digest": "sha256:5b04d502d1fa4e4b9e1fd9e56c12c966d7787155d0d3a80194963a81c8a997c1"
		}
	]
}

HTTP/1.1.201.Created
Date: Thu,.09.Aug.2018.11:28:25.GMT
Content-Type: text/plain
Content-Length: 0
Connection: keep-alive
Server: Artifactory/6.0.1
X-Artifactory-Id: artifactory_identifier
Docker-Distribution-Api-Version: registry/2.0
Docker-Content-Digest: sha256:9cbafa2ffc22c459065420626b40c5cc65195886289ce0c8969aa1c99cb9fe09
+1
Hi-Fi on Aug 9, 2018

Hi all, fyi I created a ticket for jfrog Artifactory trying to get more info as to what is causing this issue: https://www.jfrog.com/jira/browse/RTFACT-17134

+1
mzagar on Jul 18, 2018
Read more comments on GitHub
← jib: MANIFEST_INVALID when pushing manifest to quay.io
jib: NoSuchMethodError: old Apache library loaded in Gradle build environment →