spinnaker: Baking manifest using a helm chart fails with `gzip: invalid header`

Issue Summary:

I try to bake a mainfest from helm served by an internal helm-repo fails with

java.lang.IllegalStateException: Bake of HelmBakeManifestRequest(namespace=acquiring-test) failed: Error: gzip: invalid header
at com.netflix.spinnaker.rosco.manifests.helm.HelmBakeManifestService.bake(HelmBakeManifestService.java:63) ~[rosco-manifests.jar:na]
at com.netflix.spinnaker.rosco.controllers.V2BakeryController.doBake(V2BakeryController.java:22) ~[rosco-web.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]

However the helm chart is properly gzipped as suggested here.

Cloud Provider(s):

On-premise kubernetes

Environment:

CentOS 7 air-gapped

Feature Area (if this issue is UI/UX related, please tag `@spinnaker/ui-ux-team`):

rosco

Description:

Log of rosco

java.lang.IllegalStateException: Bake of HelmBakeManifestRequest(namespace=acquiring-test) failed: Error: gzip: invalid header
at com.netflix.spinnaker.rosco.manifests.helm.HelmBakeManifestService.bake(HelmBakeManifestService.java:63) ~[rosco-manifests.jar:na]
at com.netflix.spinnaker.rosco.controllers.V2BakeryController.doBake(V2BakeryController.java:22) ~[rosco-web.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.5.17.RELEASE.jar:1.5.17.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:110) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at javax.servlet.FilterChain$doFilter.call(Unknown Source) [tomcat-embed-core-8.5.34.jar:8.5.34]
at com.netflix.spinnaker.rosco.filters.SimpleCORSFilter.doFilter(SimpleCORSFilter.groovy:34) [rosco-web.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111) [spring-boot-actuator-1.5.17.RELEASE.jar:1.5.17.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.5.17.RELEASE.jar:1.5.17.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) [tomcat-embed-core-8.5.34.jar:8.5.34]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.34.jar:8.5.34]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.34.jar:8.5.34]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

Steps to Reproduce:

Setup a internal artifact helm-repo (Artifactory, Nexus)
Add repo to hal config example:

hal config artifact helm account add helm-mycompany\
 --repository https://artifactory.intra/repository/helm-mycompany/ --username=jfrog-rt-spinnaker --password

Create a pipeline with an expected artifact from helm repo
- Match Against
  - Account: helm-mycompany
  - Name: acquiring-services
  - Version: 0.0.1
  - [x] Use Default Artifact
- Default Artifact:
  - Account: helm-mycompany
  - Name: acquiring-services
  - Version: 0.0.1
Create a bake step using the expected artifact
Trigger pipeline manually

Additional Details

It is not a permission problem as I can download the artifact with the user from within the clouddriver container using wget
we use self-signed certificates in the artifact repos, but this cannot be the problem
- spinnaker is able to read the index from the repo and thus get the artifacts

hal --version
1.18.0-20190325172326

helm config (hal)

....
  helm:
    enabled: true
    accounts:
    - name: helm-mycompany
      repository: https://artifactory.intra/repository/helm-mycompany/
  ...

Spinnaker version info

1.12.6 but with custom bom

 bom_1.12.6.yml: |
version: 1.12.6
timestamp: '2019-03-14 01:34:09'
services:
  echo:
	version: 2.3.1-20190214121429
	commit: 5db9d437ca7f2fa374dcada17f77bbbb2965bd67
  clouddriver:
	version: 4.3.6-20190313030608
	commit: 5c255742c2b8797a9e2e6bb166bae927f70cdb55
  deck:
	version: 2.7.6-20190313213352
	commit: b66bc33e3a22b10b221e1b21b802cfaad7a59a41
  fiat:
	version: 1.3.2-20190128153726
	commit: daf21b24330a5f22866601559aa0f7ac99590274
  front50:
	version: 0.15.2-20190222161456
	commit: 3105e86b8c084ad6ad78507e3a5e5a427f290b99
  gate:
	version: 1.5.2-20190301030607
	commit: b238ab993ab25381ce907260879548ed74a4953f
  igor:
	version: 1.1.1-20190213190226
	commit: 63d06a5c5d55f07443dd60a81035b35cf96238e7
  kayenta:
	version: 0.6.1-20190221030610
	commit: 81d906bf8307143f40fe88f8554baa318de25ef1
  orca:
	version: 2.4.1-20190312030608
	commit: 71b214148023d88486bf7c1a4d0f59adf758db18
  rosco:
	version: 0.9.0-20190123170846
	commit: 42f81a2501de6d40676d47661579a6106b5c3e8a
  defaultArtifact: {}
  monitoring-third-party:
	version: 0.11.2-20190222030609
	commit: 232c84a8a87cecbc17f157dd180643a8b2e6067a
  monitoring-daemon:
	version: 0.11.2-20190222030609
	commit: 232c84a8a87cecbc17f157dd180643a8b2e6067a
dependencies:
  redis:
	version: 2:2.8.4-2
  consul:
	version: 0.7.5
  vault:
	version: 0.7.0
artifactSources:
  #debianRepository: https://dl.bintray.com/spinnaker-releases/debians           
  debianRepository: https://ci-tools.sc.intra/nexus3/repository/spinnaker-releases/           
  #dockerRegistry: gcr.io/spinnaker-marketplace           
  dockerRegistry: ci-tools.sc.intra:5000/spinnaker-marketplace           
  googleImageProject: marketplace-spinnaker-release           
  #gitPrefix: https://github.com/spinnaker           
  gitPrefix: https://scm.sc.intra/scm/SPIN/repos

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 34

Most upvoted comments

Apparently it seems a permission problem which is not yet clear why. We are using artifactory and the repo we have configured is an internal virtual (or group) repo which currently has the the following local repos configure

local-helm-devops
local-helm-mycompany

In the artifactory log I can see that the index.yaml is gathered with the correct user jfrog-rt-spinnaker, but then subsequent requests are done with anonymous which currently has no access:

2019-04-18 07:42:07,125 [ACCEPTED DOWNLOAD] virtual-helm:.index/-577854709/1427465087/index.yaml for jfrog-rt-spinnaker/10.128.78.24.
2019-04-18 07:42:07,206 [DENIED DOWNLOAD] local-helm-devops:acquiring-services-0.0.1.tgz for NA/10.128.78.24.
2019-04-18 07:42:07,209 [DENIED DOWNLOAD] local-helm-mycompany:acquiring-services-0.0.1.tgz for NA/10.128.78.24.

the error from rosco is a bit misleading cause actually the problem is a permission problem and not a problem with the artifact itself
it’s not clear yet why the request for the chart
- is issued against local-hel...
- done by anonymous

Obviously the problem is solved as soon we give download permissions to anonymous

papanito on Apr 18, 2019