helm: Helm repo add fails with some repositories

Hi @SteveLasker - it appears this is also a problem in Helm 3, based on user reports.

Quick update: for those using chartmuseum. charts.jetstack.io, codefresh, etc. Helm v3.3.3 and Helm v2.16.12 is now available with a fix. Please upgrade to receive the fix.

https://github.com/helm/helm/releases/tag/v3.3.3 https://github.com/helm/helm/releases/tag/v2.16.12

We’ve tested v3.3.3/v2.16.12 against a number of chart repositories, and they all seem to be working now. The only one that does not work currently is ACR.

I’ve re-titled the issue to reflect the current state of things.

Hey @mate-szalai. This is unfortunately one of the side effects of one of the security fixes.

In order to address one of the security issues, Helm now has to validate that the index.yaml file does not contain any extra fields, which chartmuseum does appear to add.

We’re looking into this right now.

It’s a little unfortunate that this security fix didn’t go through an embargo period to understand the impact and fix before it got rolled out.

Last Friday I kicked off a conversation on creating an embargo period setup for security releases.

@SteveLasker if you want a project to have an embargo setup for security releases the appropriate way to discuss it is to request it. Complaining that it doesn’t exist or that you are not on the list is an approach that usually doesn’t win favor with open source project maintainers.

I would also suggest that the next time y’all are going to extend something with Helm it may be useful to ask about patterns in doing so upstream. We could provide some guidance and hopefully avoid issues like this in the future.

The most immediate solution is to not upgrade Helm until we identify when we can get this rolled out.

This is a somewhat scary option. ACR extended the index.yaml file rather than use the built-in annotations field at the same level as the extension. So, to avoid breaking the experience for your users due to a custom extension that no longer works (when there were other ways to approach this) you are suggesting people don’t install a security update.

In the future, we are happy to help with Helm extension points (either in explaining them or discussing adding them). We would like to see ACR not end up in a broken situation where you’re telling people not to apply a Helm security release.

Engineering has confirmed Azure Public/Commercial is complete.

• Azure China and Azure Gov clouds will rollout over the next two days.
• ACR Helm Repo Security Advisory has been updated

This also fails now for Azure Container Registry helmcharts with: error unmarshaling JSON: while decoding JSON: json: unknown field “acrMetadata”

Helm 2.16.11 is affected, too.

Per the recent CNCF-sponsored security audit, the Helm maintainers needed to apply a security fix that enabled strict handling of YAML data in the Helm client.

It’s a little unfortunate that this security fix didn’t go through an embargo period to understand the impact and fix before it got rolled out. But, we are looking into it now as we do have a lot of usage. That said, to roll out a change to all Azure regions will take some time.

The helm repo behavior impacted here was an experiment that led to the OCI Artifacts work.

As noted in our ACR Roadmap, we do suggest users move to Helm 3 Registry support as it provides more scalability & reliability as all OCI compliant registries support this approach.

@jkroepke Good call. I will push a backport PR for Helm 2 shortly.

@SteveLasker, I changed the title back and closing this now.

If the Helm maintainers wish to make changes to the issue, please feel free to do so.

Woops, thanks DK. Fixed now. I’ll confirm the rollout is complete for Azure Commercial cloud regions shortly.

A quick update for those tracking.

ACR Helm Repo Security Advisory

Full details at: ACR Helm Repo Security Advisory

Hey folks, An official update, with some learnings applied:

• The ACR engineering team is validating a fix that has been deployed for canary testing on September 22, 2020
• Assuming testing goes as expected, following Azure safe deployment practices, regional deployments will start September 23, 2020
• Deployments to Azure Commercial should complete by the start of business, Monday, September 28, 2020 - please check here for final confirmation and status updates.
• Azure government clouds and Azure China should complete by September 30, 2020

Learnings:

• We have provided full insight into the issue and learnings we’re applying to an ACR Advisories page
• We also encourage the Helm community to weigh in on your views on how security notifications should be handled by Helm and other CNCF projects.

@jkroepke #8764 pushed for Helm v2 fix.