helm: 'helm dep build' requires repositories to be added, but 'helm dep update' doesn't

We discussed this on the Helm developer call today. The reason helm dep build doesn’t automatically add repos is due to security. In Helm v4 we plan to make helm dep update not automatically add them either. More details are at #9903.

We realize there is some inconvenience with having to add each repo. It’s an opt-in and awareness for security around supply chain attacks.

If there are questions around this I would be happy to try and answer them.

helm dep up should act the same as helm dep build in this respect. But, since this is already in Helm v3 this way we need to wait until Helm v4 to change it. That’s why it’s labeled this way.

I believe we should requires all repos to be added prior to updating from them. This is in the name of security. We want people who are running updates to know full well where they are coming from.

We have an url, we have a version, and we have a digest, please dont make it mandatory to add repositories as this makes things way more complicated. Security should be implemented by verifying proper digests/sha’s but not by someone adding repos.

If you don’t want to allow it by default, at least make a flag available. Otherwise this is causing a huge amount of pain for no good reason.

For the build case: We’ve got local charts that wrap upstream charts, we explicitly specify the upstream repository URL and version, it’s got the hash in the Chart.lock file. In CI I just want to run “helm dependency build” and have it download the dependencies I have verified already. Forcing me to write extra code to find all of the dependency urls and run “helm repo add blah1 <url>” for each one first is just busywork. It doesn’t improve our security at all.

For the update case: I know what repositories the chart will be pulling updates from - it’s the ones I’ve defined in Chart.yaml. I don’t want to pollute my global user namespace with repository definitions that I don’t want to use or reference directly. If this changes back to requiring me to add the repo first, I need to check through Chart.yaml, add all the repositories, do my update, remove them all again - this is busywork.

@mattfarina

what if someone successfully updated the Chart.lock file

If someone can do this, they already have access on a level allowing them to perform much more than editing the Chart.lock file.

I strongly agree with @mikebryant on this. You’re making things almost useless just because of a witch hunt.

This makes repository and dependency interaction more complicated without a good reason on top of those weren’t easy to implement in automation pipelines in the first place.

helm dep up should act the same as helm dep build in this respect. But, since this is already in Helm v3 this way we need to wait until Helm v4 to change it. That’s why it’s labeled this way.

I believe we should requires all repos to be added prior to updating from them. This is in the name of security. We want people who are running updates to know full well where they are coming from.

It’s really important that we take the security implications into consideration, since this clearly could be abused otherwise. I have not taken part in the discussions about this in the weekly meeting, so perhaps this has already been brought up. But I want to emphasize two things regarding this:

1. It is clear to me that a malicious chart author could inject a bad URL in the dependencies if we don’t require explicitly adding the repos before dep up or dep build. However, should that be part of the threat model? A malicious chart author might as well just rewrite the chart to do bad things, no need to change the dependencies. If we trust them on the templateing, why not trust them on the dependency URLs too?
2. If we do decide that the repos cannot be used without being added first, I want to propose that we introduce some mechanism that allows to automatically add them (either a flag or a separate command). Otherwise, this might become a major headache for CI use-cases.

Here is a more robust script that utilize yq.

if [ -f "./Chart.lock" ]; then
  yq --indent 0 '.dependencies | map(["helm", "repo", "add", .name, .repository] | join(" ")) | .[]' "./Chart.lock"  | sh --;
fi

Thanks for @olahouze

@pniederlag what if someone successfully updated the Chart.lock file to add a dependent chart that shouldn’t be there. And it’s out of an unexpected repo? In that case the digest would be expected. Yet, someone (or CI) trying to build the dependencies would catch the unexpected repository.

Security is should be layered and help catch successful attacks. Or, make it more difficult for attackers. There have been lots of supply chain attacks and many are coming to public light. I wouldn’t want to weaken the layers of security to help stop of catch supply chain attacks with Helm.

For anyone else who needs this to work, run this in CI before running build to workaround this issue:

i=0; for url in $(yq r Chart.yaml dependencies --tojson | jq -r .[].repository); do i=$((i+1)); helm repo add blah${i} ${url}; done

Hello

Here is a very simple script that we can use in this case When we do a helm dependency build, HELM will check the dependencies in the Chart.lock file (https://helm.sh/docs/helm/helm_dependency_build/)

You can use this script just before doing the helm dependency build in the CI/CD

if [ -f "./Chart.lock" ]; then cat ./Chart.lock | grep repository | awk '{print $2}' | while read -r line ; do helm repo add $line $line; done; fi

Exemple :

Job_A: stage: ‘lint’ script: - if [ -f “./Chart.lock” ]; then cat ./Chart.lock | grep repository | awk ‘{print $2}’ | while read -r line ; do helm repo add $line $line; done; fi - ‘helm dependency build’ - ‘helm lint .’ rules: - if: ‘$CI_PIPELINE_SOURCE == “push”’ when: ‘always’ allow_failure:

Calling this a witch hunt is way too harsh. It’s a design decision that some of us don’t agree with, not some sort of attack on individuals.

I just tried this with the latest from master and it is still an issue. https://github.com/helm/helm/issues/7214#issuecomment-613957411 doesn’t show the issue because in that example helm dep build is only run once. helm dep build succeeds when Chart.lock doesn’t exist, but fails once it does exist. That comment uses helm dep up as the second command, which runs fine. Example:

❯ ~/personal/helm/bin/helm repo list
Error: no repositories to show

❯ rm -f charts/prometheus/Chart.lock

❯ ~/personal/helm/bin/helm dep build charts/prometheus
Saving 1 charts
Downloading prometheus-operator from repo https://kubernetes-charts.storage.googleapis.com
Deleting outdated charts

❯ ~/personal/helm/bin/helm dep build charts/prometheus
Error: no repository definition for https://kubernetes-charts.storage.googleapis.com. Please add the missing repos via 'helm repo add'

❯ ~/personal/helm/bin/helm dep up charts/prometheus
Saving 1 charts
Downloading prometheus-operator from repo https://kubernetes-charts.storage.googleapis.com
Deleting outdated charts

I rebased the branch from https://github.com/helm/helm/pull/7242 against master and the changes in that branch make it work as expected:

❯ ~/personal/helm/bin/helm repo list
Error: no repositories to show

❯ rm -f charts/prometheus/Chart.lock

❯ ~/personal/helm/bin/helm dep build charts/prometheus
Saving 1 charts
Downloading prometheus-operator from repo https://kubernetes-charts.storage.googleapis.com
Deleting outdated charts

❯ ~/personal/helm/bin/helm dep build charts/prometheus
Saving 1 charts
Downloading prometheus-operator from repo https://kubernetes-charts.storage.googleapis.com
Deleting outdated charts

❯ ~/personal/helm/bin/helm dep up charts/prometheus
Saving 1 charts
Downloading prometheus-operator from repo https://kubernetes-charts.storage.googleapis.com
Deleting outdated charts

If there are questions around this I would be happy to try and answer them.

Can you talk through with me the reasoning that this is more secure?

Our company uses hundreds of helm charts, and they are all created via CI/CD, so maybe let’s limit the discussion to CI/CD created helm charts (since those are the majority of ones people depend upon).

Is the concern here that a hacker will be able to modify the helm chart location inside Chart.yaml dependencies section and not modify modify the chart dependency location specified by “help repo add X” ? This seems unlikely.

@hickeyma Yes for sure.