helm: Problems with randAlphaNum and upgrades

There are problems with using randAlphaNum in charts when upgrades occur. I’m sure this is working as expected right now, but is problematic and it should at least be documented if there is no alternate implementation right now.

If you consider the current stable/redis chart, if you don’t specify a password in your values when creating the chart a random one is created using the following data block for the secret…

data:
  {{- if .Values.redisPassword }}
  redis-password: {{ .Values.redisPassword | b64enc | quote }}
  {{- else }}
  redis-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{- end }}
{{- end -}}

So far so good. However, this value is recreated to a new random string whenever it is evaluated, which invalidates the secret - so anything which subsequently reads the secret using secretKeyRef will be given the incorrect password.

To reproduce is simple…

bash% helm install stable/redis
bash% kubectl get secret solid-ibis-redis -o jsonpath="{.data.redis-password}" | base64 --decode
YAILVFmyeK% 

bash% helm upgrade --set my.irrelevant.key=1 solid-ibis stable/redis
bash% kubectl get secret solid-ibis-redis -o jsonpath="{.data.redis-password}" | base64 --decode
Ky7iOZbC2J%

Now I can understand what is happening, but I’m interested how people should write their charts to avoid this problem. We often use mongo, redis and rabbitmq as subcharts and having a random password generated and shared across the various subcharts is pretty useful.

I was vaguely looking at hooks, but I could only create a secret on a pre-install hook, but this then gets left behind when the instance is deleted.

Any ideas…?

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 36
Comments: 30 (5 by maintainers)

Links to this issue

Question on helm chart secrets and if it is recommended for my use case

Commits related to this issue

Work around helm/helm#3053 — committed to PatrickLang/fabrikamfiber by deleted user 6 years ago
Allow to inject all credentials values via helm values and generate a random value only no value is defined This also prevent that the helm update to regenerate the values. See https://github.com/hel... — committed to ccojocar/django-DefectDojo by ccojocar 4 years ago

Most upvoted comments

For anyone who stumbles here from Google, here’s how I resolved this issue using the lookup function mentioned above:

apiVersion: v1
kind: Secret
metadata:
  name: foobar-secret
type: Opaque
{{- $previous := lookup "v1" "Secret" .Release.Namespace "foobar-secret" }}
data:
  {{- if $previous }}
  foobarPassword: {{ $previous.data.foobarPassword }}
  {{- else if .Values.foobarPassword }}
  foobarPassword: {{ .Values.foobarPassword | b64enc | quote }}
  {{- else }}
  foobarPassword: {{ randAlphaNum 40 | b64enc | quote }}
  {{- end }}

+31

codeclown on Feb 21, 2021

If someone feels like taking a crack at https://github.com/helm/helm/issues/3053#issuecomment-379826234, that would be a good feature to “pin” resources to a certain state which Tiller will not upgrade. I’d be happy to review PRs that provide this functionality.

I’d probably use a new annotation name, though. Something like:

annotations:
    "helm.sh/upgrade-policy": don't

😆

Jokes aside, "helm.sh/upgrade-policy": pin would be a great feature to have.

+21

bacongobbler on Sep 19, 2018

It seems the feature of keeping a resource only works with “helm delete” but not with “helm upgrade”

  annotations:
    "helm.sh/resource-policy": keep

That would fix the issue for reusing secrets or any other resource.

+11

beltran-rubo on Apr 9, 2018

try to use these annotations:

  annotations:
    “helm.sh/hook”: “pre-install”
    “helm.sh/hook-delete-policy”: “before-hook-creation”

reference: https://medium.com/nuvo-group-tech/move-your-certs-to-helm-4f5f61338aca

nullck on Aug 5, 2019

Maybe we should add a flag to exclude all secrets when upgrading ? Or maybe we could add a flag to exclude some ressources from upgrade based on k8s selectors or regexp pattern ?

Seems to be this would be enough for a first workaround.

jseguillon on Aug 9, 2018

Im not sure what the ultimate solution is but in this case I would generate the secrets outside and put them in the values.yaml. While not ideal, it is straightforward and less time consuming than hacking around the current constraints of the system. Its possible that the redis example is just a low friction installation method for evaluating the chart and production installations are intended to supply their secrets via values (the template does accommodate secrets from values.yaml).

jascott1 on Jan 8, 2018

Edit: This does not work: the secret is still deleted. No idea why it was there after my initial test.

A workaround is to combine .Release.IsInstall with “helm.sh/resource-policy”: keep:

{{- if .Release.IsInstall }}
---
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    "helm.sh/resource-policy": keep
type: Opaque
data:
 password: "{{ randAlphaNum 10 | b64enc }}"
{{- end }}

Drawback is that if the secret is deleted manually, it’s only re-created if the chart is re-installed.

micw on Nov 14, 2018

Maybe the solution is another type of hook or hook policy, something along the lines of “if-not-exists” where it always gets created if it isn’t there, but never gets updated if it is?

Perhaps the new lookup function introduced in Helm 3.1 could prove useful in this regard. https://helm.sh/docs/chart_template_guide/functions_and_pipelines/#using-the-lookup-function

bacongobbler on May 14, 2020

Just as a reminder, nullck’s solution works- so long as the secret never gets deleted/has to be recreated.

Maybe the solution is another type of hook or hook policy, something along the lines of “if-not-exists” where it always gets created if it isn’t there, but never gets updated if it is?

pbecotte on Dec 11, 2019

Is there any progress with this particular issue?

The use case at our end is simply, we don’t want the user to select a password for things. We want them completely random and more secure by reducing the number of people with access to the password.

aaronjpitty on Jun 15, 2018

Nullck’s referenced solution works indeed, but only for newly installing releases. If a template is added to the existing (and installed) helm chart pre-install hook will not work. Adding pre-upgrade hook will cause the same behavior as subj.

antonu17 on Jan 7, 2020

I have the same problem w/ the certificate function, e.g.: genSignedCert

On every upgrade a new certificate is generated.

http://masterminds.github.io/sprig/crypto.html

jkroepke on Sep 19, 2018

Using an annotation to prevent a resource being updated would go a long way, yes. That way in the unusual event when we’d want to recreate the secret, we’d delete it in K8S and then update the chart.

alexvicegrab on May 1, 2018