helmfile: `.Release.Labels` are not available within values templates

@voron I believe you’ve captured everything from what I imagined! Thanks for confirming.

I might change the inherit field to be a dict, and use the nested except field to support selectively inheriting only a subset of fields for more flexible nesting. WDYT?

templates:
  exchange:
    values:
    - default.yaml
    labels:
      foo
  private-exchange-svc:
    # Let helmfile merge the whole release spec and the labels field
    inherit:
      template: exchange
      # Ignore the `defalut.yaml` from the `exchange` template a.k.a replace `values` field with the same field define in this template.
      # This works like "Hey, I wanna this private-exchange-svc template to replace the whole `values` field from
      # the `exchange` template, so that this template's `values` contain `exchange.yaml` only." 
      except:
      - values
    labels:
      app: private-exchange-svc
    values:
    - exchange.yaml
releases:
  - name: private-exchange-svc-test-0
    # Let helmfile merge the whole release spec and the labels field
    inherit:
      template: private-exchange-svc
    labels:
      exchange: test

I think it will be OK if we use inherit option, but does it work without nested templates? As i understood, we will need to just simply replace yaml anchor ref sign <<: * with inherit in our release definitions?

@VladStarr I presume that’s correct! The key idea is that the original YAML spec does not support merging a single dict field (labels in our use case) across multiple YAML dicts using YAML anchors, regardless of whether they are “nested” or not. We instead support the use-case via the new feature(inherit).

To make the transition to the new feature smooth, we’ll be reverting to go-yaml v2 for the upcoming Helmfile v0.150.0. So anyone stuck in pre-v0.147.0 due to this bug should be able to upgrade to v0.150.0 without changing anything in your helmfile.yaml.

We’ll shortly add the new feature, and make it available to you in perhaps Helmfile v0.151.0 or even v0.150.0, so that you can gradually migrate to the new feature.

Maybe there is some other way to adjust our templates to keep it dry, but for now we use nested templates with labels merge between nested levels as well as labels merge/override from a release side. I suppose inherit directive should solve dry issues we cannot resolve with the pure yaml v3 now.

@VladStarr @voron FYI, #606 has been merged. You can try building helmfile from our main branch to give inherit a shot today 🎉

Hey @mumoshu, thank you for reply. Our use case with @voron is the same. It is described in my comment.

I think it will be OK if we use inherit option, but does it work without nested templates? As i understood, we will need to just simply replace yaml anchor ref sign <<: * with inherit in our release definitions?

If this workaround is going to work with go-yaml v3, it will be great

@VladStarr @voron Thanks a lot for clarifying! Let met first say that your use case alone looks totally valid to me.

What’s complicating the resolution here is that your use relied on a behavior of a specific yaml parser implementation(go-yaml v2) which does not conform to the yaml spec.

We should keep backward compatibility. But is treating a bug like a spec to keep the existing behavior the right thing to do…?

I’d be happy if we could still consider it a bug. Instead of reverting the parser to the prev version, can we add another feature to support your use case?

What if we add a new field, inherit or base or whatever that makes sense, to inherit fields from a template?

With it, I’d assume @VladStarr’s previous example could be written like this:

templates:
  exchange: &exchange
    secrets:
      - secrets/redis.yaml
      - secrets/rabbitmq.yaml
      - secrets/{{`{{ .Release.Labels.app }}`}}.yaml
      - secrets/{{`{{ .Release.Labels.app }}`}}-{{`{{ .Release.Labels.exchange }}`}}.yaml
      - secrets/{{`{{ .Release.Name }}`}}.yaml
    valuesTemplate:
      - ../common/values/{{`{{ .Release.Labels.app }}`}}.yaml.gotmpl
      - ../common/values/{{`{{ .Release.Labels.app }}`}}-{{`{{ .Release.Labels.exchange }}`}}.yaml.gotmpl
      - values/{{`{{ .Release.Labels.app }}`}}.yaml.gotmpl
      - values/{{`{{ .Release.Labels.app }}`}}-{{`{{ .Release.Labels.exchange }}`}}.yaml.gotmpl
      - values/{{`{{ .Release.Name }}`}}.yaml.gotmpl
  private-exchange-svc: &private-exchange-svc
    <<: *exchange
    labels:
      app: private-exchange-svc
      
releases:
  - name: private-exchange-svc-test-0
    # THE ONLY CHANGE YOU NEED
    inherit: private-exchange-svc
    labels:
      exchange: test

@VladStarr’s example needs a deep merge on labels only once, so as the use of inherit.

I presume @voron’s example might look a bit more complex. Hey @voron, how many “labels” fields across the templates and the release needs to be deep merged in your configuration? For Vlad’s it was two. If you need three or more, it it would look like this:

templates:
  exchange:
    labels:
      foo
  private-exchange-svc:
    # Let helmfile merge the whole release spec and the labels field
    inherit: exchange
    labels:
      app: private-exchange-svc
      
releases:
  - name: private-exchange-svc-test-0
    # Let helmfile merge the whole release spec and the labels field
    inherit: private-exchange-svc
    labels:
      exchange: test

The whole idea of labels usage for us is a deep merge of labels keys&values between templates and releases , similar to the release values from valuesTemplate + values, the case @VladStarr is showed above. We use tens of releases with just a single difference - one label, while the release template is the same. And this release template is nested from another template to keep things “dry”.

ping pong

@yxxhero thank you for your explanation & suggestion, I have tested it and it seems to work with latest helmfile. However, I understand I now need to duplicate all labels for each release and can’t trust they will be taken from template labels as before if missing.

@sisrael-dn @yxxhero If we were to allow Release.Labels as a template parameter for other fields like the release name and release namespace, how would one do the opposite? For instance, how can we enable folks who want to use Release.Labels as a parameter, while enabling others to use Release.Namespace or Release.Name as a template parameter to dynamically render the Release.Labels…? We’re basically supporting the latter pattern today I think. Fundamentally it’s impossible to support both at the same time without parsing Go template expressions embedded in each release field…?

@sisrael-dn I will work on this.