kustomize: Replacements do not work with fields that were set in a parent kustomization.

Describe the bug

replacements does not work if the source resource comes from above the replacements in the dependency graph. This is the case whether kustomizations come as Component or Kustomization files.

Note, the example below uses commonAnnotations to simplify the demonstration, however the same behavior occurs with static resources (such as a configmap.yaml file that sets an annotation statically) – and it occurs with other fields, not solely metadata.annotations.

The use case: I have a top level kustomization.yaml for each deployed environment (dev/prod/test/etc) that adds commonAnnotations to all of the resources in its tree. It then does a resources: [ ../k8s/base/ ], and in the shared kustomization, there’s a replacements entry to, for example, pull a DNS_HOSTNAME from an annotation and put it into a configmap data field, and other various resource spots it needs to go.

This worked well with vars, which seems to render all of the manifests then do the var replacements afterwards. But with replacements, kustomize does not see any fields that are created above itself.

Files that can reproduce the issue replacementsinheritance_test.go.txt

a/kustomization.yaml

kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1

commonAnnotations:
  replacement: child
 
configMapGenerator:
- name: dns-config
  options:
   disableNameSuffixHash: true
- name: deployment-config
  options:
    disableNameSuffixHash: true

replacements:
- source:
    kind: ConfigMap
    name: dns-config
    fieldPath: metadata.annotations.replacement
  targets:
  - select:
      kind: ConfigMap
      name: deployment-config
    fieldPaths:
    - data.REPLACEMENT
    options:
      create: true

b/kustomization.yaml

kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1

commonAnnotations:
  replacement: parent

resources:
- ../a

kustomize build b

Note that all of the resources have the annotation parent, but the replacements does the replacement using the prior annotations. Additionally, if you comment out the commonAnnotations from a/kustomization.yaml and then run kustomize build b, it will panic with panic: runtime error: invalid memory address or nil pointer dereference.

===== ACTUAL END ==========================================
   EXPECTED                      ACTUAL
   --------                      ------
   apiVersion: v1                apiVersion: v1
   kind: ConfigMap               kind: ConfigMap
   metadata:                     metadata:
     annotations:                  annotations:
       replacement: parent           replacement: parent
     name: dns-config              name: dns-config
   ---                           ---
   apiVersion: v1                apiVersion: v1
   data:                         data:
X    REPLACEMENT: parent           REPLACEMENT: child
   kind: ConfigMap               kind: ConfigMap
   metadata:                     metadata:
     annotations:                  annotations:
       replacement: parent           replacement: parent
     name: deployment-config       name: deployment-config

Kustomize version

{Version:kustomize/v4.2.0 GitCommit:d53a2ad45d04b0264bcee9e19879437d851cb778 BuildDate:2021-07-01T00:44:28+01:00 GoOs:darwin GoArch:amd64}

Platform

Darwin 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec 2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 29
  • Comments: 19 (8 by maintainers)

Most upvoted comments

Any updates on this issue? I was considering using vars after watching, what is apparently an old kubecon talk, and sounds like replacements might not have the same capabilities?? Thanks.

Totally agree

In this issue we’ve concluded that replacements will not be able to reference resources outside of the current kustomization stack, but tracking name references set by replacements should be supported. We can track this issue in https://github.com/kubernetes-sigs/kustomize/issues/4524 and https://github.com/kubernetes-sigs/kustomize/issues/4475.

And realistically – if I want to use a field from a resource to put into another resource, I can not think of any cases I’d want to use any value other than the FINAL RENDERED value of it.

If you see my example above, we take the annotation from a configmap, and put that into the value of another configmap. But then after all that’s done, the original configmap annotation is changed in the final product.

In what world does it make sense to read a field to reference somewhere else, when the source field isn’t yet final rendered value of that field? It just seems to be ripe for having semi-predictable inconsistencies.

Given that vars are deprecated it seems quite important to fix this

This kind of kills the point of using components to add something meaningful without having to create yet another transformer in the parent that patches the output of the component.

@natasha41575 isn’t the point of components to access the RA of parents?

This is from the KEP

A kustomization that is marked as a Component has basically the same capabilities as a normal kustomization. The main distinction is that they are evaluated after the resources of the parent kustomization (overlay or component) have been accumulated, and on top of them. This means that:

The only issue here is that replacements inside the components are accessing the not yet transformed base resources.

I’m suffering similar problem with replacements. In my case, there’s a container environment variable in a pod spec that refers to a Service by its namespace and name as ns/name. I tried to use replacements to keep this environment variable value up to date with changes to the Service’s namespace and name, since nameReference can’t handle that that composite value.

replacements works to a point, but it runs “too early.” If I change the namespace of the Service at any point “higher” in the kustomize resource/component graph, replacements fails to take the new namespace into account. This happens even when I situate the replacements field in a Component.

One of the reasons (though not the primary one) for our desire to replace vars with replacements is because vars breaks the encapsulation of each kustomization. The kustomize model is that it processes each kustomization layer one at at a time, from the bases up to the overlays. In this way, each layer is a step in the pipeline.

vars was the only transformer that was an exception; every other transformer honors this encapsulation and runs in the layer that is defined. Having an exception to the rule breaks the pipeline model and complicates things greatly.

Quoting from https://github.com/kubernetes-sigs/kustomize/issues/2052#issuecomment-762408873:

This seems to be one of the biggest pain-points of using kustomize today, as it breaks the encapsulation of the kustomization and prevents the user from composing bigger kustomizations out of smaller ones.

This is what we are trying to avoid. A user should be able to reuse bases and overlays without concerns about effects on other kustomizations in their pipeline.

As replacements is implemented today, It’s not possible to state that invariant at any level below the one in which we change the namespace or name of that Service. @natasha41575’s #4034 (comment) suggests that one day it might work more like name references.

I am open to updating name references if the replacement source came from a resource name. This would work similarly to the name prefix and suffix transformers which are able to update name references to bases in overlays. However, I think there needs to be a very, very strong case to break kustomization encapsulation more generally.

Hi,

I’m actually facing the same problem. I’m using components to add (or not) an ingress resource to my base and construct my ingress paths with the namespace name and a label.

Here is an example :

  • ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myIngress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: namespace-app.example.com
    http:
      paths:
      - backend:
          service:
            name: myService
            port:
              name: http
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - namespace-app.example.com
  • kustomize.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
- ingress.yaml

replacements:
- source:
    kind: Deployment
    name: myDeployment
    fieldPath: metadata.namespace
  targets:
  - select:
      name: myIngress
      kind: Ingress
    fieldPaths:
    - spec.rules.0.host
    - spec.tls.0.hosts.0
    options:
      delimiter: '-'
      index: 0
- source:
    kind: Deployment
    name: myDeployment
    fieldPath: metadata.labels.app
  targets:
  - select:
      name: myIngress
      kind: Ingress
    fieldPaths:
    - spec.rules.0.host
    - spec.tls.0.hosts.0
    options:
      delimiter: '-'
      index: 1

When i’m calling my base, with my component and that i add my namespace and labels with the kustomize CLI i get an error like :

fieldPath `metadata.namespace\` is missing for replacement source ~G_~V_Deployment|~X|node-dpl:metadata.namespace

While at the same time, using vars i get the result i want :

  • ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myIngress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: $(NAMESPACE)-$(APP_NAME).example.com
    http:
      paths:
      - backend:
          service:
            name: myService
            port:
              name: http
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - $(NAMESPACE)-$(APP_NAME).example.com
  • kustomize.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
- ingress.yaml

vars:
- name: APP_NAME
  objref:
    kind: Deployment
    name: myDeployment
    apiVersion: apps/v1
  fieldref:
    fieldpath: metadata.labels.app
- name: NAMESPACE
  objref:
    kind: Deployment
    name: myDeployment
    apiVersion: apps/v1
  fieldref:
    fieldpath: metadata.namespace

I’ll go with the vars for the moment even if it’s planned to deprecate. But i would like some advice to handle such things with replacements.

Kustomize version {Version:kustomize/v4.4.1 GitCommit:b2d65ddc98e09187a8e38adc27c30bab078c1dbf BuildDate:2021-11-11T23:36:27Z GoOs:linux GoArch:amd64}

In my case, I was trying to establish an invariant down in the base, where the Service referred to by this environment variable (in a Deployment pod spec) is defined. Much like name references in kustomize, I was trying to express that this environment variable’s value should always “point” at this Service, so that if someone places the Service into a different namespace or adjusts its name, the environment variable’s value would follow along accordingly.

As replacements is implemented today, It’s not possible to state that invariant at any level below the one in which we change the namespace or name of that Service. @natasha41575’s https://github.com/kubernetes-sigs/kustomize/issues/4034#issuecomment-876618258 suggests that one day it might work more like name references.