kustomize: Replacements do not work with fields that were set in a parent kustomization.
Describe the bug
replacements does not work if the source resource comes from above the replacements in the dependency graph. This is the case whether kustomizations come as Component or Kustomization files.
Note, the example below uses commonAnnotations to simplify the demonstration, however the same behavior occurs with static resources (such as a configmap.yaml file that sets an annotation statically) – and it occurs with other fields, not solely metadata.annotations.
The use case:
I have a top level kustomization.yaml for each deployed environment (dev/prod/test/etc) that adds commonAnnotations to all of the resources in its tree. It then does a resources: [ ../k8s/base/ ], and in the shared kustomization, there’s a replacements entry to, for example, pull a DNS_HOSTNAME from an annotation and put it into a configmap data field, and other various resource spots it needs to go.
This worked well with vars, which seems to render all of the manifests then do the var replacements afterwards. But with replacements, kustomize does not see any fields that are created above itself.
Files that can reproduce the issue replacementsinheritance_test.go.txt
a/kustomization.yaml
kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1
commonAnnotations:
replacement: child
configMapGenerator:
- name: dns-config
options:
disableNameSuffixHash: true
- name: deployment-config
options:
disableNameSuffixHash: true
replacements:
- source:
kind: ConfigMap
name: dns-config
fieldPath: metadata.annotations.replacement
targets:
- select:
kind: ConfigMap
name: deployment-config
fieldPaths:
- data.REPLACEMENT
options:
create: true
b/kustomization.yaml
kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1
commonAnnotations:
replacement: parent
resources:
- ../a
kustomize build b
Note that all of the resources have the annotation parent, but the replacements does the replacement using the prior annotations. Additionally, if you comment out the commonAnnotations from a/kustomization.yaml and then run kustomize build b, it will panic with panic: runtime error: invalid memory address or nil pointer dereference.
===== ACTUAL END ==========================================
EXPECTED ACTUAL
-------- ------
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
annotations: annotations:
replacement: parent replacement: parent
name: dns-config name: dns-config
--- ---
apiVersion: v1 apiVersion: v1
data: data:
X REPLACEMENT: parent REPLACEMENT: child
kind: ConfigMap kind: ConfigMap
metadata: metadata:
annotations: annotations:
replacement: parent replacement: parent
name: deployment-config name: deployment-config
Kustomize version
{Version:kustomize/v4.2.0 GitCommit:d53a2ad45d04b0264bcee9e19879437d851cb778 BuildDate:2021-07-01T00:44:28+01:00 GoOs:darwin GoArch:amd64}
Platform
Darwin 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec 2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 29
- Comments: 19 (8 by maintainers)
Totally agree
In this issue we’ve concluded that replacements will not be able to reference resources outside of the current kustomization stack, but tracking name references set by replacements should be supported. We can track this issue in https://github.com/kubernetes-sigs/kustomize/issues/4524 and https://github.com/kubernetes-sigs/kustomize/issues/4475.
And realistically – if I want to use a field from a resource to put into another resource, I can not think of any cases I’d want to use any value other than the FINAL RENDERED value of it.
If you see my example above, we take the annotation from a configmap, and put that into the value of another configmap. But then after all that’s done, the original configmap annotation is changed in the final product.
In what world does it make sense to read a field to reference somewhere else, when the source field isn’t yet final rendered value of that field? It just seems to be ripe for having semi-predictable inconsistencies.
Given that vars are deprecated it seems quite important to fix this
This kind of kills the point of using components to add something meaningful without having to create yet another transformer in the parent that patches the output of the component.
@natasha41575 isn’t the point of components to access the RA of parents?
This is from the KEP
The only issue here is that replacements inside the components are accessing the not yet transformed base resources.
I’m suffering similar problem with
replacements. In my case, there’s a container environment variable in a pod spec that refers to a Service by its namespace and name as ns/name. I tried to usereplacementsto keep this environment variable value up to date with changes to the Service’s namespace and name, sincenameReferencecan’t handle that that composite value.replacementsworks to a point, but it runs “too early.” If I change the namespace of the Service at any point “higher” in the kustomize resource/component graph,replacementsfails to take the new namespace into account. This happens even when I situate thereplacementsfield in a Component.One of the reasons (though not the primary one) for our desire to replace
varswithreplacementsis becausevarsbreaks the encapsulation of each kustomization. The kustomize model is that it processes each kustomization layer one at at a time, from the bases up to the overlays. In this way, each layer is a step in the pipeline.varswas the only transformer that was an exception; every other transformer honors this encapsulation and runs in the layer that is defined. Having an exception to the rule breaks the pipeline model and complicates things greatly.Quoting from https://github.com/kubernetes-sigs/kustomize/issues/2052#issuecomment-762408873:
This is what we are trying to avoid. A user should be able to reuse bases and overlays without concerns about effects on other kustomizations in their pipeline.
I am open to updating name references if the replacement source came from a resource name. This would work similarly to the name prefix and suffix transformers which are able to update name references to bases in overlays. However, I think there needs to be a very, very strong case to break kustomization encapsulation more generally.
Hi,
I’m actually facing the same problem. I’m using components to add (or not) an ingress resource to my base and construct my ingress paths with the namespace name and a label.
Here is an example :
When i’m calling my base, with my component and that i add my namespace and labels with the kustomize CLI i get an error like :
While at the same time, using vars i get the result i want :
I’ll go with the vars for the moment even if it’s planned to deprecate. But i would like some advice to handle such things with replacements.
Kustomize version
{Version:kustomize/v4.4.1 GitCommit:b2d65ddc98e09187a8e38adc27c30bab078c1dbf BuildDate:2021-11-11T23:36:27Z GoOs:linux GoArch:amd64}In my case, I was trying to establish an invariant down in the base, where the Service referred to by this environment variable (in a Deployment pod spec) is defined. Much like name references in kustomize, I was trying to express that this environment variable’s value should always “point” at this Service, so that if someone places the Service into a different namespace or adjusts its name, the environment variable’s value would follow along accordingly.
As
replacementsis implemented today, It’s not possible to state that invariant at any level below the one in which we change the namespace or name of that Service. @natasha41575’s https://github.com/kubernetes-sigs/kustomize/issues/4034#issuecomment-876618258 suggests that one day it might work more like name references.