kustomize: Kustomize resource ordering regression

the original motivation for this has become somewhat moot since the community has moved away from declarative things and into imperative. e.g. istio no longer supports yaml prometheus et al operators now create their crd dynamically.

this leads to a race condition, you cannot apply a system w/ a yaml doc to install prometheus followed by one that sets up a servicemonitor on some other thing. No ordering can fix this (since the order is correct). and, the system will not reconcile to the desire state (since the servicemonitor is rejected as an unknown type).

What is the solution? I dunno, does one call until kustomize; sleep 1; done ?

Did an issue for option 2 get filed?

ack, thanks.

Here’s a list of ordering changes over the last ~2 years

Mostly insertions (i.e. specifying order for a particular resource where there wasn’t an order before).

• 18 Sep 2020 #3009 put endpoints before service
• 05 Aug 2020 #2796 Order PersistentVolume before Deployment
• 12 May 2020 #2459 Relative Order of Mutating/Validating WebhookConfigs
• 20 Aug 2019 #1437 add PriorityClass to the order list
• 18 July 2019 #1369 add ResourceQuota to the order list
• 23 May 2019 #1104 Order ValidatingWebhookConfig last
• 15 May 2019 #1074 Update order of resources to include psps
• 20 May 2019 #1037 Apply LimitRange resources before workloads

MutatingWebhookConfiguration was moved from orderFirst to orderLast about a year ago in #2459 ValidatingWebhookConfiguration was moved to orderLast about two years ago in #1104

#3803 would move them back to orderFirst.

It’s been pretty stable for a year, so making a change now may have consequences for many users. As discussed above, it’s impossible for kustomize to own the answer to ‘correct’ ordering. So let’s close #3803, and leave legacy alone.

Some other options are:

1. kustomize honors --reorder none to suppress this sort, and obey FIFO. This works right now.
2. we make the existing legacy sorter accept configuration from the kustomization file. user could specify orderFirst and orderLast in that file (would need someone to write the PR for this).
3. user can write a sorting transformer plugin, and add that to their kustomization file as the last transformer to run (this is possible now, via various mechanisms - exec plugin, containerized plugin - but is the most work and slowest to execute).

If I understand correctly, the current (default) promise is: don’t care about ordering, kustomize will do it for you.

Where is that promise made? It must be retracted. It’s literally impossible right now to make one ordering that can both deploy a working stack to a cluster and universally works for everyone’s custom resources now and in the future.

Generally one needs a grander mechanism which can apply some things, waits for ready, apply more things, etc. Various things are moving in this direction, and would encapsulate kustomize or some other editor.

Thanks for the input @monopole! About:

People can order their inputs to get the desired output order, which may be the only answer. There’s no one ordering that makes sense for everyone, since no ordering can take custom resources into account.

This may indeed be the case, but it’s a big change, since it effectively changes the current promise of kustomize to manifests developers. If I understand correctly, the current (default) promise is: don’t care about ordering, kustomize will do it for you.

@mgoltzsche, @yanniszark do you agree? Someone will come along later and change it again. I’d like to make none the default value of --reorder, but it would break a bunch of people, and they’d have to reorder their configs. We could deprecate it over the course of a year or so. WDYT?

May I suggest the following:

• Keep this issue for fixing the regression in legacy ordering. I believe the original reason for ordering the webhooks last was not actually a valid reason, as described in the first comment. Webhooks should be first, to ensure they can validate/mutate subsequent resources. I have created PR https://github.com/kubernetes-sigs/kustomize/pull/3803 for this reason.
• Continue the more long-term discussion on what is the best way forward long-term for kustomize, in terms of reordering resources, in #821

Thanks! Discussing this on the earliest issue you mentioned #821

People can order their inputs to get the desired output order, which may be the only answer.

There’s no one ordering that makes sense for everyone, since no ordering can take custom resources into account.