Summary

This is an attempt to track kustomize’s performance degradation (and improvement) for several test cases, each working with a minimal set of manifests aiming to utilize a specific set of kustomize’s subroutines.

TLDR

patchesStrategicMerge is the only thing that is still slow.

Benchmark description

The tests are as follows.

1. A basic Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: test
spec:
  replicas: 1

+

kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1

resources:
- deployment.yaml

2. Same Deployment plus a JSON6902 patch:

kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1

resources:
- deployment.yaml

patches:
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 2
  target:
    group: apps
    version: v1
    kind: Deployment
    name: test

3. Same Deployment plus a strategic merge patch:

kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1

resources:
- deployment.yaml

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
     name: test
    spec:
      replicas: 2

4. A resource of an “unknown” kind

(can be e.g. SealedSecret in a real application):

apiVersion: apps/v1
kind: UnknownKind
metadata:
 name: test
spec:
  replicas: 1

It is worth mentioning that saving the patches in separate files made no difference compared to having them inline in kustomization.yaml. There is no difference between the resources having and not having the namespace attribute, either.

Running the benchmark

I have prepared a set of shell scripts designed to be run in a Docker container that execute a user-configurable number of iterations (default: 200, can be set via ITERATIONS env variable) for each of the above cases for several versions (and git revisions) of kustomize starting with v3.5.4 and ending with v5.2.1.

The mentioned benchmark supports building for linux/amd64 and linux/arm64 (meaning it should also be possible to build and run it on an ARM Mac host, but I have no hardware to verify this), and runtime has been tested on both platforms.

A zip file is attached to this ticket which contains manifests, helper scripts (beware: not necessarily safe to run outside a docker container), and Dockerfile. The versions/revisions to download or build are specified in Dockerfile, see comments inside for more information. Also see the end of this comment for an updated version.

Build and run in one command:

docker run -it $(docker build -q .)

(but it’s better to run a simple docker build . first to make sure it builds properly – unlike the one above, this will produce output.)

Test results

(also see updated results with more versions in the comments below.)

### Starting benchmark on Linux x86_64

# kustomize versions: 3.5.4 3.7.0 3.8.0 1c6481d0 00f0fd71 3.8.3 3.8.6 4.1.2 4.3.0 4.4.0 5.0.0 5.1.0 5.2.1
# iterations per test: 200
# tests: 1_no-patches 2_patches-json6902 3_patches-strategic-merge 4_no-patches-unknown-kind 
# time unit: seconds

             test: 1   test: 2   test: 3   test: 4
    v3.5.4      4.47      4.53      4.42      4.24
    v3.7.0      4.54      4.59      4.44      4.78
    v3.8.0      4.73      4.60     20.03      4.69
  1c6481d0      4.90      4.86     18.76      4.63
  00f0fd71      4.85      4.98     95.63      4.62
    v3.8.3      4.96      5.03    106.87      4.64
    v3.8.6    104.41    105.33    104.61    104.73
    v4.1.2    120.12    119.76    119.68    119.94
    v4.3.0    124.38    124.89    124.41    124.37
    v4.4.0      1.81      1.96    124.81    124.42
    v5.0.0      1.01      1.13     11.65     11.70
    v5.1.0      1.01      1.06     11.51     11.26
    v5.2.1      1.06      1.08     11.56      1.01

### Starting benchmark on Linux aarch64

# kustomize versions: kustomize/v3.5.4 kustomize/v3.7.0 kustomize/v3.8.0 1c6481d0 00f0fd71 kustomize/v3.8.3 3.8.6 4.1.2 4.3.0 4.4.0 5.0.0 5.2.1
# iterations per test: 100
# tests: 1_no-patches 2_patches-json6902 3_patches-strategic-merge 4_no-patches-unknown-kind 
# time unit: seconds

             test: 1   test: 2   test: 3   test: 4
    v3.5.4      4.61      4.41      4.33      4.29
    v3.7.0      4.98      4.79      4.78      4.84
    v3.8.0      4.81      4.87     18.38      4.87
  1c6481d0      5.01      5.43     18.73      4.90
  00f0fd71      5.05      5.14    113.08      4.89
    v3.8.3      4.95      5.12    112.45      4.75
    v3.8.6    108.89    108.59    109.17    108.97
    v4.1.2    124.71    125.93    125.55    124.82
    v4.3.0    127.36    127.53    127.15    128.09
    v4.4.0      1.92      1.99    127.89    127.31
    v5.0.0      1.34      1.45     11.82     11.94
    v5.2.1      1.32      1.41     11.84      1.34

As we can see, there was a slight (but reproducibly visible) degradation between 3.5.4 and 3.8.0 in tests 1, 2, 4, and a significant 5x degradation of test 3 (PSM) starting with 3.8.0.

In addition to the continuing gradual degradation from version to version, revision 00f0fd71 introduced further 5x degradation in test 3 (PSM) compared to 3.8.0 (revision 1c6481d0 is shown here as it comes right before 00f0fd71), see #2987 for this specific one.

Version 3.8.3 is yet slower than 00f0fd71.

Version 3.8.6 was as slow as 3.8.3 in test 3 (PSM), but introduced a 20x degradation in the rest of the tests, making them as slow as PSM (#4100).

Then there was further 20% degradation in all tests by version 4.3.0.

Improvements started to come with 4.4.0, but tests 3 and 4 stayed unfixed.

Version 5.0.0 seems to have added a serious improvement in something that affected all test cases, but tests 3 (PSM) and 4 (unknown kind), even though becoming much faster than in previous releases, remained much slower than tests 1 and 2.

Version 5.2.1 fixed test 4 (unknown kind), it came with merging PR #5076, discussed in #4569.

Conslusion

The slow patchesStrategicMerge, even though it is much faster than in older releases, still remains, and it is probably the only significant performance issue that still requires close attention.

There is, apparently, some unoptimal code specific to PSM. As we can see, there were improvements that made all modes more effective, PSM included, but relatively to the others it is still very slow.

Related issues

Here are some of the existing issues that gave hints as for what specific versions/revisions were to be tested:

• https://github.com/kubernetes-sigs/kustomize/issues/2987
• https://github.com/kubernetes-sigs/kustomize/issues/4100
• https://github.com/kubernetes-sigs/kustomize/issues/4569
• https://github.com/kubernetes-sigs/kustomize/issues/4940

…and the following might give yet another overall improvement?

• https://github.com/kubernetes-sigs/kustomize/issues/5084

Follow-up

I have added more tests to cover the Components functionality: 5) no base + plain Deployment as a Component; 6) plain Deployment in base + a JSON6902 patch as a Component; 7) same as 2, but with PSM.

Results:

Starting kustomize benchmark on Linux x86_64
kustomize versions: 
  3.5.4
  3.7.0
  3.8.0
  5.0.0
  5.2.1
iterations per test: 200
tests: 
  1_no-patches
  2_patches-json6902
  3_patches-strategic-merge
  4_no-patches-unknown-kind
  5_component-no-base-no-patches
  6_component-json6902-over-base
  7_component-PSM-over-base
time unit: seconds

             test: 1   test: 2   test: 3   test: 4   test: 5   test: 6   test: 7
    v3.5.4      4.42      4.55      4.59      4.41    (fail)    (fail)    (fail)
    v3.7.0      4.78      4.78      4.75      4.60      5.86      7.22      7.14
    v3.8.0      4.72      4.84     19.97      4.70      5.81      7.25     21.40
    v5.0.0      1.06      1.04     11.74     11.98      2.10      2.87     13.43
    v5.2.1      1.05      1.05     11.53      1.07      1.91      2.90     13.22

As we can see here, in 3.7.0, when Components were introduced, the basic case was only 1.25 times slower than the no-component plain-deployment case and the json6902 component test was 1.6 times slower than the respective no-component test.

In v5.2.1, however, this difference increased to 1.8x and 2.76x, respectively. This may indicate either that there are still possibilities for improvement in the Components code, or that the common overhead was reduced so much that what’s left are the Components’ genuine computational requirements which became more visible after shaving off the time spent on the common overhead.

It is also clear that both the component and no-component patchesStragegicMerge cases suffer from the same performance hit and should equally benefit from a single fix.

Not attaching my tests as a zip file any longer, since they are now available on github: https://github.com/shapirus/kustomize-benchmark-suite. There’s no readme, but if somebody is interested, there is a brief instruction in Dockerfile.