kustomize: Performance degradation starting with v3.8.6

Starting with v3.8.6, a serious performance degradation is observed even on a very basic setup. Version 3.8.5 (and older) was fine.

Demonstration:

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- service.yaml

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: "test"
spec:
  ports:
  - port: 80
    name: http
    targetPort: 80

I performed a test run with versions 3.8.5, 3.8.6, 3.8.7, 3.8.8, 4.2.0. While v3.8.5 is able to build the manifest in about 35 ms, v3.8.6 and v3.8.7 spend about 850 ms, and then starting with v3.8.8 and up to the latest v4.2.0 it takes about one second. Output is identical in each run, so I filtered it out to make it easier to read.

$ for ver in 3.8.5 3.8.6 3.8.7 3.8.8 4.2.0;do echo;./kustomize-v$ver version;time ./kustomize-v$ver build &>/dev/null;echo;done

{Version:kustomize/v3.8.5 GitCommit:4052cd4fd8c76a17b5f64e32509f3fba9713fe75 BuildDate:2020-10-08T02:45:59Z GoOs:linux GoArch:amd64}

real    0m0.037s
user    0m0.033s
sys     0m0.013s


{Version:kustomize/v3.8.6 GitCommit:c1747439cd8bc956028ad483cdb30d9273c18b24 BuildDate:2020-10-29T23:07:50Z GoOs:linux GoArch:amd64}

real    0m0.863s
user    0m0.926s
sys     0m0.021s


{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:14:14Z GoOs:linux GoArch:amd64}

real    0m0.853s
user    0m0.906s
sys     0m0.032s


{Version:kustomize/v3.8.8 GitCommit:72262c5e7135045ed51b01e417a7e72f558a22b0 BuildDate:2020-12-10T18:05:35Z GoOs:linux GoArch:amd64}

real    0m0.931s
user    0m1.024s
sys     0m0.013s


{Version:kustomize/v4.2.0 GitCommit:d53a2ad45d04b0264bcee9e19879437d851cb778 BuildDate:2021-06-30T22:49:26Z GoOs:linux GoArch:amd64}

real    0m0.973s
user    0m1.028s
sys     0m0.017s

It must be noted that I do not observe any performance degradation on big setups, when bases and more resources are included. Kustomize apparently spends that extra second once on startup, and then works fast processing everything. It is still a problem in my case, however, because I have a lot of microservices that have to be processed using multiple invocations of kustomize.

Platform Linux/amd64.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 17 (5 by maintainers)

Commits related to this issue

Kustomize 4.4.0 fixes the performance issues with kustomize build https://github.com/kubernetes-sigs/kustomize/issues/4100 This version mismatches that embedded with kubectl, but let's assume that i... — committed to Yolean/ystack by solsson 3 years ago

Most upvoted comments

I can confirm that the issue is now resolved.

With kustomize built from HEAD:

$ time ~/go/bin/kustomize build
apiVersion: v1
kind: Service
metadata:
  name: test
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80

real    0m0.021s
user    0m0.021s
sys     0m0.003s

shapirus on Sep 10, 2021

Another option is to re-add a hardcoded list of namespaceable resources. We may be able to automatically generate it (so that we don’t have to manually maintain the list) with the scripts that fetch OpenAPI data.

However, OpenAPI parsing being a bottleneck is a common theme so using gop, proto, or some other type of encoding scheme that is quicker to parse is probably a good idea.

I would gratefully accept any PRs that are able to achieve an improvement in OpenAPI parsing time (with benchmark tests). One requirement however would be that the public kyaml interface would have to stay the same, as there are library consumers outside of kustomize that use it.

I am seriously considering submitting a PR to revert the commit that removed the hardcoded list for the sake of performance.

natasha41575 on Aug 9, 2021

There is an improvement with Kustomize 4.4.0, but not as large as I expected.

I just tested the new kustomize release 4.4.0 against 4.3.0 and also against my hack as decribed in https://github.com/kubernetes-sigs/kustomize/issues/4100#issuecomment-895590402.

My pipeline consists of about 20 calls to kustomize build.

Result:

4.3.0: ~19 seconds
4.4.0: ~11 seconds
my hacked 4.3.0 build: ~4 seconds

This is strange, because I’ve reviewed https://github.com/kubernetes-sigs/kustomize/pull/4152 and it does essentially the same as my hack. Maybe there is a new performance degradation introduced with 4.4.0?

ChristianCiach on Sep 27, 2021