kubeconform: Could not find schema for CustomResourceDefinition

When validating apiextensions.k8s.io/v1 CustomResourceDefinition resources I get this error:

<file> - CustomResourceDefinition servicemonitors.monitoring.coreos.com failed validation: could not find schema for CustomResourceDefinition

Why is that? I thought this API was part of the Kubernetes API?

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 15
  • Comments: 15 (3 by maintainers)

Commits related to this issue

Most upvoted comments

I solved it by adding to schema-location and using Datree’s CRDs-catalog as it is mentioned in the Readme.

kubeconform -summary -output pretty  -schema-location default -schema-location "https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json"

Now as you can see everything is validated:

helm template -f apps/kube-prometheus-stack/values.yaml apps/kube-prometheus-stack| kubeconform -summary -output pretty  -schema-location default -schema-location "https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json" 
                                    
Summary: 105 resources found parsing stdin - Valid: 105, Invalid: 0, Errors: 0, Skipped: 0

This issue thread is perplexing because everyone is talking about a different thing.

When you have issue Validating a Custom Resource like ServiceMonitor or Certificate: Add this to your command line:

-schema-location "https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json"

When you have issues validating a Custom Resource Definition, which is a NATIVE Kubernetes Resource: Add this to your command line:

-schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{.NormalizedKubernetesVersion}}/{{.ResourceKind}}.json"

Your final command line should look like this:

kubeconform -output pretty -strict \
  -schema-location default \
  -schema-location "https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json" \
  -schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{.NormalizedKubernetesVersion}}/{{.ResourceKind}}.json"

Edit: Fixed below comment.

@CyDickey-msr you can also use {{.NormalizedKubernetesVersion}} in the template and define the version in kubeconform using -kubernetes-version.

The final arguments should look like this:

-kubernetes-version="your-version"
-schema-location default
-schema-location https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json
-schema-location https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{.NormalizedKubernetesVersion}}/{{.ResourceKind}}.json
-schema-location https://json.schemastore.org/{{.ResourceKind}}.json

I understand. The problem is due to the fact that the schema for CustomResourceDefinitions are not stored in the schema repository, most likely because of this

While I am quite interested in getting this to work it is likely to take me some time to figure out why this limitation is in there. If anyone wants to give this a shot before me 👍

Hi, thanks for the quick answer.

I’ve tried it out that workflow earlier today, it wasn’t entirely clear how to do it, but eventually I got it working. One thing that could be improved is to emphasize the correct FILENAME_FORMAT (i.e. not the default).

But… now, this issue, it’s about the actual CustomResourceDefinition that fails, not the ServiceMonitor resource.

To reproduce, taking the ServiceMonitor definition as an example:

$ URL=https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
$ curl -s $URL | kubeconform
stdin - CustomResourceDefinition servicemonitors.monitoring.coreos.com failed validation: could not find schema for CustomResourceDefinition
Summary: 1 resource found parsing stdin - Valid: 0, Invalid: 0, Errors: 1, Skipped: 0

If I skip the CustomResourceDefinition schema, it passes, obviously:

$ curl -s $URL | kubeconform -summary -skip CustomResourceDefinition
Summary: 1 resource found parsing stdin - Valid: 0, Invalid: 0, Errors: 0, Skipped: 1

To me, this indicates that the schema for CustomResourceDefinition cannot be found. And I think I’ve seen something similar with kubeval previously.

Is this possible to include the CRDS to main repo by default ? Like a GH actions? If so, I can work on that.

Hi Dewe, no it is not, servicemonitors.monitoring.coreos.com is part of Prometheus’ operator CRD! The workflow to validate CRDs is described in the README, https://github.com/yannh/kubeconform#converting-an-openapi-file-to-a-json-schema - could you go through it and let me know whether its understandable? It does require a little bit of manual work unfortunately.

https://github.com/yannh/kubernetes-json-schema/issues/26 seemed to fix the issue for me.

Although I do run into what Yannh mentioned above when I use multiple single file schema locations:

  -schema-location default
  -schema-location https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/\{\{.Group\}\}/\{\{.ResourceKind\}\}_\{\{.ResourceAPIVersion\}\}.json
  -schema-location https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.28.5/customresourcedefinition.json
  -schema-location https://json.schemastore.org/kustomization.json

It acts like https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.28.5/customresourcedefinition.json is the only one to exist opposed to leveraging both single file schema locations.

However this worked as expected:

-schema-location https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json
-schema-location https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v${{ inputs.kubernetes_version }}/{{.ResourceKind}}.json
-schema-location https://json.schemastore.org/{{.ResourceKind}}.json

Hi @Skaronator , unfortunately that’s not correct, -schema-location should not point to a single file - I obviously need to improve documentation. In the order you passed parameters, apis__apiextensions.k8s.io__v1_openapi.json will be used to validate all files that are not found in default or datree.
Kubeconform goes through all the schema-location in order of passing, and tries to complete the template string to find a schema in the schema location. If it finds a schema (can successfully download one), it will use this to validate the manifest. If not, it will try the next schema-location. Is this a bit more clear?

Even for your own CRDs, it’s possible to convert them to JSON and just point at them in GitHub as I’ve done here ~

https://github.com/premiscale/pass-operator/blob/master/helm/operator-crds/_json/PassSecret.json

https://github.com/premiscale/pass-operator/blob/master/.circleci/helm.yml#L16

I’m very new to this, so someone feel free to educate me if there’s a reason I shouldn’t do this.

The project I’m currently working on uses flux and has a lot of apiextensions.k8s.io/v1 with customResourceDefinitions. I probably don’t need to validate those as flux creates those files, but it’s part of a folder I’m validating. To get around this I pull the file from here and add it as another schema location and no longer get failed validation: could not find schema for CustomResourceDefinition.

Hope this helps anyone that comes across the same problem.