kubernetes-client: Creating CRDs with schema validation is broken

the K8s api server makes a lot of strict checks, for instance it fails if the schema contains default values, definitions, $ref elements, etc. One of the checks is:

	if schema.Dependencies != nil {
		for dependency, jsonSchemaPropsOrStringArray := range schema.Dependencies {
			allErrs = append(allErrs, ValidateCustomResourceDefinitionOpenAPISchema(jsonSchemaPropsOrStringArray.Schema, fldPath.Child("dependencies").Key(dependency), ssv)...)
		}
	}

source: https://github.com/kubernetes/apiextensions-apiserver/blob/master/pkg/apis/apiextensions/validation/validation.go#L671

However, even if the JSONSchemaProps object has the dependecies field set to null, the object mapper from Jackson that’s being used in here converts the null to an empty LinkedHashMap and this fails on the K8s api server. So there is no way currently to create the schema validation with the fabric8 k8s client.

I am using the .withNewOpenAPIV3SchemaLike(schema) method for the CustomResourceDefinitionBuilder.

I believe there must be some configuration of the object mapper so that it shouldn’t translate nulls to empty maps.

JSON_MAPPER.setSerializationInclusion(Include.NON_NULL);

…looks promising

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 2
  • Comments: 22 (11 by maintainers)

Commits related to this issue

Most upvoted comments

here is the reproducer:

public class Reproducer {

    public static JSONSchemaProps readSchema() {
        ObjectMapper mapper = new ObjectMapper();
        mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        URL in = Reproducer.class.getResource("/sparkCluster.json");
        if (null == in) {
            return null;
        }
        try {
            return mapper.readValue(in, JSONSchemaProps.class);
        } catch (IOException e) {
            e.printStackTrace();
            return null;
        }
    }

    public static void main(String[] args) {
        JSONSchemaProps schema = readSchema();

        CustomResourceDefinitionBuilder builder = new CustomResourceDefinitionBuilder()
                .withApiVersion("apiextensions.k8s.io/v1beta1")
                .withNewMetadata().withName("sparkclusters.radanalytics.io")
                .endMetadata()
                .withNewSpec()
                .withNewNames()
                .withKind("SparkCluster")
                .endNames()
                .withGroup("radanalytics.io")
                .withVersion("v1")
                .withScope("Namespaced")
                .withNewValidation()
                .withNewOpenAPIV3SchemaLike(schema)
                .endOpenAPIV3Schema()
                .endValidation()
                .endSpec();

        new DefaultKubernetesClient().customResourceDefinitions().createOrReplace(builder.build());
    }
}

content of sparkCluster.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "A Spark cluster configuration",
  "dependencies": null,
  "type": "object",
  "extends": {
    "type": "object",
    "existingJavaType": "io.radanalytics.operator.common.EntityInfo"
  },
  "properties": {
    "master": {
      "type": "object",
      "properties": {
        "instances": {
          "type": "integer",
          "default": "1",
          "minimum": "1"
        },
        "memory": {
          "type": "string"
        },
        "cpu": {
          "type": "string"
        },
        "labels": {
          "existingJavaType": "java.util.Map<String,String>",
          "type": "string",
          "pattern": "([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]"
        },
        "command": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "commandArgs": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    },
    "worker": {
      "type": "object",
      "properties": {
        "instances": {
          "type": "integer",
          "default": "1",
          "minimum": "0"
        },
        "memory": {
          "type": "string"
        },
        "cpu": {
          "type": "string"
        },
        "labels": {
          "existingJavaType": "java.util.Map<String,String>",
          "type": "string",
          "pattern": "([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]"
        },
        "command": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "commandArgs": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    },
    "customImage": {
      "type": "string"
    },
    "metrics": {
      "type": "boolean",
      "default": "false"
    },
    "sparkWebUI": {
      "type": "boolean",
      "default": "true"
    },
    "sparkConfigurationMap": {
      "type": "string"
    },
    "env": {
      "type": "array",
      "items": {
        "type": "object",
        "javaType": "io.radanalytics.types.Env",
        "properties": {
          "name": { "type": "string" },
          "value": { "type": "string" }
        },
        "required": ["name", "value"]
      }
    },
    "sparkConfiguration": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "value": { "type": "string" }
        },
        "required": ["name", "value"]
      }
    },
    "labels": {
      "type": "object",
      "existingJavaType": "java.util.Map<String,String>"
    },
    "historyServer": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "type": {
          "type": "string",
          "default": "sharedVolume",
          "enum": [
            "sharedVolume",
            "remoteStorage"
          ],
          "javaEnumNames": [
            "sharedVolume",
            "remoteStorage"
          ]
        },

        "sharedVolume": {
          "type": "object",
          "properties": {
            "size": {
              "type": "string",
              "default": "0.3Gi"
            },
            "mountPath": {
              "type": "string",
              "default": "/history/spark-events"
            },
            "matchLabels": {
              "type": "object",
              "existingJavaType": "java.util.Map<String,String>"
            }
          }
        },
        "remoteURI": {
          "type": "string",
          "description": "s3 bucket or hdfs path"
        }
      }
    },
    "downloadData": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string"
          },
          "to": {
            "type": "string"
          }
        },
        "required": [
          "url",
          "to"
        ]
      }
    }
  },
  "required": []
}

…but it would fail with any schema, even smaller one. Thanks for looking into this!

No no, this is an example of improved support for custom resources. For custom resource definitions it’s still the same:

https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-itests/src/test/java/io/fabric8/kubernetes/CustomResourceDefinitionIT.java