OpenMetadata: Datamodel codegen does not apply defaults for $ref

From https://github.com/open-metadata/OpenMetadata/pull/6134

There is a new definition in team.json

"definitions": {
    "teamType" : {
      "description" : "Organization is the highest level entity. An Organization has one of more Business Units, Division, or Departments. A Business Unit has one or more Divisions, or Departments. A Division has one or more Divisions or Departments. A Department has one or more Departments or users.",
      "type" : "string",
      "enum": [
        "Department",
        "Division",
        "BusinessUnit",
        "Organization"
      ],
      "default": "Department"
    }
  },

Picked up in

"teamType": {
      "description": "Team type",
      "$ref": "#/definitions/teamType"
    },

However, the pydantic model generating from here does not properly use the default in teamType. This is working correctly in Java.

Let’s see if we can play with some parameters on the CLI https://koxudaxi.github.io/datamodel-code-generator/, find a better alternative, or open a PR

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15

Most upvoted comments

Looks like it is working perfect now! Many thanks

loggerLevel: Optional[workflow.LogLevels] = Field(
        workflow.LogLevels.INFO, description='Set the logging level for the workflow.'
    )

Will be updating the version on our repo. If we see anything weird we’ll let you know, but looks perfect.

Thanks!

pmbrull on Nov 2, 2022

Hi @koxudaxi, thanks for the ping.

From our JSON Schema

"logLevels": {
      "description": "Supported logging levels",
      "javaType": "org.openmetadata.schema.metadataIngestion.LogLevels",
      "type": "string",
      "enum": ["DEBUG", "INFO", "WARN", "ERROR"],
      "default": "INFO"
    },

Now the model gets generated as

loggerLevel: Optional[workflow.LogLevels] = Field(
        workflow.LogLevels, description='Set the logging level for the workflow.'
    )

I am not sure this is correct. I would expect workflow.LogLevels.INFO. Are we missing anything here?

I believe this is the expected behavior you set in the test cases though https://github.com/koxudaxi/datamodel-code-generator/pull/876/files#diff-c3b08450ac392a35ce5120ee7381936f1c1ecedc3a60ae020d6b9d3ddb55b790R15

Thanks again! Happy to discuss 🙏

pmbrull on Oct 28, 2022

Awesome thanks! no issues at all, let me test and get back here, appreciate the help 🙏

pmbrull on Nov 2, 2022

@pmbrull Thank you for checking the version. I’m sorry, I made mistake 😢

https://github.com/koxudaxi/datamodel-code-generator/releases/tag/0.13.4 I fixed and released it as 0.13.4 🙇

https://github.com/koxudaxi/datamodel-code-generator/pull/888/files#diff-c3b08450ac392a35ce5120ee7381936f1c1ecedc3a60ae020d6b9d3ddb55b790R16-R17

koxudaxi on Oct 31, 2022

oh missed that flag you’re right! Thanks for checking and for the fast response. We’ll wait to upgrade to the following version. Thanks!

pmbrull on Oct 19, 2022

Mmhhh tried to replicate it with these couple of schemas:

bar.json

{
  "$id": "bar.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Bar",
  "type": "object",
  "definitions": {
    "logLevels": {
      "description": "Supported logging levels",
      "type": "string",
      "enum": ["DEBUG", "INFO", "WARN", "ERROR"],
      "default": "INFO"
    }  
  }
}

and foo.json

{
  "$id": "foo.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Foo",
  "type": "object",
  "properties": {
    "loggerLevel": {
      "$ref": "bar.json#/definitions/logLevels"
    }
  }
}

but foo.py gets created without passing the ref and just using the str 🤔

I might be missing something here

class Foo(BaseModel):
    loggerLevel: Optional[bar.LogLevels] = 'INFO'

pmbrull on Oct 18, 2022

hi @koxudaxi no issues at all, thanks for the fix, appreciate the time on getting this done!

Actually, I was trying to replicate an issue we are currently having with 0.13.2:

We have the ingestionPipeline.json using a $ref to another JSON Schema.
The referenced JSON Schema has some defaults in an enum.

What is happening is that the import is not properly handled. In the IngestionPipeline pydantic model we end up having an import error:

from ....metadataIngestion import workflow

[...]
    loggerLevel: Optional[workflow.LogLevels] = Field(
        LogLevels.INFO, description='Set the logging level for the workflow.'
    )
[...]

As you can see, within the Optional we have workflow.LogLevels, while this import is not being used when setting the default.

Trying to get some minified example on this that would be easier for you to reproduce.

Thanks!

pmbrull on Oct 18, 2022