fastapi: Generated schemas do not follow the OpenAPI specification (FastAPI v0.66+)

First Check

I added a very descriptive title to this issue.
I used the GitHub search to find a similar issue and didn’t find it.
I searched the FastAPI documentation, with the integrated search.
I already searched in Google “How to X in FastAPI” and didn’t find any information.
I already read and followed all the tutorial in the docs and didn’t find an answer.
I already checked if it is not related to FastAPI but to Pydantic.
I already checked if it is not related to FastAPI but to Swagger UI.
I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

I commit to help with one of those options 👆

Example Code

import json
from pydantic import BaseModel, Field
from fastapi import FastAPI
from fastapi import __version__ as fastapi_version

class Model(BaseModel):
    field: str = Field("foo", const=True)


app = FastAPI(title=f"Created with FastAPI v{fastapi_version}")


@app.get("/", response_model=Model)
def root():
    return Model(field="foo")

print(json.dumps(Model.schema(), indent=2))
print(json.dumps(app.openapi(), indent=2))

Description

With fastapi==0.68.0 and pydantic==1.8.2, the above MWE produces an invalid OpenAPI schema due to the handling of extra keys introduced in the last release. The inclusion of the const key is not supported in OpenAPI 3 (see Supported Keywords).

Here is a diff between the schema created for v0.65.2 vs v0.68 (also tested with 0.66, 0.67 with the same results).

// ...
  },
  "components": {
    "schemas": {
      "Model": {
        "title": "Model",
        "type": "object",
        "properties": {
          "field": {
            "title": "Field",
            "type": "string",
+           "const": "foo"
          }
// ...

This regression is caused by the addition of extra = "allow" to fastapi.openapi.models.Schema in (#1429) (see also the comment https://github.com/tiangolo/fastapi/pull/1429#issuecomment-889256569). This is something we ran into when updating FastAPI at https://github.com/Materials-Consortia/optimade-python-tools/pull/887.

While I would also like to be able to use extension keys (with the appropriate x- prefix) too, this change now makes it hard to use some standard pydantic features. My suggestion would be to enumerate the keys from JSON schema that OpenAPI does not support (linked above) and handle those separately in the Config for Schema. Perhaps any “extra” key that is not in this list could be prepended with a x-my-app prefix defined on the init of the FastAPI instance. I’m happy to have a go at implementing this, if desired.

Operating System

Linux

Operating System Details

No response

FastAPI Version

0.68.0

Python Version

3.8.5

Additional Context

I’ve made two gists for the OpenAPI schemas from v0.68.0 and v0.65.2:

You can pass the raw json through the online swagger validator to verify:

0.65.2: https://validator.swagger.io/validator/debug?url=https://gist.githubusercontent.com/ml-evs/99b33f68b74a0c197e970f9222faa2bc/raw/71d056de37a942bb9bc26943151c477a97aa18f4/fastapi_v0.65.2_openapi.json

$ curl -s https://validator.swagger.io/validator/debug\?url\=https://gist.githubusercontent.com/ml-evs/99b33f68b74a0c197e970f9222faa2bc/raw/71d056de37a942bb9bc26943151c477a97aa18f4/fastapi_v0.65.2_openapi.json | jq
{}

0.68.0: https://validator.swagger.io/validator/debug?url=https://gist.githubusercontent.com/ml-evs/824f9ac510f92d4e180a4e984d1da38d/raw/d25b9c3a7802702ced6e3f7a1aa60e017ac9b339/fastapi_v0.68.0_openapi.json

$ curl -s https://validator.swagger.io/validator/debug\?url\=https://gist.githubusercontent.com/ml-evs/824f9ac510f92d4e180a4e984d1da38d/raw/d25b9c3a7802702ced6e3f7a1aa60e017ac9b339/fastapi_v0.68.0_openapi.json | jq
{
  "messages": [
    "attribute components.schemas.Model.const is unexpected"
  ],
  "schemaValidationMessages": [
    {
      "level": "error",
      "domain": "validation",
      "keyword": "oneOf",
      "message": "instance failed to match exactly one schema (matched 0 out of 2)",
      "schema": {
        "loadingURI": "#",
        "pointer": "/definitions/Components/properties/schemas/patternProperties/^[a-zA-Z0-9\\.\\-_]+$"
      },
      "instance": {
        "pointer": "/components/schemas/Model"
      }
    }
  ]
}

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 3
Comments: 19 (10 by maintainers)

Commits related to this issue

Add OpenAPI warning to "Body - Fields" docs Adds a warning about generating invalid OpenAPI schemas (see #3745) when using additional keys in pydantic `Field` specifications. — committed to ml-evs/fastapi by ml-evs 2 years ago

Most upvoted comments

Hey all! Some additional info: Pydantic generates JSON Schema, and that is re-used by OpenAPI.

But OpenAPI 3.0.x is based on an old version of JSON Schema that didn’t have const and also had some additional incompatibilities. OpenAPI 3.1.x is based on JSON Schema v7, which has const.

So, about const, the ideal would be to be able to upgrade OpenAPI to 3.1.x. In fact, the currently generated OpenAPI schema is a bit more compatible with 3.1.x than 3.0.x. But I haven’t been able to update the generated version by FastAPI because Swagger UI is not compatible with 3.1.x, it doesn’t even try to render, it just explodes. So I can’t upgrade that resulting version yet. The best way to move forward for that would be to improve/add support in Swagger UI to OpenAPI 3.1.x.

Now, about extra fields included by default in the output JSON Schema, I hope a future version of Pydantic will have something like Field(schema_extra={}) instead of just including any extra args there. That would allow explicitly adding custom stuff that should be included in the output JSON Schema and avoid side effects for other use cases.

tiangolo on May 11, 2022

Just a quick note that Pydantic v2 will have its own independent field json_schema_extra instead of taking everything else and putting it in the JSON Schema output. 🎉

That will at least help understand and debug these use cases better. 🤓

tiangolo on Nov 5, 2022

awesome, thanks for that @ml-evs !

trondhindenes on Aug 9, 2022

Here’s the slightly kludgey way we got around it @trondhindenes, in case it is useful to you: https://github.com/Materials-Consortia/optimade-python-tools/pull/1131/files

Basically:

stop using const and use a regex that matches the constant value instead
wrap/reimplement pydantic.Field with our own class that itself wraps FieldInfo, which strips or prefixes invalid OpenAPI keys.

ml-evs on Aug 8, 2022

Is there any chance you could confirm this is a bug @tiangolo, 1 year on? If not, I am happy to prepare a docs PR that tries to explain the issue for other confused users. Given the (deserved) popularity of FastAPI I think this is unfortunately poisoning the OpenAPI ecosystem with invalid schemas!

Whilst @mreschke’s fix above does generate the correct OpenAPI spec, I think that it ends up disabling the pydantic validation (of e.g. const) without much extra boilerplate that redefines the entire class hierarchy down to fastapi.openapi.models.Schema.

A nice fix would be to capture this at the pydantic level with a new extra = 'prefix' handling of extra fields (or even provide a callable for how to handle them), but until then this functionality might have to be implemented in the Config for fastapi.openapi.models.Schema directly.

Unfortunately, as we are 1 year on, this may break things for people relying on extra fields in their non-standard OpenAPI schemas.

ml-evs on Apr 25, 2022

Just tested again and this bug is still affecting v0.75.1. It would be great to get some confirmation that a fix to FastAPI would be useful, perhaps following @mreschke’s workaround above. I fear that many users are producing invalid OpenAPI schemas without realising.

ml-evs on Apr 14, 2022

When using custom extra attributes on fields, which are used internally for different kinds of data processing, these values are now exposed in the api schema. In my case additionally it causes an infinite recursion.

To give some context: I’m using fastapi with django. To map pydantic model fields to a django model field, an argument orm_field is set on the pydantic model’s field pointing to the django model field.

from django.db import models
from pydantic import BaseModel

class MyDBModel(models.Model):
    id = models.CharField(primary_key=True, max_length=16)

class MyModel(BaseModel):
    id: str = Field(orm_field=MyDBModel.id)

Without patching fastapi.openapi.models.Schema’s Config setting extra = ‘ignore’ (as it was pre 0.66.0), a infinite recursion ocurrs when rendering the openapi.json

  File ".../fastapi/encoders.py", line 101 in jsonable_encoder
  File ".../fastapi/encoders.py", line 145 in jsonable_encoder
  File ".../fastapi/encoders.py", line 115 in jsonable_encoder
  File ".../fastapi/encoders.py", line 101 in jsonable_encoder
  File ".../fastapi/encoders.py", line 145 in jsonable_encoder
  File ".../fastapi/encoders.py", line 101 in jsonable_encoder
  File ".../fastapi/encoders.py", line 145 in jsonable_encoder
  File ".../fastapi/encoders.py", line 145 in jsonable_encoder
  File ".../fastapi/encoders.py", line 101 in jsonable_encoder
  File ".../fastapi/encoders.py", line 145 in jsonable_encoder
  File ".../fastapi/encoders.py", line 115 in jsonable_encoder
  File ".../fastapi/encoders.py", line 101 in jsonable_encoder

As the OpenAPI Spec only allows custom attributes with a x- prefix, I suggest to revert the change, as it is possible to declare a custom schema definition on the pydantic model using a custom config.

mstingl on Feb 26, 2022

Not sure if this helps with your particular use case…as a workaround. Obviously defaulting extra=allow will still be a pervasive issue. But you could make your own class Field(PydanticFieldInfo) (extending Pydantics FieldInfo class) and handle any extra kwargs and placing them inside a x-tra parameter before calling super. Then your Field("foo", const=True) would result in a

{
  "x-tra": {
    "const": True
  },
}

That’s basically what I did with my custom Field() class.

mreschke on Sep 22, 2021

It was a custom BaseModel and Field() override that added a few things to extra. In older FastAPI versions those were ignored, which was the desired effect. Now they show up due to extra=allow. Your issue is more complex because const is a valid parameter in pydantic/fields.py both in the def Field() and in class FieldInfo and now it shows in the OpenAPI spec, which is definitely a invalid field according to https://swagger.io/docs/specification/data-models/keywords/. The larger issue is that pydantic Field() (and therefore class FieldInfo) have that extra kwargs which pipes any undefined param right into OpenAPI spec creating a potentially invalid .json spec. The only valid extra kwargs that Field() could possibly take without breaking the spec validator is those that begin with x-, although you cannot add a dash as a parameter name, ie: name: string = Field(x-other='goodies'). I’m sure FastAPI had a good reason to add extra=allow by default, but it sure makes things more breakable.

mreschke on Sep 22, 2021