fastapi: multipart/form-data: Unable to parse complex types in a request form

Hi, I am just wondering why not add something like this. The error(type is not dict) is the value of Item comes in as string when it is multipart/form-data. This solution also works with nested classes, it also won’t unpack all fields of Item into form fields, which is not very nice if this Item class is a little complicated. Happy to create a PR if this is really a good solution.

For anyone else who needs nested models, I made this alternate approach that takes the input in the same way @tricosmo takes in the value as json. This method does do the proper validation! I really needed to take a file and complex form field and this gets the job done. It should also support multiple form properties.

fastapi code:

class TestMetaConfig(BaseModel):
    prop1: str
    prop2: list[int]


class TestModel(BaseModel):
    foo: str
    bar: int
    meta_config: TestMetaConfig


@app.post("/")
async def foo(upload_file: UploadFile = File(...), model: Json[TestModel] = Form(...)):
    with open("test.png", "wb") as fh:
        fh.write(await upload_file.read())

    logger.info(f"model.foo: {model.foo}")
    logger.info(f"model.bar: {model.bar}")
    logger.info(f"model.meta_config: {model.meta_config}")
    return {"foo": model.foo, "bar": model.bar}

Call it like this with json dumps

import json
from pathlib import Path

import requests

HERE = Path(__file__).parent.absolute()

with open(HERE / "imgs/foo.png", "rb") as fh:
    url = "http://localhost:8000/"
    files = {"upload_file": fh}
    values = {"foo": "hello", "bar": 123, "meta_config": {"prop1": "hello there", "prop2": ["general", "kenobi", 1]}}
    resp = requests.post(url, files=files, data={"model": json.dumps(values)})
    print(resp.status_code)
    print(resp.json())

Proper validation if you send in bad types like I did in my example above

{"detail": [{"loc": ["body", "model", "meta_config", "prop2", 0], "msg": "value is not a valid integer", "type": "type_error.integer"}, {"loc": ["body", "model", "meta_config", "prop2", 1], "msg": "value is not a valid integer", "type": "type_error.integer"}]}

Try this:

import inspect
from typing import Dict, Type, TypeVar, Protocol, Generic, NewType

from fastapi import Depends, FastAPI, File, Form
from pydantic import BaseModel, validator, BaseSettings, Json

app = FastAPI()

StringId = NewType('StringId', str)


def as_form(cls: Type[BaseModel]):
    """
    Adds an as_form class method to decorated models. The as_form class method
    can be used with FastAPI endpoints
    """
    new_params = [
        inspect.Parameter(
            field.alias,
            inspect.Parameter.POSITIONAL_ONLY,
            default=(Form(field.default) if not field.required else Form(...)),
        )
        for field in cls.__fields__.values()
    ]

    async def _as_form(**data):
        return cls(**data)

    sig = inspect.signature(_as_form)
    sig = sig.replace(parameters=new_params)
    _as_form.__signature__ = sig
    setattr(cls, "as_form", _as_form)
    return cls


@as_form
class Item(BaseModel):
    name: str
    another: str
    opts: Json[Dict[str, int]] = '{}'


@app.post("/test")
async def endpoint(item: Item = Depends(Item.as_form)):
    return item.dict()


if __name__ == "__main__":
    import json
    import os

    from fastapi.testclient import TestClient

    tc = TestClient(app)

    item = {"name": "vivalldi", "another": "mause"}

    data = bytearray(os.urandom(1))
    files = {"data": ("data", data, "text/csv")}

    r = tc.post("/test", data=item, files=files)
    assert r.status_code == 200, r.text
    assert r.json() == {"name": "vivalldi", "another": "mause", "opts": {}}

    files["opts"] = (None, json.dumps({"a": 2}), "application/json")
    r = tc.post("/test", data=item, files=files)
    assert r.status_code == 200
    assert r.json() == {"name": "vivalldi", "another": "mause", "opts": {"a": 2}}

The main two changes were using the Json class from pydantic, and removing the annotation from the as_form method, as otherwise pydantic would be validating the data twice - once for as_form, and once for the model itself

For anyone interested, here’s an extension of @Mause’s approach to support nested models:

def as_form(cls: Type[BaseModel]) -> Type[BaseModel]:
    """
    Adds an `as_form` class method to decorated models. The `as_form` class
    method can be used with `FastAPI` endpoints.

    Args:
        cls: The model class to decorate.

    Returns:
        The decorated class.

    """

    def make_form_parameter(field: ModelField) -> Any:
        """
        Converts a field from a `Pydantic` model to the appropriate `FastAPI`
        parameter type.

        Args:
            field: The field to convert.

        Returns:
            Either the result of `Form`, if the field is not a sub-model, or
            the result of `Depends` if it is.

        """
        if issubclass(field.type_, BaseModel):
            # This is a sub-model.
            assert hasattr(field.type_, "as_form"), (
                f"Sub-model class for {field.name} field must be decorated with"
                f" `as_form` too."
            )
            return Depends(field.type_.as_form)
        else:
            return Form(field.default) if not field.required else Form(...)

    new_params = [
        inspect.Parameter(
            field.alias,
            inspect.Parameter.POSITIONAL_ONLY,
            default=make_form_parameter(field),
        )
        for field in cls.__fields__.values()
    ]

    async def _as_form(**data):
        return cls(**data)

    sig = inspect.signature(_as_form)
    sig = sig.replace(parameters=new_params)
    _as_form.__signature__ = sig
    setattr(cls, "as_form", _as_form)
    return cls

…and an example usage:

@as_form
class Nested(BaseModel):
    foo: int
    bar: str

@as_form
class Outer(BaseModel):
    inner: Inner
    baz: float

@app.post("/test")
async def test_form(form: Outer = Depends(Outer.as_form)):
    return {"foo": form.inner.foo, "bar": form.inner.bar, "baz": form.baz}

Known Limitations: Currently, this doesn’t handle default values for fields that are sub-models. It might be possible to extend it, but in the long term, I’m in favor of merging @tricosmo’s approach.

FYI you can also wrap nested models in the Json class e.g. opts: Json[OptsModel] And if you want pydantic validation errors to propagate correctly as a fastapi request validation error instead of 500 error, you can add:

async def _as_form(**data):
    try:
        return cls(**data)
    except pydantic.ValidationError as e:
        raise fastapi.exceptions.RequestValidationError(e.raw_errors)

I’m trying to add a swagger example value for json field in the form with no luck, anyone have ideas? edit: For adding swagger examples in form fields:

openapi_schema = cls.schema()
new_params = []
for field in cls.__fields__.values():
    example = openapi_schema.get("example", {}).get(field.name)
    default = Form(field.default, example=example) if not field.required else Form(..., example=example)
    new_params.append(
        inspect.Parameter(
            field.alias,
            inspect.Parameter.POSITIONAL_ONLY,
            default=default,
        )
    )

Here’s an inheritable version of @Mause’s @as_form that is mypy compliant:

class FormBaseModel(BaseModel):
    def __init_subclass__(cls, *args, **kwargs):
        field_default = Form(...)
        new_params = []
        for field in cls.__fields__.values():
            default = Form(field.default) if not field.required else field_default
            annotation = inspect.Parameter.empty

            new_params.append(
                inspect.Parameter(
                    field.alias,
                    inspect.Parameter.POSITIONAL_ONLY,
                    default=default,
                    annotation=annotation,
                )
            )

        async def _as_form(**data):
            try:
                return cls(**data)
            except ValidationError as e:
                raise RequestValidationError(e.raw_errors)

        sig = inspect.signature(_as_form)
        sig = sig.replace(parameters=new_params)
        _as_form.__signature__ = sig  # type: ignore
        setattr(cls, "as_form", _as_form)

    @staticmethod
    def as_form(parameters=[]) -> "FormBaseModel":
        raise NotImplementedError

Adding onto this: I took @Mause’s suggestions (i think) to accomplish this goal but could not get it working such that i can multipart-form two things at once.

my code, currently grrarrgh.py:

from fastapi import FastAPI, Depends, UploadFile, File, Form
from pydantic import BaseModel, conlist, Field
import typing as t
import inspect

app = FastAPI()


# from https://github.com/tiangolo/fastapi/issues/2387#issuecomment-731662551
def as_form(cls: t.Type[BaseModel]):
    """
    Adds an as_form class method to decorated models. The as_form class method
    can be used with FastAPI endpoints
    """
    new_params = [
        inspect.Parameter(
            field.alias,
            inspect.Parameter.POSITIONAL_ONLY,
            default=(Form(field.default) if not field.required else Form(...)),
        )
        for field in cls.__fields__.values()
    ]

    async def _as_form(**data):
        return cls(**data)

    sig = inspect.signature(_as_form)
    sig = sig.replace(parameters=new_params)
    _as_form.__signature__ = sig
    setattr(cls, "as_form", _as_form)
    return cls


@as_form
class InferenceRequest(BaseModel):
    threshold: float = Field(default=.1, description="The threshold")
    render: bool = Field(
        description="a flag", default=False
    )


@app.post("/v0/infer/")
async def infer(
        files: conlist(
            UploadFile,
            min_items=1,
        ) = File(...),
        req: InferenceRequest = Depends(InferenceRequest.as_form),
):
    """get files and request data!"""
    result = dict()
    for i, f in enumerate(files):
        result[f'file {i}'] = f'{f.filename} is a {type(f)}'
    result['req'] = f'{req.dict()}:  is a {type(req)}'
    return result

Running with uvicorn grrarrgh:app --reload, the server log is:

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [8186] using statreload
INFO:     Started server process [8188]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:47576 - "POST /v0/infer/ HTTP/1.1" 200 OK

Curl as follows. Note that i specify 0.2 and true, but i get 0.1 and false:

curl -s -X  "POST"  http://127.0.0.1:8000/v0/infer/ -F 'req={"render":true, "threshold":0.2}' -F 'files=@".d1-vlcsnap-00003.jpg"' | jq .

{
  "file 0": "d1-vlcsnap-00003.jpg is a <class 'starlette.datastructures.UploadFile'>",
  "req": "{'threshold': 0.1, 'render': False}:  is a <class 'grrarrgh.InferenceRequest'>"
}

longer curl with --trace-ascii; i’m omitting the file blob for brevity

curl --trace-ascii -  -X  "POST"  http://127.0.0.1:8000/v0/infer/ -F 'req={"render":true, "threshold":0.2}' -F 'files=@".d1-vlcsnap-00003.jpg"'
Note: Unnecessary use of -X or --request, POST is already inferred.
== Info:   Trying 127.0.0.1...
== Info: TCP_NODELAY set
== Info: Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
=> Send header, 219 bytes (0xdb)
0000: POST /v0/infer/ HTTP/1.1
001a: Host: 127.0.0.1:8000
0030: User-Agent: curl/7.58.0
0049: Accept: */*
0056: Content-Length: 18807
006d: Content-Type: multipart/form-data; boundary=--------------------
00ad: ----17a4131811a5d701
00c3: Expect: 100-continue
00d9:
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 16384 bytes (0x4000)
0000: --------------------------17a4131811a5d701
002c: Content-Disposition: form-data; name="req"
0058:
005a: {"render":true, "threshold":0.2}
007c: --------------------------17a4131811a5d701
00a8: Content-Disposition: form-data; name="files"; filename="d1-vlcsn
00e8: ap-00003.jpg"
00f7: Content-Type: image/jpeg
0111:
0113: ......JFIF.....H.H.....C................................... $.'
0153: ",#..(7),01444.'9=82<.342...C...........2!.!22222222222222222222
0193: 222222222222222222222222222222..........."......................
........  < trimmed here> 
0940: .......
0949: --------------------------17a4131811a5d701--
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
<= Recv header, 37 bytes (0x25)
0000: date: Thu, 02 Sep 2021 14:28:39 GMT
<= Recv header, 17 bytes (0x11)
0000: server: uvicorn
<= Recv header, 21 bytes (0x15)
0000: content-length: 171
<= Recv header, 32 bytes (0x20)
0000: content-type: application/json
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 171 bytes (0xab)
0000: {"file 0":"d1-vlcsnap-00003.jpg is a <class 'starlette.datastruc
0040: tures.UploadFile'>","req":"{'threshold': 0.1, 'render': False}:
0080:  is a <class 'grrarrgh.InferenceRequest'>"}
{"file 0":"d1-vlcsnap-00003.jpg is a <class 'starlette.datastructures.UploadFile'>","req":"{'threshold': 0.1, 'render': False}:  is a <class 'grrarrgh.InferenceRequest'>"}== Info: Connection #0 to host 127.0.0.1 left intact

If i specify -F 'req={...};type=application/json' in curl, the Content-Disposition is updated, but nothing else changes.

@thiras I’m honestly not sure why you’re asking for that, as that isn’t a list at all, just a series of named form fields. You should really consider at that point if a form is what you want, as HTML fields really don’t support lists at all, and maybe you should just have an API endpoint that takes json

@djpetti honestly I’ll give you the same answer - forms don’t have support for nested fields either, all that is is a naming convention that can easily differ between projects