fastapi: InternalError: Packet sequence number wrong

Hello,

I came across an issue on production that I can’t replicate in the local environment. The same code path often works, but sometimes throws this error. I don’t understand, why this is happening.

I believe there is an issue with multi-thread nature of fastapi and internal usage of pymysql by databases. I’m pretty sure pymysql is single threaded.

But databases was supposed to support asyncio, correct? What am I doing wrong, please?

crud_server.py

db = databases.Database(settings.SQLALCHEMY_DATABASE_URI)

async def get_servers(is_maintenance: bool) -> List[Server]:
    query = ServerTable.select().where(ServerTable.c.is_maintenance == is_maintenance)
    if not db.is_connected:
        await db.connect()
    return await db.fetch_all(query)

ServerTable.py

metadata = MetaData(schema=settings.DBNAME_MAIN)

ServerTable = sqlalchemy.Table(
    "server",
    metadata,
    Column("id", String(15), primary_key=True),
    Column("name", String(100), nullable=False),
    Column("country", String(100), nullable=False),
    Column("group_master", String(15), nullable=True),
    Column("group_slave", String(15), nullable=True, index=True),
    Column("ip", String(length=45), nullable=False),
    Column("max_connections", Integer(), nullable=False),
    Column(
        "is_maintenance",
        Boolean(),
        nullable=False,
        server_default=text("0"),
        default=text("0"),
        index=True,
    ),
    Column("dow_maintenance", String(3), nullable=True),
    Column("hour_maintenance", Integer(), nullable=True),
    Column("minutes_maintenance", Integer(), nullable=True),
    Column("datetime_maintenance", TIMESTAMP(), nullable=True),
)

Error:

  File "/home/admin/tg/app/routers/server_status.py", line 25, in server_status
    servers = await crud_server.get_servers(is_maintenance=False)
  File "/home/admin/tg/app/database/crud_server.py", line 15, in get_servers
    return await db.fetch_all(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 140, in fetch_all
    return await connection.fetch_all(query, values)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 239, in fetch_all
    return await self._connection.fetch_all(built_query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/backends/mysql.py", line 108, in fetch_all
    await cursor.execute(query, args)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 239, in execute
    await self._query(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 457, in _query
    await conn.query(q)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 428, in query
    await self._read_query_result(unbuffered=unbuffered)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 622, in _read_query_result
    await result.read()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 1105, in read
    first_packet = await self.connection._read_packet()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 574, in _read_packet
    raise InternalError(
pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1

I would really appreciate a hint, what I’m doing wrong. Thank you so much