sqlalchemy: Random error about modifying duque when event listener is set

Describe the bug

We saw some random errors occurring in production. We are not able to reproduce them nor determine the exact circumstances.

Screenshot 2023-02-23 at 08 20 54

in the code we set a listener like this:

 @listens_for(self._db_engine, "engine_connect", named=True)
def add_connection_timeout(conn, branch):
            # will capture all checkout events of connections from the engines pool of connections

            # the timeout will persist beyond this transaction because of the connection pooling
            # however we only want to ensure it is always set during writes
            conn.connection.connection.timeout = settings.timeout

however as can be seen in the traceback the listener is not invoked during the error.

Any hint how we might be able to reproduce this are very much appreciated.

Optional link from https://docs.sqlalchemy.org which documents the behavior that is expected

No response

SQLAlchemy Version in Use

1.4.46

DBAPI (i.e. the database driver)

pyodbc

Database Vendor and Major Version

MSSQL

Python Version

3.8

Operating system

Linux

To Reproduce

we have not found a way to reproduce, it passes our tests. Works in production until it does not, a few seconds later it works again.

Error

# Copy the complete stack trace and error message here, including SQL log output if applicable.
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 120, in __init_
self.dispatch.engine_connect(self, _branch_from is not None)
L None
L ‹sqlalchemy.engine.base.Connection object at 0x7fac6cef5160>
L <member 'engine_connect of 'JoinedConnectionEventsDispatch' objects>
L <sqlalchemy.event.base. JoinedConnectionEventsDispatch object at 0x7fac6d544a00> 4 <sqlalchemy.engine. base. Connection object at 0x7fac6cef5160>
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py",line 333, in _call_.
for fn in self. listeners:
L ‹property object at Ox7fac77fba400>
LI <sqlalchemy.event.attr._JoinedListener object at Ox7fac6c3b4580>
L ‹function.
_CIsLevelDispatch._wrap_fn_for_kw.<locals>.wrap_kw at 0x7fac6db4ea60>
RuntimeError: deque mutated during iteration

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (7 by maintainers)

Most upvoted comments

the after_update event is per ORM mapped class, not per session. so for that one you’d attach it either to the specific ORM mapped class you want to listen for, or you can put it across the board on your declarative Base class, adding propagate=True so it takes effect for subclasses

https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.MapperEvents

I misspoke when I said after_update is a session event

you should attach listeners to your top level objects only, such as the engine right after you make it, or the sessionmaker. do not have event.remove in your application at all and keep the number of listeners to an absolute minimum and ensure they are permanent for the life of the application. if you want things to turn on and off within the scope of those event listeners, like to have them respond to some objects and not others, build a simple system on your end that is invoked from within your event handling functions.

before_commit is and after_update are both session events so you can put those on your sessionmaker class, where they will take effect for all Session objects.

SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

@event.listens_for("before_commit", SessionLocal)
def my_event(...):
   # ...