grequests: gevent patch_all call in grequests breaks pymysql, etc.

I was looking forward to using this awesome package with my app and gevent until I noticed that it accomplishes gevent-compatibility via gevent.monkey.patch_all():

try:
    import gevent
    from gevent import monkey as curious_george
    from gevent.pool import Pool
except ImportError:
    raise RuntimeError('Gevent is required for grequests.')

# Monkey-patch.
curious_george.patch_all(thread=False, select=False)

Unfortunately, patch_all() impacts all other packages/modules used by the app, breaking some of them that were not designed for reentrancy. For example, gevent.monkey.patch_all() broke the combination of pymysql/pooled_db (with sporadic, hard-to-reproduce failures); my app has several greenlets that on occasion make pymysql calls; I isolated the problem to gevent’s monkey-patching of the socket module in that case.

A local solution involving monkey-patching of just the requests package to use gevent’s variants of blocking API’s would greatly enhance compatibility of grequests with apps and eliminate unexpected/undesirable side-effects on other packages. For example, I recently monkey-patched HBase/Thrift via the following code snippet with the desired effect on Thrift and no harm to the rest of the app (pymysql, etc.):

# Monkey-patch Thrift's TSocket to use gevent.socket to make our HBase interface
# gevent-friendly
import gevent.socket
from thrift.transport import TSocket
TSocket.socket = gevent.socket

gevent-zeromq is another example of local monkey-patching, albeit a more complex one: https://github.com/traviscline/gevent-zeromq/blob/master/gevent_zeromq/__init__.py

Thank you, Vitaly

About this issue

  • Original URL
  • State: closed
  • Created 12 years ago
  • Reactions: 1
  • Comments: 24

Most upvoted comments

I find it really problematic for stability when an imported module all of a sudden changes so much of the environment for everything else that your app does. It’s akin to having the rug pulled from under your feet 😃

The monkeypatching also breaks PyDev’s debugger, which appears to use ordinary sockets. It would be much nicer if requests could be patched, instead of all of Python.