grequests: gevent patch_all call in grequests breaks pymysql, etc.
I was looking forward to using this awesome package with my app and gevent until I noticed that it accomplishes gevent-compatibility via gevent.monkey.patch_all():
try:
import gevent
from gevent import monkey as curious_george
from gevent.pool import Pool
except ImportError:
raise RuntimeError('Gevent is required for grequests.')
# Monkey-patch.
curious_george.patch_all(thread=False, select=False)
Unfortunately, patch_all() impacts all other packages/modules used by the app, breaking some of them that were not designed for reentrancy. For example, gevent.monkey.patch_all() broke the combination of pymysql/pooled_db (with sporadic, hard-to-reproduce failures); my app has several greenlets that on occasion make pymysql calls; I isolated the problem to gevent’s monkey-patching of the socket module in that case.
A local solution involving monkey-patching of just the requests package to use gevent’s variants of blocking API’s would greatly enhance compatibility of grequests with apps and eliminate unexpected/undesirable side-effects on other packages. For example, I recently monkey-patched HBase/Thrift via the following code snippet with the desired effect on Thrift and no harm to the rest of the app (pymysql, etc.):
# Monkey-patch Thrift's TSocket to use gevent.socket to make our HBase interface
# gevent-friendly
import gevent.socket
from thrift.transport import TSocket
TSocket.socket = gevent.socket
gevent-zeromq is another example of local monkey-patching, albeit a more complex one: https://github.com/traviscline/gevent-zeromq/blob/master/gevent_zeromq/__init__.py
Thank you, Vitaly
About this issue
- Original URL
- State: closed
- Created 12 years ago
- Reactions: 1
- Comments: 24
I find it really problematic for stability when an imported module all of a sudden changes so much of the environment for everything else that your app does. It’s akin to having the rug pulled from under your feet 😃
The monkeypatching also breaks PyDev’s debugger, which appears to use ordinary sockets. It would be much nicer if
requestscould be patched, instead of all of Python.