cachier: OSError: inotify instance limit reached

When using cachier with some long-running functions, I get the following error:

OSError: inotify instance limit reached

A fuller stack trace is in the details below. I have tried my best to obscure out proprietary information, so please forgive me if it kind of messes up the syntax.

What would be the best next step here to debug? (No pressure on your side, @shaypal5, I have no problems debugging on my own, just wanted to know your thoughts.)

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-aff6f4218d64> in <module>
----> 1 s = my_long_func("arg")
~/path/to/my/env/lib/python3.7/site-packages/cachier/core.py in func_wrapper(*args, **kwds)
169                         return _calc_entry(core, key, func, args, kwds)
170             _print(‘No entry found. No current calc. Calling like a boss.’)
–> 171             return _calc_entry(core, key, func, args, kwds)
172
173         def clear_cache():
~/path/to/my/env/lib/python3.7/site-packages/cachier/core.py in _calc_entry(core, key, func, args, kwds)
65         core.mark_entry_being_calculated(key)
66         # _get_executor().submit(core.mark_entry_being_calculated, key)
—> 67         func_res = func(*args, **kwds)
68         core.set_entry(key, func_res)
69         # _get_executor().submit(core.set_entry, key, func_res)
~/path/to/my/src.py in my_long_func.py(kwarg, another_arg)
47 @cachier(stale_after=timedelta(weeks=1))
48 def my_long_func.py(kwarg: str, another_arg: bool = False):
—> 49     res_df = wrapped_func(kwarg=kwarg, another_arg=“string”)
50     if another_arg:
51         return res_df.query(“valid_data == True”)
~/path/to/my/env/lib/python3.7/site-packages/cachier/core.py in func_wrapper(*args, **kwds)
165                     _print(‘No value but being calculated. Waiting.’)
166                     try:
–> 167                         return core.wait_on_entry_calc(key)
168                     except RecalculationNeeded:
169                         return _calc_entry(core, key, func, args, kwds)
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
191         if observer.isAlive():
192             # print(‘Timedout waiting. Starting again…’)
–> 193             return self.wait_on_entry_calc(key)
194         # print(“Returned value: {}”.format(event_handler.value))
195         return event_handler.value
~/path/to/my/env/lib/python3.7/site-packages/cachier/pickle_core.py in wait_on_entry_calc(self, key)
187             recursive=True
188         )
–> 189         observer.start()
190         observer.join(timeout=1.0)
191         if observer.isAlive():
~/path/to/my/env/lib/python3.7/site-packages/watchdog/observers/api.py in start(self)
253     def start(self):
254         for emitter in self._emitters:
–> 255             emitter.start()
256         super(BaseObserver, self).start()
257
~/path/to/my/env/lib/python3.7/site-packages/watchdog/utils/init.py in start(self)
108
109     def start(self):
–> 110         self.on_thread_start()
111         threading.Thread.start(self)
112
~/path/to/my/env/lib/python3.7/site-packages/watchdog/observers/inotify.py in on_thread_start(self)
119     def on_thread_start(self):
120         path = unicode_paths.encode(self.watch.path)
–> 121         self._inotify = InotifyBuffer(path, self.watch.is_recursive)
122
123     def on_thread_stop(self):
~/path/to/my/env/lib/python3.7/site-packages/watchdog/observers/inotify_buffer.py in init(self, path, recursive)
33         BaseThread.init(self)
34         self._queue = DelayedQueue(self.delay)
—> 35         self._inotify = Inotify(path, recursive)
36         self.start()
37
~/path/to/my/env/lib/python3.7/site-packages/watchdog/observers/inotify_c.py in init(self, path, recursive, event_mask)
186         inotify_fd = inotify_init()
187         if inotify_fd == -1:
–> 188             Inotify._raise_error()
189         self._inotify_fd = inotify_fd
190         self._lock = threading.Lock()
~/path/to/my/env/lib/python3.7/site-packages/watchdog/observers/inotify_c.py in _raise_error()
413             raise OSError(“inotify watch limit reached”)
414         elif err == errno.EMFILE:
–> 415             raise OSError(“inotify instance limit reached”)
416         else:
417             raise OSError(os.strerror(err))
</pre>
</details>

About this issue

Original URL
State: open
Created 4 years ago
Comments: 29

Most upvoted comments

@shaypal5 - I’m going to leave my analysis here for the moment (see test2.py comment with edit). It would be good if we could get some others to test this script check that it is reproducable on Linux systems (and does not trigger on non-linux systems).

Edit: I’m pretty confident in my analysis now - see “Changing limits” in the previous comment.

nhairs-lumin on Jan 16, 2024

Okay this time I can actually reproduce it.

Uses same main.py as above.

`test2.py`

### IMPORTS
### ====================================
import sys
import threading

from cachier import cachier

### GLOBALS
### ====================================
COUNT_LOCK = threading.Lock()
COUNT_SUCCESS = 0
COUNT_ERROR = 0

### MAIN
### ====================================
if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Usage: python3 test2.py <cache_dir> <threads>")
        sys.exit(1)

    @cachier(cache_dir=sys.argv[1])
    def cached_func(key) -> str:
        return "test"

    def thread_func(thread_num: int):
        global COUNT_LOCK
        global COUNT_SUCCESS
        global COUNT_ERROR
        print(f"T{thread_num}: Starting ...")
        try:
            cached_func("asdf")
            with COUNT_LOCK:
                COUNT_SUCCESS += 1
        except Exception as e:
            print(f"T{thread_num}: Error - {e!r}")
            with COUNT_LOCK:
                COUNT_ERROR += 1
        return

    print("[STARTING THREADS]")
    threads = []
    for i in range(int(sys.argv[2])):
        thread = threading.Thread(target=thread_func, args=(i,), name=f"T{i}")
        threads.append(thread)
        thread.start()

    print("[JOINING THREADS]")
    for thread in threads:
        try:
            thread.join()
        except Exception as e:
            print(f"Error joining thread {thread.name} - {e!r}")

    print("[STATS]")
    print(f"success = {COUNT_SUCCESS}")
    print(f"error = {COUNT_ERROR}")
    print(f"sanity: counts = {COUNT_SUCCESS + COUNT_ERROR}, expected {sys.argv[2]}")

Running

Ran tests using python3 main.py 1

Low count

% python3 test2.py cachier-test-5d8afbc0cbac4a32 20
[STARTING THREADS]
T0: Starting ...
T1: Starting ...
T2: Starting ...
T3: Starting ...
T4: Starting ...
T5: Starting ...
T6: Starting ...
T7: Starting ...
T8: Starting ...
T9: Starting ...
T10: Starting ...
T11: Starting ...
T12: Starting ...
T13: Starting ...
T14: Starting ...
T15: Starting ...
T16: Starting ...
T17: Starting ...
T18: Starting ...
T19: Starting ...
[JOINING THREADS]
[STATS]
success = 20
error = 0
sanity: counts = 20, expected 20

High count

% python3 test2.py cachier-test-c4366eabb7e4974a 100 
[STARTING THREADS]
T0: Starting ...
T1: Starting ...
T2: Starting ...
T3: Starting ...
T4: Starting ...
T5: Starting ...
T6: Starting ...
T7: Starting ...
T8: Starting ...
T9: Starting ...
T10: Starting ...
T11: Starting ...
T12: Starting ...
T13: Starting ...
T14: Starting ...
T15: Starting ...
T16: Starting ...
T17: Starting ...
T18: Starting ...
T19: Starting ...
T20: Starting ...
T21: Starting ...
T22: Starting ...
T23: Starting ...
T24: Starting ...
T25: Starting ...
T26: Starting ...
T27: Starting ...
T28: Starting ...
T29: Starting ...
T30: Starting ...
T31: Starting ...
T32: Starting ...
T33: Starting ...
T34: Starting ...
T35: Starting ...
T36: Starting ...
T37: Starting ...
T38: Starting ...
T39: Starting ...
T40: Starting ...
T41: Starting ...
T43: Starting ...
T44: Starting ...
T46: Starting ...
T45: Starting ...
T48: Starting ...
T47: Starting ...
T42: Starting ...
T49: Starting ...
T50: Starting ...
T51: Starting ...
T52: Starting ...
T53: Starting ...
T54: Starting ...
T55: Starting ...
T56: Starting ...
T57: Starting ...
T58: Starting ...
T59: Starting ...
T61: Starting ...
T60: Starting ...
T63: Starting ...
T64: Starting ...
T66: Starting ...
T62: Starting ...
T65: Starting ...
T67: Starting ...
T71: Starting ...
T68: Starting ...
T73: Starting ...
T70: Starting ...
T72: Starting ...
T76: Starting ...
T74: Starting ...
T75: Starting ...
T80: Starting ...
T69: Starting ...
T79: Starting ...
T81: Starting ...
T77: Starting ...
T78: Starting ...
T82: Starting ...
T83: Starting ...
T84: Starting ...
T85: Starting ...
T86: Starting ...
T87: Starting ...
T88: Starting ...
T89: Starting ...
T90: Starting ...
T91: Starting ...
T92: Starting ...
T93: Starting ...
T94: Starting ...
T95: Starting ...
T97: Starting ...
T96: Starting ...
T98: Starting ...
T99: Starting ...
[JOINING THREADS]
T69: Error - OSError(24, 'inotify instance limit reached')
T81: Error - OSError(24, 'inotify instance limit reached')
T78: Error - OSError(24, 'inotify instance limit reached')
T84: Error - OSError(24, 'inotify instance limit reached')
T37: Error - OSError(24, 'inotify instance limit reached')
T92: Error - OSError(24, 'inotify instance limit reached')
T97: Error - OSError(24, 'inotify instance limit reached')
T24: Error - OSError(24, 'inotify instance limit reached')
T56: Error - OSError(24, 'inotify instance limit reached')
T76: Error - OSError(24, 'inotify instance limit reached')
T98: Error - OSError(24, 'inotify instance limit reached')
T59: Error - OSError(24, 'inotify instance limit reached')
[STATS]
success = 88
error = 12
sanity: counts = 100, expected 100

While running this test I checked how many instances were running (see above comment for SO link to where this snippet is from).

% find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -exec sh -c 'cat $(dirname {})/../cmdline; echo ""' \; 2>/dev/null | wc -l
128

Running it again after it was done:

% find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -exec sh -c 'cat $(dirname {})/../cmdline; echo ""' \; 2>/dev/null | wc -l
40

This matches what we’d expect of only being able to handle 88 more observers (88 + 40 == 128 (aka the limit)).

Running it again with exactly 88 observers: (note 88 is for my system while it is running while I write this, with my max_user_instances)

% python3 test2.py cachier-test-54654296ffdc26c0 88
[STARTING THREADS]
T0: Starting ...
T1: Starting ...
T2: Starting ...
T3: Starting ...
T4: Starting ...
T5: Starting ...
T6: Starting ...
T7: Starting ...
T8: Starting ...
T9: Starting ...
T10: Starting ...
T11: Starting ...
T12: Starting ...
T13: Starting ...
T14: Starting ...
T15: Starting ...
T16: Starting ...
T17: Starting ...
T18: Starting ...
T19: Starting ...
T20: Starting ...
T21: Starting ...
T22: Starting ...
T23: Starting ...
T24: Starting ...
T25: Starting ...
T26: Starting ...
T27: Starting ...
T28: Starting ...
T29: Starting ...
T30: Starting ...
T31: Starting ...
T32: Starting ...
T33: Starting ...
T34: Starting ...
T35: Starting ...
T37: Starting ...
T38: Starting ...
T36: Starting ...
T40: Starting ...
T42: Starting ...
T43: Starting ...
T44: Starting ...
T46: Starting ...
T41: Starting ...
T48: Starting ...
T45: Starting ...
T50: Starting ...
T39: Starting ...
T47: Starting ...
T51: Starting ...
T49: Starting ...
T55: Starting ...
T56: Starting ...
T54: Starting ...
T58: Starting ...
T52: Starting ...
T61: Starting ...
T62: Starting ...
T63: Starting ...
T64: Starting ...
T66: Starting ...
T67: Starting ...
T68: Starting ...
T59: Starting ...
T70: Starting ...
T60: Starting ...
T72: Starting ...
T73: Starting ...
T57: Starting ...
T69: Starting ...
T77: Starting ...
T65: Starting ...
T71: Starting ...
T53: Starting ...
T81: Starting ...
T83: Starting ...
T84: Starting ...
T76: Starting ...
T78: Starting ...
T87: Starting ...
[JOINING THREADS]
T82: Starting ...
T85: Starting ...
T79: Starting ...
T74: Starting ...
T86: Starting ...
T75: Starting ...
T80: Starting ...
[STATS]
success = 88
error = 0
sanity: counts = 88, expected 88

Changing limits

Lets try changing the limits:

sudo sysctl fs.inotify.max_user_instances=200

If we now run the test again we should expect number of errors equal to number of already consumed instances (currently 40)

% python3 test2.py cachier-test-75a71579e729aa52 200

[STARTING THREADS]
T0: Starting ...
# ...
T178: Starting ...
[JOINING THREADS]
T168: Error - OSError(24, 'inotify instance limit reached')
# ...
T160: Error - OSError(24, 'inotify instance limit reached')
[STATS]
success = 160
error = 40
sanity: counts = 200, expected 200

Analysis

Assuming this has correctly identified the issue:

This error only affects Linux using inotify (this will be majority of Linux users as inotify requires Linux 2.6 which was first released in 2003).

The error is triggered when a number of Cachier Pickle cores are waiting on a "being_calculated" value (which each create a new watchdog.Observer) which exhuasts the allowed inotify instances available to a user (cat /proc/sys/fs/inotify/max_user_instances). The exact number of waiting instances depends on the max_user_instances and the number of inotify instances being consumed by other processes.

nhairs-lumin on Jan 16, 2024

Update:

I cannot reproduce the problem with the below scripts.

Why? It turns out I had a lot of hanging consumer threads that was probably taking me over the max_user_instances (cat /proc/sys/fs/inotify/max_user_instances) which is probably being reported here (instead of my incorrect assumption above of max_user_watches).

(I identified this hanging threads usings the bash find snippet in https://unix.stackexchange.com/a/15549).

Once I shutdown those consumer threads I could run the below tests with 70000 junk files which is well over my system’s max_user_watches of 65536)

Files

`main.py`

### IMPORTS
### ====================================
import os
import sys
import secrets
import shutil
import time

from cachier import cachier

### CONSTANTS
### ====================================
CACHE_DIR = f"cachier-test-{secrets.token_hex(8)}"
TIME = 20

### FUNCTIONS
### ====================================
def fill_directory(count: int = 9000) -> None:
    """Fill the cache dir with `count` junk files"""
    for i in range(count):
        filename = f"{CACHE_DIR}/{i}"
        with open(filename, "wb") as junk_file:
            junk_file.write(secrets.token_bytes(8))
    return


@cachier(cache_dir=CACHE_DIR)
def cached_func(key) -> str:
    time.sleep(TIME)
    return secrets.token_hex(8)

def run_test(junk_count: int) -> None:
    print("[ SETUP ]")
    print(f"Using cache_dir: {CACHE_DIR}")
    print(f"Creating {junk_count} junk files...")
    fill_directory(junk_count)

    print(f"[ GET READY ]")
    print(
        "The test is about to start, in another window you will need to run the "
        f"following command. You will have {TIME} seconds to run the command before "
        "the test ends."
    )
    print(f"python3 test.py {CACHE_DIR}")

    print("[ TEST RUNNING ]")
    cached_func("asdf")

    print("[ TEST ENDING ]")
    return

### MAIN
### ====================================
if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python3 main.py <junk_count>")
        sys.exit(1)
    try:
        os.mkdir(CACHE_DIR)
        run_test(int(sys.argv[1]))
    finally:
        if os.path.isdir(CACHE_DIR):
            shutil.rmtree(CACHE_DIR)

`test.py`

### IMPORTS
### ====================================
import sys
from cachier import cachier

### MAIN
### ====================================
if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python3 test.py <cache_dir>")
        sys.exit(1)

    @cachier(cache_dir=sys.argv[1])
    def cached_func(key) -> str:
        return "test"

    cached_func("asdf")

nhairs-lumin on Jan 16, 2024

@shaypal5 - I suspect that the underlying issue has not been fixed as I’ve just hit this issue.

Edit: part of my hypothesis was wrong (is not to do with number of files in cache_dir)

Details

Situation

Long running consumer completing same task (downloading image from URL), run across about 6000 tasks.

Using Cachier with Pickle core.

Versions

Python 3.11.7
Linux 6.2.0-39-generic #40~22.04.1-Ubuntu

cachier==2.2.2
watchdog==3.0.0

Traceback

 Traceback (most recent call last):
 File "/home/devuser/.local/lib/python3.11/site-packages/dramatiq/worker.py", line 485, in process_message
 res = actor(*message.args, **message.kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/devuser/.local/lib/python3.11/site-packages/dramatiq/actor.py", line 177, in __call__
 return self.fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/code/src/some_service/consumers/bars.py", line 18, in import_foo_bar
 imported_bar = importer.import_bar(foo_id, allow_update)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/code/src/some_service/bars/foo_importer.py", line 145, in import_bar
 self.import_image(
 File "/code/src/some_service/bars/foo_importer.py", line 213, in import_image
 data = self.foo.get_image_data(foo_data_id, foo_size, image_extention)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/devuser/.local/lib/python3.11/site-packages/cachier/core.py", line 308, in func_wrapper
 return core.wait_on_entry_calc(key)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/devuser/.local/lib/python3.11/site-packages/cachier/pickle_core.py", line 267, in wait_on_entry_calc
 observer.start()
 File "/home/devuser/.local/lib/python3.11/site-packages/watchdog/observers/api.py", line 261, in start
 emitter.start()
 File "/home/devuser/.local/lib/python3.11/site-packages/watchdog/utils/__init__.py", line 92, in start
 self.on_thread_start()
 File "/home/devuser/.local/lib/python3.11/site-packages/watchdog/observers/inotify.py", line 119, in on_thread_start
 self._inotify = InotifyBuffer(path, self.watch.is_recursive)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/devuser/.local/lib/python3.11/site-packages/watchdog/observers/inotify_buffer.py", line 37, in __init__
 self._inotify = Inotify(path, recursive)
 ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/devuser/.local/lib/python3.11/site-packages/watchdog/observers/inotify_c.py", line 167, in __init__
 Inotify._raise_error()
 File "/home/devuser/.local/lib/python3.11/site-packages/watchdog/observers/inotify_c.py", line 430, in _raise_error
 raise OSError(errno.EMFILE, "inotify instance limit reached")
 OSError: [Errno 24] inotify instance limit reached

Misc

Creating cached functions using the following:

self.get_image_data = cachier.cachier(
  cache_dir=".tmp/cachier",
  seperate_files=True,
  pickle_reload=False,
)(parent.get_image_data)

Cache directory

.tmp/cachier % ls -a | wc -l
8309

.tmp/cachier % ls -a | grep get_image_data | wc -l
6099

OLD INCORRECT Analysis

Here’s what I think is happening and how it might be resolved.

First - there is a limit to how many files can be watched using inotofy (which is what watchdog is using with the Linux optimal Observer). According to this page the default max_user_watches limit on most Linux systems is 8192 - this is per user so includes anything else running.

A workaround would be to increase this limit on the system being used, but this would just “postpone” the error.

Second - I believe that there are 2 scenarios that will trigger this limit being hit in general use.

a) cachier is being used across a large number of different functions b) a function is using seperate_files=True

Third - the error is only thrown if a call to a cached function results in a 'being_calculated' result - waiting for a result is the only time the observer is created . This observer watches the entire cache directory - so it doesn’t really matter which scenario fills the cache dir as long as the number of files is larger than the limit (in fact it might be possible that even filling the cache dir with non-cachier related files will trigger the error).

For the fix I don’t think anything like “adding subdirectories per function” will help because of case b) above. It’s unclear if watchdog Observers are designed to watch a single file (the current implementation is to watch the entire cache_dir and filter on the expected file) - but this might be a solution. Some other work arounds would be to change what library we use for watching, or create a directory per key (kinda gross tbh), or simply create our own polling watcher (with a small sleep interval to avoid smashing the CPU - e.g. time.sleep(0.05) # 50ms).

I’ll see if I can find a way to reproduce the problem.

nhairs-lumin on Jan 16, 2024

Oh, I forgot about your PR to fix this, @ericmjl . Please ping me again freely to make me take a look, but I suggest you two try to take a jab at this.

shaypal5 on Jun 30, 2020