netbox: Netbox API: request failed with code 500 - Cannot assing >
NetBox version
2.11.6
Python version
3.7
Steps to Reproduce
We have multiple Scripts that are running against our Netbox Instance. Some of them are utilizing read only tokens and others read_write tokens. From time to time the following exception occurs:
The request failed with code 500 Internal Server Error {'error': 'Cannot assign <SimpleLazyObject: <django.contrib.auth.models.AnonymousUser object at 0x7fe517e70780>>": "ObjectChange.user'" must be a "User" instance.', 'exception': 'ValueError', 'netbox_version': '2.11.6', 'python_version': '3.7.8'}
After that we have Changes on Objects with the change_user of the read_only token. Maybe there is any kind of race condition or something like that. Does anybody has an idea, how to further investigate this problem?
Expected Behavior
There is no exception and change user are not mixed with read_only user.
Observed Behavior
Exception occurs and change_user is read_only user
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 33 (28 by maintainers)
Commits related to this issue
- Merge pull request #7861 from netbox-community/7657-threadsafe-changelog Fixes #7657: Make request/webhook caching thread-safe — committed to netbox-community/netbox by jeremystretch 3 years ago
- Changelog for #7657 — committed to netbox-community/netbox by jeremystretch 3 years ago
Finally found some time to really dig into this today. I moved the request object and webhook queue into thread-local storage, which should resolve the “cross talk” among threads without significantly impacting multi-threaded performance.
I’ve opened draft PR #7861 to propose my changes. Please test and let me know if you encounter any issues.
And a huge thank you to @alinmear, @drygdryg, @tyler-8, @robertpenz, and everyone else who helped identify and resolve this bug!
Just for the record: we are running our installation with the changes from https://github.com/tyler-8/netbox/tree/objectchange_threadsafe till now and have no problems regarding threading so far.
We patched our Installation with changes from https://github.com/tyler-8/netbox/tree/objectchange_threadsafe and the threadlock. We updated our worker / thread gunicorn config:
The problem could not be reproduced so far. We are keeping an eye on it.
This keeps coming up, and unfortunately has proven quite difficult to squash. How are you running NetBox? With gunicorn or uWSGI? Can you share your WSGI configuration?
The current understanding (I think) is that this bug can be side-stepped by limiting each worker to a single thread. @tyler-8 please correct me if that’s wrong.