xgboost: [Discussion] XGBoost Predict is not threadsafe

I would like to start a formal discussion on potential fixes for the thread safety issue of the xgboost predict method. It is noted in a few places that the predict method is not thread safe [ in this issue and in the Python API and in the C++ API ]

A group of my colleagues at Capital One (@ahmadia, @gforsyth, @mmccarty, @stephenpardy) has come together and looked into a few possible suggested changes for this problem and would like to kick off a greater discussion on how best to improve user experience.

The best way to go about doing this would be to dive into the C++ layer and make the underlying methods thread safe. If anybody wishes to take this on, this would be the ideal change. However, this would be an intrusive set of code changes and a less intrusive just as effective change can be implemented at the Python layer.

We propose the following:

• add a lock to the existing ctypes API call so that the predict method cannot be called in a multithreaded manner unintentionally. This prevents users from shooting themselves in the foot.
• build a new, threadsafe predict method or object. This method or object will use threading.local memory to store a copy of the booster object. The primary booster object should be automatically copied into thread local memory upon fork.