sanic: LIGHTGBM pickle output not working with multiple SANIC workers (>1) but working with single worker.

I am trying to load machine learning model output with Sanic. I have loaded the output in the main method(defined globally). It works fine when I set sanic worker as 1 but not not working with multiple sanic workers when defined globally. My code waits for indefinite time for model to generate desired result.

Its works when I load model output inside the function (e.g. here in the method modelrun) even if sanic workers >= 1

It works when I load model output globally(outside the function) but only if sanic workers = 1

It doesnot work when I load model output globally(outside the function) if sanic workers > 1

Versions used

python 3.8 sanic==20.12.1 lightgbm==3.3.1 numpy==1.20.1 pandas==1.2.4 scikit-learn==1.0.2 scipy==1.6.2

Code

import pickle
import sanic

# endpoint model run
@app.route("/get_data", methods=['POST'])
async def get_model_output(request):
    output = modelrun(df, lbg_model_smote_sel_vars)
    return output

if __name__ == '__main__':
    df = pd.DataFrame()
    p_file_path = "/Users/pratiksha/radar/Radar/utils/static_data/FModel_06Jan_Smote_Sel_Vars_48.dat"
    pickle_file = open(path, 'rb')
    lbg_model_smote_sel_vars = pickle.load(pickle_file)

    app.run(host=app_host, port=int(app_port), debug=True,
            auto_reload=True, workers=int(10))

# defined in other directory
def modelrun(df_f, lbg_model_smote_sel_vars):
    training_pred_smote = lbg_model_smote_sel_vars.predict_proba(df_f)
    return training_pred_smote

stack trace

2022-01-06 16:32:02,391 - sanic.root - INFO - Goin' Fast @ http://127.0.0.1:8000
2022-01-06 16:32:02,428 - sanic.root - INFO - Starting worker [6047]
2022-01-06 16:32:02,431 - sanic.root - INFO - Starting worker [6048]
2022-01-06 16:32:02,433 - sanic.root - INFO - Starting worker [6049]
2022-01-06 16:32:02,439 - sanic.root - INFO - Starting worker [6050]
2022-01-06 16:32:02,442 - sanic.root - INFO - Starting worker [6051]
2022-01-06 16:32:02,444 - sanic.root - INFO - Starting worker [6052]
2022-01-06 16:32:02,447 - sanic.root - INFO - Starting worker [6053]
2022-01-06 16:32:02,450 - sanic.root - INFO - Starting worker [6054]
2022-01-06 16:32:02,451 - sanic.root - INFO - Starting worker [6055]
2022-01-06 16:32:02,453 - sanic.root - INFO - Starting worker [6056]

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (9 by maintainers)

Most upvoted comments

@sysu18364109 because the workers do not have a shared context. You’re consuming more memory for each individual worker, as expected, because the model is loaded in each worker.

If you want a shared worker context and want to use something prebuilt, take a look at https://github.com/ashleysommer/sanic-synchro-ctx

sjsadowski on Apr 12, 2022

Thanks, @SaidBySolo. I upgraded the Sanic version and now it is working with the context property. Still didn’t get one thing is that why the pickle was not loaded in the listener with a local variable. It is a generic python function. I did not get any errors in that process. Is there any Sanic process involved which was failing because of this and I was not getting any error for that?

Hey, I have encounter the same problem, also discovered that loading global variables that defined in __main__ into the workers doesn’t work. In my case, I have tried to load a yolov4 model in __main__ and use the same model object among these workers. But it failed, sanic workers were just blocking somewhere. If I load a new model to the context per worker in before_server_start , it works, but several times of GPU memory are consumed. So I wondered if global variables are not visible in multiple worker cases, and duplicate these variables/objects into the context is the only solution?

silveryu1999 on Apr 11, 2022