serve: Worker thread stuck in die state

@msaroufim @lxning

I can successfully repro it in my local with following setup

setup

• in config.properties, I put min/max worker=2
• start torchserve in local, waiting for it’s fully up
• have a simple python GRPC client

import sys
import grpc
import time
import argparse
import inference_pb2
import inference_pb2_grpc
import management_pb2
import management_pb2_grpc
from datetime import datetime
from google.protobuf import empty_pb2


parser = argparse.ArgumentParser()
parser.add_argument("--timeout", required=True)
parser.add_argument("--port", required=True)
parser.add_argument("-n", required=True)


if __name__ == '__main__':
    args, _ = parser.parse_known_args()
    port = args.port
    num_runs = int(args.n)

    with open('image.jpg', 'rb') as f:
        data = f.read()
    input_data = {'data': data}
    request = inference_pb2.PredictionsRequest(model_name="easyocr",
                                               input=input_data)

    channel = grpc.insecure_channel(f'localhost:{port}')
    stub = inference_pb2_grpc.InferenceAPIsServiceStub(channel)
    for i in range(num_runs):
        try:
            start = datetime.now()
            response = stub.Predictions(request, timeout=float(args.timeout))
            print("request time:", datetime.now() - start)
            print(response)
        except Exception as e:
            print(e)

• in one shell tab, run the client with normal timeout(2s) and very large number of requests, simulating consistent stream of normal requests):

python3 torchserve_grpc_client.py --port=17100 --timeout=2 -n=100000000

• in another shell tab, run the client with very short timeout(0.02s), simulating some heavy/bad requests that will eventually timeout:

python3 torchserve_grpc_client.py --port=17100 --timeout=0.02 -n=100

Result

it takes torchserve several seconds to several minutes to resolve the issue and tab1 simulation’s output is back to normal, but also likely that all workers will stuck forever and tab1 will see flood of errors.

@hgong-snap confirming we can finally repro, will get back to you with a solution soon

@hgong-snap Great, thank you for the verification.

@lxning verified onpytorch/torchserve-nightly:latest-gpu and seems issue is solved!

@hgong-snap I verified master branch on both local host and docker nightly build.

You can fetch torchserve nightly build at https://hub.docker.com/r/pytorch/torchserve-nightly/tags.

• docker pull pytorch/torchserve-nightly:latest-gpu
• docker pull pytorch/torchserve-nightly:latest-cpu

for repro, I guess best way is to cancel the request on client side? or set some short-enough GRPC timeout so that internally GRPC will cancel the request?

With that said,

1. I can’t know if a worker is in die stuck state or not.
2. Even if I know a worker is died, I can’t know which one died, nor kill/restart this specific worker

so basically there’s really no way to catch the worker die stuck error?