BentoML: bug: Http Server is not worked with Pytorch-Lightning without 'del sys.modules["prometheus_client"]'

Describe the bug

benotml serve {my_service}.py:svc --port 3001 is loaded and stdout shows log

Prometheus metrics for HTTP BentoServer from "{my_service}.py:svc" can be accessed at http://localhost:3001/metrics
2023-04-25T10:49:47+0900 [INFO] [cli] Starting development HTTP BentoServer from "{my_service}.py:svc" listening on http://0.0.0.0:3001 (Press CTRL+C to quit)

However, when I access to ‘http://0.0.0.0:3001’, it falls in infinite loading loop without any kinds of warning or error message.

When I add ‘del sys.modules[“prometheus_client”]’, the issue is resolved and I can access http server via localhost web.

I guess it is caused when we import 3rd party library using prometheus client (e.g. pytorch lightning)

To reproduce

I find that just adding a single line

import torch
import pytorch_lightning as pl

...

in your example of ‘custom_runner/torch_hub_yolov5/service.py’ triggers this bug.

(please run bentoml serve service.py:svc)

However, after add this single line, it will resolve this issue

import torch
import pytorch_lightning as pl

del sys.modules["prometheus_client"]
...

Expected behavior

No response

Environment

bentoml: 1.0.18 python: 3.8.9 pytorch-lightning: 2.0.0

About this issue

Original URL
State: open
Created a year ago
Comments: 18 (8 by maintainers)

Most upvoted comments

Here is a solution, TLDR: add the import of pytorch_lightning and torch after benotml

import sys
import bentoml
from bentoml.io import Image
from bentoml.io import PandasDataFrame


class Yolov5Runnable(bentoml.Runnable):
    SUPPORTED_RESOURCES = ("nvidia.com/gpu", "cpu")
    SUPPORTS_CPU_MULTI_THREADING = True

    def __init__(self):
        import torch # Here instead!
        import pytorch_lightning as pl
        self.model = torch.hub.load("ultralytics/yolov5", "yolov5s")

ragyhaddad on Dec 12, 2023

You may add print(sys.modules.get("prometheus_client")) in different positions to see where it is imported in the script. Like following:

import sys
print("before script:", sys.modules.get("prometheus_client"))  # 1
import bentoml
from bentoml import Service

puzzle_runner = bentoml.pytorch.get("8-puzzle:latest").to_runner()
print("after runner:", sys.modules.get("prometheus_client"))  # 2
svc = Service(
    "Test",
    runners=[puzzle_runner],
)
print("after service:", sys.modules.get("prometheus_client"))  # 3

Since I can’t reproduce the issue either, 2 and 3 should be fine. I suspect it is already imported at 1.

frostming on Jul 3, 2023