clipper: How to debug cause of "ImportError when running Container"

I am a bit stuck: I’m trying to deploy a pytorch model which runs fine on my local computer but hangs when trying to deploy it. The model’s docker container gives me this error log:

$ docker logs 4437076d479c
Starting PyTorchContainer container
Connecting to Clipper with default port: 7000
Encountered an ImportError when running container. You can use the pkgs_to_install argument when calling clipper_admin.build_model() to supply any needed Python packages.

It seems like it is not able to import some dependency. I installed the same model in a clean anaconda environment on my computer and checked exactly which dependencies I needed to install. I listed these in pkgs_to_install. I have inspected all the log files of the docker containers and I can’t find the actual error message which could give me a hint as to which dependency is missing. May some error log be getting swallowed somewhere?

The model I’m trying to run is the xinntao/ESRGAN for image super-resolution.

Dependencies: Python 3 PyTorch >= 0.4.0 Python packages: pip install numpy opencv-python

The command to deploy the model:

deploy_pytorch_model(
    clipper_conn,
    name="superresolution-model",
    version=1,
    input_type="bytes",
    func=image_enhance,
    pytorch_model=model,
    pkgs_to_install=['opencv-python','numpy','six', 'Pillow','wheel',]
    )

the clipper_service.py is where I define the model and try to deploy.

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Comments: 15 (6 by maintainers)

Most upvoted comments

The import error is because the script is importing architecture.py and architecture.py is importing block.py. Cloupickle (package we use to serialize function) is not able to pick up and bundle these scripts directly. Basically, when we deserialize the script in container, it will run something like import architecture as arch and then architecture will import block as B.

If you consolidate all three files architecture.py, block.py, and clipper_service.py into a single clipper_service file, you should be good to go.

Thank you guys for your fast response… @simon-mo I had actually tried consolidating those 3 files into one and had not gotten any further. But it’s good to know I need to do that. I still haven’t completely wrapped my head around what gets run where.

I will try building a custom model container with the latest pytorch and consolidate the files into one and let you know if it works. Thanks!