sagemaker-python-sdk: `requirements.txt` not working for PytorchModel

I have a pre-trained model, now trying to create an endpoint using Sagemaker, my folder structure like this “model.tar.gz” looks like this:

model |- config.json |- pytorch_model.bin |- special_tokens_map.json |- spiece.model |- tokenizer_config.json |- training_args.bin code |- inference.py | - requirements.txt

running following script to create endpoint:

pytorch_model = PyTorchModel( model_data='s3://mck-dl-ai-studio/answer_card/answercard.tar.gz', role=role, entry_point='inference.py', framework_version="1.3.1")

predictor = pytorch_model.deploy(instance_type='ml.t2.medium', initial_instance_count=1)

An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message “No module named ‘transformers’”. See https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/pytorch-inference-2020-07-20-16-45-51-564 in account xxxxxx for more information.

what I am missing here tried adding source_dir and py_version but no success

@Serhiy-Shekhovtsov right.

@laurenyu, it looks like we can’t store anything under /code of model.tar since the entire directory tree under code is always removed during repacking: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/utils.py#L518

The only way to get it working is to point source_dir to a folder containing requirements.txt. All the content inside source_dir is copied to opt/ml/model/code on target machine. The file specified as entry_point will also be coped there.

Another thing - since the entire content and structure of source_dir is packaged and copied to the container it doesn’t make much sense to build the model tar manually every time and then create the model using PyTorchModel.deploy. Instead, you can store the model, requirements file and the code in a folder provided as source_dir and only run PyTorchModel.deploy. (model_data is a required parameter but you can use an empty tar archive there).

@vlordier how requirements.txt can be stored there using only Sagemaker SDK with Sagemaker’s default PyTorch image? @vlordier, could you please provide an example of how to force PyTorchModel to consider requirements.txt? My code is pretty straightforward and repeats the stages described here. @laurenyu 's suggestion of having model.tar.gz file of specific structure did not work as well.

Ok, so as @vlordier wrote to be able to use requirements.txt you should use PyTorch serving container of version >=1.3.1, since 1.2.0 version doesn’t support requirements file (see sagemaker-inference-toolkit: support of requirements file was introduced in 1.1.0).

Having requirements.txt in the /code directory with a pytorch image 1.3.1 works.

On Tue, 24 Dec 2019 at 22:33, Lenar Gabdrakhmanov notifications@github.com wrote:

I’ve run into the same problem described by @dshahrokhian https://github.com/dshahrokhian. Passing directory that contains requirements file as source_dir to the PyTorchModel (as @pinal-patel https://github.com/pinal-patel suggested) does not help at all. Is there any alternative besides explicitly running package installation

import subprocessimport sys

subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “some_required_package”])

within the entry point code?

I’m using sagemaker==1.49.0.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws/sagemaker-python-sdk/issues/868?email_source=notifications&email_token=ABJQ4NKG6GKLIIWQTH75RKTQ2J5ZTA5CNFSM4H2PIIYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTUGIQ#issuecomment-568804130, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJQ4NKG3IZLXTRUCAK4X33Q2J5ZTANCNFSM4H2PIIYA .

those files should also be able to go under code/ in the model tar