ml-commons: [BUG]Model content hash can't match original hash value
What is the bug?
Model content hash can't match original hash value
How can one reproduce the bug?
I tried with the current code base. I executed the command ./gradlew run to test.
- First I upload the model
POST /_plugins/_ml/models/_upload
{
"name": "huggingface/sentence-transformers/all-MiniLM-L12-v2",
"version": "1.0.1",
"model_format": "TORCH_SCRIPT"
}
-
Then Load the model in memory. This time it works fine.
http://localhost:9200/_plugins/_ml/models/l-bnSYcBQg5VYC5uIxWy/_load -
I also generated the embedding with the loaded model and it works fine.
-
Now I unload the model:
http://localhost:9200/_plugins/_ml/models/l-bnSYcBQg5VYC5uIxWy/_unload -
And now I try to deploy again. Then I face the error:
[2023-04-03T18:42:42,801][ERROR][o.o.m.m.MLModelManager ] [integTest-0] Model content hash can't match original hash value
[2023-04-03T18:42:42,804][ERROR][o.o.m.a.f.TransportForwardAction] [integTest-0] deploy model failed on all nodes, model id: l-bnSYcBQg5VYC5uIxWy
[2023-04-03T18:42:42,804][INFO ][o.o.m.a.f.TransportForwardAction] [integTest-0] deploy model done with state: DEPLOY_FAILED, model id: l-bnSYcBQg5VYC5uIxWy
[2023-04-03T18:42:42,804][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [integTest-0] deploy model task done m-buSYcBQg5VYC5uMhUu
What is the expected behavior? Model should load again.
What is your host/environment?
- OS: [e.g. iOS]
- Version [e.g. 2.7]
- Plugins
Do you have any screenshots? If applicable, add screenshots to help explain your problem.
Do you have any additional context? Add any other context about the problem.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 23 (9 by maintainers)
That and the plugin-security.policy change. I am not able to repro it after my fix.
I am having the same problem when uploading my custom model, but then when registering the model:
Also running OpenSearch in Docker on Mac OS. I have no problems with the pre-loaded models, even when uploading them from URL. Any idea how to fix this?
Edit: Found the solution. You need to generate the checksum of your zip file with the custom model and pass it when uploading the model in the
model_content_hash_valuefield. You can generate the checksum like this:shasum -a 256 paraphrase-multilingual-mpnet-base-v2.zipThe model hash after _unload/_load differed because the zip file in
data/ml_cache/models_cache/deploy/<model_id>never got deleted and as a result, it kept getting added to the same zip file upon re-load.https://github.com/opensearch-project/ml-commons/blob/510f50be4c102027bb05e4f459ea2a9b52f62765/plugin/src/main/java/org/opensearch/ml/model/MLModelManager.java#L1080
I noticed that the zip file stayed and its size kept growing.
I found two issues related to this.
The first is that the security manager was disallowing
setReadOnly(commons-io tries to change the file permissions to be able to delete model files and directories).After I added the necessary permission to the plugin-security.policy, _unload/_load started to work.
While I was looking into why the delete was failing on Mac, I came across this issue:
https://issues.apache.org/jira/browse/IO-787
It says there was regression in commons-io 2.11 (this is the version ml-commons is currently using) specifically related to the forceDelete method that we’re using. Strangely, I don’t see what code change they made to fix that in commons-io 2.12, but since we are currently on 2.11 and it has a known issue for Mac, I am upgrading commons-io to the latest 2.15.
@juntezhang Can you create a separate issue for doing input validation on custom model registration?
had a chat with @Saikumar282 on opensearch public slack, this is an expected problem on Darwin which ML-Commons doesn’t officially support yet.
@saratvemulapalli This is happening with docker which runs on Mac OS as well. Any alternatives!!!
Thanks @dhrubo-os.
I was able to reproduce the problem.