ml-commons: [BUG]Model content hash can't match original hash value

What is the bug? Model content hash can't match original hash value

How can one reproduce the bug?

I tried with the current code base. I executed the command ./gradlew run to test.

  1. First I upload the model
POST /_plugins/_ml/models/_upload
{
  "name": "huggingface/sentence-transformers/all-MiniLM-L12-v2",
  "version": "1.0.1",
  "model_format": "TORCH_SCRIPT"
}
  1. Then Load the model in memory. This time it works fine. http://localhost:9200/_plugins/_ml/models/l-bnSYcBQg5VYC5uIxWy/_load

  2. I also generated the embedding with the loaded model and it works fine.

  3. Now I unload the model: http://localhost:9200/_plugins/_ml/models/l-bnSYcBQg5VYC5uIxWy/_unload

  4. And now I try to deploy again. Then I face the error:

[2023-04-03T18:42:42,801][ERROR][o.o.m.m.MLModelManager   ] [integTest-0] Model content hash can't match original hash value
[2023-04-03T18:42:42,804][ERROR][o.o.m.a.f.TransportForwardAction] [integTest-0] deploy model failed on all nodes, model id: l-bnSYcBQg5VYC5uIxWy
[2023-04-03T18:42:42,804][INFO ][o.o.m.a.f.TransportForwardAction] [integTest-0] deploy model done with state: DEPLOY_FAILED, model id: l-bnSYcBQg5VYC5uIxWy
[2023-04-03T18:42:42,804][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [integTest-0] deploy model task done m-buSYcBQg5VYC5uMhUu

What is the expected behavior? Model should load again.

What is your host/environment?

  • OS: [e.g. iOS]
  • Version [e.g. 2.7]
  • Plugins

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context? Add any other context about the problem.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 23 (9 by maintainers)

Most upvoted comments

That and the plugin-security.policy change. I am not able to repro it after my fix.

I am having the same problem when uploading my custom model, but then when registering the model:

{
    "task_type": "REGISTER_MODEL",
    "function_name": "TEXT_EMBEDDING",
    "state": "FAILED",
    "worker_node": [
        "Etp4k_gISGipfSbPFIZIUg"
    ],
    "create_time": 1697027593551,
    "last_update_time": 1697027809366,
    "error": "model content changed",
    "is_async": true
}

Also running OpenSearch in Docker on Mac OS. I have no problems with the pre-loaded models, even when uploading them from URL. Any idea how to fix this?

Edit: Found the solution. You need to generate the checksum of your zip file with the custom model and pass it when uploading the model in the model_content_hash_value field. You can generate the checksum like this: shasum -a 256 paraphrase-multilingual-mpnet-base-v2.zip

The model hash after _unload/_load differed because the zip file in data/ml_cache/models_cache/deploy/<model_id> never got deleted and as a result, it kept getting added to the same zip file upon re-load.

https://github.com/opensearch-project/ml-commons/blob/510f50be4c102027bb05e4f459ea2a9b52f62765/plugin/src/main/java/org/opensearch/ml/model/MLModelManager.java#L1080

I noticed that the zip file stayed and its size kept growing.

I found two issues related to this.

The first is that the security manager was disallowing setReadOnly (commons-io tries to change the file permissions to be able to delete model files and directories).

[2023-11-04T15:01:28,207][ERROR][o.o.m.e.a.DLModel        ] [integTest-0] Failed to deploy model eRPym4sBExl3TU0X9MGS
 java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
     at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
     at java.base/java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
     at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
     at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkReadExtended(UnixFileAttributeViews.java:186) ~[?:?]
     at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.readAttributes(UnixFileAttributeViews.java:253) ~[?:?]
     at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.readAttributes(UnixFileAttributeViews.java:168) ~[?:?]
     at org.apache.commons.io.file.PathUtils.setReadOnly(PathUtils.java:927) ~[commons-io-2.11.0.jar:2.11.0]
     at org.apache.commons.io.file.PathUtils.deleteFile(PathUtils.java:485) ~[commons-io-2.11.0.jar:2.11.0]
     at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:392) ~[commons-io-2.11.0.jar:2.11.0]
     at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1341) ~[commons-io-2.11.0.jar:2.11.0]
     at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:324) ~[commons-io-2.11.0.jar:2.11.0]
     at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1192) ~[commons-io-2.11.0.jar:2.11.0]
     at org.opensearch.ml.engine.algorithms.DLModel.lambda$loadModel$1(DLModel.java:256) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
     at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) [?:?]
     at org.opensearch.ml.engine.algorithms.DLModel.loadModel(DLModel.java:242) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
     at org.opensearch.ml.engine.algorithms.DLModel.initModel(DLModel.java:138) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
     at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
     at org.opensearch.ml.model.MLModelManager.lambda$deployModel$49(MLModelManager.java:960) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
     at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
     at org.opensearch.ml.model.MLModelManager.lambda$retrieveModelChunks$55(MLModelManager.java:1080) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
     at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
     at org.opensearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:78) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
     at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
     at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
     at java.base/java.lang.Thread.run(Thread.java:833) [?:?]

After I added the necessary permission to the plugin-security.policy, _unload/_load started to work.

While I was looking into why the delete was failing on Mac, I came across this issue:

https://issues.apache.org/jira/browse/IO-787

It says there was regression in commons-io 2.11 (this is the version ml-commons is currently using) specifically related to the forceDelete method that we’re using. Strangely, I don’t see what code change they made to fix that in commons-io 2.12, but since we are currently on 2.11 and it has a known issue for Mac, I am upgrading commons-io to the latest 2.15.

@juntezhang Can you create a separate issue for doing input validation on custom model registration?

@saratvemulapalli This is happening with docker which runs on Mac OS as well. Any alternatives!!!

had a chat with @Saikumar282 on opensearch public slack, this is an expected problem on Darwin which ML-Commons doesn’t officially support yet.

@saratvemulapalli This is happening with docker which runs on Mac OS as well. Any alternatives!!!

Thanks @dhrubo-os.

I was able to reproduce the problem.

[2023-06-20T13:01:03,385][ERROR][o.o.m.m.MLModelManager   ] [integTest-0] Model content hash can't match original hash value

{
                "_index": ".plugins-ml-task",
                "_id": "qXJl2ogBrrwPAYySaz6W",
                "_version": 3,
                "_seq_no": 12,
                "_primary_term": 1,
                "_score": 1.0,
                "_source": {
                    "last_update_time": 1687291263388,
                    "create_time": 1687291259798,
                    "is_async": true,
                    "function_name": "TEXT_EMBEDDING",
                    "worker_node": [
                        "bBpL_UF-SkaO6MjgEhr33w"
                    ],
                    "model_id": "pnIv2ogBrrwPAYySjz7n",
                    "state": "FAILED",
                    "task_type": "DEPLOY_MODEL",
                    "error": "{\"bBpL_UF-SkaO6MjgEhr33w\":\"model content changed\"}"
                }
            }