ml-commons: [BUG] Tag Mismatch error on VisualizationsToolIT.testVisualizationFound Windows Test

What is the bug? https://github.com/opensearch-project/ml-commons/blob/85d0c9e2b8807162de9afe7c915801b75e486064/plugin/src/test/java/org/opensearch/ml/tools/VisualizationsToolIT.java#L59-L66 There is a retry enabled on the VisualizationsToolIT.testVisualizationFound test but it seems that retry has a bit of a flaw if the underlying problem is different I am seeing that the problem here is a encryption issue. This might be the source of all of our flaky tests

VisualizationsToolIT > testVisualizationFound FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:54529/], URI [/_plugins/_ml/agents/bLjaRJQB515KRnslfdUv/_execute], status line [HTTP/1.1 500 Internal Server Error]
    {"status":500,"error":{"type":"AEADBadTagException","reason":"System Error","details":"Tag mismatch"}}
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
        at app//org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound(VisualizationsToolIT.java:74)

    java.lang.AssertionError: The response failed to meet condition after 5 attempts. Attempted to perform GET : /_plugins/_ml/models/arjaRJQB515KRnsleNWv
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.waitResponseMeetingCondition(ToolIntegrationWithLLMTest.java:103)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.checkForModelUndeployedStatus(ToolIntegrationWithLLMTest.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:74)
        at

... 

2> REPRODUCE WITH: gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound" -Dtests.seed=AD7A0603B7C68274 -Dtests.security.manager=false -Dtests.locale=fr-GN -Dtests.timezone=America/Argentina/Buenos_Aires -Druntime.java=21
  2> org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:54529/], URI [/_plugins/_ml/agents/bLjaRJQB515KRnslfdUv/_execute], status line [HTTP/1.1 500 Internal Server Error]
    {"status":500,"error":{"type":"AEADBadTagException","reason":"System Error","details":"Tag mismatch"}}
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
        at app//org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound(VisualizationsToolIT.java:74)

    java.lang.AssertionError: The response failed to meet condition after 5 attempts. Attempted to perform GET : /_plugins/_ml/models/arjaRJQB515KRnsleNWv
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.waitResponseMeetingCondition(ToolIntegrationWithLLMTest.java:103)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.checkForModelUndeployedStatus(ToolIntegrationWithLLMTest.java:89)
        at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:74)
        at

How can one reproduce the bug? This was discovered in a build failure.

What is the expected behavior? This test should pass or timeout but not have this encryption issue.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (13 by maintainers)

Most upvoted comments

Thanks for the update! Would you be willing to take that up? (adding logs)

sure, i will do two things. First add some logs to log response body, second continuing try in my local to see whether i can reproduce the error.

Adding the log here in txt format so it doesn’t expire

6_Build and Test MLCommons Plugin on linux (21).txt

Here is another example of another build failure. Linking the txt file here as well to make sure it does not expire.

7_Build and Test MLCommons Plugin on linux (21).txt

Hey @Hailong-am do you mind taking a look? Thanks

seeing VisualizationIT failing again but with different error. https://github.com/opensearch-project/ml-commons/pull/3353