ml-commons: [BUG] Tag Mismatch error on VisualizationsToolIT.testVisualizationFound Windows Test
What is the bug? https://github.com/opensearch-project/ml-commons/blob/85d0c9e2b8807162de9afe7c915801b75e486064/plugin/src/test/java/org/opensearch/ml/tools/VisualizationsToolIT.java#L59-L66 There is a retry enabled on the VisualizationsToolIT.testVisualizationFound test but it seems that retry has a bit of a flaw if the underlying problem is different I am seeing that the problem here is a encryption issue. This might be the source of all of our flaky tests
VisualizationsToolIT > testVisualizationFound FAILED
org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:54529/], URI [/_plugins/_ml/agents/bLjaRJQB515KRnslfdUv/_execute], status line [HTTP/1.1 500 Internal Server Error]
{"status":500,"error":{"type":"AEADBadTagException","reason":"System Error","details":"Tag mismatch"}}
at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
at app//org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound(VisualizationsToolIT.java:74)
java.lang.AssertionError: The response failed to meet condition after 5 attempts. Attempted to perform GET : /_plugins/_ml/models/arjaRJQB515KRnsleNWv
at org.junit.Assert.fail(Assert.java:89)
at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.waitResponseMeetingCondition(ToolIntegrationWithLLMTest.java:103)
at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.checkForModelUndeployedStatus(ToolIntegrationWithLLMTest.java:89)
at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:74)
at
...
2> REPRODUCE WITH: gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound" -Dtests.seed=AD7A0603B7C68274 -Dtests.security.manager=false -Dtests.locale=fr-GN -Dtests.timezone=America/Argentina/Buenos_Aires -Druntime.java=21
2> org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:54529/], URI [/_plugins/_ml/agents/bLjaRJQB515KRnslfdUv/_execute], status line [HTTP/1.1 500 Internal Server Error]
{"status":500,"error":{"type":"AEADBadTagException","reason":"System Error","details":"Tag mismatch"}}
at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
at app//org.opensearch.ml.tools.VisualizationsToolIT.testVisualizationFound(VisualizationsToolIT.java:74)
java.lang.AssertionError: The response failed to meet condition after 5 attempts. Attempted to perform GET : /_plugins/_ml/models/arjaRJQB515KRnsleNWv
at org.junit.Assert.fail(Assert.java:89)
at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.waitResponseMeetingCondition(ToolIntegrationWithLLMTest.java:103)
at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.checkForModelUndeployedStatus(ToolIntegrationWithLLMTest.java:89)
at org.opensearch.ml.tools.ToolIntegrationWithLLMTest.deleteModel(ToolIntegrationWithLLMTest.java:74)
at
How can one reproduce the bug? This was discovered in a build failure.
What is the expected behavior? This test should pass or timeout but not have this encryption issue.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (13 by maintainers)
sure, i will do two things. First add some logs to log response body, second continuing try in my local to see whether i can reproduce the error.
Adding the log here in txt format so it doesn’t expire
6_Build and Test MLCommons Plugin on linux (21).txt
Here is another example of another build failure. Linking the txt file here as well to make sure it does not expire.
7_Build and Test MLCommons Plugin on linux (21).txt
Hey @Hailong-am do you mind taking a look? Thanks
seeing VisualizationIT failing again but with different error. https://github.com/opensearch-project/ml-commons/pull/3353