milvus: [Bug]: Create collection failed reporting "dmlChanName by-dev-rootcoord-dml_2 and deltaChanName by-dev-rootcoord-delta_1 mis-match"

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20220225-adca79fa
- Deployment mode(standalone or cluster): cluster 
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus==2.0.1.dev3
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Create collection failed reporting “dmlChanName by-dev-rootcoord-dml_2 and deltaChanName by-dev-rootcoord-delta_1 mis-match”

Expected Behavior

Create collection successfully

Steps To Reproduce

Running weekly nightly:
https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline

Anything else?

1 log: milvus-distributed-57-pymilvus-e2e-logs.tar.gz

  1. failed timeline: [2022-02-27T02:25:40.279Z] [gw0] [ 0%] FAILED testcases/test_alias.py::TestAliasParamsInvalid::test_alias_create_alias_with_invalid_name[12-s] [2022-02-27T02:25:40.279Z] [gw3] [ 0%] FAILED testcases/test_alias.py::TestAliasParamsInvalid::test_alias_create_alias_with_invalid_name[\u4e2d\u6587]

  2. test error trace:

[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1081)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1082)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1083)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1084)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1085)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1086)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1087)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1088)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1089)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1090)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1091)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1092)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1093)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1094)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1095)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1096)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1097)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1098)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1099)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1100)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1101)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1102)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1103)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1104)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1105)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1106)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1107)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1108)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1109)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1110)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1111)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1112)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1113)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1114)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1115)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1116)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1117)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1118)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1119)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1120)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1121)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1122)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1123)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1124)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1125)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1126)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1127)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1128)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1129)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1130)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1131)[](https://ci.milvus.io:18080/jenkins/blue/organizations/jenkins/weekly-milvus-nightly-ci%2Fweekly-milvus-ci-test/detail/weekly-milvus-ci-test/57/pipeline#step-153-log-1132)[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - INFO - ci_test]: ################################################################################ (conftest.py:166)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - INFO - ci_test]: [initialize_milvus] Log cleaned up, start testing... (conftest.py:167)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - INFO - ci_test]: [setup_class] Start setup class... (client_base.py:45)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - INFO - ci_test]: *********************************** setup *********************************** (client_base.py:51)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - INFO - ci_test]: [setup_method] Start setup test case test_alias_create_alias_with_invalid_name. (client_base.py:52)

[2022-02-27T03:38:44.121Z] ------------------------------ Captured log call -------------------------------

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default'], kwargs: {'host': 'md-weekly-57-n-milvus.milvus-ci', 'port': '19530'} (api_request.py:55)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:55)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - DEBUG - ci_test]: (api_response) : True  (api_request.py:27)

[2022-02-27T03:38:44.121Z] [2022-02-27 02:25:35 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['collection_9LOQlelU', {

[2022-02-27T03:38:44.121Z]   auto_id: False

[2022-02-27T03:38:44.121Z]   description: 

[2022-02-27T03:38:44.121Z]   fields: [{

[2022-02-27T03:38:44.121Z]     name: int64

[2022-02-27T03:38:44.121Z]     description: 

[2022-02-27T03:38:44.121Z]     type: 5

[2022-02-27T03:38:44.121Z]     is_primary: True

[2022-02-27T03:38:44.121Z]     auto_id: False

[2022-02-27T03:38:44.121Z]   }, {

[2022-02-27T03:38:44.121Z]     name: float

[2022-02-27T03:38:44.121Z]     description: 

[2022-02-27T03:38:44.121Z]     type: 10

[2022-02-27T03:38:44.121Z]   }, {

[2022-02-27T03:38:44.121Z]     name: float_vector

[2022-02-27T03:38:44.121Z]     description: 

[2022-02-27T03:38:44.121Z]     type: 101

[2022-02-27T03:38:44.122Z]     params: {'dim': 128}

[2022-02-27T03:38:44.122Z]  ......, kwargs: {'consistency_level': 'Strong'} (api_request.py:55)

[2022-02-27T03:38:44.122Z] [2022-02-27 02:25:35 - ERROR - pymilvus.client.grpc_handler]: error_code: UnexpectedError

[2022-02-27T03:38:44.122Z] reason: "CreateCollection failed: dmlChanName by-dev-rootcoord-dml_1 and deltaChanName by-dev-rootcoord-delta_2 mis-match"

[2022-02-27T03:38:44.122Z]  (grpc_handler.py:131)

[2022-02-27T03:38:44.122Z] [2022-02-27 02:25:35 - ERROR - pymilvus.decorators]: RPC error: [create_collection], <BaseException: (code=1, message=CreateCollection failed: dmlChanName by-dev-rootcoord-dml_1 and deltaChanName by-dev-rootcoord-delta_2 mis-match)>, <Time:{'RPC start': '2022-02-27 02:25:35.728042', 'RPC error': '2022-02-27 02:25:35.735496'}> (decorators.py:73)

[2022-02-27T03:38:44.122Z] [2022-02-27 02:25:35 - ERROR - ci_test]: Traceback (most recent call last):

[2022-02-27T03:38:44.122Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 22, in inner_wrapper

[2022-02-27T03:38:44.122Z]     res = func(*args, **kwargs)

[2022-02-27T03:38:44.122Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 56, in api_request

[2022-02-27T03:38:44.122Z]     return func(*arg, **kwargs)

[2022-02-27T03:38:44.122Z]   File "/usr/local/lib/python3.6/site-packages/pymilvus/orm/collection.py", line 145, in __init__

[2022-02-27T03:38:44.122Z]     consistency_level=consistency_level)

[2022-02-27T03:38:44.122Z]   File "/usr/local/lib/python3.6/site-packages/pymilvus/decorators.py", line 56, in handler

[2022-02-27T03:38:44.122Z]     raise e

[2022-02-27T03:38:44.122Z]   File "/usr/local/lib/python3.6/site-packages/pymilvus/decorators.py", line 41, in handler

[2022-02-27T03:38:44.122Z]     return func(self, *args, **kwargs)

[2022-02-27T03:38:44.122Z]   File "/usr/local/lib/python3.6/site-packages/pymilvus/decorators.py", line 74, in handler

[2022-02-27T03:38:44.122Z]     raise e

[2022-02-27T03:38:44.122Z]   File "/usr/local/lib/python3.6/site-packages/pymilvus/decorators.py", line 70, in handler

[2022-02-27T03:38:44.122Z]     return func(*args, **kwargs)

[2022-02-27T03:38:44.122Z]   File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 132, in create_collection

[2022-02-27T03:38:44.122Z]     raise BaseException(status.error_code, status.reason)

[2022-02-27T03:38:44.122Z] pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=CreateCollection failed: dmlChanName by-dev-rootcoord-dml_1 and deltaChanName by-dev-rootcoord-delta_2 mis-match)>

[2022-02-27T03:38:44.122Z]  (api_request.py:35)

[2022-02-27T03:38:44.122Z] [2022-02-27 02:25:35 - ERROR - ci_test]: (api_response) : <BaseException: (code=1, message=CreateCollection failed: dmlChanName by-dev-rootcoord-dml_1 and deltaChanName by-dev-rootcoord-delta_2 mis-match)> (api_request.py:36)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 33 (33 by maintainers)

Most upvoted comments

Not yet, as we were told it only happened once last time 😃

Will take a look now that it has reproduced.

milvus started normally since 2022-02-27T02:23:56.