milvus: [Bug]: Failed to drop collection when running cases in multiple threads

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20211030-54738d2
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): rc8dev22

Current Behavior

Fail to drop collection, when running tests in multiple threads

[2021-11-01 10:18:26 - DEBUG - ci_test]: (api_request)  : [Collection.drop_partition] args: ['search_partition_0'], kwargs: {'timeout': 20} (api_request.py:44)
[2021-11-01 10:18:47 - ERROR - pymilvus.client.grpc_handler]: 
Addr [10.98.0.7:19530] drop_partition
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1635761927.003051909","description":"Error received from peer ipv4:10.98.0.7:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
	{'API start': '2021-11-01 10:18:26.998639', 'RPC start': '2021-11-01 10:18:27.002539', 'RPC error': '2021-11-01 10:18:47.003992'} (grpc_handler.py:74)
[2021-11-01 10:19:07 - ERROR - pymilvus.client.grpc_handler]: Error: <BaseException: (code=1, message=DropPartition failed: partition search_partition_0 does not exist)> (grpc_handler.py:59)
[2021-11-01 10:19:07 - ERROR - ci_test]: Traceback (most recent call last):
  File "/milvus/tests/python_client/utils/api_request.py", line 18, in inner_wrapper
    res = func(*args, **kwargs)
  File "/milvus/tests/python_client/utils/api_request.py", line 45, in api_request
    return func(*arg, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/orm/collection.py", line 904, in drop_partition
    return conn.drop_partition(self._name, partition_name, timeout=timeout, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 61, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 45, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 459, in drop_partition
    return handler.drop_partition(collection_name, partition_name, timeout)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 65, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 57, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 417, in drop_partition
    raise BaseException(response.error_code, response.reason)
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=DropPartition failed: partition search_partition_0 does not exist)>
 (api_request.py:26)
[2021-11-01 10:19:07 - ERROR - ci_test]: (api_response) : <BaseException: (code=1, message=DropPartition failed: partition search_partition_0 does not exist)> (api_request.py:27)

Expected Behavior

tests run successfully

Steps To Reproduce

run test test_search_before_after_delete with 6 threads for 10 times:
pytest --host x.x.x.x testcases/test_search_20.py::TestCollectionSearch::test_search_before_after_delete -n 6 --count 10

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (15 by maintainers)

Most upvoted comments

This issue is caused by deadlock in querynode. @bigsheeper will fix it in rc8 @yanliang567 @binbinlv