milvus: [Bug]: Load failed reporting " Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: latest
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2): latest
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
Load failed:
Logs: [standalone: artifacts-milvus-standalone-master-220-pymilvus-e2e-logs.tar.gz]
[please download from through the upper log name]
https://ci.milvus.io:18080/jenkins/job/milvus-nightly-ci/job/master/220/
Other clues:
Failed case 1:
-
Failed case: FAILED testcases/test_search_20.py::TestCollectionSearch::test_search_after_different_index_with_params[8-False-True-HNSW-params4]
-
Collection name: search_collection_w4WHSbge
-
Timeline:
[2021-11-16 15:29:18,685 - DEBUG - ci_test]: (api_request) : [Collection.load] args: [None], kwargs: {'timeout': 20} (api_request.py:44)
[2021-11-16 15:30:36,929 - ERROR - ci_test]: Traceback (most recent call last):
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 18, in inner_wrapper
res = func(*args, **kwargs)
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 45, in api_request
return func(*arg, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/orm/collection.py", line 449, in load
conn.load_collection(self._name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 58, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 322, in load_collection
return handler.load_collection("", collection_name=collection_name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 823, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 841, in wait_for_loading_collection
return self._wait_for_loading_collection_v2(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 868, in _wait_for_loading_collection_v2
raise BaseException(response.status.error_code, response.status.reason)
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc error: code = Unknown desc = collection has not been loaded to memory or load failed)>
Failed case 2:
- Failed case
[2021-11-16T15:33:08.981Z] [gw0] [ 41%] FAILED testcases/test_search_20.py::TestCollectionSearch::test_search_after_index_different_metric_type[128-True-True-IVF_FLAT-params1]
-
collection name: search_collection_sp6bGVTh
-
Timeline:
[2021-11-16 15:32:59,995 - ERROR - ci_test]: Traceback (most recent call last):
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 18, in inner_wrapper
res = func(*args, **kwargs)
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 45, in api_request
return func(*arg, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/orm/collection.py", line 449, in load
conn.load_collection(self._name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 58, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 322, in load_collection
return handler.load_collection("", collection_name=collection_name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 823, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 841, in wait_for_loading_collection
return self._wait_for_loading_collection_v2(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 868, in _wait_for_loading_collection_v2
raise BaseException(response.status.error_code, response.status.reason)
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc error: code = Unknown desc = collection has not been loaded to memory or load failed)>
Expected Behavior
Load success
Steps To Reproduce
No response
Anything else?
No response
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: latest
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2): latest
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
Load failed:
Logs: [standalone: artifacts-milvus-standalone-master-220-pymilvus-e2e-logs.tar.gz] [please download from through the upper log name] https://ci.milvus.io:18080/jenkins/job/milvus-nightly-ci/job/master/220/
Other clues:
Failed case 1:
-
Failed case: FAILED testcases/test_search_20.py::TestCollectionSearch::test_search_after_different_index_with_params[8-False-True-HNSW-params4]
-
Collection name: search_collection_w4WHSbge
-
Timeline:
[2021-11-16 15:29:18,685 - DEBUG - ci_test]: (api_request) : [Collection.load] args: [None], kwargs: {'timeout': 20} (api_request.py:44)
[2021-11-16 15:30:36,929 - ERROR - ci_test]: Traceback (most recent call last):
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 18, in inner_wrapper
res = func(*args, **kwargs)
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 45, in api_request
return func(*arg, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/orm/collection.py", line 449, in load
conn.load_collection(self._name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 58, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 322, in load_collection
return handler.load_collection("", collection_name=collection_name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 823, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 841, in wait_for_loading_collection
return self._wait_for_loading_collection_v2(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 868, in _wait_for_loading_collection_v2
raise BaseException(response.status.error_code, response.status.reason)
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc error: code = Unknown desc = collection has not been loaded to memory or load failed)>
Failed case 2:
- Failed case
[2021-11-16T15:33:08.981Z] [gw0] [ 41%] FAILED testcases/test_search_20.py::TestCollectionSearch::test_search_after_index_different_metric_type[128-True-True-IVF_FLAT-params1]
-
collection name: search_collection_sp6bGVTh
-
Timeline:
[2021-11-16 15:32:59,995 - ERROR - ci_test]: Traceback (most recent call last):
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 18, in inner_wrapper
res = func(*args, **kwargs)
File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 45, in api_request
return func(*arg, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/orm/collection.py", line 449, in load
conn.load_collection(self._name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 58, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 322, in load_collection
return handler.load_collection("", collection_name=collection_name, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 823, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
raise e
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 841, in wait_for_loading_collection
return self._wait_for_loading_collection_v2(collection_name, timeout)
File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 868, in _wait_for_loading_collection_v2
raise BaseException(response.status.error_code, response.status.reason)
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc error: code = Unknown desc = collection has not been loaded to memory or load failed)>
Expected Behavior
Load success
Steps To Reproduce
No response
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 21 (18 by maintainers)
@DragonDriver any update?