milvus: [Bug]: Querycoord panic after restarting docker with error `set empty delta channel info to meta of collection 435211660870549505`
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: master-20220811-6c3dbf0
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus==2.2.0.dev6
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
The search failed after restarting docker
RPC error: [search], <MilvusException: (code=1, message=checkIfLoaded failed when search, collection:sift_128_euclidean, partitions:[], err = GetCollectionInfo failed, collection = sift_128_euclidean, err = err: find no available querycoord, check querycoord state
Search...
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:259 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:160 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).ShowCollections
/go/src/github.com/milvus-io/milvus/internal/proxy/meta_cache.go:210 github.com/milvus-io/milvus/internal/proxy.(*MetaCache).GetCollectionInfo
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:439 github.com/milvus-io/milvus/internal/proxy.checkIfLoaded
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:[20](https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true#step:15:21)1 github.com/milvus-io/milvus/internal/proxy.(*searchTask).PreExecute
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:452 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask
/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit
)>, <Time:{'RPC start': '2022-08-11 07:04:24.678108', 'RPC error': '2022-08-11 07:04:29.858532'}>
Traceback (most recent call last):
File "scripts/second_recall_test.py", line 64, in <module>
search_test(host)
File "scripts/second_recall_test.py", line 33, in search_test
res = collection.search(
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 717, in search
res = conn.search(self._name, data, anns_field, param, limit, expr,
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 451, in search
return self._execute_search_requests(requests, timeout, **_kwargs)
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 415, in _execute_search_requests
raise pre_err
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 406, in _execute_search_requests
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=checkIfLoaded failed when search, collection:sift_128_euclidean, partitions:[], err = GetCollectionInfo failed, collection = sift_128_euclidean, err = err: find no available querycoord, check querycoord state
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:259 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:160 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).ShowCollections
/go/src/github.com/milvus-io/milvus/internal/proxy/meta_cache.go:[21](https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true#step:15:22)0 github.com/milvus-io/milvus/internal/proxy.(*MetaCache).GetCollectionInfo
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:439 github.com/milvus-io/milvus/internal/proxy.checkIfLoaded
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:201 github.com/milvus-io/milvus/internal/proxy.(*searchTask).PreExecute
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:452 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask
/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit
)>
Expected Behavior
all test cases passed
Steps To Reproduce
see https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true
Milvus Log
failed job: https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true log: https://github.com/zhuwenxing/milvus/suites/7765344016/artifacts/326509308
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (14 by maintainers)
@zhuwenxing try to reproduce on 2.1.4->master