milvus: [Bug]: Querycoord panic after restarting docker with error `set empty delta channel info to meta of collection 435211660870549505`

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20220811-6c3dbf0
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus==2.2.0.dev6
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The search failed after restarting docker

RPC error: [search], <MilvusException: (code=1, message=checkIfLoaded failed when search, collection:sift_128_euclidean, partitions:[], err = GetCollectionInfo failed, collection = sift_128_euclidean, err = err: find no available querycoord, check querycoord state

Search...
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:259 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:160 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).ShowCollections
/go/src/github.com/milvus-io/milvus/internal/proxy/meta_cache.go:210 github.com/milvus-io/milvus/internal/proxy.(*MetaCache).GetCollectionInfo
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:439 github.com/milvus-io/milvus/internal/proxy.checkIfLoaded
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:[20](https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true#step:15:21)1 github.com/milvus-io/milvus/internal/proxy.(*searchTask).PreExecute
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:452 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask
/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit
)>, <Time:{'RPC start': '2022-08-11 07:04:24.678108', 'RPC error': '2022-08-11 07:04:29.858532'}>
Traceback (most recent call last):
  File "scripts/second_recall_test.py", line 64, in <module>
    search_test(host)
  File "scripts/second_recall_test.py", line 33, in search_test
    res = collection.search(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 717, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 96, in handler
    raise e
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 92, in handler
    return func(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 74, in handler
    raise e
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/decorators.py", line 48, in handler
    return func(self, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 451, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 415, in _execute_search_requests
    raise pre_err
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 406, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=checkIfLoaded failed when search, collection:sift_128_euclidean, partitions:[], err = GetCollectionInfo failed, collection = sift_128_euclidean, err = err: find no available querycoord, check querycoord state
, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:259 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:160 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).ShowCollections
/go/src/github.com/milvus-io/milvus/internal/proxy/meta_cache.go:[21](https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true#step:15:22)0 github.com/milvus-io/milvus/internal/proxy.(*MetaCache).GetCollectionInfo
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:439 github.com/milvus-io/milvus/internal/proxy.checkIfLoaded
/go/src/github.com/milvus-io/milvus/internal/proxy/task_search.go:201 github.com/milvus-io/milvus/internal/proxy.(*searchTask).PreExecute
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:452 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask
/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit
)>

Expected Behavior

all test cases passed

Steps To Reproduce

see https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true

Milvus Log

failed job: https://github.com/zhuwenxing/milvus/runs/7781667318?check_suite_focus=true log: https://github.com/zhuwenxing/milvus/suites/7765344016/artifacts/326509308

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (14 by maintainers)

Most upvoted comments

@zhuwenxing try to reproduce on 2.1.4->master