milvus: Milvus search not working after restarting docker

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Running the reverse image search demo, the images are added to the collection and can be searched.

When docker is restarted, the collection exists, I manually check through milvus_cli. But, the collection cannot be searched.

It hangs at the milvus_client.search_vectors method call:

https://github.com/milvus-io/bootcamp/blob/master/solutions/reverse_image_search/quick_deploy/server/src/operations/search.py#L13 def do_search(table_name, img_path, model, milvus_client, mysql_cli): try: if not table_name: table_name = DEFAULT_TABLE feat = model.resnet50_extract_feat(img_path) vectors = milvus_client.search_vectors(table_name, [feat], TOP_K) vids = [str(x.id) for x in vectors[0]] paths = mysql_cli.search_by_milvus_ids(vids, table_name) distances = [x.distance for x in vectors[0]] return paths, distances except Exception as e: LOGGER.error(" Error with search : {}".format(e)) sys.exit(1)

when loading the collection via milvus_cli the load progress always remains at 0.

As per the docker compose file, the volumes are mounted and contain data generated by milvus.

Expected Behavior

Vector search should function after restarting docker

Steps To Reproduce

No response

Environment

- Milvus version: Milvusdb/milvus v2.0.0-rc6-20210910-020f109
minio/minio RELEASE.2020-12-03T00-03-10Z
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus==2.0.0rc6, milvus-cli==0.1.6
- OS(Ubuntu or CentOS): Windows 10 20H2 19042.1237
- CPU/Memory: 32GB
- GPU: Nvidia MX250
- Others:

Anything else?

No response

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 44 (16 by maintainers)

Most upvoted comments

/reopen I also have the same issue in the standalone 2.0.0-rc8 version. After I restart the containers, I can insert new records but when I make a query it not responding.

I also have tried the latest version (2.0.0) and after restart a couple of times, I have tried to make any request and got “proxy not healthy” exception

Hi Beni, Milvus 2.0.0 should be much better on stability so we recommend to use the later release. If you still hit hang please open an issue with detailed log and the team will work on the issues.

xiaofan-luan on Feb 2, 2022

My settings.

- Milvus version: Milvusdb/milvus v2.0.2
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus==2.0.1
- OS(Ubuntu or CentOS): Ubuntu 18.04
- CPU/Memory: 4Core / 32GB
- GPU: 
- Others:
   - Install guide: https://milvus.io/docs/v2.0.x/install_standalone-docker.md
   - Test guide: https://github.com/milvus-io/pymilvus/blob/master/examples/hello_milvus.py

Hi, all. I had a similar problem where the search is hanging. And as some people mentioned, I could reproduce this when I unstably stop the docker by using docker-compose down. If there is some broken in the meantime, the search is hanging although I restarted the containers well.

$ sudo docker-compose down
Removing milvus-etcd  ... done
Removing milvus-minio ... done
Removing network milvus
ERROR: error while removing network: network milvus id 8b6f2de0a4c3542fbfc9e3ba9a4129d28cfc14b83b308ede74701a53c7380c9f has active endpoints

So I tried to release the index and load it again. Then the search worked well. I don’t know why it happens, but this is my solution.

kingmbc on May 9, 2022