milvus: [Bug]: failed to connect **:53100

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: 2.3.0
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Upgrade from 2.2.12 to 2.3.0

Expected Behavior

No response

Steps To Reproduce

1.docker-compose down
2.update docker-compose.yml 
3.repull and update milvus.yml& from offcial website
4.docker-compose up -d

Milvus Log

milvus.log

Anything else?

milvus.txt docker-compose.txt

github don’t allow upload *.yml, if u want to test can use those and replace suffix

About this issue

Original URL
State: closed
Created 10 months ago
Comments: 45 (24 by maintainers)

Commits related to this issue

fix panic due to empty traceID(#26754) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> — committed to MrPresent-Han/milvus by MrPresent-Han 10 months ago
fix panic due to empty traceID(#26754) (#26808) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> — committed to milvus-io/milvus by MrPresent-Han 10 months ago
fix panic due to empty traceID(#26754) (#26808) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> — committed to MrPresent-Han/milvus by MrPresent-Han 10 months ago

Most upvoted comments

@gaoyuan5251 , I tested on my laptop and changed the trace.exporter to “”, and then the problem you issued appeared right away and the container crushed. I suspect it is still sth to do with the milvus config file.

Anyway, you helped us find a very delicate point of milvus, it’s very helpful.

MrPresent-Han on Aug 31, 2023

https://github.com/milvus-io/milvus/issues/26717 there is another issue with same problems, it seems that milvus 2.3.x really has this issue in some cases and currently, colleagues in devops haven’t reproduce the problem and locate the root cause

MrPresent-Han on Sep 25, 2023

8th in September

MrPresent-Han on Sep 8, 2023

And also, the pr regarding this flaw has been merged, and I have testified your case on my macOS. It should work out now.

MrPresent-Han on Sep 2, 2023

bro, could it have something to do with etcd? @zhagnlu

It seems no relation with etcd, bro @MrPresent-Han is focus on finding reason

but i found etcd was unhealthy status @zhagnlu

milvus-etcd quay.io/coreos/etcd:v3.5.5 “etcd -advertise-cli…” etcd 3 hours ago Up 2 hours (unhealthy) 2379-2380/tcp

docker-compose logs etcd | grep “error” milvus-etcd | {“level”:“warn”,“ts”:1693474706.6439664,“caller”:“fileutil/fileutil.go:57”,“msg”:“check file permission”,“error”:“directory “/etcd” exist, but the permission is “drwxr-xr-x”. The recommended permission is “-rwx------” to prevent possible unprivileged access to the data”}

I think this info simply imply that the directory for etcd has so high privilege, the etcd directory on my laptop is also “drwxr-xr-x”, but it runs as expected

MrPresent-Han on Aug 31, 2023