milvus: [Bug]: client search timeout

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: 2.0.2
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.0.2
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 32Cores/192GB
- GPU: 
- Others:

Current Behavior

python sdk search timeout

Expected Behavior

search service success

Steps To Reproduce

when the cluster has run about 2 weeks, this issue occured

Milvus Log

rootcoord:

[2022/05/30 20:30:00.021 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 42"]
[2022/05/30 20:30:00.023 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 43"]
[2022/05/30 20:30:00.047 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 45"]
[2022/05/30 20:30:00.183 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 44"]
[2022/05/30 20:30:00.221 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 42"]
[2022/05/30 20:30:00.223 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 43"]
[2022/05/30 20:30:00.246 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 45"]
[2022/05/30 20:30:00.384 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 44"]
[2022/05/30 20:30:00.420 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 42"]
[2022/05/30 20:30:00.423 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 43"]
[2022/05/30 20:30:00.446 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 45"]
[2022/05/30 20:30:00.585 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 44"]
[2022/05/30 20:30:00.620 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 42"]
[2022/05/30 20:30:00.623 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 43"]
[2022/05/30 20:30:00.646 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 45"]
[2022/05/30 20:30:00.783 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 44"]

proxy:

[2022/05/30 20:30:00.095 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:02.290 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:04.201 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:05.095 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:07.289 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:09.201 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:10.095 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:12.290 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:14.201 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:15.095 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:17.290 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:19.202 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:20.095 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:22.290 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:24.202 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:25.095 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:27.290 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:29.202 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/30 20:30:30.096 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]v

Anything else?

when i restart rootcoord, the rootcoord log:

[2022/05/30 22:29:12.972 +08:00] [INFO] [tso.go:121] ["sync and save timestamp"] [last=2022/05/30 22:28:57.895 +08:00] [save=2022/05/30 22:29:15.971 +08:00] [next=2022/05/30 22:29:12.971 +08:00]
[2022/05/30 22:29:12.974 +08:00] [INFO] [tso.go:121] ["sync and save timestamp"] [last=2022/05/30 22:28:56.696 +08:00] [save=2022/05/30 22:29:15.973 +08:00] [next=2022/05/30 22:29:12.973 +08:00]
[2022/05/30 22:29:13.133 +08:00] [WARN] [tso.go:177] ["clock offset is huge, check network latency and clock skew"] [jet-lag=159.479675ms] [prev-physical=2022/05/30 22:29:12.973 +08:00] [now=2022/05/30 22:29:13.133 +08:00]
[2022/05/30 22:29:13.183 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 44"]
[2022/05/30 22:29:13.221 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 42"]
[2022/05/30 22:29:13.223 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 43"]
[2022/05/30 22:29:13.246 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 45"]
[2022/05/30 22:29:13.383 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 44"]
[2022/05/30 22:29:13.421 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 42"]
[2022/05/30 22:29:13.424 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 43"]
[2022/05/30 22:29:13.447 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 45"]

when i restart proxy, the proxy log:

[2022/05/30 23:19:03.872 +08:00] [INFO] [grpclog.go:37] ["[core]Channel Connectivity change to READY"]
[2022/05/30 23:19:04.025 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433557461872672769 segmentIDs:433556518206439426 segmentIDs:433557461872672774 segmentIDs:433556518206439425 segmentIDs:433558405669453826 segmentIDs:433557461872672773 segmentIDs:433557461872672770 segmentIDs:433558405669453825 segmentIDs:433559349151662083 segmentIDs:433557461872672772 segmentIDs:433558405669453827 segmentIDs:433560292844109829 segmentIDs:433556518206439430 segmentIDs:433556518206439429 segmentIDs:433558405682561025 segmentIDs:433559349151662082 segmentIDs:433556518206439428 segmentIDs:433557461872672771 segmentIDs:433558405669453828 segmentIDs:433558405669453829 segmentIDs:433559349125709825 segmentIDs:433556518206439427 segmentIDs:433560292844109830 segmentIDs:433560292844109828 segmentIDs:433565955286630401 segmentIDs:433565955286630402 segmentIDs:433565955299475457 segmentIDs:433565955299475458 segmentIDs:433565955299475459 segmentIDs:433565955299475460 "]
[2022/05/30 23:19:04.025 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.070 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433559348955054084 segmentIDs:433558405577441281 segmentIDs:433556518140641285 segmentIDs:433558405603655683 segmentIDs:433557461806874625 segmentIDs:433559348945354753 segmentIDs:433558405590548481 segmentIDs:433559348955054083 segmentIDs:433557461819981826 segmentIDs:433557461819981825 segmentIDs:433557461819981828 segmentIDs:433556518140641284 segmentIDs:433558405603655682 segmentIDs:433559348955054081 segmentIDs:433560292765466625 segmentIDs:433560292778573825 segmentIDs:433556518140641283 segmentIDs:433557461793505281 segmentIDs:433560292765466626 segmentIDs:433560292752359426 segmentIDs:433558405590548482 segmentIDs:433556518140641282 segmentIDs:433558405603655681 segmentIDs:433556518127796225 segmentIDs:433557461819981827 segmentIDs:433556518140641281 segmentIDs:433565955090022401 segmentIDs:433565955090022402 segmentIDs:433565955103129601 segmentIDs:433565955103129602 segmentIDs:433565955103129603 segmentIDs:433565955103129604 "]
[2022/05/30 23:19:04.070 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.228 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433557461872672769 segmentIDs:433556518206439426 segmentIDs:433557461872672774 segmentIDs:433556518206439425 segmentIDs:433558405669453826 segmentIDs:433557461872672773 segmentIDs:433557461872672770 segmentIDs:433558405669453825 segmentIDs:433559349151662083 segmentIDs:433557461872672772 segmentIDs:433558405669453827 segmentIDs:433560292844109829 segmentIDs:433556518206439430 segmentIDs:433556518206439429 segmentIDs:433558405682561025 segmentIDs:433559349151662082 segmentIDs:433556518206439428 segmentIDs:433557461872672771 segmentIDs:433558405669453828 segmentIDs:433558405669453829 segmentIDs:433559349125709825 segmentIDs:433556518206439427 segmentIDs:433560292844109830 segmentIDs:433560292844109828 segmentIDs:433565955286630401 segmentIDs:433565955286630402 segmentIDs:433565955299475457 segmentIDs:433565955299475458 segmentIDs:433565955299475459 segmentIDs:433565955299475460 "]
[2022/05/30 23:19:04.229 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.273 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433559348955054084 segmentIDs:433558405577441281 segmentIDs:433556518140641285 segmentIDs:433558405603655683 segmentIDs:433557461806874625 segmentIDs:433559348945354753 segmentIDs:433558405590548481 segmentIDs:433559348955054083 segmentIDs:433557461819981826 segmentIDs:433557461819981825 segmentIDs:433557461819981828 segmentIDs:433556518140641284 segmentIDs:433558405603655682 segmentIDs:433559348955054081 segmentIDs:433560292765466625 segmentIDs:433560292778573825 segmentIDs:433556518140641283 segmentIDs:433557461793505281 segmentIDs:433560292765466626 segmentIDs:433560292752359426 segmentIDs:433558405590548482 segmentIDs:433556518140641282 segmentIDs:433558405603655681 segmentIDs:433556518127796225 segmentIDs:433557461819981827 segmentIDs:433556518140641281 segmentIDs:433565955090022401 segmentIDs:433565955090022402 segmentIDs:433565955103129601 segmentIDs:433565955103129602 segmentIDs:433565955103129603 segmentIDs:433565955103129604 "]
[2022/05/30 23:19:04.274 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.432 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433557461872672769 segmentIDs:433556518206439426 segmentIDs:433557461872672774 segmentIDs:433556518206439425 segmentIDs:433558405669453826 segmentIDs:433557461872672773 segmentIDs:433557461872672770 segmentIDs:433558405669453825 segmentIDs:433559349151662083 segmentIDs:433557461872672772 segmentIDs:433558405669453827 segmentIDs:433560292844109829 segmentIDs:433556518206439430 segmentIDs:433556518206439429 segmentIDs:433558405682561025 segmentIDs:433559349151662082 segmentIDs:433556518206439428 segmentIDs:433557461872672771 segmentIDs:433558405669453828 segmentIDs:433558405669453829 segmentIDs:433559349125709825 segmentIDs:433556518206439427 segmentIDs:433560292844109830 segmentIDs:433560292844109828 segmentIDs:433565955286630401 segmentIDs:433565955286630402 segmentIDs:433565955299475457 segmentIDs:433565955299475458 segmentIDs:433565955299475459 segmentIDs:433565955299475460 "]
[2022/05/30 23:19:04.432 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.478 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433559348955054084 segmentIDs:433558405577441281 segmentIDs:433556518140641285 segmentIDs:433558405603655683 segmentIDs:433557461806874625 segmentIDs:433559348945354753 segmentIDs:433558405590548481 segmentIDs:433559348955054083 segmentIDs:433557461819981826 segmentIDs:433557461819981825 segmentIDs:433557461819981828 segmentIDs:433556518140641284 segmentIDs:433558405603655682 segmentIDs:433559348955054081 segmentIDs:433560292765466625 segmentIDs:433560292778573825 segmentIDs:433556518140641283 segmentIDs:433557461793505281 segmentIDs:433560292765466626 segmentIDs:433560292752359426 segmentIDs:433558405590548482 segmentIDs:433556518140641282 segmentIDs:433558405603655681 segmentIDs:433556518127796225 segmentIDs:433557461819981827 segmentIDs:433556518140641281 segmentIDs:433565955090022401 segmentIDs:433565955090022402 segmentIDs:433565955103129601 segmentIDs:433565955103129602 segmentIDs:433565955103129603 segmentIDs:433565955103129604 "]
[2022/05/30 23:19:04.478 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.636 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433557461872672769 segmentIDs:433556518206439426 segmentIDs:433557461872672774 segmentIDs:433556518206439425 segmentIDs:433558405669453826 segmentIDs:433557461872672773 segmentIDs:433557461872672770 segmentIDs:433558405669453825 segmentIDs:433559349151662083 segmentIDs:433557461872672772 segmentIDs:433558405669453827 segmentIDs:433560292844109829 segmentIDs:433556518206439430 segmentIDs:433556518206439429 segmentIDs:433558405682561025 segmentIDs:433559349151662082 segmentIDs:433556518206439428 segmentIDs:433557461872672771 segmentIDs:433558405669453828 segmentIDs:433558405669453829 segmentIDs:433559349125709825 segmentIDs:433556518206439427 segmentIDs:433560292844109830 segmentIDs:433560292844109828 segmentIDs:433565955286630401 segmentIDs:433565955286630402 segmentIDs:433565955299475457 segmentIDs:433565955299475458 segmentIDs:433565955299475459 segmentIDs:433565955299475460 "]
[2022/05/30 23:19:04.637 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.682 +08:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:433559348955054084 segmentIDs:433558405577441281 segmentIDs:433556518140641285 segmentIDs:433558405603655683 segmentIDs:433557461806874625 segmentIDs:433559348945354753 segmentIDs:433558405590548481 segmentIDs:433559348955054083 segmentIDs:433557461819981826 segmentIDs:433557461819981825 segmentIDs:433557461819981828 segmentIDs:433556518140641284 segmentIDs:433558405603655682 segmentIDs:433559348955054081 segmentIDs:433560292765466625 segmentIDs:433560292778573825 segmentIDs:433556518140641283 segmentIDs:433557461793505281 segmentIDs:433560292765466626 segmentIDs:433560292752359426 segmentIDs:433558405590548482 segmentIDs:433556518140641282 segmentIDs:433558405603655681 segmentIDs:433556518127796225 segmentIDs:433557461819981827 segmentIDs:433556518140641281 segmentIDs:433565955090022401 segmentIDs:433565955090022402 segmentIDs:433565955103129601 segmentIDs:433565955103129602 segmentIDs:433565955103129603 segmentIDs:433565955103129604 "]
[2022/05/30 23:19:04.683 +08:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]
[2022/05/30 23:19:04.769 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]

ps: after i format the cluster(clear all data in minio/etcd/pulsar), the cluster work, but now the rootcoord log still has a lot of “failed to updateTimeTick” msg:

[2022/05/31 17:32:43.420 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 42"]
[2022/05/31 17:32:43.424 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 43"]
[2022/05/31 17:32:43.446 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 45"]
[2022/05/31 17:32:43.583 +08:00] [WARN] [root_coord.go:1865] ["failed to updateTimeTick"] [role=rootcoord] [msgID=0] [error="skip ChannelTimeTickMsg from un-recognized session 44"]

and the proxy log has a lot of “connetion error” msg:

[2022/05/31 17:36:17.371 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: read tcp x.x.x.x:19530->x.x.x.x:61768: read: connection reset by peer\""]
[2022/05/31 17:36:17.934 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/31 17:36:20.927 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/31 17:36:20.927 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/31 17:36:22.371 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/31 17:36:22.934 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]
[2022/05/31 17:36:25.928 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams failed to receive the preface from client: EOF\""]

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 25 (25 by maintainers)

Most upvoted comments

@BUPTAnderson sorry, but can you try to enable debug log and then re-test this again? The search request entered in proxy also entered querynode, but not sure whether it times out in proxy or querynode.

Taking msg id 433605860175276463 as an example:

proxy-18.log:125762:[2022/06/02 11:39:09.383 +08:00] [ERROR] [task_scheduler.go:501] ["Failed to post-execute task: searchTask:wait to finish failed, timeout: 433605860175276463"] [traceID=7e32b25a92135a34] [stack="github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n\t/opt/milvus/internal/proxy/task_scheduler.go:501"]
proxy-18.log:125763:[2022/06/02 11:39:09.383 +08:00] [WARN] [impl.go:2471] ["Search failed to WaitToFinish"] [error="proxy TaskCondition context Done"] [traceID=7e32b25a92135a34] [role=proxy] [msgID=433605860175276463] [db=] [collection=verificationbaseimage] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=2060] [OutputFields="[ID,User]"] [search_params="[{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"Vector\"},{\"key\":\"topk\",\"value\":\"10\"},{\"key\":\"params\",\"value\":\"{\\\"nprobe\\\":10}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [travel_timestamp=0] [guarantee_timestamp=0]
querynode-20.log:11105:[2022/06/02 11:39:09.417 +08:00] [WARN] [query_collection.go:644] ["do query failed in doUnsolvedQueryMsg because timeout, collectionID =  433568644207214593 , msgID =  433605860175276463\n"]

But I guess it’s that rootcoord didn’t forward timetick message caused that querynode didn’t update the service time, so the search request can’t be executed. Based on this assumption, you can also try eventually consistency.

yanliang567 on Jun 6, 2022