milvus: [Bug]: Proxy panic: runtime error: invalid memory address or nil pointer dereference

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.2.0
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): 2.2.0
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz / 256GB
- GPU: None
- Others:

Current Behavior

When I insert records with 300 threads simultaneously, the proxy server crashed and I see the error message attached below.

Expected Behavior

No crash occurs

Steps To Reproduce

1. I have 5 physical servers and configured a helm override_values.yml file with dataNode.replicas=4, indexNode.replicas=4, queryNode.replicas=4
2. Create a collection with 4 shards, 2 replicas
3. Insert one record with 300 threads simultaneously
4. The milvus proxy pod crashed

Milvus Log

[2022/12/12 00:01:50.396 +00:00] [ERROR] [proxy/channels_mgr.go:104] [“failed to describe collection”] [error_code=UnexpectedError] [reason=“code: CollectionNotExists, reason: can’t find collection: 0”] [stack=“github.com/milvus-io/milvus/internal/proxy.getDmlChannelsFunc.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:104\ngithub.com/milvus-io/milvus/internal/proxy.(*singleTypeChannelsMgr).getPChans\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:145\ngithub.com/milvus-io/milvus/internal/proxy.(*singleTypeChannelsMgr).getChannels\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:165\ngithub.com/milvus-io/milvus/internal/proxy.(*channelsMgrImpl).getChannels\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:330\ngithub.com/milvus-io/milvus/internal/proxy.(*insertTask).getChannels\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:99\ngithub.com/milvus-io/milvus/internal/proxy.(*insertTask).getPChanStats\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:77\ngithub.com/milvus-io/milvus/internal/proxy.(*dmTaskQueue).addPChanStats\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:262\ngithub.com/milvus-io/milvus/internal/proxy.(*dmTaskQueue).Enqueue\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:235\ngithub.com/milvus-io/milvus/internal/proxy.(*Proxy).Insert\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2539\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Insert\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:670\ngithub.com/milvus-io/milvus-proto/go-api/milvuspb._MilvusService_Insert_Handler.func1\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api@v0.0.0-20221019080323-84e9fa2f9e45/milvuspb/milvus.pb.go:9360\ngithub.com/milvus-io/milvus/internal/proxy.RateLimitInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:52\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/util/logutil/grpc_interceptor.go:22\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:70\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerHookInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:91\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:47\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/tracing/opentracing/server_interceptors.go:38\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/milvuspb._MilvusService_Insert_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api@v0.0.0-20221019080323-84e9fa2f9e45/milvuspb/milvus.pb.go:9362\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1283\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1620\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:922”] [2022/12/12 00:01:50.396 +00:00] [WARN] [proxy/task_scheduler.go:264] [“Proxy dmTaskQueue addPChanStats”] [tID=437991062989701141] [stats={}] [error=“code: CollectionNotExists, reason: can’t find collection: 0”] [2022/12/12 00:01:50.396 +00:00] [WARN] [proxy/impl.go:2540] [“Failed to enqueue insert task: code: CollectionNotExists, reason: can’t find collection: 0”] [2022/12/12 00:01:50.398 +00:00] [ERROR] [proxy/channels_mgr.go:104] [“failed to describe collection”] [error_code=UnexpectedError] [reason=“code: CollectionNotExists, reason: can’t find collection: 0”] [stack=“github.com/milvus-io/milvus/internal/proxy.getDmlChannelsFunc.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:104\ngithub.com/milvus-io/milvus/internal/proxy.(*singleTypeChannelsMgr).getPChans\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:145\ngithub.com/milvus-io/milvus/internal/proxy.(*singleTypeChannelsMgr).getChannels\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:165\ngithub.com/milvus-io/milvus/internal/proxy.(*channelsMgrImpl).getChannels\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/channels_mgr.go:330\ngithub.com/milvus-io/milvus/internal/proxy.(*insertTask).getChannels\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:99\ngithub.com/milvus-io/milvus/internal/proxy.(*dmTaskQueue).popPChanStats\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:296\ngithub.com/milvus-io/milvus/internal/proxy.(*dmTaskQueue).PopActiveTask\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:253\ngithub.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:442\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:220\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:818\ngithub.com/milvus-io/milvus/internal/proxy.(*insertTask).checkLengthOfFieldsData\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:108\ngithub.com/milvus-io/milvus/internal/proxy.(*insertTask).checkPrimaryFieldData\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:128\ngithub.com/milvus-io/milvus/internal/proxy.(*insertTask).PreExecute\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:233\ngithub.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:446”] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2bbe09f]

goroutine 4800058 [running]: github.com/milvus-io/milvus/internal/proxy.(*insertTask).checkLengthOfFieldsData(0x3?) /go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:108 +0x1f github.com/milvus-io/milvus/internal/proxy.(*insertTask).checkPrimaryFieldData(0xc002a53e00) /go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:128 +0x69 github.com/milvus-io/milvus/internal/proxy.(*insertTask).PreExecute(0xc002a53e00, {0xc0034bf780?, 0x1?}) /go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:233 +0xd96 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask(0xc000ec7860?, {0x41073e8, 0xc002a53e00}, {0x4104fd0, 0xc001d0da70}) /go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:446 +0x51b created by github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).manipulationLoop /go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:503 +0x1c5 /received request to invalidate collection meta cache

Anything else?

No response