milvus: [Bug]: Milvus start failed
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version:2.2.13
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory:
- GPU:
- Others:
Current Behavior
[2023/08/24 23:13:30.761 -04:00] [INFO] [storage/minio_chunk_manager.go:142] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
2023-08-24 23:13:30,772 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.772 CurlHttpClient [140271551506176] Curl returned error code 7 - Couldn't connect to server
2023-08-24 23:13:30,772 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.772 EC2MetadataClient [140271551506176] Http request to retrieve credentials failed
2023-08-24 23:13:30,782 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.782 CurlHttpClient [140271551506176] Curl returned error code 7 - Couldn't connect to server
2023-08-24 23:13:30,782 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.782 EC2MetadataClient [140271551506176] Http request to retrieve credentials failed
2023-08-24 23:13:30,782 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.782 EC2MetadataClient [140271551506176] Can not retrieve resource from http://169.254.169.254/latest/meta-data/placement/availability-zone
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x39a5eb5]
runtime stack:
runtime.throw({0x42caf1b?, 0x0?})
/root/tool/go/src/runtime/panic.go:1047 +0x5d fp=0x7f9383ff5ac8 sp=0x7f9383ff5a98 pc=0x151825d
runtime.sigpanic()
/root/tool/go/src/runtime/signal_unix.go:821 +0x3e9 fp=0x7f9383ff5b28 sp=0x7f9383ff5ac8 pc=0x152faa9
goroutine 370 [syscall]:
runtime.cgocall(0x3395630, 0xc002794fb0)
/root/tool/go/src/runtime/cgocall.go:157 +0x5c fp=0xc002794f88 sp=0xc002794f50 pc=0x14e297c
github.com/milvus-io/milvus/internal/util/initcore._Cfunc_InitRemoteChunkManagerSingleton({0x7f937e809000, 0x7f937e809010, 0x7f937e809020, 0x7f937e809030, 0x7f937e810000, 0x7f937e810018, 0x7f937e810020, 0x7f937e810028, 0x7f937e810030, 0x0, ...})
_cgo_gotypes.go:138 +0x58 fp=0xc002794fb0 sp=0xc002794f88 pc=0x2cc7d98
github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager(0x60f9620)
/root/code/milvus/internal/util/initcore/init_storage_config.go:78 +0x418 fp=0xc002795360 sp=0xc002794fb0 pc=0x2cc8438
github.com/milvus-io/milvus/internal/querynode.(*QueryNode).InitSegcore(0x46ed058?)
/root/code/milvus/internal/querynode/query_node.go:290 +0x23e fp=0xc0027953e0 sp=0xc002795360 pc=0x2fd02be
github.com/milvus-io/milvus/internal/querynode.(*QueryNode).Init.func1()
/root/code/milvus/internal/querynode/query_node.go:345 +0x1110 fp=0xc002795ab8 sp=0xc0027953e0 pc=0x2fd1610
sync.(*Once).doSlow(0x4257ad1?, 0x0?)
/root/tool/go/src/sync/once.go:74 +0xc2 fp=0xc002795b18 sp=0xc002795ab8 pc=0x155e122
sync.(*Once).Do(...)
/root/tool/go/src/sync/once.go:65
github.com/milvus-io/milvus/internal/querynode.(*QueryNode).Init(0x4257ad1?)
/root/code/milvus/internal/querynode/query_node.go:296 +0x5b fp=0xc002795b60 sp=0xc002795b18 pc=0x2fd04bb
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init(0xc00007ecc0)
/root/code/milvus/internal/distributed/querynode/service.go:134 +0x782 fp=0xc002795e50 sp=0xc002795b60 pc=0x31f9c22
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run(0xc000629e00?)
/root/code/milvus/internal/distributed/querynode/service.go:218 +0x25 fp=0xc002795e90 sp=0xc002795e50 pc=0x31fafa5
github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run(0xc001f903d0?)
/root/code/milvus/cmd/components/query_node.go:53 +0x2f fp=0xc002795f60 sp=0xc002795e90 pc=0x33814cf
github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()
/root/code/milvus/cmd/roles/roles.go:120 +0x18e fp=0xc002795fe0 sp=0xc002795f60 pc=0x33841ee
runtime.goexit()
/root/tool/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002795fe8 sp=0xc002795fe0 pc=0x154ffa1
created by github.com/milvus-io/milvus/cmd/roles.runComponent[...]
/root/code/milvus/cmd/roles/roles.go:104 +0x191
etcd and minio following:
version: '3.5'
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.5
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
ports:
- "2379:2379"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2022-03-17T06-34-49Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "9001:9001"
- "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
# standalone:
# container_name: milvus-standalone
# image: milvusdb/milvus:v2.3.0-beta
# command: ["milvus", "run", "standalone"]
# environment:
# ETCD_ENDPOINTS: etcd:2379
# MINIO_ADDRESS: minio:9000
# volumes:
# - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
# ports:
# - "19530:19530"
# - "9091:9091"
# depends_on:
# - "etcd"
# - "minio"
networks:
default:
name: milvus
milvus.yaml is default values
minio:
address: localhost # Address of MinIO/S3
port: 9000 # Port of MinIO/S3
accessKeyID: minioadmin # accessKeyID of MinIO/S3
secretAccessKey: minioadmin # MinIO/S3 encryption string
useSSL: false # Access to MinIO/S3 with SSL
bucketName: a-bucket # Bucket name in MinIO/S3
rootPath: files # The root path where the message is stored in MinIO/S3
useIAM: false
cloudProvider: aws
logLevel: error
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
No response
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created 10 months ago
- Comments: 17 (16 by maintainers)
https://github.com/milvus-io/milvus/issues/25990 known issue, Could you add related patch and try it