milvus: [Bug]: Milvus start failed

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.2.13
- Deployment mode(standalone or cluster):standalone 
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2023/08/24 23:13:30.761 -04:00] [INFO] [storage/minio_chunk_manager.go:142] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
2023-08-24 23:13:30,772 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.772 CurlHttpClient [140271551506176] Curl returned error code 7 - Couldn't connect to server

2023-08-24 23:13:30,772 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.772 EC2MetadataClient [140271551506176] Http request to retrieve credentials failed

2023-08-24 23:13:30,782 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.782 CurlHttpClient [140271551506176] Curl returned error code 7 - Couldn't connect to server

2023-08-24 23:13:30,782 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.782 EC2MetadataClient [140271551506176] Http request to retrieve credentials failed

2023-08-24 23:13:30,782 | INFO | default | [SEGCORE][ProcessFormattedStatement][milvus] [AWS LOG] [ERROR] 2023-08-25 03:13:30.782 EC2MetadataClient [140271551506176] Can not retrieve resource from http://169.254.169.254/latest/meta-data/placement/availability-zone

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x39a5eb5]

runtime stack:
runtime.throw({0x42caf1b?, 0x0?})
	/root/tool/go/src/runtime/panic.go:1047 +0x5d fp=0x7f9383ff5ac8 sp=0x7f9383ff5a98 pc=0x151825d
runtime.sigpanic()
	/root/tool/go/src/runtime/signal_unix.go:821 +0x3e9 fp=0x7f9383ff5b28 sp=0x7f9383ff5ac8 pc=0x152faa9

goroutine 370 [syscall]:
runtime.cgocall(0x3395630, 0xc002794fb0)
	/root/tool/go/src/runtime/cgocall.go:157 +0x5c fp=0xc002794f88 sp=0xc002794f50 pc=0x14e297c
github.com/milvus-io/milvus/internal/util/initcore._Cfunc_InitRemoteChunkManagerSingleton({0x7f937e809000, 0x7f937e809010, 0x7f937e809020, 0x7f937e809030, 0x7f937e810000, 0x7f937e810018, 0x7f937e810020, 0x7f937e810028, 0x7f937e810030, 0x0, ...})
	_cgo_gotypes.go:138 +0x58 fp=0xc002794fb0 sp=0xc002794f88 pc=0x2cc7d98
github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager(0x60f9620)
	/root/code/milvus/internal/util/initcore/init_storage_config.go:78 +0x418 fp=0xc002795360 sp=0xc002794fb0 pc=0x2cc8438
github.com/milvus-io/milvus/internal/querynode.(*QueryNode).InitSegcore(0x46ed058?)
	/root/code/milvus/internal/querynode/query_node.go:290 +0x23e fp=0xc0027953e0 sp=0xc002795360 pc=0x2fd02be
github.com/milvus-io/milvus/internal/querynode.(*QueryNode).Init.func1()
	/root/code/milvus/internal/querynode/query_node.go:345 +0x1110 fp=0xc002795ab8 sp=0xc0027953e0 pc=0x2fd1610
sync.(*Once).doSlow(0x4257ad1?, 0x0?)
	/root/tool/go/src/sync/once.go:74 +0xc2 fp=0xc002795b18 sp=0xc002795ab8 pc=0x155e122
sync.(*Once).Do(...)
	/root/tool/go/src/sync/once.go:65
github.com/milvus-io/milvus/internal/querynode.(*QueryNode).Init(0x4257ad1?)
	/root/code/milvus/internal/querynode/query_node.go:296 +0x5b fp=0xc002795b60 sp=0xc002795b18 pc=0x2fd04bb
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init(0xc00007ecc0)
	/root/code/milvus/internal/distributed/querynode/service.go:134 +0x782 fp=0xc002795e50 sp=0xc002795b60 pc=0x31f9c22
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run(0xc000629e00?)
	/root/code/milvus/internal/distributed/querynode/service.go:218 +0x25 fp=0xc002795e90 sp=0xc002795e50 pc=0x31fafa5
github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run(0xc001f903d0?)
	/root/code/milvus/cmd/components/query_node.go:53 +0x2f fp=0xc002795f60 sp=0xc002795e90 pc=0x33814cf
github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()
	/root/code/milvus/cmd/roles/roles.go:120 +0x18e fp=0xc002795fe0 sp=0xc002795f60 pc=0x33841ee
runtime.goexit()
	/root/tool/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002795fe8 sp=0xc002795fe0 pc=0x154ffa1
created by github.com/milvus-io/milvus/cmd/roles.runComponent[...]
	/root/code/milvus/cmd/roles/roles.go:104 +0x191

etcd and minio following:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    ports:
      - "2379:2379"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2022-03-17T06-34-49Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  # standalone:
  #   container_name: milvus-standalone
  #   image: milvusdb/milvus:v2.3.0-beta
  #   command: ["milvus", "run", "standalone"]
  #   environment:
  #     ETCD_ENDPOINTS: etcd:2379
  #     MINIO_ADDRESS: minio:9000
  #   volumes:
  #     - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
  #   ports:
  #     - "19530:19530"
  #     - "9091:9091"
  #   depends_on:
  #     - "etcd"
  #     - "minio"

networks:
  default:
    name: milvus

milvus.yaml is default values

minio:
  address: localhost # Address of MinIO/S3
  port: 9000 # Port of MinIO/S3
  accessKeyID: minioadmin # accessKeyID of MinIO/S3
  secretAccessKey: minioadmin # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  bucketName: a-bucket # Bucket name in MinIO/S3
  rootPath: files # The root path where the message is stored in MinIO/S3
  useIAM: false
  cloudProvider: aws
  logLevel: error

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Comments: 17 (16 by maintainers)

Most upvoted comments

https://github.com/milvus-io/milvus/issues/25990 known issue, Could you add related patch and try it