milvus: [Bug]: [scale] The insert time of first run is too long and after scale up, a search time is very large

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:v2.0.0rc8
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):v2.0.0rc8
- OS(Ubuntu or CentOS): centos7.6
- CPU/Memory: 
           AMD EPYC 7702 64-Core Processor
              123GB
- GPU: 
- Others:

Current Behavior

python SDK Blocking on interface collection.load() when add the nodes

Expected Behavior

No response

Steps To Reproduce

1. used the docker-compose to start milvus2.0 with the docker-compose.yml in computer 21
2. tested the test.py,The test is normal
3. used the docker-compose to start three milvul-nodes with the docker-compose.yml in computer 22
3. tested the test.py, Blocking on interface collection.load()

Anything else?

computer 21 docker-compose.yml


version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.0
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    #command: etcd -advertise-client-urls=http://127.0.0.1:2379,http://192.168.21.21:2379  -listen-client-urls http://192.168.21.21:2379  --data-dir /etcd
    command: etcd -advertise-client-urls=http://0.0.0.0:2379  -listen-client-urls http://0.0.0.0:2379    -listen-peer-urls http://0.0.0.0:2380 -initial-advertise-peer-urls http://0.0.0.0:2380  -initial-cluster default=http://0.0.0.0:2380 --data-dir /etcd
    ports:
      - "2379:2379"
      - "2380:2380"

  pulsar:
    container_name: milvus-pulsar
    image: apachepulsar/pulsar:2.7.3
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/pulsar:/pulsar/data
    environment:
      # bin/apply-config-from-env.py script will modify the configuration file based on the environment variables
      # nettyMaxFrameSizeBytes must be calculated from maxMessageSize + 10240 (padding)
      - nettyMaxFrameSizeBytes=104867840 # this is 104857600 + 10240 (padding)
      - defaultRetentionTimeInMinutes=10080
      - defaultRetentionSizeInMB=-1
      # maxMessageSize is missing from standalone.conf, must use PULSAR_PREFIX_ to get it configured
      - PULSAR_PREFIX_maxMessageSize=104857600
      - PULSAR_GC=-XX:+UseG1GC
    command: |
      /bin/bash -c \
      "bin/apply-config-from-env.py conf/standalone.conf && \
      exec bin/pulsar standalone --no-functions-worker --no-stream-storage"
    ports:
      - "6650:6650"

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2020-12-03T00-03-10Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://192.168.21.21:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    ports:
      - "9000:9000"

  rootcoord:
    container_name: milvus-rootcoord
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "rootcoord"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      ROOT_COORD_ADDRESS: 192.168.21.21:53100
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    depends_on:
      - "etcd"
      - "pulsar"
      - "minio"
    ports:
      - "53100:53100"

  proxy:
    container_name: milvus-proxy
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "proxy"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    ports:
      - "19888:19530"

  querycoord:
    container_name: milvus-querycoord
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "querycoord"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      QUERY_COORD_ADDRESS: 192.168.21.21:19531
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    depends_on:
      - "etcd"
      - "pulsar"
      - "minio"
    ports:
      - "19531:19531"

  querynode:
    container_name: milvus-querynode
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "querynode"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    depends_on:
      - "querycoord"

  indexcoord:
    container_name: milvus-indexcoord
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "indexcoord"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      INDEX_COORD_ADDRESS: 192.168.21.21:31000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    depends_on:
      - "etcd"
      - "pulsar"
      - "minio"
    ports:
      - "31000:31000"

  indexnode:
    container_name: milvus-indexnode
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "indexnode"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      INDEX_COORD_ADDRESS: 192.168.21.21:31000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    depends_on:
      - "indexcoord"

  datacoord:
    container_name: milvus-datacoord
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "datacoord"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      DATA_COORD_ADDRESS: 192.168.21.21:13333
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    depends_on:
      - "etcd"
      - "pulsar"
      - "minio"
    ports:
      - "13333:13333"

  datanode:
    container_name: milvus-datanode
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "datanode"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs
    depends_on:
      - "datacoord"

networks:
  default:
    name: milvus

computer 22 docker-compose.yml

version: '3.5'

services:

  querynode:
    container_name: milvus-querynode
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "querynode"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      QUERY_COORD_ADDRESS: 192.168.21.21:19531
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs


  indexnode:
    container_name: milvus-indexnode
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "indexnode"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      INDEX_COORD_ADDRESS: 192.168.21.21:31000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs


  datanode:
    container_name: milvus-datanode
    image: milvusdb/milvus:v2.0.0-rc8-20211104-d1f4106
    command: ["milvus", "run", "datanode"]
    environment:
      ETCD_ENDPOINTS: 192.168.21.21:2379
      MINIO_ADDRESS: 192.168.21.21:9000
      PULSAR_ADDRESS: pulsar://192.168.21.21:6650
      DATA_COORD_ADDRESS: 192.168.21.21:13333
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/configs:/milvus/configs

#networks:
#  default:
#    name: milvus

test.py

# Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.


import random
import time
import datetime

from pymilvus import (
    connections, list_collections,
    FieldSchema, CollectionSchema, DataType,
    Collection
)

connections.connect(host='192.168.21.21', port="19888")
dim = 640
#collection_name='test_100w_seg100W'
collection_count = 1
collection_name='test_' + str(collection_count) + 'w'
# create collection
default_fields = [
FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
]
default_schema = CollectionSchema(fields=default_fields, description="test collection")
collection = Collection(name=collection_name, schema=default_schema)
search_vec = []

def create_collection(cname = collection_name):
    print(f"\nList collections...")
    print(list_collections())


def insert_data(count = 1000000, nb = 10000):
    print("begin insert, count:", count, ",nb:", nb)

    times = int(count / nb)
    total_time = 0.0

    for j in range(times):

        vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]


        start_time = time.time()
        collection.insert(
            [
                [i for i in range(nb * j, nb * j + nb)],
                vectors
            ]
        )
        end_time = time.time()
        total_time += end_time - start_time

    print("end insert, time:", total_time)
    print(f"\nGet collection entities...")
    print(collection.num_entities)
    print("before collection load")
    collection.load()
    print("after collection load")


def create_index():
    # create index and load table
    default_index = {"index_type": "HNSW", "params": {"M": 48, "efConstruction": 500}, "metric_type": "L2"}
    print(f"\nCreate index...")
    start_time = time.time()
    collection.create_index(field_name="float_vector", index_params=default_index)
    end_time = time.time()
    print("end Create index... time:", end_time - start_time)
    print(f"\nload collection...")
    collection.load()

def drop_index():
    print(f"\nbegin drop index...")
    start_time = time.time()
    collection.drop_index()
    end_time = time.time()
    print("end drop index... time:", end_time - start_time)

def search(nq = 100, times = 10):

    topK = 1
    search_params = {"metric_type": "L2", "params": {"ef":300}}
    import time

    total_time = 0.0

    print(f"\nSearch...")
    for i in range(times):
        vectors = [[random.random() for _ in range(dim)] for _ in range(nq)]

        start_time = time.time()
        res = collection.search(
            vectors, "float_vector", search_params, topK,
            "", output_fields=["id"]
        )
        end_time = time.time()
        total_time += end_time - start_time
        print("search time:" , end_time - start_time)
    print("ave(times):", total_time / times, ", ave(nq):", total_time / times / nq)


def drop_collection():
    start_time = time.time()
    collection.drop()
    end_time = time.time()
    print(f"\ndrop colection:", collection_name, ",time:", end_time - start_time)


def collection_info():
    print("index info:",[index.params for index in collection.indexes])
    print("index info:", collection.index())
    print("index info:", collection.index)
    print("entities info:", collection.num_entities)

import argparse

parser = argparse.ArgumentParser(description='host ip')
parser.add_argument('--host', type=str, default='127.0.0.1', help='host ip')
args = parser.parse_args()

#drop_collection()
btime = datetime.datetime.now()
print("begin test : ", btime)
create_collection()
insert_data(collection_count * 10000)
search(1)
search(100)
search(2000)
create_index()
#drop_index()
search(1)
search(100)
search(2000)
#create_index()
#search()

collection_info()

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 28 (26 by maintainers)

Most upvoted comments

I’ll use the main branch test, thanks. @xige-16

+1
WordlessToContinue on Nov 25, 2021
Read more comments on GitHub
← milvus: [Bug]: rocksmq causes disk to fill up
milvus: [Bug]: Search failed when we insert data into Milvus, both datanode and querynode keep crashing →