milvus: [Bug]: log.file.rootPath failed to apply in milvus 2.0.2

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version:2.0.2
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Brought up the milvus cluster by running cmd helm install -f ./charts/milvus/values.yaml milvus-dev milvus/milvus with the following log settings:

log:
  level: "debug"
  file:
    rootPath: "/tmp"
    maxSize: 300    # MB
    maxAge: 10    # day
    maxBackups: 20
  format: "text"    # text/json

  persistence:
    mountPath: "/milvus/logs"
    ## If true, create/use a Persistent Volume Claim
    ## If false, use emptyDir
    ##
    enabled: false
    annotations:
      helm.sh/resource-policy: keep
    persistentVolumeClaim:
      existingClaim: ""
      ## Milvus Logs Persistent Volume Storage Class
      ## If defined, storageClassName: <storageClass>
      ## If set to "-", storageClassName: "", which disables dynamic provisioning
      ## If undefined (the default) or set to null, no storageClassName spec is
      ##   set, choosing the default provisioner.
      ## ReadWriteMany access mode required for milvus cluster.
      ##
      storageClass: gp3
      accessModes: ReadWriteMany
      size: 100Gi
      subPath: ""

After cluster was up, checked the queryNode but /milvus/configs/milvus.yaml showed log.file.rootPath as the empty string “”.

Expected Behavior

Expect to see logs folder auto-created under the specified path log.file.rootPath.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (9 by maintainers)

Most upvoted comments

Another question to note is that:

"{"level":"warn","ts":"2022-06-02T02:57:30.296Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"3a40c4043d8c5dc6","local-member-cluster-id":"f5ab8dc0d0e31730","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"64de156f740416f0","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d580a929db73018b"}"

When etcd member is starting, it will calculate the cluster_id based on the peer urls, we’d filter out the pod-ip.<svc-name>.svc.cluster.local dns name, using the headless svc name.

This is temporarily fixed using the following image:

etcd:
    image:
      repository: garfieldedward/etcd
      tag: "3.5.0-debian-10-r115"

After the performance test and stability test are all passed, we’ll push this image to milvusdb/etcd repository.

LoveEachDay on Jun 6, 2022

Caused by: java.lang.ClassNotFoundException: "-Xms1024m

This is the config error for pulsar zookeeper. Please keep the same format like this:

  zookeeper:
    resources:
      requests:
        memory: 1024Mi
        cpu: 0.3
    configData:
      PULSAR_MEM: >
        -Xms1024m
        -Xmx1024m
      PULSAR_GC: >
         -Dcom.sun.management.jmxremote
         -Djute.maxbuffer=10485760
         -XX:+ParallelRefProcEnabled
         -XX:+UnlockExperimentalVMOptions
         -XX:+DoEscapeAnalysis
         -XX:+DisableExplicitGC
         -XX:+PerfDisableSharedMem
         -Dzookeeper.forceSync=no

LoveEachDay on May 31, 2022