milvus: [Bug]: Upgrading to 2.2.0 breaks S3 access

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: 2.2.0
- Deployment mode(standalone or cluster): Standalone

Current Behavior

We’ve been running milvus backed by S3. When we try to upgrade to 2.2.0, we get this error:

{"level":"WARN","time":"2022/11/23 18:36:12.842 +00:00","caller":"storage/minio_chunk_manager.go:103","message":"failed to check blob bucket exist","bucket":"milvus-dev","error":"Access Denied."}

This is with no other changes to the config whatsoever, the credentials and settings are confirmed good. If we set the image back to any version prior to 2.2.0, it works again.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 21 (11 by maintainers)

Most upvoted comments

@geroz I ran into the same issue, and after inspecting the codebase to see how environment variables are loaded, I was able to solve this issue by setting the following env variables (i also use external s3 with Milvus version 2.2.3):

    extraEnv:
      - name: minio.accessKeyID
        valueFrom:
          secretKeyRef:
            name: milvus-secret
            key: accessKey
      - name: minio.secretAccessKey
        valueFrom:
          secretKeyRef:
            name: milvus-secret
            key: secretKey

This is the file where you can find all configurable variables, look for Key struct member: https://github.com/milvus-io/milvus/blob/master/internal/util/paramtable/service_param.go

ivancili on Feb 24, 2023

Not really, we went back to 2.1.4 for the time being.

geroz on Dec 9, 2022