thanos: Thanos Sidecar err="check exists: stat s3 object: Access Denied."

Hi Team,

I have configured Thanos sidecar, Thanos is in account A and s3 is in account B.
I have also created a role and instance profile for EC2 in account A to assume role in account B, so that I would not need to pass the AWS access and secret keys. The prometheus and thanos are running inside of this EC2 instance as systemd services and not docker containers.
As per the docs, I can see that the credentials are retrieved from the instance profile, but still get the below errors while uploading:

/usr/local/bin/thanos sidecar --tsdb.path=/var/lib/prometheus --prometheus.url=http://localhost:9090 --objstore.config-file /var/lib/thanos/objectstore.yaml
level=info ts=2020-12-28T10:34:58.137650889Z caller=main.go:98 msg="Tracing will be disabled"
level=info ts=2020-12-28T10:34:58.13788683Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2020-12-28T10:34:58.138189187Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2020-12-28T10:34:58.138518739Z caller=sidecar.go:291 msg="starting sidecar"
level=info ts=2020-12-28T10:34:58.138653441Z caller=reloader.go:183 component=reloader msg="nothing to be watched"
level=info ts=2020-12-28T10:34:58.138705885Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2020-12-28T10:34:58.139555567Z caller=grpc.go:116 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2020-12-28T10:34:58.13960344Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2020-12-28T10:34:58.139616301Z caller=http.go:58 service=http/server component=sidecar msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2020-12-28T10:34:58.147099692Z caller=sidecar.go:155 msg="successfully loaded prometheus external 
level=info ts=2020-12-28T10:34:58.147144958Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=warn ts=2020-12-28T10:35:00.215032014Z caller=sidecar.go:275 err="check exists: stat s3 object: Access Denied." uploaded=0
level=warn ts=2020-12-28T10:35:30.176978953Z caller=sidecar.go:275 err="check exists: stat s3 object: Access Denied." uploaded=0
level=warn ts=2020-12-28T10:36:00.170973791Z caller=sidecar.go:275 err="check exists: stat s3 object: Access Denied." uploaded=0
level=warn ts=2020-12-28T10:36:30.167065199Z caller=sidecar.go:275 err="check exists: stat s3 object: Access Denied." uploaded=0

When I manually do this, it works completely fine:

aws s3 ls s3://my-bucket
           test.txt

I can see an issue related to it #394 and also #450 merged, so I had upgraded my thanos to 0.17.2

The Role on my S3 bucket has the below permissions as given in the docs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket>/*",
                "arn:aws:s3:::<bucket>"
            ]
        }
    ]
}

Thanos, version 0.17.2

Prometheus, version 2.11.1

Environment:

OS (e.g. from /etc/os-release): NAME=“Red Hat Enterprise Linux”, VERSION=“8.3 (Ootpa)”

Could you please have a look and help me understand if my approach is correct or not so as to avoid using ACCESS and SECRET KEY and work with Instance profile. Also, please help me know if I have missed anything. Thanks a lot in advance.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 6
Comments: 27 (2 by maintainers)

Most upvoted comments

Hello Team, can anyone have a look at it please. Thanks in Advance.

abhijeetab07 on Jan 5, 2021

This still an issue?

DerrickMartinez on Oct 14, 2021

I am having same issue.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Restrict Non-https Requests", "Effect": "Deny", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::mybucket/*", "Condition": { "Bool": { "aws:SecureTransport": "false" } } } ] }

level=info ts=2022-05-19T12:14:45.237899866Z caller=factory.go:46 msg="loading bucket configuration" level=info ts=2022-05-19T12:14:45.238363784Z caller=inmemory.go:172 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt level=info ts=2022-05-19T12:14:45.238920072Z caller=options.go:24 protocol=gRPC msg="disabled TLS, key and cert must be set to enable" level=info ts=2022-05-19T12:14:45.240592638Z caller=store.go:428 msg="starting store node" level=info ts=2022-05-19T12:14:45.240665782Z caller=store.go:363 msg="initializing bucket store" level=info ts=2022-05-19T12:14:45.240755799Z caller=intrumentation.go:60 msg="changing probe status" status=healthy level=info ts=2022-05-19T12:14:45.240811358Z caller=http.go:63 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902 level=info ts=2022-05-19T12:14:45.24097428Z caller=tls_config.go:191 service=http/server component=store msg="TLS is disabled." http2=false level=warn ts=2022-05-19T12:14:45.984182456Z caller=intrumentation.go:54 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: incomplete view: 1036 errors: meta.json file exists: 01FW85KR4R5FDMG427SKHJDF8A/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FW7QW9MRDTPDM9V5J26M9JC2/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWF1B05EM9F83YNQH9QZ0A04/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWD3HEX2QNZ3XPHC3ATYY2RV/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FW912N4TT88X8H3F05C7P2XR/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FW7YR0WTGAAWKNMRHWNFB238/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWADQSCJF50X3A27V4KRXCN3/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWDZ0BWSHBK96M9RFHC7D9WG/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FW9WHJ4VHKJN7RRTNEG0W9QE/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWDAD64RE0JCRE47PF7XK8EC/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWCNT0D6YFXZKS8M3XFMV9TJ/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWC82HWYS54ADPFWY0MCFQWC/meta.json: stat s3 object: Access Denied.; meta.json file exists: 01FWECQTCTR23FJ9R96WTF0K4G/meta.json: stat s3 object: Access Denied.;

kumarganesh2814 on May 19, 2022

Hey @sherifkayad, sure!

This is the policy attached to the IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket-monitor/*",
                "arn:aws:s3:::mybucket-monitor"
            ]
        }
    ]
}

This is the resource policy configured in my S3 bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123:root",
                    "arn:aws:iam::456:root",
                    "arn:aws:iam::789:root"
                ]
            },
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket-monitor/*",
                "arn:aws:s3:::mybucket-monitor"
            ]
        }
    ]
}

Long time update: After making some changes on the architecture recently I also had to add the s3:GetObjectAcl and s3:PutObjectAcl actions because of the put_user_metadata: {"X-Amz-Acl": "bucket-owner-full-control"} setting.

Lincon-Freitas on Sep 7, 2023

A google search led me to this issue, my issue was self inflicted and i’ll post the solution to help anyone else out.

My issue was in the trust relationship and using wildcards for the KSA vs the full KSA. If you use a KSA with a wildcard like "system:serviceaccount:monitoring:prometheus-*" you must use StringLike, I ended up copy pasting StringEquals

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::1234:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/SADNAF23AS"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.us-east-1.amazonaws.com/id/SADNAF23AS:sub": "system:serviceaccount:monitoring:prometheus-*"
                }
            }
        }
    ]
}

smark88 on Feb 7, 2023

i reproduce the issue when upgrading thanos-sidecar base image to 0.28.1 then 0.29.0 all working good with 0.27.0 don’t think it’s related to s3/IAM policies unless breaking changes were introduced in 0.28.1

yassineaouadi on Nov 7, 2022

v0.14 is the working one.

jerry153fish on Mar 2, 2021