thanos: Enforcing v2 of AWS IDMS causes Access Denied error

Thanos, Prometheus and Golang version used:

name: "thanosio/thanos",
version: "0.13.0",
tag: "v0.13.0",
checksum: "sha256:743e5a7df61d67a16fd3e27fae4cae2af335cfe36f8b210240b2c14e9839b2a7"

name: "prom/prometheus",
version: "2.19.2",
tag: "v2.19.2",
checksum: "sha256:cd134bd4fca0f60ff8b4c679cebe5c5c5cf5e2da5f4886b2ae933da821915f92"

Note: I’ve also tried with the latest stable version of Thanos (0.14) but the same error appeared.

Object Storage Provider:

What happened: Following AWS security recommendations I have enforced usage of v2 IDMS on EC2 instances of Thanos Compactor. This has caused the following error to appear in logs:

{
    "caller": "runutil.go:98",
    "err": "BaseFetcher: iter bucket: Access Denied",
    "level": "error",
    "msg": "function failed. Retrying in next tick",
    "ts": "2020-08-28T20:05:24.151455425Z"
}

which seems to be coming from fetcher.go file, which in turn seems to be using Minio library. (I wonder if upgrading Minio could help at all).

Also note that when I remove the policy, error reliably disappears, so it shouldn’t be related with permissions to access a bucket.

What you expected to happen: No errors.

How to reproduce it (as minimally and precisely as possible): Enforce v2 of IDMS on Compactor EC2 instance, which includes two steps:

• set metadata_options.http_tokens to “required”
• add an IAM instance policy to Compactor Amazon EC2 instance to enforce usage of v2 of IDMS

{
  "Sid": "RequireAllEc2RolesToUseV2",
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "NumericLessThan": {
      "ec2:RoleDelivery": "2.0"
    }
  }
}

Full logs to relevant components: N/A Anything else we need to know: N/A