thanos: [Object Store Access] Context Deadline exceeded after upgrading to v0.32.4 from v0.31.0
Thanos, Prometheus and Golang version used:
- v0.31.0 --> v0.32.4
Object Storage Provider: AWS s3
What happened: We used k8s and assume role for accessing AWS s3, it worked fine with thanos v0.31.0, after upgrading to v0.32.4, we start seeing compactor complaining, seems regression related to permission.
What you expected to happen: thanos compactor can talk to AWS s3 using assumeRole.
How to reproduce it (as minimally and precisely as possible): normal s3 configuration without users/password, assume role of k8s worker to s3
Full logs to relevant components:
ts=2023-10-09T18:20:13.796433303Z caller=blocks_cleaner.go:44 level=info name=thanos-compactor msg="started cleaning of blocks marked for deletion"
ts=2023-10-09T18:20:13.797847735Z caller=blocks_cleaner.go:58 level=info name=thanos-compactor msg="cleaning of blocks marked for deletion done"
ts=2023-10-09T18:24:50.467766592Z caller=runutil.go:100 level=error name=thanos-compactor msg="function failed. Retrying in next tick" err="BaseFetcher: iter bucket: Get \"https://<redacted bucket>.s3.dualstack.us-west-2.amazonaws.com/?continuation-token=<redacted>&delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix=thanos%2Foregon-dev%2F\": context deadline exceeded"
ts=2023-10-09T18:24:50.46789616Z caller=compact.go:597 level=error name=thanos-compactor msg="retriable error" err="BaseFetcher: iter bucket: Get \"https://<redacted bucket>.s3.dualstack.us-west-2.amazonaws.com/?continuation-token=<redacted>&delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix=thanos%2Foregon-dev%2F\": context deadline exceeded"
s3 config:
"config":
"bucket": "<bucket>"
"endpoint": "s3.us-west-2.amazonaws.com"
"insecure": false
"region": "us-west-2"
"signature_version2": false
"prefix": "thanos/oregon-dev"
"type": "S3"
IAM assume role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com",
"AWS": "arn:aws:iam::<aws account id>:role/KubernetesRoles-IAMRoleWorker-<redacted>"
},
"Action": "sts:AssumeRole"
}
]
}
Anything else we need to know:
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 15 (12 by maintainers)
Also notice in the config that the user set the s3 endpoint as
s3.us-west-2.amazonaws.combut the logs shows3.dualstack.us-west-2.amazonaws.com. This another issue https://github.com/thanos-io/thanos/issues/6804it is reproducible, we are using IAM assume role in AWS instead of direct username/password, roll back to older version fixed this issue.
Like Cortex there is bucket index to help speed up the time when loading blocks. I think Thanos might be able to have similar things to avoid iterating the whole bucket every time.
Is there a way to skip old version when iterating objects in S3? Maybe that is something we can do on our side.
@jnyi We got similar issues after updating Thanos version. Now iterating the whole bucket is required due to https://github.com/thanos-io/thanos/pull/6474. The list objects request might take a long time and timed out. If you are using S3 and enable versioned bucket, you can try to clean up some old versions and try again. This ideally should improve the list performance.