vector: Providing a virtual hosted style s3 bucket endpoint causes the healthcheck to fail
A note for the community
- Please vote on this issue by adding a π reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
When providing the more modern, virtual-hosted-style bucket endpoint (https://bucket-name.s3.region-code.amazonaws.com) and the bucket name to an aws_s3 sink then the healthcheck fails indicating that the bucket cannot be found:
ERROR vector::topology::builder: Healthcheck: Failed Reason. error=Unknown bucket: "redacted-bucket-name"
The bucket name is required. However, this results in the following (incorrect) URL being generated for listing the bucket: https://bucket-name.s3.region-code.amazonaws.com/bucket-name.
Vector can still put objects in this bucket but they are all placed inside a directory named the same as the bucket because it will generate https://bucket-name.s3.region-code.amazonaws.com/bucket-name/key-prefix/key as the URL to upload to.
Given that path-style URLs are being deprecated, it would be good to support the new virtual-hosted-style URLs.
Configuration
[sources.my_source_id]
type = "file"
include = [ "/var/log/**/*.log" ]
[sinks.my_bucket]
type = "aws_s3"
inputs = [ "my_source_id" ]
bucket = "my-logs"
endpoint = "https://my-logs.s3.eu-west-1.amazonaws.com"
Version
vector 0.11.1 (v0.11.1 x86_64-unknown-linux-musl 2020-12-17)
Debug Output
No response
Example Data
No response
Additional Context
Virtual-hosted-style URLs are useful when you want to restrict egress traffic to specific domains.
References
No response
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 7
- Comments: 18 (8 by maintainers)
I believe this is happening because of the
force_path_style(true)configuration here: https://github.com/vectordotdev/vector/blob/master/src/common/s3.rs#L11I believe thatβs the case, unfortunately weβve been blocked from upgrading by a regression that was introduced a few versions ago. Working through that is on my todo list in the next few weeks IIRC.
Thanks, I will update and report back π
I definitely recommend you update to the latest Vector version (0.29.1 right now) and test your case on this version. If the issue still remains - please let us know. So old Vector versions are not supported, unfortunately.
Just tested this again on 0.31.0 and I am still seeing the same problem. Itβs odd because 0.31.0 supposedly contains this change: https://github.com/vectordotdev/vector/pull/17731 When I look at the forked AWS SDK that the hash points to then it appears to contain the change that should have fixed this: https://github.com/vectordotdev/aws-sdk-rust/blob/3d6aefb7fcfced5fc2a7e761a87e4ddbda1ee670/CHANGELOG.md#january-13th-2023 So it seems that there might be something else going on here.
I didnβt see an issue so I opened https://github.com/vectordotdev/vector/issues/17728
Do you happen to have a link to the issue so we can track it? π