vector: Providing a virtual hosted style s3 bucket endpoint causes the healthcheck to fail

A note for the community

  • Please vote on this issue by adding a πŸ‘ reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When providing the more modern, virtual-hosted-style bucket endpoint (https://bucket-name.s3.region-code.amazonaws.com) and the bucket name to an aws_s3 sink then the healthcheck fails indicating that the bucket cannot be found:

ERROR vector::topology::builder: Healthcheck: Failed Reason. error=Unknown bucket: "redacted-bucket-name"

The bucket name is required. However, this results in the following (incorrect) URL being generated for listing the bucket: https://bucket-name.s3.region-code.amazonaws.com/bucket-name.

Vector can still put objects in this bucket but they are all placed inside a directory named the same as the bucket because it will generate https://bucket-name.s3.region-code.amazonaws.com/bucket-name/key-prefix/key as the URL to upload to.

Given that path-style URLs are being deprecated, it would be good to support the new virtual-hosted-style URLs.

Configuration

[sources.my_source_id]
type = "file"
include = [ "/var/log/**/*.log" ]
[sinks.my_bucket]
type = "aws_s3"
inputs = [ "my_source_id" ]
bucket = "my-logs"
endpoint = "https://my-logs.s3.eu-west-1.amazonaws.com"

Version

vector 0.11.1 (v0.11.1 x86_64-unknown-linux-musl 2020-12-17)

Debug Output

No response

Example Data

No response

Additional Context

Virtual-hosted-style URLs are useful when you want to restrict egress traffic to specific domains.

References

No response

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 7
  • Comments: 18 (8 by maintainers)

Most upvoted comments

I believe this is happening because of the force_path_style(true) configuration here: https://github.com/vectordotdev/vector/blob/master/src/common/s3.rs#L11

It looks like support for virtual-hosted-style URLs was added to the aws rust sdk a few months back, so it might just be a question of updating the sdk? https://github.com/awslabs/aws-sdk-rust/releases/tag/release-2023-01-13

I believe that’s the case, unfortunately we’ve been blocked from upgrading by a regression that was introduced a few versions ago. Working through that is on my todo list in the next few weeks IIRC.

Thanks, I will update and report back πŸ‘

vector 0.11.1 (v0.11.1 x86_64-unknown-linux-musl 2020-12-17)

I definitely recommend you update to the latest Vector version (0.29.1 right now) and test your case on this version. If the issue still remains - please let us know. So old Vector versions are not supported, unfortunately.

Just tested this again on 0.31.0 and I am still seeing the same problem. It’s odd because 0.31.0 supposedly contains this change: https://github.com/vectordotdev/vector/pull/17731 When I look at the forked AWS SDK that the hash points to then it appears to contain the change that should have fixed this: https://github.com/vectordotdev/aws-sdk-rust/blob/3d6aefb7fcfced5fc2a7e761a87e4ddbda1ee670/CHANGELOG.md#january-13th-2023 So it seems that there might be something else going on here.

Do you happen to have a link to the issue so we can track it? πŸ‘€

I didn’t see an issue so I opened https://github.com/vectordotdev/vector/issues/17728

Do you happen to have a link to the issue so we can track it? πŸ‘€