vector: Kinesis sink endpoint bug in AWS

We are implementing Kinesis sink in AWS VPC.

Kinesis data stream is located in different AWS account and VPC than our EC2 instance with vector configuration. We have setup VPC peering between VPCs, we have adjusted security groups, added IAM roles with sts AssumeRole and configured route table accordingly. Our EC2 instance does not have internet access, so we are using VPC endpoints to access Kinesis data streams and sts. Kinesis endpoint is located on remote VPC, where Kinesis data streams are configured, but sts endpoint is local to our EC2 interneltess instance which have vector configured.

We have encountered an issue that if we configure Private DNS endpoint in vector.toml for Kinesis endpoint which is located in remote VPC, we received following error:

We have these two VPC endpoints in use: STS EP - IP 10.255.252.97 KinesIs EP - IP 10.75.2.73

EC2 instance with vector IP: 10.255.252.111

During the non-working configuration in tcpdump we do not see any traffic towards STS IP 10.255.252.97

But we can see traffic towards Kinesis EP IP 10.75.2.73 via VPC peering

The current workaround is to remove all endpoint configuration from vector.toml and this by default makes all STS commands to be sent to regional endpoint and all Kinesis commands to global endpoint kinesis.eu-central-1.amazonaws.com.

But since the EC2 does not have an internet access and cannot reach the global Kinesis endpoint, we have added DNS entry in /etc/hosts for global Kinesis endpoint to use the regional endpoint IP.

When checking .pcap with this scenario we can verify that there is traffic to both STS endpoint and Kinesis endpoint:

Long story short, and our understanding is that if we configure endpoint for kinesis sink, everything seems to be sent to that endpoint (kinesis data stream and as well as sts), we can not specify vector to use local sts endpoint instead of confiugred Kinesis data stream endpoint. By removing endpoint configuration, adding region in the config, and adding in EC2 instance hosts file to resolve that global kinesis DNS to our peering Kinesis endpoint IP, then it works, because it resolves to private IP to which we have connectivity and sts goes via localy configured endpoint in that case.