vector: Unable to get Vector to use the FIPS AWS endpoints

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The AWS Rust SDK can the ability to use the AWS FIPS service endpoints if you supply a config file with use_fips_endpoint=true or an environment variable of AWS_USE_FIPS_ENDPOINT=true. For example: https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/aws-config/src/default_provider/use_fips.rs#L26

By providing these options to Vector either through the AWS_CONFIG_FILE environment variable, directly in a config file, systemd unit file overrides and via running it on the command line, I can’t get Vector to read this file and use these options.

Our use case is for the AWS Kinesis Stream sink using an Assumed Role. The calls to AWS STS to assume the role, and the calls to Kinesis must use the FIPS endpoints as shown here https://aws.amazon.com/compliance/fips/

Configuration

Test command (also tested via systemd as below):
AWS_USE_FIPS_ENDPOINT=true VECTOR_LOG=debug vector --config=/etc/vector/vector.toml > vector.log 2>&1

# cat /etc/systemd/system/vector.service.d/override.conf
# Also attempted to set the AWS_USE_FIPS_ENDPOINT environment variable here
[Service]
Environment="AWS_CONFIG_FILE=/var/lib/vector/.aws/config"

# cat /var/lib/vector/.aws/config
[default]
use_fips_endpoint = true

# cat /root/.aws/config
[default]
use_fips_endpoint = true

Snippet from /etc/vector/vector.toml
[sinks.to_kinesis]
  type = "aws_kinesis_streams"
  inputs = ["logs"]
  region = "us-west-2"
  stream_name = "logs"
  batch.max_bytes = 5000000
  batch.max_events = 500
  buffer.type = "memory"
  buffer.max_events = 160000
  buffer.when_full = "drop_newest"

  [sinks.to_kinesis.encoding]
  codec = "json"

  [sinks.to_kinesis.healthcheck]
  enabled = false

  [sinks.to_kinesis.auth]
  assume_role = "our arn role"
  external_id = "external-id"

Version

vector 0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)

Debug Output

2023-08-24T23:10:57.664997Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-08-24T23:10:57.665109Z  INFO vector::app: Log level is enabled. level="debug"
2023-08-24T23:10:57.665122Z  WARN vector::app: DEPRECATED The openssl legacy provider provides algorithms and key sizes no longer recommended for use. Set `--openssl-legacy-provider=false` or `VECTOR_OPENSSL_LEGACY_PROVIDER=false` to disable. See https://vector.dev/highlights/2023-08-15-0-32-0-upgrade-guide/#legacy-openssl for details.
2023-08-24T23:10:57.665758Z  INFO vector::app: Loaded openssl provider. provider="legacy"
2023-08-24T23:10:57.665799Z  INFO vector::app: Loaded openssl provider. provider="default"
2023-08-24T23:10:57.666386Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-08-24T23:10:57.668290Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-08-24T23:10:57.674868Z DEBUG vector::topology::builder: Building new source. component=syslog_in
2023-08-24T23:10:57.675257Z DEBUG vector::topology::builder: Building new transform. component=syslog_logs
2023-08-24T23:10:57.679391Z DEBUG vector::topology::builder: Building new sink. component=to_file
2023-08-24T23:10:57.680762Z DEBUG vector::topology::builder: Building new sink. component=to_kinesis
2023-08-24T23:10:57.730037Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: aws_config::fs_util: loaded home directory src="HOME"
2023-08-24T23:10:57.730085Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Config)}:aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-08-24T23:10:57.730117Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Config)}:aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-08-24T23:10:57.730169Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Config)}:aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=35
2023-08-24T23:10:57.730191Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-08-24T23:10:57.730202Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-08-24T23:10:57.730217Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-08-24T23:10:57.730228Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-08-24T23:10:57.872317Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector_core::tls::settings: Fetching system root certs.
2023-08-24T23:10:57.909204Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector_core::tls::settings: Fetching system root certs.
2023-08-24T23:10:57.949259Z  INFO vector::topology::running: Running healthchecks.
2023-08-24T23:10:57.949305Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-08-24T23:10:57.949374Z  INFO vector::topology::builder: Healthcheck disabled.
2023-08-24T23:10:57.949427Z DEBUG vector::topology::running: Configuring outputs for source. component=syslog_in
2023-08-24T23:10:57.949435Z  INFO vector::topology::builder: Healthcheck passed.
2023-08-24T23:10:57.949440Z DEBUG vector::topology::running: Configuring output for component. component=syslog_in output_id=None
2023-08-24T23:10:57.949452Z DEBUG vector::topology::running: Configuring outputs for transform. component=syslog_logs
2023-08-24T23:10:57.949460Z DEBUG vector::topology::running: Configuring output for component. component=syslog_logs output_id=None
2023-08-24T23:10:57.949471Z DEBUG vector::topology::running: Connecting inputs for transform. component=syslog_logs
2023-08-24T23:10:57.949484Z DEBUG vector::topology::running: Adding component input to fanout. component=syslog_logs fanout_id=syslog_in
2023-08-24T23:10:57.949498Z DEBUG vector::topology::running: Connecting inputs for sink. component=to_kinesis
2023-08-24T23:10:57.949512Z DEBUG vector::topology::running: Adding component input to fanout. component=to_kinesis fanout_id=syslog_logs
2023-08-24T23:10:57.949522Z DEBUG vector::topology::running: Connecting inputs for sink. component=to_file
2023-08-24T23:10:57.949532Z DEBUG vector::topology::running: Adding component input to fanout. component=to_file fanout_id=syslog_logs
2023-08-24T23:10:57.949551Z DEBUG vector::topology::running: Spawning new source. key=syslog_in
2023-08-24T23:10:57.949583Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="file" component_id="syslog_in" group_id="2"
2023-08-24T23:10:57.949614Z DEBUG vector::topology::running: Spawning new transform. key=syslog_logs
2023-08-24T23:10:57.949636Z DEBUG vector::topology::running: Registered new allocation group. component_kind="transform" component_type="remap" component_id="syslog_logs" group_id="3"
2023-08-24T23:10:57.949636Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::topology::builder: Source pump supervisor starting.
2023-08-24T23:10:57.949662Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="aws_kinesis_streams" component_id="to_kinesis" group_id="4"
2023-08-24T23:10:57.949679Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::topology::builder: Source pump starting.
2023-08-24T23:10:57.949682Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="file" component_id="to_file" group_id="5"
2023-08-24T23:10:57.949707Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::topology::builder: Source starting.
2023-08-24T23:10:57.949735Z  INFO source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}: vector::sources::file: Starting file server. include=["/var/log/syslog"] exclude=[]
2023-08-24T23:10:57.949766Z  INFO vector: Vector has started. debug="false" version="0.32.1" arch="x86_64" revision="9965884 2023-08-21 14:52:38.330227446"
2023-08-24T23:10:57.949787Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-08-24T23:10:57.949872Z DEBUG transform{component_kind="transform" component_id=syslog_logs component_type=remap component_name=syslog_logs}: vector::topology::builder: Synchronous transform starting.
2023-08-24T23:10:57.949894Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector::topology::builder: Sink starting.
2023-08-24T23:10:57.949993Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::topology::builder: Sink starting.
2023-08-24T23:10:57.950055Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}: vector::utilization: utilization=0.4031854493462015
2023-08-24T23:10:57.950323Z  INFO source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-08-24T23:10:57.950509Z  INFO source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/syslog file_position=696653
2023-08-24T23:10:57.952486Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::utilization: utilization=0.003860596633269875
2023-08-24T23:10:57.954606Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::internal_events::file: Unable to open the file. path=b"/tmp/vector-2023-08-24.log" error=Permission denied (os error 13) error_code=failed_opening_file error_type="io_failed" stage="sending" internal_log_rate_limit=true
2023-08-24T23:10:57.954652Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Unable to open the file." internal_log_rate_limit=true
2023-08-24T23:10:57.954801Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector::internal_events::file: Internal log [Unable to openthe file.] is being suppressed to avoid flooding.
2023-08-24T23:10:57.954820Z ERROR sink{component_kind="sink" component_id=to_file component_type=file component_name=to_file}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2023-08-24T23:10:58.954031Z DEBUG vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=2
2023-08-24T23:10:58.955479Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role: aws_config::sts::assume_role: retrieving assumed credentials
2023-08-24T23:10:58.955562Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "environment variable not set" }))
2023-08-24T23:10:58.955593Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2023-08-24T23:10:58.955604Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-08-24T23:10:58.955620Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-08-24T23:10:58.955651Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=35
2023-08-24T23:10:58.955662Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-08-24T23:10:58.955669Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-08-24T23:10:58.955678Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-08-24T23:10:58.955685Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-08-24T23:10:58.955711Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profilecontext=the credential provider was not enabled: profile `default` did not contain credential information (CredentialsNotLoaded(CredentialsNotLoaded { source: ProfileDidNotContainCredentials { profile: "default" } }))
2023-08-24T23:10:58.955730Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=the credential provider was not enabled: $AWS_WEB_IDENTITY_TOKEN_FILE was not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "$AWS_WEB_IDENTITY_TOKEN_FILE was not set" }))
2023-08-24T23:10:58.955746Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=the credential provider was not enabled: ECS provider not configured (CredentialsNotLoaded(CredentialsNotLoaded { source: "ECS provider not configured" }))
2023-08-24T23:10:58.955755Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-08-24T23:10:58.955884Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-08-24T23:10:58.956297Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-08-24T23:10:58.956496Z DEBUG hyper::proto::h1::io: flushed 242 bytes
2023-08-24T23:10:58.956916Z DEBUG hyper::proto::h1::io: parsed 6 headers
2023-08-24T23:10:58.956928Z DEBUG hyper::proto::h1::conn: incoming body is content-length (56 bytes)
2023-08-24T23:10:58.956945Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.957075Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-08-24T23:10:58.957293Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-08-24T23:10:58.957380Z DEBUG hyper::proto::h1::io: flushed 307 bytes
2023-08-24T23:10:58.957663Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-08-24T23:10:58.957810Z DEBUG hyper::proto::h1::conn: incoming body is content-length (26 bytes)
2023-08-24T23:10:58.957820Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.957896Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=dummy-profile
2023-08-24T23:10:58.957972Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-08-24T23:10:58.958100Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-08-24T23:10:58.958181Z DEBUG hyper::proto::h1::io: flushed 333 bytes
2023-08-24T23:10:58.958414Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-08-24T23:10:58.958421Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1570 bytes)
2023-08-24T23:10:58.958433Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.958550Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata
2023-08-24T23:10:58.958565Z  INFO sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 3.012906ms)
2023-08-24T23:10:58.958740Z DEBUG hyper::client::connect::dns: resolving host="sts.us-west-2.amazonaws.com"
2023-08-24T23:10:58.960819Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:dispatch: hyper::client::connect::http: connecting to 54.240.248.204:443
2023-08-24T23:10:58.961759Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:dispatch: hyper::client::connect::http: connected to 54.240.248.204:443
2023-08-24T23:10:58.965424Z DEBUG hyper::proto::h1::io: flushed 2113 bytes
2023-08-24T23:10:58.986612Z DEBUG source{component_kind="source" component_id=syslog_in component_type=file component_name=syslog_in}:file_server: file_source::file_server: event_throughput=168.000/sec bytes_throughput=21.331k/sec ratios={"discovery": 4.567172e-5, "other": 0.00013248506, "reading": 0.0016011929, "sending": 0.0019266325, "sleeping": 0.99629456}
2023-08-24T23:10:58.991154Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-08-24T23:10:58.991171Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1526 bytes)
2023-08-24T23:10:58.991199Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:58.991245Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:dispatch: hyper::client::pool: pooling idle connection for ("https", sts.us-west-2.amazonaws.com)
2023-08-24T23:10:58.991382Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}:assume_role: aws_config::sts::assume_role: obtained assumed credentials access_key_id=Some(Some("key"))
2023-08-24T23:10:58.991398Z  INFO sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 35.926074ms)
2023-08-24T23:10:58.991532Z DEBUG hyper::client::connect::dns: resolving host="kinesis.us-west-2.amazonaws.com"
2023-08-24T23:10:58.992188Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::connect::http: connecting to 34.223.45.90:443
2023-08-24T23:10:58.993889Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::connect::http: connected to 34.223.45.90:443
2023-08-24T23:10:58.997971Z DEBUG hyper::proto::h1::io: flushed 2012 bytes
2023-08-24T23:10:59.025925Z DEBUG hyper::proto::h1::io: parsed 5 headers
2023-08-24T23:10:59.025947Z DEBUG hyper::proto::h1::conn: incoming body is content-length (169 bytes)
2023-08-24T23:10:59.025968Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-08-24T23:10:59.026012Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=1}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::pool: pooling idle connection for ("https", kinesis.us-west-2.amazonaws.com)
2023-08-24T23:10:59.026282Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=2}:request:send_operation{operation="PutRecords" service="kinesis"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: loaded credentials from cache
2023-08-24T23:10:59.026397Z DEBUG sink{component_kind="sink" component_id=to_kinesis component_type=aws_kinesis_streams component_name=to_kinesis}:request{request_id=2}:request:send_operation{operation="PutRecords" service="kinesis"}:dispatch: hyper::client::pool: reuse idle connection for ("https", kinesis.us-west-2.amazonaws.com)

Example Data

CloudTrail event logs for the AssumeRole event, showing connection to the non-FIPS endpoint

    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "sts.us-west-2.amazonaws.com"
    }

Additional Context

This isn’t related to making all the encryption in Vector FIPS compliant, like in this PR https://github.com/vectordotdev/vector/pull/18261 but only for using the FIPS endpoints as provided by the AWS SDK.

References

No response