vector: Several AWS integrations are broken in v0.46.0

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hello,

I updated vector to v0.46.0 to use it with ‘aws_kinesis_firehose’ and when I validated the service I got an error that said the following:

'2025-04-09T08:05:08.405300Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeDeliveryStream failed: dispatch failure component_kind="sink" component_type="aws_kinesis_firehose" component_id=firehose_stream
x Health check for "firehose_stream" failed: DescribeDeliveryStream failed: dispatch failure'

The moment I downgrade vector to v0.45.0 the validation returns a ‘OK’ health check.

Configuration

sinks:
  firehose_stream:
    auth: {access_key_id: REDACTED, secret_access_key: REDACTED }
    batch: {max_bytes: 250000, max_events: 500, timeout_secs: 1}
    buffer: {max_size: 268435488, type: disk, when_full: block}
    encoding: {codec: json}
    healthcheck: {enabled: true}
    inputs: [journald_aws]
    region: eu-west-3
    stream_name: redacted
    type: aws_kinesis_firehose

Version

vector 0.46.0 (x86_64-unknown-linux-gnu a7429e2 2025-04-08 16:05:37.911631718)

Debug Output


Example Data

No response

Additional Context

No response

References

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 14
  • Comments: 24 (13 by maintainers)

Most upvoted comments

We’re running into this as well with the aws_kinesis_streams component (so it is not limited to the aws_kinesis_firehose component); upgrade to 0.46.0 results in healthcheck failure:

vector[52500]: 2025-04-08T23:00:11.232539Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeStream failed: dispatch failure component_kind="sink" component_type="aws_kinesis_streams" component_id=splunk

And failure to push to the Kinesis stream:

vector[52500]: 2025-04-08T23:00:11.250876Z  WARN sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=dispatch failure internal_log_rate_limit=true
vector[52500]: 2025-04-08T23:00:12.010343Z  WARN sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.

Downgrading to 0.45.0 resolves this for us as well.

Sink config looks like:

type: aws_kinesis_streams
inputs:
  - add_keys_for_splunk
buffer:
  type: 'disk'
  max_size: 1073741824 # 1GiB
encoding:
  codec: json
auth:
  assume_role: 'arn:aws:iam::{ACCOUNT_NO}:role/{ROLE_NAME}'
  region: 'us-east-1'
stream_name: '{STREAM_NAME}'

I briefly enabled TRACE logging; there’s only one sequence that seems relevant (specifically the encountered orchestrator error message):

2025-04-08T23:00:12.008936Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:apply_configuration: aws_smithy_runtime_api::client::runtime_plugin: applying client runtime plugins
2025-04-08T23:00:12.008974Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:apply_configuration: aws_smithy_runtime::client::interceptors: running client `read_before_execution` interceptors
2025-04-08T23:00:12.008985Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:apply_configuration: aws_smithy_runtime_api::client::runtime_plugin: applying operation runtime plugins
2025-04-08T23:00:12.008993Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:apply_configuration: aws_smithy_runtime::client::interceptors: running operation `read_before_execution` interceptors
2025-04-08T23:00:12.009005Z DEBUG sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:apply_configuration: aws_smithy_runtime::client::orchestrator: timeout settings for this operation: TimeoutConfig { connect_timeout: Disabled, read_timeout: Disabled, operation_timeout: Disabled, operation_attempt_timeout: Disabled }
2025-04-08T23:00:12.009016Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}: aws_smithy_runtime::client::orchestrator: runtime_components=RuntimeComponents {REDACTED}
2025-04-08T23:00:12.009200Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}: aws_smithy_runtime::client::orchestrator: operation_timeout_config=MaybeTimeoutConfig { sleep_impl: Some(SharedAsyncSleep(TokioSleep)), timeout: None, timeout_kind: Operation }
2025-04-08T23:00:12.009208Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:try_op: aws_smithy_runtime::client::interceptors: running `modify_before_serialization` interceptors
2025-04-08T23:00:12.009213Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:try_op: aws_smithy_runtime::client::interceptors: running `read_before_serialization` interceptors
2025-04-08T23:00:12.009217Z DEBUG sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2025-04-08T23:00:12.010274Z DEBUG sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2025-04-08T23:00:12.010287Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:try_op: aws_smithy_runtime::client::interceptors: running `read_after_serialization` interceptors
2025-04-08T23:00:12.010294Z DEBUG sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:try_op: aws_smithy_runtime::client::orchestrator: encountered orchestrator error; halting
2025-04-08T23:00:12.010298Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:try_op: aws_smithy_runtime_api::client::interceptors::context: orchestrator is transitioning to the 'failure' phase from the 'BeforeTransmit' phase
2025-04-08T23:00:12.010304Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:finally_op: aws_smithy_runtime::client::interceptors: running `modify_before_completion` interceptors
2025-04-08T23:00:12.010308Z TRACE sink{component_kind="sink" component_id=splunk component_type=aws_kinesis_streams}:request{request_id=1}:request:invoke{rpc.service="kinesis" rpc.method="PutRecords"}:finally_op: aws_smithy_runtime::client::interceptors: running `read_after_execution` interceptors

There’s a line in the block that I’ve redacted (RuntimeComponents) which has a JSON-like blob that’s nearly 1600 lines when unfurled/formatted; I’ll need to review that carefully before providing it, but wanted to get the above notes into the conversation while I do that.

I’ve upgraded a few of our deployments to 0.46.1 and can confirm I no longer see these errors and logs are flowing as-expected through the aws_kinesis_streams component. Thanks for quick fix!

Ignore my comment; i missed the part where you bumped the deps. Had been looking at just after you reverted the endpoint PR.

Thank you for the custom build, I got it working with kineses, pod identity (aws eks), and assuming a role

@pront Thanks for the info & link. I can confirm that the fix candidate is working for me:

$ /usr/bin/vector --version
vector 0.47.0 (aarch64-unknown-linux-gnu cd65a57 2025-04-10 19:44:41.598162462)
$ rpm -q vector --info
Name        : vector
Version     : 0.47.0.custom.cd65a57
Release     : 1
Architecture: aarch64
Install Date: Thu Apr 10 21:24:50 2025
Group       : Applications/System
Size        : 123749074
License     : MPL-2.0
Signature   : (none)
Source RPM  : vector-0.47.0.custom.cd65a57-1.src.rpm
Build Date  : Thu Apr 10 20:24:06 2025
Build Host  : 10ad5ec296e7
URL         : https://vector.dev
Summary     : A lightweight and ultra-fast tool for building observability pipelines
Description :
A lightweight and ultra-fast tool for building observability pipelines

You should be able to just run the following in the repo root:

cargo run --config path/to/config.yaml

Thanks for this info. I was able to install cargo+git and then run:

cargo run -- --config /etc/vector/conf.d/journald.yaml

Unfortunately, my server had too little memory to compile the syn crate, so it eventually crashed, and I will have to try again with a larger server.

@pront I’ll try that sha now. I rebased my other PR (locally) on top of your branch and validated it worked yesterday, but will explicitly test your branch to minimize variables.

Config I’m testing with:

sources:
  vector_internal_metrics:
    type: "internal_metrics"
    scrape_interval_secs: 30
    tags:
      host_key: "host"

  primary_logs:
    type: "aws_s3"
    region: "us-west-2"
    compression: "gzip"
    decoding:
      codec: "json"
    sqs:
      queue_url: "https://sqs.us-west-2.amazonaws.com/...."
      client_concurrency: 10
      visibility_timeout_secs: 60
      max_number_of_messages: 1

sinks:
  primary_blackhole:
    type: blackhole
    inputs:
      - primary_logs
    print_interval_secs: 20
./target/debug/vector --version
vector 0.47.0-custom-1b1500307 (aarch64-apple-darwin debug=full)

Works as expected.

2025-04-10T14:35:45.358711Z  INFO vector::topology::builder: Healthcheck passed.
2025-04-10T14:35:45.360066Z  INFO vector: Vector has started. debug="true" version="0.47.0" arch="aarch64" revision=""
2025-04-10T14:35:45.360369Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2025-04-10T14:35:45.366725Z  INFO vector::sinks::blackhole::sink: Collected events. events=0 raw_bytes_collected=0
2025-04-10T14:35:45.382930Z  INFO source{component_kind="source" component_id=primary_logs component_type=aws_s3}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("aws-creds ** arguments redacted **"), chain: [] }