vector: Getting "Healthcheck failed" error when use aws_cloudwatch_logs component

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

It looks like the new release doesn’t like my CloudWatch logs sink configuration. I’m running Vector on AWS Amazon Linux 2 to push Tailscale logs to CloudWatch logs. This configuration has worked perfectly for months and this morning it broke when the system updated to the latest version of Vector.

systemd[1]: Starting Vector...
vector[4651]: √ Loaded ["/etc/vector/vector.toml"]
vector[4651]: √ Component configuration
vector[4651]: 2023-02-28T14:11:08.697192Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: service error component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch_logs component_name=cloudwatch_logs
vector[4651]: x Health check for "cloudwatch_logs" failed: DescribeLogGroups failed: service error
systemd[1]: vector.service: control process exited, code=exited status=78
systemd[1]: Failed to start Vector.
systemd[1]: Unit vector.service entered failed state.
systemd[1]: vector.service failed.

I’ve confirmed that I can access the CloudWatch logs from that machine using the AWS CLI, so I don’t think it’s a permissions problem. This instance has an IAM Profile assigned to it and is part of a autoscaling group that refreshes daily.

Configuration

data_dir = "/var/lib/vector"

[sources.tailscale_logs]
type = "journald"
current_boot_only = true
include_units = [ "tailscaled" ]

[sinks.cloudwatch_logs]
type = "aws_cloudwatch_logs"
inputs = [ "tailscale_logs" ]
encoding.codec = "json"
create_missing_group = false
create_missing_stream = true
group_name = "/lh/tailscale"
region = "us-east-1"
stream_name = "{{ _BOOT_ID }}"



### Version

vector 0.28.0 (aarch64-unknown-linux-gnu 971c594 2023-02-27)

### Debug Output

[root@ip-10-202-11-141 ~]# VECTOR_LOG=debug vector --config=/etc/vector/vector.toml
2023-02-28T14:23:49.130553Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-02-28T14:23:49.130704Z  INFO vector::app: Log level is enabled. level="debug"
2023-02-28T14:23:49.130778Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-02-28T14:23:49.131824Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-02-28T14:23:49.133168Z DEBUG vector::topology::builder: Building new source. component=tailscale_logs
2023-02-28T14:23:49.133485Z DEBUG vector::topology::builder: Building new sink. component=cloudwatch_logs
2023-02-28T14:23:49.140912Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2023-02-28T14:23:49.141088Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-02-28T14:23:49.141923Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-02-28T14:23:49.141993Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=29
2023-02-28T14:23:49.142028Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-02-28T14:23:49.142049Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-02-28T14:23:49.142764Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-02-28T14:23:49.142804Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-02-28T14:23:49.161569Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T14:23:49.168048Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T14:23:49.184398Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T14:23:49.190595Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T14:23:49.198227Z  INFO vector::topology::running: Running healthchecks.
2023-02-28T14:23:49.198266Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-02-28T14:23:49.198328Z DEBUG vector::topology::running: Configuring outputs for source. component=tailscale_logs
2023-02-28T14:23:49.199130Z DEBUG vector::topology::running: Configuring output for component. component=tailscale_logs output_id=None
2023-02-28T14:23:49.199860Z DEBUG vector::topology::running: Connecting inputs for sink. component=cloudwatch_logs
2023-02-28T14:23:49.199885Z DEBUG vector::topology::running: Adding component input to fanout. component=cloudwatch_logs fanout_id=tailscale_logs
2023-02-28T14:23:49.199913Z DEBUG vector::topology::running: Spawning new source. key=tailscale_logs
2023-02-28T14:23:49.199943Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="journald" component_id="tailscale_logs" group_id="2"
2023-02-28T14:23:49.199981Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="aws_cloudwatch_logs" component_id="cloudwatch_logs" group_id="3"
2023-02-28T14:23:49.200065Z  INFO vector: Vector has started. debug="false" version="0.28.0" arch="aarch64" revision="971c594 2023-02-27"
2023-02-28T14:23:49.200438Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-02-28T14:23:49.200472Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "environment variable not set" }))
2023-02-28T14:23:49.200532Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2023-02-28T14:23:49.200558Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-02-28T14:23:49.200581Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-02-28T14:23:49.200652Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=29
2023-02-28T14:23:49.200685Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-02-28T14:23:49.200705Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-02-28T14:23:49.200730Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-02-28T14:23:49.200750Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-02-28T14:23:49.200788Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump supervisor starting.
2023-02-28T14:23:49.200812Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=the credential provider was not enabled: profile `default` did not contain credential information (CredentialsNotLoaded(CredentialsNotLoaded { source: ProfileDidNotContainCredentials { profile: "default" } }))
2023-02-28T14:23:49.200837Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump starting.
2023-02-28T14:23:49.200967Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=the credential provider was not enabled: $AWS_WEB_IDENTITY_TOKEN_FILE was not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "$AWS_WEB_IDENTITY_TOKEN_FILE was not set" }))
2023-02-28T14:23:49.201004Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=the credential provider was not enabled: ECS provider not configured (CredentialsNotLoaded(CredentialsNotLoaded { source: "ECS provider not configured" }))
2023-02-28T14:23:49.201033Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-02-28T14:23:49.201203Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T14:23:49.201342Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source starting.
2023-02-28T14:23:49.201438Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::topology::builder: Sink starting.
2023-02-28T14:23:49.203055Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T14:23:49.203198Z DEBUG hyper::proto::h1::io: flushed 242 bytes
2023-02-28T14:23:49.202590Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.9762016462284366
2023-02-28T14:23:49.203401Z  INFO source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::sources::journald: Starting journalctl.
2023-02-28T14:23:49.203766Z DEBUG hyper::proto::h1::io: parsed 6 headers
2023-02-28T14:23:49.203780Z DEBUG hyper::proto::h1::conn: incoming body is content-length (56 bytes)
2023-02-28T14:23:49.204628Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T14:23:49.204796Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T14:23:49.205252Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T14:23:49.205377Z DEBUG hyper::proto::h1::io: flushed 307 bytes
2023-02-28T14:23:49.205772Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-02-28T14:23:49.205794Z DEBUG hyper::proto::h1::conn: incoming body is content-length (49 bytes)
2023-02-28T14:23:49.205807Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T14:23:49.205895Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=tailscale-account-iamrTailscaleRole-1UHIDHJGVOPRZ
2023-02-28T14:23:49.205997Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T14:23:49.209084Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T14:23:49.209276Z DEBUG hyper::proto::h1::io: flushed 356 bytes
2023-02-28T14:23:49.209870Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-02-28T14:23:49.209893Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1582 bytes)
2023-02-28T14:23:49.209912Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T14:23:49.210075Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata
2023-02-28T14:23:49.210118Z  INFO send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 11.473896ms)
2023-02-28T14:23:49.210572Z DEBUG hyper::client::connect::dns: resolving host="logs.us-east-1.amazonaws.com"
2023-02-28T14:23:49.213207Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:dispatch: hyper::client::connect::http: connecting to 3.236.94.151:443
2023-02-28T14:23:49.215510Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:dispatch: hyper::client::connect::http: connected to 3.236.94.151:443
2023-02-28T14:23:49.220306Z DEBUG hyper::proto::h1::io: flushed 2032 bytes
2023-02-28T14:23:49.224205Z DEBUG hyper::proto::h1::io: parsed 5 headers
2023-02-28T14:23:49.224228Z DEBUG hyper::proto::h1::conn: incoming body is content-length (229 bytes)
2023-02-28T14:23:49.224258Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T14:23:49.227942Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: service error component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch_logs component_name=cloudwatch_logs
2023-02-28T14:23:54.202597Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.09762018982282575
2023-02-28T14:23:59.203123Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00976204399965035
2023-02-28T14:24:04.202488Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.0009762294231395672
2023-02-28T14:24:09.202878Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00009764796036618658
2023-02-28T14:24:14.203254Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000009789994142420584                  
2023-02-28T14:24:19.202522Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.0000010040230752853468
2023-02-28T14:24:24.203802Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00000012541590831919227
2023-02-28T14:24:29.202319Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00000003756900807056255
2023-02-28T14:24:34.202474Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000000027336171243225506
2023-02-28T14:24:39.202878Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000000029191482223128206
2023-02-28T14:24:44.203012Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00000002811847175015975
2023-02-28T14:24:49.202853Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000000027832640512231938
2023-02-28T14:24:54.203120Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00000003230169365186219
2023-02-28T14:24:59.202423Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00000002825365734690971
2023-02-28T14:25:04.203705Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.0000000440347986008093
2023-02-28T14:25:09.203114Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00000003104662335151475
2023-02-28T14:25:14.203267Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000000031183804446599386



^C2023-02-28T14:25:16.864748Z  WARN source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::sources::journald: Journalctl process stopped.
2023-02-28T14:25:16.865830Z  INFO vector: Vector has stopped.
2023-02-28T14:25:16.867016Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="cloudwatch_logs, tailscale_logs" time_remaining="59 seconds left"
2023-02-28T14:25:16.868251Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source finished normally.
2023-02-28T14:25:16.868287Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump finished normally.
2023-02-28T14:25:16.869797Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump supervisor task finished normally.
2023-02-28T14:25:16.875074Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::topology::builder: Sink finished normally.

Example Data

No response

Additional Context

No response

References

No response

About this issue

Original URL
State: closed
Created a year ago
Reactions: 3
Comments: 17 (7 by maintainers)

Most upvoted comments

We also saw this on a different service:
2023-02-28T05:00:36.992307Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeDeliveryStream failed: service error component_kind="sink" component_type="aws_kinesis_firehose" component_id=firehose_logs component_name=firehose_logs
I can open a separate ticket if you like, but it seems very similar to this.

We were able to track the issue down. Unfortunately it affects all AWS components. We aim to fix it and release it as part of 0.28.1 shortly. For now, I’d recommend using 0.27.1.

jszwedko on Mar 1, 2023

We also saw this on a different service:

2023-02-28T05:00:36.992307Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeDeliveryStream failed: service error component_kind="sink" component_type="aws_kinesis_firehose" component_id=firehose_logs component_name=firehose_logs

I can open a separate ticket if you like, but it seems very similar to this.

shannoncole on Mar 1, 2023

Update! It was actually the upgrade to 0.53, hoping to have a fix by the end of the day.

spencergilbert on Feb 28, 2023

Well, I apparently didn’t run it with the correct flags or something. Running RUST_LOG=aws_smithy_http=trace vector gives me this, which does include the InvalidSignatureException error that @samidalouche is seeing on v0.28.

[root@ip-10-202-11-141 ~]# RUST_LOG=aws_smithy_http=trace vector
2023-02-28T15:12:13.090276Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-02-28T15:12:13.093994Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-02-28T15:12:13.098707Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-02-28T15:12:13.244594Z  INFO vector::topology::running: Running healthchecks.
2023-02-28T15:12:13.247553Z  INFO vector: Vector has started. debug="false" version="0.28.0" arch="aarch64" revision="971c594 2023-02-27"
2023-02-28T15:12:13.247588Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-02-28T15:12:13.252260Z  INFO source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::sources::journald: Starting journalctl.
2023-02-28T15:12:13.296709Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: service error component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch_logs component_name=cloudwatch_logs
2023-02-28T15:12:14.281053Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=1
2023-02-28T15:12:14.292588Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: service error internal_log_rate_limit=true
2023-02-28T15:12:14.292633Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2023-02-28T15:12:14.295423Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ServiceError(ServiceError { source: DescribeLogStreamsError { kind: Unhandled(Unhandled { source: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("fd193320-e53f-4f07-bac5-977e9a32d55b"), extras: {} } }), meta: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("fd193320-e53f-4f07-bac5-977e9a32d55b"), extras: {} } }, raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {"x-amzn-requestid": "fd193320-e53f-4f07-bac5-977e9a32d55b", "content-type": "application/x-amz-json-1.1", "content-length": "229", "date": "Tue, 28 Feb 2023 15:12:13 GMT", "connection": "close"}, body: SdkBody { inner: Once(Some(b"{\"__type\":\"InvalidSignatureException\",\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))) request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-02-28T15:12:14.295490Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2023-02-28T15:13:08.329746Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=6
2023-02-28T15:13:08.342241Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been rate limited 1 times.
2023-02-28T15:13:08.342268Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: service error internal_log_rate_limit=true
2023-02-28T15:13:08.342292Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2023-02-28T15:13:08.342357Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ServiceError(ServiceError { source: DescribeLogStreamsError { kind: Unhandled(Unhandled { source: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("c21db376-0074-4e15-9e13-3f06b4fb74f0"), extras: {} } }), meta: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("c21db376-0074-4e15-9e13-3f06b4fb74f0"), extras: {} } }, raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {"x-amzn-requestid": "c21db376-0074-4e15-9e13-3f06b4fb74f0", "content-type": "application/x-amz-json-1.1", "content-length": "229", "date": "Tue, 28 Feb 2023 15:13:07 GMT", "connection": "close"}, body: SdkBody { inner: Once(Some(b"{\"__type\":\"InvalidSignatureException\",\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))) request_id=2 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-02-28T15:13:08.342412Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=6 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2023-02-28T15:13:21.300819Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=3
2023-02-28T15:13:21.312694Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been rate limited 1 times.
2023-02-28T15:13:21.312723Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: service error internal_log_rate_limit=true
2023-02-28T15:13:21.312744Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2023-02-28T15:13:21.312808Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ServiceError(ServiceError { source: DescribeLogStreamsError { kind: Unhandled(Unhandled { source: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("180c4f5e-16f4-4b07-a398-71c3fdc9418b"), extras: {} } }), meta: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("180c4f5e-16f4-4b07-a398-71c3fdc9418b"), extras: {} } }, raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {"x-amzn-requestid": "180c4f5e-16f4-4b07-a398-71c3fdc9418b", "content-type": "application/x-amz-json-1.1", "content-length": "229", "date": "Tue, 28 Feb 2023 15:13:20 GMT", "connection": "close"}, body: SdkBody { inner: Once(Some(b"{\"__type\":\"InvalidSignatureException\",\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))) request_id=3 error_type="request_failed" stage="sending" internal_log_rate_limit=true
...

rnhurt on Feb 28, 2023

Thanks for checking! I’ll be looking at this today, appreciate the thorough reports.

spencergilbert on Feb 28, 2023

Here are the logs from my successful run using v0.27.1:

2023-02-28T15:04:30.278368Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-02-28T15:04:30.278507Z  INFO vector::app: Log level is enabled. level="debug"
2023-02-28T15:04:30.278569Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-02-28T15:04:30.279299Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-02-28T15:04:30.279904Z DEBUG vector::topology::builder: Building new source. component=tailscale_logs
2023-02-28T15:04:30.280148Z DEBUG vector::topology::builder: Building new sink. component=cloudwatch_logs
2023-02-28T15:04:30.280891Z DEBUG hyper_rustls::config: with_native_roots processed 124 valid and 0 invalid certs
2023-02-28T15:04:30.280936Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2023-02-28T15:04:30.280979Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-02-28T15:04:30.281014Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-02-28T15:04:30.281047Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=29
2023-02-28T15:04:30.281070Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-02-28T15:04:30.281092Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-02-28T15:04:30.281126Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-02-28T15:04:30.281148Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-02-28T15:04:30.291511Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.300081Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.318032Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.326679Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.336450Z  INFO vector::topology::running: Running healthchecks.
2023-02-28T15:04:30.336490Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-02-28T15:04:30.336518Z DEBUG vector::topology::running: Configuring outputs for source. component=tailscale_logs
2023-02-28T15:04:30.336540Z DEBUG vector::topology::running: Configuring output for component. component=tailscale_logs output_id=None
2023-02-28T15:04:30.336558Z DEBUG vector::topology::running: Connecting inputs for sink. component=cloudwatch_logs
2023-02-28T15:04:30.336575Z DEBUG vector::topology::running: Adding component input to fanout. component=cloudwatch_logs fanout_id=tailscale_logs
2023-02-28T15:04:30.336620Z DEBUG vector::topology::running: Spawning new source. key=tailscale_logs
2023-02-28T15:04:30.336653Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="journald" component_id="tailscale_logs" group_id="2"
2023-02-28T15:04:30.336692Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="aws_cloudwatch_logs" component_id="cloudwatch_logs" group_id="3"
2023-02-28T15:04:30.336750Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump supervisor starting.
2023-02-28T15:04:30.336765Z  INFO vector: Vector has started. debug="false" version="0.27.1" arch="aarch64" revision="19a51f2 2023-02-22"
2023-02-28T15:04:30.336798Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump starting.
2023-02-28T15:04:30.336827Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-02-28T15:04:30.337059Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source starting.
2023-02-28T15:04:30.337137Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::topology::builder: Sink starting.
2023-02-28T15:04:30.336998Z DEBUG aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:30.337310Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=environment variable not set
2023-02-28T15:04:30.337358Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2023-02-28T15:04:30.337383Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-02-28T15:04:30.337402Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-02-28T15:04:30.337436Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=29
2023-02-28T15:04:30.337458Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-02-28T15:04:30.337475Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-02-28T15:04:30.337498Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-02-28T15:04:30.337516Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-02-28T15:04:30.337551Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=profile `default` did not contain credential information
2023-02-28T15:04:30.337571Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=$AWS_WEB_IDENTITY_TOKEN_FILE was not set
2023-02-28T15:04:30.337599Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=ECS provider not configured
2023-02-28T15:04:30.337617Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-02-28T15:04:30.337747Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:send_operation{operation="get-token" service="imds"}: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T15:04:30.337926Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.09498622198941609
2023-02-28T15:04:30.337949Z  INFO source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::sources::journald: Starting journalctl.
2023-02-28T15:04:30.338027Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:send_operation{operation="get-token" service="imds"}: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T15:04:30.338191Z DEBUG hyper::proto::h1::io: flushed 242 bytes
2023-02-28T15:04:30.338679Z DEBUG hyper::proto::h1::io: parsed 6 headers
2023-02-28T15:04:30.338694Z DEBUG hyper::proto::h1::conn: incoming body is content-length (56 bytes)
2023-02-28T15:04:30.338716Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.338873Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T15:04:30.339019Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T15:04:30.339074Z DEBUG hyper::proto::h1::io: flushed 307 bytes
2023-02-28T15:04:30.339379Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-02-28T15:04:30.339390Z DEBUG hyper::proto::h1::conn: incoming body is content-length (49 bytes)
2023-02-28T15:04:30.339396Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.339452Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=tailscale-account-iamrTailscaleRole-1UHIDHJGVOPRZ
2023-02-28T15:04:30.339497Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T15:04:30.339602Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T15:04:30.339661Z DEBUG hyper::proto::h1::io: flushed 356 bytes
2023-02-28T15:04:30.340003Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-02-28T15:04:30.340014Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1582 bytes)
2023-02-28T15:04:30.340025Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.340105Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata
2023-02-28T15:04:30.340323Z DEBUG hyper::client::connect::dns: resolving host="logs.us-east-1.amazonaws.com"
2023-02-28T15:04:30.341742Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}: hyper::client::connect::http: connecting to 44.202.79.205:443
2023-02-28T15:04:30.344505Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}: hyper::client::connect::http: connected to 44.202.79.205:443
2023-02-28T15:04:30.347936Z DEBUG hyper::proto::h1::io: flushed 2032 bytes
2023-02-28T15:04:30.372791Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:30.373075Z DEBUG hyper::proto::h1::conn: incoming body is content-length (240 bytes)
2023-02-28T15:04:30.373225Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.373339Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:30.373707Z  INFO vector::topology::builder: Healthcheck passed.
2023-02-28T15:04:31.414132Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:31.414250Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:31.414275Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=100
2023-02-28T15:04:31.414349Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.414389Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.414464Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.414549Z DEBUG hyper::proto::h1::io: flushed 2084 bytes
2023-02-28T15:04:31.428363Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.428392Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17 bytes)
2023-02-28T15:04:31.428550Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:31.428598Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.428676Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Provided stream does not exist; creating a new one.
2023-02-28T15:04:31.428758Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "signingRegion": String("us-east-1")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.428811Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="CreateLogStream" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.428912Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="CreateLogStream" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.429012Z DEBUG hyper::proto::h1::io: flushed 2065 bytes
2023-02-28T15:04:31.471472Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.471502Z DEBUG hyper::proto::h1::conn: incoming body is empty
2023-02-28T15:04:31.471690Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="CreateLogStream" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.471752Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Stream created. name=f431275382ea4a47885bce73bceb6781
2023-02-28T15:04:31.471810Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "signingRegion": String("us-east-1")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.471854Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.471944Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.472043Z DEBUG hyper::proto::h1::io: flushed 2084 bytes
2023-02-28T15:04:31.485395Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.485424Z DEBUG hyper::proto::h1::conn: incoming body is content-length (226 bytes)
2023-02-28T15:04:31.485480Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:31.485534Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.485644Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Stream found. stream=Some("f431275382ea4a47885bce73bceb6781")
2023-02-28T15:04:31.485663Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Putting logs. token=None
2023-02-28T15:04:31.486201Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.486257Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.486445Z DEBUG hyper::client::connect::dns: resolving host="logs.us-east-1.amazonaws.com"
2023-02-28T15:04:31.486797Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::connect::http: connecting to 44.202.79.192:443
2023-02-28T15:04:31.489894Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::connect::http: connected to 44.202.79.192:443
2023-02-28T15:04:31.494727Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494789Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494845Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494880Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494917Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494949Z DEBUG hyper::proto::h1::io: flushed 13718 bytes
2023-02-28T15:04:31.554980Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.555013Z DEBUG hyper::proto::h1::conn: incoming body is content-length (80 bytes)
2023-02-28T15:04:31.555230Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:31.555279Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.555372Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49637405755031694129084131660689759327850337978875380530")
2023-02-28T15:04:35.338094Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.009927846799178427
2023-02-28T15:04:36.088027Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: tower::buffer::worker: service.ready=true processing request2023-02-28T15:04:36.088132Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:36.088157Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=2
2023-02-28T15:04:36.088265Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "signingRegion": String("us-east-1")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:36.088336Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:36.088436Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:36.088548Z DEBUG hyper::proto::h1::io: flushed 4061 bytes
2023-02-28T15:04:36.105390Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:36.105418Z DEBUG hyper::proto::h1::conn: incoming body is content-length (80 bytes)
2023-02-28T15:04:36.105450Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:36.105501Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:36.105575Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49628116737211695456459775456880034164813271731714131042")
2023-02-28T15:04:40.338219Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000992803759438506
2023-02-28T15:04:42.057163Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:42.057247Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:42.057271Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=1
2023-02-28T15:04:42.057389Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:42.057441Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:42.057535Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:42.057673Z DEBUG hyper::proto::h1::io: flushed 3110 bytes
2023-02-28T15:04:42.078286Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:42.078320Z DEBUG hyper::proto::h1::conn: incoming body is content-length (80 bytes)
2023-02-28T15:04:42.078364Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:42.078420Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:42.078488Z  INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49636311869716377971442486284532261404947306999409803410")
2023-02-28T15:04:45.338839Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00010497450932245926
2023-02-28T15:04:50.338313Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000010566938244949766
^C2023-02-28T15:04:53.560864Z  INFO vector: Vector has stopped.
2023-02-28T15:04:53.561028Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source finished normally.
2023-02-28T15:04:53.561054Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump finished normally.
2023-02-28T15:04:53.561104Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump supervisor task finished normally.
2023-02-28T15:04:53.561139Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::topology::builder: Sink finished normally.

rnhurt on Feb 28, 2023

I have encountered the same error (https://github.com/vectordotdev/vector/discussions/16627)

I reverted to this version, and the problem went away.

yum install -y vector-0.27.1-1

samidalouche on Feb 28, 2023

@rnhurt could you try v0.27.1 as well? We think this may be a regression in v0.28.0.

jszwedko on Feb 28, 2023

It looks like some sort of credentials issue, so I removed the ~/.aws directory to see if it was getting confused somewhere. The error is still present and there was no change in behaviour.

rnhurt on Feb 28, 2023

spencergilbert on Feb 28, 2023