logstash-output-amazon_es: Attempted to resurrect connection to dead ES instance, but got an error.
Hi, I have a problem with connection to the AWS ES service from logstash installed on AWS EC2. I have two instances [logstash01, logstash02] with the same configuration:
- logstash 6.8.10
- logstash-output-amazon_es (6.4.2)
- AWS ES version 6.2
The first instance is working fine but on the second one there are many warnings:
[2020-07-03T09:52:46,924][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://{my-aws-es-service}.{aws-region}.es.amazonaws.com:443/, :path=>"/"} [2020-07-03T09:52:46,927][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://{my-aws-es-service}.{aws-region}.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://{my-aws-es-service}.{aws-region}.es.amazonaws.com:443/'"}
Both EC2 instances have been assigned to the same IAM Role with full access to ES, they are both in the same VPC, subnet and their security groups have the same rules. I am able to curl ES service from both EC2s.
The configuration of amazon_es output plugin is:
output { amazon_es { hosts => ["{my-aws-es-service}.{aws-region}.es.amazonaws.com"] region => "{aws-region}" index => "logstash-%{[index]}-%{+YYYY.MM.dd}" } }
I have had this issue for couple of days and not able to resolve. Any help would be appreciated.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 13
- Comments: 15
I got this issue as well, it was due to the user lacking privileges to the root path. By adding cluster-level monitor privileges the problem was gone. Not necessarily the same issue, but the error message was the same.
UPDATE: I had been parsing local CSV files only for triage till now, so it was first time I was getting a syslog file. Uncommenting the line
http.port: 9200in/etc/elasticsearch/elasticsearch.ymlworked for me. Now no more logs about dead elasticsearch instance and logs are flowing in. Additionally make sure that the linenetwork.host: localhostis also uncommented.