fluentd-kubernetes-daemonset: Can't connect to AWS ES

I have an AWS ES instance protected by an IAM access policy. I added all my cluster nodes to said policy (by IP). Since I’m not using X-Pack, I figured I had to set the values for FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD to blank in the DaemonSet config file (fluentd-daemonset-elasticsearch.yml). Still, fluentd was unable to connect to ES:

2017-06-13 17:18:32 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-06-13 17:19:02 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:2ac3e37a42a8"

To confirm that my IAM policy was correctly set up, I curl’d my ES host from a fluentd pod and I got a successful response.

/home/fluent # curl https://<redacted>.us-west-2.es.amazonaws.com:443
{
  "name" : "Crimson Cavalier",
  "cluster_name" : "<redacted>",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "72aa8010df1a4fc849da359c9c58acba6c4d9518",
    "build_timestamp" : "2016-11-14T15:59:50Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

The solution I found was to manually edit the /fluentd/etc/fluent.conf file, remove the user and password lines and start the fluentd service again.

-   user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
-   password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"

I’m not sure what’s the best way to get this into the code though.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 17 (7 by maintainers)

Most upvoted comments

Hey, I had this same error:

fluentd Elasticsearch::Transport::Transport::Errors::Forbidden error=[403]

It was a problem with the AWS access keys. I am using the docker image cheungpat/fluentd-elasticsearch-aws:1.22 and getting the values for AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID from a k8s secret like this:

- name: AWS_ACCESS_KEY_ID
  valueFrom:
    secretKeyRef:
      name: logging-iam-user-staging
      key: aws_access_key_id
- name: AWS_SECRET_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: logging-iam-user-staging
      key: aws_secret_access_key

The problem for me was that there was quotes around the access key.

It’s worth noting that if you just see the error=[403] message with no further details then authentication wasn’t successful. In this scenario it’s nothing to do with IAM policy as it hasn’t even got that far. If you have a problem with your IAM policy you should get a more specific error like this:

2017-10-27 13:49:23 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-27 13:49:23 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] {\"Message\":\"User: arn:aws:iam::99999999999:user/staging-logs-es is not authorized to perform: es:ESHttpPost on resource: staging-logs\"}

max-rocket-internet on Oct 29, 2017

@ixai: hey, I’m also using aws es and I have it working using this: https://hub.docker.com/r/fxinnovation/aws-es-proxy/

It is a small authentication proxy to whom you can give the account credentials for accessing AWS ES and inside your cluster you define this service a being elastic search. For fluentd, heapster, etc it is completely transparent and it leverages AWS IAM auth for you. But you can’t use the FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD as these are elasticsearch specific.

By the way, I have 5 of those proxies launched otherwhize you might get throttle by AWS…

zopanix on Aug 15, 2017