logstash-output-opensearch: [BUG] - logstash-output-opensearch complains for cluster UUID while connecting to Amazon OpenSearch Serverless

Describe the bug Running the plugin gives following error when connecting with Amazon OpenSearch Serverless. [2023-01-11T17:39:47,431][ERROR][logstash.outputs.opensearch][main] Unable to retrieve OpenSearch cluster uuid {:message=>"undefined method []’ for nil:NilClass", :exception=>NoMethodError,`

This is an error message which is thrown but ingestion is working fine and I am able to see the document is OpenSearch Serverless collections.

To Reproduce Steps to reproduce the behavior:

  1. Setup and install logstash with logstash-output-opensearch plugin
  2. Setup OpenSearch Serverless collections
  3. Use below sample code to run the logstash, save file as logstash-example.conf
input {
   stdin { }
}
output {
  opensearch {
    ecs_compatibility => disabled
    index => "logstash-sample"
    hosts => "https://<add-your-collection-url>:443"
    auth_type => {
      type => 'aws_iam'
      aws_access_key_id => '<secret-key>'
      aws_secret_access_key => '<secret-access-key>'
      region => '<region>'
      service_name => 'aoss'
    }
    legacy_template => false
    default_server_major_version => 2
  }
}

  1. Run the logstash as: <path/to/your/logstash/dir>/bin/logstash -f logstash-example.conf

  2. See the error as mentioned above

Expected behavior There should not be any error and logstash should be able to index data to OpenSearch Serverless collections

Host/Environment (please complete the following information):

  • OS: MacOS
  • Version 12.6.2

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 17 (7 by maintainers)

Commits related to this issue

Most upvoted comments

@chadmyers I can confirm that logstash-output-opensearch works with AOSS and as @jomafe says this error is just a warning that can be ignored.

I basically followed the instructions in https://opensearch.org/docs/latest/tools/logstash/index/.

AWS Credentials and Endpoint

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...
export ENDPOINT=https://.....us-west-2.aoss.amazonaws.com
export AWS_REGION=us-west-2
export SERVICE=es

Latest Version of the Plugin

Built from source to make sure I have the latest.

cd ~/source/opensearch-project/logstash-output-opensearch/upstream-logstash-output-opensearch
./build.sh 
/Users/dblock/source/opensearch-project/logstash-output-opensearch/upstream-logstash-output-opensearch/builds/logstash-output-opensearch-2.0.2-arm64-darwin-21.gem

Install LogStash

wget https://artifacts.elastic.co/downloads/logstash/logstash-8.12.1-darwin-x86_64.tar.gz
tar vfxz logstash-8.12.1-darwin-x86_64.tar.gz 
cd logstash-8.12.1

Install Plugin

bin/logstash-plugin install /Users/dblock/source/opensearch-project/logstash-output-opensearch/upstream-logstash-output-opensearch/builds/logstash-output-opensearch-2.0.2-arm64-darwin-21.gem
Using bundled JDK: /Users/dblock/source/opensearch-project/logstash-issue-186/logstash-8.12.1/jdk.app/Contents/Home
warning: ignoring JAVA_TOOL_OPTIONS=-Dlog4j2.formatMsgNoLookups=true
Validating /Users/dblock/source/opensearch-project/logstash-output-opensearch/upstream-logstash-output-opensearch/builds/logstash-output-opensearch-2.0.2-arm64-darwin-21.gem
Installing logstash-output-opensearch
Installation successful

Run Logstash

 bin/logstash -e "input { stdin { } } output {
   opensearch {
    ecs_compatibility => disabled
    index  => 'logstash-logs-%{+YYYY.MM.dd}'
    hosts => ['${ENDPOINT}:443']
    auth_type => {
      type => 'aws_iam'
      aws_access_key_id => '${AWS_ACCESS_KEY_ID}'
      aws_secret_access_key => '${AWS_SECRET_ACCESS_KEY}'
      session_token => '${AWS_SESSION_TOKEN}'
      region => '${AWS_REGION}'
      service_name => '${SERVICE}'
    }
    legacy_template => false
    default_server_major_version => 2
  }
 }"

Type a few “hello world” on the console

Check Docs

$ awscurl --region=$AWS_REGION --service=$SERVICE $ENDPOINT/_cat/indices?v
health status index                                       uuid                 pri rep docs.count docs.deleted store.size pri.store.size
              logstash-logs-2024.02.13                    0Zf2oo0BiehDHTSpsLmp                  0            0         0b             0b
$ awscurl --region=$AWS_REGION --service=$SERVICE $ENDPOINT/logstash-logs-2024.02.13/_count  | jq ".count"
18

Daniel and Jordi - I haven’t had a chance to see if Logstash + AOSS is working for us yet but I wanted to thank you for your time in investigating this and I’m encouraged to see the progress you all have made.

I’ve added an IAM policy to the user allowing write actions for OpenSearch Serverless service and it started working. I thought the Data access policies from the OpenSearch Serverless service console were enough.

Thank you.