logstash-output-opensearch: [BUG] - logstash-output-opensearch complains for cluster UUID while connecting to Amazon OpenSearch Serverless
Describe the bug
Running the plugin gives following error when connecting with Amazon OpenSearch Serverless.
[2023-01-11T17:39:47,431][ERROR][logstash.outputs.opensearch][main] Unable to retrieve OpenSearch cluster uuid {:message=>"undefined method []’ for nil:NilClass", :exception=>NoMethodError,`
This is an error message which is thrown but ingestion is working fine and I am able to see the document is OpenSearch Serverless collections.
To Reproduce Steps to reproduce the behavior:
- Setup and install logstash with logstash-output-opensearch plugin
- Setup OpenSearch Serverless collections
- Use below sample code to run the logstash, save file as logstash-example.conf
input {
stdin { }
}
output {
opensearch {
ecs_compatibility => disabled
index => "logstash-sample"
hosts => "https://<add-your-collection-url>:443"
auth_type => {
type => 'aws_iam'
aws_access_key_id => '<secret-key>'
aws_secret_access_key => '<secret-access-key>'
region => '<region>'
service_name => 'aoss'
}
legacy_template => false
default_server_major_version => 2
}
}
-
Run the logstash as:
<path/to/your/logstash/dir>/bin/logstash -f logstash-example.conf -
See the error as mentioned above
Expected behavior There should not be any error and logstash should be able to index data to OpenSearch Serverless collections
Host/Environment (please complete the following information):
- OS: MacOS
- Version 12.6.2
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 17 (7 by maintainers)
Commits related to this issue
- app/vmlogs: Replace elasticsearch plugin with opensearch plugin of logstash ### Describe your changes 1. When I used the opensearch plugin in the reference example, I found a bug that caused memory ... — committed to VictoriaMetrics/VictoriaMetrics by Eason1118 2 years ago
- app/vmlogs: Replace elasticsearch plugin with opensearch plugin of logstash ### Describe your changes 1. When I used the opensearch plugin in the reference example, I found a bug that caused memory ... — committed to VictoriaMetrics/VictoriaMetrics by Eason1118 2 years ago
- app/vmlogs: Replace elasticsearch plugin with opensearch plugin of logstash ### Describe your changes 1. When I used the opensearch plugin in the reference example, I found a bug that caused memory ... — committed to truepele/VictoriaMetrics by Eason1118 2 years ago
- app/vmlogs: Replace elasticsearch plugin with opensearch plugin of logstash ### Describe your changes 1. When I used the opensearch plugin in the reference example, I found a bug that caused memory ... — committed to truepele/VictoriaMetrics by Eason1118 2 years ago
@chadmyers I can confirm that logstash-output-opensearch works with AOSS and as @jomafe says this error is just a warning that can be ignored.
I basically followed the instructions in https://opensearch.org/docs/latest/tools/logstash/index/.
AWS Credentials and Endpoint
Latest Version of the Plugin
Built from source to make sure I have the latest.
Install LogStash
Install Plugin
Run Logstash
Type a few “hello world” on the console
Check Docs
Daniel and Jordi - I haven’t had a chance to see if Logstash + AOSS is working for us yet but I wanted to thank you for your time in investigating this and I’m encouraged to see the progress you all have made.
I’ve added an IAM policy to the user allowing write actions for OpenSearch Serverless service and it started working. I thought the Data access policies from the OpenSearch Serverless service console were enough.
Thank you.