elasticsearch-rails: Sporadic error: The bulk request must be terminated by a newline

Hi,

I’m importing 500 records in rails using the bulk method, however, on and off I get the following error:

Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}

My environment: Elasticsearch version: 6.5.4 Rails: 5.1.6

elasticsearch (6.1.0)
elasticsearch-api (6.1.0)
elasticsearch-model (6.0.0)
elasticsearch-rails (6.0.0)
elasticsearch-transport (6.1.0)

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 15 (6 by maintainers)

Most upvoted comments

It’s very much Ruby 2.6, which seems to cut off a portion of the body. See the report in https://github.com/ruby/ruby/pull/2058#issuecomment-452193105, where Elasticsearch bulk payloads are specifically mentioned.

Here’s a detailed information about the bug: https://mensfeld.pl/2019/01/exploring-a-critical-netprotocol-issue-in-ruby-2-6-0p0-and-how-it-can-lead-to-a-security-problem/

Can you try on Ruby 2.6.1, which should have patched the bug?

karmi on Jan 31, 2019

@nerfologist, many thanks for the aditional information! I’m a bit mystified about the error, since both elasticsearch-ruby and elasticsearch-rails have quite a bit of unit and integration tests for bulk, which I haven’t seen failing, but I see we don’t run those against Ruby 2.6, that might indeed be the culprit. I’ll try to run them locally and see what’t going on.

karmi on Jan 30, 2019