VictoriaLogs: Victorialogs: Errors when writing via JSON stream API after version 1.1.0
Is your question request related to a specific component?
Victorialogs
Describe the question in detail
I encountered an issue when writing logs using the JSON stream API. Here are the details:
Environment:
Version: Starting from v1.1.0 Previous version (v1.0.0): Worked perfectly without issues
Problem:
Starting from version 1.1.0, writing logs via the JSON stream API results in the following errors:
2025-01-07T08:04:47.139Z info VictoriaMetrics/lib/logstorage/log_rows.go:160 ignoring too long log entry with the estimated size 3247862 bytes, since it exceeds the limit 2097152
2025-01-07T08:04:47.148Z error VictoriaMetrics/app/vlinsert/jsonline/jsonline.go:58 jsonline: cannot read line #16740 in /jsonline request: remoteAddr="11.151.222.165:37454", requestURI="/insert/jsonline?_stream_fields=stream&_time_field=date&_msg_field=log.message": cannot read the next line: unexpected EOF
Questions:
Is this issue caused by the logs being truncated when their size exceeds a certain limit? Are there any recommended solutions or configurations to avoid these errors?
Troubleshooting docs
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 19 (11 by maintainers)
Commits related to this issue
- lib/logstorage: improve error message by adding a link with the explanation why VictoriaLogs ignores logs with the size exceeding 2MB Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: improve error message by adding a link with the explanation why VictoriaLogs ignores logs with the size exceeding 2MB Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: improve error message by adding a link with the explanation why VictoriaLogs ignores logs with the size exceeding 2MB Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues... — committed to G-Research/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: improve error message by adding a link with the explanation why VictoriaLogs ignores logs with the size exceeding 2MB Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: improve error message by adding a link with the explanation why VictoriaLogs ignores logs with the size exceeding 2MB Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues... — committed to truepele/VictoriaMetrics by valyala 2 years ago
AFAIK there’s no such plan yet. @func25 could you help evaluate whether this would be a good addition?
My writing batches data to a certain size before sending it. A 5MB batch and a 10MB batch are essentially the same in terms of data; the 10MB batch is just a combination of two 5MB batches. Below is the main code for my writing. Could you please help me check if there are any issues?