VictoriaLogs: VictoriaLogs Bug: Read Timeout with stream_context
Describe the bug
The LogQL: _time:5m MarsSendServiceImpl , I can get result so quickly by UI , but append | stream_context before 4, the response is error, e.g. Read Timeout, as follow:
Other issue: https://github.com/VictoriaMetrics/victorialogs-datasource/issues/117
I think the result should response quicky and not timeout
To Reproduce
As above
Version
release 1.0.0
Logs
No response
Screenshots
No response
Used command-line flags
No response
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (12 by maintainers)
Commits related to this issue
- lib/logstorage: optimize `stream_context` pipe over log streams with tens of millions of logs `stream_context` is implemented in the way, which needs scanning all the logs for the selected log stream... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: add an ability to change the time window for searching for surrounding logs in the `stream_context` pipe Thanks to @worker24h for the idea at https://github.com/VictoriaMetrics/Victor... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: stream_context pipe: reduce the amounts of surrounding logs to check Do not check surrounding logs before the selected log if `after N` in set, and do not check logs after the selecte... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: optimize `stream_context` pipe over log streams with tens of millions of logs `stream_context` is implemented in the way, which needs scanning all the logs for the selected log stream... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: add an ability to change the time window for searching for surrounding logs in the `stream_context` pipe Thanks to @worker24h for the idea at https://github.com/VictoriaMetrics/Victor... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: stream_context pipe: reduce the amounts of surrounding logs to check Do not check surrounding logs before the selected log if `after N` in set, and do not check logs after the selecte... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: optimize `stream_context` pipe over log streams with tens of millions of logs `stream_context` is implemented in the way, which needs scanning all the logs for the selected log stream... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: add an ability to change the time window for searching for surrounding logs in the `stream_context` pipe Thanks to @worker24h for the idea at https://github.com/VictoriaMetrics/Victor... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: stream_context pipe: reduce the amounts of surrounding logs to check Do not check surrounding logs before the selected log if `after N` in set, and do not check logs after the selecte... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: optimize `stream_context` pipe over log streams with tens of millions of logs `stream_context` is implemented in the way, which needs scanning all the logs for the selected log stream... — committed to G-Research/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: add an ability to change the time window for searching for surrounding logs in the `stream_context` pipe Thanks to @worker24h for the idea at https://github.com/VictoriaMetrics/Victor... — committed to G-Research/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: stream_context pipe: reduce the amounts of surrounding logs to check Do not check surrounding logs before the selected log if `after N` in set, and do not check logs after the selecte... — committed to G-Research/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: optimize `stream_context` pipe over log streams with tens of millions of logs `stream_context` is implemented in the way, which needs scanning all the logs for the selected log stream... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: add an ability to change the time window for searching for surrounding logs in the `stream_context` pipe Thanks to @worker24h for the idea at https://github.com/VictoriaMetrics/Victor... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: stream_context pipe: reduce the amounts of surrounding logs to check Do not check surrounding logs before the selected log if `after N` in set, and do not check logs after the selecte... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: optimize `stream_context` pipe over log streams with tens of millions of logs `stream_context` is implemented in the way, which needs scanning all the logs for the selected log stream... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: add an ability to change the time window for searching for surrounding logs in the `stream_context` pipe Thanks to @worker24h for the idea at https://github.com/VictoriaMetrics/Victor... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: stream_context pipe: reduce the amounts of surrounding logs to check Do not check surrounding logs before the selected log if `after N` in set, and do not check logs after the selecte... — committed to truepele/VictoriaMetrics by valyala 2 years ago
FYI, the commit bddb0e369f7b815046c3935d3aee6134bb35f889 improves performance of the
stream_contextpipe when it is applied to logs, which belong to log streams with big number of logs (e.g. more than a million of logs). This commit will be included in the next release. In the mean time it is possible to build VictoriaLogs from source code according to these instructions and verifying whether this improves performance of thestream_contextpipe.This is a good idea! The commit 08af80ebe02cb0d299e7fc6e35cb6c48469fdcbd sets the time window for searching for surrounding logs in the log stream to one hour. This time window can be changed via
time_window <duration>option in thestream_contextpipe. For example, if you want to search for surrounding logs on a one year window, then the followingstream_contextpipe can be used:This commit will be included in the next release of VictoriaLogs.
Yes, you are right. When I selected a small stream data, it is so quickly.
I am debugging it on today, I found this line code which it has a problem.
For example, the size of
contextRowsis68022, the callbackwriteBlockwill be executed by different goroutines, then the times of executed will be68022 * N(N ===> the number of goroutines)@jiekun
@worker24h Hello Sorry for the delay. I was able to reproduce it on our dataset. Working on a fix now.
Hello, The issue is a bug? Who can fixed it?
capture trace : trace1.out.zip goroutine stack: goroutine-stack.txt
Maybe deadlock?
I added a debug log, when it printed 1711 , then the system is hold.
The logs content:
Sorry, I can’t offer your data directory, it’s too big, more than 20G,800 million pieces of data are generated every day
Are you from CHINA? Do you have WeChat? We can communicate by it @jiekun
You can search
worker24hby WeChat.