VictoriaLogs: VictoriaLogs Bug: Read Timeout with stream_context

Describe the bug

The LogQL: _time:5m MarsSendServiceImpl , I can get result so quickly by UI , but append | stream_context before 4, the response is error, e.g. Read Timeout, as follow: image

Other issue: https://github.com/VictoriaMetrics/victorialogs-datasource/issues/117

I think the result should response quicky and not timeout

To Reproduce

As above

Version

release 1.0.0

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (12 by maintainers)

Commits related to this issue

Most upvoted comments

FYI, the commit bddb0e369f7b815046c3935d3aee6134bb35f889 improves performance of the stream_context pipe when it is applied to logs, which belong to log streams with big number of logs (e.g. more than a million of logs). This commit will be included in the next release. In the mean time it is possible to build VictoriaLogs from source code according to these instructions and verifying whether this improves performance of the stream_context pipe.

I had read the source code of Loki, the log context was showed in [now - 1h, now + 1h]. Your case typically doesn’t happen

This is a good idea! The commit 08af80ebe02cb0d299e7fc6e35cb6c48469fdcbd sets the time window for searching for surrounding logs in the log stream to one hour. This time window can be changed via time_window <duration> option in the stream_context pipe. For example, if you want to search for surrounding logs on a one year window, then the following stream_context pipe can be used:

_time:5m error | stream_context before 10 time_window 1y

This commit will be included in the next release of VictoriaLogs.

As an update, we haven’t reached a conclusion on this issue in today’s meeting. We are still making progress to find a better way to optimize it. Sorry for the delay.

Please let me know how it performs on some simple streams with a small data volume so that we can confirm the cause.

Yes, you are right. When I selected a small stream data, it is so quickly. image

I am debugging it on today, I found this line code which it has a problem.
For example, the size of contextRows is 68022, the callback writeBlock will be executed by different goroutines, then the times of executed will be 68022 * N (N ===> the number of goroutines)

image

@jiekun

@worker24h Hello Sorry for the delay. I was able to reproduce it on our dataset. Working on a fix now.

Hello, The issue is a bug? Who can fixed it?
capture trace : trace1.out.zip goroutine stack: goroutine-stack.txt

Maybe deadlock?

image

image

I added a debug log, when it printed 1711 , then the system is hold.

Hi. I’m testing similar query in our victorialogs playground but unlucky cannot reproduce with the current dataset. Test query here.

May I ask if you could provide the dataset you are using?

  • It could be exported via the query API, see this doc.
  • It would be great if you could share the actual data files under its data folder. See this doc.

The logs content:

2024-11-25T05:44:48.288Z	error	VictoriaMetrics/app/vlinsert/jsonline/jsonline.go:60	jsonline: cannot read line #4620 in /jsonline request: cannot read json line, since its size exceeds -insert.maxLineSizeBytes=262144
2024-11-25T05:45:02.236Z	warn	VictoriaMetrics/app/vlselect/main.go:169	remoteAddr: "172.25.0.101:54726"; requestURI: /select/logsql/query?end=2024-11-25T05%3A44%3A31.972Z&limit=50&query=%2A+%7C+stream_context+before+4&start=2024-11-25T05%3A39%3A31.972Z; the request couldn't be executed in 30.000 seconds; possible solutions: to increase -search.maxQueryDuration=30s; to pass bigger value to 'timeout' query arg
2024-11-25T05:45:07.667Z	error	VictoriaMetrics/app/vlinsert/jsonline/jsonline.go:60	jsonline: cannot read line #5379 in /jsonline request: cannot read json line, since its size exceeds -insert.maxLineSizeBytes=262144

image

Sorry, I can’t offer your data directory, it’s too big, more than 20G,800 million pieces of data are generated every day

Are you from CHINA? Do you have WeChat? We can communicate by it @jiekun
You can search worker24h by WeChat.