VictoriaLogs: systemd-journal-upload fails to send data to VictoriaLogs
Is your question request related to a specific component?
VictoriaLogs
Describe the question in detail
I configured systemd-journal-upload like described here: https://docs.victoriametrics.com/victorialogs/data-ingestion/journald/
systemd-journal-upload fails with the following error:
Upload to http://localhost:9428/insert/journald/upload failed with code 400: remoteAddr: "127.0.0.1:56100"; requestURI: /insert/journald/upload; cannot read journald protocol data: too big data size exceeding -journald.maxRequestSize=67108864 bytes
So my journal is too big. Can’t this be a chunked upload? Is there really no way of uploading an existing journal without throwing away all the previous logs?
Troubleshooting docs
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 26 (15 by maintainers)
Commits related to this issue
- app/vlinsert/journald: add test to catch #9070 — committed to croit/VictoriaMetrics by MatthiasGrandl a year ago
- app/vlinsert/journald: add test to catch #9070 — committed to croit/VictoriaMetrics by MatthiasGrandl a year ago
- app/vlinsert/journald: parse journald logs in streaming manner This allows parsing unlimited number of logs in a single HTTP request, without the need to buffer the logs in memory. This is needed fo... — committed to VictoriaMetrics/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: parse journald logs in streaming manner This allows parsing unlimited number of logs in a single HTTP request, without the need to buffer the logs in memory. This is needed fo... — committed to VictoriaMetrics/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: follow-up for 01d413873e1b6894685a5f7afc91dcc4bf70d6fd - Add more tests, which cover various edge cases for binary encoding of log field value in journald format - Moved common... — committed to VictoriaMetrics/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: follow-up for 01d413873e1b6894685a5f7afc91dcc4bf70d6fd - Add more tests, which cover various edge cases for binary encoding of log field value in journald format - Moved common... — committed to VictoriaMetrics/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: parse journald logs in streaming manner This allows parsing unlimited number of logs in a single HTTP request, without the need to buffer the logs in memory. This is needed fo... — committed to truepele/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: follow-up for 01d413873e1b6894685a5f7afc91dcc4bf70d6fd - Add more tests, which cover various edge cases for binary encoding of log field value in journald format - Moved common... — committed to truepele/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: parse journald logs in streaming manner This allows parsing unlimited number of logs in a single HTTP request, without the need to buffer the logs in memory. This is needed fo... — committed to truepele/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: follow-up for 01d413873e1b6894685a5f7afc91dcc4bf70d6fd - Add more tests, which cover various edge cases for binary encoding of log field value in journald format - Moved common... — committed to truepele/VictoriaMetrics by valyala a year ago
- app/vlinsert/journald: follow-up for 01d413873e1b6894685a5f7afc91dcc4bf70d6fd - Add more tests, which cover various edge cases for binary encoding of log field value in journald format - Moved common... — committed to jgomezselles/VictoriaMetrics by valyala a year ago
thanks for data, i was able to reproduce it
@AndrewChubatiuk I work with @Zinurist . Here are some example logs in json format:
example.logs.tar.gz
it’s not yet ready, will let you know once it’s ready
Indeed on a fresh installation I need a much larger journal to reproduce. This is a ~1GB journal gzipped. Try the same curl reproducer as above.
Tried uploading, but the gzipped file appears to be too large to attach. I uploaded it to our company S3 https://victoriametrics9070.fra1.s3.croit.io/logs.journal.tar.gz
let me know if you feel uncomfortable downloading from there and we can figure something else out.
Just to make sure, I am running these options. Everything else is pretty much default:
-journald.streamFields=_MACHINE_ID,_SYSTEMD_UNIT,SYSLOG_IDENTIFIER,_TRANSPORTeverything else is pretty much default.