influxdb: Compaction crash loops and data loss on Raspberry Pi 3 B+ under minimal load
Following up on this post with a fresh issue to highlight worse symptoms that don’t seem explainable by a db-size cutoff (as was speculated on #6975 and elsewhere):
In the month since that post, I’ve had to forcibly mv the data/collectd directory twice to unstick influx from 1-2min crash loops that lasted days, seemingly due to compaction errors.
Today I’m noticing that my temps database (which I’ve not messed with during these collectd db problems, and gets about 5 points per second written to it) is missing large swaths of data from the 2 months I’ve been writing to it:
The last gap, between 1/14 and 1/17, didn’t exist this morning (when influx was still crash-looping, before the most recent time I ran mv /var/lib/influxdb/data/collectd ~/collectd.bak). That data was just recently discarded, it seems, possibly around the time I performed my “work-around” for the crash loop:
sudo service influxdb stop
sudo mv /var/lib/influxdb/data/collectd ~/collectd.bak
sudo service influxdb start
influx -execute 'create database collectd'
The default retention policy should not be discarding data, afaict:
> show retention policies on temps
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 true
Here’s the last ~7d of syslogs from the RPi server, 99.9% of which is logs from Influx crash-looping.
There seem to be messages about:
- WAL paths “already existing” when they were to be written to, and
- compactions failing because they couldn’t allocated memory
- that’s confusing because this RPi has consistently been using ≈½ of 1GB memory and 0% of 2GB swap, with a 64GB USB flash drive as a hard disk which is about 30% full).
Is running InfluxDB on an RPi supposed to generally work, or am I in uncharted territory just by attempting it?
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 4
- Comments: 68 (2 by maintainers)
tl;dr
Server memory resources were low enough that newly compacted TSM files were unable to be
mmaped during the final phase of a compaction.The log files were analyzed and it was determined that a low memory condition (
trace_id=0Cy_LzMW000) resulted in a failed compaction:This in turn caused temporary TSM files to be orphaned. Subsequent compactions for this group failed due to the orphaned
.tsm.tmpfiles:This issue is filed as #14058.
Low memory issues
Fixing #14058 will not address the problems that occur when additional TSM data cannot be
mmaped during a low-memory state.Due to the low memory condition, snapshots eventually began to fail, resulting in the same
mmaperrorcannot allocate memory. This causes a build up of.walfiles and for the in-memory cache to continue to grow. Eventually, the server panicked due to no available memory after the cache grew too large. On restart, the server continues to crash as it is not able to allocate sufficient memory to load the existing.tsmfiles and build the cache using the large number of existing.walfiles.Notes collected during analysis
FileStore.replaceenumerates the new files, removing the.tmpextension:https://github.com/influxdata/influxdb/blob/e9bada090f10ea04f07ab56a839690ab15328269/tsdb/engine/tsm1/file_store.go#L738
Creates a new
TSMReader:https://github.com/influxdata/influxdb/blob/e9bada090f10ea04f07ab56a839690ab15328269/tsdb/engine/tsm1/file_store.go#L763
which attempts to
mmapthe file:https://github.com/influxdata/influxdb/blob/05e7def600e85394f4826081f0f33f90b6ea0f3f/tsdb/engine/tsm1/reader.go#L1334
mmapfails withENOMEMand returnscannot allocate memory.FileStore.replacehandles the error:https://github.com/influxdata/influxdb/blob/e9bada090f10ea04f07ab56a839690ab15328269/tsdb/engine/tsm1/file_store.go#L765-L770
and renames the file back to
.tsm.tmp. The error is returned to the callee, ultimately resulting in theError replacing new TSM files:https://github.com/influxdata/influxdb/blob/aa3dfc066260674699ef8b01a4499bdcb425537a/tsdb/engine/tsm1/engine.go#L2210
The
.tsm.tmpfiles are not cleaned up, which only happens inCompactor.writeNewFiles:https://github.com/influxdata/influxdb/blob/2dd913d71bd015c9039a767176152f8ca959ab38/tsdb/engine/tsm1/compact.go#L1052-L1060
After trying the unofficial Ubuntu release… every issue I had with InfluxDB seems to have gone. Despite not actually changing any settings, the memory usage is minimal, the CPU load is nil under standard load, and the upper limit of pps being sent to the DB is extremely high (I’m seeing about 4300 pps with an ancient Intel 74 GB MLC SSD). I have a 53GB database running currently, and no issues to speak of with compacting any more. I hadn’t intended on doing this test today, but shortly after I replied the earlier comment, InfluxDB went into the dreaded endless service restart loop due to the failing compactions.
For anyone else who comes across this thread and wants a minature InfluxDB server that can actually handle a moderately sized DB… you won’t get a reliable InfluxDB without a fully 64-bit environment, as InfluxDB does not support 32-bit very well. Just do away with Raspbian and Balena and go to the Ubuntu server image as mentioned by @CJKohler. It seems much more responsive, is running significantly faster and the RPi4 is running almost cool to the touch for the first time, with InfluxDB running faster than it ever has!
@vogler I second @jjakob on the move to another server. It’s the only way to recover the data. Until the InfluxDB team addresses the way compaction operates on address-space limited devices (e.g. 32-bit OS and restrictive RAM of the RPi (even with the 4GB Pi 4, I have the same issue!)) - you have no alternative short of using another time series DB platform. I tried max-concurrent-compactions = 1, but in my case at least it still fails. I just gave up on the compaction process entirely on the Pi and just rely on occasionally shipping everything to a VM on my main PC.
I have since recovered my influx DB multiple times now using the method of transferring to a PC VM. I simply stop the services, tar the files, scp them over, start the services, within about 2 minutes the files are compacted… so I stop the services, ship back, done. The lack of solution for this issue suggests it may be easier to simply script the above solution of log shipping between hosts.
I wrote the #12362 patch to prevent the compaction issue on 32 bit systems, and I believe it does work (at the very least it did). However, my former company, the one for whom I was doing this work, ran into subsequent problems when running influxdb even with that patch, or perhaps because of it. The problem was that as the DB content got larger and larger, influxdb took longer and longer to boot up on a SBC running a microSD card. At one point it was over 5 minutes, and that kind of bootup time is just not acceptable for our application. I tried to mess with the DB configuration, trying different indexing methods and such, but I was unable to find a solution. We had had enough headaches with influx at that point to pull the plug. We eliminated it from our device entirely, which was a real headache because it was the keystone of our software architecture. And why wouldn’t it be the keystone? It’s a great convenience to use a web service to stick your data into an easily searchable database with great compression that tells the entire story of your system. Sensor data, configuration, syslogd, etc, all together in a neat package. But it just failed for us in enough ways that we had to move on. We don’t have a replacement solution, either. If you want to record data on this system, you have to plug it via crossover into a PC that is hosting an influx instance.
I think it’s true that influxdb is just targeting a different use case that we want. They want to be the DB in a the cloud that boots up and runs always, never shutting down, collecting metrics from net-capable devices, and running nearly constant queries for analytics. In many embedded cases, mine included, we just want something that boots up quickly, records data efficiently, is robust to power loss, and might have no or only sporadic internet access. I don’t even need to run queries very often; I really only look at the data if something went wrong.
Notice that these are all implementation gripes. I think the influxdb web interface is pretty good, and one reason I chose it originally was that it was one of only a few options that allowed me to specify my own timestamps. An embedded-focused alternative could keep the same interface but make different tradeoff decisions in the implementation behind the scenes.
@somera Not entirely true… after going to Ubuntu on my RPi 4 with 64-bit, I now have a 64GB database running on an SSD that has been running flawlessly for many months 24x7 now. Prior to that, the database maximum size was 3GB before compaction issues would prevent the influx service even starting. The key comes down to disabling the in-memory support by switching the sharding method in influxdb.conf:
index-version = "tsi1"By default it is “inmem”, meaning the shards sit in memory. This will cause issues with compaction, as in order to compact it needs to compact multiple shards at the same time in memory, thus depending on the size of the shard it could be too big to fit in memory (especially on a 32-bit OS). This is why your mileage may vary with adjustment of retention periods, a VERY complex topic on InfluxDB in itself - as the number of data points will impact on the amount of data requiring compaction. You might see some improvement by adjusting to shorter retention, but it heavily depends on the data being stored. I myself actually have indefinite retention running, and have no issues at all since moving away from inmem. Performance is slightly lower, but as it’s on an SSD and on a USB3 thanks to the RPi4 - it’s not all that noticeable. As it’s a RPi anyway, the memory isn’t blazing fast regardless, so I personally would not be concerned about the performance decrease unless you’re running a production workload.
IMPORTANT: do NOT change to tsi1 unless you have a decent external SSD, as the increased disk activity will destroy any standard microSD cards since most non-industrial cards don’t have any wear levelling capability, thus will hit the limit of writes the card can handle and cause the memory chips on the card itself to fail. I can speak to this from experience…
@markg85 I’ve said it before, InfluxDB cares not for our plight. To them, it’s all about Facebook-scale usage. 32-bit and IoT are definitely not their focus, we don’t even show up on the radar and that much is clear by the lack of any dev response to this thread. Maybe someone who has the skills could create a fork of InfluxDB some day, to actually handle small hardware? I see a massive market for this type of thing, since IoT sensors out in the wild are all too common, and more often than not don’t have the luxury of high-end hardware or network bandwidth. A distributed architecture with Influx running on RPIs scattered in remote farm sheds surrounded by LoRA sensors in a field, is just one such example where a solution like this could thrive.
FWIW - the real irony with influx is that after I switched to tsi1 rather than inmem for sharding, performance actually IMPROVED due to influx no longer logging thousands of errors every second to the syslog. My log files were rotating on average every hour through influx error messages. It’s definitely not what you would expect in moving away from memory cache…
@markg85 Ubuntu 64-bit (I run this, RPi3 technically can run it too). 32-bit you should avoid at all costs. If the CPU doesn’t have support for 64-bit, try my comments above to switch to tsi1 and you might be able to workaround the compaction issue. Compaction for me has been flawless ever since. It actually compacted the data that had been failing to compact for 6 months after switching it, even though it took 15 minutes to complete… much more reliable than copying back and forward to desktop hardware.
I haven’t migrated the InfluxDB off to the RPI4 yet as I haven’t yet got a suitable power setup for it yet to run stable - so I’m still battling away with the RPI3 and the InfluxDB compaction errors that typically occur every 12 days.
When I ran the compaction for my database is peaked at 1.8GB on a 64-bit OS. On the 32-bit Raspbian OS on my RPI4 w/ 4GB RAM it failed 100% of the time and restarted the service in a loop endlessly. On a 64-bit kernel (on the RPI4) - it peaked at 2.1GB this time but succeeded and has been stable since. Interestingly though - the Raspbian 64-bit kernel is JUST the kernel. It still runs a 32-bit userland - but I have no issues whatsoever with compaction on the 64-bit kernel but 100% failure rate for the 32-bit.
If I get the chance I will play about with 64-bit on RPI3, but I suspect it will make no difference due to the lack of RAM in the first place - it will just page to SD card. There are some things according to the devs of Raspbian that will run faster under 64-bit generally, but mostly the big improvements are to be had on the RPI4 due to things like USB3 and better graphics being available. There’s some functionality that breaks with 64-bit Raspbian too that are unique RPI features (e.g. certain drivers and libraries for things like Kodi) but otherwise it’s been rock solid for me.
As above - I’ve had 100% failure rate with InfluxDB compaction on 32-bit Raspbian, with 100% success with the 64-bit Raspbian kernel enabled. I wouldn’t advise running the 64-bit kernel on anything less than an RPI4 with 4 GB RAM though, as it would make very little difference and probably just page to SD card rather than providing a meaningful improvement.
Long story short… InfluxDB needs to sort out the way they use memory. The DB draws a hell of a lot of memory during general use with no load, and cannot handle compaction without chewing ridiculous amounts of memory. I’m playing with TimescaleDB as an alternative (Postgres-based) and the memory usage is negligible despite the query performance being comparable. Queries are a bit clunkier, but if the InfluxDB issues continue I’ll probably move over to TimescaleDB. InfluxDB is just too high maintenance and the devs seem to have no interest in improving the fundamental issues with memory management with this DB. I cannot believe there is no way to set an upper limit even!
@unreal4u I personally don’t use the RPi for GPIO, as I predominantly use mine for acting as mini low-power servers and have Arduino-based MCUs sending data to them over the network, but according to the maintainer of the Ubuntu image (see https://jamesachambers.com/raspberry-pi-4-ubuntu-server-desktop-18-04-3-image-unofficial/), the standard Raspbian kernel and utilities are available so I see no reason they shouldn’t work.
For InfluxDB also… be sure to set the index to file-based rather than in-memory, since a growing DB will inevitably have issues with memory caching at some point once there is enough data. You might also need to play around a bit with the retention period and shard groups to better tune how the database manages the underlying shards. I played around a lot with mine over countless hours (I’m still not happy with the performance, but it’s 90% better immediately simply by going Ubuntu). My server is logging about 47 parameters every 5 seconds into InfluxDB with no issues now, via Mosquitto MQTT into Telegraf and the CPU is almost idle, with very low memory usage now. Just be sure you have a decent MLC-based SSD attached to get the most out of it - avoid the SD card wherever you can, and be sure to move all frequently-accessed log files to the SSD rather than SD to avoid the card dying from excessive writes. I run an ancient Intel X25-M 74GB SSD for my InfluxDB via USB3 and it runs brilliantly in that setup, considering the very low power needs of that setup.
Let us know your experience with GPIO?
After trying BalenaOS for about a week, I gave up. It does provide true 64-bit support on a Pi 4, but trying to use a SSD to store the InfluxDB on was so painful that I gave up.
But the 64-bit unofficial Ubuntu 18.04 build for Raspberry Pi from James Chambers works great. And you only have to make 1 change in cmdline.txt to run it from an SSD. It can use the full 4 Gb on a Pi 4. https://jamesachambers.com/raspberry-pi-4-xubuntu-18-04-image-released-unofficial/
I have been running my torture test that sends 5000 points per second to the database with a very high cardinality (144000) and it is working great. I have sent well over 1 billion points, and the database is now 33 Gb, it has compacted it over 650 times, without problems.
@CJKohler it’s inevitable with larger databases that you will eventually run into issues with InfluxDB on the RPI, even the 4GB Pi 4 with the 64 bit kernel (I have this same configuration). I have this same issue currently with compaction failing, and there’s little you can do to resolve the situation as it is simply due to how InfluxDB manages memory (very poorly).
In essence, the developers have no interest in supporting the IoT market with InfluxDB on lightweight hardware - the platform simply assumes an infinite amount of memory and CPU is available and offers no customisable limits to limit how resources are used. You can try messing around with retention periods and other variables, but I’ve been trying that for months and still have to routinely transfer the database to a desktop PC simply to allow a compaction, otherwise the DB eventually crashes due to the size. My use case requires fine-grained data logging which unfortunately means a lot of data in each file, so I’m likely to move over to an Intel NUC or equivalent to handle InfluxDB.
It’s a real pity the developers cannot see the opportunities for configurations on lightweight hardware like this are inevitably going to be the future, especially with distributed IoT and sensor networks.
For anyone still battling with this issue… Raspbian now has an experimental 64-bit kernel available. I have seen successful compaction on my RPI 4 (4 GB RAM) since switching to that kernel. Technically the 64-bit kernel works on the 3 series too, but I would probably suggest upgrading to a RPI 4 for the extra memory, as it’s more likely to sustain larger databases in the long run.
Info on the 64-bit kernel is here: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=250730
@aemondis thx for the info!
Switch to an 64 Bit OS works for me…
Sonntag, 10. Januar 2021, 15:03 +0100 von notifications@github.com notifications@github.com:
64Bit Rasperry OS is coming: https://www.raspberrypi.org/forums/viewtopic.php?f=117&t=275370
I orderd a 8GB Pi and will test the setup…
@CJKohler I also tried TimescaleDB, and it does indeed work but the RPI is a little too underpowered to make queries performant, as postgres is a relational DB thus requires very careful queries with appropriate indexing (you need to be very careful how you write data into TimescaleDB as a result, as this impacts the placement in the indexes and partitions). I also use Grafana, so it sounds like you have a similar configuration to me; although I run Grafana on a separate RPi 3, to allow the RPi4 to dedicate itself to just InfluxDB.
The biggest problem with the 64-bit kernel with RPi is that it’s only a 64-bit kernel - but all the userspace is still 32-bit. I suspect this may be part of the issue with compaction.
There is one potential solution I haven’t yet had the time to try… that is running a balenaOS container on the RPi 4, as they have a native 64-bit image that is not produced by the RPi foundation. As BalenaOS is very lightweight container host infrastructure and can use the full memory of the host, you could then try the InfluxDB and Grafana Docker containers and it should have full native 64-bit support. Just be sure to use suitable ARM64 architecture images.
If you have the time to give it a try, let us know if this turns out to be a solution! 😃
Yes - I am still using Raspbian on it (a 32-bit OS), so the same upper memory limit issue occurs after some time. I’ve since retasked the RPI4 for other duties, so haven’t played around with it much more since, but the RPI3 is still running the InfluxDB.
I however am absolutely certain that if I was running a 64-bit OS, this compaction issue would not occur. It would however over time suffer severe performance degradation during compaction if the memory usage exceeded the 4GB physical and started paging, but it would still succeed (eventually). I have yet to hear of a stable and supported 64-bit RPi OS in any case. There are several out there, but many lose key functionality of the Raspberry Pi, such as GPIO support and require a lot of customisation to get going properly.
Interestingly… my use case for InfluxDB is logging both the telegraf system metrics and received messages via Mosquitto MQTT. All in all, I’m peaking something like 23 metrics/sec when all my MPUs are in full swing - although it does jump around a bit, since some of the sensors can only poll every ~3 seconds, whereas others are polling >4 times per second. The nature of my logging is that there are tens of different metrics, but I am also tagging them by device and sensor. Maybe that partitioning has something to do with it? I’m not sure how behind the scenes InfluxDB treats data that is tagged like this, if it does anything different at all…
Currently my data directory is 1.9GB. The wal directory is at 405MB across 11000 files (and growing rapidly). I’m already suffering the dreaded compaction issues the same day after running the last compaction, so it’s just a matter of time before it dies again…
I don’t know if it is supposed to work but I can definitely reproduce this issue on my Raspi 3B+.