thingsboard: Thingsboard crashes every couple of days. java.lang.OutOfMemoryException Java Heap Space
Hello. I have been working with thingsboard for a while. I am publishing data to thingsboard through a custom application that acts as a gateway (collects data from three devices currently) and pushes data to thingsboard after about every 8 seconds via mqtt. It works fine for a couple of days but then thingsboard crashes. The thingsboard service shows as “Active (running)” but when i try to open the thingsboard UI through the browser it does not load. My application that pushes data to thingsboard also stops publishing data as it is unable to establish a connection through mqtt. I have to restart the thingsboard service everytime this happens. Once the service is restarted, the data publishing resumes from my application. I have tried using thingsboard with the embedded HSQLDB, Cassandra and PostgreSQL but the same OutOfMemoryException : Java Heap Space seems to cause a crash. The only difference is that with Cassandra the crash happens after about three days whereas with PostgreSQL and the embedded HSQLDB, it happens within a day or two.
My thingsboard instance is hosted on a digital ocean droplet with 4GB RAM/ 2 CPUs running Ubuntu . So far i have tried increasing the memory to 1GB in the /etc/thingsboard/thingsboard.conf file by adding this line:
export JAVA_OPTS="$JAVA_OPTS -Dplatform=deb -Xms1024M -Xmx1024M", but the issue persists. I restarted the thingsboard service and monitored the memory usage of thingsboard using top and found that within 24 hours the usage increased from 17.8% to 38.0%.
Here are the ERRORs from the thingsboard.log file that occurred during two different crashes. The errors are always Java heap space errors:
2018-01-08 02:12:21,932 [nioEventLoopGroup-4-1] ERROR i.n.u.c.D.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
2018-01-08 02:24:18,326 [pool-23-thread-1] ERROR o.t.s.a.p.PluginProcessingContext - **Critical error: Java heap space**
2018-01-09 01:08:00,507 [nioEventLoopGroup-5-9] ERROR o.t.s.t.mqtt.MqttTransportHandler - [mqtt11829] Unexpected Exception
2018-01-09 01:08:02,479 [http-nio-0.0.0.0-8080-exec-1] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler processing failed; nested exception is **java.lang.OutOfMemoryError: Java heap space**] with root cause
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 52 (12 by maintainers)
To add to that, how would you clean up the data file?
@cchaz003 I am with you there. we just started evaluation on TB. first looks, it meet all our need but this could be a show stopper. I thought the problem is that the instance does not have enough memory to run TB but 8G should be sufficient. out of memory problem is common for java app. not a fan.
would love to hear any success story with TB.
Seeing something similar here, everything was fine for the first week or so but the UI has gotten progressively slower (despite occasional reboots/restarts of thingsboard). Data overload on too feeble of a server ? (rpi3)
In your case you should prevent it from crashing in the first place. Make a note of how long does it normally take on average for thingsboard to crash on your machine. then setup a cronjob to automatically restart the thingsboard service before it actually crashes. So if it crashes after 5 hours of running, then setup a cronjob to restart the service after ever 4 hours or so. It will mean that thingsboard will be offline for a few seconds every 4 hours but it is better than it crashing and being offline permanently. Hopefully the contributors of thingsboard can find a fix soon. This is the cronjob i setup for my fix for now. It restarts thingsboard at 9am and 9pm everyday. I am on ubuntu so my service command is located in /usr/sbin:
0 9,21 * * * /usr/sbin/service thingsboard restart > /dev/null 2>&1