influxdb: Influxdb 1.7.5 stops responding while ingesting data, 1.7.4 does not
While running InfluxDB 1.7.5, some time after start while ingesting data, the Influx daemon stops responding. Writes via the HTTP endpoint time out, can’t run any SELECT queries and through the influx cli, any commands that perform reads (show measurements, show tag keys, etc…) hang. There are no log messages, no CPU usage, no memory exhaustion, etc… when this happens.
Stopping influxdb leads to a hard shutdown:
Mar 29 07:30:21 influx1 systemd: Stopping InfluxDB is an open-source, distributed, time series database...
Mar 29 07:30:21 influx1 influxd: ts=2019-03-29T07:30:21.179652Z lvl=info msg="Signal received, initializing clean shutdown..." log_id=0ESuf_v0000
Mar 29 07:30:21 influx1 influxd: ts=2019-03-29T07:30:21.179766Z lvl=info msg="Waiting for clean shutdown..." log_id=0ESuf_v0000
Mar 29 07:30:21 influx1 influxd: ts=2019-03-29T07:30:21.179892Z lvl=info msg="Listener closed" log_id=0ESuf_v0000 service=snapshot
Mar 29 07:30:51 influx1 influxd: ts=2019-03-29T07:30:51.179939Z lvl=info msg="Time limit reached, initializing hard shutdown" log_id=0ESuf_v0000
Mar 29 07:30:51 influx1 systemd: Stopped InfluxDB is an open-source, distributed, time series database.
Note that this instance is in the process of backfilling with out-of-order data. I’ve now downgraded to 1.7.4 and so far it has not hung.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 10
- Comments: 30 (13 by maintainers)
At least remove
1.7.5so that others will not be affected by this.Ran into same issue here. Spin up an influx DB, set up some CQ and RPs on the database, and everything seems to work fine until a new host writes new data in (the tags contain host names). At that point everything hangs until I restart the service. Every new host will hang the system.
@benbjohnson if you give me a rpm I can test that, building from sources… is too much to ask.
should be there now. There is usually a <24 hour delay once we create the build and releases appearing in Docker hub.
Thanks @bndw 1.7.6 is being published now. https://docs.influxdata.com/influxdb/v1.7/about_the_project/releasenotes-changelog/
@benbjohnson I’d say that the fix is good, running for an hour now without any issues, previously it would hang in the first couple of minutes, nevertheless I’ll leave it running and get with another report tomorrow.