druid: historical miscalculating remaining disk capacity - segment too large exception causing segments to not load

After seeing this error in Druid historical

Caused by: com.metamx.common.ISE: Segment[timeseries_dogstatsd_counter_2018-04-04T16:00:00.000Z_2018-04-04T17:00:00.000Z_2018-04-04T16:00:00.000Z_1528210995:152,770,889] too large for storage[/var/tmp/druid/indexCache:22,010].

, we notice that the historical node stops loading new segments from realtime and the realtime nodes starts accumulating segments.

Our maxSize settings goes like this and we had enough free disk space.

druid.server.maxSize=882159184076
druid.segmentCache.locations=[{"path":"/var/tmp/druid/indexCache","maxSize":882159184076}]

Restarting Druid historical fixes the issue. We suspect that there is something going wrong with how Druid calculates the available size i.e., 22,010.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 17 (6 by maintainers)

Most upvoted comments

@egor-ryashin @tanisdlj Would it be possible for you to test this out on 0.22 ? We had something similar in one of our clusters and this patch #10884 worked for us.

a2l007 on Oct 22, 2021

its also happen to me. my remain disk size 3tb and i configured maxSize below.

druid.server.maxSize=2000000000000 druid.segmentCache.locations=[{“path”:“/var/tmp/druid/indexCache”,“maxSize”:2000000000000}]

in my case, when i use disk over 1tb, this error occured.

so i reduced druid.server.maxSize 2000000000000 to 882159184076 and scale out historical node.

druid.server.maxSize=882159184076 druid.segmentCache.locations=[{“path”:“/var/tmp/druid/indexCache”,“maxSize”:882159184076}]

finally, error is disappeared.

so i think i can use disk under (full disk size - druid.server.maxSize).

is it right?

thanks.

ps. my druid version is 0.10.1.

dsboo on Oct 14, 2019

@jkukul yes, we are sure that it was under 882159184076 bytes

sasounda on Sep 7, 2018