prometheus: Configurable minValidTime in the head block

Introduction

Head block has a concept of minValidTime wherein it won’t allow samples before 1h to be ingested (Called out of bound samples)

Proposal

Here I propose making minValidTime configurable which is solely for external users of TSDB and does not affect the functioning of Prometheus.

Sometimes we want to ingest some old data because of long outages lasting >1h. In such cases, the minValidTime is a blocker. Making it configurable will allow ingesting old data.

Out of order samples are still discarded.

Changes required

To configure minValidTime

There are 2 possible ways we could make it configurable.

1. Introduce a method SetMinValidTime() which will set it to the value passed. But this one is easy to make mistakes.

2. Pass a config, let’s say MinValidTimeGracePeriod, which will accept samples up to MinValidTimeGracePeriod back in time from the current minValidTime that we calculate.

To the compaction

Allowing old samples can interfere with head compaction. So the compaction check for Head can be changed from head.maxt - head.mint >= chunkRange * 3/2 to if len(blocks) > 0 { head.maxt - lastBlock.maxt >= chunkRange * 3/2 } else { old condition }

Vertical compaction and vertical queries needs to be enabled to take care of overlapping data.

Is it safe

Memory: It is proportional to the number of series. With m-mapping in place, taking in lots of old samples is not a problem.

Data: The vertical query and vertical compaction take care of everything. The user must be aware of what they are getting into.