IdGen: Debugging InvalidSystemClockExceptions

We’re running into sporadic InvalidSystemClockExceptions that end up requiring restarts, and we’re running out of ideas trying to find where it goes wrong.

one error example is: IdGen.InvalidSystemClockException: Clock moved backwards or wrapped around. Refusing to generate id for 4294446821 ticks from reading the code, this implies the new timestamp is behind the last generated one by 50 days, which seems too big of a difference even if there was a clock sync issue (we’ve checked and our k8s nodes are running chrony and the recorded sync errors are nowhere near that much of a difference).

The issue seems to happen once a month or so, and forces us to manually restart pods as it blocks all writes, and waiting the ticks for it to restart would take days.

Our timesource implementation

public class ClockTimeSource : ITimeSource
{
    private long previousTimestamp = 0;

    public DateTimeOffset Epoch { get; }
    public TimeSpan TickDuration => TimeSpan.FromMilliseconds(1);

    public ClockTimeSource() { }
    public ClockTimeSource(DateTimeOffset epoch) => Epoch = epoch;

    public long GetTicks()
    {
        var timestamp = (long)(DateTimeOffset.UtcNow - Epoch).TotalMilliseconds;
        if (timestamp > previousTimestamp)
        {
            previousTimestamp = timestamp;
        }
        return previousTimestamp;
    }
}

Would appreciate any pointers on where to start debugging this.