pebble: db: pebble internal error

FYI: The person you’re trying to reply to is “@lni” (L-N-I).

Got through 9 iterations of @lni’s tests with 64x parallelism while running on #580. Zero errors.

I managed to reproduce @lni’s problem with some additional instrumentation: before every restart cycle I create a checkpointXXX directory and hard link all of the DB files into that directory. The problem we’re seeing is that Pebble is running, then the test restarts into RocksDB and RocksDB creates sstables with overlapping seqnums. What the instrumentation shows is that the problem is somewhere in the WAL file itself. Here is a portion of the WAL file that RocksDB is replaying:

~ pebble wal dump --key=null --value=null checkpoint003/000017.log
...
33577(42) seq=16291 count=1
    SET(,)
33630(42) seq=16292 count=1
    SET(,)
33683(106) seq=10552 count=2
    SET(,)
    SET(,)
33796(42) seq=10554 count=1
    SET(,)

Notice the sequence numbers here. That should never happen in a WAL file. I need to do some more analysis of this WAL to determine how this is possibly happening.