RoaringBitmap: RunOptimization Bug for specific BitSet range in RoaringBitmap

Describe the bug

When using runOptimize() on a very specific sequence of continuous bits (around Integer.MAX_VALUE) and then attempt to iterate over the bitmap, this will go into an infinite loop. The sequence of “continuous” bits needs to be set in the range (lower_bound, Integer.MAX_VALUE], where, lower_bound <= Integer.MAX_VALUE - 3.

In other words, whenever more than 4 bits are set below Integer.MAX_VALUE (included), running runOptimize will mess the state such that future iteration over the bitmap loops infinitely.

To Reproduce

public void testRoaringBitmap() {
        Roaring64NavigableMap bitmap = new Roaring64NavigableMap();
        final long PRE_ADDRESSES = 4L; // (does not fail with 3L, 2L or 1L)
        final long POST_ADDRESSES = 120L; // Does not matter how many post max_value
        final long INT_MAX_VALUE = 2147483647L;
       
       for (long i = 0; i < PRE_ADDRESSES; i++) {
            bitmap.addLong(INT_MAX_VALUE - i);
        }

        for (long index = 1; index < POST_ADDRESSES; index++) {
            bitmap.addLong(INT_MAX_VALUE + index);
        }

        System.out.println("Size :: " + bitmap.getLongCardinality());

        // Serialize
        ByteBuf buffer = Unpooled.buffer();
        // Improve compression
        bitmap.runOptimize(); // This is the line causing the issue
        try (ByteBufOutputStream outputStream = new ByteBufOutputStream(buffer);
             DataOutputStream dataOutputStream =  new DataOutputStream(outputStream)){
            bitmap.serialize(dataOutputStream);
        } catch (IOException ioe) {
            throw new SerializerException("Unexpected error while serializing to a byte array");
        }

        // Deserialize
        Roaring64NavigableMap map = new Roaring64NavigableMap();
        try (ByteBufInputStream inputStream = new ByteBufInputStream(buffer)) {
            map.deserialize(inputStream);
            System.out.println("Deserialize size :: " + map.getLongCardinality());
            System.out.println();
        } catch (IOException ioe) {
            throw new SerializerException("Exception when attempting to " +
                    "deserialize stream address space.", ioe);
        }

        SortedSet<Long> addresses = new TreeSet<>(Collections.reverseOrder());
        map.forEach(e -> {
            System.out.println("Adding  :: " + e);
            addresses.add(e);
        });
    }

RoaringBitmap version:

Tested on latest version of the library as well (0.9.3)

Java version:

Java 8

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

@blacelle Should be good now. It was a silly bug.

Nevermind. Looks obvious…

  public void forEach(char msb, IntConsumer ic) {
    int high = msb << 16;
    for(int k = 0; k < this.nbrruns; ++k) {
      int base = this.getValue(k) | high;
      int le = this.getLength(k);
      for (int l = base; l <= base + le; ++l) {
        ic.accept(l);
      }
    }
  }

Can you spot the bug?