mimalloc-bench: Race condition in the rptest benchmark

Running isoalloc with the rptest benchmark resulted in a reproducible segfault, hinting at a race condition in rptest, as found by @struct:
root@67dbecee929b:/# clang -o rptest *.c -lpthread -I. -ggdb -fsanitize=thread
root@67dbecee929b:/# ./rptest 2 0 1 2 500 1000 100 8 16000
crt          2 threads random linear size [8,16000] 500 loops 1000 allocs 100 ops: ==================
WARNING: ThreadSanitizer: data race (pid=17173)
  Write of size 4 at 0x000000fec030 by main thread:
    #0 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:777:19 (rptest+0x4b51b4)
    #1 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Previous read of size 4 at 0x000000fec030 by thread T1:
    #0 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:355:10 (rptest+0x4b5a34)
    #1 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  As if synchronized via sleep:
    #0 nanosleep <null> (rptest+0x4215d8)
    #1 thread_sleep /code/jmmb/mimalloc-bench/bench/rptest/thread.c:78:2 (rptest+0x4b80f4)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:775:3 (rptest+0x4b51a0)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Location is global 'benchmark_start' of size 4 at 0x000000fec030 (rptest+0x000000fec030)

  Thread T1 (tid=17175, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

SUMMARY: ThreadSanitizer: data race /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:777:19 in benchmark_run
==================
==================
WARNING: ThreadSanitizer: data race (pid=17173)
  Read of size 8 at 0xfffff5503f68 by thread T1:
    #0 atomic_load_ptr /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:23:14 (rptest+0x4b7e10)
    #1 put_cross_thread_memory /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:312:10 (rptest+0x4b7d14)
    #2 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:469:5 (rptest+0x4b6d1c)
    #3 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  Previous write of size 8 at 0xfffff5503f68 by main thread:
    #0 atomic_store_ptr /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:28:17 (rptest+0x4b79d4)
    #1 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:767:4 (rptest+0x4b4ff4)
    #2 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  As if synchronized via sleep:
    #0 nanosleep <null> (rptest+0x4215d8)
    #1 thread_sleep /code/jmmb/mimalloc-bench/bench/rptest/thread.c:78:2 (rptest+0x4b80f4)
    #2 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:356:3 (rptest+0x4b5a4c)
    #3 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  Location is heap block of size 272 at 0xfffff5503e80 allocated by main thread:
    #0 calloc <null> (rptest+0x4228b4)
    #1 benchmark_malloc /code/jmmb/mimalloc-bench/bench/rptest/./benchmark.h:41:13 (rptest+0x4b3c5c)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:723:8 (rptest+0x4b49a8)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Thread T1 (tid=17175, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

SUMMARY: ThreadSanitizer: data race /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:23:14 in atomic_load_ptr
==================
==================
WARNING: ThreadSanitizer: data race (pid=17173)
  Atomic write of size 8 at 0xfffff5503ee0 by thread T2:
    #0 __tsan_atomic64_compare_exchange_val <null> (rptest+0x4790d4)
    #1 atomic_cas_ptr /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:77:9 (rptest+0x4b7e8c)
    #2 put_cross_thread_memory /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:315:12 (rptest+0x4b7d7c)
    #3 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:469:5 (rptest+0x4b6d1c)
    #4 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  Previous read of size 8 at 0xfffff5503ee0 by thread T1:
    #0 atomic_load_ptr /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:23:14 (rptest+0x4b7e10)
    #1 get_cross_thread_memory /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:323:10 (rptest+0x4b7c50)
    #2 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:368:14 (rptest+0x4b5b88)
    #3 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  Location is heap block of size 272 at 0xfffff5503e80 allocated by main thread:
    #0 calloc <null> (rptest+0x4228b4)
    #1 benchmark_malloc /code/jmmb/mimalloc-bench/bench/rptest/./benchmark.h:41:13 (rptest+0x4b3c5c)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:723:8 (rptest+0x4b49a8)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Thread T2 (tid=17176, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Thread T1 (tid=17175, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

SUMMARY: ThreadSanitizer: data race (/code/jmmb/mimalloc-bench/bench/rptest/rptest+0x4790d4) in __tsan_atomic64_compare_exchange_val
==================
==================
WARNING: ThreadSanitizer: data race (pid=17173)
  Read of size 4 at 0x000000fec034 by thread T2:
    #0 atomic_load32 /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:38:14 (rptest+0x4b7a20)
    #1 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:473:8 (rptest+0x4b6d3c)
    #2 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  Previous atomic write of size 4 at 0x000000fec034 by thread T1:
    #0 __tsan_atomic32_fetch_add <null> (rptest+0x4735a0)
    #1 atomic_incr32 /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:52:9 (rptest+0x4b7dc4)
    #2 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:511:3 (rptest+0x4b6d9c)
    #3 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  Location is global 'benchmark_threads_sync' of size 4 at 0x000000fec034 (rptest+0x000000fec034)

  Thread T2 (tid=17176, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Thread T1 (tid=17175, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

SUMMARY: ThreadSanitizer: data race /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:38:14 in atomic_load32
==================
==================
WARNING: ThreadSanitizer: data race (pid=17173)
  Read of size 4 at 0xfffff5503ee8 by main thread:
    #0 atomic_load32 /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:38:14 (rptest+0x4b7a20)
    #1 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:789:38 (rptest+0x4b5258)
    #2 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Previous atomic write of size 4 at 0xfffff5503ee8 by thread T2:
    #0 __tsan_atomic32_fetch_add <null> (rptest+0x4735a0)
    #1 atomic_add32 /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:62:9 (rptest+0x4b7c04)
    #2 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:398:5 (rptest+0x4b601c)
    #3 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  As if synchronized via sleep:
    #0 nanosleep <null> (rptest+0x4215d8)
    #1 thread_sleep /code/jmmb/mimalloc-bench/bench/rptest/thread.c:78:2 (rptest+0x4b80f4)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:784:3 (rptest+0x4b51fc)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Location is heap block of size 272 at 0xfffff5503e80 allocated by main thread:
    #0 calloc <null> (rptest+0x4228b4)
    #1 benchmark_malloc /code/jmmb/mimalloc-bench/bench/rptest/./benchmark.h:41:13 (rptest+0x4b3c5c)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:723:8 (rptest+0x4b49a8)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Thread T2 (tid=17176, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

SUMMARY: ThreadSanitizer: data race /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:38:14 in atomic_load32
==================
==================
WARNING: ThreadSanitizer: data race (pid=17173)
  Read of size 4 at 0xfffff5503f70 by main thread:
    #0 atomic_load32 /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:38:14 (rptest+0x4b7a20)
    #1 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:789:38 (rptest+0x4b5258)
    #2 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Previous atomic write of size 4 at 0xfffff5503f70 by thread T2:
    #0 __tsan_atomic32_fetch_add <null> (rptest+0x4735a0)
    #1 atomic_add32 /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:62:9 (rptest+0x4b7c04)
    #2 benchmark_worker /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:460:32 (rptest+0x4b6b54)
    #3 thread_entry /code/jmmb/mimalloc-bench/bench/rptest/thread.c:29:2 (rptest+0x4b7fc4)

  As if synchronized via sleep:
    #0 nanosleep <null> (rptest+0x4215d8)
    #1 thread_sleep /code/jmmb/mimalloc-bench/bench/rptest/thread.c:78:2 (rptest+0x4b80f4)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:784:3 (rptest+0x4b51fc)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Location is heap block of size 272 at 0xfffff5503e80 allocated by main thread:
    #0 calloc <null> (rptest+0x4228b4)
    #1 benchmark_malloc /code/jmmb/mimalloc-bench/bench/rptest/./benchmark.h:41:13 (rptest+0x4b3c5c)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:723:8 (rptest+0x4b49a8)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

  Thread T2 (tid=17176, running) created by main thread at:
    #0 pthread_create <null> (rptest+0x4241b4)
    #1 thread_run /code/jmmb/mimalloc-bench/bench/rptest/thread.c:41:12 (rptest+0x4b7f14)
    #2 benchmark_run /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:772:29 (rptest+0x4b5128)
    #3 main /code/jmmb/mimalloc-bench/bench/rptest/rptest.c:901:9 (rptest+0x4b7ba8)

SUMMARY: ThreadSanitizer: data race /code/jmmb/mimalloc-bench/bench/rptest/./atomic.h:38:14 in atomic_load32
==================
........756290 memory ops/CPU second (59MiB peak, 3MiB -> 58MiB bytes sample, 1375% overhead)
ThreadSanitizer: reported 6 warnings
About this issue

Original URL
State: open
Created 3 years ago
Comments: 20 (12 by maintainers)
Most upvoted comments

The upstream benchmark repo has been updated to use stdatomic on gcc/clang and an atomic for the start variable - https://github.com/mjansson/rpmalloc-benchmark/commit/65f9ae0475ba02fcf29e0525d8a3930b084a24e0
mjansson on Dec 22, 2021
Unfortunately Visual Studio still doesn’t support C11 atomics, but I’ll change upstream the benchmark start flag to an atomic and use <stdatomic.h> on non-MSVC compilers
mjansson on Dec 13, 2021