bitcoin: cmake inconsistently overriding `-O3` (sometimes)

Using master @ d73f37dda221835b5109ede1b84db2dc7c4b74a1.

The following seems incorrect, because

make -C depends/ NO_QT=1 NO_WALLET=1 NO_ZMQ=1 -j18 CFLAGS="-O3" CXXFLAGS="-O3"
cmake -B build --toolchain /root/ci_scratch/depends/aarch64-unknown-linux-gnu/toolchain.cmake
<snip>
C++ compiler flags .................... -O3 -O2 -g
Linker flags .......................... -O3 -O2 -g

is overriding the user-provided -O3.

If you configure by providing an explicit -DCMAKE_BUILD_TYPE=RelWithDebInfo, you get:

cmake -B build --toolchain /root/ci_scratch/depends/aarch64-unknown-linux-gnu/toolchain.cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo
C++ compiler flags .................... -O3
Linker flags .......................... -O3

No -O2 at all, so the user provided -O3 is used (also missing -g for debug info?).

If you configure with -DCMAKE_BUILD_TYPE=Release:

cmake -B build --toolchain /root/ci_scratch/depends/aarch64-unknown-linux-gnu/toolchain.cmake -DCMAKE_BUILD_TYPE=Release
C++ compiler flags .................... -O3 -O2
Linker flags .......................... -O3 -O2

Now we are back to (inconsistently) overriding the user.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (25 by maintainers)

Commits related to this issue

Most upvoted comments

See also #32337.

Ignoring the current impl details, the main question to be answered here is: Do we consider -O3 to be harmful/dangerous?

Is it the case that the primary danger for us would be the compiler optimising out undefined behaviour? If so, then our compiler warnings and native_asan ci test which has the undefined sanitizer enabled should (in theory) catch said instances of UB?

Currently we intercept flags for the Release type and downgrade to -O2. This would be pretty unexpected behavior and unnecessary unless we believe that -O3 is potentially harmful/dangerous.

I think I might have the opposite view here, in that for our software specifically; that we should actively downgrade to “known good” configuration when users who are self-compiling will likely by-default choose Release, as you mention.

As far as I know, we can’t claim we fully understand the implications of -O3 on our codebase in terms of optimizations like:

  • Loop vectorization
  • More aggressive loop optimizations
  • Memory aliasing assumptions
  • etc.

In particular we will of course be most worried about -O3 “optimizing” away cryptographic security checks, or inadvertently creating new side-channel attacks, etc.

For someone who knows what they are doing and really wants (to test?) -O3, it doesn’t seem a lot of work to use our almost currently-documented workaround:

cmake -B build -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-O3 -g"

<snip>
C++ compiler flags .................... -Wthread-safety -Wthread-safety-analysis -O3 -g -Wall -Wextra -std=c++20 -fPIC -fdebug-prefix-map=/home/will/src/core/bitcoin/src=. -fmacro-prefix-map=/home/will/src/core/bitcoin/src=. -Wall -Wextra -Wgnu -Wformat -Wformat-security -Wvla -Wshadow-field -Wthread-safety -Wloop-analysis -Wredundant-decls -Wunused-member-function -Wdate-time -Wconditional-uninitialized -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wdocumentation -Wself-assign -Wundef -Wno-unused-parameter -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection
Linker flags .......................... -Wthread-safety -Wthread-safety-analysis -O3 -g -Wall -Wextra -fuse-ld=mold -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie

I also benchmarked O2 vs O3 using benchcoin (where the O3 build also included an O3 build of libsecp), which found a partial IBD (block 800k -> 855k) using -O3 to be at most 4% faster when running with a 32GB dbcache:

  O2 (base)
  Time (mean ± σ):     885.754 s ±  4.830 s    [User: 761.173 s, System: 94.122 s]
  Range (min … max):   881.841 s … 891.152 s    3 runs

  O3 (head)
  Time (mean ± σ):     852.058 s ±  8.978 s    [User: 731.521 s, System: 93.994 s]
  Range (min … max):   843.264 s … 861.210 s    3 runs

  Summary
  head (bafb9b1e75c74a4a003a5e35ead50ea0b8a76018) ran
    1.04 ± 0.01 times faster than base (57616211dc0aa0b3cc79cfb38340e1ebf3018a37)

And with a default dbcache of 450MB only a meagre 2% faster.

  O2 (base)
  Time (mean ± σ):     1333.866 s ± 47.501 s    [User: 1026.961 s, System: 177.274 s]
  Range (min … max):   1285.064 s … 1379.948 s    3 runs

  O3 (head)
  Time (mean ± σ):     1306.266 s ± 11.358 s    [User: 944.099 s, System: 170.093 s]
  Range (min … max):   1294.300 s … 1316.897 s    3 runs

  Summary
  head (bafb9b1e75c74a4a003a5e35ead50ea0b8a76018) ran
    1.02 ± 0.04 times faster than base (57616211dc0aa0b3cc79cfb38340e1ebf3018a37)

In summary, for such a small potential maximum benefit, accepting the unknowns don’t personally feel worth it to me. I think if the automatic downgrade and workaround can be documented, and a fix provided for the current -DCMAKE_CXX_FLAGS_RELEASE bug, we would be in a nice conservative position, where advanced users can still build Release with -O3 too, if they really want to, but otherwise we will override the flag for Release.

PS. The binaries for the benchcoin run are available here in case anyone would like to run their own tests with them.

PPS. It seems that Linus as recently as 2022 still does not want to entertain the option of an -O3 Kernel: https://lore.kernel.org/lkml/CA+55aFz2sNBbZyg-_i8_Ldr2e8o9dfvdSfHHuRzVtP2VMAUWPg@mail.gmail.com/ outside of manually setting special build flags (much like our current “workaround” described above).

Ah, I see. It’s: replace_cxx_flag_in_config(Release -O3 -O2)

I suppose that should only happen if the -O3 isn’t coming from an explicitly set CMAKE_CXX_FLAGS_RELEASE.