openssl: atexit is also unusable on AIX

Kurt Roeckx wrote: I think the cleanup function was introduced at the same time as the atexit, and there were 1 or more other cleanup functions before that that now do nothing. Functionality to initialize and clean library is existing since beginning. Macro or set of functions or single function is just different implementation. In general clean exit could ensure that resources are freed. It is not about cases related to cryptography library. It could be free of resources like file descriptors or sockets. Note cryptography library may use such resources indirectly. Regards, Roumen

The use of atexit changed at 3.3 with new test introductions and changed existing behaviour of OpenSSL regardless of what was there “in the beginning”. This broke some platforms as evidence from the number of atexit cases here. A potentially reasonable solution would have been to undo the change, but that was not deemed acceptable for some reason. As a result, the no-exit build option was introduced to save various platforms that did not behave exactly like gcc/glibc. While atexit is generally acceptable in a standalone project, OpenSSL is embedded in too many other projects and changes to atexit usage has negative impacts.

My opinion is that atexit has no business in OpenSSL given the behavioural variances in the atexit semantics when DLLs are involved. 3.3 or 4.0 should remove atexit completely and leave it up to the using projects instead, as an formal API change instead of what happened. This is just my opinion based on fighting this particular C capability for decades on various platform.

A correct leak detector should incorporate reachability into how it reports things. We primarily use LSan, which gets this right, and this says valgrind does too: https://developers.redhat.com/blog/2021/04/23/valgrind-memcheck-different-ways-to-lose-your-memory

It sounds then like the reports will be limited to people misunderstanding valgrind’s output, and they can be pointed to this document. And you can always tell them to call OPENSSL_cleanup, if they are really truly insistent.

It’ll be tedious to deal with that confusion, but the status quo is crashes (or perhaps even exploitable memory errors!) in real world applications because OpenSSL’s use of atexit doesn’t match your callers use of threads.

It also sounds like OPENSSL_cleanup’s documentation should be fixed. It’s not a mandate on applications, but an optional call for applications that really want to waste cycles on program shutdown. It probably needs to come with some warnings too. In my experience with OpenSSL callers, it’s quite common for library code to call such cleanup functions, not realizing the implications for other libraries in the same address space that use OpenSSL. For BoringSSL, we intentionally make all the global cleanup functions into no-ops because they don’t actually do anything useful, and it’s so common for folks to get this wrong.