openssl: Build errors for buildtests in parallel builds due to missing dependency

When using parallel make with enabled buildtests, we sometimes see failures like

In file included from test/buildtest_obj_mac.c:10:
include/openssl/obj_mac.h:12: error: unterminated #ifndef
  12 | #ifndef OPENSSL_OBJ_MAC_H

This happens because test/buildtest_obj_mac.c does not have an explicit dependency on include/openssl/obj_mac.h which is regenerated in parallel due to the rules added in commit d040a1b9a028e89f0a33b36fb99f0151d2fdd4c3. The rebuild path in the makefile is:

build_sw
->
_build_sw
->
build_libs_nodep
->
libcrypto.pc
->
libcrypto.so
->
libcrypto.so.3
->
libcrypto.ld
->
util/libcrypto.num
->
include/openssl/obj_mac.h (via CRYPTOHEADERS)

Since there is no dependency between this chain and the compilation of build_test_obj_mac.c into test/buildtest_c_obj_mac-bin-buildtest_obj_mac.o, they can run in parallel. In rare cases, this leads to the situation where regeneration of obj_mac.h is still taking place while compilation reads that (partial) file causing above mentioned error.

I am not sure how to fix this issue. Once solution would be to add an explicit dependency between the compilation step and the generated header.

Note that this problem does not occur always since parallel builds are non-deterministic.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 33 (33 by maintainers)

Commits related to this issue

build.info: make sure to generate crypto objects obj_dat.h etc. first Fixes #19087 — committed to siemens/openssl by DDvO 2 years ago
Revert part of "Makefile: Generate crypto objects only as far as needed" This reverts the adaptations to generate_crypto_objects that were done by commit 89c39bd8f6528ad268c1c0d0c9e196ba4396941b. Fi... — committed to siemens/openssl by DDvO 2 years ago
Revert (most of) "Makefile: Generate crypto objects only as far as needed" This reverts all adaptations by commit 89c39bd8f6528ad268c1c0d0c9e196ba4396941b on Configurations/unix-Makefile.tmpl except ... — committed to siemens/openssl by DDvO 2 years ago
Revert (most of) "Makefile: Generate crypto objects only as far as needed" This reverts all adaptations by commit 89c39bd8f6528ad268c1c0d0c9e196ba4396941b on Configurations/unix-Makefile.tmpl except ... — committed to siemens/openssl by DDvO 2 years ago
Revert (most of) "Makefile: Generate crypto objects only as far as needed" This reverts all adaptations by commit 89c39bd8f6528ad268c1c0d0c9e196ba4396941b on Configurations/unix-Makefile.tmpl except ... — committed to sftcd/openssl by DDvO 2 years ago
Revert (most of) "Makefile: Generate crypto objects only as far as needed" This reverts all adaptations by commit 89c39bd8f6528ad268c1c0d0c9e196ba4396941b on Configurations/unix-Makefile.tmpl except ... — committed to beldmit/openssl by DDvO 2 years ago

Most upvoted comments

Well, be careful, make starts a new shell process for each recipe line. So variables you set on one line are lost on the next line. So you would need to join the recipe lines using \ to have the use the same shell:

foo: bar baz
    line1; \
    line2; \
    line3

ifranzki on Aug 30, 2022

Anyway, good point that atomicity of generating include/openssl/obj_mac.h is a complication here. I suppose this can be solved by doing the stage-wise generation on a tmp file (that is local per make thread) and then renaming or copying it to include/openssl/obj_mac.h.

That might work. But, rename the file to its final destination. Don’t copy, that’s just recreating the problem.

levitte on Aug 30, 2022

include/openssl/obj_mac.h exists in the source tree. But regeneration while the object is used does not sound like a good idea. What if regeneration actually changes the file? I am still not 100% sure why in our internal CI systems sometimes we recreate that file and sometimes not. But if it is regenerated, it should be regenerated before it is ever used.

juergenchrist on Aug 30, 2022