ksh: ksh cannot be successfully built with macOS Command Line Tools 14.2

Fails with the below even on a clean checkout. macOS 12.6.2 (not 12.6.1), on Intel and M1.

+ mamake -C cmd/builtin install
+ iffe -v -c 'cc  -Os ' ref -I/Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/include/ast -I/Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/include /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libast.a -liconv /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libcmd.a -lmd -lutil /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libast.a -liconv -liconv : run /Users/mwilson/src/ksh-test/ksh/src/cmd/builtin/features/pty
iffe: test: is sys/types.h a header ... yes
iffe: test: is -liconv a library ... yes
iffe: test: is /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libast.a a library ... yes
iffe: test: is -lutil a library ... yes
iffe: test: is -lmd a library ... yes
iffe: test: is /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libcmd.a a library ... yes
iffe: test: is pty.h a header ... no
iffe: test: is libutil.h a header ... no
iffe: test: is sys/pty.h a header ... no
iffe: test: is sys/ptyio.h a header ... no
iffe: test: is sys/vty.h a header ... no
iffe: test: is sys/ioctl.h a header ... yes
iffe: test: is stropts.h a header ... no
iffe: test: is openpty a library function ... yes
iffe: test: is _getpty a library function ... no
iffe: test: is ptsname a library function ... yes
iffe: test: is stdlib.h a header ... yes
iffe: test: is grantpt a library function ... yes
iffe: test: is unlockpt a library function ... yes
iffe: test: is posix_openpt a library function ... yes
iffe: test: is termios.h a header ... yes
iffe: test: is cfmakeraw a library function ... yes
iffe: test: is -lm a library ... yes
iffe: test: output{ ... }end ... 
iffe: test: output{ ... }end ... 
iffe: Output block failed to compile. Export IFFEFLAGS=-d1 to debug.
mamake [cmd/builtin]: *** exit code 1 making FEATURE/pty
mamake: *** exit code 1 making cmd/builtin
mamake: *** exit code 1 making all
package: make failed at Mon Jan  9 19:36:06 PST 2023 in /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64
[729] mbp13 $

With IFFEFLAGS=-d1

+ mamake -C cmd/builtin install
+ iffe -d1 -v -c 'cc  -Os ' ref -I/Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/include/ast -I/Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/include /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libast.a -liconv /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libcmd.a -lmd -lutil /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libast.a -liconv -liconv : run /Users/mwilson/src/ksh-test/ksh/src/cmd/builtin/features/pty
./20560.c:1:2: error: expected identifier or '('
(;
 ^
./20560.c:1:2: error: expected ')'
./20560.c:1:1: note: to match this '('
(;
^
2 errors generated.
iffe: test: is sys/types.h a header ...
iffe: ... yes
iffe: test: is -liconv a library ...
iffe: ... yes
iffe: test: is /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libast.a a library ...
iffe: ... yes
iffe: test: is -lutil a library ...
iffe: ... yes
iffe: test: is -lmd a library ...
iffe: ... yes
iffe: test: is /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/lib/libcmd.a a library ...
iffe: ... yes
iffe: test: is pty.h a header ...
iffe: ... no
iffe: test: is libutil.h a header ...
iffe: ... no
iffe: test: is sys/pty.h a header ...
iffe: ... no
iffe: test: is sys/ptyio.h a header ...
iffe: ... no
iffe: test: is sys/vty.h a header ...
iffe: ... no
iffe: test: is sys/ioctl.h a header ...
iffe: ... yes
iffe: test: is stropts.h a header ...
iffe: ... no
./20560.c:31:46: warning: cast to smaller integer type 'unsigned int' from 'char *' [-Wpointer-to-int-cast]
int main(){char* i = (char*)&FoobaR; return ((unsigned int)i)^0xaaaa;}
                                             ^~~~~~~~~~~~~~~
1 warning generated.
Undefined symbols for architecture arm64:
  "_FoobaR", referenced from:
      _main in 20560-e9ecd6.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
iffe: test: is openpty a library function ...
./20560.c:39:31: error: use of undeclared identifier 'openpty'
static _IFFE_fun i=(_IFFE_fun)openpty;int main(){return ((unsigned int)i)^0xaaaa;}
                              ^
./20560.c:39:58: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)openpty;int main(){return ((unsigned int)i)^0xaaaa;}
                                                         ^~~~~~~~~~~~~~~
1 warning and 1 error generated.
./20560.c:39:58: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)openpty;int main(){return ((unsigned int)i)^0xaaaa;}
                                                         ^~~~~~~~~~~~~~~
1 warning generated.
iffe: ... yes
iffe: test: is _getpty a library function ...
./20560.c:40:31: error: use of undeclared identifier '_getpty'
static _IFFE_fun i=(_IFFE_fun)_getpty;int main(){return ((unsigned int)i)^0xaaaa;}
                              ^
./20560.c:40:58: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)_getpty;int main(){return ((unsigned int)i)^0xaaaa;}
                                                         ^~~~~~~~~~~~~~~
1 warning and 1 error generated.
./20560.c:40:58: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)_getpty;int main(){return ((unsigned int)i)^0xaaaa;}
                                                         ^~~~~~~~~~~~~~~
1 warning generated.
Undefined symbols for architecture arm64:
  "__getpty", referenced from:
      _main in 20560.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
iffe: ... no
iffe: test: is ptsname a library function ...
./20560.c:40:58: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)ptsname;int main(){return ((unsigned int)i)^0xaaaa;}
                                                         ^~~~~~~~~~~~~~~
1 warning generated.
iffe: ... yes
iffe: test: is stdlib.h a header ...
iffe: ... yes
iffe: test: is grantpt a library function ...
./20560.c:43:58: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)grantpt;int main(){return ((unsigned int)i)^0xaaaa;}
                                                         ^~~~~~~~~~~~~~~
1 warning generated.
iffe: ... yes
iffe: test: is unlockpt a library function ...
./20560.c:44:59: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)unlockpt;int main(){return ((unsigned int)i)^0xaaaa;}
                                                          ^~~~~~~~~~~~~~~
1 warning generated.
iffe: ... yes
iffe: test: is posix_openpt a library function ...
./20560.c:45:63: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)posix_openpt;int main(){return ((unsigned int)i)^0xaaaa;}
                                                              ^~~~~~~~~~~~~~~
1 warning generated.
iffe: ... yes
iffe: test: is termios.h a header ...
iffe: ... yes
iffe: test: is cfmakeraw a library function ...
./20560.c:47:60: warning: cast to smaller integer type 'unsigned int' from '_IFFE_fun' (aka 'int (*)()') [-Wpointer-to-int-cast]
static _IFFE_fun i=(_IFFE_fun)cfmakeraw;int main(){return ((unsigned int)i)^0xaaaa;}
                                                           ^~~~~~~~~~~~~~~
1 warning generated.
iffe: ... yes
iffe: test: is -lm a library ...
iffe: ... yes
iffe: test: output{ ... }end ...
dyld[20744]: Library not loaded: '/usr/lib/libmd.dylib'
  Referenced from: '/Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/src/cmd/builtin/20560.exe'
  Reason: tried: '/usr/lib/libmd.dylib' (no such file), '/usr/local/lib/libmd.dylib' (no such file)
iffe: line 496: 20744: Abort
iffe: ... 
iffe: test: output{ ... }end ...
dyld[20758]: Library not loaded: '/usr/lib/libmd.dylib'
  Referenced from: '/Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64/src/cmd/builtin/20560.exe'
  Reason: tried: '/usr/lib/libmd.dylib' (no such file), '/usr/local/lib/libmd.dylib' (no such file)
iffe: line 496: 20758: Abort
iffe: ... 
iffe: Output block failed to compile. Export IFFEFLAGS=-d1 to debug.
mamake [cmd/builtin]: *** exit code 1 making FEATURE/pty
mamake: *** exit code 1 making cmd/builtin
mamake: *** exit code 1 making all
package: make failed at Mon Jan  9 19:39:17 PST 2023 in /Users/mwilson/src/ksh-test/ksh/arch/darwin.arm64-64

[731] mbp13 $

About this issue

Original URL
State: closed
Created a year ago
Comments: 46

Commits related to this issue

build: Work around Apple Xcode linker breakage The macOS 13 SDK that comes with Xcode.app includes a libmd stub that fools the linker into thinking the md (message digest) library is actually install... — committed to ksh93/ksh by McDutchie a year ago

Most upvoted comments

Confirming that CLT 14.2 can now be used to build ksh on macOS 12.4.

posguy99 on Feb 22, 2023

The workaround should be in the same Mamfiles that Apple patched, but I’m not happy with simply deleting the test for libmd. It should do what you say – actually attempt to execute the test programs, instead of being satisfied with just being able to compile them.

The patch below should accomplish this (by adding && ./1.${COTEMP}.x to execute the test binary, which should do nothing but return 0).

Patch v1

diff --git a/src/lib/libcmd/Mamfile b/src/lib/libcmd/Mamfile
index 5aa4e2478..540dbab63 100644
--- a/src/lib/libcmd/Mamfile
+++ b/src/lib/libcmd/Mamfile
@@ -39,8 +39,8 @@ make install
 				exec - esac
 				exec - continue
 				exec - elif test ! -f ${INSTALLROOT}/lib/lib$i.a
-				exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
-				exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+				exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+				exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
 				exec - *$x*) continue ;;
 				exec - esac
 				exec - ;;
diff --git a/src/lib/libsum/Mamfile b/src/lib/libsum/Mamfile
index ed011c588..27048ef66 100644
--- a/src/lib/libsum/Mamfile
+++ b/src/lib/libsum/Mamfile
@@ -39,8 +39,8 @@ make install
 				exec - esac
 				exec - continue
 				exec - elif test ! -f ${INSTALLROOT}/lib/lib$i.a
-				exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
-				exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+				exec - then case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} -L${INSTALLROOT}/lib ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
+				exec - *$x*) case `{ ${CC} ${mam_cc_FLAGS} ${CCFLAGS} ${LDFLAGS} -o 1.${COTEMP}.x 1.${COTEMP}.o -l$i 2>&1 && ./1.${COTEMP}.x || echo '' "$x" ;} | sed -e 's/[][()+@?]/#/g' || :` in
 				exec - *$x*) continue ;;
 				exec - esac
 				exec - ;;

McDutchie on Feb 21, 2023

So, this is not our bug. XCode breaks the system.

Agreed. As @tvalenta pointed out, the stub that the macOS 13 SDK installs fools the lazy linker into thinking the library is there, but it isn’t in the dylib cache on macOS 12, so at runtime it goes looking for it and fails.

Can it be worked around, though? Can iffe actually try to execute the resulting executable, and trap on the error that would be generated? I could swear that some other iffe test does that. Doing that would not be fatal to any other platform, and would work on macOS.

posguy99 on Feb 21, 2023

Ok, yes, applying both src__lib__libcmd__Mamfile.diff and src__lib__libsum__Mamfile.diff allows ksh to build on macOS 12.4 + CLT 14.2.

So that’s how Apple must have gotten around the build failure.

posguy99 on Feb 17, 2023

On my 2015 MacBook (with the way nicer keyboard), I’m running macOS 12.6.3. https://github.com/apple-oss-distributions/libmd shows activity for libmd that coincides with macOS 13.0. With installation of XCode CLT 14.2, the SDK defaults to SDK version 13.1, which includes the file MacOSX13.1.sdk/usr/lib/libmd.tbd. However, I find no evidence that libmd.dylib exists in macOS 12.6.3, and based on the history of the apple-oss-distributions/libmd repository, don’t see anything to make me believe it’s been included with macOS for years (the previous commit was 2008-05-03).

On my M1 MacBook running 13.x, I have no trouble building ksh93 with CLT 14.2, using the MacOS 13.1 SDK. I haven’t dumped the dylib cache, but minimal tests indicate that libmd.dylib is indeed in the OS. I don’t have an Intel MacBook that can run Ventura handy, but I can get my hands on one and will see if I can find the library there, too.

On my Intel MacBook with 12.6.3, I have some of the older SDKs (11.3 and 12.3). If I run xcrun --sdk macosx12.3 sh bin/package make, the build doesn’t detect libmd as being an available library and completes – skipping over libsum and libcmd.

Running xcrun --sdk macosx13.1 sh bin/package make fails with libmd being linked to the test binary, but not actually existing.

Based on https://firefox-source-docs.mozilla.org/widget/cocoa/sdks.html, I tried xcrun --sdk macosx13.1 sh bin/package make CCFLAGS="-mmacosx-version-min=12.0" but it also failed. I couldn’t find any magic sauce in usr/include/Availability*.h files that referenced libmd routines. In the 13.1 SDK, the only header files I found with references to the ksh routines from ksh93/ksh/src/lib/libsum/sum-lmd.c were in the SDK’s include file CommonCrypto/CommonDigest.h.

Could it be entirely serendipitous that ksh compiles at all on macOS 13.0?

tvalenta on Feb 4, 2023