tensorflow: TF 2.11.0/2.12 fails to build in MacOS 13 - XCode 14.1 - issue with ld linker

@bhack, @feranick, and @sushreebarsa

It’s not a compiler issue, it’s a linker (ld) issue. As I said it seems Apple folks changed ld recently. I managed to build recent master branch with Xcode 14.1 by replacing the ld with the one from Xcode 13.x. My Xcode 14.1 is installed at /Application, I backed up the ld in

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/

and replaced it with the one from 13.

The ld from Xcode 13.4.1 works for me

$ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-764
BUILD 11:29:01 May 17 2022
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 13.1.6, (clang-1316.0.21.2.5) (static support for 28, runtime is 28)
TAPI support using: Apple TAPI version 13.1.6 (tapi-1316.0.7.3)

ld from 14.x, e.g, 14.1 doesn’t work

@(#)PROGRAM:ld  PROJECT:ld64-820.1
BUILD 18:42:42 Sep 11 2022
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 14.0.0, (clang-1400.0.29.202) (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 14.0.0 (tapi-1400.0.11)

Unfortunately, it seems publicly available ld64 source code is 609, https://github.com/apple-oss-distributions/ld64/tree/ld64-609. I cannot find ld64 7xx and 8xx. So I don’t know what exactly the issue is.

I can confirm that XCode 14.3 with `ld’ version 857 now works, and compilation is successful. I am going to close this as resolved.

Would anyone have a workable version of ld to share? I can’t install XCode 13.x once 14.x is in place with MacOS 13.x is installed…

Hi @feranick this is fixed in https://developer.apple.com/services-account/download?path=/Developer_Tools/Xcode_14.3_beta_2/Xcode_14.3_beta_2.xip. Can you please check and verify?

I’ve tried with Xcode_14.3_Release_Candidate and now TF 2.12 compilation failed with

ERROR: /private/var/tmp/_bazel_alex/dc1a9368c8e4ba5b96348c2850b37ab0/external/boringssl/BUILD:161:11: Compiling src/crypto/x509/t_x509.c [for host] failed: (Exit 1): cc_wrapper.sh failed: error executing command external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics ... (remaining 44 arguments skipped)
external/boringssl/src/crypto/x509/t_x509.c:321:18: error: variable 'l' set but not used [-Werror,-Wunused-but-set-variable]
    int ret = 0, l, i;
                 ^
1 error generated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build

The issue still applies to TF 2.12-rc0. Same error.

  (cd /private/var/tmp/_bazel_feranick/50b852099a3bf3aaa184abce166f8e34/execroot/org_tensorflow && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=13.1 \
    PATH=/opt/local/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/usr/local/bin/:/Users/feranick/Documents/Work/c/android-sdk/platform-tools:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/opt/X11/bin:/Library/Apple/usr/bin \
    XCODE_VERSION_OVERRIDE=14.2.0.14C18 \
    ZERO_AR_DATE=1 \
  external/local_config_cc/cc_wrapper.sh @bazel-out/host/bin/tensorflow/python/_pywrap_tensorflow_internal.so-2.params)
# Configuration: 2d4c6c40372f31a14f2a66c81d2ffdcff66f4f42f7d84d50ade0310423a8d138
# Execution platform: @local_execution_config_platform//:platform
ld: malformed trie, terminalSize extends beyond trie data file 'bazel-out/host/bin/_solib_darwin_x86_64/_U_S_Stensorflow_Spython_C_Upywrap_Utensorflow_Uinternal.so_Ucclib___Utensorflow/libtensorflow_cc.2.dylib'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error in child process '/usr/bin/xcrun'. 1
external/local_config_cc/cc_wrapper.sh: line 69: 25937 Abort trap: 6           "$(/usr/bin/dirname "$0")"/wrapped_clang "$@"
Target //tensorflow/tools/pip_package:build_pip_package failed to build
ERROR: /Users/feranick/Desktop/tensorflow/tensorflow/tools/pip_package/BUILD:73:10 Middleman _middlemen/tensorflow_Stools_Spip_Upackage_Ssimple_Uconsole-runfiles failed: (Aborted): cc_wrapper.sh failed: error executing command 
  (cd /private/var/tmp/_bazel_feranick/50b852099a3bf3aaa184abce166f8e34/execroot/org_tensorflow && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=13.1 \
    PATH=/opt/local/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/usr/local/bin/:/Users/feranick/Documents/Work/c/android-sdk/platform-tools:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/opt/X11/bin:/Library/Apple/usr/bin \
    XCODE_VERSION_OVERRIDE=14.2.0.14C18 \
    ZERO_AR_DATE=1 \
  external/local_config_cc/cc_wrapper.sh @bazel-out/host/bin/tensorflow/python/_pywrap_tensorflow_internal.so-2.params)
# Configuration: 2d4c6c40372f31a14f2a66c81d2ffdcff66f4f42f7d84d50ade0310423a8d138
# Execution platform: @local_execution_config_platform//:platform
INFO: Elapsed time: 7766.822s, Critical Path: 1024.08s
INFO: 16824 processes: 4348 internal, 12476 local.```

Either way works, it is an environment variable for specifying the active developer directory for Xcode. If using a .bazelrc file, include build --action_env DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer in it.

Not sure if this issue is related but we have seen ld: malformed trie errors when trying to build with Xcode command line tools. The issue goes away if built with the full Xcode. (see related comment)

It seems that XCode 14.x just ships a problematic linker, indeed, as downgrading to commandline tools 13.x (@(#)PROGRAM:ld PROJECT:ld64-764) resolves the issue, even if the deployment target/platform version is still 12.0 🤷

Ah, thats unfortunate. Thank you for the insight!

Can you please point to a howto guide on using the full XCode for compilation (rather than using the command tools? I’d be happy to try if this works.

Set DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer (path to Xcode) and try running the build again.

Not sure if this issue is related but we have seen ld: malformed trie errors when trying to build with Xcode command line tools. The issue goes away if built with the full Xcode. (see related comment)

Hi guys, Xcode 14.2 RC is out. Maybe it will help you?