pandas: build failure with Xcode 10 - libstdc++ not supported anymore

Code Sample, a copy-pastable example if possible

(pandas-dev) bilbo:pandas-rb robertbuckley$ python setup.py build_ext --inplace -j 4

running build_ext
building 'pandas._libs.window' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -Wno-unused-function
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
pandas/_libs/window.cpp:655:10: fatal error: 'ios' file not found
#include "ios"
         ^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1

Problem description

I setup a dev environment on Mac OS 10.13 according to https://pandas.pydata.org/pandas-docs/stable/contributing.html#creating-a-development-environment and all was well

Upgraded to 10.14 (mojave) and Xcode 10, and now i get the above failure.

Xcode 10 release notes includes the following deprecation notice

Building with libstdc++ was deprecated with Xcode 8 and is not supported in Xcode 10 when targeting iOS. C++ projects must now migrate to libc++ and are recommended to set a deployment target of macOS 10.9 or later, or iOS 7 or later. Besides changing the C++ Standard Library build setting, developers should audit hard-coded linker flags and target dependencies to remove references to libstdc++ (including -lstdc++, -lstdc++.6.0.9, libstdc++.6.0.9.tbd, and libstdc++.6.0.9.dylib). Project dependencies such as static archives that were built against libstdc++ will also need to be rebuilt against libc++. (40885260)

Some workaround suggestions are at https://stackoverflow.com/questions/52425766/stdlibc-headers-not-found-error-on-xcode-10 and the answer its linked to.

I dont have access to Xcode 9 anymore, so i can’t try the workaround of copying the old libs from there

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 23 (22 by maintainers)

Commits related to this issue

Most upvoted comments

Hi, I got to the bottom of this issue, in the end, after a trawl through clang, distutils and python make/config files.

The compiler standard library defaults to either libstdc++ or libc++, depending on the targetted macOS version - libstdc++ for 10.8 and below, and libc++ for 10.9 and above. This is determined by the environment variable MACOSX_DEPLOYMENT_TARGET or the compiler option-mmacosx-version-min, defaulting to the system version otherwise.

When distuils builds extensions on macOS, it setsMACOSX_DEPLOYMENT_TARGET to the version that python was compiled with, even if the host system / Xcode is newer.

Recent macOS versions of python have a 64-bit only variant built for 10.9 (python.org), and a universal 64/32-bit variant built for 10.6 (python.org) or 10.7 (conda). I am running the conda universal variant, so distutils targets macOS 10.7, despite my system being 10.14, with Xcode 10 which doesn’t install libstdc++. Hence the failure I saw.

It may work for some users, for 1 of 2 reasons:

  • libstdc++ is lying around somewhere from a previous Xcode 9 install
  • when running a recent 64-bit only variant of python, which targets macOS 10.9 and hence libc++

I’ve pushed a revised change which sets the targeted macOS to 10.9, when running on a 10.9 or above system, with a python/distutils which targets pre-10.9. This should not break builds on pre-10.9 systems, unlike my previous approach. It fixes the issue on my system. CI is running now.

Please let me know if you’d like me to submit it as a PR (assuming CI passes)

+24
robbuckley on Jan 6, 2019

Not sure if it’s the same issue or if this is a good fix, but I also couldn’t build after updating to mojave (I got my error at a different point though). I fixed it by adding the argument extra_link_args=["-stdlib=libc++", "-mmacosx-version-min=10.9"] to the initialization of Extension object obj here.

https://github.com/pandas-dev/pandas/blob/d78bd7a65f87022bae5021d36c8656c0ea5a7b76/setup.py#L672-L680

Credit for idea here

Here’s my error log before performing the above step:

g++ -bundle -undefined dynamic_lookup -L/Users/admin/anaconda3/envs/pandas-dev/lib -arch x86_64 -L/Users/admin/anaconda3/envs/pandas-dev/lib -arch x86_64 -L/usr/local/opt/libarchive/lib -I/usr/local/opt/libarchive/include -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -o /Users/admin/Documents/Projects/pandas-jz/pandas/_libs/window.cpython-36m-darwin.so
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1
+6
JustinZhengBC on Nov 3, 2018

@robbuckley I experienced the same issue. Thanks for the PR!

+2
JimJeon on Dec 16, 2018

i managed to repro the issue on my system with the following sequence of steps:

  1. uninstalling Xcode.app and command line tools (rm -rf /Library/Developer)
  2. removing /usr/include. This needs ’System Integrity Protection’ to be disabled, see (https://superuser.com/questions/995360/missing-usr-include-in-os-x-el-capitan#1009364)
  3. install ‘Command Line Tools (mac OS 10.14) for Xcode 10.1 from https://developer.apple.com/download/more/ (needs a [free] apple developer login)

After this I could reproduce the original failure mode i reported, and could get the extensions compiling OK using a variant of the fix proposed by @JustinZhengBC:

obj = Extension('pandas.{name}'.format(name=name),
                sources=sources,
                depends=data.get('depends', []),
                include_dirs=include,
                language=data.get('language', 'c'),
                define_macros=data.get('macros', macros),
                extra_compile_args=extra_compile_args + ['-stdlib=libc++'],
                extra_link_args=['-stdlib=libc++']

Probably these changes should be wrapped with a check for Mac OS >= 10.9 to avoid breaking other platforms.

If i added only the part in extra_link_args, I see the compiler error i originally reported:

building 'pandas._libs.window' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead
      [-Wstdlibcxx-not-found]
pandas/_libs/window.cpp:655:10: fatal error: 'ios' file not found
#include "ios"
         ^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1

If i added only the extra_compile_args part, the compile succeeds but the linker would throw an error:

building 'pandas._libs.window' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -Wno-unused-function -stdlib=libc++
In file included from pandas/_libs/window.cpp:663:
In file included from /Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:
In file included from /Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:
/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: 
      "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by " \
 ^
pandas/_libs/window.cpp:45351:27: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'Py_ssize_t' (aka 'long')
      [-Wsign-compare]
  __pyx_t_2 = ((__pyx_t_1 != __pyx_v_N) != 0);
                ~~~~~~~~~ ^  ~~~~~~~~~
2 warnings generated.
g++ -bundle -undefined dynamic_lookup -L/Users/robertbuckley/anaconda3/envs/pandas-dev/lib -arch x86_64 -L/Users/robertbuckley/anaconda3/envs/pandas-dev/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -o /Users/robertbuckley/python-scripts/pandas-rb/pandas/_libs/window.cpython-36m-darwin.so
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1

My working theory is that the older version of Xcode command line tools adds the headers into /usr/include and the libstdc++ binaries into somewhere like /usr/lib, and that subsequent uninstalls of Xcode / Command Line Tools don’t remove them.

possibly useful liinks

libc++ https://libcxx.llvm.org/ LLVM (clang) changes where the warning stdlibcxx-not-found was added https://reviews.llvm.org/D48297 extensive discussion on conda-forge https://github.com/conda-forge/conda-forge.github.io/issues/183

+2
robbuckley on Nov 7, 2018

Hey, I got the same errors but could not find /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg on macOS Catalina Version 10.15.2 with Xcode 11.2.1

+1
OswaldoBornemann on Dec 12, 2019

hi,

i managed to get it building by deleting Xcode, and dowloading + installing Xcode 9.4.1 Command Line Tools from the Apple Developer website.

There is still something funny going on, because I then installed Xcode 10 Command Line Tools (after first renaming the old ones at /Library/Developer/CommandLineTools). While I could use sudo xcode-select -s <path> to switch between the 2 versions, and clang --version confirmed it was switching the compiler version at least, I am suspicious that the libstdc++ libs and headers from 9.4.1 have been left somewhere thats not affected by the switch. So there may be some scenarios where this breaks (eg fresh install of Mojave).

I’ll continue to investigate, if thats OK

+1
robbuckley on Nov 1, 2018
Read more comments on GitHub
← pandas: BUG:to_pickle() raises TypeError when compressing large dataframe
pandas: BUILD: missing test data in 2.1.0 sdist/install →