grpc: gRPC does not compile on Mac OS Catalina (10.15.6), Big Sur (11.0.1)
What version of gRPC and what language are you using?
Several versions of gRPC are used, and this error is reproduced in all of them:
- 1.27.1
- 1.33.2
- 1.34.0-dev0 (from the nightly build channel) This problem has been observed with the following Python versions:
- Python 2.7.0
- Python 3.8.2
- Python 3.9.0
What operating system (Linux, Windows,…) and version?
This problem is observed in both macOS Catalina (10.15.6) and macOS Big Sur (11.0.1).
What runtime / compiler are you using (e.g. python version or version of gcc)
For macOS Catalina:
Python 3.8.5 (default)
[Clang 6.0 (clang-600.0.57)] on darwin
For macOS Big Sur:
Python 3.9.0 (default)
[Clang 12.0.0 (clang-1200.0.31.1)] on darwin
What did you do?
Several approaches, trying to observe or correct the problem itself. It was first discovered when installing kubemq
, a library that has grpcio
as a direct dependency. Then, the problem arises when installing it, either by poetry
(which uses pip) or directly by pip
:
poetry install kubemq
Or directly,poetry install grpcio
This problem evidently arises the same, if usingpip
, as mentioned above.
What did you expect to see?
A successful installation of either kubemq
, or grpcio
.
What did you see instead?
Here is the full error message, plus the stack trace:
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -D_WIN32_WINNT=1536 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -Ithird_party/boringssl-with-bazel/src/include -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/zlib -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/user/Developer/dev/external/grpc/env/include -I/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c third_party/zlib/gzlib.c -o python_build/temp.macosx-11.0-x86_64-3.9/third_party/zlib/gzlib.o -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
third_party/zlib/gzlib.c:252:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
LSEEK(state->fd, 0, SEEK_END); /* so gzoffset() is correct */
^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
# define LSEEK lseek
^
third_party/zlib/gzlib.c:252:9: note: did you mean 'fseek'?
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
# define LSEEK lseek
^
/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include/stdio.h:162:6: note: 'fseek' declared here
int fseek(FILE *, long, int);
^
third_party/zlib/gzlib.c:258:24: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
state->start = LSEEK(state->fd, 0, SEEK_CUR);
^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
# define LSEEK lseek
^
third_party/zlib/gzlib.c:359:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
# define LSEEK lseek
^
third_party/zlib/gzlib.c:400:15: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
# define LSEEK lseek
^
third_party/zlib/gzlib.c:496:14: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
offset = LSEEK(state->fd, 0, SEEK_CUR);
^
third_party/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
# define LSEEK lseek
^
5 errors generated.
creating None/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/tmpfh161rfz
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include -I/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/user/Developer/dev/external/grpc/env/include -I/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/tmpfh161rfz/a.c -o None/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/tmpfh161rfz/a.o
Traceback (most recent call last):
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
self.spawn(compiler_so + cc_args + [src, '-o', obj] +
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
_classic_spawn(self, command)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 87, in spawn
raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 268, in build_extensions
build_ext.build_ext.build_extensions(self)
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
_build_ext.build_extension(self, ext)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
objects = self.compiler.compile(sources,
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 251, in new_compile
return old_compile(obj, src, ext, cc_args, extra_postargs,
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/setup.py", line 471, in <module>
setuptools.setup(
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
return distutils.core.setup(**attrs)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/install.py", line 546, in run
self.run_command('build')
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 87, in run
_build_ext.run(self)
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 272, in build_extensions
raise CommandError(
commands.CommandError: Failed `build_ext` step:
Traceback (most recent call last):
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 117, in _compile
self.spawn(compiler_so + cc_args + [src, '-o', obj] +
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
_classic_spawn(self, command)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/ccompiler.py", line 910, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 87, in spawn
raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 268, in build_extensions
build_ext.build_ext.build_extensions(self)
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/Users/user/Developer/dev/external/grpc/env/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
_build_ext.build_extension(self, ext)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
objects = self.compiler.compile(sources,
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/src/python/grpcio/commands.py", line 251, in new_compile
return old_compile(obj, src, ext, cc_args, extra_postargs,
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/user/Developer/dev/external/grpc/env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/setup.py'"'"'; __file__='"'"'/private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-req-build-7tkqxa9h/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/2d/t29hxdf92t7gv0rjybndvvs00000gn/T/pip-record-6ckz6lgw/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/Developer/dev/external/grpc/env/include/site/python3.9/grpcio Check the logs for full command output.
Anything else we should know about your project / environment?
Insofar as I have observed, the key here is the LSEEK complaint: I presume Clang tries to execute zlib code that is not compliant with proper C99 expression.
As a proposal, to solve this problem temporarily would be to pass a Wnoimplicit-function-declaration
as a C flag during compilation to ignore this issue, and then further address the offending section on zlib
in the future.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 29
- Comments: 36 (4 by maintainers)
Workaround for MacOS Big Sur(Intel and Arm) with Python 3.9.0.
Finally made it work by running:
I’m very tired now, so to be honest the only thing I did was just copy-pasting every env variable from every issue I found ☺️. Main references: #24248 #24449
For me:
I’m seeing similar issues with zlib on MacOS Big Sur
For me, all it took was a
python -m pip install --upgrade pip
to make it work.@VagifMammadaliyev - I introduced another error with
GRPC_PYTHON_BUILD_SYSTEM_RE2=true
The following works for me:
GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true
@VagifMammadaliyev you are a hero!
One additional comment - i had to make sure my pip was up to date before the commend in https://github.com/grpc/grpc/issues/24677#issuecomment-729983060 worked for me.
On M1 ARM Mac it is possible to run into all kinds of issues. You must not use GNU binary utils or a version of Python built with the GNU binary utils. I had installed Python via
pyenv
which used the binary utils installed byhomebrew
. In the end here is what worked for me:The easiest way to know whether you are using the wrong binary utils is to check the output of
$(which ranlib) -V
. If you are using the Apple provided binary utils as you should, you will see the following error (which is a good sign):Hope this helps someone!
I had a similar issue, my specific case being
google-cloud-bigquery
being unable to installgrpcio
using pipenv on Big Sur.A quick fix that worked for me was running
Afterwards, clang errors were no longer thrown and I was able to install
grpcio
andgoogle-cloud-bigquery
via pipenv. Not sure if this fix applies to the original issue but hope it helps!The above didn’t work for me for whatever reason but, changing the macos version of the wheel worked for me:
Here i’m using 1.33.2 on cpython 3.8. FOr other versions, check out https://pypi.org/project/grpcio/1.33.2/#files
This has been fixed on master and will be included in
1.39.0
, which will be released a week from now.I am also encountering these errors pertaining to zlib on Mac OS Big Sur. For the time being I was able to get around these build errors by making the following changes in my local zlib files:
lseek
(line 14) tofseek
#include <unistd.h>
after line 6#include <unistd.h>
after line 6This is a workaround and may have un-intended consequences.
Upgrading pip to 21.1.1 solved my issue https://issues.apache.org/jira/browse/ARROW-10744.
Thanks @VagifMammadaliyev for this workaround. This issue was freaking me out from past 2 days and i just came across your workaround and it worked. Once again thanks a lot.
Wonderful solution, @VagifMammadaliyev! You have really made my day 👍🏼 Beers are on me whenever you visit Berlin! 🍻
Thanks @VagifMammadaliyev - this gave me a huge headache. Your solution finally worked. Hoping the team addresses this.