tensorflow: PyPI tensorflow-macos wheels 2.9.[012] and 2.10.0 don't work on Apple silicon + Rosetta

I’m supposed to be able to install and run PyPI’s tensorflow-macos on Apple silicon with Rosetta, right? At least I was able to with 2.8.0. All versions in PyPI since then don’t work because they are compiled with AVX extension which Rosetta doesn’t support. See details in “Click to expand!” below for repro and verification steps. I tested 2.9.[012] and 2.9.10. I suspect newer packages were compiled incorrectly with AVX.

In an x86 Rosetta-translated Python virtualenv, I expect to be able to

  1. pip install tensorflow-macos==AFFECTED_VERSIONS
  2. successfully run python -c "import tensorflow as tf; print('TensorFlow version:', tf.__version__)" without getting SIGKILL (Illegal instruction) error
Click to expand!

Issue Type

Bug

Source

binary

Tensorflow Version

2.9.0, 2.9.1, 2.9.2, 2.10.0

Custom Code

No

OS Platform and Distribution

Apple silicon, M1 Macbook Pro, macOS 12.6

Mobile device

No response

Python version

3.8.12

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

$ python -c "import platform;print(platform.machine()); import tensorflow as tf; print('TensorFlow version:', tf.__version__)"
x86_64
Illegal instruction: 4

Standalone code to reproduce the issue

# Install Rosetta 2
softwareupdate --install-rosetta

# Install a parallel x86_64 homebrew to /usr/local/bin/brew
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# Update environment variables to use x86_64 homebrew and libraries.
eval "$(/usr/local/bin/brew shellenv)"

# Make sure which brew returns /usr/local/bin/brew instead of /opt/homebrew/bin/brew before continuing.

# Install Python 3.8
arch -x86_64 brew install openssl readline sqlite3 xz zlib pyenv pyenv-virtualenv

# Check brew installed packages are actually x86_64.
file $(brew --prefix openssl)/bin/openssl
# You should see x86_64 in the output string. Like so
# /usr/local/opt/openssl@3/bin/openssl: Mach-O 64-bit executable x86_64

# Configure pyenv.
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

# Check pyenv is configured correctly by running a few commands. pyenv should show you a help message. pyenv versions should show you the Python versions you have installed.

# We install Python 3.8.12 here, but any 3.8 version works.
# Install pyenv-alias to customize Python version names in case you want a 3.8.12 for arm64.
git clone https://github.com/s1341/pyenv-alias.git $(pyenv root)/plugins/pyenv-alias
VERSION_ALIAS="3.8.12-x86" arch -x86_64 pyenv install 3.8.12

# Check that the Python version is x86_64.
pyenv shell 3.8.12-x86
python -c "import platform;print(platform.machine())"
# You should see x86_64 instead of arm64.

# create new virtual env from that x86 Python binary
pyenv virtualenv -p python3.8 3.8.12-x86 tf-macos-2.9.1

# activate the env
pyenv activate tf-macos-2.9.1

# We install tensorflow-macos 2.9.1 here but all affected versions (2.9.0, 2.9.1, 2.9.2, 2.10.0) have the same error
pip install tensorflow-macos==2.9.1

# try to import tensorflow. This should error with "illegal instruction"
python -c "import tensorflow as tf; print('TensorFlow version:', tf.__version__)"

Relevant log output

pyenv virtualenv -p python3.8 3.8.12-x86 tf-macos-2.9.1
created virtual environment CPython3.8.12.final.0-64 in 327ms
  creator CPython3Posix(dest=/Users/dxia/.pyenv/versions/3.8.12-x86/envs/tf-macos-2.9.1, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/dxia/Library/Application Support/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Looking in links: /var/folders/sr/ryl6yttx739282tn918mf3l00000gn/T/tmptufecni3
Requirement already satisfied: setuptools in /Users/dxia/.pyenv/versions/3.8.12-x86/envs/tf-macos-2.9.1/lib/python3.8/site-packages (65.4.1)
Requirement already satisfied: pip in /Users/dxia/.pyenv/versions/3.8.12-x86/envs/tf-macos-2.9.1/lib/python3.8/site-packages (22.2.2)

/tmp via 🐍 v3.8.12 (tf-macos-2.9.1) on ☁️  dxia@spotify.com took 2s
❯ pyenv activate tf-macos-2.9.1
pyenv-virtualenv: prompt changing not working for fish.

/tmp via 🐍 v3.8.12 (tf-macos-2.9.1) on ☁️  dxia@spotify.com
❯ pip install tensorflow-macos==2.9.1
Looking in indexes: https://pypi.spotify.net/simple/
Collecting tensorflow-macos==2.9.1
  Downloading https://artifactory.spotify.net/artifactory/api/pypi/pypi/packages/packages/9c/04/21d13a3fdaf9ec28cab6844e8a2b8c1993b04350d95f0edc6c17ea6d7588/tensorflow_macos-2.9.1-cp38-cp38-macosx_11_0_x86_64.whl (237.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 237.5/237.5 MB 6.1 MB/s eta 0:00:00
...
Installing collected packages: tensorboard-plugin-wit, pyasn1, libclang, keras, flatbuffers, zipp, wrapt, urllib3, typing-extensions, termcolor, tensorflow-estimator, tensorboard-data-server, six, rsa, pyparsing, pyasn1-modules, protobuf, oauthlib, numpy, MarkupSafe, idna, gast, charset-normalizer, certifi, cachetools, absl-py, werkzeug, requests, packaging, opt-einsum, keras-preprocessing, importlib-metadata, h5py, grpcio, google-pasta, google-auth, astunparse, requests-oauthlib, markdown, google-auth-oauthlib, tensorboard, tensorflow-macos
Successfully installed MarkupSafe-2.1.1 absl-py-1.3.0 astunparse-1.6.3 cachetools-5.2.0 certifi-2022.9.24 charset-normalizer-2.1.1 flatbuffers-1.12 gast-0.4.0 google-auth-2.13.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.50.0 h5py-3.7.0 idna-3.4 importlib-metadata-5.0.0 keras-2.9.0 keras-preprocessing-1.1.2 libclang-14.0.6 markdown-3.4.1 numpy-1.23.4 oauthlib-3.2.2 opt-einsum-3.3.0 packaging-21.3 protobuf-3.19.6 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-3.0.9 requests-2.28.1 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tensorboard-2.9.1 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-estimator-2.9.0 tensorflow-macos-2.9.1 termcolor-2.0.1 typing-extensions-4.4.0 urllib3-1.26.12 werkzeug-2.2.2 wrapt-1.14.1 zipp-3.10.0

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python -m pip install --upgrade pip

/tmp via 🐍 v3.8.12 (tf-macos-2.9.1) on ☁️  dxia@spotify.com took 1m23s
python -c "import platform;print(platform.machine()); import tensorflow as tf; print('TensorFlow version:', tf.__version__)"
x86_64
fish: Job 1, 'python -c "import platform;prin…' terminated by signal SIGILL (Illegal instruction)

# presence of vmovups means wheel was compiled with AVX which Rosetta doesn't support
/tmp via 🐍 v3.8.12 (tf-macos-2.9.1) on ☁️  dxia@spotify.com took 1m23s
❯ objdump -d /Users/dxia/.pyenv/versions/tf-macos-2.9.1/lib/python3.8/site-packages/tensorflow/core/platform/_cpu_feature_guard.so | grep vmovups | head
    50ed: c5 f8 10 00                  	vmovups	(%rax), %xmm0
    50fa: c5 f8 11 00                  	vmovups	%xmm0, (%rax)
    6114: c5 f8 11 07                  	vmovups	%xmm0, (%rdi)
    66c4: c5 fc 11 00                  	vmovups	%ymm0, (%rax)
    70ab: c5 fc 11 43 48               	vmovups	%ymm0, 72(%rbx)
    71a1: c5 fc 11 43 48               	vmovups	%ymm0, 72(%rbx)
    73b6: c5 f8 10 45 a8               	vmovups	-88(%rbp), %xmm0
    73bb: c5 f8 11 04 24               	vmovups	%xmm0, (%rsp)
    7792: c5 fc 11 40 10               	vmovups	%ymm0, 16(%rax)
    7797: c5 fc 11 00                  	vmovups	%ymm0, (%rax)
error: write on a pipe with no reader

/tmp via 🐍 v3.8.12 (tf-macos-2.9.1) on ☁️  dxia@spotify.com
❯ objdump -d /Users/dxia/.pyenv/versions/tf-macos-2.9.1/lib/python3.8/site-packages/tensorflow/python/util/_tf_stack.so | grep vmovups | head
    6165: c5 f8 11 40 10               	vmovups	%xmm0, 16(%rax)
    63f7: c5 fc 11 40 20               	vmovups	%ymm0, 32(%rax)
    63fc: c5 fc 11 40 40               	vmovups	%ymm0, 64(%rax)
    6401: c5 fc 11 40 60               	vmovups	%ymm0, 96(%rax)
    6406: c5 fc 11 80 80 00 00 00      	vmovups	%ymm0, 128(%rax)
    640e: c5 fc 11 80 c0 00 00 00      	vmovups	%ymm0, 192(%rax)
    6416: c5 fc 11 80 e0 00 00 00      	vmovups	%ymm0, 224(%rax)
    641e: c5 fc 11 80 38 01 00 00      	vmovups	%ymm0, 312(%rax)
    6426: c5 fc 11 80 20 01 00 00      	vmovups	%ymm0, 288(%rax)
    642e: c5 fc 11 80 00 01 00 00      	vmovups	%ymm0, 256(%rax)
error: write on a pipe with no reader

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (8 by maintainers)

Most upvoted comments

@mohantym thanks for reopening and for the tip. Just to be clear, your suggestion to use tensorflow-metal is for Apple silicon native (i.e. arm64) without Rosetta translation, right? And it requires conda and pip packages?

Unfortunately, my company currently only uses PyPI instead of conda. So we cannot conda install -c apple tensorflow-deps according to the Apple doc here. Moreover, even if we did use a hybrid conda + PyPI environment (which isn’t recommended), we’d need to first install pip packages since our software depends on tfx which pulls in tensorflow as a transitive dependency. This is further against the conda doc’s recommendations (“Use pip only after conda”). Then we’d need to run conda install --force-reinstall tensorflow=2.9.1 to replace the bad PyPI tensorflow-macos with the osx-arm64 ones from conda-forge. And these conda-forge packages are uploaded by usernames like ngam and uwe.korn whose affiliations with official maintainer teams from Google and Apple are unclear.

full conda + pip virtualenv repro steps

P.S. Your links for 2 and 3 above

  1. After install conda and install tensorflow-deps
  2. Test case executed with git_58587.py

seem to be the same screenshot. Is 3 supposed to be different?

+2
davidxia on Dec 12, 2022

Thanks for the update.

  1. We would love to use native macosx arm64 builds. But my team depends not only on tensorflow (usually only a transitive dependency) but also tfx and a lot of other tensorflow related packages like tfx-bsl, tensorflow-data-validation, etc. In order to use native macosx arm64 builds, all these other tensorflow packages will also need to provide macosx arm64 builds. So we are stuck using macosx x86 + Rosetta.
  2. I thought the whole point of Google and/or Apple creating an entirely new tensorflow-macos PyPI package is to provide macosx x86 builds that Rosetta can run? Otherwise I can just use tensorflow macosx x86 builds which are here on PyPI, right?

Happy to provide more details.

Thanks @davidxia. This makes sense. We will look into enabling Rosetta support in upcoming release. I will update here.

+1
kulinseth on Nov 10, 2022

Thanks @rishikasinha-tf and @learning-to-play for pointing out this issue. Apologize for delay in response.

To clarify, I’m just seeking clarity on whether I can expect tensorflow-macos to support Apple silicon + Rosetta. Totally fine if the decision by Google and/or Apple is no after a certain version. I just want an explicit and transparent notification of this. Thanks.

@davidxia . This is a good question. Let me provide some context (you might already be familiar or would have guessed). As you mentioned Rosetta doesn’t support AVX, so TF built with AVX enabled will have issue which you pointed out. Till TF 2.8 we were building the packages : CC_OPT_FLAGS=“-march=core2 -Wno-sign-compare” and then we switched to because we were not building the TF base optimally for x86 builds.
CC_OPT_FLAGS=“-march=native -Wno-sign-compare”

In order to be consistent with Tensorflow base builds we switched to using the native flag which is akin to -c opt recommended by Google as well.

We have couple of options.

  1. Use the native ARM64 builds. Can you please provide reason why you need to use Rosetta and can’t use the native builds ?
  2. We can add back the core2 flag which will not be the most optimal build, which I would like to avoid if we can.

If there is a compelling reason we can go with option (2). Please let me know your thoughts.

+1
kulinseth on Nov 10, 2022

@sushreebarsa I just downloaded the tensorflow_macos-2.9.2-cp38-cp38-macosx_11_0_x86_64.whl file from PyPI here, unzipped it, and ran objdump -d /tmp/tensorflow_macos-2.9.2/tensorflow/python/util/_tf_stack.so | grep vmovups | head and still see it has AVX instructions. So it’s not resolved for this wheel. I didn’t check the many others though.

❯ objdump -d /tmp/tensorflow_macos-2.9.2/tensorflow/python/util/_tf_stack.so | grep vmovups | head
    6165: c5 f8 11 40 10               	vmovups	%xmm0, 16(%rax)
    63f7: c5 fc 11 40 20               	vmovups	%ymm0, 32(%rax)
    63fc: c5 fc 11 40 40               	vmovups	%ymm0, 64(%rax)
    6401: c5 fc 11 40 60               	vmovups	%ymm0, 96(%rax)
    6406: c5 fc 11 80 80 00 00 00      	vmovups	%ymm0, 128(%rax)
    640e: c5 fc 11 80 c0 00 00 00      	vmovups	%ymm0, 192(%rax)
    6416: c5 fc 11 80 e0 00 00 00      	vmovups	%ymm0, 224(%rax)
    641e: c5 fc 11 80 38 01 00 00      	vmovups	%ymm0, 312(%rax)
    6426: c5 fc 11 80 20 01 00 00      	vmovups	%ymm0, 288(%rax)
    642e: c5 fc 11 80 00 01 00 00      	vmovups	%ymm0, 256(%rax)

I think all the tensorflow_macos-(2.9.0, 2.9.1, 2.9.2, 2.10.0)-cp(38, 39, 310)...86_64.whls on PyPI that have AVX instructions will need to be replaced, right?

+1
davidxia on Oct 30, 2022
Read more comments on GitHub
← tensorflow: Pylint incorrectly identifies tensorflow public API functions in tensorflow 2.2+
tensorflow: Python Configuration Error: 'PYTHON_BIN_PATH' environment variable is not set →