tokenizers: import error Symbol not found: ____chkstk_darwin

same issue here, try pip install tokenizers==0.7.0, then it works

Can’t see any of your screenshots. Anyway, you can install tokenizers even if pip says otherwise:

$ pip install transformers
Collecting transformers
  Using cached https://files.pythonhosted.org/packages/27/3c/91ed8f5c4e7ef3227b4119200fc0ed4b4fd965b1f0172021c25701087825/transformers-3.0.2-py3-none-any.whl
Collecting filelock (from transformers)
  Using cached https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Collecting sentencepiece!=0.1.92 (from transformers)
  Using cached https://files.pythonhosted.org/packages/f2/e2/813dff3d72df2f49554204e7e5f73a3dc0f0eb1e3958a4cad3ef3fb278b7/sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl
Collecting regex!=2019.12.17 (from transformers)
  Using cached https://files.pythonhosted.org/packages/c7/10/a5f40a296a199dda90432ba0af4dc4221aa5cde31c62b6088c88df971e9e/regex-2020.7.14-cp37-cp37m-manylinux2010_x86_64.whl
Collecting tokenizers==0.8.1.rc1 (from transformers)
  Using cached https://files.pythonhosted.org/packages/02/59/68c7e3833f535615fb97d33ffcb7b30bbf62bc7477a9c59cd19ad8535d72/tokenizers-0.8.1rc1-cp37-cp37m-manylinux1_x86_64.whl
Collecting tqdm>=4.27 (from transformers)
  Using cached https://files.pythonhosted.org/packages/af/88/7b0ea5fa8192d1733dea459a9e3059afc87819cb4072c43263f2ec7ab768/tqdm-4.48.0-py2.py3-none-any.whl
Collecting packaging (from transformers)
  Using cached https://files.pythonhosted.org/packages/46/19/c5ab91b1b05cfe63cccd5cfc971db9214c6dd6ced54e33c30d5af1d2bc43/packaging-20.4-py2.py3-none-any.whl
Collecting requests (from transformers)
  Using cached https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl
Collecting numpy (from transformers)
  Using cached https://files.pythonhosted.org/packages/50/8f/29d5688614f9bba59931683d5d353d738d4a3007833219ee19c455732753/numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl
Collecting sacremoses (from transformers)
  Using cached https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz
Collecting pyparsing>=2.0.2 (from packaging->transformers)
  Using cached https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl
Collecting six (from packaging->transformers)
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting idna<3,>=2.5 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl
Collecting chardet<4,>=3.0.2 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl
Collecting click (from sacremoses->transformers)
  Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
Collecting joblib (from sacremoses->transformers)
  Using cached https://files.pythonhosted.org/packages/51/dd/0e015051b4a27ec5a58b02ab774059f3289a94b0906f880a3f9507e74f38/joblib-0.16.0-py3-none-any.whl
Installing collected packages: filelock, sentencepiece, regex, tokenizers, tqdm, pyparsing, six, packaging, idna, chardet, certifi, urllib3, requests, numpy, click, joblib, sacremoses, transformers
  Running setup.py install for sacremoses ... done
Successfully installed certifi-2020.6.20 chardet-3.0.4 click-7.1.2 filelock-3.0.12 idna-2.10 joblib-0.16.0 numpy-1.19.1 packaging-20.4 pyparsing-2.4.7 regex-2020.7.14 requests-2.24.0 sacremoses-0.0.43 sentencepiece-0.1.91 six-1.15.0 tokenizers-0.8.1rc1 tqdm-4.48.0 transformers-3.0.2 urllib3-1.25.10
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ pip install tokenizers==0.8.1
Collecting tokenizers==0.8.1
  Using cached https://files.pythonhosted.org/packages/25/fc/775a802d0d59707a0a6448b0457c7222bd3d65a8232fbd02d904bae255b5/tokenizers-0.8.1-cp37-cp37m-manylinux1_x86_64.whl
transformers 3.0.2 has requirement tokenizers==0.8.1.rc1, but you'll have tokenizers 0.8.1 which is incompatible.
Installing collected packages: tokenizers
  Found existing installation: tokenizers 0.8.1rc1
    Uninstalling tokenizers-0.8.1rc1:
      Successfully uninstalled tokenizers-0.8.1rc1
Successfully installed tokenizers-0.8.1
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ python
Python 3.7.3 (default, Apr 15 2020, 10:51:02)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenizers
>>> tokenizers.__version__
'0.8.1'
>>> import transformers
>>> transformers.__version__
'3.0.2'

See above, it says it is incompatible, and yet it works 🙂

Yes, it is actually compatible, but pip says otherwise because it thinks only 0.8.1.rc1 works with transformers. Anyway, you can install 0.8.1 which I am sure will work!

Awesome! Thank you @sepal, this is extremely helpful.

@Narsil Yes, totally related! I think in our case it is due to our dependencies. If we use only full rust crates, then it seems to work fine for older versions of macOS even when building from the last one. It probably gets funky when we start linking to external libs because I guess the linker does only include what’s necessary for the last version of macOS.

So I guess, this is related to the addition of the onig dependency, as it was working right before we added it. Maybe libc would cause the same thing actually. We should be able to fix it with MACOSX_DEPLOYMENT_TARGET=10.9 added in the env, at least I hope so! (Like here: https://github.com/rust-lang/rustup/issues/24)

It only happens in High Sierra, I tried the same thing in a linux system and everything went well

Could it be linked to that ? https://github.com/nodegui/nodegui/issues/391#issuecomment-582788859 Seems building with github actions by defaults make binaries incompatible with older macos versions.