transformers: Cannot install Transformers version >2.3.0 with pip on CentOS
🐛 Bug
I cannot install pip install transformers for a release newer than 2.3.0. The install errors out when trying to install tokenizers. This is similar to another issue, except I have a Rust Compiler in my environment so I do not see: "error: can not find Rust Compiler".
Information
Model I am using (Bert, XLNet …):
N/A
Language I am using the model on (English, Chinese …):
N/A
The problem arises when using:
- the official example scripts: (give details below)
- my own modified scripts: (give details below)
The tasks I am working on is:
- an official GLUE/SQUaD task: (give the name)
- my own task or dataset: (give details below)
To reproduce
Steps to reproduce the behavior:
pip install transformers
which leads to the following error:
Building wheels for collected packages: tokenizers
Building wheel for tokenizers (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /home/johnmg/t2t/bin/python /home/johnmg/t2t/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpcvv0fpj6
cwd: /tmp/pip-install-d2wcoxbe/tokenizers
Complete output (221 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/tokenizers
copying tokenizers/__init__.py -> build/lib/tokenizers
creating build/lib/tokenizers/models
copying tokenizers/models/__init__.py -> build/lib/tokenizers/models
creating build/lib/tokenizers/decoders
copying tokenizers/decoders/__init__.py -> build/lib/tokenizers/decoders
creating build/lib/tokenizers/normalizers
copying tokenizers/normalizers/__init__.py -> build/lib/tokenizers/normalizers
creating build/lib/tokenizers/pre_tokenizers
copying tokenizers/pre_tokenizers/__init__.py -> build/lib/tokenizers/pre_tokenizers
creating build/lib/tokenizers/processors
copying tokenizers/processors/__init__.py -> build/lib/tokenizers/processors
creating build/lib/tokenizers/trainers
copying tokenizers/trainers/__init__.py -> build/lib/tokenizers/trainers
creating build/lib/tokenizers/implementations
copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib/tokenizers/implementations
copying tokenizers/implementations/char_level_bpe.py -> build/lib/tokenizers/implementations
copying tokenizers/implementations/byte_level_bpe.py -> build/lib/tokenizers/implementations
copying tokenizers/implementations/bert_wordpiece.py -> build/lib/tokenizers/implementations
copying tokenizers/implementations/base_tokenizer.py -> build/lib/tokenizers/implementations
copying tokenizers/implementations/__init__.py -> build/lib/tokenizers/implementations
copying tokenizers/__init__.pyi -> build/lib/tokenizers
copying tokenizers/models/__init__.pyi -> build/lib/tokenizers/models
copying tokenizers/decoders/__init__.pyi -> build/lib/tokenizers/decoders
copying tokenizers/normalizers/__init__.pyi -> build/lib/tokenizers/normalizers
copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib/tokenizers/pre_tokenizers
copying tokenizers/processors/__init__.pyi -> build/lib/tokenizers/processors
copying tokenizers/trainers/__init__.pyi -> build/lib/tokenizers/trainers
running build_ext
running build_rust
Updating crates.io index
warning: unused manifest key: target.x86_64-apple-darwin.rustflags
Compiling proc-macro2 v1.0.8
Compiling unicode-xid v0.2.0
Compiling syn v1.0.15
Compiling libc v0.2.67
Compiling autocfg v1.0.0
Compiling lazy_static v1.4.0
Compiling cfg-if v0.1.10
Compiling semver-parser v0.7.0
Compiling memchr v2.3.3
Compiling serde v1.0.104
Compiling maybe-uninit v2.0.0
Compiling ryu v1.0.2
Compiling regex-syntax v0.6.14
Compiling getrandom v0.1.14
Compiling scopeguard v1.1.0
Compiling unicode-width v0.1.7
Compiling itoa v0.4.5
Compiling bitflags v1.2.1
Running `rustc --crate-name unicode_xid /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=d0c8361b0afb9c55 -C extra-filename=-d0c8361b0afb9c55 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=97f59661e87a2bff -C extra-filename=-97f59661e87a2bff --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/proc-macro2-97f59661e87a2bff -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=be4245bf41be9154 -C extra-filename=-be4245bf41be9154 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/syn-be4245bf41be9154 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=4aa17b314a9f9392 -C extra-filename=-4aa17b314a9f9392 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/libc-4aa17b314a9f9392 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name autocfg /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0b99959d54eb5a43 -C extra-filename=-0b99959d54eb5a43 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name lazy_static /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=7fe90463f0542b89 -C extra-filename=-7fe90463f0542b89 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name semver_parser /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-parser-0.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ce71380f50d590b6 -C extra-filename=-ce71380f50d590b6 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name cfg_if /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=1b04fa8f4baea64e -C extra-filename=-1b04fa8f4baea64e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=682166ccfd58c578 -C extra-filename=-682166ccfd58c578 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memchr-682166ccfd58c578 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=e3191056f1858817 -C extra-filename=-e3191056f1858817 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/serde-e3191056f1858817 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=99bcd9a60d46382c -C extra-filename=-99bcd9a60d46382c --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/maybe-uninit-99bcd9a60d46382c -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=e704a6b7a71f3d7a -C extra-filename=-e704a6b7a71f3d7a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/ryu-e704a6b7a71f3d7a -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name regex_syntax /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=feb44197369905d4 -C extra-filename=-feb44197369905d4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="std"' -C metadata=c2394b8b43d330b2 -C extra-filename=-c2394b8b43d330b2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/getrandom-c2394b8b43d330b2 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name scopeguard /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=8a63ed9d96488c18 -C extra-filename=-8a63ed9d96488c18 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name unicode_width /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=397b6227577b65ae -C extra-filename=-397b6227577b65ae --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name itoa /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/itoa-0.4.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=b9ca519a13df71bf -C extra-filename=-b9ca519a13df71bf --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Compiling ppv-lite86 v0.2.6
Compiling rayon-core v1.7.0
Compiling unindent v0.1.5
Compiling version_check v0.9.1
Compiling strsim v0.8.0
Compiling vec_map v0.8.1
Compiling either v1.5.3
Compiling number_prefix v0.3.0
Compiling smallvec v1.2.0
Compiling ansi_term v0.11.0
Compiling unicode_categories v0.1.1
Compiling spin v0.5.2
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' -C metadata=74dee2c088f4fdf7 -C extra-filename=-74dee2c088f4fdf7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/bitflags-74dee2c088f4fdf7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name ppv_lite86 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=a66cdba604de2e91 -C extra-filename=-a66cdba604de2e91 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=01d02e6ac28bd2a9 -C extra-filename=-01d02e6ac28bd2a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/rayon-core-01d02e6ac28bd2a9 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name version_check /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/version_check-0.9.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=cb4ea0451d56bc0d -C extra-filename=-cb4ea0451d56bc0d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name unindent /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unindent-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=54ee88ab0038e61b -C extra-filename=-54ee88ab0038e61b --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name strsim /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/strsim-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=97f24fc34d9d28cd -C extra-filename=-97f24fc34d9d28cd --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name vec_map /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/vec_map-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=135a62f8a977656a -C extra-filename=-135a62f8a977656a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name either /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/either-1.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6fcca30f288e7f70 -C extra-filename=-6fcca30f288e7f70 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name number_prefix /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/number_prefix-0.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c617a5b231fd33f2 -C extra-filename=-c617a5b231fd33f2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --edition=2018 --crate-name smallvec /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.2.0/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5a1b93c8a07d924a -C extra-filename=-5a1b93c8a07d924a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name ansi_term /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=f9c541f6d3ce7af3 -C extra-filename=-f9c541f6d3ce7af3 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name unicode_categories /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode_categories-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=3376acf28b9791fe -C extra-filename=-3376acf28b9791fe --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Running `rustc --crate-name spin /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/spin-0.5.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6c5053f023d06140 -C extra-filename=-6c5053f023d06140 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
Compiling thread_local v1.0.1
Running `rustc --crate-name thread_local /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=7cdd46b26d4f9805 -C extra-filename=-7cdd46b26d4f9805 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --cap-lints allow`
Compiling textwrap v0.11.0
Running `rustc --crate-name textwrap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/textwrap-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0f312c8508ee8e9d -C extra-filename=-0f312c8508ee8e9d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --cap-lints allow`
Compiling semver v0.9.0
Running `rustc --crate-name semver /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-0.9.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=21f81775743ad422 -C extra-filename=-21f81775743ad422 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern semver_parser=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsemver_parser-ce71380f50d590b6.rmeta --cap-lints allow`
Compiling unicode-normalization-alignments v0.1.12
Running `rustc --crate-name unicode_normalization_alignments /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-normalization-alignments-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=2f95bcef5d770e35 -C extra-filename=-2f95bcef5d770e35 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern smallvec=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsmallvec-5a1b93c8a07d924a.rmeta --cap-lints allow`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/rayon-core-01d02e6ac28bd2a9/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memchr-682166ccfd58c578/build-script-build`
Running `rustc --crate-name memchr /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=916f9f60d041f29f -C extra-filename=-916f9f60d041f29f --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
Compiling rustc_version v0.2.3
Running `rustc --crate-name rustc_version /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc_version-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=63af08b6d5f0b1a9 -C extra-filename=-63af08b6d5f0b1a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern semver=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsemver-21f81775743ad422.rmeta --cap-lints allow`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/getrandom-c2394b8b43d330b2/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/bitflags-74dee2c088f4fdf7/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/maybe-uninit-99bcd9a60d46382c/build-script-build`
Compiling crossbeam-utils v0.7.2
Compiling crossbeam-epoch v0.8.2
Compiling num-traits v0.2.11
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=5b24f08aed575110 -C extra-filename=-5b24f08aed575110 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-utils-5b24f08aed575110 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=515968de6557b6b7 -C extra-filename=-515968de6557b6b7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-epoch-515968de6557b6b7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=618c6828911959ab -C extra-filename=-618c6828911959ab --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/num-traits-618c6828911959ab -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/ryu-e704a6b7a71f3d7a/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/serde-e3191056f1858817/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/libc-4aa17b314a9f9392/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/syn-be4245bf41be9154/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/proc-macro2-97f59661e87a2bff/build-script-build`
Running `rustc --crate-name bitflags /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=95683541c46a6653 -C extra-filename=-95683541c46a6653 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg bitflags_const_fn`
Running `rustc --crate-name maybe_uninit /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=169701ffcef7a104 -C extra-filename=-169701ffcef7a104 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg derive_copy --cfg repr_transparent --cfg native_uninit`
Compiling c2-chacha v0.2.3
Running `rustc --edition=2018 --crate-name c2_chacha /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/c2-chacha-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=db6d7fc899faf453 -C extra-filename=-db6d7fc899faf453 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ppv_lite86=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libppv_lite86-a66cdba604de2e91.rmeta --cap-lints allow`
Running `rustc --crate-name ryu /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=81c16096c65f1d25 -C extra-filename=-81c16096c65f1d25 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg integer128 --cfg must_use_return --cfg maybe_uninit`
Running `rustc --edition=2018 --crate-name proc_macro2 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=ce62abe820ec95ab -C extra-filename=-ce62abe820ec95ab --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern unicode_xid=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_xid-d0c8361b0afb9c55.rmeta --cap-lints allow --cfg use_proc_macro --cfg wrap_proc_macro`
Running `rustc --crate-name libc /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5d9b252ed56b1945 -C extra-filename=-5d9b252ed56b1945 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN`
Compiling aho-corasick v0.7.8
Running `rustc --crate-name aho_corasick /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5f8df0f6460a66c2 -C extra-filename=-5f8df0f6460a66c2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern memchr=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemchr-916f9f60d041f29f.rmeta --cap-lints allow`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/num-traits-618c6828911959ab/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-utils-5b24f08aed575110/build-script-build`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-epoch-515968de6557b6b7/build-script-build`
Compiling memoffset v0.5.3
Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=8ebd67a7766256e7 -C extra-filename=-8ebd67a7766256e7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memoffset-8ebd67a7766256e7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern rustc_version=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librustc_version-63af08b6d5f0b1a9.rlib --cap-lints allow`
Compiling quote v1.0.2
Running `rustc --edition=2018 --crate-name quote /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=5dd3b63b3c37ba50 -C extra-filename=-5dd3b63b3c37ba50 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --cap-lints allow`
Running `rustc --crate-name num_traits /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=682c148c69950086 -C extra-filename=-682c148c69950086 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg has_i128`
Compiling num_cpus v1.12.0
Compiling termios v0.3.1
Compiling clicolors-control v1.0.1
Compiling atty v0.2.14
Running `rustc --edition=2018 --crate-name getrandom /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=3bcce62cba29d0a1 -C extra-filename=-3bcce62cba29d0a1 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
Running `rustc --crate-name num_cpus /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num_cpus-1.12.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5375d111fec819b4 -C extra-filename=-5375d111fec819b4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
Running `rustc --crate-name termios /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/termios-0.3.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=d64097ba20dddbc5 -C extra-filename=-d64097ba20dddbc5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
Running `rustc --crate-name clicolors_control /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clicolors-control-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="terminal_autoconfig"' -C metadata=f95aedfd36305d68 -C extra-filename=-f95aedfd36305d68 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
Running `rustc --crate-name atty /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=61c34de20facc8fb -C extra-filename=-61c34de20facc8fb --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
Running `rustc --edition=2018 --crate-name syn /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=fb7a652ed3ecc931 -C extra-filename=-fb7a652ed3ecc931 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rmeta --extern unicode_xid=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_xid-d0c8361b0afb9c55.rmeta --cap-lints allow --cfg syn_disable_nightly_tests`
Compiling clap v2.33.0
Running `rustc --crate-name clap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.33.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=4d1679758f5cc3c5 -C extra-filename=-4d1679758f5cc3c5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ansi_term=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libansi_term-f9c541f6d3ce7af3.rmeta --extern atty=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libatty-61c34de20facc8fb.rmeta --extern bitflags=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libbitflags-95683541c46a6653.rmeta --extern strsim=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libstrsim-97f24fc34d9d28cd.rmeta --extern textwrap=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libtextwrap-0f312c8508ee8e9d.rmeta --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --extern vec_map=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libvec_map-135a62f8a977656a.rmeta --cap-lints allow`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memoffset-8ebd67a7766256e7/build-script-build`
Compiling rand_core v0.5.1
Running `rustc --edition=2018 --crate-name rand_core /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="getrandom"' --cfg 'feature="std"' -C metadata=4adb25904fdd70df -C extra-filename=-4adb25904fdd70df --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern getrandom=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libgetrandom-3bcce62cba29d0a1.rmeta --cap-lints allow`
Running `rustc --crate-name memoffset /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ee652fbed0600815 -C extra-filename=-ee652fbed0600815 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg memoffset_maybe_uninit --cfg memoffset_doctests`
Running `rustc --crate-name crossbeam_utils /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=71c95db82240db48 -C extra-filename=-71c95db82240db48 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --cap-lints allow --cfg has_min_const_fn --cfg has_atomic_u8 --cfg has_atomic_u16 --cfg has_atomic_u32 --cfg has_atomic_u64`
Compiling rand_chacha v0.2.1
Running `rustc --edition=2018 --crate-name rand_chacha /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_chacha-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=164a44df65235912 -C extra-filename=-164a44df65235912 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern c2_chacha=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libc2_chacha-db6d7fc899faf453.rmeta --extern rand_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_core-4adb25904fdd70df.rmeta --cap-lints allow`
Compiling rand v0.7.3
Running `rustc --edition=2018 --crate-name rand /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="getrandom"' --cfg 'feature="getrandom_package"' --cfg 'feature="libc"' --cfg 'feature="std"' -C metadata=102d035e4ca6c699 -C extra-filename=-102d035e4ca6c699 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern getrandom_package=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libgetrandom-3bcce62cba29d0a1.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --extern rand_chacha=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_chacha-164a44df65235912.rmeta --extern rand_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_core-4adb25904fdd70df.rmeta --cap-lints allow`
Compiling crossbeam-queue v0.2.1
Running `rustc --crate-name crossbeam_epoch /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=4cd0c2190306aa4a -C extra-filename=-4cd0c2190306aa4a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern maybe_uninit=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmaybe_uninit-169701ffcef7a104.rmeta --extern memoffset=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemoffset-ee652fbed0600815.rmeta --extern scopeguard=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libscopeguard-8a63ed9d96488c18.rmeta --cap-lints allow --cfg has_min_const_fn`
Running `rustc --crate-name crossbeam_queue /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-queue-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=e5afea70501509a9 -C extra-filename=-e5afea70501509a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --cap-lints allow`
Compiling regex v1.3.4
Running `rustc --crate-name regex /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="thread_local"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=40c5630aef8afe3e -C extra-filename=-40c5630aef8afe3e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern aho_corasick=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libaho_corasick-5f8df0f6460a66c2.rmeta --extern memchr=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemchr-916f9f60d041f29f.rmeta --extern regex_syntax=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex_syntax-feb44197369905d4.rmeta --extern thread_local=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libthread_local-7cdd46b26d4f9805.rmeta --cap-lints allow`
Compiling crossbeam-deque v0.7.3
Running `rustc --crate-name crossbeam_deque /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6fff16ed40375025 -C extra-filename=-6fff16ed40375025 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_epoch=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_epoch-4cd0c2190306aa4a.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern maybe_uninit=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmaybe_uninit-169701ffcef7a104.rmeta --cap-lints allow`
Running `rustc --edition=2018 --crate-name rayon_core /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5e812e4a0a947026 -C extra-filename=-5e812e4a0a947026 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_deque-6fff16ed40375025.rmeta --extern crossbeam_queue=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_queue-e5afea70501509a9.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern num_cpus=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libnum_cpus-5375d111fec819b4.rmeta --cap-lints allow`
Compiling rayon v1.3.0
Running `rustc --edition=2018 --crate-name rayon /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=35b9846c719edc99 -C extra-filename=-35b9846c719edc99 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_deque-6fff16ed40375025.rmeta --extern either=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libeither-6fcca30f288e7f70.rmeta --extern rayon_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librayon_core-5e812e4a0a947026.rmeta --cap-lints allow`
Compiling console v0.9.2
Running `rustc --edition=2018 --crate-name console /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/console-0.9.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode-width"' -C metadata=a0e410a25b05d297 -C extra-filename=-a0e410a25b05d297 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern clicolors_control=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libclicolors_control-f95aedfd36305d68.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --extern termios=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libtermios-d64097ba20dddbc5.rmeta --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --cap-lints allow`
Compiling indicatif v0.14.0
Running `rustc --edition=2018 --crate-name indicatif /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indicatif-0.14.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=af81cb79d58cbea4 -C extra-filename=-af81cb79d58cbea4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern console=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libconsole-a0e410a25b05d297.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern number_prefix=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libnumber_prefix-c617a5b231fd33f2.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --cap-lints allow`
Compiling pyo3-derive-backend v0.8.5
Running `rustc --edition=2018 --crate-name pyo3_derive_backend /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-derive-backend-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=4e1d8a522a8e0abc -C extra-filename=-4e1d8a522a8e0abc --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rmeta --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rmeta --cap-lints allow`
Compiling serde_derive v1.0.104
Compiling proc-macro-hack v0.5.11
Compiling ctor v0.1.12
Compiling ghost v0.1.1
Compiling inventory-impl v0.1.5
Compiling pyo3cls v0.8.5
Running `rustc --crate-name serde_derive /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.104/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 --cfg 'feature="default"' -C metadata=c97a5ca23329a0e7 -C extra-filename=-c97a5ca23329a0e7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
Running `rustc --edition=2018 --crate-name proc_macro_hack /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=6c301fa525410f51 -C extra-filename=-6c301fa525410f51 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
Running `rustc --edition=2018 --crate-name ctor /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ctor-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=6d15aecbd8ecf9a9 -C extra-filename=-6d15aecbd8ecf9a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
Running `rustc --edition=2018 --crate-name ghost /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ghost-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=a7fa8d8cb581322e -C extra-filename=-a7fa8d8cb581322e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
Running `rustc --edition=2018 --crate-name inventory_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-impl-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=07295296bec98a10 -C extra-filename=-07295296bec98a10 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
Running `rustc --edition=2018 --crate-name pyo3cls /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3cls-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=4e128ea32e108d4e -C extra-filename=-4e128ea32e108d4e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern pyo3_derive_backend=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libpyo3_derive_backend-4e1d8a522a8e0abc.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
Compiling paste-impl v0.1.7
Compiling indoc-impl v0.3.4
Running `rustc --edition=2018 --crate-name paste_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=0100b2f59cf859eb -C extra-filename=-0100b2f59cf859eb --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
Running `rustc --edition=2018 --crate-name indoc_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-impl-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=b6ef86cd971e397d -C extra-filename=-b6ef86cd971e397d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --extern unindent=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunindent-54ee88ab0038e61b.rlib --cap-lints allow`
Compiling inventory v0.1.5
Running `rustc --edition=2018 --crate-name inventory /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=81f198a01c0ec25f -C extra-filename=-81f198a01c0ec25f --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ctor=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libctor-6d15aecbd8ecf9a9.so --extern ghost=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libghost-a7fa8d8cb581322e.so --extern inventory_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libinventory_impl-07295296bec98a10.so --cap-lints allow`
Compiling indoc v0.3.4
Running `rustc --edition=2018 --crate-name indoc /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=3e8e6670af4a3851 -C extra-filename=-3e8e6670af4a3851 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern indoc_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libindoc_impl-b6ef86cd971e397d.so --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --cap-lints allow`
Compiling paste v0.1.7
Running `rustc --edition=2018 --crate-name paste /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=331a39d39ec573ee -C extra-filename=-331a39d39ec573ee --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern paste_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libpaste_impl-0100b2f59cf859eb.so --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --cap-lints allow`
Running `rustc --crate-name serde /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=ef69b68005b70c62 -C extra-filename=-ef69b68005b70c62 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern serde_derive=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_derive-c97a5ca23329a0e7.so --cap-lints allow --cfg ops_bound --cfg core_reverse --cfg de_boxed_c_str --cfg de_boxed_path --cfg de_rc_dst --cfg core_duration --cfg integer128 --cfg range_inclusive --cfg num_nonzero --cfg core_try_from --cfg num_nonzero_signed --cfg std_atomic64 --cfg std_atomic`
Compiling serde_json v1.0.48
Running `rustc --edition=2018 --crate-name serde_json /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.48/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5007cd62ba6989a5 -C extra-filename=-5007cd62ba6989a5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern itoa=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libitoa-b9ca519a13df71bf.rmeta --extern ryu=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libryu-81c16096c65f1d25.rmeta --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rmeta --cap-lints allow`
Compiling tokenizers v0.7.0 (/tmp/pip-install-d2wcoxbe/tokenizers/tokenizers-lib)
Running `rustc --edition=2018 --crate-name tokenizers tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=02f2af0a4056c877 -C extra-filename=-02f2af0a4056c877 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern clap=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libclap-4d1679758f5cc3c5.rmeta --extern indicatif=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libindicatif-af81cb79d58cbea4.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern rand=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand-102d035e4ca6c699.rmeta --extern rayon=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librayon-35b9846c719edc99.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --extern regex_syntax=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex_syntax-feb44197369905d4.rmeta --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rmeta --extern serde_json=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_json-5007cd62ba6989a5.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_normalization_alignments-2f95bcef5d770e35.rmeta --extern unicode_categories=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_categories-3376acf28b9791fe.rmeta`
Compiling pyo3 v0.8.5
Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.5/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="python3"' -C metadata=7ff152acc5305eee -C extra-filename=-7ff152acc5305eee --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rlib --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rlib --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rlib --extern serde_json=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_json-5007cd62ba6989a5.rlib --extern version_check=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libversion_check-cb4ea0451d56bc0d.rlib --cap-lints allow`
Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee/build-script-build`
error: failed to run custom build command for `pyo3 v0.8.5`
Caused by:
process didn't exit successfully: `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Error: pyo3 requires a nightly or dev version of Rust.', /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.5/build.rs:542:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
warning: build failed, waiting for other jobs to finish...
error: build failed
cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module pyo3/python3 --release --verbose -- --crate-type cdylib
error: cargo failed with code: 101
----------------------------------------
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
Expected behavior
A successful install of transformers.
Environment info
transformersversion:>2.3.0- Platform:
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- Python version:
3.7.4 - PyTorch version (GPU?):
1.4.0 - Tensorflow version (GPU?): N/A.
- Using GPU in script?: N/A.
- Using distributed or parallel set-up in script?: N/A.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (5 by maintainers)
Hey guys, @JohnGiorgi @Shane-Neeley , I think I figured out what was happening to you (to me also). At the time of your question, I think there was a high possibility that in the “setup.py” of transformers’ source code, there was a line writing “tokenizers=0.5.2” ; but in fact this old version will bother the update of transformers. But when I install the newest tokenizers by pip (pip install -U tokenizers), you will get a tokenizers==0.7.0. That’s why at the time of your question, there were always conflits and bugs about this tokenizers (I also installed Rust and setuptool-rust, it was always the same error).
And now they just corrected this line in the setup.py. So I just suggest you to
Then voilà ! You’ll get a brand new smoothy transformers.