iree: Error when compiling the Bert large model
Describe the bug Hello, I have an error trying to compile the Bert large model. The error is as below
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/distribute/merge_call_interim.py:51:0: error: 'util.global.store' op global type mismatch; global __iree_flow___sm_node20__optimizer.iter is 'tensor<i64>' but store is 'tensor<i32>'
/home/vivian/.local/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py:671:0: note: called from
/home/vivian/iree-samples/ModelCompiler/nlp_models/bert_small_gen.py:60:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py:1136:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/eager/function.py:3831:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:677:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py:1161:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/eager/function.py:3130:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/eager/function.py:3292:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/eager/function.py:3036:0: note: called from
/home/vivian/.local/lib/python3.9/site-packages/tensorflow/python/distribute/merge_call_interim.py:51:0: note: see current operation: "util.global.store"(%2036) {global = @__iree_flow___sm_node20__optimizer.iter} : (tensor<i32>) -> ()
compilation from source to vm failed
To Reproduce
- Generate the Bert large model using the script from here https://github.com/google/iree-samples/blob/main/ModelCompiler/nlp_models/bert_large_gen.py
- Run command with CUDA NVIDIA V100
./iree-compile -iree-input-type=mhlo -iree-mlir-to-vm-bytecode-module -iree-hal-target-backends=cuda -iree-cuda-llvm-target-arch=sm_70 -iree-hal-cuda-disable-loop-nounroll-wa /home/vivian/iree-samples/ModelCompiler/nlp_models/model.mlir -o model.vmfb
Additional context
- IREE was built with this commit cbea6b2670f56ba20134408b222d41c911f36cf8
- IREE pip packages 20220325.87
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 37 (13 by maintainers)
Commits related to this issue
- Fix mnist_training Colab notebook. (#9066) See [this discussion on Discord](https://discord.com/channels/689900678990135345/689900680009482386/972318238056775710) about this error, and https://github... — committed to iree-org/iree by ScottTodd 2 years ago
- Fix TF models TF models needed some workarounds from https://github.com/google/iree/issues/8644 — committed to powderluv/iree-samples by powderluv 2 years ago
Great, I’ll look into flipping these so the small one tests again and we can add it to testing (unless someone already did while I was otherwise occupied 🙂). The i64 here is only loop count (and should be completely unnecessary that big), so glad this then gets to next part