iree: Bad dispatch outputs from SDXL VAE
What happened?
I’ve narrowed a numerics issue in a model down to a bad dispatch. The whole model outputs zeros but this dispatch is producing some NAN
s and some Inf
s.
Steps to reproduce your issue
iree-compile --iree-hal-target-backends=llvm-cpu --iree-input-type=torch --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --iree-llvmcpu-target-cpu-features=host --iree-llvmcpu-target-triple=x86_64-linux-gnu --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-opt-const-eval=false --iree-opt-const-expr-hoisting=false --iree-llvmcpu-enable-ukernels=all stable_diffusion_xl_base_1_0_vae.mlir -o cpu_vae.vmfb --iree-flow-trace-dispatch-tensors
- Observe the output of dispatch 208
What component(s) does this issue relate to?
MLIR, Runtime
Version information
da982154aebccb41c1cf9bf5594097a2e6906b19
Additional context
No response
About this issue
- Original URL
- State: open
- Created 4 months ago
- Comments: 34 (20 by maintainers)
whoa, first legit find from the suite and it hasn’t landed yet! high five
All the patches are landed to IREE, @gpetters94 could you help verify if the issue is addressed?
No, it’s ok, I’m thinking we might end up narrowing to a very similar dispatch after updating to the attention-retaining IR
I’m doing a few breaks to see where zeroes start. The dispatch graph shows these dispatches getting pretty huge:
They are good outputs at least up to
--iree-flow-break-dispatch=@main:242