iree: Issue with Only Vulkan-Related Test Codes Failing on Android Device
I’ve progressed through building IREE and running tests on Android, but I’ve encountered an issue where only Vulkan-related test codes are consistently failing. I’ve checked the Vulkan information on Android and didn’t find any apparent issues, leaving me unsure of the cause.
Environment Details:
Android Version: 13 Vulkan apiVersion: 1.1.128 Build Steps:
Retrieved the code through Git using the x64_x86 Cross Tools Command Prompt for VS 2022. Executed vcvarsall.bat x64. Set the Android NDK path. Cloning and Submodule Update:
Git clone and git submodule update --init. Building for Host:
cmake -GNinja -B ../iree-build/ \ -DCMAKE_INSTALL_PREFIX=../iree-build/install \ -DCMAKE_BUILD_TYPE=RelWithDebInfo \ . cmake --build ../iree-build/ --target install
Building for Android:
cmake -GNinja -B ../iree-build-android/ \ -DCMAKE_TOOLCHAIN_FILE="%ANDROID_NDK%/build/cmake/android.toolchain.cmake" \ -DIREE_HOST_BIN_DIR="%CD%/../iree-build/install/bin" \ -DANDROID_ABI="arm64-v8a" \ -DANDROID_PLATFORM="android-31" \ -DIREE_BUILD_COMPILER=OFF . cmake --build ../iree-build-android/
Testing:
Built test dependencies with cmake --build …/iree-build-android/ --target iree-test-deps. Ensured the Android device was visible with adb devices. Ran tests using ctest --test-dir …/iree-build-android/ --output-on-failure. Result: The detailed results are lengthy, but in summary, all tests related to Vulkan failed, whereas others passed successfully. This discrepancy is puzzling, especially given that the Vulkan setup on the Android device seems correct according to specifications.
I’m seeking insights into why this might be happening, whether it’s a known issue with specific Android versions or Vulkan API levels, or if there are additional configurations or steps I should consider to resolve these test failures.
` Total Test time (real) = 457.62 sec
The following tests FAILED: 5 - iree/tests/e2e/linalg/check_vulkan-spirv_vulkan_conv2d.mlir_on_android_device (Failed) 6 - iree/tests/e2e/linalg/check_vulkan-spirv_vulkan_i4_to_f32.mlir_on_android_device (Failed) 8 - iree/tests/e2e/linalg/check_winograd_vulkan-spirv_vulkan_conv2d.mlir_on_android_device (Failed) 21 - iree/tests/e2e/linalg_ext_ops/check_vulkan-spirv_vulkan_scatter.mlir_on_android_device (Failed) 22 - iree/tests/e2e/linalg_ext_ops/check_vulkan-spirv_vulkan_sort.mlir_on_android_device (Failed) 23 - iree/tests/e2e/linalg_ext_ops/check_vulkan-spirv_vulkan_winograd_input.mlir_on_android_device (Failed) 24 - iree/tests/e2e/linalg_ext_ops/check_vulkan-spirv_vulkan_winograd_output.mlir_on_android_device (Failed) 61 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_abs.mlir_on_android_device (Failed) 62 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_add.mlir_on_android_device (Failed) 63 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_dot.mlir_on_android_device (Failed) 64 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_reduce_min.mlir_on_android_device (Failed) 65 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_torch_index_select_high_rank.mlir_on_android_device (Failed) 66 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_torch_index_select_negative.mlir_on_android_device (Failed) 67 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_torch_index_select_scalar.mlir_on_android_device (Failed) 68 - iree/tests/e2e/regression/check_regression_vulkan-spirv_dynamic_torch_index_select_vector.mlir_on_android_device (Failed) 69 - iree/tests/e2e/regression/check_regression_vulkan-spirv_i1_inlined_constant.mlir_on_android_device (Failed) 70 - iree/tests/e2e/regression/check_regression_vulkan-spirv_linalg_ops.mlir_on_android_device (Failed) 71 - iree/tests/e2e/regression/check_regression_vulkan-spirv_reduction_broadcast_elementwise.mlir_on_android_device (Failed) 72 - iree/tests/e2e/regression/check_regression_vulkan-spirv_scalar_computation.mlir_on_android_device (Failed) 73 - iree/tests/e2e/regression/check_regression_vulkan-spirv_softmax.mlir_on_android_device (Failed) 74 - iree/tests/e2e/regression/check_regression_vulkan-spirv_strided_slice.mlir_on_android_device (Failed) 75 - iree/tests/e2e/regression/check_regression_vulkan-spirv_transpose.mlir_on_android_device (Failed) 196 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_abs.mlir_on_android_device (Failed) 197 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_add.mlir_on_android_device (Failed) 198 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_batch_norm_inference.mlir_on_android_device (Failed) 199 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_bitcast_convert.mlir_on_android_device (Failed) 200 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_broadcast.mlir_on_android_device (Failed) 201 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_broadcast_add.mlir_on_android_device (Failed) 202 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_broadcast_in_dim.mlir_on_android_device (Failed) 203 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_clamp.mlir_on_android_device (Failed) 204 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_compare.mlir_on_android_device (Failed) 205 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_complex.mlir_on_android_device (Failed) 206 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_concatenate.mlir_on_android_device (Failed) 207 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_constant.mlir_on_android_device (Failed) 208 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_convert.mlir_on_android_device (Failed) 209 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_convolution.mlir_on_android_device (Failed) 210 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_cosine.mlir_on_android_device (Failed) 211 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_divide.mlir_on_android_device (Failed) 212 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_dot.mlir_on_android_device (Failed) 213 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_dot_bf16.mlir_on_android_device (Failed) 214 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_dot_general.mlir_on_android_device (Failed) 215 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_dynamic_slice.mlir_on_android_device (Failed) 216 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_dynamic_update_slice.mlir_on_android_device (Failed) 217 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_exponential.mlir_on_android_device (Failed) 218 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_exponential_minus_one.mlir_on_android_device (Failed) 219 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_finite.mlir_on_android_device (Failed) 220 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_floor.mlir_on_android_device (Failed) 221 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_gather.mlir_on_android_device (Failed) 222 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_iota.mlir_on_android_device (Failed) 223 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_log.mlir_on_android_device (Failed) 224 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_log_plus_one.mlir_on_android_device (Failed) 225 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_maximum.mlir_on_android_device (Failed) 226 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_minimum.mlir_on_android_device (Failed) 227 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_multiply.mlir_on_android_device (Failed) 228 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_negate.mlir_on_android_device (Failed) 229 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_pad.mlir_on_android_device (Failed) 230 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_philox.mlir_on_android_device (Failed) 231 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_pow.mlir_on_android_device (Failed) 232 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_reduce.mlir_on_android_device (Failed) 233 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_reduce_window.mlir_on_android_device (Failed) 234 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_remainder.mlir_on_android_device (Failed) 235 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_reshape.mlir_on_android_device (Failed) 236 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_rng_normal.mlir_on_android_device (Failed) 237 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_rng_uniform.mlir_on_android_device (Failed) 238 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_round.mlir_on_android_device (Failed) 239 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_rsqrt.mlir_on_android_device (Failed) 240 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_scatter.mlir_on_android_device (Failed) 241 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_scatter_dynamic.mlir_on_android_device (Failed) 242 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_select.mlir_on_android_device (Failed) 243 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_sine.mlir_on_android_device (Failed) 244 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_slice.mlir_on_android_device (Failed) 245 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_sort.mlir_on_android_device (Failed) 246 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_sqrt.mlir_on_android_device (Failed) 247 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_subtract.mlir_on_android_device (Failed) 248 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_tanh.mlir_on_android_device (Failed) 249 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_three_fry.mlir_on_android_device (Failed) 250 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_torch_index_select.mlir_on_android_device (Failed) 251 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_transpose.mlir_on_android_device (Failed) 252 - iree/tests/e2e/stablehlo_ops/check_vulkan-spirv_vulkan_while.mlir_on_android_device (Failed) 266 - iree/tests/e2e/tensor_ops/check_vulkan-spirv_vulkan_extract_slice.mlir_on_android_device (Failed) 267 - iree/tests/e2e/tensor_ops/check_vulkan-spirv_vulkan_tensor_insert_slice.mlir_on_android_device (Failed) 390 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_abs.mlir_on_android_device (Failed) 391 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_add.mlir_on_android_device (Failed) 392 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_arithmetic_right_shift.mlir_on_android_device (Failed) 393 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_bitwise_and.mlir_on_android_device (Failed) 394 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_bitwise_or.mlir_on_android_device (Failed) 395 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_bitwise_xor.mlir_on_android_device (Failed) 396 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_ceil.mlir_on_android_device (Failed) 397 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_clamp.mlir_on_android_device (Failed) 398 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_clz.mlir_on_android_device (Failed) 399 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_const.mlir_on_android_device (Failed) 400 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_equal.mlir_on_android_device (Failed) 401 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_exp.mlir_on_android_device (Failed) 402 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_floor.mlir_on_android_device (Failed) 403 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_fully_connected.mlir_on_android_device (Failed) 404 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_gather.mlir_on_android_device (Failed) 405 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_greater.mlir_on_android_device (Failed) 406 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_greater_equal.mlir_on_android_device (Failed) 407 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_if.mlir_on_android_device (Failed) 408 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_log.mlir_on_android_device (Failed) 409 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_logical_left_shift.mlir_on_android_device (Failed) 410 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_logical_right_shift.mlir_on_android_device (Failed) 411 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_matmul.mlir_on_android_device (Failed) 412 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_max_pool.mlir_on_android_device (Failed) 413 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_maximum.mlir_on_android_device (Failed) 414 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_minimum.mlir_on_android_device (Failed) 415 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_mul.mlir_on_android_device (Failed) 416 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_mul_shift.mlir_on_android_device (Failed) 417 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_negate.mlir_on_android_device (Failed) 418 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_pad.mlir_on_android_device (Failed) 419 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_reciprocal.mlir_on_android_device (Failed) 420 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_reduce.mlir_on_android_device (Failed) 421 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_reshape.mlir_on_android_device (Failed) 422 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_rsqrt.mlir_on_android_device (Failed) 423 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_select.mlir_on_android_device (Failed) 424 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_sigmoid.mlir_on_android_device (Failed) 425 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_sub.mlir_on_android_device (Failed) 426 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_table.mlir_on_android_device (Failed) 427 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_tanh.mlir_on_android_device (Failed) 428 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_transpose.mlir_on_android_device (Failed) 429 - iree/tests/e2e/tosa_ops/check_vulkan-spirv_vulkan_while.mlir_on_android_device (Failed) 430 - iree/tests/e2e/vulkan_specific/check_vulkan-spirv_vulkan_f16_add_f16.mlir_on_android_device (Failed) 431 - iree/tests/e2e/vulkan_specific/check_vulkan-spirv_vulkan_f16_dot_f16.mlir_on_android_device (Failed) 432 - iree/tests/e2e/vulkan_specific/check_vulkan-spirv_vulkan_conv_conv.mlir_on_android_device (Failed) 433 - iree/tests/e2e/vulkan_specific/check_vulkan-spirv_vulkan_i64_emulation_add_i64.mlir_on_android_device (Failed) 434 - iree/tests/e2e/vulkan_specific/check_vulkan-spirv_vulkan_i64_emulation_mul_i64.mlir_on_android_device (Failed) 435 - iree/tools/test/iree_run_module_correctness_test_on_android_device (Failed) 519 - iree/hal/drivers/vulkan/cts/vulkan_command_buffer_test_on_android_device (Failed) 520 - iree/hal/drivers/vulkan/cts/vulkan_command_buffer_dispatch_test_on_android_device (Failed) 521 - iree/hal/drivers/vulkan/cts/vulkan_command_buffer_push_constants_test_on_android_device (Failed) 524 - iree/hal/drivers/vulkan/cts/vulkan_event_test_on_android_device (Failed) 526 - iree/hal/drivers/vulkan/cts/vulkan_file_test_on_android_device (Failed) 528 - iree/hal/drivers/vulkan/cts/vulkan_semaphore_test_on_android_device (Failed) 545 - iree/modules/check/test/check_vulkan-spirv_vulkan_success.mlir_on_android_device (Failed) 572 - iree/samples/simple_embedding/simple_embedding_vulkan_test_on_android_device (Failed) Errors while running CTest `
About this issue
- Original URL
- State: open
- Created 7 months ago
- Comments: 19
Okay, this is highlighting some gaps in our logging and error handling. The CTS tests should be giving more information.
We could log the status values here, ideally with full stack traces: https://github.com/openxla/iree/blob/f97aa4dec694c6c484d8eb35fc8dc9ba14459544/runtime/src/iree/hal/cts/cts_test_base.h#L45-L66
I’d like to know why creation is failing “gracefully” in here: https://github.com/openxla/iree/blob/main/runtime/src/iree/hal/drivers/vulkan/vulkan_driver.cc.
I don’t have an environment set up to test this on Android right now, but these flags on
iree-run-module
(e.g.iree-run-module --dump_devices
) might give more info: https://github.com/openxla/iree/blob/f97aa4dec694c6c484d8eb35fc8dc9ba14459544/runtime/src/iree/tooling/device_util.c#L169-L175I suspect this Vulkan device is missing some of the features that IREE requires (e.g. timeline semaphores - https://github.com/openxla/iree/issues/4494 has some background there, and https://github.com/KhronosGroup/Vulkan-ExtensionLayer supposedly can polyfill support for it).
As for the tests with
check_
in their names, I think those are just assuming that device creation will succeed and crashing if it fails. Those should also handle/log the errors. Might just be a matter of changing which macro is used here: https://github.com/openxla/iree/blob/f97aa4dec694c6c484d8eb35fc8dc9ba14459544/runtime/src/iree/modules/check/check_test.cc#L68-L72Ooof, that’s in LLVM, one of our more complicated dependencies. I wonder if others upstream there have issues/workarounds/docs for it.
For reference, here is a recent CI run on a Pixel 6 that includes Vulkan tests: https://github.com/openxla/iree/actions/runs/7219849588/job/19672528897 (Mostly to show that the tests can pass)