Tensile: [Bug]: MI250X performance degrades with non-uniform data

Issue description

Image

I’m in the process of profiling and optimizing an application for LUMI-G’s partition with thousands MI250X GPUs. When performing a classical matrix-to-matrix multiplication $C_{mn} = A_{mk} B_{kn} $ of square matrices ($n=k=m$) with FP64 data (doubles), I noticed something very odd with rocBlas GEMM routines of ROCm 6.0, namely:

  • I’m observing a loss of 40% performance when using non-uniform data on this classical benchmark test.

If uniform data is to be used, I recover 90% of the theoretical performance (~38 TFLOPS). This seems like a bug in the GEMM routines or a feature in the device that is out of the user control. I have reached the supercomputer support team but no satisfactory answers had been provided. Thus, I reach directly to the rocBLAS team to request an investigation.

Expected behavior

I expected to reach somewhere near to 80-90% of the theoretical performance of LUMI’s MI250X devices.

Log-files and source code

Here, I provide the source code for reproduction, output logs, CmakeLists and configuration scripts to run and reproduce data on the figure above.

Attach devBlas.zip

Additional context

This snippet was developed and tested in LUMI-G, an AMD certified supercomputer.

devBlas.zip

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (2 by maintainers)

Most upvoted comments

So rocblas-bench can be driven in two ways (command line config or yaml file config). For command line see rocblas-bench --help. –initialization hpl is how you would set the intialization. However for --yaml it is based on configuration from the rocblas_common.yaml file so you need to add an argument to your squareGEMMs_NN yaml. initialization: hpl This can be after your iters: value. This should be initialization with the uniform range [-0.5, 0.5], otherwise likely rand_int is used as the initializer so would be uniform range [0.0,10.0]. You can also verify the numeric range of your random input in your sample code to see in does not contain NaNs and matches what I would expect looked like a [0.0,1.0] generator.

The effect of “warm-up” looks weirdly large to me. @wme7, are sure you are running the same calculations? Your python script from a couple of days ago seems like it’s using single precision sgemms, while earlier you had dgemms.

@wme7 , as the other folks pointed out, rand_int is faster than trig_float as most of of the bits of the FP64 representation will be zeros. With less bit flipping you can do more compute for the same power budget. I predict that for FP32 you would have seen less of a diference but still a difference.

I’m not sure if the precision of the sampling is enough to assess the shifts in the frequency. I’d say though that should be more pronounced if you use the two dies simultaneously as they share the same power budget. Using a single die will allow it to take a bit more power than it would if the other die would be used. So, for your tunning you should eventually take this in consideration.

Yes, rand_int runs faster.

hpl and trig_float are both floating point numbers, but internally we use trig_float now, there should not be a diff in performance though. You also need to correctly set your iterations. To get the max frequency, you need to warmup the GPU (cold_iters), then to benchmark the performance, you need to set the hot iteration (iters). As a rule of thumb, try to design the # iterations so that your run each benchmark at least for 500ms ~1000ms cold, and 500ms ~1000ms hot. To do so, you can run your current benchmark once, and develop a short script to update iteration for each config (500000/us). Of course iteration counts would be different for rand_int and trig_float.

On your initialization question, performance depends on the frequency, which itslef, depends on the GPU power and temperature. If the binary representations of your numbers in the matrices have more “1” bits, it draws more power, more termperature, and lowers freq.

To collect freq while running rocblas-bench, you can use this flag: ROCBLAS_BENCH_FREQ=1.

You can also read this blog: https://rocm.blogs.amd.com/software-tools-optimization/measuring-max-achievable-flops-part2/README.html

I get you. we can initialize the matrices for rocblas-bench using three configurations:

  • hlp : with values in the range [-0.5,0.5],
  • rand_int : random integer values in the range [0.0,10.0], and
  • trig_float : discrete values within the range of [-1.0, 1.0].

With this in mind, I can also modify my snippet ( a copy is provided in the description of the issue) so that I can repeat the test with my snippet and compare with the rocblas-bench results.