oneDNN: Get unexpected GPU memory usage on B60 platform

Summary

The GPU memory is not released at B60 platform when executing GEMM.

The same behavior is not observed in Arc A770 platform.

Version

onednn_verbose,v1,info,oneDNN v3.6.1 (commit e72f65d70e36f552f902d35614aef7aa54f3c796)
onednn_verbose,v1,info,cpu,runtime:DPC++,nthr:120
onednn_verbose,v1,info,cpu,isa:Intel AVX-512 with float16, Intel DL Boost and bfloat16 support and Intel AMX with bfloat16 and 8-bit integer support
onednn_verbose,v1,info,gpu,runtime:DPC++
onednn_verbose,v1,info,cpu,engine,0,backend:OpenCL,name:Intel(R) Xeon(R) w9-3595X,driver_version:2024.18.12
onednn_verbose,v1,info,gpu,engine,0,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,1,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,2,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,3,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,graph,backend,0:dnnl_backend

Environment

oneDNN includes hardware-specific optimizations and may behave differently on depending on the compiler and build environment. Include the following information to help reproduce the issue:

  • CPU: Intel® Xeon® w9-3595X
  • OS version (uname -a): 6.11.0-26-generic # 26~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 17 19:20:47 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
  • Compiler version (gcc --version): 13.2.0
  • CMake version (cmake --version): cmake version 4.0.2

Steps to reproduce

The problem is reproducible within image: intel/oneapi-basekit:2025.0.1-0-devel-ubuntu24.04

Code to reproduce:

//onednn_kernel.cpp

/*******************************************************************************
* Copyright 2023-2024 Intel Corporation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
*     http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*******************************************************************************/


#include <cassert>
#include <cctype>
#include <cmath>
#include <cstdio>
#include <iostream>
#include <random>
#include <stdexcept>
#include <vector>

#include "oneapi/dnnl/dnnl.hpp"
#ifndef DNNL_SYCL_HPP
#define DNNL_SYCL_HPP

#include "oneapi/dnnl/dnnl_sycl.hpp"
// #include <ittnotify.h>

#endif /* DNNL_SYCL_HPP */

using namespace dnnl;

namespace {

void init_vector(std::vector<float> &v) {
    std::mt19937 gen;
    std::uniform_real_distribution<float> u(0, 1);
    for (auto &e : v)
        e = u(gen);
}

} // namespace

int number_of_runs = 100;

// Create a MatMul primitive descriptor for the following op:
// C_f32 = A_f32 * (B_u4 - zp_B) * sc_B[:]
//
// Here:
// - Matrices A and C are of f32 data type.
// - The B matrix is stored as int8_t, its zero point is zp_B, and all its
//   dimensions are known. This matrix can be a matrix of compressed weights
//   in an MLP topology.
// - The weights scaling values are not known at the primitive creation time.
matmul::primitive_desc matmul_pd_create(
        int64_t M, int64_t N, int64_t K, int64_t G, const engine &eng) {

    memory::desc a_md({M, K}, memory::data_type::f32, {K, 1}); // M x K layout
    memory::desc b_md({K, N}, memory::data_type::u4, memory::format_tag::any);
    memory::desc c_md({M, N}, memory::data_type::f32, {N, 1}); // M x N layout

    // Create attributes and indicate that the alpha and zero points are
    // runtime parameters
    primitive_attr attr;
    // Set scales with multiple scales along K and N dimensions and with groups along K.
    attr.set_scales(DNNL_ARG_WEIGHTS,
            /* mask */ (1 << 0) + (1 << 1), {G, 1}, memory::data_type::f32);
    // Set a single zero point with u4 data type.
    attr.set_zero_points(
            DNNL_ARG_WEIGHTS, /* mask */ 0, {}, memory::data_type::u8);
    // Set fpmath mode with `apply_to_int=true` to apply fpmath mode behavior to
    // integral primitives (in this example, matmul).
    attr.set_fpmath_mode(fpmath_mode::bf16, true);

    // Create a MatMul primitive descriptor
    return matmul::primitive_desc(eng, a_md, b_md, c_md, attr);
}

// Read from handle, write to memory
inline void write_to_dnnl_memory(void *handle, dnnl::memory &mem) {
    dnnl::engine eng = mem.get_engine();
    size_t size = mem.get_desc().get_size();

    if (!handle) throw std::runtime_error("handle is nullptr.");

#ifdef DNNL_WITH_SYCL
    bool is_cpu_sycl = (DNNL_CPU_RUNTIME == DNNL_RUNTIME_SYCL
            && eng.get_kind() == dnnl::engine::kind::cpu);
    bool is_gpu_sycl = (DNNL_GPU_RUNTIME == DNNL_RUNTIME_SYCL
            && eng.get_kind() == dnnl::engine::kind::gpu);
    if (is_cpu_sycl || is_gpu_sycl) {
        auto mkind = dnnl::sycl_interop::get_memory_kind(mem);
        if (mkind == dnnl::sycl_interop::memory_kind::buffer) {
            auto buffer = dnnl::sycl_interop::get_buffer<uint8_t>(mem);
            auto dst = buffer.get_host_access();
            uint8_t *dst_ptr = dst.get_pointer();
            if (!dst_ptr)
                throw std::runtime_error("get_pointer returned nullptr.");
            for (size_t i = 0; i < size; ++i)
                dst_ptr[i] = ((uint8_t *)handle)[i];
        } else {
            assert(mkind == dnnl::sycl_interop::memory_kind::usm);
            uint8_t *dst_ptr = (uint8_t *)mem.get_data_handle();
            if (!dst_ptr)
                throw std::runtime_error("get_data_handle returned nullptr.");
            if (is_cpu_sycl) {
                for (size_t i = 0; i < size; ++i)
                    dst_ptr[i] = ((uint8_t *)handle)[i];
            } else {
                auto sycl_queue
                        = dnnl::sycl_interop::get_queue(dnnl::stream(eng));
                sycl_queue.memcpy(dst_ptr, handle, size).wait();
            }
        }
        return;
    }
#endif
#if DNNL_GPU_RUNTIME == DNNL_RUNTIME_OCL
    if (eng.get_kind() == dnnl::engine::kind::gpu) {
        void *mapped_ptr = mem.map_data();
        if (mapped_ptr) std::memcpy(mapped_ptr, handle, size);
        mem.unmap_data(mapped_ptr);
        return;
    }
#endif

    if (eng.get_kind() == dnnl::engine::kind::cpu) {
        uint8_t *dst = static_cast<uint8_t *>(mem.get_data_handle());
        if (!dst) throw std::runtime_error("get_data_handle returned nullptr.");
        for (size_t i = 0; i < size; ++i)
            dst[i] = ((uint8_t *)handle)[i];
        return;
    }

    assert(!"not expected");
}


void prepare_input(memory &A_f32_mem, memory &sc_B_mem, memory &zp_B_mem) {
    int64_t M = A_f32_mem.get_desc().get_dims()[0];
    int64_t N = sc_B_mem.get_desc().get_dims()[0];
    int64_t K = A_f32_mem.get_desc().get_dims()[1];
    int64_t NUM_G = sc_B_mem.get_desc().get_dims()[1];

    std::vector<float> A_f32(M * K);
    init_vector(A_f32);

    std::vector<float> sc_B(NUM_G * N);
    init_vector(sc_B);

    uint8_t zp_B = 0;

    write_to_dnnl_memory(A_f32.data(), A_f32_mem);
    write_to_dnnl_memory(&zp_B, zp_B_mem);
    write_to_dnnl_memory(sc_B.data(), sc_B_mem);
}

using fp16=sycl::half;
#include <chrono>
void infer(const matmul &matmul_p, int64_t M, int64_t N, int64_t K, int64_t G,
        const memory &B_u4_mem, const engine &eng) {
    // input of the current layer / operation
    memory A_f32_mem({{M, K}, memory::data_type::f32, {K, 1}}, eng);
    // De-quantization parameters (eg. Scale and Shift)
    const int64_t n_groups = K / G;
    memory sc_B_mem({{N, n_groups}, memory::data_type::f32, {1, N}}, eng);
    memory zp_B_mem({{1}, memory::data_type::f32, {1}}, eng);

    // the function below fills dnnl::memory with some values
    // these memories, typically, come from the previous layers / operations
    // with meaningful data inside
    prepare_input(A_f32_mem, sc_B_mem, zp_B_mem);

    // output - no initialization required
    memory C_f32_mem({{M, N}, memory::data_type::f32, {N, 1}}, eng);

    stream s(eng);
    // __itt_string_handle* first_task_handle =
    //   __itt_string_handle_create("ondnn s4 matmul");
    // if(domain)__itt_task_begin(domain, __itt_null, __itt_null, first_task_handle);
    auto start = std::chrono::high_resolution_clock::now();
    for (int run = 0; run < number_of_runs; ++run){
        matmul_p.execute(s,
                {{DNNL_ARG_SRC, A_f32_mem}, {DNNL_ARG_WEIGHTS, B_u4_mem},
                        {DNNL_ARG_DST, C_f32_mem},
                        {DNNL_ARG_ATTR_SCALES | DNNL_ARG_WEIGHTS, sc_B_mem},
                        {DNNL_ARG_ATTR_ZERO_POINTS | DNNL_ARG_WEIGHTS,
                                zp_B_mem}});
        s.wait();
    }

    auto end = std::chrono::high_resolution_clock::now();
    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
    std::cout << "ondnn s4 matmul execution time: " << duration.count() << " ms" << std::endl;
    // if(domain)__itt_task_end(domain);
}

void linear_xpu_forward_onednn_s4_gemv(
        size_t input_size,
        size_t state_size,
        size_t output_size,
        sycl::queue& q
        //  __itt_domain* domain
){
    engine eng(dnnl::engine::kind::gpu, 0);

    const int64_t K = state_size;
    const int64_t N = output_size;
    const int64_t M = input_size;
    // Quantization Group size for scales. Must be divisible by 32.
    const int64_t G = 64;

    auto matmul_pd = matmul_pd_create(M, N, K, G, eng);

    // Original weights stored as float in a known format
    std::vector<float> B_f32(K * N);
    init_vector(B_f32);

    // Pre-packed weights stored as int8_t
    memory B_u4_mem(matmul_pd.weights_desc(), eng);
    {
        stream s(eng);
        memory B_f32_mem(
                {{K, N}, memory::data_type::f32, memory::format_tag::ab}, eng);
        write_to_dnnl_memory(B_f32.data(), B_f32_mem);
        reorder(B_f32_mem, B_u4_mem).execute(s, B_f32_mem, B_u4_mem);
        s.wait();
    }

    matmul matmul_p(matmul_pd);

    infer(matmul_p, M, N, K, G, B_u4_mem, eng);  
}

int main()
{
    try
    {
        sycl::gpu_selector selector;
        sycl::queue q(selector);


        // 模拟的输入参数(可以根据实际模型修改)
        const size_t M = 4096;
        const size_t K = 10240;
        const size_t N = 8192; 



        for (int i = 0; i < 100; ++i)
        {
            std::cout << "Run #" << i + 1 << std::endl;
            linear_xpu_forward_onednn_s4_gemv(
                M,
                K,
                N,
                q);
        }

        // std::cout << "Execution finished successfully.\n";
    }
    catch (const std::exception &e)
    {
        std::cerr << "Error occurred: " << e.what() << std::endl;
        return 1;
    }

    return 0;
}
icpx -fsycl -std=c++20 onednn_kernel.cpp   -ldnnl -ltbb   -o example
./example

Observed behavior

At each run, we can observed that the GPU memory usage is increased a little bit and never get released. I am not sure if this is the correct behavior or not.

Expected behavior

Memory get released.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 20 (6 by maintainers)

Most upvoted comments

Hi, I have reproduced your result. Thank you for your help~

I guess use the newest oneapi with this specific version of oneDNN together fixes the problem.

I will close this issue once I have verified this in our workload~

@gc-fu, I was able to reproduce a memory leak in the oneapi-2025.1 release, but it seems all memory leaks have subsequently been fixed on the oneDNN v3.8 branch. Could you check to see if you get the same results from the following process?

To begin with, I installed the oneapi-2025.1 basekit via apt on a machine with a battlemage GPU. I also modified the reproducer to log memory usage with the following:

diff --git a/examples/bug.cpp b/examples/bug.cpp
index 482318ca65..851162b24b 100644
--- a/examples/bug.cpp
+++ b/examples/bug.cpp
@@ -26,6 +26,8 @@
 #include <stdexcept>
 #include <vector>

+#include <level_zero/zes_api.h>
+
 #include "oneapi/dnnl/dnnl.hpp"
 #ifndef DNNL_SYCL_HPP
 #define DNNL_SYCL_HPP
@@ -50,6 +52,53 @@ void init_vector(std::vector<float> &v) {

 int number_of_runs = 100;

+void dump_memory_usage() {
+    static const zes_mem_handle_t memory = []() {
+        if (zesInit(0) != ZE_RESULT_SUCCESS) {
+            std::cerr << "Failed to initialize Sysman\n";
+            exit(1);
+        }
+
+        uint32_t driverCount = 0;
+        zesDriverGet(&driverCount, nullptr);
+        std::vector<zes_driver_handle_t> drivers(driverCount);
+        zesDriverGet(&driverCount, drivers.data());
+
+        if (driverCount != 1) {
+            std::cerr << "Unsupported driver count\n";
+            exit(1);
+        }
+        int idx = 0;
+        uint32_t deviceCount = 0;
+        zesDeviceGet(drivers[idx], &deviceCount, nullptr);
+        std::vector<zes_device_handle_t> devices(deviceCount);
+        zesDeviceGet(drivers[idx], &deviceCount, devices.data());
+        if (deviceCount != 1) {
+            std::cerr << "Unsupported device count\n";
+            exit(1);
+        }
+
+        uint32_t memoryCount = 0;
+        zesDeviceEnumMemoryModules(devices[idx], &memoryCount, nullptr);
+        std::vector<zes_mem_handle_t> memories(memoryCount);
+        zesDeviceEnumMemoryModules(devices[idx], &memoryCount, memories.data());
+        if (memoryCount != 1) {
+            std::cerr << "Unsupported memory count\n";
+            exit(1);
+        }
+        return memories[0];
+    }();
+
+    zes_mem_state_t mem_state = {};
+    mem_state.stype = ZES_STRUCTURE_TYPE_MEM_STATE;
+    if (zesMemoryGetState(memory, &mem_state) != ZE_RESULT_SUCCESS) {
+        std::cerr << "Failed get get memory state\n";
+        exit(1);
+    }
+    printf("Memory Usage: Used - %ld, Free - %ld\n",
+            mem_state.size - mem_state.free, mem_state.free);
+}
+
 // Create a MatMul primitive descriptor for the following op:
 // C_f32 = A_f32 * (B_u4 - zp_B) * sc_B[:]
 //
@@ -241,6 +290,7 @@ void linear_xpu_forward_onednn_s4_gemv(

 int main()
 {
+    dump_memory_usage();
     try
     {
         sycl::gpu_selector selector;
@@ -262,6 +312,7 @@ int main()
                 K,
                 N,
                 q);
+            dump_memory_usage();
         }

         // std::cout << "Execution finished successfully.\n";

I then compiled and ran this against the basekit oneDNN distribution which demonstrates a memory leak:

$ source /opt/intel/oneapi/2025.1/oneapi-vars.sh
$ icpx -I -fsycl -std=c++20 bug.cpp -ldnnl -ltbb -lOpenCL -lsycl -lze_loader -Wl,--disable-new-dtags,-rpath,${LD_LIBRARY_PATH} -o bug_2025.1
$ sudo setcap cap_perfmon=ep bug_2025.1 # Enable permissions to access memory usage
$ ONEDNN_PRIMITIVE_CACHE_CAPACITY=0 ./bug_2025.1
Memory Usage: Used - 30814208, Free - 12778590208
Run #1
ondnn s4 matmul execution time: 5820 ms
Memory Usage: Used - 569016320, Free - 12240388096
Run #2
ondnn s4 matmul execution time: 5800 ms
Memory Usage: Used - 581226496, Free - 12228177920
Run #3
ondnn s4 matmul execution time: 5807 ms
Memory Usage: Used - 593436672, Free - 12215967744
Run #4
ondnn s4 matmul execution time: 5821 ms
Memory Usage: Used - 605646848, Free - 12203757568
Run #5
ondnn s4 matmul execution time: 5834 ms
Memory Usage: Used - 617857024, Free - 12191547392

I subsequently rebuilt the reproducer to use a oneDNN rls-v3.8 build, which showed that the memory has been resolved as memory usage stabilized after 2 iterations:

$ source /opt/intel/oneapi/2025.1/oneapi-vars.sh
$ icpx -I -fsycl -v -std=c++20 bug.cpp -ldnnl -ltbb -lOpenCL -lsycl -lze_loader -Wl,--disable-new-dtags,-rpath,<dnnl_v3.8_build>/src:${LD_LIBRARY_PATH} -o bug_dnnl_v3.8 && sudo setcap cap_perfmon=ep bug_dnnl_v3.8
$ ONEDNN_PRIMITIVE_CACHE_CAPACITY=0 ./bug_dnnl_v3.8
Memory Usage: Used - 30814208, Free - 12778590208
Run #1
ondnn s4 matmul execution time: 5819 ms
Memory Usage: Used - 563564544, Free - 12245839872
Run #2
ondnn s4 matmul execution time: 5815 ms
Memory Usage: Used - 564686848, Free - 12244717568
Run #3
ondnn s4 matmul execution time: 5834 ms
Memory Usage: Used - 564686848, Free - 12244717568
Run #4
ondnn s4 matmul execution time: 5843 ms
Memory Usage: Used - 564686848, Free - 12244717568
Run #5
ondnn s4 matmul execution time: 5848 ms
Memory Usage: Used - 564686848, Free - 12244717568

I additionally checked running bug_dnnl_v3.8 with a primitive cache size of 2, and saw memory use stabilize after 6 iterations, so there do not appear to be any leaks associated with the primitive cache either.

@gc-fu Thanks for checking, just in case, are you testing rls-v3.8? Older oneDNN versions had a memory leak for that particular scenario with different engines that was fixed in 71b25d3.

Yep, I double-checked the environment variable:

Image

Also, my oneDNN branch:

root@ws-arc-002:/llm/oneDNN# git branch
  main
* rls-v3.8

We are investigating on our end as well and will report back as soon as we find the root cause.