oneDNN: Get unexpected GPU memory usage on B60 platform
Summary
The GPU memory is not released at B60 platform when executing GEMM.
The same behavior is not observed in Arc A770 platform.
Version
onednn_verbose,v1,info,oneDNN v3.6.1 (commit e72f65d70e36f552f902d35614aef7aa54f3c796)
onednn_verbose,v1,info,cpu,runtime:DPC++,nthr:120
onednn_verbose,v1,info,cpu,isa:Intel AVX-512 with float16, Intel DL Boost and bfloat16 support and Intel AMX with bfloat16 and 8-bit integer support
onednn_verbose,v1,info,gpu,runtime:DPC++
onednn_verbose,v1,info,cpu,engine,0,backend:OpenCL,name:Intel(R) Xeon(R) w9-3595X,driver_version:2024.18.12
onednn_verbose,v1,info,gpu,engine,0,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,1,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,2,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,3,backend:Level Zero,name:Intel(R) Graphics [0xe210],driver_version:1.6.32961,binary_kernels:enabled
onednn_verbose,v1,info,graph,backend,0:dnnl_backend
Environment
oneDNN includes hardware-specific optimizations and may behave differently on depending on the compiler and build environment. Include the following information to help reproduce the issue:
- CPU: Intel® Xeon® w9-3595X
- OS version (
uname -a): 6.11.0-26-generic # 26~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 17 19:20:47 UTC 2 x86_64 x86_64 x86_64 GNU/Linux - Compiler version (
gcc --version): 13.2.0 - CMake version (
cmake --version): cmake version 4.0.2
Steps to reproduce
The problem is reproducible within image: intel/oneapi-basekit:2025.0.1-0-devel-ubuntu24.04
Code to reproduce:
//onednn_kernel.cpp
/*******************************************************************************
* Copyright 2023-2024 Intel Corporation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*******************************************************************************/
#include <cassert>
#include <cctype>
#include <cmath>
#include <cstdio>
#include <iostream>
#include <random>
#include <stdexcept>
#include <vector>
#include "oneapi/dnnl/dnnl.hpp"
#ifndef DNNL_SYCL_HPP
#define DNNL_SYCL_HPP
#include "oneapi/dnnl/dnnl_sycl.hpp"
// #include <ittnotify.h>
#endif /* DNNL_SYCL_HPP */
using namespace dnnl;
namespace {
void init_vector(std::vector<float> &v) {
std::mt19937 gen;
std::uniform_real_distribution<float> u(0, 1);
for (auto &e : v)
e = u(gen);
}
} // namespace
int number_of_runs = 100;
// Create a MatMul primitive descriptor for the following op:
// C_f32 = A_f32 * (B_u4 - zp_B) * sc_B[:]
//
// Here:
// - Matrices A and C are of f32 data type.
// - The B matrix is stored as int8_t, its zero point is zp_B, and all its
// dimensions are known. This matrix can be a matrix of compressed weights
// in an MLP topology.
// - The weights scaling values are not known at the primitive creation time.
matmul::primitive_desc matmul_pd_create(
int64_t M, int64_t N, int64_t K, int64_t G, const engine &eng) {
memory::desc a_md({M, K}, memory::data_type::f32, {K, 1}); // M x K layout
memory::desc b_md({K, N}, memory::data_type::u4, memory::format_tag::any);
memory::desc c_md({M, N}, memory::data_type::f32, {N, 1}); // M x N layout
// Create attributes and indicate that the alpha and zero points are
// runtime parameters
primitive_attr attr;
// Set scales with multiple scales along K and N dimensions and with groups along K.
attr.set_scales(DNNL_ARG_WEIGHTS,
/* mask */ (1 << 0) + (1 << 1), {G, 1}, memory::data_type::f32);
// Set a single zero point with u4 data type.
attr.set_zero_points(
DNNL_ARG_WEIGHTS, /* mask */ 0, {}, memory::data_type::u8);
// Set fpmath mode with `apply_to_int=true` to apply fpmath mode behavior to
// integral primitives (in this example, matmul).
attr.set_fpmath_mode(fpmath_mode::bf16, true);
// Create a MatMul primitive descriptor
return matmul::primitive_desc(eng, a_md, b_md, c_md, attr);
}
// Read from handle, write to memory
inline void write_to_dnnl_memory(void *handle, dnnl::memory &mem) {
dnnl::engine eng = mem.get_engine();
size_t size = mem.get_desc().get_size();
if (!handle) throw std::runtime_error("handle is nullptr.");
#ifdef DNNL_WITH_SYCL
bool is_cpu_sycl = (DNNL_CPU_RUNTIME == DNNL_RUNTIME_SYCL
&& eng.get_kind() == dnnl::engine::kind::cpu);
bool is_gpu_sycl = (DNNL_GPU_RUNTIME == DNNL_RUNTIME_SYCL
&& eng.get_kind() == dnnl::engine::kind::gpu);
if (is_cpu_sycl || is_gpu_sycl) {
auto mkind = dnnl::sycl_interop::get_memory_kind(mem);
if (mkind == dnnl::sycl_interop::memory_kind::buffer) {
auto buffer = dnnl::sycl_interop::get_buffer<uint8_t>(mem);
auto dst = buffer.get_host_access();
uint8_t *dst_ptr = dst.get_pointer();
if (!dst_ptr)
throw std::runtime_error("get_pointer returned nullptr.");
for (size_t i = 0; i < size; ++i)
dst_ptr[i] = ((uint8_t *)handle)[i];
} else {
assert(mkind == dnnl::sycl_interop::memory_kind::usm);
uint8_t *dst_ptr = (uint8_t *)mem.get_data_handle();
if (!dst_ptr)
throw std::runtime_error("get_data_handle returned nullptr.");
if (is_cpu_sycl) {
for (size_t i = 0; i < size; ++i)
dst_ptr[i] = ((uint8_t *)handle)[i];
} else {
auto sycl_queue
= dnnl::sycl_interop::get_queue(dnnl::stream(eng));
sycl_queue.memcpy(dst_ptr, handle, size).wait();
}
}
return;
}
#endif
#if DNNL_GPU_RUNTIME == DNNL_RUNTIME_OCL
if (eng.get_kind() == dnnl::engine::kind::gpu) {
void *mapped_ptr = mem.map_data();
if (mapped_ptr) std::memcpy(mapped_ptr, handle, size);
mem.unmap_data(mapped_ptr);
return;
}
#endif
if (eng.get_kind() == dnnl::engine::kind::cpu) {
uint8_t *dst = static_cast<uint8_t *>(mem.get_data_handle());
if (!dst) throw std::runtime_error("get_data_handle returned nullptr.");
for (size_t i = 0; i < size; ++i)
dst[i] = ((uint8_t *)handle)[i];
return;
}
assert(!"not expected");
}
void prepare_input(memory &A_f32_mem, memory &sc_B_mem, memory &zp_B_mem) {
int64_t M = A_f32_mem.get_desc().get_dims()[0];
int64_t N = sc_B_mem.get_desc().get_dims()[0];
int64_t K = A_f32_mem.get_desc().get_dims()[1];
int64_t NUM_G = sc_B_mem.get_desc().get_dims()[1];
std::vector<float> A_f32(M * K);
init_vector(A_f32);
std::vector<float> sc_B(NUM_G * N);
init_vector(sc_B);
uint8_t zp_B = 0;
write_to_dnnl_memory(A_f32.data(), A_f32_mem);
write_to_dnnl_memory(&zp_B, zp_B_mem);
write_to_dnnl_memory(sc_B.data(), sc_B_mem);
}
using fp16=sycl::half;
#include <chrono>
void infer(const matmul &matmul_p, int64_t M, int64_t N, int64_t K, int64_t G,
const memory &B_u4_mem, const engine &eng) {
// input of the current layer / operation
memory A_f32_mem({{M, K}, memory::data_type::f32, {K, 1}}, eng);
// De-quantization parameters (eg. Scale and Shift)
const int64_t n_groups = K / G;
memory sc_B_mem({{N, n_groups}, memory::data_type::f32, {1, N}}, eng);
memory zp_B_mem({{1}, memory::data_type::f32, {1}}, eng);
// the function below fills dnnl::memory with some values
// these memories, typically, come from the previous layers / operations
// with meaningful data inside
prepare_input(A_f32_mem, sc_B_mem, zp_B_mem);
// output - no initialization required
memory C_f32_mem({{M, N}, memory::data_type::f32, {N, 1}}, eng);
stream s(eng);
// __itt_string_handle* first_task_handle =
// __itt_string_handle_create("ondnn s4 matmul");
// if(domain)__itt_task_begin(domain, __itt_null, __itt_null, first_task_handle);
auto start = std::chrono::high_resolution_clock::now();
for (int run = 0; run < number_of_runs; ++run){
matmul_p.execute(s,
{{DNNL_ARG_SRC, A_f32_mem}, {DNNL_ARG_WEIGHTS, B_u4_mem},
{DNNL_ARG_DST, C_f32_mem},
{DNNL_ARG_ATTR_SCALES | DNNL_ARG_WEIGHTS, sc_B_mem},
{DNNL_ARG_ATTR_ZERO_POINTS | DNNL_ARG_WEIGHTS,
zp_B_mem}});
s.wait();
}
auto end = std::chrono::high_resolution_clock::now();
auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
std::cout << "ondnn s4 matmul execution time: " << duration.count() << " ms" << std::endl;
// if(domain)__itt_task_end(domain);
}
void linear_xpu_forward_onednn_s4_gemv(
size_t input_size,
size_t state_size,
size_t output_size,
sycl::queue& q
// __itt_domain* domain
){
engine eng(dnnl::engine::kind::gpu, 0);
const int64_t K = state_size;
const int64_t N = output_size;
const int64_t M = input_size;
// Quantization Group size for scales. Must be divisible by 32.
const int64_t G = 64;
auto matmul_pd = matmul_pd_create(M, N, K, G, eng);
// Original weights stored as float in a known format
std::vector<float> B_f32(K * N);
init_vector(B_f32);
// Pre-packed weights stored as int8_t
memory B_u4_mem(matmul_pd.weights_desc(), eng);
{
stream s(eng);
memory B_f32_mem(
{{K, N}, memory::data_type::f32, memory::format_tag::ab}, eng);
write_to_dnnl_memory(B_f32.data(), B_f32_mem);
reorder(B_f32_mem, B_u4_mem).execute(s, B_f32_mem, B_u4_mem);
s.wait();
}
matmul matmul_p(matmul_pd);
infer(matmul_p, M, N, K, G, B_u4_mem, eng);
}
int main()
{
try
{
sycl::gpu_selector selector;
sycl::queue q(selector);
// 模拟的输入参数(可以根据实际模型修改)
const size_t M = 4096;
const size_t K = 10240;
const size_t N = 8192;
for (int i = 0; i < 100; ++i)
{
std::cout << "Run #" << i + 1 << std::endl;
linear_xpu_forward_onednn_s4_gemv(
M,
K,
N,
q);
}
// std::cout << "Execution finished successfully.\n";
}
catch (const std::exception &e)
{
std::cerr << "Error occurred: " << e.what() << std::endl;
return 1;
}
return 0;
}
icpx -fsycl -std=c++20 onednn_kernel.cpp -ldnnl -ltbb -o example
./example
Observed behavior
At each run, we can observed that the GPU memory usage is increased a little bit and never get released. I am not sure if this is the correct behavior or not.
Expected behavior
Memory get released.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 20 (6 by maintainers)
Hi, I have reproduced your result. Thank you for your help~
I guess use the newest oneapi with this specific version of oneDNN together fixes the problem.
I will close this issue once I have verified this in our workload~
@gc-fu, I was able to reproduce a memory leak in the oneapi-2025.1 release, but it seems all memory leaks have subsequently been fixed on the oneDNN v3.8 branch. Could you check to see if you get the same results from the following process?
To begin with, I installed the oneapi-2025.1 basekit via apt on a machine with a battlemage GPU. I also modified the reproducer to log memory usage with the following:
I then compiled and ran this against the basekit oneDNN distribution which demonstrates a memory leak:
I subsequently rebuilt the reproducer to use a oneDNN rls-v3.8 build, which showed that the memory has been resolved as memory usage stabilized after 2 iterations:
I additionally checked running
bug_dnnl_v3.8with a primitive cache size of 2, and saw memory use stabilize after 6 iterations, so there do not appear to be any leaks associated with the primitive cache either.Yep, I double-checked the environment variable:
Also, my oneDNN branch:
We are investigating on our end as well and will report back as soon as we find the root cause.