onnxruntime: [WebGPU] DETR Object Detection Model extremely slow due to multiple copies between CPU and GPU

Describe the issue

As described in the title, using the WebGPU execution provider is extremely slow for this DETR object detection model: https://huggingface.co/Xenova/detr-resnet-50. The unquantized version is around 160MB (quantized version is 40MB, but I am not testing that one here).

In comparison with the WASM EP, it’s around 2 times slower (WebGPU is ~14 seconds, WASM is ~8 seconds).

Enabling logging and profiling shows that this is due to numerous copies between the GPU and CPU during model execution.

Log file: webgpu_detr.log

Although, most of the time (12 seconds) can be traced back to a single copy (line 38689 of the log file):

    log.ts:17 [V,2023-11-26T17:52:10.647Z][WebGPU] GpuDataManager.release(id=815), gpuDataId=815
>>> log.ts:17 [V,2023-11-26T17:52:10.648Z][WebGPU] jsepCopyGpuToCpu: gpuDataId=816, dataOffset=273302656, size=625
    log.ts:17 [V,2023-11-26T17:52:22.976Z][WebGPU] GpuDataManager.release(id=816), gpuDataId=816
    log.ts:17 [V,2023-11-26T17:52:22.976Z][WebGPU] GpuDataManager.create(size=625) => id=817
    log.ts:17 [V,2023-11-26T17:52:22.976Z][WebGPU] jsepCopyCpuToGpu: dataOffset=345775616, gpuDataId=817, size=625
    log.ts:17 [V,2023-11-26T17:52:22.977Z][WebGPU] GpuDataManager.upload(id=817)
    log.ts:17 [V,2023-11-26T17:52:22.977Z][WebGPU] jsepRun: kernel=343388760, contextDataOffset=343725312

@fs-eire @guschmue

To reproduce

import 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.16.3/dist/ort.webgpu.min.js'

ort.env.wasm.wasmPaths='https://cdn.jsdelivr.net/npm/onnxruntime-web@1.16.3/dist/'

ort.env.logLevel = 'verbose';
ort.env.debug = true;
ort.env.webgpu.profilingMode = 'default';


const options = {
    executionProviders: ['webgpu'],
    graphOptimizationLevel: 'disabled',
}


const response = await fetch('https://huggingface.co/Xenova/detr-resnet-50/resolve/main/onnx/model.onnx');
const buffer = new Uint8Array(await response.arrayBuffer());


const session = await ort.InferenceSession.create(buffer, options);
console.log(session);

// Dummy inputs
const inputs = {
    pixel_values: new ort.Tensor('float32', new Float32Array(1 * 3 * 800 * 800), [1, 3, 800, 800]),
    pixel_mask: new ort.Tensor('int64', new BigInt64Array(1 * 64 * 64).fill(1n), [1, 64, 64])
}

const start = performance.now();
const output = await session.run(inputs);
const end = performance.now();
console.log(end - start, output);

Urgency

High - blocks WebGPU usage in Transformers.js

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.16.3

Execution Provider

‘webgpu’ (WebGPU)