tfjs: method `toPixels` is not optimized for `webgl` or `webgpu` backends

method fromPixels uses optimized code paths depending on the browser and using webgl calls results in 10x faster performance than fallback using canvas.getImageData

however, method toPixels does not have any optimizations and relies strictly on downloading tensor, creating ImageData object from it and then drawing it onto canvas
(implementation is in src/tfjs-core/src/opts/browser.ts:toPixels

for models where result is already in GPU memory, tensor download is by far the most expensive (and unnecessary) operation

for example, look at the following timing values (in ms):

prepare: 1 //includes call to fromPixels and normalizing inputs to -1…1
inference: 2 // call to await model.executeAsync
process: 0 // includes denormalizing and converting data from rgb to rgba
download: 147 // this is just await tensor.data()
draw: 2 // actual canvas.drawImage
total: 153

as you can see this may be an extreme case, but 96% of time is spent on unnecessary download of data from gpu memory

ask is to implement optimized webgl and webgpu path for toPixels method

environment: tfjs 3.19.0 with chrome 103

About this issue

Original URL
State: open
Created 2 years ago
Comments: 32 (14 by maintainers)

Most upvoted comments

@pyu10055 gpu-pipeline example is 500+ lines of code split into 6 js files (not modules) that are visible to each other only via global namespace - this kind of project structure is a nightmare to follow, opposite of what an example should be

and actual processing method takes gl texture from tensor and returns gl framebuffer that is later used for drawing on canvas, so its not even directly applicable without modifying tons of code.

let just say that my feature request stands - to provide out-of-the-box optimized toPixels method.

vladmandic on Jul 28, 2022

@FabioRomagnolo It worked for me. This solution is the best. Thanks for your codes.

19521178 on Dec 22, 2022

got a working example with single typescript module (200 loc, with full strong typing) and with a single exported method: source code: https://github.com/vladmandic/anime/blob/main/src/gldraw.ts

usage:

const data = tensor.dataToGPU({ customTexShape: [canvas.width, canvas.height] }); // get pointer to tensor texture on gpu
drawTexture(canvas, data.texture); // draw texture on canvas
tf.dispose(data.tensorRef); // dispose tensor

vladmandic on Aug 3, 2022

@vladmandic I ever wrote a PR to optimize toPixels. After some internal discussions, dataToGPU is a more powerful interface for users. So I suspended that work. It seems that it’s still useful to provide an efficient way for toPixels. One question here is what kind of canvas would you like to draw to? A 2d-canvas or let the backend to decide the canvas context (webgl/webgpu context canvas)? The original toPixels seems directly use it as a 2d-canvas.

qjia7 on Jul 28, 2022