tfjs: Memory Leak with GPU & tf.node.decodeImage

Windows 10 20H2, Cuda 10.0, Cudnn v7.6.5, TFJS v2.7.0

Issue Running the following code fills my 1080s memory causing issues when performing any actual useful code

const { node: { decodeImage }, tidy, engine } = require('@tensorflow/tfjs-node-gpu')
const fs = require('fs')

async function main () {
  engine().startScope()
  const buffer = fs.readFileSync('./test.jpg')
  const tensor = tidy(() => decodeImage(Buffer.from(buffer)).toFloat().expandDims())
  tensor.dispose()
  engine().endScope()
  setTimeout(() => {
  }, 5000)
}
main()

I tried without scopes & tidy. Same result. GPU usage goes back to normal after the timeout ends & the script exists.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 5
  • Comments: 15 (4 by maintainers)

Most upvoted comments

same issue on tf-js-node cpu

+4
ColorBuffer on Mar 26, 2021

I’m having the same problem with the tfjs-node CPU backend, but only for certain JPEGs shot on iPhones. Some cause a memory leak, some don’t.

If it helps, here’s the header of a leaky image:

FF D8 FF E1 00 22 45 78 69 66 00 00 4D 4D 00 2A 00 00 00 08 00 01 01 12 00 03 00 00 00 01 00 01 00 00 00 00 00 00 FF ED 00 38 50 68 6F 74 6F 73 68 6F 70 20 33 2E 30 00 38 42 49 4D 04 04 00 00 00 00 00 00 38 42 49 4D 04 25 00 00 00 00 00 10 D4 1D 8C D9 8F 00 B2 04 E9 80 09 98 EC F8 42 7E FF E2 02 34 49 43 43 5F 50 52 4F 46 49 4C 45 00 01 01 00 00 02 24 61 70 70 6C 04 00 00 00 6D 6E 74 72 52 47 42 20 58 59 5A 20 07 E1 00 07 00 07 00 0D 00 16 00 20 61 63 73 70 41 50 50 4C 00 00 00 00 41 50 50 4C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F6 D6 00 01 00 00 00 00 D3 2D 61 70 70 6C CA 1A 95 82 25 7F 10 4D 38 99 13 D5 D1 EA 15 82 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0A 64 65 73 63 00 00 00 FC 00 00 00 65 63 70 72 74 00 00 01 64 00 00 00 23 77 74 70 74 00 00 01 88 00 00 00 14 72 58 59 5A 00 00 01 9C 00 00 00 14 67 58 59 5A 00 00 01 B0 00 00 00 14 62 58 59 5A 00 00 01 C4 00 00 00 14 72 54 52 43 00 00 01 D8 00 00 00 20 63 68 61 64 00 00 01 F8 00 00 00 2C 62 54 52 43 00 00 01 D8 00 00 00 20 67 54 52 43 00 00 01 D8 00 00 00 20 64 65 73 63 00 00 00 00 00 00 00 0B 44 69 73 70 6C 61 79 20 50 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 78 74 00 00 00 00 43 6F 70 79 72 69 67 68 74 20 41 70 70 6C 65 20 49 6E 63 2E 2C 20 32 30 31 37 00 00 58 59 5A 20 00 00 00 00 00 00 F3 51 00 01 00 00 00 01 16 CC 58 59 5A 20 00 00 00 00 00 00 83 DF 00 00 3D BF FF FF FF BB 58 59 5A 20 00 00 00 00 00 00 4A BF 00 00 B1 37 00 00 0A B9 58 59 5A 20 00 00 00 00 00 00 28 38 00 00 11 0B 00 00 C8 B9 70 61 72 61 00 00 00 00 00 03 00 00 00 02 66 66 00 00 F2 A7 00 00 0D 59 00 00 13 D0 00 00 0A 5B 73 66 33 32 00 00 00 00 00 01 0C 42 00 00 05 DE FF FF F3 26 00 00 07 93 00 00 FD 90 FF FF FB A2 FF FF FD A3 00 00 03 DC 00 00 C0 6E FF C0 00 11 08 0F C0 0B D0 03 01 22 00 02 11 01 03 11 01 FF C4 00 1F 00 00 01 05 01 01 01 01 01 01 00 00 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B FF C4 00 B5 10 00 02 01 03 03 02 04 03 05 05 04 04 00 00 01 7D 01 02 03 00 04 11 05 12 21 31 41 06 13 51 61 07 22 71 14 32 81 91 A1 08 23 42 B1 C1 15 52 D1 F0 24 33 62 72 82 09 0A 16 17 18 19 1A 25 26 27 28 29 2A 34 35 36 37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5 D6 D7 D8 D9 DA E1 E2 E3 E4 E5 E6 E7 E8 E9 EA F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FF C4 00 1F 01 00 03 01 01 01 01 01 01 01 01 01 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B FF C4 00 B5 11 00 02 01 02 04 04 03 04 07 05 04 04 00 01 02 77 00 01 02 03 11 04 05 21 31 06 12 41 51 07 61 71 13 22 32 81 08 14 42 91 A1 B1 C1 09 23 33 52 F0 15 62 72 D1 0A 16 24 34 E1 25 F1 17 18 19 1A 26 27 28 29 2A 35 36 37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 82 83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5 D6 D7 D8 D9 DA E2 E3 E4 E5 E6 E7 E8 E9 EA F2 F3 F4 F5 F6 F7 F8 F9 FA FF DB 00 43 00 04 04 04 04 04 04 06 04 04 06 09 06 06 06 09 0C 09 09 09 09 0C 0F 0C 0C 0C 0C 0C 0F 12 0F 0F 0F 0F 0F 0F 12 12 12 12 12 12 12 12 15 15 15 15 15 15 19 19 19 19 19 1C 1C 1C 1C 1C 1C 1C 1C 1C 1C FF DB 00 43 01 04 05 05 07 07 07 0C 07 07 0C 1D 14 10 14 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D FF DD 00 04 00 BD

And the header of a non-leaky image:

FF D8 FF E1 00 16 45 78 69 66 00 00 4D 4D 00 2A 00 00 00 08 00 00 00 00 00 00 FF E2 02 34 49 43 43 5F 50 52 4F 46 49 4C 45 00 01 01 00 00 02 24 61 70 70 6C 04 00 00 00 6D 6E 74 72 52 47 42 20 58 59 5A 20 07 E1 00 07 00 07 00 0D 00 16 00 20 61 63 73 70 41 50 50 4C 00 00 00 00 41 50 50 4C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F6 D6 00 01 00 00 00 00 D3 2D 61 70 70 6C CA 1A 95 82 25 7F 10 4D 38 99 13 D5 D1 EA 15 82 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0A 64 65 73 63 00 00 00 FC 00 00 00 65 63 70 72 74 00 00 01 64 00 00 00 23 77 74 70 74 00 00 01 88 00 00 00 14 72 58 59 5A 00 00 01 9C 00 00 00 14 67 58 59 5A 00 00 01 B0 00 00 00 14 62 58 59 5A 00 00 01 C4 00 00 00 14 72 54 52 43 00 00 01 D8 00 00 00 20 63 68 61 64 00 00 01 F8 00 00 00 2C 62 54 52 43 00 00 01 D8 00 00 00 20 67 54 52 43 00 00 01 D8 00 00 00 20 64 65 73 63 00 00 00 00 00 00 00 0B 44 69 73 70 6C 61 79 20 50 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 78 74 00 00 00 00 43 6F 70 79 72 69 67 68 74 20 41 70 70 6C 65 20 49 6E 63 2E 2C 20 32 30 31 37 00 00 58 59 5A 20 00 00 00 00 00 00 F3 51 00 01 00 00 00 01 16 CC 58 59 5A 20 00 00 00 00 00 00 83 DF 00 00 3D BF FF FF FF BB 58 59 5A 20 00 00 00 00 00 00 4A BF 00 00 B1 37 00 00 0A B9 58 59 5A 20 00 00 00 00 00 00 28 38 00 00 11 0B 00 00 C8 B9 70 61 72 61 00 00 00 00 00 03 00 00 00 02 66 66 00 00 F2 A7 00 00 0D 59 00 00 13 D0 00 00 0A 5B 73 66 33 32 00 00 00 00 00 01 0C 42 00 00 05 DE FF FF F3 26 00 00 07 93 00 00 FD 90 FF FF FB A2 FF FF FD A3 00 00 03 DC 00 00 C0 6E FF DB 00 84 00 09 06 07 10 10 10 1F 10 10 10 15 1A 20 20 10 0F 17 10 10 10 1E 28 20 16 1A 20 0D 28 1E 20 28 28 28 1A 28 2F 31 20 30 20 29 31 0E 1E 25 38 25 26 2E 40 41 2B 2C 28 25 2C 2E 3C 30 34 29 27 41 21 01 0A 0A 0A 0F 10 0D 0D 0D 15 11 2B 0D 15 15 29 2D 31 3B 2A 36 31 2D 33 29 33 34 25 37 3A 2D 35 32 33 2A 25 21 2B 29 31 30 36 2C 33 36 32 2F 36 37 2C 31 26 33 2C 33 25 2C 31 31 2E 26 2F 30 24 39 FF C0 00 11 08 03 6C 03 6C 03 01 22 00 02 11 01 03 11 01 FF C4 00 1C 00 00 03 01 01 01 01 01 01 00 00 00 00 00 00 00 00 00 01 02 03 04 05 06 07 08 FF C4 00 3C 10 00 02 02 01 03 02 04 05 02 05 04 01 04 02 03 01 00 01 02 11 03 04 21 31 12 41 05 51 61 81 13 22 71 91 A1 32 B1 06 14 42 C1 F0 52 D1 E1 F1 62 72 82 92 C2 23 33 15 43 A2 16 FF C4 00 18 01 01 01 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 01 02 03 04 FF C4 00 1D 11 01 01 01 01 00 03 01 01 01 00 00 00 00 00 00 00 00 01 11 02 12 21 31 41 03 61

Seems like it could have something to do with Exif data?

+2
decahedron1 on Apr 17, 2021
Read more comments on GitHub
← tfjs: Local fetch not working in android tfjs-react-native API
tfjs: method `toPixels` is not optimized for `webgl` or `webgpu` backends →