tensorflow: Memory leaks in repeated model training despite garbage collection and session clearing

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes, see below
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows and Linux
  • TensorFlow installed from (source or binary): pip
  • TensorFlow version (use command below): v2.3.0-54-gfcc4b966f1 2.3.1
  • Python version: 3.8.5
  • CUDA/cuDNN version: none
  • GPU model and memory: none

Describe the current behavior The code below leaks memory.

From my local, CPU-only system:

iteration 0: rss 368 MB
iteration 1: rss 400 MB  # new maximum
iteration 2: rss 402 MB  # new maximum
iteration 3: rss 439 MB  # new maximum
iteration 4: rss 431 MB
iteration 5: rss 444 MB  # new maximum
iteration 6: rss 442 MB
iteration 7: rss 447 MB  # new maximum
iteration 8: rss 445 MB
iteration 9: rss 461 MB
iteration 10: rss 449 MB  # new maximum
iteration 11: rss 462 MB  # new maximum
iteration 12: rss 460 MB
iteration 13: rss 466 MB  # new maximum
iteration 14: rss 463 MB
iteration 15: rss 483 MB  # new maximum
iteration 16: rss 468 MB
iteration 17: rss 490 MB  # new maximum
iteration 18: rss 476 MB
iteration 19: rss 489 MB
iteration 20: rss 478 MB
iteration 21: rss 491 MB  # new maximum
iteration 22: rss 480 MB
iteration 23: rss 492 MB  # new maximum
iteration 24: rss 480 MB
iteration 25: rss 497 MB  # new maximum
iteration 26: rss 481 MB
iteration 27: rss 493 MB
iteration 28: rss 483 MB
iteration 29: rss 493 MB
iteration 30: rss 486 MB
iteration 31: rss 488 MB
iteration 32: rss 487 MB
iteration 33: rss 489 MB
iteration 34: rss 498 MB  # new maximum
iteration 35: rss 487 MB
iteration 36: rss 494 MB
iteration 37: rss 493 MB
iteration 38: rss 489 MB
iteration 39: rss 500 MB  # new maximum
iteration 40: rss 487 MB
iteration 41: rss 501 MB  # new maximum
iteration 42: rss 494 MB
iteration 43: rss 493 MB
iteration 44: rss 502 MB  # new maximum
iteration 45: rss 510 MB  # new maximum
iteration 46: rss 503 MB
iteration 47: rss 497 MB
iteration 48: rss 494 MB
iteration 49: rss 507 MB
iteration 50: rss 497 MB
iteration 51: rss 507 MB
iteration 52: rss 499 MB
iteration 53: rss 499 MB
iteration 54: rss 507 MB
iteration 55: rss 497 MB
iteration 56: rss 507 MB
iteration 57: rss 500 MB
iteration 58: rss 501 MB
iteration 59: rss 508 MB
iteration 60: rss 498 MB
iteration 61: rss 507 MB
iteration 62: rss 501 MB
iteration 63: rss 502 MB
iteration 64: rss 510 MB
iteration 65: rss 501 MB
iteration 66: rss 511 MB  # new maximum
iteration 67: rss 503 MB
iteration 68: rss 502 MB
iteration 69: rss 512 MB
iteration 70: rss 500 MB
iteration 71: rss 512 MB  # new maximum
iteration 72: rss 505 MB
iteration 73: rss 506 MB
iteration 74: rss 512 MB
iteration 75: rss 522 MB  # new maximum
iteration 76: rss 509 MB
iteration 77: rss 507 MB
iteration 78: rss 507 MB
iteration 79: rss 514 MB
iteration 80: rss 525 MB  # new maximum
iteration 81: rss 514 MB
iteration 82: rss 508 MB
iteration 83: rss 507 MB
iteration 84: rss 516 MB
iteration 85: rss 527 MB  # new maximum
iteration 86: rss 513 MB
iteration 87: rss 511 MB
iteration 88: rss 509 MB
iteration 89: rss 516 MB
iteration 90: rss 508 MB
iteration 91: rss 511 MB
iteration 92: rss 514 MB
iteration 93: rss 509 MB
iteration 94: rss 518 MB
iteration 95: rss 508 MB
iteration 96: rss 513 MB
iteration 97: rss 514 MB
iteration 98: rss 509 MB
iteration 99: rss 519 MB

From Colab:

iteration 0: rss 547 MB
iteration 1: rss 652 MB  # new maximum
iteration 2: rss 671 MB  # new maximum
iteration 3: rss 674 MB  # new maximum
iteration 4: rss 674 MB
iteration 5: rss 674 MB
iteration 6: rss 674 MB
iteration 7: rss 675 MB  # new maximum
iteration 8: rss 680 MB  # new maximum
iteration 9: rss 680 MB
iteration 10: rss 689 MB  # new maximum
iteration 11: rss 689 MB
iteration 12: rss 689 MB
iteration 13: rss 689 MB
iteration 13: rss 689 MB
iteration 14: rss 689 MB
iteration 15: rss 689 MB
iteration 16: rss 689 MB
iteration 17: rss 689 MB
iteration 18: rss 689 MB
iteration 19: rss 689 MB
iteration 20: rss 689 MB
iteration 21: rss 689 MB
iteration 22: rss 689 MB
iteration 23: rss 689 MB
iteration 24: rss 689 MB
iteration 25: rss 689 MB
iteration 26: rss 689 MB
iteration 27: rss 689 MB
iteration 28: rss 689 MB
iteration 29: rss 689 MB
iteration 30: rss 689 MB
iteration 31: rss 689 MB
iteration 32: rss 689 MB
iteration 33: rss 689 MB
iteration 34: rss 689 MB
iteration 35: rss 691 MB  # new maximum
iteration 36: rss 691 MB
iteration 37: rss 697 MB  # new maximum
iteration 38: rss 702 MB  # new maximum
iteration 39: rss 704 MB  # new maximum
iteration 40: rss 704 MB
iteration 41: rss 704 MB
iteration 42: rss 704 MB
iteration 43: rss 704 MB
iteration 44: rss 704 MB
iteration 45: rss 704 MB
iteration 46: rss 704 MB
iteration 47: rss 704 MB
iteration 48: rss 704 MB
iteration 49: rss 704 MB
iteration 50: rss 704 MB
iteration 51: rss 704 MB
iteration 52: rss 704 MB
iteration 53: rss 704 MB
iteration 54: rss 704 MB
iteration 55: rss 704 MB
iteration 56: rss 704 MB
iteration 57: rss 704 MB
iteration 58: rss 704 MB
iteration 59: rss 704 MB
iteration 60: rss 704 MB
iteration 61: rss 704 MB
iteration 62: rss 704 MB
iteration 63: rss 704 MB
iteration 64: rss 704 MB
iteration 65: rss 704 MB
iteration 66: rss 704 MB
iteration 67: rss 704 MB
iteration 68: rss 704 MB
iteration 69: rss 704 MB
iteration 70: rss 704 MB
iteration 71: rss 704 MB
iteration 72: rss 704 MB
iteration 73: rss 704 MB
iteration 74: rss 704 MB
iteration 75: rss 704 MB
iteration 76: rss 705 MB  # new maximum
iteration 77: rss 705 MB
iteration 78: rss 705 MB
iteration 79: rss 705 MB
iteration 80: rss 705 MB
iteration 81: rss 705 MB
iteration 82: rss 706 MB  # new maximum
iteration 83: rss 706 MB
iteration 84: rss 713 MB  # new maximum
iteration 85: rss 713 MB
iteration 86: rss 713 MB
iteration 87: rss 713 MB
iteration 88: rss 719 MB  # new maximum
iteration 89: rss 719 MB
iteration 90: rss 719 MB
iteration 91: rss 719 MB
iteration 92: rss 719 MB
iteration 93: rss 719 MB
iteration 94: rss 719 MB
iteration 95: rss 719 MB
iteration 96: rss 719 MB
iteration 97: rss 720 MB  # new maximum
iteration 98: rss 720 MB
iteration 99: rss 720 MB

Describe the expected behavior Memory consumptions stays constant

Standalone code to reproduce the issue

import gc
import os

import numpy as np
import psutil
import tensorflow as tf

tf.get_logger().setLevel("ERROR")  # Suppress "tf.function retracing" warnings
process = psutil.Process(os.getpid())
for i in range(100):
    # do some work
    model = tf.keras.applications.mobilenet.MobileNet()
    model.compile(loss="mse")
    x = tf.zeros((1, *model.input.shape[1:]))
    y = tf.zeros((1, *model.output.shape[1:]))
    history = model.fit(x=x, y=y, verbose=0)
    
    # clean up
    _ = gc.collect()
    tf.keras.backend.clear_session()
    
    # show memory usage
    print(f"iteration {i}: rss {process.memory_info().rss >> 20} MB")

Other info / logs

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 1
  • Comments: 26 (8 by maintainers)

Most upvoted comments

Note: in the example above,

  • when commenting out the model.fit line, or
  • when working with constant data (assigning x and y outside the loop),

memory consumption stays constant.

This example reproduces the issue with a much smaller network. Note that this example does use constant data:

import gc
import os

import psutil
import tensorflow as tf
from tensorflow import keras

tf.get_logger().setLevel("ERROR")  # Suppress "tf.function retracing" warnings
process = psutil.Process(os.getpid())
prev_mem = 0
first_mem = 0
for i in range(100):
    # do some work
    in_layer = keras.layers.Input(shape=(1,))
    hidden_layer = in_layer
    for _ in range(200):
        hidden_layer = keras.layers.Dense(1)(hidden_layer)
    out_layer = hidden_layer
    model = keras.Model(inputs=in_layer, outputs=out_layer)
    model.compile(loss="mse")
    history = model.fit(x=[0], y=[0], verbose=0)
    
    # clean up
    _ = gc.collect()
    keras.backend.clear_session()
    
    # show memory usage
    mem = process.memory_info().rss
    if i == 0:
        first_mem = mem
    print(
        f"iteration {i}: rss {mem >> 20} MB ({(mem - prev_mem) >> 10:+} KB; "
        + f"{((mem - first_mem) // max(1, i)) >> 10:+} KB/it.)"
    )
    prev_mem = mem


Playing with the number of hidden layers and the number of nodes suggests this is related to the number of the layers, not the number of parameters:

  • 0 hidden layers leads to constant memory consumption
  • 1 hidden layer leads to several KB leak per iteration, even with thousands of nodes
  • 200 hidden layers with a single node (400 parameters) lead to several MB leaking per iteration. This is the case in the extreme example above, from Colab:
iteration 0: rss 590 MB (+605152 KB; +0 KB/it.)
iteration 1: rss 786 MB (+200200 KB; +200200 KB/it.)
iteration 2: rss 790 MB (+4108 KB; +102154 KB/it.)
iteration 3: rss 803 MB (+13300 KB; +72536 KB/it.)
iteration 4: rss 807 MB (+4476 KB; +55521 KB/it.)
iteration 5: rss 813 MB (+5652 KB; +45547 KB/it.)
iteration 6: rss 814 MB (+784 KB; +38086 KB/it.)
iteration 7: rss 817 MB (+3140 KB; +33094 KB/it.)
iteration 8: rss 821 MB (+4376 KB; +29504 KB/it.)
iteration 9: rss 822 MB (+1152 KB; +26354 KB/it.)
iteration 10: rss 825 MB (+3384 KB; +24057 KB/it.)
iteration 11: rss 827 MB (+1840 KB; +22037 KB/it.)
iteration 12: rss 828 MB (+1140 KB; +20296 KB/it.)
iteration 13: rss 831 MB (+2636 KB; +18937 KB/it.)
iteration 14: rss 832 MB (+1320 KB; +17679 KB/it.)
...

@ghsanti no idea. Note all of the above is 4 years old.

Are you using gc after clear_session?:

 tf.keras.backend.clear_session()
 gc.collect()

This (now using 2.4.0-dev20201002, twice the number of iterations as before, but still non-eager) seems to confirm that there is no unlimited leaking, even though memory usage increases over a long period of 15 minute before becoming stable:

image

2.4.0-dev20201002
iteration 0: rss 576 MB (+589980 KB; +0 KB/it.)
iteration 1: rss 640 MB (+66248 KB; +66248 KB/it.)
iteration 2: rss 653 MB (+13272 KB; +39760 KB/it.)
iteration 3: rss 672 MB (+18668 KB; +32729 KB/it.)
iteration 4: rss 687 MB (+15744 KB; +28483 KB/it.)
iteration 5: rss 696 MB (+8804 KB; +24547 KB/it.)
iteration 6: rss 709 MB (+14108 KB; +22807 KB/it.)
iteration 7: rss 729 MB (+19676 KB; +22360 KB/it.)
iteration 8: rss 717 MB (-11960 KB; +18070 KB/it.)
iteration 9: rss 756 MB (+40036 KB; +20510 KB/it.)
iteration 10: rss 721 MB (-36272 KB; +14832 KB/it.)
iteration 11: rss 823 MB (+104888 KB; +23019 KB/it.)
iteration 12: rss 744 MB (-80428 KB; +14398 KB/it.)
iteration 13: rss 826 MB (+83092 KB; +19682 KB/it.)
iteration 14: rss 824 MB (-1768 KB; +18150 KB/it.)
iteration 15: rss 843 MB (+20136 KB; +18282 KB/it.)
iteration 16: rss 834 MB (-9228 KB; +16563 KB/it.)
iteration 17: rss 793 MB (-42044 KB; +13116 KB/it.)
iteration 18: rss 826 MB (+32968 KB; +14218 KB/it.)
iteration 19: rss 843 MB (+18204 KB; +14428 KB/it.)
iteration 20: rss 837 MB (-6944 KB; +13360 KB/it.)
iteration 21: rss 852 MB (+15700 KB; +13471 KB/it.)
iteration 22: rss 843 MB (-8920 KB; +12453 KB/it.)
iteration 23: rss 801 MB (-43616 KB; +10015 KB/it.)
iteration 24: rss 831 MB (+30976 KB; +10889 KB/it.)
iteration 25: rss 850 MB (+19196 KB; +11221 KB/it.)
iteration 26: rss 843 MB (-7000 KB; +10520 KB/it.)
iteration 27: rss 861 MB (+18324 KB; +10809 KB/it.)
iteration 28: rss 852 MB (-9300 KB; +10091 KB/it.)
iteration 29: rss 866 MB (+15068 KB; +10263 KB/it.)
iteration 30: rss 859 MB (-7688 KB; +9664 KB/it.)
iteration 31: rss 872 MB (+13084 KB; +9774 KB/it.)
iteration 32: rss 862 MB (-9812 KB; +9162 KB/it.)
iteration 33: rss 876 MB (+13840 KB; +9304 KB/it.)
iteration 34: rss 866 MB (-9348 KB; +8756 KB/it.)
iteration 35: rss 881 MB (+15032 KB; +8935 KB/it.)
iteration 36: rss 838 MB (-43928 KB; +7466 KB/it.)
iteration 37: rss 850 MB (+11628 KB; +7579 KB/it.)
iteration 38: rss 842 MB (-7940 KB; +7170 KB/it.)
iteration 39: rss 850 MB (+8424 KB; +7203 KB/it.)
iteration 40: rss 844 MB (-6588 KB; +6858 KB/it.)
iteration 41: rss 851 MB (+7832 KB; +6882 KB/it.)
iteration 42: rss 847 MB (-4784 KB; +6604 KB/it.)
iteration 43: rss 854 MB (+7516 KB; +6625 KB/it.)
iteration 44: rss 849 MB (-5292 KB; +6354 KB/it.)
iteration 45: rss 855 MB (+6528 KB; +6358 KB/it.)
iteration 46: rss 851 MB (-4416 KB; +6124 KB/it.)
iteration 47: rss 859 MB (+8172 KB; +6167 KB/it.)
iteration 48: rss 855 MB (-4216 KB; +5951 KB/it.)
iteration 49: rss 784 MB (-72688 KB; +4346 KB/it.)
iteration 50: rss 822 MB (+39308 KB; +5045 KB/it.)
iteration 51: rss 834 MB (+11772 KB; +5177 KB/it.)
iteration 52: rss 832 MB (-1468 KB; +5049 KB/it.)
iteration 53: rss 841 MB (+9408 KB; +5132 KB/it.)
iteration 54: rss 838 MB (-3788 KB; +4966 KB/it.)
iteration 55: rss 846 MB (+8300 KB; +5027 KB/it.)
iteration 56: rss 843 MB (-2616 KB; +4891 KB/it.)
iteration 57: rss 850 MB (+6932 KB; +4926 KB/it.)
iteration 58: rss 846 MB (-4244 KB; +4768 KB/it.)
iteration 59: rss 852 MB (+6292 KB; +4794 KB/it.)
iteration 60: rss 849 MB (-3120 KB; +4662 KB/it.)
iteration 61: rss 855 MB (+6020 KB; +4684 KB/it.)
iteration 62: rss 852 MB (-3212 KB; +4557 KB/it.)
iteration 63: rss 796 MB (-57268 KB; +3576 KB/it.)
iteration 64: rss 831 MB (+36168 KB; +4085 KB/it.)
iteration 65: rss 841 MB (+9976 KB; +4176 KB/it.)
iteration 66: rss 840 MB (-968 KB; +4098 KB/it.)
iteration 67: rss 845 MB (+5716 KB; +4122 KB/it.)
iteration 68: rss 845 MB (-636 KB; +4052 KB/it.)
iteration 69: rss 848 MB (+2896 KB; +4035 KB/it.)
iteration 70: rss 847 MB (-544 KB; +3970 KB/it.)
iteration 71: rss 851 MB (+3832 KB; +3968 KB/it.)
iteration 72: rss 850 MB (-1080 KB; +3898 KB/it.)
iteration 73: rss 854 MB (+4040 KB; +3900 KB/it.)
iteration 74: rss 852 MB (-1528 KB; +3826 KB/it.)
iteration 75: rss 856 MB (+3876 KB; +3827 KB/it.)
iteration 76: rss 854 MB (-1680 KB; +3754 KB/it.)
iteration 77: rss 857 MB (+2764 KB; +3741 KB/it.)
iteration 78: rss 857 MB (-304 KB; +3690 KB/it.)
iteration 79: rss 859 MB (+2584 KB; +3676 KB/it.)
iteration 80: rss 857 MB (-2572 KB; +3598 KB/it.)
iteration 81: rss 859 MB (+2604 KB; +3585 KB/it.)
iteration 82: rss 859 MB (-408 KB; +3537 KB/it.)
iteration 83: rss 861 MB (+2436 KB; +3523 KB/it.)
iteration 84: rss 861 MB (+104 KB; +3483 KB/it.)
iteration 85: rss 863 MB (+1328 KB; +3457 KB/it.)
iteration 86: rss 863 MB (-20 KB; +3417 KB/it.)
iteration 87: rss 864 MB (+1476 KB; +3394 KB/it.)
iteration 88: rss 864 MB (-184 KB; +3354 KB/it.)
iteration 89: rss 865 MB (+1332 KB; +3331 KB/it.)
iteration 90: rss 865 MB (-184 KB; +3292 KB/it.)
iteration 91: rss 867 MB (+1964 KB; +3277 KB/it.)
iteration 92: rss 867 MB (-416 KB; +3237 KB/it.)
iteration 93: rss 868 MB (+1920 KB; +3223 KB/it.)
iteration 94: rss 868 MB (-284 KB; +3186 KB/it.)
iteration 95: rss 870 MB (+1560 KB; +3169 KB/it.)
iteration 96: rss 869 MB (-896 KB; +3126 KB/it.)
iteration 97: rss 870 MB (+1492 KB; +3109 KB/it.)
iteration 98: rss 870 MB (-112 KB; +3077 KB/it.)
iteration 99: rss 871 MB (+1096 KB; +3057 KB/it.)
iteration 100: rss 871 MB (-108 KB; +3025 KB/it.)
iteration 101: rss 872 MB (+1220 KB; +3007 KB/it.)
iteration 102: rss 872 MB (-4 KB; +2978 KB/it.)
iteration 103: rss 803 MB (-71068 KB; +2259 KB/it.)
iteration 104: rss 846 MB (+44628 KB; +2666 KB/it.)
iteration 105: rss 851 MB (+4836 KB; +2687 KB/it.)
iteration 106: rss 853 MB (+1680 KB; +2677 KB/it.)
iteration 107: rss 856 MB (+2756 KB; +2678 KB/it.)
iteration 108: rss 856 MB (+624 KB; +2659 KB/it.)
iteration 109: rss 859 MB (+2728 KB; +2660 KB/it.)
iteration 110: rss 860 MB (+844 KB; +2643 KB/it.)
iteration 111: rss 862 MB (+2288 KB; +2640 KB/it.)
iteration 112: rss 862 MB (+32 KB; +2617 KB/it.)
iteration 113: rss 864 MB (+2612 KB; +2616 KB/it.)
iteration 114: rss 864 MB (-40 KB; +2593 KB/it.)
iteration 115: rss 866 MB (+2116 KB; +2589 KB/it.)
iteration 116: rss 867 MB (+44 KB; +2567 KB/it.)
iteration 117: rss 867 MB (+968 KB; +2553 KB/it.)
iteration 118: rss 868 MB (+912 KB; +2539 KB/it.)
iteration 119: rss 870 MB (+1592 KB; +2532 KB/it.)
iteration 120: rss 871 MB (+804 KB; +2517 KB/it.)
iteration 121: rss 872 MB (+1496 KB; +2509 KB/it.)
iteration 122: rss 871 MB (-720 KB; +2482 KB/it.)
iteration 123: rss 873 MB (+2104 KB; +2479 KB/it.)
iteration 124: rss 873 MB (-776 KB; +2453 KB/it.)
iteration 125: rss 875 MB (+2228 KB; +2451 KB/it.)
iteration 126: rss 874 MB (-1232 KB; +2422 KB/it.)
iteration 127: rss 876 MB (+2504 KB; +2422 KB/it.)
iteration 128: rss 875 MB (-1284 KB; +2394 KB/it.)
iteration 129: rss 878 MB (+2844 KB; +2397 KB/it.)
iteration 130: rss 877 MB (-1112 KB; +2370 KB/it.)
iteration 131: rss 879 MB (+2620 KB; +2372 KB/it.)
iteration 132: rss 878 MB (-896 KB; +2347 KB/it.)
iteration 133: rss 880 MB (+1708 KB; +2342 KB/it.)
iteration 134: rss 879 MB (-848 KB; +2319 KB/it.)
iteration 135: rss 880 MB (+1412 KB; +2312 KB/it.)
iteration 136: rss 880 MB (-300 KB; +2293 KB/it.)
iteration 137: rss 882 MB (+1960 KB; +2290 KB/it.)
iteration 138: rss 881 MB (-1408 KB; +2263 KB/it.)
iteration 139: rss 884 MB (+2948 KB; +2268 KB/it.)
iteration 140: rss 882 MB (-1580 KB; +2241 KB/it.)
iteration 141: rss 885 MB (+2588 KB; +2243 KB/it.)
iteration 142: rss 858 MB (-27564 KB; +2033 KB/it.)
iteration 143: rss 864 MB (+6172 KB; +2062 KB/it.)
iteration 144: rss 864 MB (+644 KB; +2052 KB/it.)
iteration 145: rss 867 MB (+3176 KB; +2060 KB/it.)
iteration 146: rss 868 MB (+616 KB; +2050 KB/it.)
iteration 147: rss 870 MB (+2244 KB; +2052 KB/it.)
iteration 148: rss 870 MB (-348 KB; +2035 KB/it.)
iteration 149: rss 873 MB (+2788 KB; +2040 KB/it.)
iteration 150: rss 872 MB (-780 KB; +2022 KB/it.)
iteration 151: rss 875 MB (+2772 KB; +2027 KB/it.)
iteration 152: rss 875 MB (-36 KB; +2013 KB/it.)
iteration 153: rss 877 MB (+2232 KB; +2014 KB/it.)
iteration 154: rss 877 MB (+192 KB; +2003 KB/it.)
iteration 155: rss 879 MB (+1888 KB; +2002 KB/it.)
iteration 156: rss 879 MB (-48 KB; +1989 KB/it.)
iteration 157: rss 823 MB (-56824 KB; +1614 KB/it.)
iteration 158: rss 864 MB (+42200 KB; +1871 KB/it.)
iteration 159: rss 869 MB (+4748 KB; +1889 KB/it.)
iteration 160: rss 870 MB (+948 KB; +1883 KB/it.)
iteration 161: rss 874 MB (+3932 KB; +1896 KB/it.)
iteration 162: rss 874 MB (+392 KB; +1887 KB/it.)
iteration 163: rss 876 MB (+2348 KB; +1889 KB/it.)
iteration 164: rss 876 MB (-200 KB; +1877 KB/it.)
iteration 165: rss 879 MB (+2496 KB; +1880 KB/it.)
iteration 166: rss 879 MB (-224 KB; +1868 KB/it.)
iteration 167: rss 880 MB (+1840 KB; +1868 KB/it.)
iteration 168: rss 881 MB (+676 KB; +1861 KB/it.)
iteration 169: rss 882 MB (+992 KB; +1855 KB/it.)
iteration 170: rss 883 MB (+644 KB; +1848 KB/it.)
iteration 171: rss 885 MB (+2044 KB; +1849 KB/it.)
iteration 172: rss 883 MB (-1608 KB; +1829 KB/it.)
iteration 173: rss 885 MB (+1876 KB; +1830 KB/it.)
iteration 174: rss 864 MB (-21160 KB; +1697 KB/it.)
iteration 175: rss 869 MB (+4488 KB; +1713 KB/it.)
iteration 176: rss 872 MB (+3212 KB; +1722 KB/it.)
iteration 177: rss 876 MB (+3948 KB; +1734 KB/it.)
iteration 178: rss 876 MB (+156 KB; +1726 KB/it.)
iteration 179: rss 879 MB (+3068 KB; +1733 KB/it.)
iteration 180: rss 878 MB (-716 KB; +1719 KB/it.)
iteration 181: rss 881 MB (+3380 KB; +1729 KB/it.)
iteration 182: rss 881 MB (-564 KB; +1716 KB/it.)
iteration 183: rss 881 MB (-128 KB; +1706 KB/it.)
iteration 184: rss 880 MB (-620 KB; +1693 KB/it.)
iteration 185: rss 883 MB (+3292 KB; +1702 KB/it.)
iteration 186: rss 830 MB (-54556 KB; +1399 KB/it.)
iteration 187: rss 871 MB (+41540 KB; +1614 KB/it.)
iteration 188: rss 873 MB (+2188 KB; +1617 KB/it.)
iteration 189: rss 876 MB (+3456 KB; +1627 KB/it.)
iteration 190: rss 878 MB (+1516 KB; +1626 KB/it.)
iteration 191: rss 879 MB (+1448 KB; +1625 KB/it.)
iteration 192: rss 880 MB (+732 KB; +1621 KB/it.)
iteration 193: rss 882 MB (+2028 KB; +1623 KB/it.)
iteration 194: rss 882 MB (+528 KB; +1617 KB/it.)
iteration 195: rss 884 MB (+1940 KB; +1619 KB/it.)
iteration 196: rss 885 MB (+1040 KB; +1616 KB/it.)
iteration 197: rss 885 MB (+352 KB; +1609 KB/it.)
iteration 198: rss 885 MB (-212 KB; +1600 KB/it.)
iteration 199: rss 886 MB (+1072 KB; +1598 KB/it.)
mprof: Sampling memory every 0.1s
running as a Python program...

I have try run_eagerly=True for completeness, and I am puzzled. That version uses 2-4 times fewer memory and is roughly 10 times as fast.

image

I also tested the run time of individual iterations to rule out differing numbers of iterations or other stupid mistakes:

>>> model.compile(loss="mse"), timeit.timeit(lambda: model.fit(x=[0], y=[0], verbose=0), number=3)
(None, 19.40625670000003)
>>> model.compile(loss="mse", run_eagerly=True), timeit.timeit(lambda: model.fit(x=[0], y=[0], verbose=0), number=3)
(None, 1.3771592999999598)
>>> model.compile(loss="mse"), timeit.timeit(lambda: model.fit(x=[0], y=[0], verbose=0), number=3)
(None, 16.871414200000004)
>>> model.compile(loss="mse", run_eagerly=True), timeit.timeit(lambda: model.fit(x=[0], y=[0], verbose=0), number=3)
(None, 1.3355119000000286)
>>> model.compile(loss="mse"), timeit.timeit(lambda: model.fit(x=[0], y=[0], verbose=0), number=3)
(None, 19.290036299999997)

(And this result reproduces on 2.3.1/Linux/GPU.)

@bhack I have, since reporting this issue, found an unrelated bug in my (GPU-based, Linux) environment that may have resulted in (and has since resolved) the OOM crashes that I have experienced. So first of all, I do not have a pressing issue that needs resolving. Also, tf-nightly is clearly better than 2.3.1 in at least one case, so let’s hope 2.4 includes the relevant fixes and performance stays where it is in this case.

However, there may still be an issue in general. I also ran mprof now for the code in https://github.com/tensorflow/tensorflow/issues/43702#issuecomment-702069826, and this is the result (Windows, CPU, 2.3.1):

image

iteration 0: rss 555 MB (+568336 KB; +0 KB/it.)
iteration 1: rss 655 MB (+102644 KB; +102644 KB/it.)
iteration 2: rss 668 MB (+13936 KB; +58290 KB/it.)
iteration 3: rss 698 MB (+30728 KB; +49102 KB/it.)
iteration 4: rss 714 MB (+15920 KB; +40807 KB/it.)
iteration 5: rss 730 MB (+16896 KB; +36024 KB/it.)
iteration 6: rss 720 MB (-10324 KB; +28300 KB/it.)
iteration 7: rss 763 MB (+44004 KB; +30543 KB/it.)
iteration 8: rss 752 MB (-11972 KB; +25229 KB/it.)
iteration 9: rss 771 MB (+19708 KB; +24615 KB/it.)
iteration 10: rss 789 MB (+18512 KB; +24005 KB/it.)
iteration 11: rss 801 MB (+12616 KB; +22969 KB/it.)
iteration 12: rss 837 MB (+36504 KB; +24097 KB/it.)
iteration 13: rss 805 MB (-32320 KB; +19757 KB/it.)
iteration 14: rss 802 MB (-3504 KB; +18096 KB/it.)
iteration 15: rss 858 MB (+57864 KB; +20747 KB/it.)
iteration 16: rss 854 MB (-4848 KB; +19147 KB/it.)
iteration 17: rss 821 MB (-33520 KB; +16049 KB/it.)
iteration 18: rss 853 MB (+32664 KB; +16972 KB/it.)
iteration 19: rss 873 MB (+20176 KB; +17141 KB/it.)
iteration 20: rss 867 MB (-5840 KB; +15992 KB/it.)
iteration 21: rss 885 MB (+18284 KB; +16101 KB/it.)
iteration 22: rss 879 MB (-5608 KB; +15114 KB/it.)
iteration 23: rss 837 MB (-43208 KB; +12578 KB/it.)
iteration 24: rss 867 MB (+30256 KB; +13315 KB/it.)
iteration 25: rss 887 MB (+20796 KB; +13614 KB/it.)
iteration 26: rss 884 MB (-3176 KB; +12968 KB/it.)
iteration 27: rss 900 MB (+16940 KB; +13115 KB/it.)
iteration 28: rss 892 MB (-8400 KB; +12347 KB/it.)
iteration 29: rss 909 MB (+16752 KB; +12499 KB/it.)
iteration 30: rss 900 MB (-8200 KB; +11809 KB/it.)
iteration 31: rss 914 MB (+13468 KB; +11862 KB/it.)
iteration 32: rss 907 MB (-6352 KB; +11293 KB/it.)
iteration 33: rss 923 MB (+16040 KB; +11437 KB/it.)
iteration 34: rss 913 MB (-10332 KB; +10797 KB/it.)
iteration 35: rss 928 MB (+15464 KB; +10930 KB/it.)
iteration 36: rss 921 MB (-6960 KB; +10433 KB/it.)
iteration 37: rss 933 MB (+11584 KB; +10464 KB/it.)
iteration 38: rss 929 MB (-3824 KB; +10088 KB/it.)
iteration 39: rss 938 MB (+9608 KB; +10076 KB/it.)
iteration 40: rss 931 MB (-6980 KB; +9649 KB/it.)
iteration 41: rss 941 MB (+10136 KB; +9661 KB/it.)
iteration 42: rss 937 MB (-4548 KB; +9323 KB/it.)
iteration 43: rss 948 MB (+11060 KB; +9363 KB/it.)
iteration 44: rss 937 MB (-11484 KB; +8890 KB/it.)
iteration 45: rss 951 MB (+14796 KB; +9021 KB/it.)
iteration 46: rss 943 MB (-8172 KB; +8647 KB/it.)
iteration 47: rss 954 MB (+11132 KB; +8700 KB/it.)
iteration 48: rss 947 MB (-7348 KB; +8366 KB/it.)
iteration 49: rss 958 MB (+11264 KB; +8425 KB/it.)
iteration 50: rss 952 MB (-6196 KB; +8132 KB/it.)
iteration 51: rss 961 MB (+9632 KB; +8162 KB/it.)
iteration 52: rss 957 MB (-4104 KB; +7926 KB/it.)
iteration 53: rss 964 MB (+7192 KB; +7912 KB/it.)
iteration 54: rss 961 MB (-2864 KB; +7712 KB/it.)
iteration 55: rss 970 MB (+8984 KB; +7735 KB/it.)
iteration 56: rss 966 MB (-4168 KB; +7523 KB/it.)
iteration 57: rss 972 MB (+5988 KB; +7496 KB/it.)
iteration 58: rss 970 MB (-1888 KB; +7334 KB/it.)
iteration 59: rss 974 MB (+4396 KB; +7284 KB/it.)
iteration 60: rss 973 MB (-1084 KB; +7145 KB/it.)
iteration 61: rss 979 MB (+6168 KB; +7129 KB/it.)
iteration 62: rss 975 MB (-3812 KB; +6952 KB/it.)
iteration 63: rss 983 MB (+7544 KB; +6962 KB/it.)
iteration 64: rss 980 MB (-3372 KB; +6800 KB/it.)
iteration 65: rss 988 MB (+8604 KB; +6828 KB/it.)
iteration 66: rss 984 MB (-3968 KB; +6664 KB/it.)
iteration 67: rss 992 MB (+8552 KB; +6693 KB/it.)
iteration 68: rss 988 MB (-4888 KB; +6522 KB/it.)
iteration 69: rss 996 MB (+8040 KB; +6544 KB/it.)
iteration 70: rss 990 MB (-5264 KB; +6376 KB/it.)
iteration 71: rss 999 MB (+8432 KB; +6405 KB/it.)
iteration 72: rss 997 MB (-2108 KB; +6286 KB/it.)
iteration 73: rss 1005 MB (+9104 KB; +6325 KB/it.)
iteration 74: rss 1002 MB (-3692 KB; +6190 KB/it.)
iteration 75: rss 1009 MB (+7152 KB; +6202 KB/it.)
iteration 76: rss 1006 MB (-2660 KB; +6086 KB/it.)
iteration 77: rss 1014 MB (+7556 KB; +6105 KB/it.)
iteration 78: rss 1008 MB (-5256 KB; +5959 KB/it.)
iteration 79: rss 1016 MB (+7316 KB; +5976 KB/it.)
iteration 80: rss 1012 MB (-3388 KB; +5859 KB/it.)
iteration 81: rss 1022 MB (+9920 KB; +5909 KB/it.)
iteration 82: rss 1017 MB (-4788 KB; +5779 KB/it.)
iteration 83: rss 1026 MB (+8492 KB; +5812 KB/it.)
iteration 84: rss 1022 MB (-3464 KB; +5701 KB/it.)
iteration 85: rss 1030 MB (+7852 KB; +5726 KB/it.)
iteration 86: rss 1027 MB (-3152 KB; +5623 KB/it.)
iteration 87: rss 1033 MB (+5996 KB; +5628 KB/it.)
iteration 88: rss 1031 MB (-1732 KB; +5544 KB/it.)
iteration 89: rss 1039 MB (+7896 KB; +5570 KB/it.)
iteration 90: rss 1036 MB (-3064 KB; +5474 KB/it.)
iteration 91: rss 997 MB (-39496 KB; +4980 KB/it.)
iteration 92: rss 995 MB (-1824 KB; +4906 KB/it.)
iteration 93: rss 1002 MB (+6668 KB; +4925 KB/it.)
iteration 94: rss 998 MB (-3544 KB; +4835 KB/it.)
iteration 95: rss 1007 MB (+8732 KB; +4876 KB/it.)
iteration 96: rss 1003 MB (-4284 KB; +4781 KB/it.)
iteration 97: rss 1011 MB (+8868 KB; +4823 KB/it.)
iteration 98: rss 1008 MB (-3732 KB; +4735 KB/it.)
iteration 99: rss 1013 MB (+5104 KB; +4739 KB/it.)

I will check with more iterations if the decrease around iteration 90/100 hints at some hard memory limit or whether memory is growing even further, but already what happens until iteration 90 tells me that memory is growing when it probably should not.

@markodjordjic: thanks, but I’m not quite willing to invest that effort.

By the way, each of your many individual messages triggers an individual email notification. Could you try to post your thoughts in a combined message? This is not an IRC chat here 😉 You could also edit past message if you want to add anything.

@bersbersbers I do not see a fix for this any time soon.

May I ask what makes you think that? You reported the other issue just 3 days ago, and as far as I can see, did not provide a working reproduction code.

In meanwhile use this as a workaround: https://www.tensorflow.org/guide/keras/customizing_what_happens_in_fit

This is ten pages of documentation - which section is the “workaround” (other than implementing the whole fitting routine myself)?