talos: ResourceExhaustedError after several iterations in a grid search

First off, make sure to check your support options.

The preferred way to resolve usage related matters is through the docs which are maintained up-to-date with the latest version of Talos.

If you do end up asking for support in a new issue, make sure to follow the below steps carefully.

1) Confirm the below

I have looked for an answer in the Docs
My Python version is 3.5 or higher
I have searched through the issues Issues for a duplicate
I’ve tested that my Keras model works as a stand-alone

2) Include the output of:

talos.__version__ == 0.6.7

3) Explain clearly what you are trying to achieve

I am running a grid search that gives 36 rounds. After about 4 or 5 rounds, during a model.fit I suddenly get hit by a ResourceExhaustedError. I think this is very odd given that I am able to complete at least 3 rounds of fitting on the GPU (with a model and batch size that takes up pretty much all the gpu memory), so it seems that there is a small but significant memory leak somewhere. Any ideas what it could be?

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 33 (12 by maintainers)

Most upvoted comments

Sure! I use custom keras.utils.Sequence data generators, so I have two dummy variables for my scan command as shown below:

  dummy_x = np.empty((1, BATCH_SIZE, 208, 208))
   dummy_y = np.empty((1, BATCH_SIZE))

   scan_object = ta.Scan(
       x=dummy_x,
       y=dummy_y,
       disable_progress_bar=False,
       print_params=True,
       model=talos_model,
       params=p,
       experiment_name="talos/" + date_string,
       reduction_method='gamify',
   )

I will take a look at talos 1.0 right away!

bjtho08 on Apr 23, 2020

I would love to, but that option crashes my python kernel, so it’s not really possible. This is a long-standing Keras bug, I believe.

bjtho08 on Apr 23, 2020