tensorforce: Unable to train for many episodes: RAM usage too high!

Hi @AlexKuhnle, I have some trouble training a ppo agent. Basically, I’m able to train it for only very few episodes (e.g. 4, 8). If I increase the number of episodes, my laptop will crash or freeze due to running out of RAM.

I have a linux machine with 16GB of RAM. Tensorflow 2.1.0 (cpu-only) and Tensorforce 0.5.4. The agent I’m trying to train is defined as follows:

policy_network = dict(type='auto', size=64, depth=2, 
                      final_size=256, final_depth=2, internal_rnn=10)
        
agent = Agent.create(
            agent='ppo', 
            environment=environment, 
            max_episode_timesteps=200,
            network=policy_network,
            # Optimization
            batch_size=4, 
            update_frequency=1, 
            learning_rate=1e-3, 
            subsampling_fraction=0.5,
            optimization_steps=5,:
            # Reward estimation
            likelihood_ratio_clipping=0.2, discount=0.99, estimate_terminal=False,
            # Critic
            critic_network='auto',
            critic_optimizer=dict(optimizer='adam', multi_step=10, learning_rate=1e-3),
            # Exploration
            exploration=0.0, variable_noise=0.0,
            # Regularization
            l2_regularization=0.0, entropy_regularization=0.0,
        )

The environment is a custom one I’ve made: it has a complex state space (i.e. an image and some feature vectors), and a simple action space (i.e. five float actions).

I use a Runner to train the agent:

runner = Runner(agent, environment, max_episode_timesteps=200, num_parallel=None)
runner.run(num_episodes=100)

As you can see from the above code snippet, I’d like to train my agent for (at least) 100 episodes but the system crashes after completing episode 4.

I noticed that, during training, every batch_size episodes (4 in my case) Tensorforce allocates an additional amount of 6/7 GB of RAM which causes the system to crash: my OS uses 1 GB + 2/3 GB the environment simulator + 3/4 GB for the agent and environment.

This is what happens (slightly before freezing):

Is this behaviour normal? Just to be sure, I tested a similar (but simpler) agent on the CartPole environment for 300 episodes and it works fine with very little memory overhead. How it’s possible?

Thank you in advance.