tensorboard: ProfilerNotRunningError: Cannot stop profiling. No profiler is running.

I encountered an error when using “keras.callbacks.TensorBoard(logdir)”

windows 10
tensorflow-gpu==2.0.0

*The following error message : ProfilerNotRunningError Traceback (most recent call last) C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in on_start(self, model, callbacks, use_samples, verbose, mode) 667 try: –> 668 yield 669 finally:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs) 323 training_context=training_context, –> 324 total_epochs=epochs) 325 cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs) 173 cbks.make_logs(model, batch_logs, batch_outs, mode) –> 174 step += 1 175

C:\ProgramData\Anaconda3\lib\contextlib.py in exit(self, type, value, traceback) 87 try: —> 88 next(self.gen) 89 except StopIteration:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in on_batch(self, step, mode, size) 699 self.callbacks._call_batch_hook( –> 700 mode, ‘end’, step, batch_logs) 701 self.progbar.on_batch_end(step, batch_logs)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in _call_batch_hook(self, mode, hook, batch, logs) 234 batch_hook = getattr(callback, hook_name) –> 235 batch_hook(batch, logs) 236 self._delta_ts[hook_name].append(time.time() - t_before_callbacks)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in on_train_batch_end(self, batch, logs) 517 # For backwards compatibility. –> 518 self.on_batch_end(batch, logs=logs) 519

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in on_batch_end(self, batch, logs) 1602 if self._is_tracing: -> 1603 self._log_trace() 1604 elif (not self._is_tracing and

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in _log_trace(self) 1636 step=self._total_batches_seen, -> 1637 profiler_outdir=os.path.join(self.log_dir, ‘train’)) 1638 self._is_tracing = False

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\ops\summary_ops_v2.py in trace_export(name, step, profiler_outdir) 1217 if profiler: -> 1218 _profiler.save(profiler_outdir, _profiler.stop()) 1219

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\profiler.py in stop() 97 raise ProfilerNotRunningError( —> 98 ‘Cannot stop profiling. No profiler is running.’) 99 if context.default_execution_mode == context.EAGER_MODE:

ProfilerNotRunningError: Cannot stop profiling. No profiler is running.

During handling of the above exception, another exception occurred:

ProfilerNotRunningError Traceback (most recent call last) <ipython-input-28-889f4adfe2f3> in <module>() 27 history = model.fit(x_train_scaled, y_train, epochs=10, 28 validation_data=(x_valid_scaled, y_valid), —> 29 callbacks = callbacks)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs) 726 max_queue_size=max_queue_size, 727 workers=workers, –> 728 use_multiprocessing=use_multiprocessing) 729 730 def evaluate(self,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs) 370 total_epochs=1) 371 cbks.make_logs(model, epoch_logs, eval_result, ModeKeys.TEST, –> 372 prefix=‘val_’) 373 374 return model.history

C:\ProgramData\Anaconda3\lib\contextlib.py in exit(self, type, value, traceback) 97 value = type() 98 try: —> 99 self.gen.throw(type, value, traceback) 100 except StopIteration as exc: 101 # Suppress StopIteration unless it’s the same exception that

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in on_start(self, model, callbacks, use_samples, verbose, mode) 669 finally: 670 # End of all epochs –> 671 self.callbacks._call_end_hook(mode) 672 673 @tf_contextlib.contextmanager

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in call_end_hook(self, mode) 256 “”"Helper function for on{train|test|predict}_end methods.“”" 257 if mode == ModeKeys.TRAIN: –> 258 self.on_train_end() 259 elif mode == ModeKeys.TEST: 260 self.on_test_end()

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in on_train_end(self, logs) 373 “”" 374 for callback in self.callbacks: –> 375 callback.on_train_end(logs) 376 377 def on_test_begin(self, logs=None):

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in on_train_end(self, logs) 1619 def on_train_end(self, logs=None): 1620 if self._is_tracing: -> 1621 self._log_trace() 1622 self._close_writers() 1623

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\callbacks.py in log_trace(self) 1635 name='batch%d’ % self._total_batches_seen, 1636 step=self._total_batches_seen, -> 1637 profiler_outdir=os.path.join(self.log_dir, ‘train’)) 1638 self._is_tracing = False 1639

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\ops\summary_ops_v2.py in trace_export(name, step, profiler_outdir) 1216 1217 if profiler: -> 1218 _profiler.save(profiler_outdir, _profiler.stop()) 1219 1220 trace_off()

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\profiler.py in stop() 96 if _profiler is None: 97 raise ProfilerNotRunningError( —> 98 ‘Cannot stop profiling. No profiler is running.’) 99 if context.default_execution_mode == context.EAGER_MODE: 100 context.async_wait()

ProfilerNotRunningError: Cannot stop profiling. No profiler is running.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 5
  • Comments: 18

Commits related to this issue

Most upvoted comments

use the parameter profile_batch for example: callbacks = [tensorflow.keras.callbacks.TensorBoard(log_dir=logdir, histogram_freq=1, profile_batch = 100000000)]

I have just had a similar issue (same error), i recalled that that i read somewhere there is an issue reading or accessing file paths on windows i think.

Try changing

logdir = './cnn_selu_callbacks'

to

logdir = os.join.path('cnn_selu_callbacks')

This works for me name = "logs\\model" Two backslashes instead of one forward slash.

Also, I’m sure it’s os.path.join and not os.join.path

I have just had a similar issue (same error), i recalled that that i read somewhere there is an issue reading or accessing file paths on windows i think.

Try changing

logdir = './cnn_selu_callbacks'

to

logdir = os.join.path('cnn_selu_callbacks')

And passing logdir to keras.callbacks.TensorBoard(log_dir=logdir), as in my case this was line that was the issue.

I’m not sure how this will effect the output_model_file in your code though.

use the parameter profile_batch for example: callbacks = [tensorflow.keras.callbacks.TensorBoard(log_dir=logdir, histogram_freq=1, profile_batch = 100000000)]

This really helps me, thanks bro

I had similar issue. it was windows path problem. changing ‘./logs’ to ‘.\\logs’ solved it

os.path is kind of passing. hope tensorflow to support pathlib soon!~

@shliang0603 Are you encountering this issue on Windows?

For me profile_batch=0 also works. In Tensorflow help it is said: “Set profile_batch=0 to disable profiling. Must run in TensorFlow eager mode.” So maybe you need somehow to turn on eager mode to enable profiling