tensorflow: tf.data.experimental.enable_debug_mode() doesn't enable breakpoint

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Desktop
TensorFlow installed from (source or binary): pip
TensorFlow version (use command below): v1.12.1-53831-ga8b6d5ff93a 2.5.0-rc0
Python version: Python 3.6.8
CUDA/cuDNN version: CPU
GPU model and memory: CPU

Describe the current behavior

import tensorflow as tf

tf.data.experimental.enable_debug_mode()
tf.config.run_functions_eagerly(True)


def func(x):
    x = x + 1  # BREAKPOINT HERE
    return x


dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3])
dataset = dataset.map(func)
# dataset = dataset.map(lambda x: tf.py_function(func, [x], tf.int32))  # doesn't work either
for item in dataset:
    print(item)

The breakpoint is not hit. (using pycharm professional 2020.3 )

Describe the expected behavior

I expect that the breakpoint is hit

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 19 (15 by maintainers)

Most upvoted comments

In PyCharm, one can trigger a breakpoint in a function called by Dataset.map() by adding the following line within that function:

import pydevd; pydevd.settrace()

Example:

import tensorflow as tf

# haven't tested whether these are needed or not
tf.data.experimental.enable_debug_mode()
tf.config.run_functions_eagerly(True)


def func(x):
    import pydevd; pydevd.settrace()
    x = x + 1  # BREAKPOINT HERE
    return x


dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3])
dataset = dataset.map(func)
for item in dataset:
    print(item)

Also note that you will not need to install pydevd separately, as it is bundled with PyCharm. Simply add that line and run debugging.

(found in this StackOverflow answer)

dniku on Jan 31, 2022

When running in debug mode, for technical reasons, tf.data executes user-defined functions using a background thread.
The implementation of pdb.set_trace() uses signal, which as per the error message is not supported in the main thread. Either Python 3.8 removed the use of signal in pdb or added support for signal in background threads.

jsimsa on Apr 20, 2021

Thank you Jay. I was running the repro with an internal build of TensorFlow that uses Python 3.6, which explains the error.

I was able to find a solution to the problem I was encountering. For the example to work with Python 3.6 or Python 3.7, one has to do pdb.Pdb(nosigint=True).set_trace() instead of pdb.set_trace().

So to summarize, the following program illustrates how to add breakpoints to tf.data user-defined functions:

import pdb
import tensorflow as tf

tf.data.experimental.enable_debug_mode()

def func(x):
    pdb.Pdb(nosigint=True).set_trace()
    x = x + 1
    return x


dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3])
dataset = dataset.map(func)

for item in dataset:
    print(item)

@cgebbe please let us know if this addresses your question and if it does, feel free to close the issue. Thanks.

jsimsa on Apr 13, 2021

I have tested it with installing tf-nightly and adding break point to the source code in post-mortem debug mode with pdb. And the behavior can be re-produced. If we add break point within the function, it is not enabled, while if we add break point in other places (like add a simple print “hello, world” line), it is enabled.

jayxiaojieshi on Apr 13, 2021

@jayxiaojieshi could you please help investigate this issue? thank you

jsimsa on Apr 9, 2021