tensorflow: tf2.11 tf.profiler.experimental doesn't generate trace.json.gz. Possible to have a python script to get the json.gz from xplane.pb?

Click to expand!

Issue Type

Bug

Have you reproduced the bug with TF nightly?

Yes

Source

binary

Tensorflow Version

tf2.11

Custom Code

Yes

OS Platform and Distribution

Linux Ubuntu 22.04

Mobile device

No response

Python version

3.9

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

In tf2.10, when use the tf.profiler.experimental.start(logdir) and tf.profiler.experimental.stop(). It will generate many files as bellow in logdir folder.
.
├── events.out.tfevents.XXXX.profile-empty
└── plugins
    └── profile
        └── 2022_11_29_08_25_43
            ├── XXXX.input_pipeline.pb
            ├── XXXX.kernel_stats.pb
            ├── XXXX.memory_profile.json.gz
            ├── XXXX.overview_page.pb
            ├── XXXX.tensorflow_stats.pb
            ├── XXXX.trace.json
            └── XXXX.xplane.pb

But in tf2.11, it only generate these files.
.
├── events.out.tfevents.XXXX.profile-empty
└── plugins
    └── profile
        ├── 2022_11_29_08_32_10
        │   └── XXXX.xplane.pb

Possible to derive json.gz file from xplane.pb with some python scripts?

Standalone code to reproduce the issue

NA

Relevant log output

No response

About this issue

Original URL
State: closed
Created a year ago
Comments: 15 (10 by maintainers)

Most upvoted comments

You could use pywrap_profiler functions defined in https://github.com/tensorflow/tensorflow/blob/f62622072d3b6de1790861ba749a1bfa043bd1dd/tensorflow/python/profiler/internal/profiler_wrapper.cc to extract the trace.json.

References to how this can be invoked can be found here https://github.com/tensorflow/profiler/blob/85dcfd10656d623330b11c3bbb8afed6418ec533/plugin/tensorboard_plugin_profile/convert/raw_to_tool_data.py

This would look like

pywrap_profiler.xspace_to_tools_data(xspace_file_path, "trace_viewer")

cliveverghese on May 25, 2023

@cliveverghese’s comment got me to finally get the .json out of this annoying file format. I made this Python script (with a patch from @louie-tsai):

import argparse

# Create argument parser
parser = argparse.ArgumentParser(description="Convert raw file to tool data")
parser.add_argument("input_file", help="Input file path")
parser.add_argument("output_file", help="Output JSON file path")

# Parse the arguments
args = parser.parse_args()

# Process the input file
print("\033[32mImport TensorFlow...\033[0m")
import tensorboard_plugin_profile.convert.raw_to_tool_data as rttd
input_file = args.input_file
print("\033[32mXSpace to Tool Data...\033[0m")
tv = rttd.xspace_to_tool_data([input_file], "trace_viewer^", {'tqx': ''})

if isinstance(tv, tuple):
    tv = tv[0]

# Write the processed data to the output file
output_file = args.output_file
print("\033[32mWriting file...\033[0m")
with open(output_file, "w") as f:
    f.write(tv)

print("\033[32mDone!\033[0m")

mcourteaux on Jun 9, 2023

output of xspace_to_tool_data is tuple data format for my case, so I need to only get first item of the tuple data.

louie-tsai on Jun 2, 2023

I have replicated the reported behaviour. With Tf 2.10v I found many files along with xplane.pb being built but with TF2.12 and tf-nightly there is only one file i.e. xplane.pb. I have attached minimal code snippet for replicating the behaviour in attached gist.

With TF 2.10:

With TF2.12:

@louie-tsai , I need to check the reasons for this difference in behaviour and will update you if ay workaround. Thanks

SuryanarayanaY on Apr 12, 2023