clearml: ClearML is not saving scalars/images when using Tensorflow Object detection API - TF2.2

This issue is related to this thread: https://clearml.slack.com/archives/CTK20V944/p1610457717141400

To reproduce: git clone https://github.com/glemarivero/raccoon_dataset.git setup a virtualenv run sh autoinstall.sh Generate tfrecords:

python generate_tfrecord.py --output_path images.tfrecord --csv_input raccoon_labels.csv --image_dir images

Run training:

python model_main_tf2.py --model_dir=models/ --pipeline_config_path=pipeline.config

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 20 (13 by maintainers)

Most upvoted comments

Just to add my +1, the scenario just described by @bmartinn is the same that I experience.

asmithaxe on Mar 24, 2021

Hi @glemarivero Thank you again for the code! I am able to run the training session. I think it reproduces the issue, as I’m only getting the “steps_per_sec” scalar graph, and I’m missing the “Loss/BoxClassifierLoss/localization_loss” etc. Let me dig deeper and see what’s going on.

bmartinn on Mar 23, 2021