tf-faster-rcnn: RuntimeWarning: invalid value encountered in log targets_dw = np.log(gt_widths / ex_widths) Command terminated by signal 11
i use my own datasets replace the voc2007 and have some issue. Can you please suggest solutions?
here is the log.
##`+ echo Logging output to experiments/logs/vgg16_voc_2007_trainval__vgg16.txt.2017-05-26_14-23-40 Logging output to experiments/logs/vgg16_voc_2007_trainval__vgg16.txt.2017-05-26_14-23-40
- set +x
- ‘[’ ‘!’ -f output/vgg16/voc_2007_trainval/default/vgg16_faster_rcnn_iter_70000.ckpt.index ‘]’
- [[ ! -z ‘’ ]]
- CUDA_VISIBLE_DEVICES=0
- time python ./tools/trainval_net.py --weight data/imagenet_weights/vgg16.ckpt --imdb voc_2007_trainval --imdbval voc_2007_test --iters 70000 --cfg experiments/cfgs/vgg16.yml --net vgg16 --set ANCHOR_SCALES ‘[8,16,32]’ ANCHOR_RATIOS ‘[0.5,1,2]’ TRAIN.STEPSIZE 50000
Called with args:
Namespace(cfg_file=‘experiments/cfgs/vgg16.yml’, imdb_name=‘voc_2007_trainval’, imdbval_name=‘voc_2007_test’, max_iters=70000, net=‘vgg16’, set_cfgs=[‘ANCHOR_SCALES’, ‘[8,16,32]’, ‘ANCHOR_RATIOS’, ‘[0.5,1,2]’, ‘TRAIN.STEPSIZE’, ‘50000’], tag=None, weight=‘data/imagenet_weights/vgg16.ckpt’)
Using config:
{‘ANCHOR_RATIOS’: [0.5, 1, 2],
‘ANCHOR_SCALES’: [8, 16, 32],
‘DATA_DIR’: ‘/media/y/B0AAA15CAAA11FB8/linux/tf-faster-rcnn/data’,
‘DEDUP_BOXES’: 0.0625,
‘EPS’: 1e-14,
‘EXP_DIR’: ‘vgg16’,
‘GPU_ID’: 0,
‘MATLAB’: ‘matlab’,
‘PIXEL_MEANS’: array([[[ 102.9801, 115.9465, 122.7717]]]),
‘POOLING_MODE’: ‘crop’,
‘POOLING_SIZE’: 7,
‘RESNET’: {‘BN_TRAIN’: False, ‘FIXED_BLOCKS’: 1, ‘MAX_POOL’: False},
‘RNG_SEED’: 3,
‘ROOT_DIR’: ‘/media/y/B0AAA15CAAA11FB8/linux/tf-faster-rcnn’,
‘TEST’: {‘BBOX_REG’: True,
‘HAS_RPN’: True,
‘MAX_SIZE’: 1000,
‘MODE’: ‘nms’,
‘NMS’: 0.3,
‘PROPOSAL_METHOD’: ‘gt’,
‘RPN_NMS_THRESH’: 0.7,
‘RPN_POST_NMS_TOP_N’: 300,
‘RPN_PRE_NMS_TOP_N’: 6000,
‘RPN_TOP_N’: 5000,
‘SCALES’: [600],
‘SVM’: False},
‘TRAIN’: {‘ASPECT_GROUPING’: False,
‘BATCH_SIZE’: 256,
‘BBOX_INSIDE_WEIGHTS’: [1.0, 1.0, 1.0, 1.0],
‘BBOX_NORMALIZE_MEANS’: [0.0, 0.0, 0.0, 0.0],
‘BBOX_NORMALIZE_STDS’: [0.1, 0.1, 0.2, 0.2],
‘BBOX_NORMALIZE_TARGETS’: True,
‘BBOX_NORMALIZE_TARGETS_PRECOMPUTED’: True,
‘BBOX_REG’: True,
‘BBOX_THRESH’: 0.5,
‘BG_THRESH_HI’: 0.5,
‘BG_THRESH_LO’: 0.0,
‘BIAS_DECAY’: False,
‘DISPLAY’: 20,
‘DOUBLE_BIAS’: True,
‘FG_FRACTION’: 0.25,
‘FG_THRESH’: 0.5,
‘GAMMA’: 0.1,
‘HAS_RPN’: True,
‘IMS_PER_BATCH’: 1,
‘LEARNING_RATE’: 0.001,
‘MAX_SIZE’: 1000,
‘MOMENTUM’: 0.9,
‘PROPOSAL_METHOD’: ‘gt’,
‘RPN_BATCHSIZE’: 256,
‘RPN_BBOX_INSIDE_WEIGHTS’: [1.0, 1.0, 1.0, 1.0],
‘RPN_CLOBBER_POSITIVES’: False,
‘RPN_FG_FRACTION’: 0.5,
‘RPN_NEGATIVE_OVERLAP’: 0.3,
‘RPN_NMS_THRESH’: 0.7,
‘RPN_POSITIVE_OVERLAP’: 0.7,
‘RPN_POSITIVE_WEIGHT’: -1.0,
‘RPN_POST_NMS_TOP_N’: 2000,
‘RPN_PRE_NMS_TOP_N’: 12000,
‘SCALES’: [600],
‘SNAPSHOT_ITERS’: 5000,
‘SNAPSHOT_KEPT’: 3,
‘SNAPSHOT_PREFIX’: ‘vgg16_faster_rcnn’,
‘STEPSIZE’: 50000,
‘SUMMARY_INTERVAL’: 180,
‘TRUNCATED’: False,
‘USE_ALL_GT’: True,
‘USE_FLIPPED’: True,
‘USE_GT’: False,
‘WEIGHT_DECAY’: 0.0005},
‘USE_GPU_NMS’: False}
Loaded dataset
voc_2007_trainvalfor training Set proposal method: gt Appending horizontally-flipped training examples… voc_2007_trainval gt roidb loaded from /media/y/B0AAA15CAAA11FB8/linux/tf-faster-rcnn/data/cache/voc_2007_trainval_gt_roidb.pkl done Preparing training data… done 1528 roidb entries Output will be saved to/media/y/B0AAA15CAAA11FB8/linux/tf-faster-rcnn/output/vgg16/voc_2007_trainval/defaultTensorFlow summaries will be saved to/media/y/B0AAA15CAAA11FB8/linux/tf-faster-rcnn/tensorboard/vgg16/voc_2007_trainval/defaultLoaded datasetvoc_2007_testfor training Set proposal method: gt Preparing training data… voc_2007_test gt roidb loaded from /media/y/B0AAA15CAAA11FB8/linux/tf-faster-rcnn/data/cache/voc_2007_test_gt_roidb.pkl done 328 validation roidb entries Filtered 0 roidb entries: 1528 -> 1528 Filtered 0 roidb entries: 328 -> 328 2017-05-26 14:24:11.316553: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn’t compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2017-05-26 14:24:11.316569: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn’t compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-05-26 14:24:11.316572: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn’t compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2017-05-26 14:24:11.316575: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn’t compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2017-05-26 14:24:11.316577: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn’t compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. Solving… /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " Loading initial model weights from data/imagenet_weights/vgg16.ckpt Varibles restored: vgg_16/conv1/conv1_1/biases:0 Varibles restored: vgg_16/conv1/conv1_2/weights:0 Varibles restored: vgg_16/conv1/conv1_2/biases:0 Varibles restored: vgg_16/conv2/conv2_1/weights:0 Varibles restored: vgg_16/conv2/conv2_1/biases:0 Varibles restored: vgg_16/conv2/conv2_2/weights:0 Varibles restored: vgg_16/conv2/conv2_2/biases:0 Varibles restored: vgg_16/conv3/conv3_1/weights:0 Varibles restored: vgg_16/conv3/conv3_1/biases:0 Varibles restored: vgg_16/conv3/conv3_2/weights:0 Varibles restored: vgg_16/conv3/conv3_2/biases:0 Varibles restored: vgg_16/conv3/conv3_3/weights:0 Varibles restored: vgg_16/conv3/conv3_3/biases:0 Varibles restored: vgg_16/conv4/conv4_1/weights:0 Varibles restored: vgg_16/conv4/conv4_1/biases:0 Varibles restored: vgg_16/conv4/conv4_2/weights:0 Varibles restored: vgg_16/conv4/conv4_2/biases:0 Varibles restored: vgg_16/conv4/conv4_3/weights:0 Varibles restored: vgg_16/conv4/conv4_3/biases:0 Varibles restored: vgg_16/conv5/conv5_1/weights:0 Varibles restored: vgg_16/conv5/conv5_1/biases:0 Varibles restored: vgg_16/conv5/conv5_2/weights:0 Varibles restored: vgg_16/conv5/conv5_2/biases:0 Varibles restored: vgg_16/conv5/conv5_3/weights:0 Varibles restored: vgg_16/conv5/conv5_3/biases:0 Varibles restored: vgg_16/fc6/biases:0 Varibles restored: vgg_16/fc7/biases:0 Loaded. Fix VGG16 layers… /media/y/B0AAA15CAAA11FB8/linux/tf-faster-rcnn/tools/…/lib/model/bbox_transform.py:26: RuntimeWarning: invalid value encountered in log targets_dw = np.log(gt_widths / ex_widths) Command terminated by signal 11 62.03user 5.27system 0:57.96elapsed 116%CPU (0avgtext+0avgdata 3723648maxresident)k 382896inputs+16outputs (296major+3462186minor)pagefaults 0swaps`
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (1 by maintainers)
If you have checked the xmin ymin xmax ymax and ensure that xmin>0 and xmax<width, ymin>0 and ymax<height, but the problem is still there. Maybe you can try delete the file in /data/chache and rerun the code.
if your dataset’s bbox xmin = 0 or ymin = 0, you should change code in pascal_voc.py # x1 = float(bbox.find(‘xmin’).text) - 1 # y1 = float(bbox.find(‘ymin’).text) - 1 # x2 = float(bbox.find(‘xmax’).text) - 1 # y2 = float(bbox.find(‘ymax’).text) - 1 to
# x1 = float(bbox.find(‘xmin’).text) # y1 = float(bbox.find(‘ymin’).text) # x2 = float(bbox.find(‘xmax’).text) # y2 = float(bbox.find(‘ymax’).text) if your dataset’s bbox xmax= width or ymin = height ,you should change code in imdb.py # boxes[:, 0] = widths[i] - oldx2 - 1 # boxes[:, 2] = widths[i] - oldx1 - 1 to boxes[:, 0] = widths[i] - oldx2 boxes[:, 2] = widths[i] - oldx1 if your dataset’s bbox xmax > width or ymax > height, you should delete it or relabel it