DeepReg: error while using "gmi" for the loss

Hi @ciphercharly the fix has been integrated into the main branch now, feel free to test again 😉 Please reopen this ticket if there’s still error!

tested quickly, seems to run without errors with custom model/data too 👍

great! will try asap to run the mr_heart demo and then my custom one too (I have limited access to the machine where I can run this … likely Monday). I am working with pelvic MRI multi-mode images from a set of patients (same machine for all of them), and I need to register intra-subjects the different channels, so I use the ‘paired’ scenario (but for now I am using only the T2 and vibe channels)

Sure, we might merge the fix into main soon, but I will re-open this ticket if it got closed automatically 😉 as the bug is solved using demo data.

update: trying to run predictions on the model trained with gmi results in this error

coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2020-10-16 09:52:42.484648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-16 09:52:42.484686: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-16 09:52:42.484713: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-16 09:52:42.484740: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-16 09:52:42.484766: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-16 09:52:42.484792: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-16 09:52:42.484818: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-16 09:52:42.505351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1, 2
2020-10-16 09:52:42.505395: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-16 09:52:43.898745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-16 09:52:43.898791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 1 2 
2020-10-16 09:52:43.898801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N Y Y 
2020-10-16 09:52:43.898807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 1:   Y N Y 
2020-10-16 09:52:43.898814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 2:   Y Y N 
2020-10-16 09:52:43.906845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 29461 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0)
2020-10-16 09:52:43.909464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 29682 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:08:00.0, compute capability: 7.0)
2020-10-16 09:52:43.911828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30129 MB memory) -> physical GPU (device: 2, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2020-10-16 09:52:52.785402: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-16 09:52:54.375699: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-16 09:53:07.676152: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_grad_ops_3d.cc:1170 : Invalid argument: Conv3DBackpropInputOp: input and out_backprop must have the same batch sizeinput batch: 3outbackprop batch: 1 batch_dim: 0
Traceback (most recent call last):
  File "demos/paired_MMIV/demo_predict.py", line 26, in <module>
    save_png=True,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/predict.py", line 334, in predict
    save_png=save_png,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/predict.py", line 81, in predict_on_dataset
    outputs_dict = model.predict(x=inputs_dict)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 130, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1599, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 814, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Input to reshape is a tensor with 1769472 values, but the requested shape has 5308416
         [[node DDFRegistrationModelWithoutLabel/tf_op_layer_Reshape/Reshape (defined at /home/charlie/3DREG-tests/DeepReg/deepreg/predict.py:81) ]] [Op:__inference_predict_function_7614]

Function call stack:
predict_function

but this could be related to my setup/data (same exact setup/code for training/prediction with only change ‘lncc’ works tho), I have to try with one of the demos

Hi @ciphercharly, I created a new conda env from scratch and I did find a missing package which is scipy, then it’s all working fine including the prediction.

My commands are:

python demos/grouped_mr_heart/demo_data.py

deepreg_train --gpu "" --config_path demos/grouped_mr_heart/grouped_mr_heart.yaml --log_dir grouped_mr_heart

deepreg_predict --gpu "" --config_path demos/grouped_mr_heart/grouped_mr_heart.yaml --ckpt_path logs/grouped_mr_heart/save/weights-epoch6.ckpt --save_png --mode test

with the following config

dataset:
  dir:
    train: "demos/grouped_mr_heart/dataset/train"
    valid: "demos/grouped_mr_heart/dataset/val"
    test: "demos/grouped_mr_heart/dataset/test"
  format: "nifti"
  type: "grouped" # paired / unpaired / grouped
  labeled: false
  intra_group_prob: 1
  intra_group_option: "unconstrained" # forward / backward / unconstrained
  sample_image_in_group: true
  image_shape: [32, 32, 28]

train:
  # define neural network structure
  model:
    method: "ddf" # the registration method, value should be ddf / dvf / conditional
    backbone: "local" # value should be local / global / unet
    local:
      num_channel_initial: 16 # number of initial channel in local net, controls the size of the network
      extract_levels: [0, 1, 2, 3]

  # define the loss function for training
  loss:
    dissimilarity:
      image:
        name: "gmi"
        weight: 1.0
      label:
        weight: 0.0
        name: "multi_scale"
        multi_scale:
          loss_type: "dice"
          loss_scales: [0, 1, 2, 4, 8, 16]
        single_scale:
          loss_type: "cross-entropy"
    regularization:
      weight: 100 # weight of regularization loss
      energy_type: "bending" # value should be bending / gradient-l1 / gradient-l2

  # define the optimizer
  optimizer:
    name: "adam" # value should be adam / sgd / rms
    adam:
      learning_rate: 1.0e-4

  preprocess:
    batch_size: 4
    shuffle_buffer_num_batch: 1 # shuffle_buffer_size = batch_size * shuffle_buffer_num_batch

  # other training hyper-parameters
  epochs: 6 # number of training epochs
  save_period: 2 # the model will be saved every `save_period` epochs.

If you are able to reproduce the error, could you please share your config and commands? By the way which data are you using?

I haven’t saved the message but the pip error (which btw contained a line about the recent changes in the packages dependency system, so could be due to that) stated that dataclasses was missing and required by another package… fslpy? if I am not mistaken? am sure it started by f… only other option is flake8

Sure, will do a new conda Env or using docker to check this. Super thx!

I haven’t saved the message but the pip error (which btw contained a line about the recent changes in the packages dependency system, so could be due to that) stated that dataclasses was missing and required by another package… fslpy? if I am not mistaken? am sure it started by f… only other option is flake8