slideflow: Concatenate error with multimodal input
Description
Hello,
I have a couple of clinical variables (categorical and float) that I want to use as additional input. I list them as: multi_input = [“age”, “sex”, “height”, “weight”]. Anyway, when I try to train a (very basic) model, I receive an error saying that tensor shapes don’t match:
ConcatOp : Ranks of all input tensors should match: shape[0] = [16,4,1] vs. shape[1] = [16,2048] [[{{node model/input_merge/concat}}]] [Op:__inference_train_function_35947]
I believe there’s something wrong with the way how the clinical variables are introduced into the other layers (slide_feature_input)? How can I change it? Or am I missing something trivial? Would be great if anybody could help! Cheers.
To Reproduce
Steps to reproduce the behavior:
1. Commands
import slideflow as sf
P = sf.load_project('project')
hp = sf.ModelParams(
tile_px=299,
tile_um=100,
)
multi_input = ["age", "sex", "height", "weight"]
P.train(
'category',
params=hp,
val_strategy='none',
input_header= multi_input,
)
2. Output
[11:45:54] INFO Training model category-HP0…
INFO Hyperparameters: {
“augment”: “xyrj”,
“batch_size”: 16,
“drop_images”: false,
“dropout”: 0,
“early_stop”: false,
“early_stop_method”: “loss”,
“early_stop_patience”: 0,
“epochs”: [
3
],
“hidden_layer_width”: 500,
“hidden_layers”: 0,
“include_top”: true,
“l1”: 0.0,
“l1_dense”: 0.0,
“l2”: 0.0,
“l2_dense”: 0.0,
“learning_rate”: 0.0001,
“learning_rate_decay”: 0,
“learning_rate_decay_steps”: 100000,
“loss”: “sparse_categorical_crossentropy”,
“manual_early_stop_batch”: null,
“manual_early_stop_epoch”: null,
“model”: “xception”,
“normalizer”: null,
“normalizer_source”: null,
“optimizer”: “Adam”,
“pooling”: “max”,
“tile_px”: 299,
“tile_um”: 100,
“toplayer_epochs”: 0,
“trainable_layers”: 0,
“training_balance”: “category”,
“uq”: false,
“validation_balance”: “none”
}
INFO Val settings: {
“strategy”: “none”,
“k_fold”: 3,
“k”: null,
“k_fold_header”: null,
“fraction”: null,
“source”: null,
“annotations”: null,
“filters”: null,
“dataset”: null
}
INFO Using 687 training TFRecords, 0 validation
INFO Adding input variable age as float
INFO Adding input variable sex as float
INFO Adding input variable height as float
INFO Adding input variable weight as float
[11:46:28] INFO Training with both images and 4 categories of slide-level input
2023-05-21 11:46:28.822013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30948 MB memory: -> device: 0, nam
e: Tesla V100-SXM2-32GB, pci bus id: 0000:61:00.0, compute capability: 7.0
[11:46:29] INFO Using pretraining from imagenet
Model: “model”
Layer (type) Output Shape Param # Connected to
tile_image (InputLayer) [(None, 299, 299, 3 0 []
)]
xception (Functional) (None, 2048) 20861480 [‘tile_image[0][0]’]
slide_feature_input (InputLaye [(None, 4)] 0 []
r)
post_convolution (Activation) (None, 2048) 0 [‘xception[0][0]’]
input_merge (Concatenate) (None, 2052) 0 [‘slide_feature_input[0][0]’,
‘post_convolution[0][0]’]
logits-0 (Dense) (None, 2) 4106 [‘input_merge[0][0]’]
out-0 (Activation) (None, 2) 0 [‘logits-0[0][0]’]
Total params: 20,865,586
Trainable params: 20,811,058
Non-trainable params: 54,528
[11:46:37] INFO Beginning training
Epoch 1/3
2023-05-21 11:46:51.242360: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:428] Loaded cuDNN version 8201
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 3378, in train
self._train_hp(
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 709, in _train_hp
self._train_split(dataset, hp, val_settings, s_args)
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 933, in _train_split
project_utils._train_worker(
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project_utils.py”, line 147, in _train_worker
results = trainer.train(train_dts, val_dts, **training_kw)
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/model/tensorflow.py”, line 1925, in train
self.model.fit(
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/tensorflow/python/eager/execute.py”, line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
Detected at node ‘model/input_merge/concat’ defined at (most recent call last): File “<stdin>”, line 1, in <module> File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 3378, in train self._train_hp( File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 709, in _train_hp self._train_split(dataset, hp, val_settings, s_args) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 933, in _train_split project_utils._train_worker( File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project_utils.py”, line 147, in _train_worker results = trainer.train(train_dts, val_dts, **training_kw) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/model/tensorflow.py”, line 1925, in train self.model.fit( File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1650, in fit tmp_logs = self.train_function(iterator) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1249, in train_function return step_function(self, iterator) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1233, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1222, in run_step outputs = model.train_step(data) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1023, in train_step y_pred = self(x, training=True) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 561, in call return super().call(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/base_layer.py”, line 1132, in call outputs = call_fn(inputs, *args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 96, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/functional.py”, line 511, in call return self._run_internal_graph(inputs, training=training, mask=mask) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/functional.py”, line 668, in _run_internal_graph outputs = node.layer(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/base_layer.py”, line 1132, in call outputs = call_fn(inputs, *args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 96, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/layers/merging/base_merge.py”, line 196, in call return self._merge_function(inputs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/layers/merging/concatenate.py”, line 134, in _merge_function return backend.concatenate(inputs, axis=self.axis) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/backend.py”, line 3572, in concatenate return tf.concat([to_dense(x) for x in tensors], axis) Node: ‘model/input_merge/concat’ ConcatOp : Ranks of all input tensors should match: shape[0] = [16,4,1] vs. shape[1] = [16,2048] [[{{node model/input_merge/concat}}]] [Op:__inference_train_function_35947]
Expected behavior
Successful training of a multimodal model.
Environment:
- Slideflow Version (e.g., 1.0): 2.0.3-post1
- OS (e.g., Ubuntu): Ubuntu 20.04.5
- How you installed Slideflow (
pip, source): pip install slideflow[tf] cucim cupy-cuda11x - Python version: 3.9
- CUDA/cuDNN version: 11.6
- GPU models and configuration: Tesla V100-SXM2-32GB
- Any other relevant information:
Additional context
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17
Commits related to this issue
- Fix multi-modal input with continuous clinical variables (Tensorflow) [#282] - Fixes errors when using continuous clinical variables as additional slide input with Tensorflow backend - Separate patch... — committed to slideflow/slideflow by jamesdolezal 3 years ago
- Fix multi-modal input with continuous clinical variables (Tensorflow) [#282] - Fixes errors when using continuous clinical variables as additional slide input with Tensorflow backend - Separate patch... — committed to slideflow/slideflow by jamesdolezal 3 years ago
Hi Jinny - thanks for the question, this could be better clarified in the documentation.
The best way to determine what data was used for training/validation is to view the
slide_manifest.csvfile created in the model folder during training. This is a CSV file with three columns - the slide name, the outcome label, and the dataset (training/validation).In addition to manually viewing the file, you can quickly and automatically pull the list of slides that were used for model training or validation using
sf.util.get_slides_from_model_manifest(), specifying whether you want to retrieve the training or validation slides using the parameterdataset:You can then use create a dataset from only those slides, and use that dataset for generating heatmaps, or other downstream tasks:
To answer your question more directly though, the
splits.jsonhas the slides/tfrecords split into the number of groups equal to your cross-fold (in your case, 3). For k-fold 1, the first group (A) is validation, and the remainder is training (B+C). For k-fold 2, the second group (B) is validation, and the remainder (A+C) is training. And so on.I appreciate you asking, I realize now that I failed to include this information in the documentation. I’ll add a section in the documentation explaining this more clearly.
Let me know if that makes sense or if I can help clarify further!
Training MIL models with linear outcomes is under development! (see PR https://github.com/jamesdolezal/slideflow/pull/287). The plan is to add this in version 2.1, which is still 1-2 months out.