slideflow: Concatenate error with multimodal input

Description

Hello,

I have a couple of clinical variables (categorical and float) that I want to use as additional input. I list them as: multi_input = [“age”, “sex”, “height”, “weight”]. Anyway, when I try to train a (very basic) model, I receive an error saying that tensor shapes don’t match:

ConcatOp : Ranks of all input tensors should match: shape[0] = [16,4,1] vs. shape[1] = [16,2048] [[{{node model/input_merge/concat}}]] [Op:__inference_train_function_35947]

I believe there’s something wrong with the way how the clinical variables are introduced into the other layers (slide_feature_input)? How can I change it? Or am I missing something trivial? Would be great if anybody could help! Cheers.

To Reproduce

Steps to reproduce the behavior:

1. Commands

import slideflow as sf
P = sf.load_project('project')
hp = sf.ModelParams(
    tile_px=299,
    tile_um=100,
)
multi_input = ["age", "sex", "height", "weight"]
P.train(
        'category',
        params=hp,
        val_strategy='none',
        input_header= multi_input,
        )

2. Output

[11:45:54] INFO Training model category-HP0…
INFO Hyperparameters: {
“augment”: “xyrj”,
“batch_size”: 16,
“drop_images”: false,
“dropout”: 0,
“early_stop”: false,
“early_stop_method”: “loss”,
“early_stop_patience”: 0,
“epochs”: [
3
],
“hidden_layer_width”: 500,
“hidden_layers”: 0,
“include_top”: true,
“l1”: 0.0,
“l1_dense”: 0.0,
“l2”: 0.0,
“l2_dense”: 0.0,
“learning_rate”: 0.0001,
“learning_rate_decay”: 0,
“learning_rate_decay_steps”: 100000,
“loss”: “sparse_categorical_crossentropy”,
“manual_early_stop_batch”: null,
“manual_early_stop_epoch”: null,
“model”: “xception”,
“normalizer”: null,
“normalizer_source”: null,
“optimizer”: “Adam”,
“pooling”: “max”,
“tile_px”: 299,
“tile_um”: 100,
“toplayer_epochs”: 0,
“trainable_layers”: 0,
“training_balance”: “category”,
“uq”: false,
“validation_balance”: “none”
}
INFO Val settings: {
“strategy”: “none”,
“k_fold”: 3,
“k”: null,
“k_fold_header”: null,
“fraction”: null,
“source”: null,
“annotations”: null,
“filters”: null,
“dataset”: null
}
INFO Using 687 training TFRecords, 0 validation
INFO Adding input variable age as float
INFO Adding input variable sex as float
INFO Adding input variable height as float
INFO Adding input variable weight as float
[11:46:28] INFO Training with both images and 4 categories of slide-level input
2023-05-21 11:46:28.822013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30948 MB memory: -> device: 0, nam e: Tesla V100-SXM2-32GB, pci bus id: 0000:61:00.0, compute capability: 7.0
[11:46:29] INFO Using pretraining from imagenet

Model: “model”

Layer (type) Output Shape Param # Connected to

tile_image (InputLayer) [(None, 299, 299, 3 0 []
)]

xception (Functional) (None, 2048) 20861480 [‘tile_image[0][0]’]

slide_feature_input (InputLaye [(None, 4)] 0 []
r)

post_convolution (Activation) (None, 2048) 0 [‘xception[0][0]’]

input_merge (Concatenate) (None, 2052) 0 [‘slide_feature_input[0][0]’,
‘post_convolution[0][0]’]

logits-0 (Dense) (None, 2) 4106 [‘input_merge[0][0]’]

out-0 (Activation) (None, 2) 0 [‘logits-0[0][0]’]

Total params: 20,865,586
Trainable params: 20,811,058
Non-trainable params: 54,528

[11:46:37] INFO Beginning training
Epoch 1/3
2023-05-21 11:46:51.242360: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:428] Loaded cuDNN version 8201
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 3378, in train
self._train_hp(
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 709, in _train_hp
self._train_split(dataset, hp, val_settings, s_args)
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 933, in _train_split
project_utils._train_worker(
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project_utils.py”, line 147, in _train_worker
results = trainer.train(train_dts, val_dts, **training_kw)
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/model/tensorflow.py”, line 1925, in train
self.model.fit(
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/tensorflow/python/eager/execute.py”, line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node ‘model/input_merge/concat’ defined at (most recent call last): File “<stdin>”, line 1, in <module> File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 3378, in train self._train_hp( File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 709, in _train_hp self._train_split(dataset, hp, val_settings, s_args) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project.py”, line 933, in _train_split project_utils._train_worker( File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/project_utils.py”, line 147, in _train_worker results = trainer.train(train_dts, val_dts, **training_kw) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/slideflow/model/tensorflow.py”, line 1925, in train self.model.fit( File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1650, in fit tmp_logs = self.train_function(iterator) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1249, in train_function return step_function(self, iterator) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1233, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1222, in run_step outputs = model.train_step(data) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 1023, in train_step y_pred = self(x, training=True) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/training.py”, line 561, in call return super().call(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/base_layer.py”, line 1132, in call outputs = call_fn(inputs, *args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 96, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/functional.py”, line 511, in call return self._run_internal_graph(inputs, training=training, mask=mask) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/functional.py”, line 668, in _run_internal_graph outputs = node.layer(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/engine/base_layer.py”, line 1132, in call outputs = call_fn(inputs, *args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/utils/traceback_utils.py”, line 96, in error_handler return fn(*args, **kwargs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/layers/merging/base_merge.py”, line 196, in call return self._merge_function(inputs) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/layers/merging/concatenate.py”, line 134, in _merge_function return backend.concatenate(inputs, axis=self.axis) File “/data/jjung23/miniconda3/envs/sf_2/lib/python3.9/site-packages/keras/backend.py”, line 3572, in concatenate return tf.concat([to_dense(x) for x in tensors], axis) Node: ‘model/input_merge/concat’ ConcatOp : Ranks of all input tensors should match: shape[0] = [16,4,1] vs. shape[1] = [16,2048] [[{{node model/input_merge/concat}}]] [Op:__inference_train_function_35947]

Expected behavior

Successful training of a multimodal model.

Environment:

  • Slideflow Version (e.g., 1.0): 2.0.3-post1
  • OS (e.g., Ubuntu): Ubuntu 20.04.5
  • How you installed Slideflow (pip, source): pip install slideflow[tf] cucim cupy-cuda11x
  • Python version: 3.9
  • CUDA/cuDNN version: 11.6
  • GPU models and configuration: Tesla V100-SXM2-32GB
  • Any other relevant information:

Additional context

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 17

Commits related to this issue

Most upvoted comments

Hi Jinny - thanks for the question, this could be better clarified in the documentation.

The best way to determine what data was used for training/validation is to view the slide_manifest.csv file created in the model folder during training. This is a CSV file with three columns - the slide name, the outcome label, and the dataset (training/validation).

In addition to manually viewing the file, you can quickly and automatically pull the list of slides that were used for model training or validation using sf.util.get_slides_from_model_manifest(), specifying whether you want to retrieve the training or validation slides using the parameter dataset:

import slideflow as sf

model_path = '/path/to/saved_model'

val_slides = sf.util.get_slides_from_model_manifest(model_path, dataset='validation')

You can then use create a dataset from only those slides, and use that dataset for generating heatmaps, or other downstream tasks:

P = sf.Project(...)
val_dataset = P.dataset(..., filters={'slide': val_slides})

To answer your question more directly though, the splits.json has the slides/tfrecords split into the number of groups equal to your cross-fold (in your case, 3). For k-fold 1, the first group (A) is validation, and the remainder is training (B+C). For k-fold 2, the second group (B) is validation, and the remainder (A+C) is training. And so on.

I appreciate you asking, I realize now that I failed to include this information in the documentation. I’ll add a section in the documentation explaining this more clearly.

Let me know if that makes sense or if I can help clarify further!

Training MIL models with linear outcomes is under development! (see PR https://github.com/jamesdolezal/slideflow/pull/287). The plan is to add this in version 2.1, which is still 1-2 months out.