starfish: `write_experiment_json()` doesn't handle non-flat experiment directory structures correctly
Hi all,
So I’ve encountered an issue when running write_experiment_json() in ‘no copy’ mode (see:
https://github.com/spacetx/starfish/pull/1053)
The resulting output .json files that starfish produces all assume that image tiles reside in the experiment root directory (see: https://github.com/spacetx/slicedimage/blob/3456f86a10cc9c89996a394cb75a03457a9bf6bc/slicedimage/io.py#L303).
However, our data (and likely many other group’s data) do not live in a flat directory structure. Ours happens to look like:
expt_root_dir
| - Round_1_dir
| | - FOV_1_dir
| | | - channel_1_z_1.tiff
| | ...
| | - FOV_2_dir
| | | - channel_1_z_1.tiff
| | ...
| ...
| - Round_2_dir
| | - FOV_1_dir
| | | - channel_1_z_1.tiff
| | ...
| ...
...
Although this issue arises because Starfish can no longer dictate a flat data structure (by forcing copying), copying has to be optional as the overhead of creating a new copy of huge ‘local’ datasets is unreasonable.
From an end-user perspective, having image data organized into subdirectories is also much easier to deal with than having to deal with one folder with thousands of tiffs.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (7 by maintainers)
Commits related to this issue
- Allow partitions to be written to any relative path. Currently, we only take the basename of the partition path generated by `partition_path_generator`. We should accept any relative path. This cha... — committed to spacetx/slicedimage by deleted user 5 years ago
- Allow partitions to be written to any relative path. Currently, we only take the basename of the partition path generated by `partition_path_generator`. We should accept any relative path. This cha... — committed to spacetx/slicedimage by deleted user 5 years ago
- Permit writing of images to hierarchical on-disk representations. Rather than writing the basename of a referenced file, calculate the relative path from the current file and record that. Adds a tes... — committed to spacetx/slicedimage by deleted user 5 years ago
- Permit writing of images to hierarchical on-disk representations. Rather than writing the basename of a referenced file, calculate the relative path from the current file and record that. Adds a tes... — committed to spacetx/slicedimage by deleted user 5 years ago
- Permit writing of images to hierarchical on-disk representations. Rather than writing the basename of a referenced file, calculate the relative path from the current file and record that. Adds a tes... — committed to spacetx/slicedimage by deleted user 5 years ago
- Allow partitions to be written to any relative path. Currently, we only take the basename of the partition path generated by `partition_path_generator`. We should accept any relative path. This cha... — committed to spacetx/slicedimage by deleted user 5 years ago
- Permit writing of images to hierarchical on-disk representations. Rather than writing the basename of a referenced file, calculate the relative path from the current file and record that. Adds a tes... — committed to spacetx/slicedimage by deleted user 5 years ago
- Permit writing of images to hierarchical on-disk representations. Rather than writing the basename of a referenced file, calculate the relative path from the current file and record that. Adds a tes... — committed to spacetx/slicedimage by deleted user 5 years ago
- Permit writing of images to hierarchical on-disk representations. (#87) Rather than writing the basename of a referenced file, calculate the relative path from the current file and record that. Adds... — committed to spacetx/slicedimage by ttung 5 years ago
- Test to verify that inplace experiment construction works. This should nail the coffin on #1116, though there may still be usability improvements we can deliver. — committed to spacetx/starfish by deleted user 5 years ago
- Test to verify that inplace experiment construction works. (#1131) This should nail the coffin on #1116, though there may still be usability improvements we can deliver. — committed to spacetx/starfish by ttung 5 years ago
- Permit writing of images to hierarchical on-disk representations. (#87) Rather than writing the basename of a referenced file, calculate the relative path from the current file and record that. Adds... — committed to spacetx/slicedimage by ttung 5 years ago
After checking out starfish to 32ba1f30e2d0485387b5eb57dfa99f0c4f7b12b8 and selectively merging with 131608e901507dabbe1d11597f8b846fddce951b, things look to be working for me now. There was no need for me to change how FOVs are named.
Many thanks @ttung!