netcdf-c: `read` operations cause too many disk reads with unlimited dimensions
@lesserwhirls - the problem as far as I can tell exists in netcdf-java as well.
This is a repeat of this issue which can probably be closed in favor of this one.
The problem: we’re hosting netcdf-classic files with in EFS and reading them from EC2, so we’re charged per bytes read. The charges have been unexpectedly high (order of magnitude) for reads from our datasets with unlimited dimensions.
I’ve attached the results of some testing I’ve done locally on my mac using fs-usage, input files, the output of those tests, and the python script I used to count bytes read. The short summary is output from reading a fixed dimension dataset are about what we expect with both ncdump and toolsui, but way too high when reading from a record dataset.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20 (9 by maintainers)
Can anyone suggest some form of caching that might help alleviate this problem?
I’ll take a look when I get back to Colorado next week, when I have access to a MacOS machine with
sudo. Thanks!It should be portable to a mac! The quick synopsis is:
fs_usage -w -f filesysand >> the results into a text filencdumpcommand I’m testing in a another windowfs_usagefs~usageand run again on next test caseThen once I had all the logs I parsed them with python regex to find the number of bytes recorded as
readbyncdump. I tried with grep and awk, but it was easier for me with python, I’m sure you’d have better luck than I there.A couple of things I’ve learned while staring at these logs for hours:
preadinstead ofread(which totally makes sense but that caught me up for a while when I was only searching for reads)Interesting; I’m taking a look at this now. I’m in rural Oregon (again), forgive the late response. I’m curious if the same behavior is observed with h5dump; it will be helpful to figure out if this is something in
libnetcdforlibhdf5.