netcdf-c: `read` operations cause too many disk reads with unlimited dimensions

@lesserwhirls - the problem as far as I can tell exists in netcdf-java as well.

This is a repeat of this issue which can probably be closed in favor of this one.

The problem: we’re hosting netcdf-classic files with in EFS and reading them from EC2, so we’re charged per bytes read. The charges have been unexpectedly high (order of magnitude) for reads from our datasets with unlimited dimensions.

I’ve attached the results of some testing I’ve done locally on my mac using fs-usage, input files, the output of those tests, and the python script I used to count bytes read. The short summary is output from reading a fixed dimension dataset are about what we expect with both ncdump and toolsui, but way too high when reading from a record dataset.

test_disk_reads.zip

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 20 (9 by maintainers)

Most upvoted comments

Can anyone suggest some form of caching that might help alleviate this problem?

I’ll take a look when I get back to Colorado next week, when I have access to a MacOS machine with sudo. Thanks!

It should be portable to a mac! The quick synopsis is:

  • run fs_usage -w -f filesys and >> the results into a text file
  • run the ncdump command I’m testing in a another window
  • end fs_usage
  • change the output file for fs~usage and run again on next test case

Then once I had all the logs I parsed them with python regex to find the number of bytes recorded as read by ncdump. I tried with grep and awk, but it was easier for me with python, I’m sure you’d have better luck than I there.

A couple of things I’ve learned while staring at these logs for hours:

  • recording reads only is easier to parse, but if you record everything you can also see when ncdump opens and closes the actual dataset file, which is also useful.
  • With netcdf4, all the data reads are pread instead of read (which totally makes sense but that caught me up for a while when I was only searching for reads)

Interesting; I’m taking a look at this now. I’m in rural Oregon (again), forgive the late response. I’m curious if the same behavior is observed with h5dump; it will be helpful to figure out if this is something in libnetcdf or libhdf5.