netcdf-c: nc_get_var_string and ncdump throw HDF error on an NC_STRING variable with one unlimited dimension and written with an offset

I am able to create this NC_STRING variable with one limited and one unlimited dimension, and write to it some data with an offset without issue. But when trying to read it back with nc_get_var_string, I am getting a NC_EHDFERR (-101). When trying to use ncdump on the file, I also see “NetCDF: HDF error”. However, I am able to read it fine with nc_get_vars_string.

Here are the repro steps (attached here as well: repro.cpp.zip):

#include <iostream>
#include <vector>
#include "netcdf.h"

void checkErrorCode(int status, const char* message){
    if (status != NC_NOERR){
        std::cout << "Error code: " << status << " from " << message << std::endl;
    }
}

int main(int argc, const char * argv[]) {
    

    // set up data
    
    const char** charPointers = new const char*[4];
    std::string* stringData = new std::string[4];
    
    stringData[0] = "a";
    charPointers[0] = stringData[0].c_str();
    
    stringData[1] = "aa";
    charPointers[1] = stringData[1].c_str();
    
    stringData[2] = "b";
    charPointers[2] = stringData[2].c_str();
    
    stringData[3] = "bb";
    charPointers[3] = stringData[3].c_str();
    
    
    int ncid;
    int retval;
    
    // -- WRITE --
    
    retval = nc_create("myfile.nc", NC_NETCDF4, &ncid);
    checkErrorCode(retval, "nc_create");
    
    // Define dimensions
    int dimid_a;
    retval = nc_def_dim(ncid, "a", NC_UNLIMITED, &dimid_a);
    checkErrorCode(retval, "nc_def_dim for 'a'");
    
    int dimid_b;
    retval = nc_def_dim(ncid, "b", 10, &dimid_b);
    checkErrorCode(retval, "nc_def_dim for 'b'");

    int dimids[2] = {dimid_b, dimid_a};
    
    // Define variable
    int varid;
    retval = nc_def_var(ncid, "var", NC_STRING, 2, dimids, &varid);
    checkErrorCode(retval, "nc_def_var");
        
    
    // Put variable
    size_t start[2] = {0, 1};
    size_t count[2] = {2, 2};
    ptrdiff_t stride[2] = {1, 1};
    
    retval = nc_put_vars_string(ncid, varid, start, count, stride, charPointers);
    checkErrorCode(retval, "nc_put_vars_string");
        

    retval = nc_close(ncid);
    checkErrorCode(retval, "nc_close(1)");
    
    
    // -- READ --
    retval = nc_open("myfile.nc", NC_NOWRITE, &ncid);
    checkErrorCode(retval, "nc_open");
    
    // get dimensions
    int ndims, dimids_read[NC_MAX_VAR_DIMS];
    retval = nc_inq_var(ncid, varid, NULL, NULL, &ndims, dimids_read, NULL);
    checkErrorCode(retval, "nc_inq_var");

    // calculate num elements to read
    size_t dimlen = 0;
    size_t num_items = 1;
    for ( int j = 0; j < ndims; ++j ) {
        retval = nc_inq_dimlen(ncid, dimids_read[j], &dimlen);
        checkErrorCode(retval, "nc_inq_dimlen");
        num_items *= dimlen;
    }
    
    // get var
    char** readStrings = new char*[num_items];
    retval = nc_get_var_string(ncid, varid, readStrings);
    checkErrorCode(retval, "nc_get_var_string");

    retval = nc_close(ncid);
    checkErrorCode(retval, "nc_close(2)");
    
    return retval;
}

The output when run:

$ ./a.out 
Error code: -101 from nc_get_var_string
$ ls
a.out*		myfile.nc	repro.cpp
$ ncdump myfile.nc 
netcdf myfile {
dimensions:
	a = UNLIMITED ; // (3 currently)
	b = 10 ;
variables:
	string var(b, a) ;
data:

 var =
NetCDF: HDF error
Location: file <..>/netcdf/ncdump/vardata.c; line 478

As far as I could tell, to reproduce the issue there has to be >1 dimension, one has to be unlimited, and when writing data the offset needs to be in the unlimited dimension. I also only see this with NC_STRING data.

I am seeing this with netcdf 4.7.4 and the above repro steps are run on macOS 10.15.6 (but we are seeing it on other platforms as well).

Is this a known bug or limitation? I know nc_get_var_string has a warning about using it with unlimited dimensions, but this issue seems specific to NC_STRING

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 18 (3 by maintainers)

Most upvoted comments

@krisfed thanks for checking in; we are working through. It can be tricky to determine what will make it into the release, given the resources we have to work with, but we will try to have this in the 4.9.1 release 😃

Thank you Dennis for the confirmation! No worries, I realize this is an edge-case-y issue…

That is my guess. So we need to look at the string code and look for ways it is handled differently than vlen.

I did an experiment in which I used char(*) t (i.e. VLEN of t) as the type of the variable. This should be roughly equivalent to using NC_STRING type because internally this is how NC_STRING is stored. Assuming I got it right, then this works to produce the file. However ncdump crashes on it for some reason. I am investigating this latter problem separately. But h5dump on myfile.nc appears correct. I am attaching my program for verification (I converted it to C). It has extension .txt instead of .c so that github will allow the attachment. svlen.txt