netcdf-c: DAP server error when trying to subset a variable

We are seeing an inconsistent NC_EDAPSVC error when trying read a subset from variables in http://opendap.nccs.nasa.gov:80/dods/GEOS-5/fp/0.25_deg/assim/inst3_3d_asm_Np file.

For example, the error happens if trying to subset into the variable right away, but doesn’t happen if you first read the entire variable. In this file, seems like all variables except for lev, lat, and lon have this issue.

Here is standalone reproduction code that uses “time” variable as an example:

#include <iostream>
#include <vector>
#include "netcdf.h"

void checkErrorCode(int status, const char* funcName){
    if (status != NC_NOERR){
        std::cout << "Error code: " << status << " from " << funcName << std::endl;
        std::cout << nc_strerror(status) << std::endl << std::endl;
    }
}

int main(int argc, const char * argv[]) {

    // open file
    int ncid = 0;
    int retval = nc_open("http://opendap.nccs.nasa.gov:80/dods/GEOS-5/fp/0.25_deg/assim/inst3_3d_asm_Np", NC_NOWRITE, &ncid);
    checkErrorCode(retval, "nc_open");
    
    // get ID for the variable
    int varid = 0;
    retval = nc_inq_varid(ncid, "time", &varid);
    checkErrorCode(retval, "nc_inq_varid");
    std::cout <<  "Variable ID is " << varid << std::endl;
    
    
//    // read the whole variable - works fine
//    // (it is a 1D variable with total count of 21214)
//    std::cout <<  std::endl << "Reading the whole 'time' variable" << std::endl;
//    const int totalCount = 21214;
//    std::vector<double> wholeVar(totalCount, 0.0);
//    
//    retval = nc_get_var_double(ncid, varid, wholeVar.data());
//    checkErrorCode(retval, "nc_get_var_double");
    
    // read a subset of the variable - errors out
    // (it is a 1D variable with total count of 21214)
    std::cout <<  std::endl << "Reading a subset of the 'time' variable" << std::endl;
    const int subsetCount = 10;
    std::vector<double> subsetVar(subsetCount, 0.0);
    size_t start[] = {0};
    size_t count[] = {subsetCount};
    ptrdiff_t stride[] = {1};
    
    retval = nc_get_vars_double(ncid, varid, start, count, stride, subsetVar.data());
    checkErrorCode(retval, "nc_get_vars_double");
    
    
    // read the whole variable - works fine
    // (it is a 1D variable with total count of 21214)
    std::cout <<  std::endl << "Reading the whole 'time' variable" << std::endl;
    const int totalCount = 21214;
    std::vector<double> wholeVar(totalCount, 0.0);
    
    retval = nc_get_var_double(ncid, varid, wholeVar.data());
    checkErrorCode(retval, "nc_get_var_double");
    
    // close file
    retval = nc_close(ncid);
    checkErrorCode(retval, "nc_close");
    
    return retval;
}

Here is the output this code produces (the subsetting errors out, but reading the whole variable works):

% ./readSubsetFromDAP.o                                                         
Variable ID is 0

Reading a subset of the 'time' variable
oc_open: server error retrieving url: code=0 message="constraint parsing failed; The variable `time%5b0' was not found in the dataset."Error code: -70 from nc_get_vars_double
NetCDF: DAP server error


Reading the whole 'time' variable

But if I switch the order and read the whole variable first and then do the subsetting, the error does not happen at all. I’ve also seen that sometimes subsetting in a way that still reads the whole variable (start=0, stride=1, count= dimension size) does not error, but sometimes it does.

I see some potentially related existing discussion about URL encoding and special characters like “[” (“%5b” mentioned in the error above): https://github.com/Unidata/netcdf-c/issues/1425#issuecomment-508207239 .

Would appreciate any suggestions on what might be the issue here and any possible workarounds!

We are using netCDF 4.9.2.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 24 (1 by maintainers)

Most upvoted comments

Ok, the error message is being generated by the server and recognized by the DAP2 code in libnetcdf-c. So, the problem must be investigated on the server side.

Actually, it is documented in the NUG (see DAP2.md in the Unidata/netcdf repo).

Ok, apparently a feature got lost from the documentation, so there is a work around. Try adding ‘encode=’ to the query as a fragment. So for example:

http://opendap.nccs.nasa.gov:80/dods/GEOS-5/fp/0.25_deg/assim/inst3_3d_asm_Np.dmr?time[0:10]#encode=

The encode flag turns off URL (%xx) encoding for a request and then turns on selected encoding. So for example:

  • encode= turns off all encoding
  • encode=path causes special characters in the URL path to be encoded
  • encode=query ditto for encoding query part of the URL
  • encode=path,query to encode both
  • encode=all is same as path,query

Not yet I got side tracked on another issue. I have time today.

Let me investigate.

oc_open: server error retrieving url: code=0 message=“not an available dataset”

This indicates that the client is getting an HTTP code of 0 from the server. I would have thought this was impossible, so I need to dig deeper…

Hi @DennisHeimbigner - sorry to nudge you, but anything you can say so far? Any additional data we can provide? Thank you for taking a look!

Kris

@DennisHeimbigner is there anything here that might be server side that occurs to you immediately?

Let me take a look.