netcdf-c: DAP server error when trying to subset a variable
We are seeing an inconsistent NC_EDAPSVC error when trying read a subset from variables in http://opendap.nccs.nasa.gov:80/dods/GEOS-5/fp/0.25_deg/assim/inst3_3d_asm_Np file.
For example, the error happens if trying to subset into the variable right away, but doesn’t happen if you first read the entire variable. In this file, seems like all variables except for lev, lat, and lon have this issue.
Here is standalone reproduction code that uses “time” variable as an example:
#include <iostream>
#include <vector>
#include "netcdf.h"
void checkErrorCode(int status, const char* funcName){
if (status != NC_NOERR){
std::cout << "Error code: " << status << " from " << funcName << std::endl;
std::cout << nc_strerror(status) << std::endl << std::endl;
}
}
int main(int argc, const char * argv[]) {
// open file
int ncid = 0;
int retval = nc_open("http://opendap.nccs.nasa.gov:80/dods/GEOS-5/fp/0.25_deg/assim/inst3_3d_asm_Np", NC_NOWRITE, &ncid);
checkErrorCode(retval, "nc_open");
// get ID for the variable
int varid = 0;
retval = nc_inq_varid(ncid, "time", &varid);
checkErrorCode(retval, "nc_inq_varid");
std::cout << "Variable ID is " << varid << std::endl;
// // read the whole variable - works fine
// // (it is a 1D variable with total count of 21214)
// std::cout << std::endl << "Reading the whole 'time' variable" << std::endl;
// const int totalCount = 21214;
// std::vector<double> wholeVar(totalCount, 0.0);
//
// retval = nc_get_var_double(ncid, varid, wholeVar.data());
// checkErrorCode(retval, "nc_get_var_double");
// read a subset of the variable - errors out
// (it is a 1D variable with total count of 21214)
std::cout << std::endl << "Reading a subset of the 'time' variable" << std::endl;
const int subsetCount = 10;
std::vector<double> subsetVar(subsetCount, 0.0);
size_t start[] = {0};
size_t count[] = {subsetCount};
ptrdiff_t stride[] = {1};
retval = nc_get_vars_double(ncid, varid, start, count, stride, subsetVar.data());
checkErrorCode(retval, "nc_get_vars_double");
// read the whole variable - works fine
// (it is a 1D variable with total count of 21214)
std::cout << std::endl << "Reading the whole 'time' variable" << std::endl;
const int totalCount = 21214;
std::vector<double> wholeVar(totalCount, 0.0);
retval = nc_get_var_double(ncid, varid, wholeVar.data());
checkErrorCode(retval, "nc_get_var_double");
// close file
retval = nc_close(ncid);
checkErrorCode(retval, "nc_close");
return retval;
}
Here is the output this code produces (the subsetting errors out, but reading the whole variable works):
% ./readSubsetFromDAP.o
Variable ID is 0
Reading a subset of the 'time' variable
oc_open: server error retrieving url: code=0 message="constraint parsing failed; The variable `time%5b0' was not found in the dataset."Error code: -70 from nc_get_vars_double
NetCDF: DAP server error
Reading the whole 'time' variable
But if I switch the order and read the whole variable first and then do the subsetting, the error does not happen at all. I’ve also seen that sometimes subsetting in a way that still reads the whole variable (start=0, stride=1, count= dimension size) does not error, but sometimes it does.
I see some potentially related existing discussion about URL encoding and special characters like “[” (“%5b” mentioned in the error above): https://github.com/Unidata/netcdf-c/issues/1425#issuecomment-508207239 .
Would appreciate any suggestions on what might be the issue here and any possible workarounds!
We are using netCDF 4.9.2.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 24 (1 by maintainers)
Ok, the error message is being generated by the server and recognized by the DAP2 code in libnetcdf-c. So, the problem must be investigated on the server side.
Actually, it is documented in the NUG (see DAP2.md in the Unidata/netcdf repo).
Ok, apparently a feature got lost from the documentation, so there is a work around. Try adding ‘encode=’ to the query as a fragment. So for example:
The encode flag turns off URL (%xx) encoding for a request and then turns on selected encoding. So for example:
Not yet I got side tracked on another issue. I have time today.
Let me investigate.
This indicates that the client is getting an HTTP code of 0 from the server. I would have thought this was impossible, so I need to dig deeper…
Hi @DennisHeimbigner - sorry to nudge you, but anything you can say so far? Any additional data we can provide? Thank you for taking a look!
Kris
Let me take a look.