datacube-core: nodata for product definition doesnt match dtype

247 | ga_srtm_dem1sv1_0 | {"platform": {"code": "Space Shuttle Endeavour"}, "instrument": {"name": "SIR"}, "product_type": "DEM"} |                 1 | {"name": "ga_srtm_dem1sv1_0", "storage": {"crs": "EPSG:4326", "resolution": {"latitude": -0.00027777777778, "longitude": 0.00027777777778}}, "metadata": {"platform": {"code": "Space Shuttle Endeavour"}, "instrument": {"name": "SIR"}, "product_type": "DEM"}, "description": "DEM 1sec Version 1.0", "measurements": [{"name": "dem", "dtype": "float32", "units": "metre", "nodata": -340282350000000000000000000000000000000}, {"name": "dem_s", "dtype": "float32", "units": "metre", "nodata": -340282350000000000000000000000000000000}, {"name": "dem_h", "dtype": "float32", "units": "metre", "nodata": -340282350000000000000000000000000000000}], "metadata_type": "eo"} | 2020-09-25 04:29:16.760552+00 | ows      | 2021-07-01 01:07:29.571853+00
(1 row)

eo3-validate type handling needs update

About this issue

Original URL
State: open
Created 2 years ago
Comments: 15 (14 by maintainers)

Most upvoted comments

The real value is probably: -340282346638528859811704183484516925440 which is numpy.finfo(numpy.float32).min, but it lost precision when converting to string via scientific notation.

Our validator is correctly failing the product — should this issue go to ODC core?

@jeremyh

test in numpy_value_fits_dtype

        return np.all(np.array([value], dtype=dtype) == [value])

doesn’t check what it claims, it checks that value can round-trip to numpy and back, and that’s only appropriate for integer types. Value as supplied, is probably incorrect because of precision loss, but it is surely not outside of valid range for float32 number.

Kirill888 on Nov 9, 2022