TileDB-Py: Reading sparse tiledb array abruptly exits
I have a large sparse array with about 400 million entries (3D point entries with x,y,z as dimensions and some point information like intensity, etc. as attributes). Unfortunatley when I want to read the pointcloud with point_cloud = db.df[:] the python instance just exits without an error.
My first thought was not enough memory, but the I have 64GB of memory and during processing it raised to only about 25GB, so I can rule out “out of memory”
I have new insights to this issue: the tiledb query only exits, if I have the following settings for creating the array:
- tiledb.default_ctx({“sm.dedup_coords”: “true”})
- allows_duplicates=False,
tiledb.default_ctx({"sm.dedup_coords": "true"})
dom = tiledb.Domain(
tiledb.Dim(
name="X",
domain=(np.finfo(np.float64).min, np.finfo(np.float64).max),
dtype=np.float64,
filters=filter_list,
),
tiledb.Dim(
name="Y",
domain=(np.finfo(np.float64).min, np.finfo(np.float64).max),
dtype=np.float64,
filters=filter_list,
),
tiledb.Dim(
name="Z",
domain=(np.finfo(np.float64).min, np.finfo(np.float64).max),
dtype=np.float64,
filters=filter_list,
),
)
# Create schema
schema = tiledb.ArraySchema(
domain=dom,
attrs=[
tiledb.Attr(name="Intensity", dtype=np.uint16),
tiledb.Attr(name="Label", dtype=np.uint16),
tiledb.Attr(name="PointId", dtype=np.uint64),
],
cell_order="hilbert",
tile_order=None,
capacity=100000,
sparse=True,
allows_duplicates=False,
)
# Create the TileDB array
tiledb.SparseArray.create(tiledb_array_name, schema)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (5 by maintainers)
Thanks @weidinger-c, we have very little documentation on Hilbert. I filed a ticket to get that fixed soon.
On your question in your second message, how did you rebuilt tiledb? It should be fixed.
Thank you, we are investigating.
https://app.shortcut.com/tiledb-inc/story/44758