TileDB-Py: Reading sparse tiledb array abruptly exits

I have a large sparse array with about 400 million entries (3D point entries with x,y,z as dimensions and some point information like intensity, etc. as attributes). Unfortunatley when I want to read the pointcloud with point_cloud = db.df[:] the python instance just exits without an error. My first thought was not enough memory, but the I have 64GB of memory and during processing it raised to only about 25GB, so I can rule out “out of memory”

I have new insights to this issue: the tiledb query only exits, if I have the following settings for creating the array:

  • tiledb.default_ctx({“sm.dedup_coords”: “true”})
  • allows_duplicates=False,
tiledb.default_ctx({"sm.dedup_coords": "true"})

dom = tiledb.Domain(
    tiledb.Dim(
        name="X",
        domain=(np.finfo(np.float64).min, np.finfo(np.float64).max),
        dtype=np.float64,
        filters=filter_list,
    ),
    tiledb.Dim(
        name="Y",
        domain=(np.finfo(np.float64).min, np.finfo(np.float64).max),
        dtype=np.float64,
        filters=filter_list,
    ),
    tiledb.Dim(
        name="Z",
        domain=(np.finfo(np.float64).min, np.finfo(np.float64).max),
        dtype=np.float64,
        filters=filter_list,
    ),
)
# Create schema
schema = tiledb.ArraySchema(
    domain=dom,
    attrs=[
        tiledb.Attr(name="Intensity", dtype=np.uint16),
        tiledb.Attr(name="Label", dtype=np.uint16),
        tiledb.Attr(name="PointId", dtype=np.uint64),
    ],
    cell_order="hilbert",
    tile_order=None,
    capacity=100000,
    sparse=True,
    allows_duplicates=False,
)
# Create the TileDB array
tiledb.SparseArray.create(tiledb_array_name, schema)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (5 by maintainers)

Most upvoted comments

Thanks @weidinger-c, we have very little documentation on Hilbert. I filed a ticket to get that fixed soon.

On your question in your second message, how did you rebuilt tiledb? It should be fixed.