datachain: Empty save() does not works as query

Empty save() in datachain query fails with Internal error on creating dataset error.

Query example:

from datachain import DataChain

DataChain.from_storage("gs://dvcx-datacomp-small/metadata", anon=True).save()

This happens because if name is not set, temp dataset name will be used. At the same time, after save() is finishes, temp datasets are cleaned up.

Some explanation:

Empty save is needed because operations are lazy. Chains are not executed immediately and intermediate results are not stored. For example, if we want to create dc_filtered_1 and dc_embeddings from dc, without save() intermediate dc chain will be executed twice, for each children.

Couple options here:

  1. Skip saved temp dataset on cleanup
  2. Use exec() instead of empty save(). Need to check if this is proper replace + need to add exec() to docs.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 25 (25 by maintainers)

Most upvoted comments

Note that this happens as a part of session cleanup, not during DatasetQuery.cleanup().

https://github.com/iterative/datachain/blob/0eb79599a0d774d89b5ff5a34db2b3dcba9a5016/src/datachain/query/session.py#L70

I was thinking that it could be a small fix tbh - not sure on this.

If it’s small fix - yes, that’s wold be great. Otherwise, I’d just improve the message.

It might be inefficient (extra table that will GCed almost right away), but it’s not clear why it should not be possible.

I mean… yes, adjusting an error message is just a shortcut. The assumption is - we don’t want spend time on this now (does not seem a priority for now) and we can assume that users do not do save() when compute-persist is not needed.

from_storage(...).save() or save("mycats").save() ideally should work just fine (a copy of a table is not even needed - can be optimized).

This is expected behavior, just error message is bad / wrong.

Temp datasets should be used inside a query for optimization purposed, and not as a result of a query. We should just catch this error and write better error message.

okay, so, on the Studio side - what do we do with a query result (if there is no save() at the end) - how / where do we save it?

On Studio, we always save the dataset (with catalog.query(save=True).

https://github.com/iterative/datachain/blob/49141a5b069c7d13d90a1798659faacf494e1e97/src/datachain/catalog/catalog.py#L1988-L1990

We need this for showing preview results.

Here the fix is to disable cleanup on session, which fixes the error.

We can discuss what we should do about dangling datasets. But that is already the case.

I think we should disable Session cleanup on Studio inside the scripts.

We can decide if we want to delete the datasets or hide it, but that should happen after the script runs.