datachain: Empty save() does not works as query
Empty save() in datachain query fails with Internal error on creating dataset error.
Query example:
from datachain import DataChain
DataChain.from_storage("gs://dvcx-datacomp-small/metadata", anon=True).save()
This happens because if name is not set, temp dataset name will be used. At the same time, after save() is finishes, temp datasets are cleaned up.
Some explanation:
Empty save is needed because operations are lazy. Chains are not executed immediately and intermediate results are not stored. For example, if we want to create
dc_filtered_1anddc_embeddingsfromdc, withoutsave()intermediatedcchain will be executed twice, for each children.
Couple options here:
- Skip saved temp dataset on cleanup
- Use
exec()instead of emptysave(). Need to check if this is proper replace + need to addexec()to docs.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 25 (25 by maintainers)
Note that this happens as a part of session cleanup, not during
DatasetQuery.cleanup().https://github.com/iterative/datachain/blob/0eb79599a0d774d89b5ff5a34db2b3dcba9a5016/src/datachain/query/session.py#L70
If it’s small fix - yes, that’s wold be great. Otherwise, I’d just improve the message.
I mean… yes, adjusting an error message is just a shortcut. The assumption is - we don’t want spend time on this now (does not seem a priority for now) and we can assume that users do not do save() when compute-persist is not needed.
from_storage(...).save()orsave("mycats").save()ideally should work just fine (a copy of a table is not even needed - can be optimized).This is expected behavior, just error message is bad / wrong.
Temp datasets should be used inside a query for optimization purposed, and not as a result of a query. We should just catch this error and write better error message.
On Studio, we always save the dataset (with
catalog.query(save=True).https://github.com/iterative/datachain/blob/49141a5b069c7d13d90a1798659faacf494e1e97/src/datachain/catalog/catalog.py#L1988-L1990
We need this for showing preview results.
Here the fix is to disable cleanup on session, which fixes the error.
We can discuss what we should do about dangling datasets. But that is already the case.
@shcheklein the expected behaviour is in the docstring:
https://github.com/iterative/datachain/blob/49141a5b069c7d13d90a1798659faacf494e1e97/src/datachain/lib/dc.py#L562-L571
I think we should disable Session cleanup on Studio inside the scripts.
We can decide if we want to delete the datasets or hide it, but that should happen after the script runs.
Looks like this is raised on Studio side.
https://github.com/iterative/studio/blob/6e7ed3fe9b30ef2631b7cee97e5eb621263e155b/backend/datachain_worker/tasks/run_job.py#L367