datachain: Column type inference fails after using mutate to subtract two columns
Description
I am creating two columns with cosine_distance and then I try to create a new column by subtracting one from the other. The code looks like this:
from sqlalchemy import func
embeddings_differences = (
DataChain
.from_dataset("embeddings")
.mutate(
query_sim_new = 1 - cosine_distance(C.document.embeddings_new, embedded_query_new),
query_sim_old = 1 - cosine_distance(C.document.embeddings_old, embedded_query_old),
)
.mutate(abs_difference = func.abs(C.query_sim_old - C.query_sim_new))
.save("embeddings-differences")
)
After running this I get this error:
CompileError: (in table 'ds_embeddings-differences_1', column 'abs_difference'): Can't generate DDL for NullType(); did you forget to specify a type on this Column?
This fails even if I remove func.abs, i.e. when this is just an ordinary subtraction of the two columns. If I try to save the dataset after the first .mutate, load it and then save it as a different dataset after calling the second .mutate it fails as well.
If I simply try to show the results with .show or convert it to pandas, it works as expected.
How to reproduce:
Run this notebook and see the output of the cell with the code above.
Version Info
0.2.12.dev36+g0b3cf72
Python 3.12.4
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 16 (16 by maintainers)
@ilongin thank you! This is a minimum reproducible example:
@tibor-mach created new issue for
order_byaftermutate… looks like this wasn’t working even beforeDataChain(when we had onlyDatasetQuery) but no one have ran into it before you now. https://github.com/iterative/datachain/issues/166@ilongin the workaround does seem to work but it also creates some new issues, in my example above, it breaks the
.order_bymethod with the following error. It seems it thinks that the value is textual even though I cast it as Float.The code is the same as above, only that one cell is rewritten like this:
@ilongin hmm, this is kind of awkward, particularly with these simple operations it is kind of a hassle to have to manually cast the output. I wonder if we couldn’t just include our custom automatic inference.
But I will use the workaround for now ,thanks!
@tibor-mach Added PR https://github.com/iterative/datachain/pull/152
This PR just adds better error handling for this as I don’t think we can fix it in better way. The workaround, and only solution I think for now, is to manually cast to specific type when type cannot be infered, like in subtract example. In theory we could have complex logic to infer type ourselves if sqlalchemy fails to do so, but I have a feeling it would be too complicated as there is a reason sqlalchemy doesn’t do it.
Until we have
castfunction inDataChain(PR is merged) you can use it fromSQLAlchemydirectly, e.gfrom sqlalchemy import cast. Examples of how to cast to specific types are in the PRThe main cause for this is that subtract (and division) binary operations ends up as
NullType()in SQLAlchemy for some reasonExample:
Trying to investigate why is this the case and how can be fixed