datafusion: `stride` is not optional for new `array_slice` UDF

Describe the bug

The array_slice UDF takes 4 parameters.

https://github.com/apache/datafusion/blob/96487ea0cbb7901a1e4aa18fdf6deb8961319fea/datafusion/functions-array/src/extract.rs#L55-L61

Which means that args.len() is always 4 in array_slice_inner, even when called with stride = Expr::Null or stride = Expr::Int64(None)

https://github.com/apache/datafusion/blob/96487ea0cbb7901a1e4aa18fdf6deb8961319fea/datafusion/functions-array/src/extract.rs#L289-L293

To Reproduce

I encountered this while creating the python wrapper for datafusion-python and had to set stride=1 else the function would panic.

https://github.com/apache/datafusion-python/blob/7ad526caf140f7eb76ec541c903027d49693b58f/src/functions.rs#L104-L111

Expected behavior

The stride behavior in the UDF should match what’s available in the CLI

CREATE TEMPORARY VIEW data3 AS VALUES ([1.0, 2.0, 3.0, 3.0]), ([4.0, 5.0, 3.0]), ([6.0]);
❯ select * from data3;
+----------------------+
| column1              |
+----------------------+
| [1.0, 2.0, 3.0, 3.0] |
| [4.0, 5.0, 3.0]      |
| [6.0]                |
+----------------------+

> select * from array_slice(data3.column1, -1, 2)
+-----------------------------------------------+
| array_slice(data3.column1,Int64(-1),Int64(2)) |
+-----------------------------------------------+
| []                                            |
| []                                            |
| [6.0]                                         |
+-----------------------------------------------+

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

I submitted a formal PR #10469 to address this.

I recall hitting something like this with the substr/substring function. One would think they would be identical however they were not (since rust doesn’t do variadic functions nor does it allow defaults for args ala scala or Kotlin). For the substr it was expanded to two functions that internally used a vec<> and handled the logic of using the optional 3rd argument there.

    #[doc = "substring from the `position` to the end"]
    pub fn substr(string: Expr, position: Expr) -> Expr {
        super::substr().call(vec![string, position])
    }

    #[doc = "substring from the `position` with `length` characters"]
    pub fn substring(string: Expr, position: Expr, length: Expr) -> Expr {
        super::substr().call(vec![string, position, length])
    }
    ```

First, I’d like to emphasize that if passing stride=1 to the UDF worked the same as not providing a stride in the SQL api, then this would be just porcelain / a nicety. Reposting the example from: https://github.com/apache/datafusion/issues/10425

CREATE TEMPORARY VIEW data3 AS VALUES ([1.0, 2.0, 3.0, 3.0]), ([4.0, 5.0, 3.0]), ([6.0]);
❯ select * from data3;
+----------------------+
| column1              |
+----------------------+
| [1.0, 2.0, 3.0, 3.0] |
| [4.0, 5.0, 3.0]      |
| [6.0]                |
+----------------------+
❯ select array_slice(column1, -1, 2) from data3;
+-----------------------------------------------+
| array_slice(data3.column1,Int64(-1),Int64(2)) |
+-----------------------------------------------+
| []                                            |
| []                                            |
| [6.0]                                         |
+-----------------------------------------------+
❯ select array_slice(column1, -1, 2, 1) from data3;

thread 'main' panicked at /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-data-51.0.0/src/transform/primitive.rs:31:43:
range end index 9 out of range for slice of length 8

Now to this issue, I agree that I like the explicit arguments versus passing a Vec, and I would hope the solution doesn’t require keeping two separate UDFs.

I created a draft with what I would think is the natural rust solution in #10450. I don’t know if this violates some datafusion constraint, but all the tests pass.

#[doc = "returns a slice of the array."]
pub fn array_slice(array: Expr, begin: Expr, end: Expr, stride: Option<Expr>) -> Expr {
    if let Some(stride) = stride {
        Expr::ScalarFunction(ScalarFunction::new_udf(
            array_slice_udf(),
            vec![array, begin, end, stride],
        ))
    } else {
        Expr::ScalarFunction(ScalarFunction::new_udf(
            array_slice_udf(),
            vec![array, begin, end],
        ))
    }
}

If that is not possible, then a second proposal would be to amend array_slice_inner w/ some null check.

    let stride = if args_len == 4 & is_not_null(&args[3]) {
        Some(as_int64_array(&args[3])?)
    } else {
        None
    };

But again, this is just a nice-to-have if passing stride=1 worked the same as not providing it in the SQL API.

I agree it is helpful if we support expr_fn::array_slice(array, begin, end). We can’t only provide three args for it now

But I can’t think of any good solution to this

What you are expecting is something not supported in rust – variadic function

fn array_slice(array:Expr, begin:Expr, end:Expr) {
    // with stride 1
}

fn array_slice(array:Expr, begin:Expr, end:Expr, stride:Expr) {
    // with stride
}

I think the issue here should be handled in datafusion-python

I encountered this while creating the python wrapper for datafusion-python and had to set stride=1 else the function would panic.

@Michael-J-Ward Did you successfully create expression without stride=1 before 38? It would be surprising if it works previously.