duckdb: union_by_name is vastly slower/fails over httpfs

What happens?

A query as simple as select max(date) from read_parquet('s3://bucket/**/*.parquet', hive_partitioning=true, union_by_name=true) becomes unusably slow, seemingly, as the set of files to scan increases.

• I am enabling union_by_name preemptively, because i know that the set of columns being written to these files may increase over time, which will lead to historically written files lacking newer columns.

• by “unusably slow” i mean, firstly it takes a long time, but also it begins producing duckdb.IOException: IO Error: Connection error for HTTP HEAD to <bucket http url>

• In my particular example date is a hive partition key, which should mean (i think) that there’s not even any data to scan.

To Reproduce

S3 bucket with structure like:

date=2023-01-01/part0.parquet
date=2023-01-02/part0.parquet
...
date=2023-06-20/part0.parquet

File can be as simple as (parquet equivalent): date, id\n{date},1

For 300 dates worth of this ^: locally, select max(date) from read_parquet('s3://bucket/**/*.parquet', hive_partitioning=true, union_by_name=true) takes ~4m, whereas union_by_name=false is essentially instant.

OS:

aarch64

DuckDB Version:

0.8.0/0.8.1

DuckDB Client:

Python

Full Name:

Dan Cardin

Affiliation:

Known

Have you tried this on the latest master branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree