rill: `rill init --db ` does not add DuckDB tables to the Rill catalog

To reproduce w/ the AdBids dataset:

Download the AdBids dataset: wget https://raw.githubusercontent.com/rilldata/rill-developer/main/web-local/test/data/AdBids.csv
Start DuckDB w/ a persistent database file: duckdb test.db
Create a DuckDB table: CREATE TABLE adbids AS SELECT * FROM read_csv_auto('AdBids.csv');
Exit duckdb: .exit
Initialize Rill with the DuckDB file: rill init --db test.db
Start rill: rill start
Observe that queries to the adbids table work, but there are no sources listed in the assets sidebar
Exit Rill, open test.db w/ DuckDB, and see that there are no entries in rill.catalog

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (16 by maintainers)

Most upvoted comments

@ericpgreen2 I defer to @hamilton here, but my take is that I do not think that is worth the effort to pull this code from our application because we will want to get back to this and fix it. We will always want a CLI path for existing duckDB databases just like we do for the new remote sources. If users want to try an undocumented and unsupported CLI feature for a Tech Preview of our tool I think that is ok.

magorlick on Nov 2, 2022

I’ve got functionality for discovering existing tables and adding them to the catalog (and exposing them over the catalog APIs) in a branch that I’m merging this week. That should allow us to render existing tables in the sidebar. I would argue for listing them under “sources” – maybe add a small icon or something to differentiate them.

I initially also thought we should consider existing tables as sources, but have changed my mind. On the backend, they’re easier to deal with as a separate concept. On the frontend, it gives us more flexibility regarding whether to treat them similarly or differently from sources. And in the future, we might want/need to support tables in Rill (conceptually, they’re slightly different than sources and models).

About Postgres, I agree we should consider it a source. We could add a “push down queries” property or something like that on the source to let the user decide whether data should be copied into DuckDB or queried through the foreign data wrapper.

begelundmuller on Oct 31, 2022

@ericpgreen2 that seems sensible to me. In terms of workflow, people generally will be interested in bringing their whole duckdb db along for the ride, since often they have already put in just enough data to do the thing they care about. How do we make it easy for them to jump-start Rill Developer w/ a fresh set of tables?

@begelundmuller has some thinking generally around this. First step is to figure out the right conceptual model & follow it.

hamilton on Oct 31, 2022