sqlalchemy: half-supported ENUM type in DuckDB raises unrelated error message from re.sub
Describe the bug
The current version of DuckDB can create tables with an ENUM type which is not yet listed in its types table, so _get_columns_info returns NULL/None for the format_type, which is passed into re.sub which causes it to raise an exception that is not very relevant to the error.
Ideally it would raise an exception saying that the type is unsupported.
To Reproduce
import ibis
con = ibis.duckdb.connect()
sql = [
"CREATE TYPE enum_t AS ENUM('a', 'b');",
"CREATE TABLE tmp (enum_col enum_t);",
"INSERT INTO tmp VALUES ('b');",
"INSERT INTO tmp VALUES ('a');",
]
for s in sql:
con.raw_sql(s)
t = con.table('tmp')
Error
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(338)from_call()
-> result: Optional[TResult] = func()
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(259)<lambda>()
-> lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_hooks.py(265)__call__()
-> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_manager.py(80)_hookexec()
-> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(60)_multicall()
-> return outcome.get_result()
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_result.py(60)get_result()
-> raise ex[1].with_traceback(ex[2])
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(39)_multicall()
-> res = hook_impl.function(*args)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(174)pytest_runtest_call()
-> raise e
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(166)pytest_runtest_call()
-> item.runtest()
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/python.py(1761)runtest()
-> self.ihook.pytest_pyfunc_call(pyfuncitem=self)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_hooks.py(265)__call__()
-> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_manager.py(80)_hookexec()
-> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(60)_multicall()
-> return outcome.get_result()
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_result.py(60)get_result()
-> raise ex[1].with_traceback(ex[2])
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(39)_multicall()
-> res = hook_impl.function(*args)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/python.py(192)pytest_pyfunc_call()
-> result = testfunction(**testargs)
/home/saul/git/ibis/ibis/backends/duckdb/tests/test_datatypes.py(152)test_enum_type()
-> t = con.table('tmp')
/home/saul/git/ibis/ibis/backends/base/sql/alchemy/__init__.py(422)table()
-> sqla_table = self._get_sqla_table(
/home/saul/git/ibis/ibis/backends/duckdb/__init__.py(315)_get_sqla_table()
-> table = super()._get_sqla_table(name, schema, **kwargs)
/home/saul/git/ibis/ibis/backends/base/sql/alchemy/__init__.py(377)_get_sqla_table()
-> return sa.Table(name, self.meta, schema=schema, autoload=autoload)
<string>(2)__new__()
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py(309)warned()
-> return fn(*args, **kwargs)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(616)__new__()
-> with util.safe_reraise():
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py(70)__exit__()
-> compat.raise_(
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/util/compat.py(208)raise_()
-> raise exception
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(612)__new__()
-> table._init(name, metadata, *args, **kw)
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(687)_init()
-> self._autoload(
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(722)_autoload()
-> conn_insp.reflect_table(
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py(774)reflect_table()
-> for col_d in self.get_columns(
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py(497)get_columns()
-> col_defs = self.dialect.get_columns(
<string>(2)get_columns()
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py(55)cache()
-> ret = fn(self, con, *args, **kw)
> /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/base.py(3879)get_columns()
-> column_info = self._get_column_info(
/home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/base.py(3916)_get_column_info()
-> attype = re.sub(r"\(.*\)", "", format_type)
/home/saul/mambaforge/envs/ibis/lib/python3.10/re.py(209)sub()
-> return _compile(pattern, flags).sub(repl, string, count)
Versions
- OS: Linux/Ubuntu 22.04
- Python: 3.10.6
- SQLAlchemy: 1.4.41
- Database: DuckDB
- DBAPI (eg: psycopg, cx_oracle, mysqlclient): duckdb
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (9 by maintainers)
I agree. Since there are a multitude of databases that use pg-like semantic, the postgresql dialect should try to be defensive if it’s not overly inconvinient
OK see that’s pretty easy, we can do that
try it on your end, see if it works, then show me the patch and I can write a unit test on this end