sqlalchemy: half-supported ENUM type in DuckDB raises unrelated error message from re.sub

Describe the bug

The current version of DuckDB can create tables with an ENUM type which is not yet listed in its types table, so _get_columns_info returns NULL/None for the format_type, which is passed into re.sub which causes it to raise an exception that is not very relevant to the error.

Ideally it would raise an exception saying that the type is unsupported.

To Reproduce

import ibis                                                                                                                           
    con = ibis.duckdb.connect()                                                                                                           
    sql = [                                                                                                                               
        "CREATE TYPE enum_t AS ENUM('a', 'b');",                                                                                          
        "CREATE TABLE tmp (enum_col enum_t);",                                                                                            
        "INSERT INTO tmp VALUES ('b');",                                                                                                  
        "INSERT INTO tmp VALUES ('a');",                                                                                                  
    ]                                                                                                                                     
    for s in sql:                                                                                                                         
        con.raw_sql(s)                                                                                                                    
                                                                                                                                          
    t = con.table('tmp')

Error

  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(338)from_call()
-> result: Optional[TResult] = func()
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(259)<lambda>()
-> lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_hooks.py(265)__call__()
-> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_manager.py(80)_hookexec()
-> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(60)_multicall()
-> return outcome.get_result()
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_result.py(60)get_result()
-> raise ex[1].with_traceback(ex[2])
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(39)_multicall()
-> res = hook_impl.function(*args)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(174)pytest_runtest_call()
-> raise e
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/runner.py(166)pytest_runtest_call()
-> item.runtest()
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/python.py(1761)runtest()
-> self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_hooks.py(265)__call__()
-> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_manager.py(80)_hookexec()
-> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(60)_multicall()
-> return outcome.get_result()
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_result.py(60)get_result()
-> raise ex[1].with_traceback(ex[2])
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/pluggy/_callers.py(39)_multicall()
-> res = hook_impl.function(*args)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/_pytest/python.py(192)pytest_pyfunc_call()
-> result = testfunction(**testargs)
  /home/saul/git/ibis/ibis/backends/duckdb/tests/test_datatypes.py(152)test_enum_type()
-> t = con.table('tmp')
  /home/saul/git/ibis/ibis/backends/base/sql/alchemy/__init__.py(422)table()
-> sqla_table = self._get_sqla_table(
  /home/saul/git/ibis/ibis/backends/duckdb/__init__.py(315)_get_sqla_table()
-> table = super()._get_sqla_table(name, schema, **kwargs)
  /home/saul/git/ibis/ibis/backends/base/sql/alchemy/__init__.py(377)_get_sqla_table()
-> return sa.Table(name, self.meta, schema=schema, autoload=autoload)
  <string>(2)__new__()
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py(309)warned()
-> return fn(*args, **kwargs)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(616)__new__()
-> with util.safe_reraise():
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py(70)__exit__()
-> compat.raise_(
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/util/compat.py(208)raise_()
-> raise exception
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(612)__new__()
-> table._init(name, metadata, *args, **kw)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(687)_init()
-> self._autoload(
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/sql/schema.py(722)_autoload()
-> conn_insp.reflect_table(
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py(774)reflect_table()
-> for col_d in self.get_columns(
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py(497)get_columns()
-> col_defs = self.dialect.get_columns(
  <string>(2)get_columns()
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py(55)cache()
-> ret = fn(self, con, *args, **kw)
> /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/base.py(3879)get_columns()
-> column_info = self._get_column_info(
  /home/saul/mambaforge/envs/ibis/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/base.py(3916)_get_column_info()
-> attype = re.sub(r"\(.*\)", "", format_type)
  /home/saul/mambaforge/envs/ibis/lib/python3.10/re.py(209)sub()
-> return _compile(pattern, flags).sub(repl, string, count)

Versions

  • OS: Linux/Ubuntu 22.04
  • Python: 3.10.6
  • SQLAlchemy: 1.4.41
  • Database: DuckDB
  • DBAPI (eg: psycopg, cx_oracle, mysqlclient): duckdb

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (9 by maintainers)

Commits related to this issue

Most upvoted comments

I agree. Since there are a multitude of databases that use pg-like semantic, the postgresql dialect should try to be defensive if it’s not overly inconvinient

OK see that’s pretty easy, we can do that

try it on your end, see if it works, then show me the patch and I can write a unit test on this end