dbt-core: [CT-2672] [Bug] Cannot import name 'Unpack' from 'typing_extensions' (dbt-core 1.5.1)

I can file a ticket; I work on the adapter specifically, so I don’t have a lot of insight into the DBR.

ImportError: cannot import name 'Unpack' from 'typing_extensions' (/databricks/python/lib/python3.9/site-packages/typing_extensions.py)

Oh, I’ve come across something like this after typing-extensions upgrade and made a fix in this commit:

• https://github.com/Fatal1ty/mashumaro/commit/a5d95b8fa9123907bbc4263c35071b394ba4f0f1

It will be included in the upcoming release this or next week.

Hi, any updates on this issue? We are running into the same thing. We are implementing Programmatic Invocation here as well and have big plans for it! To isolate this problem I use this small bit of Python code and simply run it from the Databricks Workspace on a cluster that only has dbt-databricks 1.6.6 installed:

print("Starting")

from dbt.cli.main import dbtRunner, dbtRunnerResult

dbt = dbtRunner()
cli_args = ['--version']
res: dbtRunnerResult = dbt.invoke(cli_args)

print("The End")

I have tested it with different LTS Databricks runtime versions (12.2 and 13.3), different Access modes (Shared and Single) and with or without installing typing_extenstions 4.7.1. All using dbt-databricks 1.6.6. These are the results:

My complete stack trace when I get the error looks like this:

ImportError: cannot import name 'override' from 'typing_extensions' (/databricks/python/lib/python3.10/site-packages/typing_extensions.py)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
File <command-824890481590461>, line 3
      1 print("Starting")
----> 3 from dbt.cli.main import dbtRunner, dbtRunnerResult
      5 dbt = dbtRunner()
      6 cli_args = ['--version']

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/__init__.py:1
----> 1 from .main import cli as dbt_cli  # noqa

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/main.py:14
      6 import click
      7 from click.exceptions import (
      8     Exit as ClickExit,
      9     BadOptionUsage,
     10     NoSuchOption,
     11     UsageError,
     12 )
---> 14 from dbt.cli import requires, params as p
     15 from dbt.cli.exceptions import (
     16     DbtInternalException,
     17     DbtUsageException,
     18 )
     19 from dbt.contracts.graph.manifest import Manifest

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py:3
      1 import dbt.tracking
      2 from dbt.version import installed as installed_version
----> 3 from dbt.adapters.factory import adapter_management, register_adapter
      4 from dbt.flags import set_flags, get_flag_dict
      5 from dbt.cli.exceptions import (
      6     ExceptionExit,
      7     ResultExit,
      8 )

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/adapters/factory.py:8
      5 from pathlib import Path
      6 from typing import Any, Dict, List, Optional, Set, Type
----> 8 from dbt.adapters.base.plugin import AdapterPlugin
      9 from dbt.adapters.protocol import AdapterConfig, AdapterProtocol, RelationProtocol
     10 from dbt.contracts.connection import AdapterRequiredConfig, Credentials

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/adapters/base/__init__.py:6
      4 from dbt.contracts.connection import Credentials  # noqa: F401
      5 from dbt.adapters.base.meta import available  # noqa: F401
----> 6 from dbt.adapters.base.connections import BaseConnectionManager  # noqa: F401
      7 from dbt.adapters.base.relation import (  # noqa: F401
      8     BaseRelation,
      9     RelationType,
     10     SchemaSearchMap,
     11 )
     12 from dbt.adapters.base.column import Column  # noqa: F401

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/adapters/base/connections.py:35
     26 import dbt.exceptions
     27 from dbt.contracts.connection import (
     28     Connection,
     29     Identifier,
   (...)
     33     AdapterResponse,
     34 )
---> 35 from dbt.contracts.graph.manifest import Manifest
     36 from dbt.adapters.base.query_headers import (
     37     MacroQueryStringSetter,
     38 )
     39 from dbt.events import AdapterLogger

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/contracts/graph/manifest.py:27
     24 from typing_extensions import Protocol
     25 from uuid import UUID
---> 27 from dbt.contracts.graph.nodes import (
     28     BaseNode,
     29     Documentation,
     30     Exposure,
     31     GenericTestNode,
     32     GraphMemberNode,
     33     Group,
     34     Macro,
     35     ManifestNode,
     36     Metric,
     37     ModelNode,
     38     DeferRelation,
     39     ResultNode,
     40     SemanticModel,
     41     SourceDefinition,
     42     UnpatchedSourceDefinition,
     43 )
     44 from dbt.contracts.graph.unparsed import SourcePatch, NodeVersion, UnparsedVersion
     45 from dbt.contracts.graph.manifest_upgrade import upgrade_manifest_json

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/contracts/graph/nodes.py:61
     59 from dbt_semantic_interfaces.references import MetricReference as DSIMetricReference
     60 from dbt_semantic_interfaces.type_enums import MetricType, TimeGranularity
---> 61 from dbt_semantic_interfaces.parsing.where_filter_parser import WhereFilterParser
     63 from .model_config import (
     64     NodeConfig,
     65     SeedConfig,
   (...)
     72     SemanticModelConfig,
     73 )
     76 # =====================================================================
     77 # This contains the classes for all of the nodes and node-like objects
     78 # in the manifest. In the "nodes" dictionary of the manifest we find
   (...)
     95 # Various parent classes and node attribute classes
     96 # ==================================================

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_semantic_interfaces/parsing/where_filter_parser.py:14
      7 from dbt_semantic_interfaces.call_parameter_sets import (
      8     FilterCallParameterSets,
      9     ParseWhereFilterException,
     10 )
     11 from dbt_semantic_interfaces.parsing.where_filter.parameter_set_factory import (
     12     ParameterSetFactory,
     13 )
---> 14 from dbt_semantic_interfaces.parsing.where_filter.where_filter_dimension import (
     15     WhereFilterDimensionFactory,
     16 )
     17 from dbt_semantic_interfaces.parsing.where_filter.where_filter_entity import (
     18     WhereFilterEntityFactory,
     19 )
     20 from dbt_semantic_interfaces.parsing.where_filter.where_filter_time_dimension import (
     21     WhereFilterTimeDimensionFactory,
     22 )

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_semantic_interfaces/parsing/where_filter/where_filter_dimension.py:5
      1 from __future__ import annotations
      3 from typing import List, Optional, Sequence
----> 5 from typing_extensions import override
      7 from dbt_semantic_interfaces.errors import InvalidQuerySyntax
      8 from dbt_semantic_interfaces.protocols.protocol_hint import ProtocolHint

On dbt-core==1.6.1 and dbt-core==1.6.2 I get something similar (on databricks): ImportError: cannot import name 'override' from 'typing_extensions' (/databricks/python/lib/python3.10/site-packages/typing_extensions.py)

Fixed by adding typing_extensions==4.7.1 as a dependency

I got the same issue when running on Databricks Runtime 13.3 LTS. But on Databricks Runtime 12.2 LTS it worked out fine. I did not try to fix typing_extensions==4.7.1 as you did but rather tried to update to the newest version of typing_extensions which did not make it work.

This was tried with dbt-core 1.6.2 and dbt-databricks 1.6.4 + 1.6.5.

Here my full stack trace for further investigation for the devs:

Traceback (most recent call last):
  File "/databricks/python3/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/databricks/python/lib/python3.10/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
  File "/databricks/python/lib/python3.10/site-packages/dbt/cli/main.py", line 13, in <module>
    from dbt.cli import requires, params as p
  File "/databricks/python/lib/python3.10/site-packages/dbt/cli/requires.py", line 3, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter
  File "/databricks/python/lib/python3.10/site-packages/dbt/adapters/factory.py", line 8, in <module>
    from dbt.adapters.base.plugin import AdapterPlugin
  File "/databricks/python/lib/python3.10/site-packages/dbt/adapters/base/__init__.py", line 6, in <module>
    from dbt.adapters.base.connections import BaseConnectionManager  # noqa: F401
  File "/databricks/python/lib/python3.10/site-packages/dbt/adapters/base/connections.py", line 35, in <module>
    from dbt.contracts.graph.manifest import Manifest
  File "/databricks/python/lib/python3.10/site-packages/dbt/contracts/graph/manifest.py", line 27, in <module>
    from dbt.contracts.graph.nodes import (
  File "/databricks/python/lib/python3.10/site-packages/dbt/contracts/graph/nodes.py", line 61, in <module>
    from dbt_semantic_interfaces.parsing.where_filter_parser import WhereFilterParser
  File "/databricks/python/lib/python3.10/site-packages/dbt_semantic_interfaces/parsing/where_filter_parser.py", line 14, in <module>
    from dbt_semantic_interfaces.parsing.where_filter.where_filter_dimension import (
  File "/databricks/python/lib/python3.10/site-packages/dbt_semantic_interfaces/parsing/where_filter/where_filter_dimension.py", line 5, in <module>
    from typing_extensions import override
ImportError: cannot import name 'override' from 'typing_extensions' (/databricks/python/lib/python3.10/site-packages/typing_extensions.py)

Hope that helps to fix this issue.

@dbeatty10

I think it’s worth to exclude only those versions that break something in dbt-core. Not taking into account the transition to major version 3, I remember that 3.0.2 and 3.1.0 were problematic, so they are candidates for exclusion. On the other hand if you rely on some functionality that exists only from a version X then you should obviously pin that version as the minimal one and remove all exclusions before X as redundant:

mashumaro[msgpack]>=X

If I were in your place, I would be looking at pull requests from dependabot and either: a) exclude package versions not passing the tests leaving the minimal version X b) bump the minimal version X to the following patch version that passes the tests

The option “a” is better for users because it makes it easier to use dbt-core in combination with other packages that may also depend on mashumaro. But if dbt-core is positioned as the product rather than as a library than option “b” or even strict pin to the specific version of a package is also a good choice.

@thijs-nijhuis-shell sounds good, thanks. If there’s some weird interaction on those warnings that turns out to be something we need to fix on the dbt side please do open a separate issue. Hopefully you can get an update that makes those warnings stop.

@tlento , thanks for fixing it for 1.7.x as well and all the effort you put into this.

Unfortunately, my workaround to suppress the warnings doesn’t work after all. When promoting the warnings to errors the dbt library is not loaded (completely) which makes sense of course. For some reason, when I set the warnings to ‘ignore’ instead of error like the python documentation says the warnings are still shown. I have no idea how to solve or workaround this.

The link you added about 13.3 runtime points to 13.3LTS ML. We use 13.3LTS and that one doesn’t have the wrapt package installed it seems. I also don’t see it when doing the deptreelist.

I also don’t get the warning (or the error that is solved now) when running locally. I am going to create a Issue in the dbt-databricks github for this.

dbt-semantic-interfaces 0.2.3 has been deployed: https://pypi.org/project/dbt-semantic-interfaces/0.2.3/

At this point, doing a pip upgrade (for existing installations) or doing a fresh install of dbt-core 1.6 or a corresponding adapter package should pin the typing-extensions version to something past 4.4.0.

As mentioned earlier, this will NOT fix issues with 1.5 installs, and even on 1.6 any installation that clobbers the typing_extensions version may still cause problems, but at least if you install dbt-core it should now update typing-extensions appropriately.

Thanks to @Fatal1ty for submitting the quick patch on the DSI side and everyone here (and especially @thijs-nijhuis-shell ) for the detailed info on what was broken.

The databricks 13.3 runtime appears happy with typing_extensions >= 4.3, so we will merge the relevant fixes into the dbt-semantic-interfaces package today, with plans to deploy the backport tomorrow.

This won’t affect existing installations of dbt-core or dbt-databricks, as an update to typing-extensions will not automatically trigger, but new installations should update versions accordingly. Worth noting, the databricks runtime install might still clobber the typing-extensions version on pre-installed dbt-databricks packages, so it could be necessary to install/update dbt-databricks (or even typing-extensions) after the runtime has spun up.

I’ll update here when the backport is deployed, as that should address most issues people are encountering with dbt-core 1.6.

Thanks! I will make sure to test it on our end when available.

@QMalcolm and I have done some further digging, and this comment https://github.com/dbt-labs/dbt-core/issues/7828#issuecomment-1780640653 was extremely helpful in our investigation, so big thanks to @thijs-nijhuis-shell

What appears to be happening is Databricks in particular is configuring its initial python environment with typing-extensions set to 4.1/4.3, and a subsequent install of dbt-core 1.6 will not update this because 1.6 is set to require ~=4.0 (by way of dbt-semantic-interfaces).

Since everything works if you install dbt-core (or typing-extensions 4.7.1) first, this suggests databricks is actually perfectly happy with 4.7 or whatever.

However, that databricks 13.3 overwrites the typing-extensions module with 4.3 (while 12.2 leaves that well enough alone) is a bit of a cause for concern. Like, why would it do that?

We’re ready to backport the dbt-semantic-interfaces change to address the issue with 1.6, but we’d like to confirm that Databricks 13.3 is happy with a later version of typing_extensions before we merge and deploy. We’ll update here once we’re done.

@Almaz-Murzabekov I guess this should not be required when you install packages with the %pip magic or the init cluster script.

The root cause of this issue comes from dbt-semantic-interfaces:

• https://github.com/dbt-labs/dbt-semantic-interfaces/issues/193

Hey @thijs-nijhuis-shell, I don’t know your setup in Databricks, but if you’re using code inside dbx notebooks, please also try this command before running any dbt-api call (restarting python process)

# Restart python process to ensure new installed packages are used
dbutils.library.restartPython()

TBH, dunno why, but it’s required if you’re installing deps inside the notebook

@thijs-nijhuis-shell I guess it would be helpful if you could find out for each runtime / mode what version of package typing_extensions is used. e.g. run pip freeze | grep typing_extensions and put the version number in a table as given above. Maybe the python version would help as well (python --version). I guess we should boil that down since here we are in the dbt-core repo, not the Databricks repo.