sqlfluff: Unable to lint Ephemeral model - Errors with "dbt.exceptions.InternalException: Asked to compile ... node, but it has no compiled form"

Expected Behaviour

If I try to run sqlfluff lint . to lint the entire dbt project, I expect to get the linted output with the style violations and line numbers. I do not expect Python related stacktrace errors.

Observed Behaviour

If I try to run sqlfluff lint . on the entire dbt project, I get errors for some models (34 models error like this out of 663 total).

WARNING    Unable to lint models/analytics/entitlements/sl_web_user_entitlement_base.sql due to an internal error. Please report this as an issue with your query's contents and stacktrace below!
To hide this warning, add the failing file to .sqlfluffignore
Traceback (most recent call last):
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/linter.py", line 1260, in lint_path
    target_file.read(), fname=fname, fix=fix, config=config
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/linter.py", line 1052, in lint_string
    parsed = self.parse_string(in_str=in_str, fname=fname, config=config)
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/linter.py", line 794, in parse_string
    in_str=in_str, fname=fname, config=config
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/templaters/dbt.py", line 188, in process
    return self._unsafe_process(fname, in_str, config)
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/templaters/dbt.py", line 234, in _unsafe_process
    manifest=self.dbt_manifest,
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/dbt/compilation.py", line 502, in compile_node
    node = self._compile_node(node, manifest, extra_context)
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/dbt/compilation.py", line 398, in _compile_node
    compiled_node = _compiled_type_for(node).from_dict(data)
  File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/dbt/compilation.py", line 40, in _compiled_type_for
    f'Asked to compile {type(model)} node, but it has no compiled form'
dbt.exceptions.InternalException: Asked to compile <class 'dbt.contracts.graph.compiled.CompiledModelNode'> node, but it has no compiled form

I would not expect these errors to occur.

The folder structure is:

surfline-dbt-data-models
|── analysis
|── data
|── dbt_modules
|── logs
|── macros
|── models
│   |── analytics
│   │   |── more_analytics_folders
│   │   └── entitlements
│   │       |── schema.yml
│   │       └── sl_web_user_entitlement_base.sql
│   └── more_models_folders
|── snapshot
|── target
|── tests
|── .envrc
|── .gitignore
|── .sqlfluff
|── .sqlfluffignore
|── dbt_project.yml
|── environment.yml
|── packages.yml
└── README.md

To add to the confusion, if I run…

▶ sqlfluff lint models/analytics/entitlements
== [models/analytics/entitlements/sl_web_user_entitlement_base.sql] FAIL
L:  10 | P:  78 | L001 | Unnecessary trailing whitespace.
L:  11 | P:  75 | L001 | Unnecessary trailing whitespace.
L:  21 | P:  87 | L008 | Commas should be followed by a single whitespace unless
                       | followed by a comment.
L:  21 | P: 118 | L016 | Line is too long
L:  22 | P:   1 | L001 | Unnecessary trailing whitespace.
L:  29 | P:  15 | L009 | Files must end with a trailing newline.

or

▶ sqlfluff lint models/analytics/entitlements/sl_web_user_entitlement_base.sql
== [models/analytics/entitlements/sl_web_user_entitlement_base.sql] FAIL
L:  10 | P:  78 | L001 | Unnecessary trailing whitespace.
L:  11 | P:  75 | L001 | Unnecessary trailing whitespace.
L:  21 | P:  87 | L008 | Commas should be followed by a single whitespace unless
                       | followed by a comment.
L:  21 | P: 118 | L016 | Line is too long
L:  22 | P:   1 | L001 | Unnecessary trailing whitespace.
L:  29 | P:  15 | L009 | Files must end with a trailing newline.

…it lints fine.

Steps to Reproduce

See above.

Version

Include the output of sqlfluff --version along with your Python version

▶ sqlfluff --version
sqlfluff, version 0.4.1

I have python, dbt, and sqlfluff installed via a conda virtual environment (and virtual env is activated in VScode). The conda env is defined as:

name: surfline-dbt-dev
channels:
  - conda-forge
  - defaults
dependencies:
  - pip=20.1.1
  - python=3.7.3
  - pip:
    - dbt==0.19.0
    - sqlfluff[dbt]

Configuration

# For SQLFluff Rules reference, see:
# https://docs.sqlfluff.com/en/stable/rules.html#rules-reference
[sqlfluff]
verbose = 0
nocolor = False
dialect = postgres
templater = dbt
rules = None
exclude_rules = None
recurse = 0
output_line_length = 80
runaway_limit = 10
ignore = parsing
ignore_templated_areas = True

[sqlfluff:indentation]
indented_joins = False
template_blocks_indent = True

# Some rules can be configured directly from the config common to other rules.
[sqlfluff:rules]
tab_space_size = 4
max_line_length = 80
indent_unit = space
comma_style = trailing
allow_scalar = True
single_table_references = consistent
only_aliases = True

# Some rules have their own specific config.
# All SQLFluff rules can be found at: https://docs.sqlfluff.com/en/stable/rules.html#rules-reference
# When a rule in not listed below, we inherit the default behavior from above.
[sqlfluff:rules:L003]
lint_templated_tokens = True

[sqlfluff:rules:L010]  # Keywords
capitalisation_policy = lower

[sqlfluff:rules:L014]  # Unquoted identifiers
capitalisation_policy = lower

[sqlfluff:rules:L016]
# Setting to True allows us to copy/paste long URLs as comments
ignore_comment_lines = True

[sqlfluff:rules:L030]  # Function names
capitalisation_policy = lower

[sqlfluff:rules:L038]
select_clause_trailing_comma = forbid

[sqlfluff:rules:L040]  # Null & Boolean Literals
capitalisation_policy = lower

[sqlfluff:rules:L042]
# By default, allow subqueries in from clauses, but not join clauses.
forbid_subquery_in = join

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 20 (13 by maintainers)

Most upvoted comments

Same issue. Is there a way to tell sqlfluff to skip ephemeral tables?

I am seeing very similar errors, and noticed something that might help shed some light on the error: for me, the errors occur only for dbt models for which the materialization is set to ephemeral. It does not matter if I apply that materialization inline through {{ config(materialized = 'ephemeral') }} or if I make it part of the dbt_project.yml: anytime materialization is set to ephemeral, I get the stacktrace attached below for each ephemeral file.

The “Asked to compile node but it has no compiled form” piece of the error makes me think that there might be differences in how dbt treats ephemeral vs. actually-materialized models, whereas sqlfluff might expect every .sql file to also have a compiled form?

WARNING    Unable to lint <models> due to an internal error. Please report this as an issue with your query's contents and stacktrace below!
To hide this warning, add the failing file to .sqlfluffignore
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sqlfluff/core/linter.py", line 1259, in lint_path
    self.lint_string(
  File "/usr/local/lib/python3.8/site-packages/sqlfluff/core/linter.py", line 1052, in lint_string
    parsed = self.parse_string(in_str=in_str, fname=fname, config=config)
  File "/usr/local/lib/python3.8/site-packages/sqlfluff/core/linter.py", line 793, in parse_string
    templated_file, templater_violations = self.templater.process(
  File "/usr/local/lib/python3.8/site-packages/sqlfluff/core/templaters/dbt.py", line 188, in process
    return self._unsafe_process(fname, in_str, config)
  File "/usr/local/lib/python3.8/site-packages/sqlfluff/core/templaters/dbt.py", line 232, in _unsafe_process
    node = self.dbt_compiler.compile_node(
  File "/usr/local/lib/python3.8/site-packages/dbt/compilation.py", line 479, in compile_node
    node = self._compile_node(node, manifest, extra_context)
  File "/usr/local/lib/python3.8/site-packages/dbt/compilation.py", line 379, in _compile_node
    compiled_node = _compiled_type_for(node).from_dict(data)
  File "/usr/local/lib/python3.8/site-packages/dbt/compilation.py", line 39, in _compiled_type_for
    raise InternalException(
dbt.exceptions.InternalException: Asked to compile <class 'dbt.contracts.graph.compiled.CompiledModelNode'> node, but it has no compiled form

I think this issue is still unfixed. When running sqlfluff lint <path-to-model-1.sql> <path-to-model-2.sql> and one of the models is depending on an ephemeral model the same exception is triggered.

When both models are ephemeral there is no issue, the issue only happens when 1) more than one .sql model is passed and one or more of them depend on other ephemeral models.

I tested this in 0.6.8.

Thanks!

Hi @barrywhart .

I have recreated the issue as described on 16th Jul 2021 by @johansjob . Code and some further explanation below.

dbt_project.yml . This is the one generated by dbt init, but I took out the model definitions, so all models in these examples will be materialized as view unless specified in the model files.

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'my_new_project'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'default'

# These configurations specify where dbt should look for different types of files.
# The `source-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
    - "target"
    - "dbt_modules"

.sqlfluff . I don’t believe any of this is relevant to the issue, except templater = dbt, of course. But I put some rules so we can see them passing/failing.

[sqlfluff]
dialect = snowflake
templater = dbt
# dbt templating does not keep trailing new lines (L009)
exclude_rules = L009

[sqlfluff:rules:L010]
capitalisation_policy = upper

[sqlfluff:rules:L014]
capitalisation_policy = lower

dbt model files:

These are in two folders:

  • error folder demonstrates the erroneous behaviour. Use sqlfluff lint models/error
  • ok folder demonstrates things working as expected. Use sqlfluff lint models/ok

models/error/b.sql

select * 
from {{ ref('c') }}
where id = 1

models/error/c.sql

{{ config(materialized='ephemeral') }}

select 1 as id

models/ok/x.sql

{{ config(materialized='ephemeral') }}

select 1 as id

models/ok/y.sql

select * 
from {{ ref('x') }}
where id = 1

I’m probably just repeating what @johansjob reported, but here’s the conditions that cause the error:

  1. A model is ephemeral AND
  2. Another model depends on it AND
  3. The dependent model’s name comes BEFORE the ephemeral model`s name in the sorted order.

Hence: b.sql depends on a.sql causes error. y.sql depends on x.sql works ok. sqlfluff lint models/error/c.sql works. I assume because the dependent b.sql is out of scope.

I also confirmed @johansjob finding folders are irrelevant to this issue. I could put b.sql and c.sql in separate folders and still get the error. Makes sense, given that dbt’s DAG does not consider folders, only file names.

I hope this helps and happy to provide more input.

@GClunies thanks for checking 😃 I updated the issue title