sqlfluff: Unable to lint Ephemeral model - Errors with "dbt.exceptions.InternalException: Asked to compile ... node, but it has no compiled form"
Expected Behaviour
If I try to run sqlfluff lint .
to lint the entire dbt
project, I expect to get the linted output with the style violations and line numbers. I do not expect Python related stacktrace errors.
Observed Behaviour
If I try to run sqlfluff lint .
on the entire dbt
project, I get errors for some models (34 models error like this out of 663 total).
WARNING Unable to lint models/analytics/entitlements/sl_web_user_entitlement_base.sql due to an internal error. Please report this as an issue with your query's contents and stacktrace below!
To hide this warning, add the failing file to .sqlfluffignore
Traceback (most recent call last):
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/linter.py", line 1260, in lint_path
target_file.read(), fname=fname, fix=fix, config=config
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/linter.py", line 1052, in lint_string
parsed = self.parse_string(in_str=in_str, fname=fname, config=config)
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/linter.py", line 794, in parse_string
in_str=in_str, fname=fname, config=config
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/templaters/dbt.py", line 188, in process
return self._unsafe_process(fname, in_str, config)
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/sqlfluff/core/templaters/dbt.py", line 234, in _unsafe_process
manifest=self.dbt_manifest,
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/dbt/compilation.py", line 502, in compile_node
node = self._compile_node(node, manifest, extra_context)
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/dbt/compilation.py", line 398, in _compile_node
compiled_node = _compiled_type_for(node).from_dict(data)
File "/opt/anaconda3/envs/surfline-dbt-dev/lib/python3.7/site-packages/dbt/compilation.py", line 40, in _compiled_type_for
f'Asked to compile {type(model)} node, but it has no compiled form'
dbt.exceptions.InternalException: Asked to compile <class 'dbt.contracts.graph.compiled.CompiledModelNode'> node, but it has no compiled form
I would not expect these errors to occur.
The folder structure is:
surfline-dbt-data-models
|── analysis
|── data
|── dbt_modules
|── logs
|── macros
|── models
│ |── analytics
│ │ |── more_analytics_folders
│ │ └── entitlements
│ │ |── schema.yml
│ │ └── sl_web_user_entitlement_base.sql
│ └── more_models_folders
|── snapshot
|── target
|── tests
|── .envrc
|── .gitignore
|── .sqlfluff
|── .sqlfluffignore
|── dbt_project.yml
|── environment.yml
|── packages.yml
└── README.md
To add to the confusion, if I run…
▶ sqlfluff lint models/analytics/entitlements
== [models/analytics/entitlements/sl_web_user_entitlement_base.sql] FAIL
L: 10 | P: 78 | L001 | Unnecessary trailing whitespace.
L: 11 | P: 75 | L001 | Unnecessary trailing whitespace.
L: 21 | P: 87 | L008 | Commas should be followed by a single whitespace unless
| followed by a comment.
L: 21 | P: 118 | L016 | Line is too long
L: 22 | P: 1 | L001 | Unnecessary trailing whitespace.
L: 29 | P: 15 | L009 | Files must end with a trailing newline.
or
▶ sqlfluff lint models/analytics/entitlements/sl_web_user_entitlement_base.sql
== [models/analytics/entitlements/sl_web_user_entitlement_base.sql] FAIL
L: 10 | P: 78 | L001 | Unnecessary trailing whitespace.
L: 11 | P: 75 | L001 | Unnecessary trailing whitespace.
L: 21 | P: 87 | L008 | Commas should be followed by a single whitespace unless
| followed by a comment.
L: 21 | P: 118 | L016 | Line is too long
L: 22 | P: 1 | L001 | Unnecessary trailing whitespace.
L: 29 | P: 15 | L009 | Files must end with a trailing newline.
…it lints fine.
Steps to Reproduce
See above.
Version
Include the output of sqlfluff --version
along with your Python version
▶ sqlfluff --version
sqlfluff, version 0.4.1
I have python
, dbt
, and sqlfluff
installed via a conda
virtual environment (and virtual env is activated in VScode). The conda env is defined as:
name: surfline-dbt-dev
channels:
- conda-forge
- defaults
dependencies:
- pip=20.1.1
- python=3.7.3
- pip:
- dbt==0.19.0
- sqlfluff[dbt]
Configuration
# For SQLFluff Rules reference, see:
# https://docs.sqlfluff.com/en/stable/rules.html#rules-reference
[sqlfluff]
verbose = 0
nocolor = False
dialect = postgres
templater = dbt
rules = None
exclude_rules = None
recurse = 0
output_line_length = 80
runaway_limit = 10
ignore = parsing
ignore_templated_areas = True
[sqlfluff:indentation]
indented_joins = False
template_blocks_indent = True
# Some rules can be configured directly from the config common to other rules.
[sqlfluff:rules]
tab_space_size = 4
max_line_length = 80
indent_unit = space
comma_style = trailing
allow_scalar = True
single_table_references = consistent
only_aliases = True
# Some rules have their own specific config.
# All SQLFluff rules can be found at: https://docs.sqlfluff.com/en/stable/rules.html#rules-reference
# When a rule in not listed below, we inherit the default behavior from above.
[sqlfluff:rules:L003]
lint_templated_tokens = True
[sqlfluff:rules:L010] # Keywords
capitalisation_policy = lower
[sqlfluff:rules:L014] # Unquoted identifiers
capitalisation_policy = lower
[sqlfluff:rules:L016]
# Setting to True allows us to copy/paste long URLs as comments
ignore_comment_lines = True
[sqlfluff:rules:L030] # Function names
capitalisation_policy = lower
[sqlfluff:rules:L038]
select_clause_trailing_comma = forbid
[sqlfluff:rules:L040] # Null & Boolean Literals
capitalisation_policy = lower
[sqlfluff:rules:L042]
# By default, allow subqueries in from clauses, but not join clauses.
forbid_subquery_in = join
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 20 (13 by maintainers)
Same issue. Is there a way to tell sqlfluff to skip ephemeral tables?
I am seeing very similar errors, and noticed something that might help shed some light on the error: for me, the errors occur only for dbt models for which the materialization is set to
ephemeral
. It does not matter if I apply that materialization inline through{{ config(materialized = 'ephemeral') }}
or if I make it part of thedbt_project.yml
: anytime materialization is set to ephemeral, I get the stacktrace attached below for each ephemeral file.The “Asked to compile node but it has no compiled form” piece of the error makes me think that there might be differences in how dbt treats ephemeral vs. actually-materialized models, whereas sqlfluff might expect every
.sql
file to also have a compiled form?I think this issue is still unfixed. When running
sqlfluff lint <path-to-model-1.sql> <path-to-model-2.sql>
and one of the models is depending on an ephemeral model the same exception is triggered.When both models are ephemeral there is no issue, the issue only happens when 1) more than one .sql model is passed and one or more of them depend on other ephemeral models.
I tested this in 0.6.8.
Thanks!
Hi @barrywhart .
I have recreated the issue as described on 16th Jul 2021 by @johansjob . Code and some further explanation below.
dbt_project.yml
. This is the one generated bydbt init
, but I took out the model definitions, so all models in these examples will be materialized asview
unless specified in the model files..sqlfluff
. I don’t believe any of this is relevant to the issue, excepttemplater = dbt
, of course. But I put some rules so we can see them passing/failing.dbt model files:
These are in two folders:
error
folder demonstrates the erroneous behaviour. Usesqlfluff lint models/error
ok
folder demonstrates things working as expected. Usesqlfluff lint models/ok
models/error/b.sql
models/error/c.sql
models/ok/x.sql
models/ok/y.sql
I’m probably just repeating what @johansjob reported, but here’s the conditions that cause the error:
ephemeral
ANDephemeral
model`s name in the sorted order.Hence:
b.sql
depends ona.sql
causes error.y.sql
depends onx.sql
works ok.sqlfluff lint models/error/c.sql
works. I assume because the dependentb.sql
is out of scope.I also confirmed @johansjob finding folders are irrelevant to this issue. I could put
b.sql
andc.sql
in separate folders and still get the error. Makes sense, given that dbt’s DAG does not consider folders, only file names.I hope this helps and happy to provide more input.
@GClunies thanks for checking 😃 I updated the issue title