sqlfluff: dbt.exceptions.InternalException: Asked to compile node, but it has no compiled form
Search before asking
- I searched the issues and found no similar issues.
What Happened
While linting any dbt model the error is raised part-way through linting (apparently after all the fail lines have been produced and displayed).
I’ve tried sqlfluff versions
0.10.1 - no error 0.11.0, 0.13.2, 1.0.0, 1.1.0 - error thrown
Expected Behaviour
sqlfluff reports linting failures.
Observed Behaviour
SQL model materialising as a view. This is really the WEB_USERS model from the fivetran segment dbt package, just renamed to use our naming standard.
with
SOURCE as (
select
*
from
{{ ref('SG__WEB_USERS__UNION') }}
),
RENAMED as (
select
_DBT_SOURCE_RELATION,
EMAIL,
{{ sv_utils.get_email_domain('EMAIL') }} as EMAIL_DOMAIN,
{{ sv_utils.is_valid_email('EMAIL') }} as IS_EMAIL_VALID,
ID as USER_ID,
PERSON_ID,
USERNAME,
EMAIL_ADDRESS,
{{ sv_utils.get_email_domain('EMAIL_ADDRESS') }} as EMAIL_ADDRESS_DOMAIN,
{{ sv_utils.is_valid_email('EMAIL_ADDRESS') }} as IS_EMAIL_ADDRESS_VALID,
"NAME" as FULL_NAME,
SHORT_NAME,
FIRST_NAME,
INITIALS,
SURNAME,
LAST_NAME,
USER_TYPE,
CONTEXT_IP as IP,
-- CONTEXT_LIBRARY_NAME, -- not useful
SUBSCRIPTIONS,
CONTEXT_PAGE_SEARCH as PAGE_URL_QUERY,
CONTEXT_PAGE_URL as PAGE_URL,
{{ dbt_utils.get_url_host('CONTEXT_PAGE_URL') }} as PAGE_URL_HOST,
-- CONTEXT_LIBRARY_VERSION, -- not useful
CONTEXT_PAGE_REFERRER as PAGE_REFERRER,
replace(
{{ dbt_utils.get_url_host('CONTEXT_PAGE_REFERRER') }},
'www.',
''
) as PAGE_REFERRER_HOST,
CONTEXT_PAGE_TITLE as PAGE_TITLE,
CONTEXT_LOCALE as LOCALE,
CONTEXT_PAGE_PATH as PAGE_PATH,
CONTEXT_CAMPAIGN_SOURCE as UTM_SOURCE,
CONTEXT_CAMPAIGN_MEDIUM as UTM_MEDIUM,
CONTEXT_CAMPAIGN_NAME as UTM_CAMPAIGN,
BISLR_ID,
CONTEXT_USER_AGENT as USER_AGENT,
case
when lower(context_user_agent) like '%android%' then 'Android'
else replace(
{{ dbt_utils.split_part(
dbt_utils.split_part('CONTEXT_USER_AGENT',"'('",2),"' '",1)
}},
';','')
end as device,
UUID_TS,
RECEIVED_AT as RECEIVED_AT_DATE_TIME,
"ROLE" as COMPANY_ROLE,
AGENCY_ID,
ORGANISATION_ID,
COMPANY_ID,
COMPANY,
AGENCY_NAME,
ORGANISATION_NAME,
COMPANY_NAME,
ORGANISATION_TRADING_STYLES,
COMPANY_FINANCIAL_FORCE_ID,
COMPANY_CURRENCIES,
COMPANY_RISK_PROFILES,
AGENCIES,
HAS_LTD_TERMS,
AGENCY_ACCOUNT_STATUS,
PROPOSAL_STATE,
HAS_LABS_ACCESS,
HAS_PAYE_TERMS,
AGENCY_PAYROLL_PROVIDER,
AML_STATUS,
AGENCY_SIGN_UP_STATE,
TIME_SHEETS_AVAILABLE,
PICTURE_URL,
AGENCY_HAS_FULL_ACCESS,
ONLINE_TIMESHEET_PROVIDER,
ROLES,
ROLES_IS_SUPER_USER,
ROLES_IS_CLIENT_AUTHORISER,
ROLES_IS_SONOVATE_ADMIN,
ROLES_IS_CANDIDATE,
PERMISSIONS as USERS_PERMISSIONS,
PERMISSIONS_IS_CLIENT_AUTHORISER,
PERMISSIONS_IS_SONOVATE_ADMIN,
PERMISSIONS_IS_CANDIDATE,
"PROFILE" as USERS_PROFILE,
"SESSION_ID" as USERS_SESSION_ID,
AUTHENTICATED,
IMAGE_URL,
AUTHENTICATION_TOKEN,
TOKEN,
PERSON_TOKEN,
LANGUAGE_CODE,
SETTINGS_DATE_FORMAT,
SETTINGS_SUBSCRIPTIONS,
SETTINGS_TIME_ZONE,
SETTINGS_LANGUAGE,
SETTINGS_TIME_FORMAT,
SETTINGS_COUNTRY_CODE,
SETTINGS_WEEK_START
from
SOURCE
),
FINAL as (
select
*,
case
when DEVICE = 'iPhone' then 'iPhone'
when DEVICE = 'Android' then 'Android'
when DEVICE in ('iPad', 'iPod') then 'Tablet'
when DEVICE in ('Windows', 'Macintosh', 'X11')
then 'Desktop'
else 'Uncategorized' end as DEVICE_CATEGORY
from
RENAMED
)
select
*
from
FINAL
Example verbose output
How to reproduce
Just a basic sqlfluff lint operation
sqlfluff lint .\models\base\segment\SG__WEB_USERS.sql
Dialect
snowflake
Version
0.11.0 0.13.2 1.0.1 1.1.0
Configuration
.sqlfluff
# dbt templating does not keep trailing new lines (L009)
[sqlfluff]
verbose = 3
templater = dbt
dialect = snowflake
exclude_rules = L009,L031,L034
ignore_templated_areas = True
sql_file_exts = .sql,.sql.j2,.dml,.ddl,.SQL
[sqlfluff:rules]
tab_space_size = 2
[sqlfluff:rules:L010]
capitalisation_policy = lower
[sqlfluff:rules:L014]
extended_capitalisation_policy = upper
[sqlfluff:rules:L016]
max_line_length = 180
[sqlfluff:rules:L019]
comma_style = leading
[sqlfluff:rules:L030]
extended_capitalisation_policy = lower
[sqlfluff:rules:L040]
capitalisation_policy = lower
[sqlfluff:templater:dbt]
apply_dbt_builtins = True
load_macros_from_path=./dbt_modules/dbt_utils/macros,./macros
[sqlfluff:templater:dbt:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable) %}item{% endmacro %}
dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}
[sqlfluff:templater:jinja:context]
num_things=456
tbl_name=my_table
my_list=['a', 'b', 'c']
MY_LIST=("d", "e", "f")
[sqlfluff:templater:jinja]
apply_dbt_builtins = True
load_macros_from_path=./dbt_modules/dbt_utils/macros
[sqlfluff:templater:jinja:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable) %}item{% endmacro %}
dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}
.sqlfluffignore
target/
dbt_modules/
macros/
Are you willing to work on and submit a PR to address the issue?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 50 (28 by maintainers)
Ok - unfortunately I don’t think git supports installing direct from subdirectories of git repositories (you’ll need a new version of
sqlfluff-templater-dbtnot the rootsqlfluffproject).Can you download the code for that branch and install from the local version.
You can install the templater locally by running:
What you should get instead of getting
dbt.exceptions.InternalExceptiondirectly is that dbt will skip the file and carry on - but will also give you a warning with the details of exactly which file (or files) are causing the issue. This should at least enable you to use the tool - but also know which files are causing issues.@reubster - that’s a very surprising issue. It looks like it’s an issue with importing
mashumarobefore any work is done at all in dbt or in sqlfluff. Given the changes made in that branch, where there are no changes to any of the imports, it seems strange that we would have triggered an import error.One thing I do know is that support for
dbt-snowflakein python 310 isn’t live yet (see supported versions on pypi), and part of that is due to dependencies, so while it’s a bit of a shot in the dark, would you be able to try the same thing but in python 3.7, 3.8 or 3.9? I can see that your testing env is a python 3.10 env. I know 3.10 is coming for that package but it’s still in pre-release.@barrywhart I might need your help here, especially on option 1.
I think we have three possible options here:
nodeis already aCompiledModelNode. This feels like the easiest, but I’m not sure how the tracing monkey patch would function properly in that case._unsafe_process(), so we never re-process a file. That dodges the problem here, but potentially leaves us open to issues if we ever get aCompiledModelNodeon the first attempt.I’m going to make a PR to better understand option 3.
hi @alanmcruickshank
That worked a treat. I’ve downloaded main and installed. I need to install both sqlfluff and the templater. Re-running the lint I now get.
The changes I’ve suggested on that branch above are actually now merged into the main branch (we decided they’re actually broadly useful and applicable beyond just debugging). I’ll see if we can cut a new release soon which we can then use for testing without needing to install from source.
Barry, from my point of view… I don’t get the error in 0.10.1, and I do get the error in 0.11.0 - which is the release that includes the ephemeral fix.
Mine too.
I have now;
I get the same error. I get this error for any model.
Things I note from the trace;
Stuck figuring out what to try next. Is it viable for me to download the source and run it in a debug mode?