salt: Can't use tpldir or slspath in template file

Workaround described here: https://github.com/saltstack-formulas/mysql-formula/pull/184#issuecomment-398193878

@max-arnold compile_template_str is designed to take a string, while compile_template is designed to take a path to a template file.

With regard to tpldir, tpldot, and tplfile, these are also SLS-specific. That is, they are different depending on what SLS file is being processed. When compile_template is operating on a template, it is not aware of the URL used to fetch the template, just the contents of the template. So, outside of the context of an SLS file (which has a known relative path), there is no good way for the templating system to know what these values should be. And also, keep in mind that these values are different depending on the path of the SLS file!

When I need to use these variables in a jinja template, I just pass them in the context argument, like so:

/etc/foo.conf:
  file.managed:
    - source: salt://apps/foo/foo.conf.jinja
    - template: jinja
    - context:
        tpldir: {{ tpldir }}

@myii (salt-formulas org maintainer) informed me about a problem with the tpldir, slspath, tplfile, and tpldot template variables when using the new map.jinja troubleshooting module introduced in https://github.com/saltstack/salt/pull/55253.

I’m not 100% familiar with the code (I just backported it to Salt Neon), but on a first glance it looks like the problem is caused by using the salt.template.compile_template_str in modules/jinja.py. To evaluate a map.jinja file it renders the following inline template string that imports the specified map.jinja:

{{% from "{path}" import {value} with context %}}
{{{{ {value} | tojson }}}}

(1) The above snippet gets saved into a tempfile (something like /tmp/__salt.tmp.f1t0s2c_). That is probably the reason why these path variables aren’t defined.

(2) The second part of the problem is that a map.jinja could be imported in different sls files that are located in a formula root or in a subfolder. This produces different values of the above mentioned template variables.

(3) And the final issue is that these variables have different values depending on the presence of with context import clause.

Below is a reproducible test case:

% tree formula

formula
├── init.sls
├── map.jinja
└── subfolder
    └── init.sls

init.sls (both files are the same):

{% do salt.log.warning('INSIDE INIT:') %}
{% do salt.log.warning('VAR tpldir="{}"'.format(tpldir)) %}
{% do salt.log.warning('VAR slspath="{}"'.format(slspath)) %}
{% do salt.log.warning('VAR tplfile="{}"'.format(tplfile)) %}
{% do salt.log.warning('VAR tpldot="{}"'.format(tpldot)) %}

{% do salt.log.warning('WITHOUT CONTEXT:') %}
{% from "formula/map.jinja" import defaults %}

{% do salt.log.warning('WITH CONTEXT:') %}
{% from "formula/map.jinja" import defaults with context %}

map.jinja:

{% do salt.log.warning('INSIDE MAP:') %}
{% do salt.log.warning('VAR tpldir="{}"'.format(tpldir)) %}
{% do salt.log.warning('VAR slspath="{}"'.format(slspath)) %}
{% do salt.log.warning('VAR tplfile="{}"'.format(tplfile)) %}
{% do salt.log.warning('VAR tpldot="{}"'.format(tpldot)) %}

{% set defaults = {} %}

Below is how issues (2) and (3) mainfest themselves:

% sudo salt-call state.apply formula -l warning

[WARNING ] INSIDE INIT:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula"
[WARNING ] VAR tplfile="formula/init.sls"
[WARNING ] VAR tpldot="formula"
[WARNING ] WITHOUT CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula"
[WARNING ] VAR tplfile="formula/map.jinja"
[WARNING ] VAR tpldot="formula"
[WARNING ] WITH CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula"
[WARNING ] VAR tplfile="formula/init.sls"
[WARNING ] VAR tpldot="formula"

% sudo salt-call state.apply formula.subfolder -l warning

[WARNING ] INSIDE INIT:
[WARNING ] VAR tpldir="formula/subfolder"
[WARNING ] VAR slspath="formula/subfolder"
[WARNING ] VAR tplfile="formula/subfolder/init.sls"
[WARNING ] VAR tpldot="formula.subfolder"
[WARNING ] WITHOUT CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula/subfolder"
[WARNING ] VAR tplfile="formula/map.jinja"
[WARNING ] VAR tpldot="formula"
[WARNING ] WITH CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula/subfolder"
[WARNING ] VAR slspath="formula/subfolder"
[WARNING ] VAR tplfile="formula/subfolder/init.sls"
[WARNING ] VAR tpldot="formula.subfolder"

And below is the issue (1) with the new jinja.py module:

% sudo salt-call jinja.load_map formula/map.jinja defaults

[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="."
[WARNING ] VAR slspath=""
[WARNING ] VAR tplfile=""
[WARNING ] VAR tpldot=""

I made an experimental patch to modules/jinja.py that improves the behavior:

diff --git a/salt/modules/jinja.py b/salt/modules/jinja.py
index eb23991c37..f5760b5767 100644
--- a/salt/modules/jinja.py
+++ b/salt/modules/jinja.py
@@ -52,12 +52,16 @@ def load_map(path, value):
         {{% from "{path}" import {value} with context %}}
         {{{{ {value} | tojson }}}}
         '''.format(path=path, value=value))
-    return salt.template.compile_template_str(
-        tmplstr,
+    return salt.template.compile_template(
+        ':string:',
         salt.loader.render(__opts__, __salt__),
         __opts__['renderer'],
         __opts__['renderer_blacklist'],
-        __opts__['renderer_whitelist'])
+        __opts__['renderer_whitelist'],
+        input_data=tmplstr,
+        tmplpath=path,
+        sls=path
+    )

% sudo salt-call jinja.load_map formula/map.jinja defaults

[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula/map/jinja"
[WARNING ] VAR tplfile="formula/map.jinja"
[WARNING ] VAR tpldot="formula"

However, the slspath is obviously incorrect.

One potential workaround is to introduce the tplroot variable (see https://github.com/saltstack/salt/pull/51814, however if you place a formula into a deeply nested dir (e.g., blah/blah/blah/blah/template-formula), the tplroot variable will be just blah and all the import paths inside the formula will be broken.

TLDR: salt-formulas need path-independent ways to reference map.jinja files (and also *.yaml and *.json data files) inside of *.sls files. Also we need a simple way to debug these files in isolation. Plus, all template variables should be documented: https://github.com/saltstack/salt/issues/50925

CC: @terminalmage You wrote the original jinja.py troubleshooting module and have much more experience in Salt renderer subsystem. Maybe you can provide some clues on how to fix this? Also, are there any reasons why you used the salt.template.compile_template_str function instead of salt.template.compile_template?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

Where map.jinja is imported with context, here’s another workaround:

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{#- Start imports as #}
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}