salt: Unable to handle symlinks in GitFS repositories

Salt:

               Salt: 2014.7.1
             Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
             Jinja2: 2.7.2
           M2Crypto: 0.21.1
     msgpack-python: 0.3.0
       msgpack-pure: Not Installed
           pycrypto: 2.6.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.0.1
               RAET: Not Installed
                ZMQ: 4.0.4
               Mako: 0.9.1

OS: Ubuntu 14.04.2 LTS (amd64)

GitFS backend: python-git-0.3.2~RC1-3

Using symlinks seems to break Salt in several places.

What I’m trying to do:

Setup a GitFS repository salt-formulas which contains multiple git subtree clones (originally wanted to use submodules, see also #13664) of public formula repos. The reasons for not simply adding the repositories of the public formulas to the SaltMaster’s fileserver config are:

• adding/removing a salt-formula would require changes of the master config and therefore a restart of the master
• Untested code from public repositories would be used directly for our critical infrastructure. By proxy-ing them through our own repo as subtree modules we can control which upstream version is used and test it in a separate branch/environment before.

This way, I wanted to have a directory structure like this:

├── samba -> samba-formula/samba
└── samba-formula
    ├── LICENSE
    ├── pillar.example
    ├── README.rst
    └── samba
        ├── client.sls
        ├── config.sls
        ├── files
        │   └── smb.conf
        ├── init.sls
        └── map.jinja

As the formulas internally often use absolute references such as include: samba.config, having samba-formula in the root doesn’t work. Besides that, being able to include the formula in own states by simply including samba instead of samba-formula/samba would also be a plus. So either the content of samba-formula needs to be moved to the repository’s root (which would then break the repository and make further updates impossible) or a symlink samba pointing to samba-formula/samba needs to be created.

This would have made it possible to have a repository of repositories which could be comfortably managed and kept in sync with their upstreams.

Using a symlink to not break the repositories leads to several issues:

Jinja: Jinja is unable to render anything as it can’t find the file referenced through a symlink:

    Rendering SLS 'dev-feature-linux_domain_membership :samba-formula/samba.config' failed: Jinja error: samba/map.jinja
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 286, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 132, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: samba/map.jinja

; line 1

---
{% from "samba/map.jinja" import samba with context %}    <======================

include:
  - samba

samba_config:
[...]

SLS: Referencing those formulas in states fails like this:

include:
    - samba

… fails like this:

    Specified SLS ntp in saltenv dev-feature-linux_domain_membership is not available on the salt master

So it seems like at some point symlinks from GitFS sources aren’t handled properly at all and end up invisible/unusable for consumers such as the renderer or state module.