salt: Modules synchronization fails if git backend is enabled with python3 and gitpython

I’m testing salt with python 3 and I have a problem with our custom modules. It was working without any problem with python 2.7. Is there required any changes in config file or is it a bug in salt?

Modules are called e.g. in jinja templates via salt['x.y'](args) and salt returns error: Unable to manage file: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'x.y'

Sync failed too: Run:

salt-run saltutil.sync_all

Output:

[ERROR   ] Failed to sync clouds module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync modules module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync states module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync grains module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync renderers module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync returners module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync output module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync proxy module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync runners module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync wheel module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync engines module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync thorium module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync queues module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync pillar module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync utils module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync sdb module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync cache module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync fileserver module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync tops module: string argument expected, got 'bytes'
[ERROR   ] Failed to sync tokens module: string argument expected, got 'bytes'
...

Salt master error log:
[ERROR   ] Error in function _file_list:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/master.py", line 1776, in run_func
    ret = getattr(self, func)(load)
  File "/usr/lib/python3/dist-packages/salt/utils/decorators/__init__.py", line 597, in wrapped
    return function(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 752, in file_list
    ret.update(self.servers[fstr](load))
  File "/usr/lib/python3/dist-packages/salt/fileserver/gitfs.py", line 185, in file_list
    return _gitfs().file_list(load)
  File "/usr/lib/python3/dist-packages/salt/utils/gitfs.py", line 2909, in file_list
    return self._file_lists(load, 'files')
  File "/usr/lib/python3/dist-packages/salt/utils/gitfs.py", line 2886, in _file_lists
    repo_files, repo_symlinks = repo.file_list(load['saltenv'])
  File "/usr/lib/python3/dist-packages/salt/utils/gitfs.py", line 1286, in file_list
    file_blob.stream_data(stream)
  File "/usr/lib/python3/dist-packages/git/objects/base.py", line 119, in stream_data
    stream_copy(istream, ostream)
  File "/usr/lib/python3/dist-packages/git/util.py", line 127, in stream_copy
    destination.write(chunk)

Setup

salt master config for modules:

extension_modules: /srv/modules/

/srv/modules and /srv/modules/modules are symlinks to dir with our modules (*.py files)

Versions Report

Salt minion is on the same server as salt master and in the same version:

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Jan 19 2017, 14:11:04)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.4
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-6-amd64
         system: Linux
        version: debian 9.4

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 34 (9 by maintainers)

Most upvoted comments

I recently continued my Salt Master project I left of in January 2019. Debian 9 amd64 , deb http://repo.saltstack.com/py3/debian/9/amd64/2018.3 stretch main, python3-git installed,

gitfs_provider: gitpython
fileserver_backend:
  - gitfs
  - roots

I did some changes to my public GitHub repo, salt-run fileserver.update and it fails, which it didn’t the last time I was working on the project.

root@salt:~# salt-run fileserver.update
Exception occurred in runner fileserver.update: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/client/mixins.py", line 387, in _low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/runners/fileserver.py", line 352, in update
    fileserver.update(back=backend)
  File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 500, in update
    self.servers[fstr]()
  File "/usr/lib/python3/dist-packages/salt/fileserver/gitfs.py", line 133, in update
    _gitfs().update(remotes)
  File "/usr/lib/python3/dist-packages/salt/utils/gitfs.py", line 2421, in update
    self.find_file
  File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 277, in reap_fileserver_cache_dir
    ret = find_func(filename, saltenv=saltenv)
  File "/usr/lib/python3/dist-packages/salt/utils/gitfs.py", line 2764, in find_file
    blob, blob_hexsha, blob_mode = repo.find_file(repo_path, tgt_env)
  File "/usr/lib/python3/dist-packages/salt/utils/gitfs.py", line 1341, in find_file
    file_blob.stream_data(stream)
  File "/usr/lib/python3/dist-packages/git/objects/base.py", line 119, in stream_data
    stream_copy(istream, ostream)
  File "/usr/lib/python3/dist-packages/git/util.py", line 127, in stream_copy
    destination.write(chunk)
TypeError: string argument expected, got 'bytes'

What must I do, to continue my Salt Master project with GitHub repos? I especially rely on the offical formulas.

root@salt:~# salt --versions-report
Salt Version:
           Salt: 2018.3.4
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: 0.24.5
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.2
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.9
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-9-amd64
         system: Linux
        version: debian 9.9