Description of Issue/Question

Trying to push file from minion to master:

File size: 28 bytes

: salt 'minion' cp.push /root/myip.txt

minion: False

All minions (400) have the same issue. Minion versions range from 2015.5.3 up to 2016.3.3. The only minion that does work is the minion running locally on the salt-master.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

The salt master has this setting in /etc/salt/master file_recv: True

Both Salt master and salt minion processes have been restarted many times.

Steps to Reproduce Issue

Sorry for the log file blast, but I didn’t want to leave anything out

Master debug log (log_level: all) 2016-10-20 17:13:04,004 [salt.utils.event ][DEBUG ][6535] Sending event - data = {‘tgt_type’: ‘glob’, ‘jid’: ‘20161020171304002177’, ‘tgt’: ‘MINION_ID’, ‘_stamp’: ‘2016-10-20T22:13:04.003420’, ‘user’: ‘sudo_USER’, ‘arg’: [‘/root/myip.txt’], ‘fun’: ‘cp.push’, ‘minions’: [‘MINION_ID’]} 2016-10-20 17:13:04,005 [salt.utils.event ][TRACE ][6507] get_event() received = {‘tag’: ‘20161020171304002177’, ‘data’: {‘_stamp’: ‘2016-10-20T22:13:04.002262’, ‘minions’: [‘MINION_ID’]}} 2016-10-20 17:13:04,005 [salt.returners ][INFO ][6535] Using default for mysql ssl_key 2016-10-20 17:13:04,005 [salt.utils.reactor ][DEBUG ][6507] Gathering reactors for tag 20161020171304002177 2016-10-20 17:13:04,006 [salt.returners ][INFO ][6535] Using default for mysql ssl_cert 2016-10-20 17:13:04,006 [salt.returners ][INFO ][6535] Using default for mysql port 2016-10-20 17:13:04,006 [salt.utils.event ][TRACE ][6507] get_event() received = {‘tag’: ‘salt/job/20161020171304002177/new’, ‘data’: {‘tgt_type’: ‘glob’, ‘jid’: ‘20161020171304002177’, ‘tgt’: ‘MINION_ID’, ‘_stamp’: ‘2016-10-20T22:13:04.003420’, ‘user’: ‘sudo_USER’, ‘arg’: [‘/root/myip.txt’], ‘fun’: ‘cp.push’, ‘mini ons’: [‘MINION_ID’]}} 2016-10-20 17:13:04,007 [salt.returners ][INFO ][6535] Using default for mysql ssl_ca 2016-10-20 17:13:04,007 [salt.utils.reactor ][DEBUG ][6507] Gathering reactors for tag salt/job/20161020171304002177/new 2016-10-20 17:13:04,007 [salt.loaded.int.returner.mysql ][DEBUG ][6535] Trying to reuse MySQL connection pool 2016-10-20 17:13:04,011 [salt.master ][INFO ][6535] User sudo_USER Published command cp.push with jid 20161020171304002177 2016-10-20 17:13:04,011 [salt.master ][DEBUG ][6535] Published command details {‘tgt_type’: ‘glob’, ‘jid’: ‘20161020171304002177’, ‘tgt’: ‘MINION_ID’, ‘ret’: ‘’, ‘user’: ‘sudo_USER’, ‘arg’: [‘/root/myip.txt’], ‘fun’: ‘cp.push’} 2016-10-20 17:13:04,017 [salt.client ][TRACE ][24378] func get_cli_event_returns() 2016-10-20 17:13:04,023 [salt.utils.lazy ][DEBUG ][24378] LazyLoaded mysql.get_load 2016-10-20 17:13:04,028 [salt.utils.lazy ][DEBUG ][24378] LazyLoaded config.option 2016-10-20 17:13:04,028 [salt.returners ][INFO ][24378] Using default for mysql ssl_key 2016-10-20 17:13:04,028 [salt.returners ][INFO ][24378] Using default for mysql ssl_cert 2016-10-20 17:13:04,029 [salt.returners ][INFO ][24378] Using default for mysql port 2016-10-20 17:13:04,029 [salt.returners ][INFO ][24378] Using default for mysql ssl_ca 2016-10-20 17:13:04,029 [salt.loaded.int.returner.mysql][DEBUG ][24378] Generating new MySQL connection pool 2016-10-20 17:13:04,051 [salt.client ][DEBUG ][24378] get_iter_returns for jid 20161020171304002177 sent to set([‘MINION_ID’]) will timeout at 17:13:09.050518 2016-10-20 17:13:04,052 [salt.utils.event ][TRACE ][24378] _get_event() waited 0 seconds and received nothing 2016-10-20 17:13:04,064 [salt.utils.event ][TRACE ][24378] get_event() received = {‘tag’: ‘salt/job/20161020171304002177/new’, ‘data’: {‘tgt_type’: ‘glob’, ‘jid’: ‘20161020171304002177’, ‘tgt’: ‘MINION_ID’, ‘_stamp’: ‘2016-10-20T22:13:04.003420’, ‘user’: ‘sudo_USER’, ‘arg’: [‘/root/myip.txt’], ‘fun’: ‘cp.push’, ‘minions’: [‘MINION_ID’]}} 2016-10-20 17:13:04,072 [salt.master ][TRACE ][6560] AES payload received with command _file_recv 2016-10-20 17:13:04,073 [salt.master ][TRACE ][6560] Master function call _file_recv took 1.90734863281e-05 seconds 2016-10-20 17:13:04,086 [salt.master ][TRACE ][6551] AES payload received with command _return 2016-10-20 17:13:04,087 [salt.utils.job ][INFO ][6551] Got return from MINION_ID for job 20161020171304002177 2016-10-20 17:13:04,088 [salt.utils.event ][DEBUG ][6551] Sending event - data = {‘fun_args’: [‘/root/myip.txt’], ‘jid’: ‘20161020171304002177’, ‘return’: False, ‘retcode’: 0, ‘success’: True, ‘cmd’: ‘_return’, ‘_stamp’: ‘2016-10-20T22:13:04.087391’, ‘fun’: ‘cp.push’, ‘id’: ‘MINION_ID’} 2016-10-20 17:13:04,089 [salt.returners ][INFO ][6551] Using default for mysql ssl_key 2016-10-20 17:13:04,089 [salt.returners ][INFO ][6551] Using default for mysql ssl_cert 2016-10-20 17:13:04,090 [salt.utils.event ][TRACE ][6507] get_event() received = {‘tag’: ‘salt/job/20161020171304002177/ret/MINION_ID’, ‘data’: {‘fun_args’: [‘/root/myip.txt’], ‘jid’: ‘20161020171304002177’, ‘return’: False, ‘retcode’: 0, ‘success’: True, ‘cmd’: ‘_return’, ‘_stamp’: ‘2016-10-20T22:13:04.087391’, ‘fun’: ‘cp.push’, ‘id’: ‘MINION_ID’}} 2016-10-20 17:13:04,090 [salt.utils.event ][TRACE ][24378] get_event() received = {‘tag’: ‘salt/job/20161020171304002177/ret/MINION_ID’, ‘data’: {‘fun_args’: [‘/root/myip.txt’], ‘jid’: ‘20161020171304002177’, ‘return’: False, ‘retcode’: 0, ‘success’: True, ‘cmd’: ‘_return’, ‘_stamp’: ‘2016-10-20T22:13:04.087391’, ‘fun’: ‘cp.push’, ‘id’: ‘MINION_ID’}} 2016-10-20 17:13:04,090 [salt.utils.reactor ][DEBUG ][6507] Gathering reactors for tag salt/job/20161020171304002177/ret/MINION_ID 2016-10-20 17:13:04,090 [salt.returners ][INFO ][6551] Using default for mysql port 2016-10-20 17:13:04,091 [salt.returners ][INFO ][6551] Using default for mysql ssl_ca 2016-10-20 17:13:04,091 [salt.client ][DEBUG ][24378] jid 20161020171304002177 return from MINION_ID 2016-10-20 17:13:04,091 [salt.loaded.int.returner.mysql ][DEBUG ][6551] Trying to reuse MySQL connection pool 2016-10-20 17:13:04,095 [salt.master ][TRACE ][6551] Master function call _return took 0.00797820091248 seconds 2016-10-20 17:13:04,096 [salt.utils.lazy ][DEBUG ][24378] LazyLoaded nested.output 2016-10-20 17:13:04,097 [salt.output ][TRACE ][24378] data = {‘MINION_ID’: False} 2016-10-20 17:13:04,198 [salt.utils.event ][TRACE ][24378] _get_event() waited 0 seconds and received nothing 2016-10-20 17:13:04,199 [salt.client ][DEBUG ][24378] jid 20161020171304002177 found all minions set([‘MINION_ID’])

Minion debug log (log_level: all) 2016-10-20 17:17:17,516 [salt.minion ][INFO ][8164] User sudo_USER Executing command cp.push with jid 20161020171304002177 2016-10-20 17:17:17,517 [salt.minion ][DEBUG ][8164] Command details {‘tgt_type’: ‘glob’, ‘jid’: ‘20161020171304002177’, ‘tgt’: ‘MINION_ID’, ‘ret’: ‘’, ‘user’: ‘sudo_USER’, ‘arg’: [‘/root/myip.txt’], ‘fun’: ‘cp.push’} 2016-10-20 17:17:17,540 [salt.minion ][INFO ][8283] Starting a new job with PID 8283 2016-10-20 17:17:17,543 [salt.utils.lazy ][DEBUG ][8283] LazyLoaded cp.push 2016-10-20 17:17:17,545 [salt.minion ][TRACE ][8283] Executors list [‘direct_call.get’] 2016-10-20 17:17:17,546 [salt.utils.lazy ][DEBUG ][8283] LazyLoaded direct_call.get 2016-10-20 17:17:17,546 [salt.loaded.int.module.cp ][DEBUG ][8283] Trying to copy ‘/root/myip.txt’ to master 2016-10-20 17:17:17,546 [salt.crypt ][DEBUG ][8283] Initializing new SAuth for (‘/etc/salt/pki/minion’, ‘MINION_ID’, ‘tcp://MASTER_IP:4506’) 2016-10-20 17:17:17,550 [salt.crypt ][DEBUG ][8283] Loaded minion key: /etc/salt/pki/minion/minion.pem 2016-10-20 17:17:17,556 [salt.transport.zeromq ][DEBUG ][8283] Initializing new AsyncZeroMQReqChannel for (‘/etc/salt/pki/minion’, ‘MINION_ID’, ‘tcp://MASTER_IP:4506’, ‘aes’) 2016-10-20 17:17:17,557 [salt.crypt ][DEBUG ][8283] Initializing new AsyncAuth for (‘/etc/salt/pki/minion’, ‘MINION_ID’, ‘tcp://MASTER_IP:4506’) 2016-10-20 17:17:17,559 [salt.transport.zeromq ][TRACE ][8283] Inserted key into loop_instance_map id 43644256 for key (‘/etc/salt/pki/minion’, ‘MINION_ID’, ‘tcp://MASTER_IP:4506’, ‘aes’) and process 8283 2016-10-20 17:17:17,571 [salt.loaded.int.module.cp ][ERROR ][8283] cp.push Failed transfer failed. Ensure master has ‘file_recv’ set to ‘True’ and that the file is not larger than the ‘file_recv_size_max’ setting on the master. 2016-10-20 17:17:17,573 [salt.minion ][DEBUG ][8283] Minion return retry timer set to 7 seconds (randomized) 2016-10-20 17:17:17,573 [salt.minion ][INFO ][8283] Returning information for job: 20161020171304002177 2016-10-20 17:17:17,574 [salt.transport.zeromq ][DEBUG ][8283] Initializing new AsyncZeroMQReqChannel for (‘/etc/salt/pki/minion’, ‘MINION_ID’, ‘tcp://MASTER_IP:4506’, ‘aes’) 2016-10-20 17:17:17,574 [salt.crypt ][DEBUG ][8283] Initializing new AsyncAuth for (‘/etc/salt/pki/minion’, ‘MINION_ID’, ‘tcp://MASTER_IP:4506’) 2016-10-20 17:17:17,578 [salt.transport.zeromq ][TRACE ][8283] Inserted key into loop_instance_map id 43644112 for key (‘/etc/salt/pki/minion’, ‘MINION_ID’, ‘tcp://MASTER_IP:4506’, ‘aes’) and process 8283 2016-10-20 17:17:17,594 [salt.minion ][TRACE ][8283] ret_val = None 2016-10-20 17:17:22,168 [salt.utils.process ][TRACE ][8164] Process manager iteration

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Master version:

Salt Version: Salt: 2016.3.3

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 1.4.1 gitdb: 0.5.4 gitpython: 0.3.2 RC1 ioflo: Not Installed Jinja2: 2.2.1 libgit2: 0.20.0 libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.6 mysql-python: 1.2.3c1 pycparser: Not Installed pycrypto: 2.6.1 pygit2: 0.20.3 Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56) python-gnupg: 0.3.8 PyYAML: 3.11 PyZMQ: 14.5.0 RAET: Not Installed smmap: 0.8.1 timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.0.5

System Versions: dist: centos 6.6 Final machine: x86_64 release: 2.6.32-504.el6.x86_64 system: Linux version: CentOS 6.6 Final

Minion version:

Salt Version: Salt: 2016.3.2

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: unknown libgit2: Not Installed libnacl: Not Installed M2Crypto: 0.20.2 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pygit2: Not Installed Python: 2.6.6 (r266:84292, Feb 22 2013, 00:00:18) python-gnupg: 0.3.8 PyYAML: 3.10 PyZMQ: 14.3.1 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 3.2.5

System Versions: dist: centos 6.4 Final machine: x86_64 release: 2.6.32-358.el6.x86_64 system: Linux version: CentOS 6.4 Final