salt: [ERROR ] Salt request timed out. If this error persists, worker_threads may need to be increased.

I don’t understand what’s going on here. When I try to target using grains I get the following error and it takes quite a while to time out.

[root@i-8d------ master.d]# time salt -G 'role:eam-ejcron' test.ping

[ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.

Failed to authenticate!  This is most likely because this user is not permitted to execute commands, but there is a small possibility that a disk error occurred (check disk/inode usage).

real    0m35.388s
user    0m0.751s
sys     0m0.258s
[root@i-8d------ master.d]# 

If I ping an instance that I know has that role by “name” it works, still pretty slow.

[root@i-8d------ master.d]# time  salt "i-30------" test.ping
i-30------:
    True
real    0m12.682s
user    0m0.808s
sys     0m0.268s
[root@i-8d------ master.d]# 

Verifying that the instance has the grain I think it should.

[root@i-8d------ master.d]# time salt 'i-30------' cmd.run 'grep role /etc/salt/grains'
i-30------
    role: eam-ejcron
real    0m5.138s
user    0m0.671s
sys     0m0.245s
[root@i-8d------ master.d]# time  salt 'i-30------' grains.get role             
i-30------:
    eam-ejcron
real    0m4.445s
user    0m0.772s
sys     0m0.330s
[root@i-8d------ master.d]#

Version of the saltmaster follows. It is an m3.medium in EC2.

[root@i-8d------ master.d]# salt --versions-report
         Salt: 2014.7.5
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: 2.2.1
         M2Crypto: 0.20.2
         msgpack-python: 0.1.13
         msgpack-pure: Not Installed
         pycrypto: 2.0.1
         libnacl: Not Installed
         PyYAML: 3.10
         ioflo: Not Installed
         PyZMQ: 14.3.1
         RAET: Not Installed
         ZMQ: 4.0.4
         Mako: Not Installed
[root@i-8d------ master.d]#

Plenty of free disk space and inodes.

[root@i-8d------ master.d]# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvde1      7.9G  4.6G  3.4G  58% /
[root@i-8d------ master.d]# df -i /
Filesystem     Inodes IUsed  IFree IUse% Mounted on
/dev/xvde1     524288 63094 461194   13% /
[root@i-8d------ master.d]#

worker_threads setting

[root@i-8d------ master.d]# grep worker_threads /etc/salt/master.d/jobs.conf
worker_threads: 16
[root@i-8d------ master.d]#