salt: [3006.x][BUG] Salt-API unstable after upgrade from 3004.2 to 3006.1

Description Salt-API unstable after upgrade from 3004.2 to 3006.1

Setup Customer is not using Aria Config components, only Salt open

Enabled eauth=“auto” on one of our salt-master’s in our failing -test environment.

external_auth: auto: saltuser: *admin_acls

then attempted to curl the salt-api. note the time it took to complete: 0m0.293s time curl -v -sSk https://s12345.svr.us.dog.net:8000/login -H ‘Accept: application/json’ -m 45 -d eauth=auto -d username=saltuser -d password=saltuser

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior enabled eauth=“auto” on one of our salt-master’s in our failing -test environment.

external_auth: auto: saltuser: *admin_acls

then attempted to curl the salt-api. note the time it took to complete: 0m0.293s

time curl -v -sSk https://s12345.svr.us.dog.net:8000/login -H ‘Accept: application/json’ -m 45 -d eauth=auto -d username=saltuser -d password=saltuser

About to connect() to s12345.svr.us.dog.net port 8000 (#0) Trying 192.168.173.29… Connected to s12345.svr.us.dog.net (192.168.173.29) port 8000 (#0) Initializing NSS with certpath: sql:/etc/pki/nssdb skipping SSL peer certificate verification SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 Server certificate: subject: CN=cassandra-test.dpg.net,OU=gti,O=Dog,L=Kat,ST=Ohio,C=US start date: Jul 20 10:16:27 2022 GMT expire date: Jul 20 10:16:27 2023 GMT common name: cassandra-test.dog.net issuer: CN=PSIN0P551,DC=exchad,DC=dog,DC=net

POST /login HTTP/1.1 User-Agent: curl/7.29.0 Host: 121345.svr.us.dog.net:8000 Accept: application/json Content-Length: 46 Content-Type: application/x-www-form-urlencoded

upload completely sent off: 46 out of 46 bytes < HTTP/1.1 200 OK < Content-Type: application/json < Server: CherryPy/18.6.1 < Date: Thu, 22 Jun 2023 18:36:43 GMT < Allow: GET, HEAD, POST < Access-Control-Allow-Origin: * < Access-Control-Expose-Headers: GET, POST < Access-Control-Allow-Credentials: true < X-Auth-Token: 441098ff2f9d91ec9c47bfa18c99f34e61a6eefb < Vary: Accept-Encoding < Content-Length: 210 < Set-Cookie: session_id=441098ff2f9d91ec9c47bfa18c99f34e61a6eefb; expires=Fri, 23 Jun 2023 04:36:43 GMT; Max-Age=36000; Path=/ < Connection #0 to host s12345.svr.us.dog.net left intact Unknown macro: {“return”} real 0m0.293s user 0m0.083s sys 0m0.074s

I then proceeded to run that curl command a few more times with similar results. However, after a minute or two, the request was unable to complete in under 45 seconds:

time curl -v -sSk https://mouse.svr.us.dog.net:8000/login -H ‘Accept: application/json’ -m 45 -d eauth=auto -d username=saltuser -d password=saltuser

About to connect() to mouse.svr.us.dog.net port 8000 (#0) Trying 192.169.200.136… Connected to mouse.svr.us.dog.net (192.169.200.136) port 8000 (#0) Initializing NSS with certpath: sql:/etc/pki/nssdb skipping SSL peer certificate verification SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 Server certificate: subject: CN=cassandra-test.dog.net,OU=gti,O=Dog,L=Kat,ST=Ohio,C=US start date: Jul 20 10:16:27 2022 GMT expire date: Jul 20 10:16:27 2023 GMT common name: cassandra-test.dpg.net issuer: CN=PSIN0P551,DC=exchad,DC=dog,DC=net

POST /login HTTP/1.1 User-Agent: curl/7.29.0 Host: mouse.svr.us.dog.net:8000 Accept: application/json Content-Length: 46 Content-Type: application/x-www-form-urlencoded

upload completely sent off: 46 out of 46 bytes Operation timed out after 45001 milliseconds with 0 out of -1 bytes received Closing connection 0 curl: (28) Operation timed out after 45001 milliseconds with 0 out of -1 bytes received real 0m45.014s user 0m0.088s sys 0m0.076s

1. Expected behavior:

   • Salt-API should work as expected

2. Versions Report:

   • mouse: Salt Version: Salt: 3006.1

Python Version: Python: 3.10.11 (main, May 5 2023, 02:31:54) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.10 gitpython: 3.1.31 Jinja2: 3.1.2 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 5.4.1 PyZMQ: 23.2.0 relenv: 0.12.3 smmap: 5.0.0 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: rhel 7.9 Maipo locale: utf-8 machine: x86_64 release: 3.10.0-1160.83.1.el7.x86_64 system: Linux version: Red Hat Enterprise Linux Server 7.9 Maipo

s12345: Salt Version: Salt: 3006.1

Python Version: Python: 3.10.11 (main, May 5 2023, 02:31:54) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.10 gitpython: 3.1.31 Jinja2: 3.1.2 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 5.4.1 PyZMQ: 23.2.0 relenv: 0.12.3 smmap: 5.0.0 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: rhel 7.9 Maipo locale: utf-8 machine: x86_64 release: 3.10.0-1160.83.1.el7.x86_64 system: Linux version: Red Hat Enterprise Linux Server 7.9 Maipo

Milkcart: Salt Version: Salt: 3006.1

Python Version: Python: 3.10.11 (main, May 5 2023, 02:31:54) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.10 gitpython: 3.1.31 Jinja2: 3.1.2 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 5.4.1 PyZMQ: 23.2.0 relenv: 0.12.3 smmap: 5.0.0 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: rhel 7.9 Maipo locale: utf-8 machine: x86_64 release: 3.10.0-1160.83.1.el7.x86_64 system: Linux version: Red Hat Enterprise Linux Server 7.9 Maipo

1. Additional context:
   • noticed number of exceptions in the event bus for minion data refresh events. I am not sure if this relevant but just mentioning it in the hope that it could be of some use.

minion/refresh/horse

{“Minion data cache refresh”: “horse”, “_stamp”: “2023-06-22T14:05:47.065752”} _salt_error {“pretag”: null, “cmd”: “_minion_event”, “tag”: “_salt_error”, “data”:

{“message”: “The minion function caused an exception”, “args”: [“The minion function caused an exception”], “_stamp”: “2023-06-22T14:05:47.391652”} , “id”: “carthorse”, “_stamp”: “2023-06-22T14:05:47.426772”}

–THROUGH TEMPLATE Salt Stack Default, PR is created by sfry, reviewed by jonessean,lskrzypczak,christopherw and submitted by jonessean–

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Versions Report

PASTE HERE

Additional context Add any other context about the problem here.