SaltGUI: Job listing breaks after using x509v2 states on salt 3006.1

I’d like too I just haven’t had time to thoroughly post the issue.

The workaround should go in netapi/rest_cherrypy/app.py (forget about json_out.py)

add the following function (anywhere in that file):

def fix_data_for_json(o, level): 
    if type(o) is dict: 
        n = {} 
        for k,v in o.items(): 
            n[k] = fix_data_for_json(v, level+1) 
        return n
    if type(o) is list:
        n = [fix_data_for_json(x, level+1) for x in o]
        return n
    if type(o) is bytes:
        # convert bytes-string into a representation of it
        return 'bytes:' + ''.join(format(b, '02x') for b in o)
    return o

add the following code in function hypermedia_handler, just before response = out(ret):

        # first eliminate all binary fields
        # known to exist in /stats
        # and in x509.sign_remote_certificate
        ret = fix_data_for_json(ret, 1)

Restart both salt-api and salt-master.

The result no longer crashes the salt-api, and thus SaltGUI can present the results. In this case, the presence of the public_key in the function arguments means that the display on the screen for an individual job is lousy and will need improvement. see https://github.com/erwindon/SaltGUI/pull/533

In theory I should be able to trigger this with a curl command similar to what saltgui is requesting correct?

Is this the related issue you were referring to? https://github.com/saltstack/salt/issues/59620

I also notice in trace log_level I can see a byte string for the ‘public_key’ is in bytes in the salt-api logs:

2023-07-14 00:01:26,294 [salt.utils.event :32  ][TRACE   ][4236] get_event() received = {'data': {'fun': 'runner.jobs.list_jobs', 'jid': '20230714000125247475', 'user': 'saltuser1', 'fun_args': [], '_stamp': '2023-07-14T00:01:26.285480', 'return': {'20230713235509404634': {'Function': 'saltutil.sync_all', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:09.404634'}, '20230713235510710927': {'Function': 'saltutil.refresh_pillar', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:10.710927'}, '20230713235511310291': {'Function': 'mine.update', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:11.310291'}, '20230713235512168953': {'Function': 'state.sls', 'Arguments': ['mine_hosts', {'__kwarg__': True, 'queue': False, 'concurrent': False}], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:12.168953'}, '20230713235521421624': {'Function': 'grains.items', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'saltuser1', 'StartTime': '2023, Jul 13 23:55:21.421624'}, '20230713235522759582': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:22.759582'}, '20230713235523474794': {'Function': 'test.version', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:23.474794'}, '20230713235525637575': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:25.637575'}, '20230713235528704888': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:28.704888'}, '20230713235550263847': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:50.263847'}, '20230713235554012710': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:55:54.012710'}, '20230713235604699929': {'Function': 'grains.items', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'saltuser1', 'StartTime': '2023, Jul 13 23:56:04.699929'}, '20230713235605656092': {'Function': 'test.version', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:56:05.656092'}, '20230713235606169578': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 13 23:56:06.169578'}, '20230714000028210876': {'Function': 'statels.sls', 'Arguments': ['trigger'], 'Target': 'saltmaster', 'Target-type': 'glob', 'User': 'sudo_root', 'StartTime': '2023, Jul 14 00:00:28.210876'}, '20230714000033355193': {'Function': 'state.sls', 'Arguments': ['trigger'], 'Target': 'saltmaster', 'Target-type': 'glob', 'User': 'sudo_root', 'StartTime': '2023, Jul 14 00:00:33.355193'}, '20230714000035334537': {'Function': 'x509.sign_remote_certificate', 'Arguments': ['server', {'digest': 'sha256', 'signing_private_key': None, 'signing_private_key_passphrase': None, 'signing_cert': None, 'public_key': b'0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xc2/\xec\xbf\xe7G\xa0\x8e\xb5m\xda+g\xe9\xd3H\xf4\x89x\xe6nC\x82\x0c(Z\xa0\x92\xd2\x8a\xab\xd8f\xa2\xee\x19C\xdf\x9f\xa8\x99uKw\x18\xfa3!>\x01A\x7fJ*\x9f\x1b\xe9X\xff\x88*D\xd3M\x97\x1d\xado\n+\x0c\xbe\x13\xa8h\xec\xca\xd1"\xbcXF\xb7>\x95\x83I\x1a\x85OoP\x84\xfe\xf9|\xcd\xdd{\xed\xcc\xb3;v\x0fJ\xf1\x93\xe7t\xc0\xf5\xe2\x89\xcfd\x87OCJl+\x15\x9aiVL\xdb.\x06\xa4:\xca!\x12\xde\xb9=\xc1\xba\x08\xf5Gy\x08\x1b\xd6C\xc6\x90\xa8!\x07>\xdf8\x96YQ\xc1\x16\xca]\x9e\x02\xe3\x0b\x16\xa5\xf9:\x93\xaa\xe3\x11X\xfc\x1c\x1f\x92\x06\xf5i5o\x0b@\xd9n)\xd7\x08Y\x1f\x0c\xa3\x9e\x9bLDVH\xcck\x89\xe2\xe0r\xcd5\x8e\x1c\xce\xf8\xb3\xa9\x85\xd6\xb6\xc2\xa6\xa1\x8d+\xabg4;i\xd8\xec\xf27\x81W_\xec\xef\xd9\x11\xc5_~$T\xa5\x84\n5F\xc1\xe40D\xe9\x07\x02\x03\x01\x00\x01', 'csr': None, 'subject': None, 'serial_number': None, 'not_before': None, 'not_after': None, 'days_valid': 365, 'CN': 'saltmaster.vagrant.lan'}, False], 'Target': 'saltmaster', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 14 00:00:35.334537'}, '20230714000038524520': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 14 00:00:38.524520'}, '20230714000125993037': {'Function': 'saltutil.running', 'Arguments': [], 'Target': '*', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2023, Jul 14 00:01:25.993037'}}, 'success': True}, 'tag': 'salt/run/20230714000125247475/ret'}
2023-07-14 00:01:26,294 [salt.transport.ipc:372 ][DEBUG   ][4236] Closing IPCMessageSubscriber instance
2023-07-14 00:01:26,295 [salt.loaded.int.netapi.rest_cherrypy.app:917 ][DEBUG   ][4236] Could not serialize the return data from Salt.
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py", line 913, in hypermedia_handler
    response = out(ret)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/json.py", line 137, in dumps
    return json_module.dumps(obj, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/saltstack/salt/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

I can trim down a salt state to trigger this easily if needed. I realized my state was based around having pillar and such already configured.

that would help me too since I could not reproduce the effect even with the info from https://github.com/erwindon/SaltGUI/issues/532#issuecomment-1634839759

What are the next steps here? Should I make a ticket for salt-api or?

indeed! see https://github.com/saltstack/salt/issues but you need to provide a really simple reproducer with the ticket. something that does not need other sls includefiles or directories like /etc/clusterca, output for one minion only, etc. I would appreciate it when you post the url of that ticket here. btw, I will close this ticket after that since there is nothing I can change in SaltGUI that helps against this

I can trim down a salt state to trigger this easily if needed. I realized my state was based around having pillar and such already configured.

HTTP code 500 indicated the the server crashed (here for this request only) typically that means a python exception was raised while working on the request but an error the backend-code itself is not that fatal, by failure in the conversion to json is since that happens only in salt-api (and not in salt commands) the dev-team does not always spot it

Ack sorry. Here’s the right one?

/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py

that is the full filename for me too

no you need to select the “500” line. the line with “events?token=” is just the event-stream (see page “Events”) that one works fine, it was just interrupted

I am currently trying to narrow this down to where it starts failing…

@tazaki

I’m sure this error is in salt-api itself. It would also appear if you use the salt-api with another application (e.g. in a shell script using curl/wget)

salt-api uses a REST interface that has a JSON encoding style. but internally, saltstack uses Python code which has the ability to use data that cannot be converted to JSON when that data ends up in a response that triggers the error you see.

We have seen this type of error before in the /stats api while also using setting collect_stats=true. in that specific case, the return data had a bytes-string as the value for some field. I think that one was solved btw

can you supply me with a minimal states-file that triggers the error? can you see which api-call triggers the error? should be visible in the debug-view of the browser in section “network”.