keylime: Python 3 Port - TypeError: Object of type bytes is not JSON serializable

Hi @lukehinds

It looks like py3 now exports byte arrays from base64.b64encode instead of ASCII strings when performing its base64 encoding. We will need to convert this byte array back to ASCII for this to be JSON serializeable, using .decode('ascii').

For https://github.com/keylime/keylime/blob/e7c153a16fa47e3b411ddbcd18f04a66b37e8898/keylime/ca_util.py#L448

Change base64.b64encode(crypto.generate_random_key()) to base64.b64encode(crypto.generate_random_key()).decode('ascii') so it can be serialized.

Can you see if that fixes the issue for you? We might need to do this everywhere base64 is performed …

@lukehinds – can you try bytes(nonce, encoding="utf8").hex() and see if that works?

@jetwhiz, yep. Just encoding. Almost have all the tests passing, so will be making a PR shortly I hope.

fixed the above: contents = b"random garbage to test as payload"

@lukehinds – does this mean the issue you posted is fixed now? was it another encoding issue?

Hi @lukehinds

Have you updated the fork arguments for the testing script:

https://github.com/keylime/keylime/blob/94328329d2dad5d5a659168865d2e73347a48ce4/test/test_restful.py#L69-L70

This should be changed to python3 so that the services are all kicked off with the python3 interpreter.

Hi @lukehinds

Apparently py3 now returns a float when dividing two integers, so you’ll have to switch to using floor division to get py2 behavior back. Try updating this line and see if that works:

        alg_size = Hash_Algorithms.get_hash_size(hash_alg) // 4

Note that this same problem will likely pop up in check_quote as well (and maybe other places): https://github.com/keylime/keylime/blob/94328329d2dad5d5a659168865d2e73347a48ce4/keylime/tpm2.py#L967-L973

Bravo! I’m sure you can breathe a sigh of relief now :p The tpm1.py changes should be pretty much the same as tpm2.py, and after that will be the WebApp: we’ll need to switch to using a yaml parser now it sounds, maybe https://github.com/nodeca/js-yaml? I can help with that part after everything else stabilizes.

Thanks Charlie, that did it. I was using application/x-yaml , and text/x-yaml resolved it!

I think sockserver is trying to return a string that is encoded with a particular encoding (e.g,. UTF-8) as a browser response for the REST call. You’ll have to export dict as a yaml string and then encode that as UTF-8 before passing this into json_response (such as with yaml.dump).

So for instance:

def echo_yaml_response(handler,code,status=None,results=None):
    """Takes a package and returns it to the user w/ full HTTP headers"""
    if handler is None or code is None:
        return False
    if status is None:
        status = httplib.responses[code]
    if results is None:
        results = {}
    
    yaml_res = {'code': code, 'status': status, 'results' : results}
    yaml_response = yaml.dump(yaml_res, encoding='utf-8')
    
    if isinstance(handler, BaseHTTPRequestHandler):
        handler.send_response(code)
        handler.send_header('Content-Type', 'text/x-yaml; charset=utf-8')
        handler.end_headers()
        handler.wfile.write(yaml_response)
        return True
    elif isinstance(handler, tornado.web.RequestHandler):
        handler.set_status(code)
        handler.set_header('Content-Type', 'text/x-yaml; charset=utf-8')
        handler.write(yaml_response)
        handler.finish()
        return True
    else:
        return False

(untested, but something along those lines)

Note in the code above that you need to change the Content-type headers being returned in these locations (L225 and L231) to text/x-yaml; charset=utf-8 so the requesters know how to deal with the response correctly.

All of the receivers must be changed as well to parse YAML instead of JSON, such as here: cloud_agent.py#L190. Probably with something like yaml_body = yaml.load(post_body).

Support for YAML in the tornado_requests class would also be a good idea: tornado_requests.py#L61 so that these calls are still possible: registrar_client.py#L104.

due an update.

So we can ignore the above woes, which have all been solved by having yaml handle private.json and tmpdata.json. This works well as it accepts strings and bytes, unlike json, so we are no loner encoding and decoding every time.

I am not towards the end of the registration process. Adding this as I might have appeared to have gone quiet and taking some time over this migration.

i think it isn’t deep into the c code. It’s in the callback to provide the password. i think this is what is causing the problem https://github.com/keylime/keylime/blob/a5473dca8ce500182b95f41e2cc291d825c50075/keylime/ca_util.py#L77