core: Template and JSON parse exception when using bitwise_and filter with binary data

The problem

I have a binary payload received via MQTT which isn’t getting parsed because of an exception with the bitwise_and filter.

Additionally, I created an automation which parses the value from the incoming binary data, but this throws an ‘unable to parse’ exception.

• Incoming data: b'\xc2\x9b' (i.e. 0xC29B)

MQTT sensor configuration:

sensor
  - platform: mqtt
    unique_id: "mysensor"
    name: "Temperature"
    state_topic: "up/mysensor/115"
    device_class: "temperature"
    unit_of_measurement: "°C"
    value_template: "{{ value | bitwise_and(4095) }}"

I have tried using bitwise_and(b'\x0f\xff') and converting the incoming value to a float which also doesn’t work.

MQTT was designed with small payloads in mind, so to me it makes more sense to use binary payloads over json strings but this doesn’t seem to be supported in HA. I spotted this unit test: https://github.com/home-assistant/core/blob/1bcd62cd32ff84d3a56d6bfd88589410e1eb9891/tests/components/mqtt/test_init.py#L458-L469 https://github.com/home-assistant/core/blob/1bcd62cd32ff84d3a56d6bfd88589410e1eb9891/homeassistant/components/mqtt/subscription.py#L89

Why is UTF8/ASCII the only supported payload type? I can’t seem to find this in the mqtt integration documentation. Can’t we specify the payload type in the sensor config or have it converted to something that is usable by HA?

What version of Home Assistant Core has the issue?

core-2021.11.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

No response

Link to integration documentation on our website

No response

Example YAML snippet

id: '1637503671681'
alias: MQTT test
description: ''
trigger:
  - platform: mqtt
    topic: up/mysensor/115
    id: up-mysensor-115
    encoding: ''
condition: []
action:
  - event: mqttdata
    event_data:
      raw: '{{ trigger.payload }}'
      temperature: '{{ trigger.payload | bitwise_and(4095) }}'
mode: single

Anything in the logs that might be useful for us?

2021-11-21 14:25:31 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on up/mysensor/115: b'\xc2\x9b'
2021-11-21 14:25:31 ERROR (MainThread) [homeassistant.util.logging] Exception in message_received when handling msg on 'up/mysensor/115': '^{'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 37, in wrapper
    msg_callback(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/sensor.py", line 227, in message_received
    _update_state(msg)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/sensor.py", line 192, in _update_state
    payload = template.async_render_with_possible_json_value(
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 561, in async_render_with_possible_json_value
    return _render_with_context(
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 1698, in _render_with_context
    return template.render(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 1597, in bitwise_and
    return first_value & second_value
TypeError: unsupported operand type(s) for &: 'str' and 'int'

2021-11-21 14:25:31 ERROR (MainThread) [homeassistant.components.websocket_api.messages] Unable to serialize to JSON. Bad data found at $.event(Event: mqttdata).data.raw=b'\xc2\x9b'(<class 'bytes'>
2021-11-21 14:25:31 WARNING (Recorder) [homeassistant.components.recorder] Event is not JSON serializable: <Event mqttdata[L]: raw=b'\xc2\x9b'>

...

2021-11-21 15:27:31 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on up/mysensor/115: b'\x02\x8c'
2021-11-21 15:27:31 WARNING (MainThread) [homeassistant.components.mqtt] Can't decode payload b'\x02\x8c' on up/mysensor/115 with encoding utf-8 (for <Job HassJobType.Callback <function MqttSensor._subscribe_topics.<locals>.message_received at 0x7f1fc42e9790>>

Additional information

No response