core: Ingress for Add-ons broken after update to 2023.9.0

The problem

After the update to Home Assistant 2023.9.0 Ingress for all Add-ons seems to be broken. I can no longer open ESPHome or File editor from Home Assistant UI. The UI presents me an “500 Internal Server Error” and inside the supervisor logs a bunch of exceptions is thrown. Reverting back to Home Assistant 2023.8.4 resolves the issues.

What version of Home Assistant Core has the issue?

core-2023.9.0

What was the last working version of Home Assistant Core?

core-2023.8.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

[31m23-09-07 12:14:35 ERROR (MainThread) [aiohttp.server] Error handling request [0m
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/aiohttp.py", line 139, in sentry_app_handle
    reraise(*_capture_exception(hub))
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/_compat.py", line 60, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/aiohttp.py", line 129, in sentry_app_handle
    response = await old_handle(self, request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 185, in block_bad_requests
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 201, in system_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 220, in token_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 280, in core_proxy
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/ingress.py", line 141, in handler
    return await self._handle_request(request, addon, path, session_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/ingress.py", line 217, in _handle_request
    async with self.sys_websession.request(
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 558, in _request
    resp = await req.send(conn)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 670, in send
    await writer.write_headers(status_line, self.headers)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_writer.py", line 129, in write_headers
    buf = _serialize_headers(status_line, headers)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aiohttp/_http_writer.pyx", line 132, in aiohttp._http_writer._serialize_headers
    _safe_header(to_str(val))
  File "aiohttp/_http_writer.pyx", line 109, in aiohttp._http_writer.to_str
    raise TypeError("Cannot serialize non-str key {!r}".format(s))
TypeError: Cannot serialize non-str key None

Additional information

System Information

version	core-2023.9.1
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.11.5
os_name	Linux
os_version	6.1.45
arch	x86_64
timezone	Europe/Berlin
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	5000
Installed Version	1.32.1
Stage	running
Available Repositories	1284
Downloaded Repositories	12

Home Assistant Cloud

logged_in	true
subscription_expiration	October 18, 2023 at 2:00 AM
relayer_connected	true
relayer_region	eu-central-1
remote_enabled	false
remote_connected	false
alexa_enabled	false
google_enabled	false
remote_server	eu-central-1-5.ui.nabu.casa
certificate_status	ready
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 10.5
update_channel	stable
supervisor_version	supervisor-2023.08.3
agent_version	1.5.1
docker_version	23.0.6
disk_total	30.8 GB
disk_used	19.2 GB
healthy	true
supported	true
board	ova
supervisor_api	ok
version_api	ok
installed_addons	Advanced SSH & Web Terminal (15.0.7), RaspberryMatic CCU (3.71.12.20230826), SQLite Web (3.9.2), ESPHome (2023.8.3), Mosquitto broker (6.3.1), Mealie (Omni): (v1.0.0-beta-5-4), File editor (5.6.0)

Dashboards

dashboards	3
resources	4
views	12
mode	storage

Recorder

oldest_recorder_run	September 7, 2023 at 9:22 AM
current_recorder_run	September 9, 2023 at 7:26 AM
estimated_db_size	1918.15 MiB
database_engine	sqlite
database_version	3.41.2