core: New manually configured MQTT entities with unique IDs don't show up in HA

The problem

If I define a new entity with the MQTT integration by editing a yaml file to include a unique_id field, that entity does not show up in the entities list after a reload operation. If I edit the yaml to remove the unique_id entry in the yaml and reload, the new entity shows up and the entity_id has a “_2” suffix on it. If I edit the yaml again and restore the unique_id, the entity disappears. This occurs for most if not all platforms for MQTT including sensor, binary_sensor, fan, and cover.

What version of Home Assistant Core has the issue?

System Information version | core-2023.11.3 – | – installation_type | Home Assistant OS dev | false hassio | true docker | true user | root virtualenv | false python_version | 3.11.6 os_name | Linux os_version | 6.1.59 arch | x86_64 timezone | America/Chicago config_dir | /config

Home Assistant Community Store

GitHub API | ok – | – GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 5000 Installed Version | 1.33.0 Stage | running Available Repositories | 1357 Downloaded Repositories | 5

Home Assistant Cloud

logged_in | true – | – subscription_expiration | February 11, 2024 at 6:00 PM relayer_connected | true relayer_region | us-east-1 remote_enabled | true remote_connected | true alexa_enabled | true google_enabled | false remote_server | us-east-1-1.ui.nabu.casa certificate_status | ready can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Home Assistant Supervisor

host_os | Home Assistant OS 11.1 – | – update_channel | stable supervisor_version | supervisor-2023.11.6 agent_version | 1.6.0 docker_version | 24.0.6 disk_total | 125.3 GB disk_used | 26.4 GB healthy | true supported | true board | ova supervisor_api | ok version_api | ok installed_addons | Acurite to Home Assistant (0.3.23), Studio Code Server (5.14.2), Z-Wave JS (0.4.1), Samba share (12.2.0), Terminal & SSH (9.8.1), TasmoBackup (1.05.00), Network UPS Tools (0.12.2), Piper (1.4.0), Home Assistant Google Drive Backup (0.112.1), InfluxDB (4.8.0), Node-RED (16.0.2), ESPHome (2023.11.6), Advanced SSH & Web Terminal (16.0.1), openWakeWord (1.8.2), Whisper (1.0.0)

Dashboards

dashboards | 4 – | – resources | 2 views | 15 mode | storage

Recorder

oldest_recorder_run | December 2, 2023 at 4:27 AM – | – current_recorder_run | December 3, 2023 at 8:38 AM estimated_db_size | 563.47 MiB database_engine | sqlite database_version | 3.41.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

MQTT

Link to integration documentation on our website

https://www.home-assistant.io/integrations/mqtt

Diagnostics information

When the entity_id is not present in HA, there are no events in the log that indicate why (default logging level). If I set logging level to debug, I can see some references to the entity I’m trying to create, but I don’t understand what they mean. I’m attaching a log file (level debug) that was collected after re-enabling the unqiue_id on the motion sensor in the yaml snippet below. home-assistant_2023-12-04T11-52-06.524Z.zip

I have tried to see if the form of the unique_id I specify affects the problem and it does not appear to. I’ve used text-only, numeric-only and text+numeric IDs with no difference in behavior.

Example YAML snippet

- name: "Garage Motion 12"
  device_class: motion
  state_topic: "home/garage/sensorArray/state"
  availability_topic: "home/garage/sensorArray/availability"
  value_template: "{{ value_json.forward.motion }}"
  payload_on: "ON"
  payload_off: "OFF"
  payload_available: "online"
  payload_not_available: "offline"
  unique_id: 10003

Anything in the logs that might be useful for us?

I can't nail down the version of core where this started but I'd say that it was released around 5 months ago.  I've been creating new MQTT entities without unique IDs since then.

Additional information

No response