core: Logbook exclude filters produce unexpected results when combined with recorder filters

I also encountered the problem today. No matter what I try, my sensors cannot be excluded from the logbook integration, even though I am not interested in them being displayed.

Still an issue

as this is not a critical issue I am fine with it for now, but would try to keep the conversation alive. Do you have a link to the discussion at hand?

The conversations were on discord, and sadly I’ve lost track of them at this point.

If a clear scope results from such a conversation I could invest some time to start working on the implementation

I think the next step would be to start an architecture discussion here so everyone can get aligned on a solution: https://github.com/home-assistant/architecture/discussions

Not sure if it is a conflict from user perspective although it might be a “conflict” in the overlayed filters from technical perspective.

It seems to be impossible to exclude something in logbook which has been included in recorder. I created 2 simple input_booleans to test the behaviour with this config which you might describe as “conflict”:

recorder:
  include:
    entity_globs:
      - input_boolean.test*

logbook:
  exclude:
    entities:
      - input_boolean.test_exclude
  include:
    entities:
      - input_boolean.test_include

I would expect that a simple change to input_boolean.test_include is included in the recorder, but not in the logbook. As far as I remember this was possible before PR https://github.com/home-assistant/core/pull/72728 was introduced.

Logbook filters are overlayed on top of recorder filters. If you have a conflict in the two filters they won’t work as expected

Referring to PR #72728 it seems to be impossible to define an exclude for an entity for the logbook if this entity has been included via recorder, no matter by which filter type (domain, entity_glob, entity).

Feels like a Bug to me…

Logbook filters are overlayed on top of recorder filters. If you have a conflict in the two filters they won’t work as expected

@bdraco I tried to dig a bit into the logbook source code and in detail into the PR #72728 because I wanted to understand what this change exactly does. Here are my findings and I would love to hear especially your feedback:

• The purpose of the PR in the database which have been set by the recorder filter rules.
• To achieve this the filter rules for recorder are merged with the logbook rules.
• With this change no logbook entries are stored for entries that are exluded by recorder rules (which is imo ok).
• On the other side, as stated in some comments in this thread, it is not longer possible to exclude entries from logbook which have been included by recorder rules. Imo this case was completely forgotten when implementing the change.
• The documentation does not describe the current behaviour because “exclude” is not possible:

By default, the logbook will use the same filter as the recorder. To limit which entities are being exposed to Logbook, you can use the include and exclude parameters.

Conclusion: Merging the filter rules was a bad idea. Imo the recorder rules should only be applied for storing the logbook entries into db and the logbook rules should only be applied for returning them to UI, … That way no merge is needed.

This is a real nightmare, I wasted a lot of time trying to get the logbook to work properly before I read this.

It’s counter intuitive, I think there should at least be a note in the documentation.

It seems to be impossible to exclude something in logbook which has been included in recorder.

That’s correct, includes always win in this case.

There’s been some talk about redoing the whole filtering system, but the conversations have died off a few months ago.

Just to set expectations, unless a champion appears to work on this, I expect the current state of things is how it’s likely to be for the foreseeable future.

After updating to 2022.11 it’s getting even worse for me, the excluded entity in logbook is not just wrongly logged in the logbook, the logbook also seems to log triggers from other entities which does not make sense at all.

Both entities are independent template sensors, why the heck does the logbook think one triggered the other 😆

EDIT: solved, this happens if multiple template sensors are generated from attributes of the same entity, they share the trigger context although no explicit trigger is defined. Got rid of this issue by defining a trigger on the attribute I want to template on.

This is the template binary sensor, just for completeness:

- binary_sensor:
  - unique_id: heizkreislauf_wohnzimmer
    name: 'Heizkreislauf Wohnzimmer'
    device_class: opening
    state: >-
      {{ 'on' if is_state_attr('homematic.ccu3', 'Thermostat_Wohnzimmer', true) else 'off' }}
    availability: "{{ states('homematic.ccu3') != 'unavailable' }}"