core: Riemann Sum integration uses broken defaults

The problem

The trapezoidal method (the default) of the Reimann Sum integration is completely incompatible with the way Home Assistant stores values.

See:

I also see the issue come up on Reddit every now and then.

Every time the workaround is “use the left method”. But it’s not just those specific cases that need a different method - the default of trapezoidal is simply wrong for practically any use case. With the way Home Assistant does not store or report updates to states with the same value, it will give inaccurate results for any sensor that could have repeated values. And that would be pretty-much every sensor out there.

The current warning in the documentation is not enough - it is not just in some specific cases that left should be used - it should be used in practically all cases. And even the “Energy” example does not use left, and that is where we most often see the issue come up.

I’d recommend changing the default to left. Even though that could be considered a “breaking change”, I’d estimate it would fix many more cases than it would break.

Alternatively, require always setting the method, and make it clear in the documentation that trapezoid should only be used in some specific cases. This requirement could be phased in over time, by first showing a deprecation message for config that doesn’t specify the method.

The right method would probably be even worse, but at least most users should not attempt to use it.

Another related issue, but more difficult to solve:

What version of Home Assistant Core has the issue?

core-2023.4.6

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Riemann sum integral

Link to integration documentation on our website

https://www.home-assistant.io/integrations/integration/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 7
  • Comments: 15 (2 by maintainers)

Most upvoted comments

The point the OP makes is valid, not just because HA does not store unchanged values. Battery sensors will typically only send a new value when the value changes significantly. So left Riemann is best for that too. Any sensor that updates on change - left is best.

Only on time based sampling trapezoid is better, but that is rarely the case. One could argue that time based sampling on something that requires a Riemann integral is by itself a bad choice unless the samples are very frequent.

Any sensor that has long constant values in HA need left. That is basically any sensor that can be 0 for a significant duration, such as wattage on devices, solar energy… again, most of what HA uses it for. And even if trapezoid would work, I would argue left would work pretty well too. The reverse is definitely not the case.

And did any one mention all graphs in HA also display the left Riemann equivalent, not trapezoid? That is not without reason.

The Problem still exists

For left, it still only updates when the underlying value changes, but when it does update it will have the correct value.

For the issue of the update only happening when the value changes, see https://github.com/home-assistant/core/issues/88940. That is a more difficult one to solve - likely only by changing the implementation to be timer-based instead of / in addition to event-based.

With trapezoid, there is no way for the value to be correct without using a timer-based implementation. So having it as the default without fixing that is a big issue.

related question, why isn’t it a timer-based implementation? only updating when the value changes introduces confusion and problems such as this one, numbers not changing until it receives a different value and the whole sensor becoming “unknown” when it receives a value of 0.

This is what is happening on my side:

Schermata 2023-05-10 alle 08 35 57

Schermata 2023-05-10 alle 08 36 07

Schermata 2023-05-10 alle 08 51 38

this is the implementation:

sensor:
  - platform: template
    sensors:
      smartplugs_consumption:
        friendly_name: Smartplugs Consumption
        icon_template: 'mdi:power-socket-eu'
        # TPLink, we delegate consumption calculation to total device nr (for different consumptions on/off
        value_template: >-
          {% set dev = namespace(value=0) %}
          {% set meross = namespace(total = 0) -%}
          # count plugs
          {% for pwr in states.switch -%}
            {% if pwr.entity_id | regex_search('^switch\.meross.*$') -%}
              {% set meross.total = meross.total + 1  -%}
            {% endif -%}
          {% endfor -%}
          # smart plug consumption is 0.8W each
          {% set dev.value = meross.total * 0.8 %}
          # let's add some 0.01 consumption to total consumption int every 5 mins to make riemann integration happy
          {% set current_minute = strptime(states('sensor.time'), "%H:%M").minute %}
            {% if current_minute % 5 == 0 %}
              {% set dev.value = dev.value | int + 0.01 %}
            {% endif %}
          {{ dev.value }}

        daily_smartplugs_rounded:
          value_template: >-
            {{ "%.2f"|format(states('sensor.daily_smartplugs_peak') | float) }}
          attribute_templates:
            cost: >-
              {{ "%.2f"|format(states('sensor.daily_smartplugs_peak') | float * 0.246) }}
            last_reset: "{{ now().replace(day=now().day -1, hour=22, minute=0, second=0, microsecond=0)  }}"
            state_class: 'total'
            meter_period: daily
          unit_of_measurement: kWh
          device_class: energy

  - platform: integration
    source: sensor.smartplugs_consumption
    name: smartplugs_instant
    unit_prefix: k
    round: 2
    method: left

utility_meter:
  daily_smartplugs:
    source: sensor.smartplugs_instant
    cycle: daily
    tariffs:
      - peak
      - offpeak

the entity in the picture is “daily_smartplugs_rounded”, but the smartplugs_consumption (not rounded) has exactly the same saw-like pattern and value. though it get displayed correctly (i.e. positive) on the energy dashboard.

Don’t know, maybe i’ts a templating error or something, Tried to set daily_smartplugs_rounded to state_class: ‘total’ or state_class: ‘total_increment’ (and remove last_reset, as per docs), but is still have this weird behaviour i did not have before 2023.5.

It would update on 1st sensor change