core: Derivative sensor doesn't update correctly for non-changing values

Pretty amazing this issue goes back 2 years. It seems pretty obvious that because the source sensor doesn’t update when the value no longer changes, thus the derivative sensor, which needs more than 1 data point, doesn’t update either until another value gets sent from the source. It seems most people that “fix” this issue does so like the above, with some means to force an update. I’m no different.

In my case, I created a template sensor of the source and added an attribute that updates every minute. Then I based my derivative sensor off of this.

- sensor: 
     - name: "bathroom humidity"
       unit_of_measurement: "%"     
       state: "{{ state_attr('sensor.wiser_roomstat_bathroom', 'humidity') }}"
       attributes:
         attribute: "{{ now().minute }}" 

Hope that helps someone.

I’m so astounded that many are focusing on “how would I implement” and not starting with the obvious… A “Derivative” measures the rate of change occurring. My humidity sensor sends a reading every 60 seconds. HA DISCARDS repetitive readings. The “15 minute window” derivative of an hour of, say, 45% humidity, is 0. No question.
The “15 minute window” derivative of “no readings” is 0. No question.

I understand the concern for “non-reporting sensors”. But since HA drops the repetitive values, it seems best to work with what we have. No data within the window? Report the derivative as ‘0.0’. Only one reading within the window? Report the derivative as ‘0.0’.

It almost feels like the derivative helper is not written with HA in mind, since HA, by default, discards repeated readings, yet a derivative sensor, by its nature, needs multiple readings INCLUDING repeated readings.

This happens because to calculate the derivate it uses the last known values and if new data comes in the older values are discarded (of time window). In your case your sensor didn’t admit any data for over a day so the derivative is based on that data from a day ago.

What is the value of the time_window parameter in this case? In such way it looks ridiculous.

I am not sure whether we want to change this logic. If we did, the following happens, you have a time window of 15 seconds, and if data comes in every (let’s say) 20 seconds, the derivative would never be able to be calculated because you would have only one point.

This is the point! Of course, in case no data received during last 20 seconds with 15 sec update interval in terms of approximation definitely means that flow is zero! Otherwise, it is just another realisation of old-known integration statistics sensor.

Yeah it gets to that last value and doesn’t calculate the derivative is zero until one more value is updated. So it gets close to 0 and is trending that way, but that last final step can take hours (however long it takes for the source sensor to update one more time).

It’s the same behavior as with the trend integration, which is where I stole the work around above.

https://community.home-assistant.io/t/add-force-update-support-to-template-sensor/106901/2

I presume that to get a derivative of 0, you need two consecutive values of the same number, but the way HA tends to work until specified otherwise is that it won’t send a second value from a device until the value changes. So the more I think about it, the more it seems the fault of HA as a whole and how derivatives work, and not the code or integration.

@afaucogney already posted :

IMO, there is no issue. You are looking for a perfect derivative mechanism in a sampled world, that’s not possible. Every derivative of a sampled signal is an approximation, because the sampled signal is also an approximation.

I fully agree with @divanikus - the problem is that the function shows value at the current moment but the value is being calculated on the past sensor values. Ir order to automate the solution is the value out-of-date, the integration can use time window parameter.

having the same on my server

TLDR: I propose to update this integration as described in the final chapter of this loooong post.

---

I would like to give my view on the topic, which is based on mathematical insight:

Let me first start with a bit of explanation of how I approached this topic.

How I mathematically approach sensor values in HA

Sensor values are non-uniform (not equi-distant, i.e. not always with equal time between them) samples of the value a real-world “function” f(t) at a specific time. Since sample values are digital numbers, these are an approximation of the real value of the function f(t). Furthermore, consecutive equal values are ignored by HA, meaning we can interpret our samples as a list of pairs (t1,v1),(t2,v2),(t3,v3),..., where the sequence t is increasing, and any value of v is always different from the previous one.

This means that even if the method by which new values are obtained is a polling method with a regular interval, the samples may still be non-uniform because HA throws out equal values. Thus, any method we think of to process data should always assume the data is non-uniform, because even though we might know that data is checked regularly, we never know when the next different value comes in.

Now, based on the sample list (t1,v1),(t2,v2),(t3,v3),... we want to create a “best representation” function g(t) that resembles f(t) as close as possible.

So, how do we obtain a best representation g(t) of the function f(t) from its unique samples? There are multiple ways to do this:

1. The value of g(t) is based on only sample values that are in the past, i.e. only those samples (ti,vi) with ti <= t. We call this a “causal” relation, because every change is caused by and can be computed from only those things that have already happened, for instance:
   • Method last: g(t) is equal to the value of the last recorded (unique) sample.
2. The value of g(t) is based on previous and future values of f(t), for instance:
   • Method linear: g(t) is a straight line between the previous and next (unique) samples.

The problem with any method that would fall in the category of item 2 is that they use future values of f(t), which means that at time t we cannot compute g(t) yet, because we also need at least 1 future sample f(t') for some t'> t, which we cannot know yet.

Let me pose an assumption, without trying to clarify too much why I think it is true:

• Any reconstruction method you can think of that is “causal” and computable from the unique samples is most probably a “causal, linear & time-invariant filter” applied to the reconstruction function g(t) from the method last. This basically means it is a weigted average of the previous values, where the weights depend on how long each sample was active, and how far it is in the past.

The derivative integration

So back to this integration. In my opinion, there is 1 property that would need to be satisfied for this integration to be useful, and that is:

• The “Fundamental theorem of calculus”: Calculate the integral from 0 to x of the derivative f'(t) of a function f(t), and for all x the outcome will be exactly f(x) - f(0), i.e. the original function will come out, although possibly with an offset.
  • So lets apply this to HA: If we apply the Riemann sum integral integration to the output sensor of our Derivative integration, a sensor with the same values as the original sensor should be the output. Note: since we assume g(t) is based on reconstruction method last, the only riemann sum integral method that makes sense is to use left-integration, which is what I will assume from here on.

So, based on all of the above, I can now properly explain why I personally have a couple of “issues” with the current implementation of the Derivative integration:

• If no time_window is configured, the integration calculates a new derivative value at the moment when the next value comes in, from the previous value (t1,v1) and the new value (t2,v2), simply by calculating the rate of change (v2 - v1) / (t2 - t1). Though this is a good calculation to calculate the average derivative in the time window [t1, t2], the problem I have with it is that the integration applies this value at time t2, whereas (assuming we use the reconstruction method last) it actually makes more sense to apply this value at time t1. But that would mean at time t1 the value v2 at time t2 would need to be known, and thus we do not have a “causal” sensor!
• The “Fundamental theorem of calculus” does not apply to this integration:
  • If no time_window is configured, then the following example shows that the property is not guaranteed: Assume a cumulative sensor that has values (0,0),(2,1),(5,4). Then the derivative will calculate the following derivative sensor values: (0,0),(2,0.5),(5,1), which, using left-integration results in the integrated values (0,0),(2,0),(5,1.5), which is not equal to the original. (Note: right-integration would actually result in the original sensor values, but that does not match the assumption that our reconstruction function is based on method last)
  • To make things worse, regardless of whether a time_window is given or not, if no sensor values take place for a long time, the derivative never resets to 0 (which is the main problem of this issue, of course), so taking the example sensor values above, if then the next value of (1005, 5) comes in, the derivative integration will add sample (1000, 0.001), and the riemann sum integral integration will add sample (1005,1.5 + 1 * (1005 - 5))=(1005, 1001.5), which is not even close!

Let me give an example of what the derivative samples should look like in my opinion. Assume the samples are (0,0),(2,1),(5,4),(12,5) and time_window=4. Then the derivative samples should be:

• Initial state 0. Lets denote it as (-inf, 0)
• At time 0: Sample (0,0) enters the time_window. New derivative sample: (0,0).
  • The first sample we receive cannot give us any information, because it is the first sample so it does not yet denote a change since there is no previous value. We shall assume that the original sensor had this value for all past.
• At time 2: Sample (2,1) enters the time_window. New derivative sample: (2,0.25).
  • The second sample we receive has a difference of 1 w.r.t the sensor value time_window seconds ago, so the average derivative is 1/time_window = 1/4 = 0.25.
• At time 4: Sample (0,0) leaves the time_window. New derivative sample: (4,0.25).
  • The first sample is now outside the time window, but due the the last-interpretation, its value is still applicable in the first 2 seconds of the window. This might seem like a strange thing to do, because it is the same value as the previous sample, but below we shall see that it makes sense to update the derivative also once a sample goes outside the time_window, not just when it enters it.
• At time 5: Sample (5,4) enters the time_window. New derivative sample: (5,1).
  • The third sample has a difference of 4 w.r.t. the sensor value time_window seconds ago, which was time 1, i.e. when sample (0,0) was active, this its value was 0, so the derivative should be (4-0)/4 = 1.
• At time 6: Sample (2,1) leaves the time_window. New derivative sample: (6, 0.75).
  • The second sample leaves the time window, so the value at the “end” of the window now changes from 0 to 1, so our derivative changes from (4-0)/4 to (4-1)/4.
• At time 9: Sample (5,4) leaves the time_window. New derivative sample: (9, 0).
  • The third sample leaves the time window. This means no more samples are “inside” the window, and sample (5,4) is active throughout its entirety, so the derivative should become 0.
• At time 12: Sample (12,5) enters the time_window. New derivative sample: (12, 0.25).
• At time 16: Sample (12,5) leaves the time_window. New derivative sample: (16, 0).

So our derivative samples become: (0,0),(2,0.25),(5,1),(6,0.75),(9,0),(12,0.25),(16,0) (we dropped the duplicate (4,0.25), as HA would do).

Now lets ty to left-integrate this using the riemann sum integral integration:

• Sample (0,0): Integral = 0 => sample (0,0)
• Sample (2,0.25): Integral += 2 * 0 => sample (2,0)
• Sample (5,1): Integral += 3*0.25 => sample (5,0.75)
• Sample (6,0.75): Integral += 1*1 => sample (6,1.75)
• Sample (9,0): Integral += 3*0.75 => sample (9,4)
• Sample (12,0.25): Integral += 3*0 => sample (12,4)
• Sample (16,0): Integral += 4*0.25 => sample (16,5)

Wait what?!? That is not even close to the original list of samples! Indeed, but due to the time_window we apply, what we are actually doing is applying a 4-second moving average to the original sensor before calculating its derivative. Suppose we would calculate a 4-second moving average of the original sample list every second:

• Sample (0,0) => sample (0,0)
• Time 1 => sample (1,0)
• Sample (2,1) => sample (2,0)
  • This is because in the previous 4 seconds the average value was 0.
• Time 3 => sample (3, 0.25)
• Time 4 => sample (4, 0.5)
• Sample (5,4) => sample (5,0.75)
  • In the previous 4 seconds the value was 0 for 1 second (in the interval [1,2]) and 1 for 3 seconds (in the interval [2,5]), so the average is (0*1 + 1*3)/4=0.75.
• Time 6 => sample (6,1.75)
  • 1 for 3 seconds and 4 for 1 second, so (3*1 + 1*4)/4=7/4=1.75
• Time 7 => sample (7,2.5)
• Time 8 => sample (8,3.25)
• Time 9 => sample (9,4)
• Time 10 => sample (10,4)
• Time 11 => sample (11,4)
• Sample (12,5) => sample (12,4)
• Time 13 => sample (13,4.25)
• Time 14 => sample (14,4.5)
• Time 15 => sample (15,4.75)
• Time 16 => sample (16,5)

And if we cross-check the times at which the riemann sum integral integration calculated a new value, it matches 100% with the above list.

My proposed update

So, basically I would say that this integration needs an update as follows:

• If no time_window is given, it should default to 1, as otherwise we are creating a non-derivative sensor.
• The integration should trigger on when a new sample arrives and when it leaves the time_window interval. This means the derivative will have 2 times as many state changes as the original sensor, but this comes with the added benefit that it actually makes sense as a derivative!
  • This should be achievable by using the async_call_later function, with a delay of exactly time_window.

@afaucogney Would you be so kind as to read the above rationale and comment on whether you think this is a good improvement of this integration? If so, let me know, I can start working on implementing it on relatively short notice.

Did you try the time_window ? I’m sur this is what you are looking for !

Possibly, it is what I need. I read the manual but It’s not clear how it works. What value should I use for the time_window? Is it a time delta that derivative uses for calculation of increment? In this case I think it’s better to use the time interval that my sensor is being updated (15 sec).

When you say ‘there is no change’ : What is the diff between “no change” and “waiting for the next value”. How can the component know if before getting the new value:

I’m not agree with you, since we are talking about physical conception ‘derivative’ what means the speed of value changing versus time line, no matter what is the reason of such changes, either “no change” or “waiting for the next value”. This is the nature of any derivative. But in case you start to matter regarding the reason of changes you mean statistics but not derivative. Home Assistant already has statistics sensor which does work exactly the way you tell about.

+1

Regarding the original issue @Spirituss described: I had the same problem that the derivative integration didn’t update values when my source sensor values were constant. I use this integration for calculating the power (in kW) based on the energy (kWh) which my energy meters provide. The latter is collected by using the RESTful Sensor integration:

rest:
  - resource: http://192.168.60.11/cm?cmnd=status%2010
    scan_interval: 30
    sensor:
      - name: "Verbrauch Normalstrom"
        state_class: total_increasing
        device_class: energy
        unit_of_measurement: kWh
        value_template: >
          {% set v = value_json.StatusSNS.normal.bezug_kwh %}
          {% if float(v) > 0 -%}
            {{ v }}
          {%- endif %}

sensor:
  - platform: derivative
    source: sensor.verbrauch_normalstrom
    name: "Verbrauch Normalstrom Leistung"
    time_window: "00:03:00"
    unit_time: h
    unit: kW

I guess the value updates didn’t take place because hass didn’t write values of the source sensor in the database. I haven’t verified this, I just took a look at the Prometheus metric hass_last_updated_time_seconds of the source sensor which I collect. As you can see, the source sensor didn’t update for quite some time:

I could fix it by adding force_update: true to the sensor specification of the rest integration. Now it seems the source sensor (sensor.verbrauch_normalstrom) values are updated regularly – even when the value doesn’t change (which can’t be seen on my screenshots):

Just wanted to quickly post this solution in case someone else finds this issue and uses the restful integration. Maybe other integrations provide similar functionality.

@basnijholt @afaucogney I believe that the problem lies in the plotting. Since people usually use such sensors for plotting, it’s highly desirable to see when change stops. It means that if the time window has exceeded and no new values are present, sensor should report 0, null, undef whatever, but not the last value. The statistics integration does this by issuing a timer for time window time and resetting sensor’s value on it’s exceeding.

Maybe you miss configure it, please post your configuration, and the output. Extract of the datatable would also be suitable.

Config:

sensor:
  - name: raw_water_drink_filter_kitchen_flow
    platform: derivative
    source: sensor.raw_water_drink_filter_kitchen
    round: 2
    unit_time: min
    time_window: "00:00:15"

The sensor sensor.raw_water shows nothing for long time:

But derivative sensor which physically means flow is still showing non-zero value:

It is definitely not the problem of the approximation, but the obvious mistake in the calculation algorithm realisation.

If your case doesn’t work with time_widow, feel free to open a PR, we can look on that.

I added time_window to my sensors and nothing has changed. Derivatives show the same value as before.

statistics sensor runs periodically regardless if there are any changes in the sensor. This also means that it doesn’t track changes during the period.

derivate sensor (and integration sensor in which it is based) tracks changes in the source sensor. That means that if the source sensor doesn’t change, the derivate sensor will keep it’s value for long periods of time.

One possible solution is to combine both methods: track changes and periodically read the source sensor to detect “no changes”.