core: Utility Meter doesn't account for HA downtime

FWIW. Anyone still reading this thread.

A year later, with some additional wisdom and experience. I do not recommend using my alternative approach. Although it does achieve its intention, it has its own set of drawbacks. More importantly. @dgomes 's point of view is the RIGHT way to approach this. The “limitations” of the utility_meter is based on flawed data, not a flawed component. If we were to try and address every “bad data” scenario, we’d have bloated code in EVERY component.

There are GOOD ways of creating a buffer component that will “fix” the data before the utility meter does its job.

@dgomes

No problem. Thx for the tutoring us monkeys.

Anyone who would like an less than optimal “utility_meter” I will refine and maintain the code here: https://github.com/tiaanv/hass-utility_meter_alternative

@dgomes

After spending a bit of time in the code, and looking at how it works, I have now come to the conclusion that the problem might actually be bigger. A component not having an old_state is remediable, as you mentioned by addressing the functionality of the sensor in it’s code, and should definitely be addressed in that way.

The biggest problem with the way it is written now, is that it expects well behaved data ALL THE TIME. This simply cannot be guaranteed, because even if the code for a sensor (e.g. mqtt) lies within control of HA devs, the actual data that feeds it does NOT.

It is a BAD idea to calculate a diff on each sensor update, and add that to the total.

If we were to have a single bad/out of sequence value, greater that the current value, the utility meter will not only calculate and report an incorrect total, it will also stop working from that point onward. Whilst I can agree, that the sensor that feeds this BAD data is to blame, I firmly believe in developing robust code as far as possible, and I am of the opinion this is not, as has been proven.

The utility_meter component, as it is, can surely work for some cases, where the sensor input data is robust, but I, for one, can’t use it, since I don’t have control of the input sensor data, and it does sometimes misbehave. Pity.

I think a more robust method is to “snapshot” the opening balance form the source, (on init, or after reset), and simple set the state to the difference between that snapshot, and the current value. this would make it rather immune to a single misbehaving value. I have modified the existing meter code for my own use, and am testing it as a custom component now. If it proves to be more reliable, I will propose this as an alternative in the future.

Final note. I appreciate the level of commitment and effort that you and the rest of the contributors are doing. The mere fact that there is a utility_meter component is super awesome. Thank you for even taking the time to respond to these comments. So often the users (including myself) sit on the other side, and have this EXPECTATION. Mine is rather admiration!

Yes, I noticed that zip file but a dedicated git for just that component unzipped might be better for the community and future updates?

edit: I had to scroll even more up. found your git page:

Anyone who would like an less than optimal “utility_meter” I will refine and maintain the code here: https://github.com/tiaanv/hass-utility_meter_alternative

@dgomes I am using the utility meter extensively, and had an issue with one of them not adding up values quite correctly, and came across this thread.

I fully understand the operation of the utility meter, and think as it stands it is perfectly functional, however I do tend to agree with some comments above. The biggest issue being that the way the meter works at the moment that any “glitch” in the source sensor (old_state vs new_state) will render useless data in the meter.

I took a look at the source for the utility meter sensor.py

diff = Decimal(new_state.state) - Decimal(old_state.state) could potentially look something like: diff = Decimal(new_state.state) - Decimal(self._last_source_state) where last_source_state is captured after the source is read

The sensor would have a much wider use-case if it did not depend on the old_state value of the source to calculate the diff. Perhaps a potential enhancement would be to maintain the “old_state” internally, and use that to calculate the diff. That way the dependency on the source to have old_state is removed.

It seems to me that this would have virtually no negative effect, and satisfy the requirements above.

I am happy to contribute to the project, from a development perspective, and will do so in time, but I am still noob in HASS terms, and would like to get to know a bit more…

I think this can fall in the “enhancement” category rather than “bug” @tollerm @bouwew

I clearly miss the point because I do not expect that my power consumption is zero when my sensor is offline 😃 What I expect is that the utility-meter simply subtracts from the current value of a sensor the value of the same sensor at some specified time. And that specified time is changed every day for daily measurements and every month for monthly measurements. In current implementation the utility-meter is doing something completely different. As result it produces wrong values if there was not communication with the source sensor, while the information is not lost, because the source sensor integrates continuously the consumed property such as power, volume, mass etc. Thus the utility-meter is not useful for me.

Today my Pi that collects the data from the DSMR sensor was offline during 4 hours and as result of this accident the utility-meter lost 5 kWh of the energy. Trying to find the reason for this phenomenon, I spent a few hours digging in my configuration file and python code of this component, and finally arrived to this conversation. If I understood this thread correctly, the utility-meter is doing the following: It differentiates the integral of a function using the saved state, than integrates again to get this integral. Clearly, with this algorithm it can work correctly only if the input function is continuous and differentiable that is not the case. It is entirely possible to miss short pulses of high energy consumption. That’s why all energy consumption meters have analog integrators! In my opinion the algorithm used in the utility meter is completely wrong and need to be changed.

@bouwew I think your missing the point that the dsmr.py file is part of homeassistant therefor we are in control of changing it and implementing the restorestate function. As you already mentioned, we should tackle this issue at the source and the dsmr.py is our source in this case. Instead of changing this utility_meter, which should just record specific values, we should improve the dsmr python script which read and stores values from our DSMR (correct me if I’m wrong @dgomes .

In any case, I will contact you soon via discord @dgomes

Thank you for your feedback, I always enjoy a technical discussion 😃

I will just give you my general thought about this topic: if problems arise, they need to be solved at the source. Anything you do “'after” will only cause a mess. You should not try to solve the problems a source is having in the Utility-Meter source-code. If you implement a fix for one, you will break something for another.

About your comments:

• “it can only work when the monitored sensor is always there” - on the contrary, if the sensor is not there it will only loose a small amount of information, not the full period as your proposed solution.

I understand what you want to achieve but I don’t like the result: inaccurate sensor values. The method I propose can cause the loss of one full period but the chance this happens is very small. You could find a way in the middle by keeping the logging ongoing and use the “best” reference value: either the value stored at the end of the period or either the first available value since the end of period. If no trouble happens at one “end of period”, the value provided by the utility-meter sensor is always correct, when the monitored sensor provides data. If something happens during the period, that data will be lost but the utility-meter sensor will still show the correct value once the sensor returns. On the odd chance that something happens during the end of period, the utility-meter sensor can “recover” by using a first obtained value from the monitored sensor when the "something happens"is over.

If you implement the algorithm like this, you will fix the error reported in this topic, you will make me happy and you will end up with the best of both worlds and an improved sensor, in my opinion 😉

To continue our discussion in this topic: https://community.home-assistant.io/t/utility-meter-keep-values-after-restart/97712/26

As I see it, the algorithm you describe in the above comment is not the best choice, it can only work when the monitored sensor is always there. When it disappears for a while, for whatever reason, the utility-meter sensor-values will be off/wrong. I think the algorithm of the Utility-Meter sensor should be like this: it must save (to a non-volatile storage medium) the value of the monitored sensor at the end of each period (daily, weekly, monthly, yearly), that is the reference value that then can be used to determine the value of the utility-meter sensor (= present value of monitored sensor - saved value of monitored sensor at end of last period). Nothing else needs to be saved, only this reference value. This save-action must always be successful, so the server should not be offline/rebooting, etc during the end of the chosen monitor period(s). This is the responsibility of the user. When anything happens in-between, it does not matter because the utility-meter sensor can always calculate the correct value when the monitored sensor reappears, because the reference-value is still there.

Do you understand what I mean?