core: TOD incorrect after midnight restart

@ahaverty

Does anyone have a workaround or alternative integration that follows the sun? I’m hitting this regularly, as I’m doing a good bit of late night tweaking recently.

  - binary_sensor:
    # tod workaround
    - name: "Night"
      unique_id: tod_night
      state: >
        {% set cur_sunset = iif(state_attr("sun.sun", "next_setting")>state_attr("sun.sun", "next_rising")
        ,as_datetime(state_attr("sun.sun", "next_setting"))-timedelta(days=1)
        ,as_datetime(state_attr("sun.sun", "next_setting")))%}

        {% if 
        now()>cur_sunset
        and now()<as_datetime(state_attr("sun.sun", "next_rising"))
        -%}
          on
        {%- else -%}
          off
        {%- endif %}

I conducted some tests, and confirm the issue arises when the binary sensor’s state was set to ‘on’ the previous day. Performing a restart on the same day does not alter the state, but doing so the following day resets it to its default state, ‘off’. My solution was to define a Day sensor instead Night sensor:

binary_sensor:
  - platform: tod
    name: Trevina Dia
    before: sunset
    before_offset: "00:10:00"
    after: sunrise
    after_offset: "-00:10:00"

@ahaverty I tried your workaround this night. It also works on my HA instance 😄. Here’s the comparison:

I made a restart at 00:30 in the morning. You can see that the workaround sensor stays on after the restart and the original one doesn’t.

However I did a slight modification to your approach. I did not specify a delay_off of 2 minutes in the template sensor. Instead I have the right tod sensor trigger at 00:00 with an offset of -00:02:00. I think that makes it a bit easier. Here’s my config:

binary_sensor:
  - platform: tod
    name: Jalousien Nachtmodus Rechts
    after: sunset
    after_offset: "00:35:00"
    before: "23:59"
  - platform: tod
    name: Jalousien Nachtmodus Links
    after: "00:00"
    after_offset: "-00:02"
    before: sunrise
    before_offset: "-00:35:00"

template:
  - binary_sensor:
      - name: "Jalousien Nachtmodus workaround"
        unique_id: template_binary_Sensor_jalousien_nachtmodus_workaround_unique_id
        state: >
          {{ (is_state('binary_sensor.jalousien_nachtmodus_links', 'on') or is_state('binary_sensor.jalousien_nachtmodus_rechts', 'on')) }}

PS: I named my sensors wrong and confounded left with right, but it still works, lol.

@zzachattack2 I agree that the non-functioning part of tod should be removed, but how to do that with no code maintainer for it? Also, there is the problem that tod might break completely in the future if there are any changes to core functionality it relies on.

@ahaverty Thanks for that workaround, it’s the easiest to setup I have seen so far. Will try it out too.

Very untested, but I think I’ve come up with a decent workaround for the likes of a “night” sensor that relies on sunrise and sunset:

binary_sensor:
  - platform: tod
    name: left_night
    after: sunset
    before: "23:59"

  - platform: tod
    name: right_night
    after: "00:00"
    before: sunrise
    before_offset: "-00:02"

template:
    - binary_sensor:
      - name: "Night"
        unique_id: night
        delay_off:
          minutes: 2
        state: >
          {{ (is_state('binary_sensor.left_night', 'on') or is_state('binary_sensor.right_night', 'on')) }}

i.e: The left_night and right_night I don’t intend to use anywhere other than in the template’s state for binary_sensor.night above.

I’ve tested restarting just after midnight, and see binary_sensor.right_night is set to ‘on’ as expected. I’ve gone with 23:59 to be safe, but added a delay_off of 2 minutes so that the night sensor doesn’t actually change to ‘off’ too early or trigger any automations just before midnight.

And the -2min before_offset is to counteract that delay_off. So it should function exactly like the standard night sensor, but without the nightime restart issues.

I just wanted to confirm that the bug is still present in the newest version 2023.2.0, so no need for @ahaverty to downgrade your testing version.

Here’s the result from a scheduled restart at 00h30 AM on the prior version 2023.1.7:

The same happens with a scheduled restart at 00h30 AM on version 2023.2.0 :

In both instances, the sensor was only turned on until 00h30 AM in the morning and it turned off after the restart

I have more information listed in the 3rd comment on this but I will include my config again here. I’m US-based with a US timezone currently running on a raspberry pi 4 and do indeed have this issue, so I don’t think being non-US is a cause. I had this issue on a pi 3 as well.

binary_sensor:
  - platform: tod
    name: Night
    after: sunset
    before: sunrise

@wafliron to add to this, I’m running Home Assistant OS on a raspberry pi 4 while also experiencing the issue.

@ahaverty FWIW I experience this issue in Australia too - reliably reproducible with an after-midnight HA reboot. Another data point towards it potentially being something to do with non-US timezones - or perhaps non-US locales/languages?

Random thought, but possible it’s got something to do with the environment HA is running in too? I’m using Home Assistant Operating System running as an ESXi virtual machine (think I downloaded an OVF from somewhere and used that, but so long ago I don’t remember properly).

Looking at your unit-tests everything seems to be in-line with the other unit-tests. Maybe there is something odd with the force stop and restart where it initializes itself with a different time when restarting. Maybe you should set the time again after the restart. This is just a wild guess and I’m not that great at programming in the first place

Edit: Stupid suggestion from me with setting the time again after a restart. This should have no effect because the init during restart is what causes the problem. But still… is it using the correct testing time after the restart?

Thanks @frenck 👍

For anyone else trying to fix this: I took a stab at it, but unfortunately, I can’t even reproduce the issue in unit tests.

As far as I understood, my test case here should fail after the HA restart, based on what I’ve experienced with my actual HA instance restarting just after midnight: https://github.com/ahaverty/core/commit/ecb5353025a9bc14cac41594796406801db9b362

I did see there was a recent cleanup of TOD in #79412, so I tested against an older version of HA (an Oct 2022 release) just in case that PR has already fixed this issue and we haven’t noticed since, but I still couldn’t get my test to fail on an older Oct release.

My theory on when the issue occurs (and what I was trying to fail in unit tests):

• When a sun-based TOD sensor starts on the day previous to a restart, but ends after the restart I’ve found that sun-based TOD sensors work fine for restarts during the day, for configurations like ‘daytime’, so my guess is there’s something not being picked up during initialisation of the sensors after a restart. I’m just unfortunately not knowledgeable enough on HA to pinpoint/reproduce it here in code.

My theories on why I can’t get the test to fail:

• I’m misunderstanding what hass.async_stop & hass.async_start actually does
  • Perhaps the async stop/start isn’t a realistic replica of a real HA reboot.
• This restart issue is more complex to reproduce than I expected (I did also include offsets in my test, just in case that’s required for this issue, but the test still passes with them)

EDIT: No, I have an obvious issue with a couple of things with my test case above. It looks like the suite is setup for sunsets happening before midnight for some reason. ~I’m now able to fail the test when I change the timezone and location to Ireland, against the Oct release. I’ll look at testing against dev branch next, and see if I can add a fix also (i.e the hard part 😬)~

EDIT 2: yes, there’s something odd going on with this test suite if I’m reading this correctly. The timezone is set to the US, but the lat/lng is in Poland. So the sunrise/sunsets aren’t going to be realistic at all 🤔

EDIT 3: I’m still unable to reproduce this in unit tests unfortunately. As far as I can tell, I’m calling the correct restart methods. I’ve also fixed the suite’s lat/lng to match the US Pacific timezone, so that sunrise/sunset happen on different days (as typically expected for most of the world). If anyone else wants to try fix, perhaps have a look at my attempt first: https://github.com/home-assistant/core/compare/dev...ahaverty:core:fix/TOD-incorrect-after-midnight-restart

EDIT 4: Nope, I’ve gone back to an Oct '22 release again with my latest test fixes, and can’t reproduce the issue in unit tests. I’ve also tried:

• Testing with my exact TOD configuration, with multiple TOD sensors (below).
• Starting the hass async_stop before midnight, jumping to after midnight, and starting hass again

I’m stumped…

My TOD config for reference:

config = {
        "binary_sensor": [
            {
                "platform": "tod",
                "name": "Night",
                "after": "sunset",
                "before": "sunrise",
            },
            {
                "platform": "tod",
                "name": "Day",
                "after": "sunrise",
                "before": "sunset",
            },
            {
                "platform": "tod",
                "name": "Evening",
                "after": "sunset",
                "after_offset": "-00:30",
                "before": "00:30",
            },
            {
                "platform": "tod",
                "name": "Outdoor Evening",
                "after": "sunset",
                "before": "23:30",
            },
            {
                "platform": "tod",
                "name": "Day",
                "after": "sunrise",
                "before": "sunset",
            },
            {
                "platform": "tod",
                "name": "Early Morning",
                "after": "sunrise",
                "before": "sunrise",
                "before_offset": "02:00"
            }
        ]
    }

Issue is still present on Home Assistant 2023.1.2. I had an unforeseen restart in the middle of the night and all my sunbased tod-sensors simply turned off. I included a pic of the tod-sensor handling my covers. All my covers stayed closed this morning instead of reacting to sunrise.

@DmytroKorniienko thanks for sharing! I’m stuck though with some more complex TOD’s that are important to me for outdoor light automations. I had a go at modifying your code for evening for example, but couldn’t find a way to get the after_offset working properly.

after_offset requires more complex logic and I’m not implemented it for my automations, but can share one more example:

  - binary_sensor:
    # from 21:00 to 5 minutes before next_rising
    - name: "Light Control"
      unique_id: tod_light_control
      state: >
        {% set cur_sunset = iif(state_attr("sun.sun", "next_setting")>state_attr("sun.sun", "next_rising")
        ,as_datetime(state_attr("sun.sun", "next_setting"))-timedelta(days=1)
        ,as_datetime(state_attr("sun.sun", "next_setting")))%}
        {% set cur_start = strptime(as_timestamp(cur_sunset) | timestamp_custom('%Y-%m-%d 00:00:00%z'),'%Y-%m-%d %H:%M:%S%z') %}

        {% if 
        now()>cur_start+timedelta(hours=21, minutes=00)
        and now()<as_datetime(state_attr("sun.sun", "next_rising"))-timedelta(hours=0, minutes=5)
        -%}
          on
        {%- else -%}
          off
        {%- endif %}

The problem is still present with version 2022.12.8. My home assistant instance suffers from regular unplanned restarts during the night, the cause of which I haven’t found yet. This would not really be an issue if it weren’t for all my time of day sensors having a wrong state after the restart. Is there really no fix for this?