core: Nest SDM Camera stops streaming

The problem

After running for a couple of minutes ~ 5- 10, the stream of all my Nest cameras stops with the following error, running the latest PR https://github.com/home-assistant/core/pull/42556 .

Environment

arch	x86_64
chassis	desktop
dev	false
docker	true
docker_version	19.03.12
hassio	true
host_os	Ubuntu 18.04.4 LTS
installation_type	Home Assistant Supervised
os_name	Linux
os_version	4.15.0-115-generic
python_version	3.8.6
supervisor	2020.10.1
timezone	Europe/Amsterdam
version	0.117.1
virtualenv	false

Home Assistant Core release with the issue: 0.117.1
Last working Home Assistant Core release (if known):
Operating environment (OS/Container/Supervised/Core): Supervised
Integration causing this issue: Nest (SDM)
Link to integration documentation on our website: https://www.home-assistant.io/integrations/nest/

Problem-relevant `configuration.yaml`

Traceback/Error logs

2020-11-03 17:53:00 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.139645789842976] Error handling message: Unknown error
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 18, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 592, in ws_camera_stream
    source = await camera.stream_source()
  File "/config/custom_components/nest/camera_sdm.py", line 102, in stream_source
    new_stream = await self._stream.extend_rtsp_stream()
  File "/usr/local/lib/python3.8/site-packages/google_nest_sdm/camera_traits.py", line 86, in extend_rtsp_stream
    resp.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 941, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://smartdevicemanagement.googleapis.com/v1/enterprises/xxxxxxxx-63f6-46a0-8c2a-7c8890d13546/devices/xxxxxx57JXge-Zwpzf8fXdMsTtpnv_smNTA_8sytnm6xc9J2Qq0FvSxxxxxxxxzwc_HVy4IIkj0hvIgMxe2V:executeCommand')

2020-11-03 17:54:54 ERROR (stream_worker) [homeassistant.components.stream.worker] Stream connection failed: rtsps://stream-eu1-delta.dropcam.com:443/sdm_live_stream/xxxxxxxxxJxxoGXx2i97HFm10RyLNycYeWisB4bgScQBMPhen1Y6rE4eUnpMXNpkXxw-nRtxEi7vii1DjInFKgBjta8y4rcz6gduoAFSRmMFPf08QbW5rue9E7LjE2tFyLwv8SVa_XJWEwEu7pZejd7FFTseWD7aHzi2DKolnqKMGryWOXhYfmgr4cJAGKRBn?auth=g.0.eyJraWQiOiIyMzhiNTUxZmMyM2EyM2Y4M2E2ZTE3MmJjZTg0YmU3ZjgxMzAzMmM4IiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJuZXN0X2lkOm5lc3QtcGhvZW5peC1wcm9kOjEyNDk1NjM0IiwiaXNzIjoibmVzdC1zZWN1cml0eS1hdXRocHJveHkiLCJwb2wiOiIzcC1vYXV0aC1zY29wZS1BUElfU0RNX1NFUlZJQ0UtY2xpZW50LTc4NDQ2NDI5MzkxMC1jaWI3cWI2NGdrZjBzbWRhMXU1NWNlM29rMGIwbG1qZS5hcHBzLmdvb2dsZXVzZXJjb250ZW50LmNvbSIsImV4cCI6MTYwNDQyMjM3M30.EIulM7xELUk7LrrLnu9p_bZ3RC2WzcflCWBodwW-d9MA5ofOHtn9kDFWXjlxvTLcZ7--LsSBE41ejDp1aEFXlHzJeuAQdt-aqVliguz0qbCv4_hOcEDIJPJmvy2tAfu75bQhKxTMlgAX3z9qjGfIOXv64O3P4DCCst0lYYAsnk0Q6kFJKd3mMY7dU5Ml8AuHLza9wA8XgqNd_t8xf0ow1Ze0CLSbz9CP4y8Ry2KDGqhA5l4NshjTYnypNHhjZVb1boS2b_b06onbwi58coj_aAAq1CIheIsERocU8tlmkE1Vsa_YzikJNPyZJLJS6isuMMnefJFAyGbvzWeX7Im41w
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 54, in stream_worker
    _stream_worker_internal(hass, stream, quit_event)
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 71, in _stream_worker_internal
    container = av.open(stream.source, options=stream.options, timeout=STREAM_TIMEOUT)
  File "av/container/core.pyx", line 354, in av.container.core.open
  File "av/container/core.pyx", line 225, in av.container.core.Container.__cinit__
  File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.HTTPNotFoundError: [Errno 875574520] Server returned 404 Not Found: 'rtsps://stream-eu1-delta.dropcam.com:443/sdm_live_stream/xxxxxxxxxuo3JxxoGXx2i97HFm10RyLNycYeWisB4bgScQBMPhen1Y6rE4eUnpMXNpkXxw-nRtxEi7vii1DjInFKgBjta8y4rcz6gduoAFSRmMFPf08QbW5rue9E7LjE2tFyLwv8SVa_XJWEwEu7pZejd7FFTseWD7aHzi2DKolnqKMGryWOXhYfmgr4cJAGKRBn?auth=g.0.eyJraWQiOiIyMzhiNTUxZmMyM2EyM2Y4M2E2ZTE3MmJjZTg0YmU3ZjgxMzAzMmM4IiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJuZXN0X2lkOm5lc3QtcGhvZW5peC1wcm9kOjEyNDk1NjM0IiwiaXNzIjoibmVzdC1zZWN1cml0eS1hdXRocHJveHkiLCJwb2wiOiIzcC1vYXV0aC1zY29wZS1BUElfU0RNX1NFUlZJQ0UtY2xpZW50LTc4NDQ2NDI5MzkxMC1jaWI3cWI2NGdrZjBzbWRhMXU1NWNlM29rMGIwbG1qZS5hcHBzLmdvb2dsZXVzZXJjb250ZW50LmNvbSIsImV4cCI6MTYwNDQyMjM3M30.EIulM7xELUk7LrrLnu9p_bZ3RC2WzcflCWBodwW-d9MA5ofOHtn9kDFWXjlxvTLcZ7--LsSBE41ejDp1aEFXlHzJeuAQdt-aqVliguz0qbCv4_hOcEDIJPJmvy2tAfu75bQhKxTMlgAX3z9qjGfIOXv64O3P4DCCst0lYYAsnk0Q6kFJKd3mMY7dU5Ml8AuHLza9wA8XgqNd_t8xf0ow1Ze0CLSbz9CP4y8Ry2KDGqhA5l4NshjTYnypNHhjZVb1boS2b_b06onbwi58coj_aAAq1CIheIsERocU8tlmkE1Vsa_YzikJNPyZJLJS6isuMMnefJFAyGbvzWeX7Im41w'; last error log: [rtsp] method DESCRIBE failed: 404 Not Found
2020-11-03 17:54:55 ERROR (stream_worker) [homeassistant.components.stream.worker] Stream connection failed: rtsps://stream-eu1-delta.dropcam.com:443/sdm_live_stream/xxxxxxxxxxO_3yn6nA07oVKjrfUjzkaupzzvSsEZreKPsScQBMPhenxypiOaMWMUKRDX0Gvea6EaYi9oYEwmRFeZEbReuVoOmqREI-Js6SOzxTL3AIumPtIdCwSqxKEgzzM64LicyKX8QIOXrNH1EBo1rXR6DbSH6qx7GX4LE2SxKOe0FPMOPavlYp0SP5FNIDXuhO?auth=g.0.eyJraWQiOiIyMzhiNTUxZmMyM2EyM2Y4M2E2ZTE3MmJjZTg0YmU3ZjgxMzAzMmM4IiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJuZXN0X2lkOm5lc3QtcGhvZW5peC1wcm9kOjEyNDk1NjM0IiwiaXNzIjoibmVzdC1zZWN1cml0eS1hdXRocHJveHkiLCJwb2wiOiIzcC1vYXV0aC1zY29wZS1BUElfU0RNX1NFUlZJQ0UtY2xpZW50LTc4NDQ2NDI5MzkxMC1jaWI3cWI2NGdrZjBzbWRhMXU1NWNlM29rMGIwbG1qZS5hcHBzLmdvb2dsZXVzZXJjb250ZW50LmNvbSIsImV4cCI6MTYwNDQyMjM3NH0.C3Ph72FvmxI352o2sRB7XYGxGxBbN-UWWf69pV2P3T8wOj3FpVJPIHpKM3K9IcHyi_Dn1n4BABkiZ-4e8D34lgY2DTDhJg3Ai55kEPMQCmZXfewiSYJojVhDwuBea65BbmXBHGOKK6e4WTjT9gwQqWiC84zxoIb60T1fdvi0f4tpE0sL2I1ZHrhSUUAfT58yGM3OHscPalCd9SvnrfL4uQkfVnEH0aOP81aLzqfCIfdchpZJdT9G2ZA3Xf5BcsB5uqCgW_H3F5sFNQpY-bSVeRB67wpx39eQJhDuJCI3hAOr4w7OailF8Z6SSbgd_NxDXhK8I4uj7PWwUUZmR3tYxA
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 54, in stream_worker
    _stream_worker_internal(hass, stream, quit_event)
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 71, in _stream_worker_internal
    container = av.open(stream.source, options=stream.options, timeout=STREAM_TIMEOUT)
  File "av/container/core.pyx", line 354, in av.container.core.open
  File "av/container/core.pyx", line 225, in av.container.core.Container.__cinit__
  File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.HTTPNotFoundError: [Errno 875574520] Server returned 404 Not Found: 'rtsps://stream-eu1-delta.dropcam.com:443/sdm_live_stream/xxxxxxxxxxx6nA07oVKjrfUjzkaupzzvSsEZreKPsScQBMPhenxypiOaMWMUKRDX0Gvea6EaYi9oYEwmRFeZEbReuVoOmqREI-Js6SOzxTL3AIumPtIdCwSqxKEgzzM64LicyKX8QIOXrNH1EBo1rXR6DbSH6qx7GX4LE2SxKOe0FPMOPavlYp0SP5FNIDXuhO?auth=g.0.eyJraWQiOiIyMzhiNTUxZmMyM2EyM2Y4M2E2ZTE3MmJjZTg0YmU3ZjgxMzAzMmM4IiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJuZXN0X2lkOm5lc3QtcGhvZW5peC1wcm9kOjEyNDk1NjM0IiwiaXNzIjoibmVzdC1zZWN1cml0eS1hdXRocHJveHkiLCJwb2wiOiIzcC1vYXV0aC1zY29wZS1BUElfU0RNX1NFUlZJQ0UtY2xpZW50LTc4NDQ2NDI5MzkxMC1jaWI3cWI2NGdrZjBzbWRhMXU1NWNlM29rMGIwbG1qZS5hcHBzLmdvb2dsZXVzZXJjb250ZW50LmNvbSIsImV4cCI6MTYwNDQyMjM3NH0.C3Ph72FvmxI352o2sRB7XYGxGxBbN-UWWf69pV2P3T8wOj3FpVJPIHpKM3K9IcHyi_Dn1n4BABkiZ-4e8D34lgY2DTDhJg3Ai55kEPMQCmZXfewiSYJojVhDwuBea65BbmXBHGOKK6e4WTjT9gwQqWiC84zxoIb60T1fdvi0f4tpE0sL2I1ZHrhSUUAfT58yGM3OHscPalCd9SvnrfL4uQkfVnEH0aOP81aLzqfCIfdchpZJdT9G2ZA3Xf5BcsB5uqCgW_H3F5sFNQpY-bSVeRB67wpx39eQJhDuJCI3hAOr4w7OailF8Z6SSbgd_NxDXhK8I4uj7PWwUUZmR3tYxA'; last error log: [rtsp] method DESCRIBE failed: 404 Not Found
2020-11-03 17:54:56 ERROR (stream_worker) [libav.rtsp] method DESCRIBE failed: 404 Not Found
2020-11-03 17:54:56 ERROR (stream_worker) [libav.rtsp] method DESCRIBE failed: 404 Not Found

Additional information

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 57 (23 by maintainers)

Most upvoted comments

@nickd70s thanks! I’ll use your poke as an excuse to give a short update on the plan:

Improve test coverage for the stream worker - In progress: https://github.com/home-assistant/core/pull/44161
Change camera / stream interaction to support a callback to fetch the latest URL - In progress: https://github.com/home-assistant/core/pull/43718
Refactor the stream worker to make it easier to hold state in the future - Paused, waiting on above PRs: https://github.com/home-assistant/core/pull/44000
Update stream worker to restart on url changes, and preserve PTS / DTS - Not started

Overall reviews are expected to go a bit slow over the holidays, but rest assured this is still top of mind. Cheers!

allenporter on Dec 21, 2020

@uvjustin I would love your quick thoughts on how to make nest place nicely with the stream component, and discuss some potential options. Is that something you could give me some advice on?

Background: The nest API has an RPC that generates a streaming url (contains an ?auth= param), and another RPC that extends the lifetime of the stream giving a new ?auth= param to use instead. That is, the stream is valid for 5 minutes and it needs to use a new url.

This is the logic in the nest integration for refreshing the stream source

Problem: I don’t believe the existing camera + stream API expects this behavior, where a nest cam stream url changes over time. Once a stream is started, it uses a fixed stream url, and will start serving errors after awhile. Additionally, new calls to view the nest cam stream end up creating additional streams since the stream url itself is used as a cache key.

When thinking about possible solutions, I have a couple suggestions to help start the conversation, but very flexible and open to better ideas:

Support passing a callback into the stream to get back either the current url (or current set of url parameters to replace)
Support a different unique id in the stream cache, either based on something passed in like an entity id or the base of the url without an auth parameter.
Have an explicit timestamp of when the stream should expire that the stream worker is aware of. Could explicitly restart the stream, rather than waiting for an authentication failure
It probably also needs to consider not backing off on retry, either by looking for a specific error code, or because it knows it explicitly hit the expiration time

allenporter on Nov 28, 2020

i also see a lot of : ERROR (stream_worker) [homeassistant.components.stream.worker] Error opening stream rtsps://stream-eu1-delta.dropcam.com:443/

is that related to this issue here?

pergolafabio on Dec 15, 2020

This is really strange because it kind of works. There are valid credentials to get streams and new stream urls, and some event messages are being received. Yet some event messages are serving errors and fetching the streams themselves sometimes fails.

My best idea is to focus on the subscriber side errors for now, and try to add some more instrumentation and debugging.

allenporter on Nov 14, 2020