alerta: Expired heartbeats don't create alerts

Issue Summary

Running alerta heartbeats --alert does not create any alerts even if there are expired heartbeats available. Partial output looks like this:

ID                                    ORIGIN         CUSTOMER    TAGS    ATTRIBUTES                  CREATED              RECEIVED             SINCE    TIMEOUT    LATENCY    MAX LATENCY    STATUS
------------------------------------  -------------  ----------  ------  --------------------------  -------------------  -------------------  -------  ---------  ---------  -------------  --------
b006e2f8-7ac4-48de-a20f-1fc722042d2f  fimfcn1        Paragon             {'environment': 'paragon'}  2020/06/23 08:57:03  2020/06/23 08:57:03  0:00:41  120s       0ms        2000ms         ok
ec270a57-f6bf-47bc-a626-3a44b91bfe26  jhworkstation  Paragon             {'environment': 'paragon'}  2020/06/23 08:18:03  2020/06/23 08:18:03  0:39:41  120s       0ms        2000ms         expired
b7bc3db2-d8a0-41f7-a270-ece4a0548eb1  ps1            Paragon             {'environment': 'paragon'}  2020/06/23 08:57:02  2020/06/23 08:57:02  0:00:43  120s       0ms        2000ms         ok
a219e7e6-b117-49eb-970e-3f5832f08328  pcweb2         Paragon             {'environment': 'paragon'}  2020/06/23 08:57:01  2020/06/23 08:57:01  0:00:43  120s       0ms        2000ms         ok
....
....
Alerting 37 heartbeats  [####################################]  100%

Environment

OS: Ubuntu 18.04 LTS
API version: 8.0.0
Deployment: docker
Database: latest PostGres docker image
Server config: Auth enabled? Yes Auth provider? Gitlab Customer views? Yes
web UI version: 8.0.0
CLI version: 7.5.6

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 35 (17 by maintainers)

Most upvoted comments

Probably found the issue:

                if b.status == 'expired':  # aka. "stale"
                    severity = b.attributes.pop('severity', severity)
                    client.send_alert()
                elif b.status == 'slow':
                    severity = b.attributes.pop('severity', severity)
                    client.send_alert()
                else:
                    severity = b.attributes.pop('severity', default_normal_severity)
                    client.send_alert()

The variable name severity is used twice: as a function parameter ANA locally. The loop above goes through all the OK heartbeats and for the first one, the severity is set to default_normal_severity, which is normal. When it then goes into the first branch for the expired heartbeat which comes later in the loop, the severity defaults to severity which meanwhile got set to normal.

Looks like using different variable names for the function argument and the local variable should fix this.

jurgenhaas on Jun 28, 2020

Ok. Sounds like bug. Will investigate. Thanks.

satterly on Jun 27, 2020