manageiq: Events don't show on Middleware Provider Timeline

Description

Hello,

I’ve been trying to see Middleware Providers Events on Timeline and I couldn’t see any. Although everything seems to be fine on Hawkular Services side. Group triggers, Group member, Events are being generated fine and connection between the MIQ and Provider seems to be fine as well.

Environment

ManageIQ Docker instance using docker:latest
Hawkular Services instance usign hawkular-services:latest

Samples of Logs

Event Sample :

[ { "eventType": "EVENT", "tenantId": "hawkular", "id": "MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1-1502156325254-4c581377-1c55-4474-8f72-4d1f31f56d1f", "ctime": 1502156325254, "dataSource": "_none_", "dataId": "MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1", "category": "TRIGGER", "text": "Test-Alert1-Instance1", "context": { "dataId.hm.prefix": "hm_g_", "dataId.hm.type": "gauge", "miq.alert_profiles": "22", "resource_path": "/t;hawkular/f;7402c000-6df6-46ae-9e79-9b4f71aa0ce4/r;EAP7-Standalone~~" }, "tags": { "miq.event_type": "hawkular_alert", "miq.resource_type": "MiddlewareServer" }, "trigger": { "tenantId": "hawkular", "id": "MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1", "name": "Test-Alert1-Instance1 for EAP7-Standalone", "description": "Test-Alert1-Instance1", "type": "MEMBER", "eventType": "EVENT", "eventCategory": null, "eventText": null, "severity": "MEDIUM", "context": { "dataId.hm.prefix": "hm_g_", "dataId.hm.type": "gauge", "miq.alert_profiles": "22", "resource_path": "/t;hawkular/f;7402c000-6df6-46ae-9e79-9b4f71aa0ce4/r;EAP7-Standalone~~" }, "tags": { "miq.event_type": "hawkular_alert", "miq.resource_type": "MiddlewareServer" }, "autoDisable": false, "autoEnable": false, "autoResolve": false, "autoResolveAlerts": true, "autoResolveMatch": "ALL", "dataIdMap": { "WildFly Memory Metrics~Heap Max": "hm_g_MI~R~[7402c000-6df6-46ae-9e79-9b4f71aa0ce4/EAP7-Standalone~~]~MT~WildFly Memory Metrics~Heap Max", "WildFly Memory Metrics~Heap Used": "hm_g_MI~R~[7402c000-6df6-46ae-9e79-9b4f71aa0ce4/EAP7-Standalone~~]~MT~WildFly Memory Metrics~Heap Used" }, "memberOf": "MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28", "enabled": true, "firingMatch": "ANY", "source": "_none_" }, "dampening": { "tenantId": "hawkular", "triggerId": "MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1", "triggerMode": "FIRING", "type": "STRICT", "evalTrueSetting": 1, "evalTotalSetting": 1, "evalTimeSetting": 0, "dampeningId": "hawkular-MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1-FIRING" }, "evalSets": [ [ { "evalTimestamp": 1502156325254, "dataTimestamp": 1502156342001, "type": "COMPARE", "condition": { "tenantId": "hawkular", "triggerId": "MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1", "triggerMode": "FIRING", "type": "COMPARE", "conditionSetSize": 2, "conditionSetIndex": 1, "conditionId": "hawkular-MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1-FIRING-2-1", "dataId": "hm_g_MI~R~[7402c000-6df6-46ae-9e79-9b4f71aa0ce4/EAP7-Standalone~~]~MT~WildFly Memory Metrics~Heap Used", "operator": "GT", "data2Id": "hm_g_MI~R~[7402c000-6df6-46ae-9e79-9b4f71aa0ce4/EAP7-Standalone~~]~MT~WildFly Memory Metrics~Heap Max", "data2Multiplier": 0.2 }, "value1": 365691568, "value2": 1366294528 }, { "evalTimestamp": 1502156325254, "dataTimestamp": 1502156342001, "type": "COMPARE", "condition": { "tenantId": "hawkular", "triggerId": "MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1", "triggerMode": "FIRING", "type": "COMPARE", "conditionSetSize": 2, "conditionSetIndex": 2, "conditionId": "hawkular-MiQ-region-2ecdd959-4b31-45c1-bc5a-afc9ca8e9fcf-ems-1d05c128-af7f-409d-8855-4ec373930bc6-alert-28-1-FIRING-2-2", "dataId": "hm_g_MI~R~[7402c000-6df6-46ae-9e79-9b4f71aa0ce4/EAP7-Standalone~~]~MT~WildFly Memory Metrics~Heap Used", "operator": "LT", "data2Id": "hm_g_MI~R~[7402c000-6df6-46ae-9e79-9b4f71aa0ce4/EAP7-Standalone~~]~MT~WildFly Memory Metrics~Heap Max", "data2Multiplier": 0.15 }, "value1": 365691568, "value2": 1366294528 } ] ] } ]

MIQ Server Provider with Data:

MIQ Provider Timeline with no events:

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 23 (16 by maintainers)

Most upvoted comments

It’s broken here: https://github.com/ManageIQ/manageiq/blob/53c1704a56db62a1751665f52323c0e496f32669/app/models/miq_alert.rb#L569

because target is the server object and, because of STI, target.class.name will have the full class name ManageIQ::Providers::Hawkular::MiddlewareManager::MiddlewareServer. It should be better to use is_a? method.

It’s also broken here: https://github.com/ManageIQ/manageiq/blob/53c1704a56db62a1751665f52323c0e496f32669/app/models/middleware_server.rb#L36

because alert id’s can now be different.

Aaahh! 😱 It’s broken in a lot of places.

israel-hdez on Aug 28, 2017

@cfcosta @jshaughn @gbaufake I went back to this because I was on something else that, to some extent, depends on this. I could find several problems:

First, look at this line of code in event catcher. It uses ::Hawkular::Alerts::Alert to partition events, but it should be ::Hawkular::Alerts::Event. This is my fault. Sorry 😞 Once this is fixed, events will be stored in miq database but they still won’t appear in timeline.
Events won’t appear in the timeline because they are created in Hawkular with tag miq.event_type == hawkular_alert (see source here). However, we don’t have it listed in our settings.yml file and that explains why it’s ignored when querying the timeline. I found this comment on event catcher and it makes me think that hidding those events/alerts was on purpose. But I don’t know exactly the background on that.
At least for servers in domain mode, member triggers are being created to metrics like hm_g_MI~R~[master.Unnamed+Domain/Local~/host=master/server=server-one]~MT~WildFly Memory Metrics~Heap Max. Notice the + character in Unnamed+Domain. This means that member triggers are being associated to metrics that doesn’t exist and events will never raise in these cases. This is also a bug, but I didn’t track the code to find where it is.
And, finally, event catcher is using EmsEvent to store events. I read the code of that class a little and what I understand is that EmsEvent is for logging events in timeline. This means that event catcher is just recording hidden timeline events and it’s completly ignoring the configuration in the alert. Probably, event catcher should be doing something else to process the events; but I don’t know.

So, right now I don’t know if this is an issue. I think we need more background on what should be doing the event catcher with the raised events. But right now, it would be accumulating them in miq the database completly hidden to the user and doing nothing else with those events 👎

israel-hdez on Sep 5, 2017