sensu-go: Entity definition not updated in round robin proxy checks
Hi !
When executing a round robin check for proxy entities, the entities definition does not seem to be updated when some attributes are modified (tested with annotations and subscriptions) and the resulting command with token substitution uses old attributes.
Expected Behavior
Updated annotations, subscriptions, etc. of proxy entities are expected to be used by round robin proxy checks tokens substitution (i.e. an IP address).
Current Behavior
Round robin proxy checks use old annotations on tokens substitution, as seen in events (but the entity definition in events is up to date).
Non round-robin checks with the same definition work as expected.
Steps to Reproduce (for bugs)
Using Docker, create two containers (a backend and an agent) using this docker-compose file:
version: "3"
services:
backend:
image: sensu/sensu:5.21.0
hostname: backend
command: sensu-backend start --log-level debug
ports:
- 3000:3000
- 8080:8080
volumes:
- ./backend:/var/lib/sensu
networks:
sensu:
agent:
image: sensu/sensu:5.21.0
hostname: agent
command: sensu-agent start --backend-url 'ws://backend:8081' --subscriptions proxy
depends_on:
- backend
networks:
sensu:
networks:
sensu:
Create a proxy entity and a dummy check:
---
type: CheckConfig
api_version: core/v2
metadata:
name: ping
spec:
check_hooks: null
command: ping {{ .annotations.ip_address }} -c 3
interval: 15
proxy_requests:
entity_attributes:
- entity.entity_class == 'proxy'
publish: true
round_robin: true
subscriptions:
- proxy
timeout: 10
---
type: Entity
api_version: core/v2
metadata:
annotations:
ip_address: 127.0.0.1
name: proxy
spec:
entity_class: proxy
Let this check get executed by the agent, the event returns:
$ sensuctl event info proxy ping --format json | jq .entity.metadata.annotations.ip_address
"127.0.0.1"
$ sensuctl event info proxy ping --format json | jq .check.command
"ping 127.0.0.1 -c 3"
Modify the ip_address
annotation of the proxy entity:
---
type: Entity
api_version: core/v2
metadata:
annotations:
ip_address: 8.8.8.8
name: proxy
spec:
entity_class: proxy
Wait until it gets executed again, the entity definition is updated in the event, but the check does not use the
$ sensuctl event info proxy ping --format json | jq .entity.metadata.annotations.ip_address
"8.8.8.8"
$ sensuctl event info proxy ping --format json | jq .check.command
"ping 127.0.0.1 -c 3"
Those logs are visible in the backend, but do not solve the issue:
{"component":"cache","level":"debug","msg":"rebuilding the cache for resource type *v2.Silenced","time":"2020-06-25T14:52:19Z"}
{"component":"cache","level":"debug","msg":"rebuilding the cache for resource type *v2.Entity","time":"2020-06-25T14:52:19Z"}
The same steps but with a check with round_robin: false
work as expected.
Context
In real life, i use proxy entities with many annotations (usernames, password, ip addresses, etc.) as well as subscriptions
to filter them with proxy_requests.entity_attributes
. Many agents run those proxy checks, so round_robin
is a must have.
Your Environment
- Sensu version used: 5.21.0
- Installation method: Docker
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 19 (3 by maintainers)
All executions of the check show this problem (have seen it last for days). The only way i found for now is to restart the backends, so the new entities definitions are cached.
We are facing exactly same issue. What we do, as we don’t want to restart the sensu backend, we delete and create proxy check - then all new definitions are propagated properly. Even when check interval is changed on fly (without backend restart or delete/create), it has no effect and check is scheduled all time according “buffered” value when it was loaded completely from scratch very last time (which is even service restart of delete/create) I’d say this is pretty serious as it breaks the generic concept of Sensu management.