sensu-go: Entity definition not updated in round robin proxy checks

Hi !

When executing a round robin check for proxy entities, the entities definition does not seem to be updated when some attributes are modified (tested with annotations and subscriptions) and the resulting command with token substitution uses old attributes.

Expected Behavior

Updated annotations, subscriptions, etc. of proxy entities are expected to be used by round robin proxy checks tokens substitution (i.e. an IP address).

Current Behavior

Round robin proxy checks use old annotations on tokens substitution, as seen in events (but the entity definition in events is up to date).

Non round-robin checks with the same definition work as expected.

Steps to Reproduce (for bugs)

Using Docker, create two containers (a backend and an agent) using this docker-compose file:

version: "3"

services:
  backend:
    image: sensu/sensu:5.21.0
    hostname: backend
    command: sensu-backend start --log-level debug
    ports:
      - 3000:3000
      - 8080:8080
    volumes:
      - ./backend:/var/lib/sensu
    networks:
      sensu:

  agent:
    image: sensu/sensu:5.21.0
    hostname: agent
    command: sensu-agent start --backend-url 'ws://backend:8081' --subscriptions proxy
    depends_on:
      - backend
    networks:
      sensu:

networks:
  sensu:

Create a proxy entity and a dummy check:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: ping
spec:
  check_hooks: null
  command: ping {{ .annotations.ip_address }} -c 3
  interval: 15
  proxy_requests:
    entity_attributes:
    - entity.entity_class == 'proxy'
  publish: true
  round_robin: true
  subscriptions:
  - proxy
  timeout: 10

---
type: Entity
api_version: core/v2
metadata:
  annotations:
    ip_address: 127.0.0.1
  name: proxy
spec:
  entity_class: proxy

Let this check get executed by the agent, the event returns:

$ sensuctl event info proxy ping --format json | jq .entity.metadata.annotations.ip_address
"127.0.0.1"
$ sensuctl event info proxy ping --format json | jq .check.command                         
"ping 127.0.0.1 -c 3"

Modify the ip_address annotation of the proxy entity:

---
type: Entity
api_version: core/v2
metadata:
  annotations:
    ip_address: 8.8.8.8
  name: proxy
spec:
  entity_class: proxy

Wait until it gets executed again, the entity definition is updated in the event, but the check does not use the

$ sensuctl event info proxy ping --format json | jq .entity.metadata.annotations.ip_address
"8.8.8.8"
$ sensuctl event info proxy ping --format json | jq .check.command                         
"ping 127.0.0.1 -c 3"

Those logs are visible in the backend, but do not solve the issue:

{"component":"cache","level":"debug","msg":"rebuilding the cache for resource type *v2.Silenced","time":"2020-06-25T14:52:19Z"}
{"component":"cache","level":"debug","msg":"rebuilding the cache for resource type *v2.Entity","time":"2020-06-25T14:52:19Z"}

The same steps but with a check with round_robin: false work as expected.

Context

In real life, i use proxy entities with many annotations (usernames, password, ip addresses, etc.) as well as subscriptions to filter them with proxy_requests.entity_attributes. Many agents run those proxy checks, so round_robin is a must have.

Your Environment

• Sensu version used: 5.21.0
• Installation method: Docker