portainer: Unable to access stats for containers (Windows Server, Docker Swarm)

Bug description If I try to open the stats for any of the containers in our portainer-managed Docker Swarms (Windows Server), I get an error “Failure. Cannot read properties of null (reading ‘find’)”

Expected behavior I see the stats

Portainer Logs

2021-10-05T10:49:21.675940100Z level=info msg="2021/10/05 10:49:21 Instance already has defined environments. Skipping the environment defined via CLI."
2021-10-05T10:49:21.723484000Z level=info msg="2021/10/05 10:49:21 Instance already has an administrator user defined. Skipping admin password related flags."
2021-10-05T10:49:21.723484000Z level=info msg="2021/10/05 10:49:21 [INFO] [cmd,main] Starting Portainer version 2.9.0"
2021-10-05T10:49:21.724409900Z level=info msg="2021/10/05 10:49:21 [DEBUG] [chisel, monitoring] [check_interval_seconds: 10.000000] [message: starting tunnel management process]"
2021-10-05T10:49:21.724409900Z level=info msg="2021/10/05 10:49:21 [INFO] [http,server] [message: starting HTTPS server on port :9443]"
2021-10-05T10:49:21.724409900Z level=info msg="2021/10/05 10:49:21 [DEBUG] [internal,init] [message: start initialization monitor ]"
2021-10-05T10:49:21.724409900Z level=info msg="2021/10/05 10:49:21 [INFO] [http,server] [message: starting HTTP server on port :9000]"
2021-10-05T10:49:21.724409900Z 2021/10/05 10:49:21 server: Reverse tunnelling enabled
2021-10-05T10:49:21.724409900Z 2021/10/05 10:49:21 server: Fingerprint 31:29:dd:e5:a4:36:0b:a7:28:ae:1e:13:7a:42:0b:a8
2021-10-05T10:49:21.724409900Z 2021/10/05 10:49:21 server: Listening on 0.0.0.0:8000...
2021-10-05T10:50:45.282873300Z level=info msg="2021/10/05 10:50:45 http error: Invalid JWT token (err=Invalid JWT token) (code=401)"
2021-10-05T10:50:45.606515100Z level=info msg="2021/10/05 10:50:45 http error: Invalid JWT token (err=Invalid JWT token) (code=401)"
2021-10-05T10:50:45.797799200Z level=info msg="2021/10/05 10:50:45 http error: Invalid JWT token (err=Invalid JWT token) (code=401)"
2021-10-05T10:50:56.891958700Z level=info msg="2021/10/05 10:50:56 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: c66d4600-30d3-4efc-ace4-a1cae3e838f4] [resource_id: buildagent202451-temp_qr3orxi11izpe5nbda8jix4vg]"
2021-10-05T10:50:56.984244800Z level=info msg="2021/10/05 10:50:56 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: c66d4600-30d3-4efc-ace4-a1cae3e838f4] [resource_id: buildagent212452-temp_qr3orxi11izpe5nbda8jix4vg]"
2021-10-05T10:50:56.984244800Z level=info msg="2021/10/05 10:50:56 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: 8c31aab7-678a-4669-9f2b-08a5ea67a78a] [resource_id: 7fe51417f5b7ba7d2391062828be300a4bfb630c88a17327cedadb8517c0011a]"
2021-10-05T10:50:57.023308800Z level=info msg="2021/10/05 10:50:57 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: 8c31aab7-678a-4669-9f2b-08a5ea67a78a] [resource_id: buildagent30972409-temp_qr3orxi11izpe5nbda8jix4vg]"
2021-10-05T10:50:57.024746600Z level=info msg="2021/10/05 10:50:57 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: 8c31aab7-678a-4669-9f2b-08a5ea67a78a] [resource_id: yddhbpf1ppjx15n4ttnvag3cx]"
2021-10-05T10:50:57.044676600Z level=info msg="2021/10/05 10:50:57 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: c66d4600-30d3-4efc-ace4-a1cae3e838f4] [resource_id: buildagent392449-temp_qr3orxi11izpe5nbda8jix4vg]"
2021-10-05T10:50:57.126845800Z level=info msg="2021/10/05 10:50:57 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: c66d4600-30d3-4efc-ace4-a1cae3e838f4] [resource_id: buildagent662450-temp_qr3orxi11izpe5nbda8jix4vg]"
2021-10-05T10:50:58.582105300Z level=info msg="2021/10/05 10:50:58 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: 8c31aab7-678a-4669-9f2b-08a5ea67a78a] [resource_id: rpboe3y18s3fyrsllygpusn1q]"
2021-10-05T10:50:58.606863900Z level=info msg="2021/10/05 10:50:58 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: 8c31aab7-678a-4669-9f2b-08a5ea67a78a] [resource_id: rpboe3y18s3fyrsllygpusn1q]"
2021-10-05T10:51:11.810697500Z level=info msg="2021/10/05 10:51:11 http error: Invalid JWT token (err=Invalid JWT token) (code=401)"
2021-10-05T10:51:11.847726200Z level=info msg="2021/10/05 10:51:11 http error: Invalid JWT token (err=Invalid JWT token) (code=401)"
2021-10-05T10:51:24.053992300Z level=info msg="2021/10/05 10:51:24 [WARN] [http,proxy,docker] [message: unknown user name in access control label, ignoring access control rule for this user] [name: 8c31aab7-678a-4669-9f2b-08a5ea67a78a] [resource_id: buildagent31072410-temp_qr3orxi11izpe5nbda8jix4vg]"

Steps to reproduce the issue:

1. Go to portainer -> services
2. Extend one of the services
3. Click on the stats icon
4. See error

Please also see the following screen recording, showing the problem on multiple different services and containers:

Technical details:

• Portainer version: 2.9.0
• Docker version (managed by Portainer): 20.10.4, 20.10.5 and 20.10.6 (Swarm with 3 managers and 3 workers)
• Kubernetes version (managed by Portainer): -
• Platform (windows/linux): windows
• Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):

  agent:
    image: portainer/agent:2.9.0
    environment:
      AGENT_CLUSTER_ADDR: tasks.agent
    volumes:
      - source: '\\.\pipe\docker_engine'
        target: '\\.\pipe\docker_engine'
        type: npipe
      - source: 'f:/dockerdata/volumes'
        target: 'C:/ProgramData/docker/volumes'
        type: bind
    networks:
      - agent-network
    deploy:
      mode: global
      placement:
        constraints:
          - node.platform.os == windows

  portainer:
    image: portainer/portainer-ce:2.9.0
    command: -H tcp://tasks.agent:9001 --tlsskipverify --admin-password-file 'c:\\secrets\\adminPwd'
    volumes:
      - s:/portainer-data:c:/data
    networks:
      - agent-network
      - traefik-public
    deploy:
      placement:
        constraints:
          - node.role == manager
          - node.role == manager
      labels:
        - traefik.enable=true
        - traefik.http.routers.portainer.rule=Host(`***redacted***`) && PathPrefix(`/portainer/`)
        - traefik.http.routers.portainer.entrypoints=websecure
        - traefik.http.routers.portainer.tls.certresolver=myresolver
        - traefik.http.routers.portainer.service=portainer@docker
        - traefik.http.services.portainer.loadBalancer.server.scheme=http
        - traefik.http.services.portainer.loadBalancer.server.port=9000
        - traefik.http.middlewares.portainer.stripprefix.prefixes=/portainer
        - traefik.http.middlewares.limit.buffering.maxRequestBodyBytes=500000000
        - traefik.http.routers.portainer.middlewares=portainer@docker, limit@docker
    depends_on:
      - traefik
    secrets:
      - source: adminPwd
        target: "c:/secrets/adminPwd"

• Browser: Edge
• Use Case (delete as appropriate): Using Portainer in a Commerical setup.
• Have you reviewed our technical documentation and knowledge base? Yes

Additional context It used to work with previous versions, but I can’t say at which moment it started to fail. We have multiple Swarms set up fully automated and the problem seems to happen everywhere (I didn’t check each one, but the first five I checked all show the same behavior).