compose: [BUG] Secret mounts in `/run/secrets` throws an error: Could not find the file / in container

Description

This error started just recently, especially when updating to latest Docker Engine / Docker Compose versions.

When a service is mentioned with user: "1000" the container that have explicit users defined to them e.g. grafana in their images fail to mount the secrets in compose file to the respective /run/secrets directory in the container on boot.

Previously, it was known that a getent was performed on the host to match whether the user ID matches and a container is spun up in order to make /run/secrets readable by a the container’s user (if it not root).

Steps To Reproduce

Compose file

services:
  grafana:
    image: docker.io/grafana/grafana-oss:9.5.1
    container_name: komponist_grafana
    environment:
      - GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    secrets:
      - grafana_admin_username
      - grafana_admin_password
    logging:
      options:
        max-size: "5m"
    ports:
     - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
    grafana_admin_username:
      environment: GRAFANA_ADMIN_USERNAME
    grafana_admin_password:
      environment: GRAFANA_ADMIN_PASSWORD

Environment Variables

.env file

GRAFANA_ADMIN_USERNAME=admin
GRAFANA_ADMIN_PASSWORD=testGrafana

Steps

1. Upon docker compose up the following error occurs:

   Container komponist_grafana  Creating                                                                           
   Error response from daemon: Could not find the file / in container 
   9714fd659bd2eb795855f9fa292d7e76f3a06fdd40a16dfd47e5c53f759758a9
   

2. Upon forcing an up again using docker compose up the following logs show up:

   komponist_grafana  | Getting secret GF_SECURITY_ADMIN_PASSWORD from 
   /run/secrets/grafana_admin_password
   komponist_grafana  | /run.sh: line 59: /run/secrets/grafana_admin_password: No such file or directory
   komponist_grafana exited with code 1
   

3. Upon removing the user value from the Compose file, error still persists

Compose Version

v2.18.1

Docker Environment

Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 229
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.44GiB

Anything else?

Work-around / Solution

The only way to get the container up is to figure out from the container what the user is using an whoami / id and place this in the user of the compose file.

Working docker-compose.yml

services:
  grafana:
    image: docker.io/grafana/grafana-oss:9.5.1
    container_name: komponist_grafana
    environment:
      - GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    secrets:
      - grafana_admin_username
      - grafana_admin_password
    logging:
      options:
        max-size: "5m"
    ports:
     - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
    user: grafana
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
    grafana_admin_username:
      environment: GRAFANA_ADMIN_USERNAME
    grafana_admin_password:
      environment: GRAFANA_ADMIN_PASSWORD

I am not sure if this is a Docker Compose Bug or a Docker Engine thing from the start of Docker Engine v24.x.x.

NOTE: This problem does not persist when the container’s user is root

Happy to help reproduce any other examples with similar logic.