rclone: Broken S3 Bucket Listing with Strict Access Policies

What is the problem you are having with rclone?

S3 bucket listing is broken with 1.54 when strict access policies are used.

What is your rclone version (output from rclone version)

rclone v1.54.0
- os/arch: linux/amd64
- go version: go1.15.7

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Ubuntu 18.04.5

Which cloud storage system are you using? (e.g. Google Drive)

Minio S3

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone-v1.54.0 --s3-no-check-bucket --config=/home/darthshadow/rclone/rclone.conf lsd ZenStorage-Inbound:zd-movies-non-english/movies-non-english/

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2021/02/02 16:52:57 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone-v1.54.0" "--s3-no-check-bucket" "--config=/home/darthshadow/rclone/rclone.conf" "lsd" "ZenStorage-Inbound:zd-movies-non-english/movies-non-english/" "-vv"]
2021/02/02 16:52:57 DEBUG : Using config file from "/home/darthshadow/rclone/rclone.conf"
2021/02/02 16:52:57 DEBUG : Creating backend with remote "ZenStorage-Inbound:zd-movies-non-english/movies-non-english/"
2021/02/02 16:52:57 Failed to create file system for "ZenStorage-Inbound:zd-movies-non-english/movies-non-english/": Forbidden: Forbidden
        status code: 403, request id: 165FFC768E1DA1F7, host id:

---

The access policies allow the user to only access a specific subpath in the bucket like: zd-movies-non-english/movies-non-english/German/movies/. There are multiple such policies for specific paths in the bucket.

This works perfectly with 1.53.4.

2021/02/02 17:15:31 DEBUG : rclone: Version "v1.53.4" starting with parameters ["rclone-v1.53.4" "--s3-no-check-bucket" "--config=/home/darthshadow/rclone/rclone.conf" "lsd" "ZenStorage-Inbound:zd-movies-non-english/movies-non-english/" "-vv"]
2021/02/02 17:15:31 DEBUG : Using config file from "/home/darthshadow/rclone/rclone.conf"
2021/02/02 17:15:31 DEBUG : Creating backend with remote "ZenStorage-Inbound:zd-movies-non-english/movies-non-english/"
2021/02/02 17:15:31 DEBUG : fs cache: renaming cache item "ZenStorage-Inbound:zd-movies-non-english/movies-non-english/" to be canonical "ZenStorage-Inbound:zd-movies-non-english/movies-non-english"
           0 2021-02-02 17:15:31        -1 Bollywood
           0 2021-02-02 17:15:31        -1 Danish
           0 2021-02-02 17:15:31        -1 Dutch
           0 2021-02-02 17:15:31        -1 German
           0 2021-02-02 17:15:31        -1 Movies-4k-Nordic
           0 2021-02-02 17:15:31        -1 Swedish
2021/02/02 17:15:31 DEBUG : 3 go routines active

Dumping the HTTP Requests shows an extra request from 1.53.4 which is missing in 1.54:

GET /zd-movies-non-english?delimiter=%2F&max-keys=1000&prefix=movies-non-english%2F HTTP/1.1