minio-go: Iterating more than 1k objects failed with ListObjectsV2 but works with ListObjects

Given this simple (possibly not very go-like) function where I iterate through all files, looking for a BACKUP file, and then return the s3://bucket/prefix for a given one among other things.

    doneCh := make(chan struct{})
    defer close(doneCh)
    backups := make([]Backup, 0)
    // iterate through all objects looking for .../BACKUP which indicates a finished backup
    objectCh := s.client.ListObjectsV2(s.config.Bucket, "", true, doneCh)
    for object := range objectCh {
        if object.Err != nil {
            return nil, fmt.Errorf("unable to get backups: %s", object.Err)
        }
        if path.Base(object.Key) == "BACKUP" {
            var backupType types.BackupType
            if strings.Contains(object.Key, types.Incremental.String()) {
                backupType = types.Incremental
            } else if strings.Contains(object.Key, types.Full.String()) {
                backupType = types.Full
            }
            u, _ := url.Parse("")
            u.Scheme = "s3"
            u.Host = s.config.Bucket
            u.Path = path.Dir(object.Key)
            b := Backup{
                ID:       u.Path,
                Location: s.addAuthorization(u.String()),
                Typ:      backupType,
            }
            backups = append(backups, b)
        }
    }
    sort.SliceStable(backups, func(i int, j int) bool {
        return backups[i].Location < backups[j].Location
    })

When using ListObjectsV2 in this manner I kept getting the following error:

Truncated response should have continuation token set

But I get no such error and my application works using ListObjects.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (8 by maintainers)

Most upvoted comments

@harshavardhana thanks for your reply , I just found a work-around for me , I can use the mc tags/RELEASE.2019-10-02T19-41-02Z according to this issue https://github.com/minio/mc/issues/3067

You can use this version if you encounter this same error.

LeoQuote on Feb 24, 2020

I’ve experienced same problem and ListObjects also worked for me. My code is straight from example:

	client, err := minio.New(endpoint, accessKey, secKey, ssl)
	if err != nil {
		log.Fatal(err)
	}

	doneCh := make(chan struct{})
	defer close(doneCh)
	recursive := true
	objectCh := client.ListObjectsV2(bucket, "", recursive, doneCh)

	for object := range objectCh { 
	  // delete stuff
}

Changing it to V1 or narrowing scope with second argument does the trick.

After taking look at code, I think the culprit should be S3 vendor (DigitalOcean Spaces in my case) not setting token: https://github.com/minio/minio-go/blob/753e5f73af12800035498a3efc6828e943143c5c/api-list.go#L247-L249

zarbis on Jul 11, 2019

Thanks @gigatexal we’ll take a look

nitisht on Jul 2, 2019