n3dr: Retry failed requests and do not exit when a single artifact download failed

First of all, n3dr is AWESOME!

What would you like to be added:

  • Retry failed requests
  • Log a failed download and proceed to the next one.

Why is this needed:

On systems with a huge number of artifacts it takes a while to download everything and it’s confusing when the whole process ended up because of a short network issue or in cases when artifact data is corrupted.

Examples:

# docker logs a5e206740344
time="2020-07-10T11:08:32Z" level=info msg="HomeDir: '/home/jenkins'"
time="2020-07-10T16:30:11Z" level=info msg="Assembling downloadURLs 'releases'"
 64656 / 74888 time="2020-07-10T21:06:13Z" level=fatal msg="Get \"https://nexus/service/rest/v1/assets?repository=releases&continuationToken=55258bea3c6fae8a1c9b8bc8d0c74cdc\": EOF"
 64735 / 74888   86.44% 43m17

That’s sad when after 10 hours collecting URLs the process failed with EOF.

Or

time="2020-07-12T08:46:39Z" level=fatal msg="URL: 'https://nexus/repository/releases/archetype-catalog.xml' does not seem to contain an artifactName"

This file could be uploaded by mistake, I don’t remember, but that’s not a reason to stop the backup process.

Or sometimes it happens: data file has been damaged on a filesystem or removed, so the backup process should not be interrupted, e.g.

 12 / 4442 [>------------------------------------------------------------------------------------------------------------------------------------------]   0.27% 01m01s
FATA[0279] ResponseCode: '500' and Message '500 javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit' for URL: https://nexus-dev/repository/thirdparty/com/oracle/oim/wlthint3client/10.3.6.0/wlthint3client-10.3.6.0.pom 

Thank you!

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 4
  • Comments: 17 (10 by maintainers)

Commits related to this issue

Most upvoted comments

For completeness: I tested version 6.0.0, which successfully solves the problem I mentioned in this issue. Thanks, @030!

I am running Nexus version OSS 3.26.1-02.

I do not recall uploading this file myself. It is located in a group repository. I think it may have been automatically generated by Nexus. See: https://stackoverflow.com/questions/22837962/how-to-update-archetype-catalog-in-nexus-repository