n3dr: Retry failed requests and do not exit when a single artifact download failed
First of all, n3dr is AWESOME!
What would you like to be added:
- Retry failed requests
- Log a failed download and proceed to the next one.
Why is this needed:
On systems with a huge number of artifacts it takes a while to download everything and it’s confusing when the whole process ended up because of a short network issue or in cases when artifact data is corrupted.
Examples:
# docker logs a5e206740344
time="2020-07-10T11:08:32Z" level=info msg="HomeDir: '/home/jenkins'"
time="2020-07-10T16:30:11Z" level=info msg="Assembling downloadURLs 'releases'"
64656 / 74888 time="2020-07-10T21:06:13Z" level=fatal msg="Get \"https://nexus/service/rest/v1/assets?repository=releases&continuationToken=55258bea3c6fae8a1c9b8bc8d0c74cdc\": EOF"
64735 / 74888 86.44% 43m17
That’s sad when after 10 hours collecting URLs the process failed with EOF.
Or
time="2020-07-12T08:46:39Z" level=fatal msg="URL: 'https://nexus/repository/releases/archetype-catalog.xml' does not seem to contain an artifactName"
This file could be uploaded by mistake, I don’t remember, but that’s not a reason to stop the backup process.
Or sometimes it happens: data file has been damaged on a filesystem or removed, so the backup process should not be interrupted, e.g.
12 / 4442 [>------------------------------------------------------------------------------------------------------------------------------------------] 0.27% 01m01s
FATA[0279] ResponseCode: '500' and Message '500 javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit' for URL: https://nexus-dev/repository/thirdparty/com/oracle/oim/wlthint3client/10.3.6.0/wlthint3client-10.3.6.0.pom
Thank you!
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 4
- Comments: 17 (10 by maintainers)
Commits related to this issue
- [GH-134] Backup an NPM repository. — committed to 030/n3dr by 030 4 years ago
- [GH-134] NPM recursion. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Go Channel. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Get repository type. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Download NPM artifacts when calling repositories subcommand. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Some cleanup. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Several superfuous tests removed. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Several superfuous tests removed. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Backup tests were broken. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Golang CI test issues resolved. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Recursion 404 allow otherwise npm download will not succeed. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Always check error explicitly. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Golang 1.15.6. — committed to 030/n3dr by 030 4 years ago
- [GH-134] 3.29.0 integration test broken currently. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Reenabled test. — committed to 030/n3dr by 030 4 years ago
- Merge pull request #170 from 030/134-npm [GH-134] Backup an NPM repository. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Snap version 3.5.2. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Snap version 5.5.2. — committed to 030/n3dr by 030 4 years ago
- [GH-134] Backup an NPM repository. — committed to 030/n3dr by 030 4 years ago
- [GH-134] NPM recursion. — committed to 030/n3dr by 030 4 years ago
For completeness: I tested version
6.0.0
, which successfully solves the problem I mentioned in this issue. Thanks, @030!I am running Nexus version
OSS 3.26.1-02
.I do not recall uploading this file myself. It is located in a group repository. I think it may have been automatically generated by Nexus. See: https://stackoverflow.com/questions/22837962/how-to-update-archetype-catalog-in-nexus-repository