n3dr: Not able to upload downloaded repository content

What happened: I’m trying to migrate a maven2 repository from a Nexus 3 to another one. Download seems to be fine. But I cannot upload the downloaded repository content to the new one. It fails with the following error:

FATA[0004]/home/runner/work/n3dr/n3dr/cmd/n3dr/upload.go:23 main.glob..func3() HTTPStatusCode: '400'; ResponseMessage: '[{"id":"groupId","message":"Missing required component field 'Group ID'"},{"id":"artifactId","message":"Missing required component field 'Artifact ID'"},{"id":"version","message":"Missing required component field 'Version'"}]'; ErrorMessage: '<nil>' 

Though it fails with this error, it uploads the artifact it prints the error for and then fails. That artifact has the following files inside its folder:

xxx-client-5.0.0.jar
xxx-client-5.0.0.module
xxx-client-5.0.0.pom

What you expected to happen: The upload process should complete successfully.

How to reproduce it (as minimally and precisely as possible):

Download the repository content:

n3dr backup -n https://nexus.example.com -r my-maven-releases -u readonly-user -p userPwd --directory-prefix $(pwd)/my-maven-releases

Then upload the content to the repository inside the new nexus instance:

n3dr upload -n https://new-nexus.example.com -u admin -p adminPwd -r my-maven-releases --directory-prefix $(pwd)/my-maven-releases

Anything else we need to know?:

Environment:

  • source nexus version: OSS 3.28.1-01
  • target nexus version: OSS 3.29.2-02
  • n3dr version (use n3dr -v): nd3r version refs/tags/6.0.1 (snap version)
  • OS (e.g: cat /etc/os-release): Ubuntu 20.04.2 LTS
  • Kernel (e.g. uname -a): 5.4.0-1037-aws #39-Ubuntu SMP Thu Jan 14 02:56:06 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 20 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Reported error:

DEBU[0003]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:95 
github.com/030/n3dr/internal/artifacts.Nexus3.multipartUpload() URL                                           
multipart="maven2.asset1=@my-maven-releases/my-maven-releases/com/example/product/proj-client/5.0.0-2-gcac5af6/proj-client-5.0.0-2-gcac5af6.jar,
maven2.asset1.extension=jar,
maven2.asset1.classifier=gcac5af6,
maven2.asset2=@my-maven-releases/my-maven-releases/com/example/product/proj-client/5.0.0-2-gcac5af6/proj-client-5.0.0-2-gcac5af6.module,
maven2.asset2.extension=module,
maven2.asset2.classifier=gcac5af6,
maven2.asset3=@my-maven-releases/my-maven-releases/com/example/product/proj-client/5.0.0-2-gcac5af6/proj-client-5.0.0-2-gcac5af6.pom,
maven2.asset3.extension=pom,
maven2.asset3.classifier=gcac5af6" 
url="https://nexus.my.com/service/rest/v1/components?repository=my-maven-releases"
FATA[0004]/home/runner/work/n3dr/n3dr/cmd/n3dr/upload.go:23 main.glob..func3() HTTPStatusCode: '400'; 
ResponseMessage: '[{"id":"groupId","message":"Missing required component field 'Group ID'"},
{"id":"artifactId","message":"Missing required component field 'Artifact ID'"},
{"id":"version","message":"Missing required component field 'Version'"}]'; ErrorMessage: '<nil>'

Reproduced error:

@030 - looks like it’s much better now. My upload command has finished and I got only 3 errors like this: Missing required component field which are OK in my case (as they’re kind of test/dirty builds)

Thank you for this awesome tool! 🚀

😃 My mistake. I placed a binary in /usr/local/bin and that was before the snap in $PATH. I deleted it and now I have the correct version. I’ll try and let you know.

Hi @030 , skip really saves the day, thanks! But I’ve seen many regex related errors. Here is one:

DEBU[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:119 github.com/030/n3dr/internal/artifacts.Nexus3.multipartUpload() URL                                           multipart= url="https://nexus.myapp.com/service/rest/v1/components?repository=myapp-maven-releases"                                                                               
ERRO[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:299 github.com/030/n3dr/internal/artifacts.(*Nexus3).readMavenFilesAndUpload() skipErrors: 'true'. Error: 'The string should at least contain a '=', but was: '''                                                                                                                                         
DEBU[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:283 github.com/030/n3dr/internal/artifacts.(*Nexus3).readMavenFilesAndUpload() 113 Detecting artifacts in folder 'myapp-maven-releases/myapp-maven-releases/com/myorg/myproj/common-analytics/5.9.0-6-gefcc5b9'                                                                             
ERRO[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:104 github.com/030/n3dr/internal/artifacts.artifactTypeDetector() skipErrors: 'true'. Error: 'Check whether regexVersion: '([\d\.-]+\d)' and regexClassifier: '([a-zA-Z]+)?' match the artifact: 'myapp-maven-releases/myapp-maven-releases/com/myorg/myproj/common-analytics/5.9.0-6-gefcc5
b9/common-analytics-5.9.0-6-gefcc5b9.jar''                                                                                                                                                                                                                                                                                                                                   ERRO[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:104 github.com/030/n3dr/internal/artifacts.artifactTypeDetector() skipErrors: 'true'. Error: 'Check whether regexVersion: '([\d\.-]+\d)' and regexClassifier: '([a-zA-Z]+)?' match the artifact: 'myapp-maven-releases/myapp-maven-releases/com/myorg/myproj/common-analytics/5.9.0-6-gefcc5
b9/common-analytics-5.9.0-6-gefcc5b9.module''                                                                                                                                                                                                                                                                                                                                ERRO[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:104 github.com/030/n3dr/internal/artifacts.artifactTypeDetector() skipErrors: 'true'. Error: 'Check whether regexVersion: '([\d\.-]+\d)' and regexClassifier: '([a-zA-Z]+)?' match the artifact: 'myapp-maven-releases/myapp-maven-releases/com/myorg/myproj/common-analytics/5.9.0-6-gefcc5b9/common-analytics-5.9.0-6-gefcc5b9.pom''                                                                                                                                                                                                                                                                                                                                   
ERRO[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:291 github.com/030/n3dr/internal/artifacts.(*Nexus3).readMavenFilesAndUpload() skipErrors: 'true'. Error: 'The sb.String() should not be empty. Verify whether the path: 'myapp-maven-releases/myapp-maven-releases/com/myorg/myproj/common-analytics/5.9.0-6-gefcc5b9' contains artifacts' 
INFO[0474]/home/runner/work/n3dr/n3dr/internal/artifacts/upload.go:296 github.com/030/n3dr/internal/artifacts.(*Nexus3).readMavenFilesAndUpload() 113 Upload ''

Btw, that last 2 parts of a.b.c.d-6-abc123 is generated by palantir’s gradle git plugin. 6 denotes the changes after last tag and the last bit is the summary of commit sha.

It’d also be nice if you could add an option to skip errors when a parameter is specified like below:

n3dr upload ... -s ...

There are multiple cases I can think of for this:

  • One repo may have disabled overwriting of artifacts (and subsequent trials such as in this ticket) would cause all upload to fail
  • One artifact may really have issues but the others would be fine?

There could be a regex mismatch, I will add proj-client-5.0.0-2-gcac5af6.pom to the integration test to inspect the issue.