nexus-public: All Blob stores are offline after migration to v3.73.0: unable to migrate secrets error
We have just recently migrated our Nexus OSS service from v3.72.0 to v3.73.0. Immediately after Nexus has started we noticed that all S3 blob stores are reported as not started. Further investigation uncovered the following error message for each blob store we have. The same error appears when I attempt to modify blob store configuration via API or UI.
2024-10-11 15:58:26,282+0000 ERROR [quartz-9-thread-5] *SYSTEM org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask - Failed to run task 'Migrate existing secrets into a single source (secrets table).'
org.sonatype.nexus.security.secrets.SecretMigrationException: unable to migrate secrets for blobstore: xxxxxxxx
at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.migrate(BlobStoreConfigSecretsMigrator.java:69)
at org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask.execute(SecretsMigrationTask.java:51)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:105)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.sonatype.nexus.crypto.internal.error.CipherException: last block incomplete in decryption
at org.sonatype.nexus.crypto.internal.LegacyCipherFactoryImpl$PbeCipherImpl.transform(LegacyCipherFactoryImpl.java:114)
at org.sonatype.nexus.crypto.internal.LegacyCipherFactoryImpl$PbeCipherImpl.decrypt(LegacyCipherFactoryImpl.java:103)
at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl.decryptLegacy(SecretsServiceImpl.java:306)
at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl.doDecrypt(SecretsServiceImpl.java:229)
at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl.access$0(SecretsServiceImpl.java:227)
at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl$SecretImpl.decrypt(SecretsServiceImpl.java:364)
at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.maybeMigrateSecret(BlobStoreConfigSecretsMigrator.java:85)
at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.migrate(BlobStoreConfigSecretsMigrator.java:65)
... 14 common frames omitted
Caused by: javax.crypto.IllegalBlockSizeException: last block incomplete in decryption
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2263)
at org.sonatype.nexus.crypto.internal.LegacyCipherFactoryImpl$PbeCipherImpl.transform(LegacyCipherFactoryImpl.java:110)
... 21 common frames omitted
2024-10-11 15:58:26,283+0000 INFO [quartz-9-thread-5] *SYSTEM org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask - Task complete
I have not found a workaround. I have additionally followed the release note instructions and created an encryption key using a nexus.secrets.file property and migrated secrets using REST API call. But the problem with blob store persists.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 15
@garesgit is correct, there’s a bug here that causes Repo to sometimes try to decrypt something that’s not encrypted, so the upgrade step fails. Apologies for the pain here, we have a fix in progress that is scheduled to come out in our next monthly release (scheduled for the first week of November).
Guys, the same thing happened to me with the AWS S3 blobs after enabling the secret. I just generated new IAM tokens, updated them in the blob store, saved, and it came back online.
Okay, 3.74 is out with a fix. Apologies for the friction trying to upgrade. Thanks for coming back and posting that it worked for you, @mokrinsky.
I had the same issue and with the new 3.74.0 release everything went back to normal.
Same error here. Downgrading to version 3.72.0.
Ended up downgrading and restoring the database.
For replacing AmazonS3Factory.java I would need to build an own container.
Just a clue of what could be going on here. I tried to find the diffs and it looks like the issue is happening on the AmazonS3Factory.java (I having issues with the AWS S3 repos). Doing a diff with the previous version I can see that there is decrypt method call over the AWS
secretAccessKey. In previous version there was not and it was passed thorugh clear (see below diff with previous version). I checked the DB and thesecretAccessKeyis not encrypted (it is in cleartext). Due just code change in the AmazonS3Factory.java and as a matter of test, I have replaced the AmazonS3Factory.class into the nexus-blobstore-s3-3.73.0-12.jar with the class of v3.72-04 and the S3 repos started fine with only this change. So, wondering if the code is trying to decrypt something that is not encrypted and that is why we are getting that error.