verdaccio: "publish --force" and "unpublish VERSION" seem to be broken

Describe the bug

I tried to look at #1359 and #1405 but couldn’t find a solution. I’m using ~~https://github.com/Remitly/verdaccio-s3-storage~~ https://registry.verdaccio.org/-/web/detail/verdaccio-aws-s3-storage as a storage plugin, but you can comment the whole storage key in configuration and it will be the same (except for one thing that is pointed out below).

npm publish --force

If I try to overwrite an existing package version with a new one:

npm publish --force --registry MY_REPO

The result is:

  • old package is removed from the remote storage and the web UI doesn’t show it
  • the npm command fails with:
npm ERR! code EPUBLISHCONFLICT
npm ERR! publish fail Cannot publish over existing version.
npm ERR! publish fail Update the 'version' field in package.json and try again.
npm ERR! publish fail 
npm ERR! publish fail To automatically increment version numbers, see:
npm ERR! publish fail     npm help version

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/marco/.npm/_logs/2019-08-10T13_46_04_312Z-debug.log
  • running the same exact command again publishes successfully

npm unpublish --force SPECIFIC_VERSION

If I try to unpublish the package first to avoid errors:

npm unpublish "@test/my-pack@1.0.0-beta" --force --registry MY_REPO

The result is:

  • npm exits with no errors and says that the package has been removed
  • the package still shows up on the AWS S3 storage and on the web UI
  • trying to publish the same package from now on will result in failures (this doesn’t happen when using local storage: publishing again works as with the command above)

The only way I found to recover from this situation was:

  • removing the .tgz archive manually from the storage
  • restart the verdaccio server

These commands don’t work as expected.

To Reproduce

I’m using the following Dockerfile to speed up testing:

FROM verdaccio/verdaccio:latest

USER root

ENV NODE_ENV=production

# Install "envsubst" to replace ENV_VARS in config
RUN apk add gettext libintl

# Install AWS S3 Plugin
RUN yarn add verdaccio-aws-s3-storage --production=true --no-lockfile --registry https://registry.verdaccio.org

# Copy configuration and .htpasswd
COPY ./config.yml  $VERDACCIO_APPDIR/conf/config.yaml

# Fix permissions
RUN chown -R $VERDACCIO_USER_UID:root $VERDACCIO_APPDIR/conf/config.yaml \
    && chmod -R g=u $VERDACCIO_APPDIR/conf/config.yaml

USER $VERDACCIO_USER_UID

# Set user credentials, replace ENV_VARS in config and finally run Verdaccio
CMD echo "$HTPASSWD" > /verdaccio/storage/htpasswd \
    && envsubst \$AWS_S3_BUCKET,\$AWS_S3_PATH_PREFIX,\$AWS_DEFAULT_REGION,\$AWS_S3_ENDPOINT,\$AWS_S3_PATH_STYLE,\$AWS_ACCESS_KEY_ID,\$AWS_SECRET_ACCESS_KEY < $VERDACCIO_APPDIR/conf/config.yaml > $VERDACCIO_APPDIR/conf/config.yaml \
    && $VERDACCIO_APPDIR/bin/verdaccio --config $VERDACCIO_APPDIR/conf/config.yaml --listen $VERDACCIO_PROTOCOL://0.0.0.0:$VERDACCIO_PORT

This docker-compose.yml should run your registry:

version: '2.1'

services:
  verdaccio:
    image: testing/private-verdaccio-s3:latest
    environment:
      AWS_S3_BUCKET: 
      AWS_S3_PATH_PREFIX: 
      AWS_ACCESS_KEY_ID: 
      AWS_SECRET_ACCESS_KEY: 
      AWS_DEFAULT_REGION:
      AWS_S3_ENDPOINT: 
      AWS_S3_PATH_STYLE: "true"
      # User is demo:demo
      HTPASSWD: |
        demo:$apr1$uq2Uw53i$dS7GoGvNuupumWvXjOJ.R1
    ports:
      - "4874:4873"

Configuration file config.yml:

#
# This is the config file used for the docker images.
# It allows all users to do anything, so don't use it on production systems.
#
# Do not configure host and port under `listen` in this file
# as it will be ignored when using docker.
# see https://verdaccio.org/docs/en/docker#docker-and-custom-port-configuration
#
# Look here for more config file examples:
# https://github.com/verdaccio/verdaccio/tree/master/conf
#

# path to a directory with all packages
storage: /verdaccio/storage/data

# comment this section entirely to disable AWS S3
store:
  aws-s3-storage:
    bucket: $AWS_S3_BUCKET
    keyPrefix: $AWS_S3_PATH_PREFIX
    region: $AWS_DEFAULT_REGION
    endpoint: $AWS_S3_ENDPOINT
    s3ForcePathStyle: $AWS_S3_PATH_STYLE
    accessKeyId: $AWS_ACCESS_KEY_ID
    secretAccessKey: $AWS_SECRET_ACCESS_KEY

web:
  title: Verdaccio

auth:
  htpasswd:
    file: /verdaccio/storage/htpasswd
    max_users: -1

security:
  api:
    jwt:
      sign:
        expiresIn: 60d
        notBefore: 1
  web:
    sign:
      expiresIn: 7d
      notBefore: 1

# a list of other known repositories we can talk to
uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '**':
    access: $all
    publish: $all
    unpublish: $all

    # if package is not available locally, proxy requests to 'npmjs' registry
    proxy: npmjs

middlewares:
  audit:
    enabled: true

# log settings
logs:
  - { type: stdout, format: pretty, level: http }

listen:
  - 0.0.0.0:4873

Put these 3 files in a folder called verdaccio-test, then build and run everything:

cd /path/to/verdaccio-test
docker build -t testing/private-verdaccio-s3:latest .
docker-compose up -d

# Attach to logs
docker logs -f verdaccio-test_verdaccio_1

# use http://127.0.0.1:4874

Expected behavior

Packages should be deleted correctly.

Conclusion

I hope that everything is clear enough, please feel free to ask for further information.

Thank you for your time.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 31 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Here’s the breakdown for all bugs mentioned in the issue.

TLDR: jump to the last paragraph.

Bug 1 - npm unpublish -f failed with error, but package is actually removed by verdaccio.

Fact 2 - npm unpublish [-f] pkg@version work as expect after patching PR https://github.com/verdaccio/monorepo/pull/289

Bug 3 - npm publish -f (republish) failed with error, but the package@version file is removed by verdaccio (npm publish -f == npm unpublish -f). Thus run the publish command again will succeed, because verdaccio thinks you’re publishing a new version.

  • storage: all
  • registry.npm.org behaviors
    • republish same version is not allowed, returns error 403 - “Cannot publish over previously published version”
    • unpublish pkg@version is allowed. However, publish just unpublished pkg@version is not allowed, returns error 403. That means you can not use this trick to overwrite a published version. This is not considered as a bug, because it is consistent as above.
    • the only way to fix issues in a published version is to publish a new version instead.
  • reason
    • when work with registry.npm.org
      • the npm client requests publish -f, the server returns a 403 (permission error)
      • the npm client stops here
      npm http fetch PUT 403 https://registry.npmjs.org/com.bastianblokland.enumgenerator 6995ms
      npm verb stack Error: 403 Forbidden - PUT https://registry.npmjs.org/com.bastianblokland.enumgenerator - You cannot publish over the previously published versions: 1.6.1.
      npm verb stack     at C:\Program Files\nodejs\node_modules\npm\node_modules\npm-registry-fetch\check-response.js:104:15
      npm verb stack     at processTicksAndRejections (internal/process/task_queues.js:93:5)
      npm verb statusCode 403
      
    • when work with verdaccio,
      • the npm client requests publish -f, the server returns 409 (conflict error)
      • the npm client continue to work because of -f, fetches package info, the server returns 200
      • the npm client fetches package info again, the server returns 304 (no change)
      • the npm client requests to delete the pkg@version, the server returns 201 (successfully deleted)
      • the npm client raises version conflict error. Here the issue, the npm client shall skip the version check, because of -f.
      • this end up with the result that pkg@version being removed, with no pkg@version get published. (aka publish -f pkg@version == unpubish pkg@version)
      npm http fetch PUT 409 http://127.0.0.1:4873/com.bastianblokland.enumgenerator 78ms
      npm http fetch GET 200 http://127.0.0.1:4873/com.bastianblokland.enumgenerator?write=true 30ms
      npm WARN publish Forced publish over com.bastianblokland.enumgenerator@1.6.1
      npm http fetch GET 304 http://127.0.0.1:4873/com.bastianblokland.enumgenerator?write=true 35ms (from cache)
      npm http fetch DELETE 201 http://127.0.0.1:4873/com.bastianblokland.enumgenerator/-rev/3-3da368dddcba208e 13ms
      npm verb stack Error: Cannot publish com.bastianblokland.enumgenerator@undefined over existing version.
      npm verb stack     at ConflictError (C:\Program Files\nodejs\node_modules\npm\node_modules\libnpmpublish\publish.js:211:24)
      npm verb stack     at patchMetadata (C:\Program Files\nodejs\node_modules\npm\node_modules\libnpmpublish\publish.js:164:11)
      npm verb stack     at C:\Program Files\nodejs\node_modules\npm\node_modules\libnpmpublish\publish.js:63:22
      npm verb stack     at tryCatcher (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\util.js:16:23)
      npm verb stack     at Promise._settlePromiseFromHandler (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:517:31)
      npm verb stack     at Promise._settlePromise (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:574:18)
      npm verb stack     at Promise._settlePromise0 (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:619:10)
      npm verb stack     at Promise._settlePromises (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:699:18)
      npm verb stack     at _drainQueueStep (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\async.js:138:12)
      npm verb stack     at _drainQueue (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\async.js:131:9)
      npm verb stack     at Async._drainQueues (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\async.js:147:5)
      npm verb stack     at Immediate.Async.drainQueues [as _onImmediate] (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\async.js:17:14)
      npm verb stack     at processImmediate (internal/timers.js:439:21)
      npm verb pkgid com.bastianblokland.enumgenerator
      npm verb cwd C:\Users\favoy\Documents\projects\verdaccio\enum-generator-unity
      npm verb Windows_NT 10.0.18363
      npm verb argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "--registry" "http://127.0.0.1:4873" "publish" "-f" "--verbose"
      npm verb node v12.13.1
      npm verb npm  v6.12.1
      npm ERR! code EPUBLISHCONFLICT
      npm ERR! publish fail Cannot publish over existing version.
      npm ERR! publish fail Update the 'version' field in package.json and try again.
      npm ERR! publish fail
      npm ERR! publish fail To automatically increment version numbers, see:
      npm ERR! publish fail     npm help version
      npm verb exit [ 1, true ]
      
    • fix: won’t fix. It is not a verdaccio server issue, but a npm client issue.

Based on the analysis

  • I suggest verdaccio merge https://github.com/verdaccio/monorepo/pull/289 to fix npm unpublish -f bug for s3 storage.
  • @liarco because of the publish -f bug is not fixable by verdaccio. I suggest you run npm unpublish pkg@version, then npm publish to simulate the npm publish -f.

@liarco I have verified the issue only appears in S3, and fixed in https://github.com/verdaccio/monorepo/pull/289, see the PR description for details.

@flyfishMT

Can you explain how to fix the corrupted metadata? Thanks!

The metadata is just a JSON file, quite simple to read, if you want to get rid of one specific version, you need to remove it from different sections.

Json_Parser_Online

This is something that handles the client, the fields added by verdaccio are prefixed with underscore eg: _distfiles.

If you don’t have published packages in the X package, then just do as @liarco suggested, remove the package.json file or the whole folder.

Hi @juanpicado, I left a feedback yesterday about this: https://github.com/verdaccio/monorepo/pull/289#issuecomment-569282128

There’s still the problem with publish -f but it cannot be fixed (as @favoyang explained), by the way the 404 error on unpublish si gone!

Thank you! 😊

We will check next week, sorry guys I am super busy these days.

@liarco looks the same.

First of all, thank you very much for your time. I really appreciate your help with this issue.

Later today I’m gonna send you feedback about what you asked plus some updates that I got after further testing.

Absolutely! The files are exactly the ones that I shared above, but I’m gonna create a demo repo in a matter of minutes. Thank you.

In the mean while I tested one more time to be sure and it’s still the same:

My console:

MacBook-Air-di-Marco:test marco$ npm publish --force --registry http://127.0.0.1:4874
npm WARN using --force I sure hope you know what you are doing.
npm notice 
npm notice 📦  @test/beta-pack@0.0.0-dev
npm notice === Tarball Contents === 
npm notice 49B  index.js    
npm notice 229B package.json
npm notice 20B  README.md   
npm notice === Tarball Details === 
npm notice name:          @test/beta-pack                         
npm notice version:       0.0.0-dev                               
npm notice package size:  373 B                                   
npm notice unpacked size: 298 B                                   
npm notice shasum:        0999bc22c3e87051fa3bc945ca39cc276e9b0b3b
npm notice integrity:     sha512-G+b1QLXVycRYC[...]KEUVGCUkov2GQ==
npm notice total files:   3                                       
npm notice 
+ @test/beta-pack@0.0.0-dev
MacBook-Air-di-Marco:test marco$ npm publish --force --registry http://127.0.0.1:4874
npm WARN using --force I sure hope you know what you are doing.
npm notice 
npm notice 📦  @test/beta-pack@0.0.0-dev
npm notice === Tarball Contents === 
npm notice 49B  index.js    
npm notice 229B package.json
npm notice 20B  README.md   
npm notice === Tarball Details === 
npm notice name:          @test/beta-pack                         
npm notice version:       0.0.0-dev                               
npm notice package size:  373 B                                   
npm notice unpacked size: 298 B                                   
npm notice shasum:        0999bc22c3e87051fa3bc945ca39cc276e9b0b3b
npm notice integrity:     sha512-G+b1QLXVycRYC[...]KEUVGCUkov2GQ==
npm notice total files:   3                                       
npm notice 
npm WARN publish Forced publish over @test/beta-pack@0.0.0-dev
npm ERR! code EPUBLISHCONFLICT
npm ERR! publish fail Cannot publish over existing version.
npm ERR! publish fail Update the 'version' field in package.json and try again.
npm ERR! publish fail 
npm ERR! publish fail To automatically increment version numbers, see:
npm ERR! publish fail     npm help version

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/marco/.npm/_logs/2019-08-10T15_21_47_692Z-debug.log
MacBook-Air-di-Marco:test marco$ npm publish --force --registry http://127.0.0.1:4874
npm WARN using --force I sure hope you know what you are doing.
npm notice 
npm notice 📦  @test/beta-pack@0.0.0-dev
npm notice === Tarball Contents === 
npm notice 49B  index.js    
npm notice 229B package.json
npm notice 20B  README.md   
npm notice === Tarball Details === 
npm notice name:          @test/beta-pack                         
npm notice version:       0.0.0-dev                               
npm notice package size:  373 B                                   
npm notice unpacked size: 298 B                                   
npm notice shasum:        0999bc22c3e87051fa3bc945ca39cc276e9b0b3b
npm notice integrity:     sha512-G+b1QLXVycRYC[...]KEUVGCUkov2GQ==
npm notice total files:   3                                       
npm notice 
+ @test/beta-pack@0.0.0-dev
MacBook-Air-di-Marco:test marco$ 

Container logs:

 http <-- 201, user: liarco(192.168.160.1), req: 'PUT /@test%2fbeta-pack', bytes: 1380/53
 http <-- 409, user: liarco(192.168.160.1), req: 'PUT /@test%2fbeta-pack', error: this package is already present
 http <-- 200, user: liarco(192.168.160.1), req: 'GET /@test%2fbeta-pack?write=true', bytes: 0/592
 http <-- 304, user: liarco(192.168.160.1), req: 'GET /@test%2fbeta-pack?write=true', bytes: 0/0
 http <-- 201, user: liarco(192.168.160.1), req: 'DELETE /@test%2fbeta-pack/-rev/3-ef8a0450baebcd97', bytes: 0/30
 http <-- 201, user: liarco(192.168.160.1), req: 'PUT /@test%2fbeta-pack', bytes: 1380/53