verdaccio: aws-s3-storage Publish fails when optional accessKeyId & secretAccessKey are not provided

Describe the bug aws-s3-storage - Publish fails when optional accessKeyId & secretAccessKey are not provided. This does not happen with Verdaccio V3 and it’s S3 storage plugin on our Cloud Set up.

Looking in the Verdaccio log the plugin is reported as installed however throws the errors below

 warn --- config file  - /etc/verdaccio/config.yaml
 warn --- Plugin successfully loaded: verdaccio-aws-s3-storage
 error-=- s3: [_getData] err: no such package available
 error-=- s3: [_getData] err 404 create new database
 warn --- Plugin successfully loaded: verdaccio-htpasswd
 warn --- Plugin successfully loaded: verdaccio-audit
 warn --- http address - http://0.0.0.0:4873/ - verdaccio/4.0.0
(node:702) [DEP0106] DeprecationWarning: crypto.createDecipher is deprecated.

Despite this the S3 Bucket is access and the verdaccio-s3-db.json is uploaded to the bucket inside the prefixed folder specified

When I try to publish the verdaccio log produces this output whether I log in or not; this particular output I have logged in using npm via ssh

 http <-- 201, user: phil(212.58.232.179 via 127.0.0.1), req: 'PUT /-/user/org.couchdb.user:phil', byt
es: 127/73
 http <-- 201, user: phil(212.58.232.179 via 127.0.0.1), req: 'PUT /-/user/org.couchdb.user:phil', byt
es: 127/73
 error-=- s3: [S3PackageManager _getData] aws The specified key does not exist.
 error-=- s3: [S3PackageManager _getData] The specified key does not exist.
 error-=- s3: [S3PackageManager readPackage] no such package available
 error-=- error reading  package.json: no such package available
 http <-- 500, user: phil(212.58.232.179 via 127.0.0.1), req: 'PUT /s3-contacts-upload-demo', error: I
nternal Server Error
 http <-- 500, user: phil(212.58.232.179 via 127.0.0.1), req: 'PUT /s3-contacts-upload-demo', error: I
nternal Server Error

To Reproduce

  1. Add verdaccio-aws-s3-storage to an EC2 inside a vpn behind a network load balancer with permissions to write to the bucket.
  2. The Bucket has ‘Block all public access’ turn on
  3. Access is via the bucket owner’s Canonical ID
  4. The bucket policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::193231631354:role/int-ise-sandbox-verdaccio-infrastructure-EC2Role-V88WA4WD8IMB"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::int-bbc-npm-storage-poc3",
                "arn:aws:s3:::int-bbc-npm-storage-poc3/*"
            ]
        }
    ]
}
  1. No CORS configuration
  2. Upload is done via ssh and developer certificates
npm config set registry 'int-DCA-verdaccio-v4-nlb-53aae13bfe866c8a.elb.eu-west-1.amazonaws.com'
npm set ca null
alias npm="npm --registry https://int-dca-verdaccio-v4-nlb-53aae13bfe866c8a.elb.eu-west-1.amazonaws.com/ --cert=\"$(cat /etc/pki/certificate.pem)\" --key=\"$(cat /etc/pki/certificate.pem)\" --cafile=/etc/pki/tls/certs/ca-bundle.crt"
npm publish

Expected behavior I would expect to be able to publish as I can with verdaccio v3 and the s3 plugin for that version

Configuration File (cat ~/.config/verdaccio/config.yaml) I have used both storage: ./storage & storage: verdaccio/storage with the same results; this is the same config I use for the Verdaccio v3 with the appropriate s3 plugin for that version. Only difference is the plugin name under ‘store’

storage: ./storage
store:
 aws-s3-storage:
    bucket: int-bbc-npm-storage-poc3
    keyPrefix: ISE
    region: eu-west-1 # optional, will use aws s3's default behavior if not specified
    s3ForcePathStyle: false # optional, will use path style URLs for S3 objects

packages:
  '@ise-release-internal/*':
    access: bbc
    publish: bbc
    proxy: npmjs
  '@ise-release--external/*':
    access: $all
    publish: bbc
    proxy: npmjs
  '**':
    access: $all
    publish: $all
    proxy: npmjs
uplinks:
  npmjs:
    url: https://registry.npmjs.org/
    cache: true
logs:
  - {type: stdout, format: pretty, level: http}
web:
  enable: true
  title: BBC CDA NPM Registry
  scope:
  primary_color: "#323232"
listen:
    0.0.0.0:4873
max_body_size: 100mb
auth:
  htpasswd:
    file: /usr/lib/verdaccio/htpasswd
middlewares:
  audit:
    enabled: true

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

So my last comment was based on the errors that people had noted, but I have managed to get v5 29.2 working with an assumed role via our kubernetes setup. No errors and 26th using the guidelines for version 5 migration, got it working with no errors.

Thanks for the update!

Know what you mean about too many request!

P.

On 20 Feb 2020, at 15:45, Juan Picado @jotadeveloper notifications@github.com wrote:

Not much from my side, sorry have to many request, hopefully someone else can look into this or well, just wait a bit more I clean up my queue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/verdaccio/monorepo/issues/321?email_source=notifications&email_token=AEBW5JTNOHGZIY7BUCOHWNTRD2QQJA5CNFSM4KRNBD7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMO2PHA#issuecomment-589146012, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEBW5JSPREUB4N36NCI2Z3LRD2QQJANCNFSM4KRNBD7A.

Hey @juanpicado hope this is more readable

Describe the bug aws-s3-storage - Publish fails when optional accessKeyId & secretAccessKey are not provided. This does not happen with Verdaccio V3 and it’s S3 storage plugin on our Cloud Set up.

Looking in the Verdaccio log the plugin is reported as installed however throws the errors below

warn — config file - /etc/verdaccio/config.yaml warn — Plugin successfully loaded: verdaccio-aws-s3-storage error-=- s3: [_getData] err: no such package available error-=- s3: [_getData] err 404 create new database warn — Plugin successfully loaded: verdaccio-htpasswd warn — Plugin successfully loaded: verdaccio-audit warn — http address - http://0.0.0.0:4873/ - verdaccio/4.0.0 (node:702) [DEP0106] DeprecationWarning: crypto.createDecipher is deprecated.

Despite this the S3 Bucket is access and the verdaccio-s3-db.json is uploaded to the bucket inside the prefixed folder specified

When I try to publish the verdaccio log produces this output whether I log in or not; this particular output I have logged in using npm via ssh

http <-- 201, user: phil(212.58.232.179 via 127.0.0.1), req: ‘PUT /-/user/org.couchdb.user:phil’, byt es: 127/73 http <-- 201, user: phil(212.58.232.179 via 127.0.0.1), req: ‘PUT /-/user/org.couchdb.user:phil’, byt es: 127/73 error-=- s3: [S3PackageManager _getData] aws The specified key does not exist. error-=- s3: [S3PackageManager _getData] The specified key does not exist. error-=- s3: [S3PackageManager readPackage] no such package available error-=- error reading package.json: no such package available http <-- 500, user: phil(212.58.232.179 via 127.0.0.1), req: ‘PUT /s3-contacts-upload-demo’, error: I nternal Server Error http <-- 500, user: phil(212.58.232.179 via 127.0.0.1), req: ‘PUT /s3-contacts-upload-demo’, error: I nternal Server Error>

To Reproduce

  1. Add verdaccio-aws-s3-storage to an EC2 inside a vpn behind a network load balancer with permissions to write to the bucket.
  2. The Bucket has ‘Block all public access’ turn on
  3. Access is via the bucket owner’s Canonical ID
  4. The bucket policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::193231631354:role/int-ise-sandbox-verdaccio-infrastructure-EC2Role-V88WA4WD8IMB" }, "Action": "s3:", "Resource": [ "arn:aws:s3:::int-bbc-npm-storage-poc3", "arn:aws:s3:::int-bbc-npm-storage-poc3/" ] } ] }
  5. No CORS configuration
  6. Upload is done via ssh and developer certificates

Expected behavior I would expect to be able to publish as I can with verdaccio v3 and the s3 plugin for that version

Configuration File (cat ~/.config/verdaccio/config.yaml) I have used both storage: ./storage & storage: verdaccio/storage with the same results; this is the same config I use for the Verdaccio v3 with the appropriate s3 plugin for that version. Only difference is the plugin name under ‘store’

storage: ./storage store: aws-s3-storage: bucket: int-bbc-npm-storage-poc3 keyPrefix: ISE region: eu-west-1 # optional, will use aws s3’s default behavior if not specified s3ForcePathStyle: false # optional, will use path style URLs for S3 objects packages: ‘@ise-release-internal/’: access: bbc publish: bbc proxy: npmjs ‘@ise-release–external/’: access: $all publish: bbc proxy: npmjs ‘**’: access: $all publish: $all proxy: npmjs uplinks: npmjs: url: https://registry.npmjs.org/ cache: true logs: {type: stdout, format: pretty, level: http} web: enable: true title: BBC CDA NPM Registry scope: primary_color: “#323232” listen: 0.0.0.0:4873 max_body_size: 100mb auth: htpasswd: file: /usr/lib/verdaccio/htpasswd middlewares: audit: enabled: true