unpkg: Getting 503's and (incorrect) 404's
Somewhat related to https://github.com/unpkg/unpkg.com/issues/35.
I’m working on an in-browser NPM client for System that intelligently downloads all of the files the app needs from Unpkg (instead of downloading everything w/ the actual tarball). An important note is that we are not downloading UMD/dist builds, but instead downloading all files required from the main/module/typings field in the package JSON. For a large package like Angular it will typically make ~100-200 requests within a 5s period, and performance/reliability typically isn’t a problem.
However, I’ve noticed that Unpkg will send back 503’s and incorrect 404’s from time to time, and I’m thinking that it might be due to a race condition that happens when Unpkg encounters a package it’s seen for the first time. For example, about an hour ago I performed an install of glamorous@3.13.1 and got back a 503 for this file — but after refreshing that file’s URL, the file contents were successfully sent back.
Other times, instead of a 503, I’ll get back a 404 with the Not found: file "/folder/file.ext" in package x@0.0.0 (with actual file path & package name in error) — but upon refreshing the URL, the file contents come back totally fine.
@mjackson are you aware of any issues that might be causing this? (i.e. rate limits, race conditions, etc). I’m planning on running unpkg locally and seeing if I can replicate the errors there w/ logging, but any info you can provide would be much appreciated.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 28 (12 by maintainers)
@EricSimons your editor looks really cool! I was playing with it a little and tried to install a few packages in there, but after trying to install one larger package it seems like your api went down and wouldn’t return anything other than 504. Sorry if I broke anything 😆
If you can’t get this to work reliably with unpkg, I’d love to see if you experience the same problems when downloading files from https://cdn.jsdelivr.net/npm/ - we took a lot of time making sure requests are handled correctly in every edge case we could think of.
Just made a PR (#39) that should fix this issue, @mjackson lmk what you think!
Just read through the Medium post, looks pretty awesome @EricSimons 😃 best of luck with the launch! I’m definitely working on getting that bug fixed and I apologize for being so slow. I should’ve had a fix out by now. Been busy this summer! I will definitely be trying to get it fixed this week.
@mjackson I think it could it be another race condition introduced from
https://github.com/unpkg/unpkg.com/commit/4aba460a6309ece485603bab3017742b1443e5b7#diff-6e41c60c01cb116a8dc152f784096f77R25
where requests for a package that is currently being extracted by another request aren’t waiting until the extraction fully completes, and instead getting a local fs cache-hit while the extraction is still taking place for that file, causing the server to send only what’s been extracted so far.
Just double checked and I don’t even have rate limiting enabled on my Cloudflare account
I don’t believe that rate limiting should ever be a problem on unpkg.com. I haven’t set up rate limiting on the CDN, but maybe Cloudflare does some rate limiting by default. I’ll double check with them.
I’m thinking it could be a race condition somewhere. When a lot of requests come in for the same package we have to do a bunch of work to download it, decompress it, save it to disk, and then serve the file. I wonder if the problem could be that many requests for the same package trigger multiple fetches and the files are being overwritten on disk, causing a race condition. In this case, ideally only the first request should trigger the fetch and every subsequent request should be able to just wait until that work is done. But who knows if this is actually the problem or not.
But ya, you should be able to just download and run everything locally. Hopefully that will let you reproduce the error. And please let me know if you have trouble running it locally. You’ll probably need to run a Redis server and set the
$REDIS_URLenvironment variable to boot everything. I can’t remember off hand if there’s any other specific setup you’ll need.