dist: 402 Payment Required from cloudsmith

It’s strange that we’ve hit our 2TB bandwidth quota in merely 3 days. Our apt repo appears to be abused as part of CI builds. If any of you are using the apt package in CI, please change your approach.

Until then, we’re working on a solution.

N.B. Please refrain from making more comments of “me too” unless you’re adding more details or context.

Hey everyone 👋

@francislavoie - I’ve increased the limit once more to get everyone unblocked. I’m also going to take some time to look at our logs to determine where the majority of requests are coming from.

It’s strange that we’ve hit our 2TB bandwidth quota in merely 3 days

There are many public mirrors that would host your repository for free, with no bandwidth limits. Have you considered a distributed approach?

Can you elaborate?

For example, the many organizations listed here or here provide their services for free, and would very likely be happy to host your repo as well. Then you just need to refer users to a nearby healthy mirror.

I gave up using Cloudsmith and started to install from releases directly (Ubuntu 22.04)

CADDY_VERSION="2.7.6"
CADDY_ARCH="amd64"

wget -q "https://github.com/caddyserver/caddy/releases/download/v${CADDY_VERSION}/caddy_${CADDY_VERSION}_linux_${CADDY_ARCH}.deb"
apt install "./caddy_${CADDY_VERSION}_linux_${CADDY_ARCH}.deb"

We’re also going to ask that anyone using the Cloudsmith repos in CI to use xcaddy or to download from GitHub releases instead.

We’re not sure but it seems likely that an extremely busy CI or testing environment is relentlessly downloading Caddy from Cloudsmith.

Talk to Fastly. They can front the Cloudsmith repo as a CDN. It was something I worked with them on for the Alpine Linux package CDN.

@compuguy the best solution we can offer for now is that you download the .deb from our Github releases (which is what we upload to Cloudsmith anyway).

We’re still waiting for communication from Cloudsmith about this.

This is due to abusive users hammering Cloudsmith. There’s absolutely no way we received 2.5TB of legitimate traffic within 3 days. For the record, here’s our usage tracking for the past year:

/cc @lskillen @BartoszBlizniak from Cloudsmith

Cloudsmith has notified me that they have implemented measures, so we appreciate their help and cooperation 😃

It might be still more helpful to talk to CDNs, who have expertise and host projects like these anyway. Importantly they also have expertise in dealing with abuse, and still a desire for good availability and speed, as developer targeted mindshare.

Fastly definitely comes to mind, runs mirrors, Cloudflare might.

This seems like a separate program, https://www.fastly.com/fast-forward.