go: meta: high rate of 5XX errors from go.googlesource.com since 2021-09-08

#!watchflakes
post <- `https://go\.googlesource\.com.*: The requested URL returned error: 5\d\d`

Some of the cmd/go long-mode tests end up cloning repos controlled by the Go project, such as golang.org/x/sys.

Other long-mode tests send requests to a dedicated test server (vcs-test.golang.org), various GitHub repos, packages hosted on rsc.io, and gopkg.in. The failure rate against vcs-test.golang.org is, as far as I can tell, negligible. The GitHub and gopkg.in tests for the most part check integration with services outside the control of the Go project. The rsc.io tests have a low but nontrivial failure rate (filed as #49954), and could perhaps be migrated to vcs-test.golang.org if need be.

However, the dependency on go.googlesource.com for the x repos is not easy to avoid, and as far as I can tell its failure rate since September has dwarfed all of those other services. Many of these failures already result in long (2m+) hangs that could otherwise push the test over its deadline, so I don’t think it would be viable to just add retries (#28194) — especially given that if the problem could be fixed by retrying, the server could presumably do that retry internally itself.

I would rather not lose coverage or add complexity to the tests by fetching the Go repos only through proxy.golang.org — @golang/release, is it viable to get a higher SLA on go.googlesource.com?

---

greplogs --dashboard -md -l -e 'https://go\.googlesource\.com.* 5\d\d\b' --since=2021-01-01

2022-01-10T17:06:19-9e7ea35/linux-386-longtest 2022-01-05T21:22:03-2b39d86/linux-386-longtest 2021-12-08T15:30:52-9fe77de/linux-amd64-longtest 2021-12-03T21:23:11-b3e1fbf/linux-386-longtest 2021-12-02T23:34:26-5f65520/linux-386-longtest 2021-12-02T22:03:11-d3bf627/linux-386-longtest 2021-12-02T20:49:56-bbe1be5/linux-386-longtest 2021-12-02T20:45:47-3c6295d/linux-386-longtest 2021-12-01T17:13:34-08ecdf7/linux-386-longtest 2021-11-30T20:04:58-7ccbcc9/linux-386-longtest 2021-11-30T18:49:39-5f63f16/linux-amd64-longtest 2021-11-30T18:09:02-931d80e/linux-386-longtest 2021-11-30T18:09:02-931d80e/linux-amd64-longtest 2021-11-29T22:02:48-2f6d382/linux-386-longtest 2021-11-24T21:11:45-b77f5f9/linux-386-longtest 2021-11-24T21:09:36-7e5331a/linux-amd64-longtest 2021-11-23T19:31:05-0f64c21/linux-amd64-longtest 2021-11-19T21:36:17-0f75a9e/linux-amd64-longtest 2021-11-16T14:33:48-50dac3b/linux-386-longtest 2021-11-11T20:20:42-3e94140/linux-386-longtest 2021-11-11T13:58:28-d76b1ac/linux-386-longtest 2021-11-10T18:24:14-6406e09/linux-amd64-longtest 2021-11-09T22:58:24-578ada4/linux-386-longtest 2021-11-09T19:01:20-526b2ef/linux-386-longtest 2021-11-09T18:23:16-55e6e82/linux-amd64-longtest 2021-11-08T16:44:14-7ee3f14/linux-amd64-longtest 2021-11-05T17:47:28-6f32d20/linux-386-longtest 2021-11-05T00:52:04-1c4cfd8/linux-386-longtest 2021-11-04T21:52:36-8ad0a7e/linux-386-longtest 2021-11-04T14:17:18-901bf29/linux-386-longtest 2021-11-03T20:30:17-7f2463c/linux-amd64-longtest 2021-11-02T13:43:24-4d7bf41/linux-386-longtest 2021-11-01T19:42:30-5f1bcd1/linux-386-longtest 2021-11-01T15:55:29-0ae4c7f/linux-amd64-longtest 2021-10-29T19:35:47-5ec139f/linux-386-longtest 2021-10-29T18:56:29-6afdf01/linux-386-longtest 2021-10-29T18:35:59-5ec2427/linux-386-longtest 2021-10-29T18:35:20-9ac1ee2/linux-386-longtest 2021-10-29T18:23:03-413672f/linux-386-longtest 2021-10-28T20:27:44-5e21728/linux-amd64-longtest 2021-10-28T16:54:29-5fce1d9/linux-386-longtest 2021-10-28T01:15:26-b2fe2eb/linux-386-longtest 2021-10-27T21:37:54-749f6e9/linux-386-longtest 2021-10-27T20:24:48-a155a30/linux-386-longtest 2021-10-27T20:03:17-68bd512/linux-386-longtest 2021-10-27T19:34:49-ffd2284/linux-386-longtest 2021-10-27T19:18:20-514ebae/linux-386-longtest 2021-10-27T18:08:49-bb49eb3/linux-386-longtest 2021-10-26T20:12:32-f8779b9/linux-amd64-longtest 2021-10-26T17:10:45-76cef81/linux-amd64-longtest 2021-10-26T14:24:17-283d8a3/linux-386-longtest 2021-10-25T17:46:41-9012996/linux-amd64-longtest 2021-10-22T00:57:18-9ff91b9/linux-amd64-longtest 2021-10-21T19:08:43-b8da7e4/linux-386-longtest 2021-10-21T02:35:47-fca46d0/linux-386-longtest 2021-10-12T20:20:41-d032b2b/linux-386-longtest 2021-10-12T14:32:53-36a265a/linux-386-longtest 2021-10-05T19:28:36-7ae83c8/linux-386-longtest 2021-10-01T14:30:33-8d494b0/linux-386-longtest 2021-09-27T19:42:34-315cec2/linux-386-longtest 2021-09-21T14:02:52-7d67f8d/linux-amd64-longtest 2021-09-15T17:32:52-6196979/linux-386-longtest 2021-09-15T17:32:52-6196979/linux-amd64-longtest 2021-09-15T16:32:27-72bb818/linux-386-longtest 2021-09-15T16:32:27-72bb818/linux-amd64-longtest 2021-09-14T23:20:18-2933c45/linux-386-longtest 2021-09-14T23:07:15-137543b/linux-amd64-longtest 2021-09-14T23:03:28-3a72175/linux-amd64-longtest 2021-09-13T21:48:41-42057e9/linux-386-longtest 2021-09-13T20:42:44-81a4fe6/linux-386-longtest 2021-09-09T01:25:40-d4579c2-d62866e/darwin-arm64-11_0-toothrot 2021-09-08T16:59:47-d419f9c/linux-amd64-longtest 2021-09-08T16:19:36-409434d/linux-386-longtest 2021-05-26T22:43:54-1d5298d/linux-amd64-longtest