go: x/build/cmd/gomote: 502 Bad Gateway error

I often encounter this 502 Bad Gateway error while actively working with a buildlet created using gomote. I guess this is because I was holding the buildlet too long (~30min) and there is a hidden time limit on each buildlet. If so, we need a better error message than this:

$ gomote ls user-hakim-windows-amd64-2016-0
Error running ls: 502 Bad Gateway:

And, gomote list still shows the lease is not yet expired. That’s misleading.

$ gomote list
user-hakim-windows-amd64-2016-0	windows-amd64-2016	host-windows-amd64-2016	expires in 29m49.611152102s

About this issue

Original URL
State: open
Created 6 years ago
Reactions: 2
Comments: 17 (14 by maintainers)

Commits related to this issue

cmd/coordinator: make gomote proxy 502 error include detail Go 1.11 added ReverseProxy.ErrorHandler; use it to make the httputil.ReverseProxy failures print the underlying error back to the (trusted)... — committed to golang/build by bradfitz 5 years ago
buildlet, cmd/coordinator: stop using legacy Request.Cancel for cancellation Maybe this will solve the golang/go#28365 problems. But at least it gets us into codepaths that are known & trusted, and r... — committed to golang/build by bradfitz 5 years ago
cmd/coordinator: make gomote proxy 502 error include detail Go 1.11 added ReverseProxy.ErrorHandler; use it to make the httputil.ReverseProxy failures print the underlying error back to the (trusted)... — committed to codebien/build by bradfitz 5 years ago
buildlet, cmd/coordinator: stop using legacy Request.Cancel for cancellation Maybe this will solve the golang/go#28365 problems. But at least it gets us into codepaths that are known & trusted, and r... — committed to codebien/build by bradfitz 5 years ago

Most upvoted comments

To add to my previous message, I’ve noticed something very strange. I had spun up 25 windows-amd64-2016 builders and they ran fine for a few minutes. Then all but 3 of them died with errors like in my previous message. My gopool program kept trying to recreate the 22 failed builders, but I never saw one that lasted more than 2.1 seconds after creation. All but the 3 kept failing with 502 errors. I believe exactly the same thing had happened in a previous run, though I wasn’t watching it as closely.

While this was happening, I saw the following on farmer:

host-windows-amd64-2016: 25 waiting (oldest 9m55s, newest 5s, progress 7s)
   gomote: 22 (oldest 44s, newest 5s)
   try: 3 (oldest 9m55s, newest 9m55s)

aclements on Jan 20, 2022

Per above — I think this is fixed. Setting to WaitingForInfo to see if anyone chimes in with recent gomote trouble.

bcmills on Jun 21, 2021