concourse: Jobs error with ContainerNotFoundError

A few of our jobs have errored with the following message:

Backend error: Exit status: 404, message: {"Type":"ContainerNotFoundError","Message":"unknown handle: 7uv7ag0a28j","Handle":"7uv7ag0a28j"}

Our deployment of concourse is running on AWS but the jobs are all failing on a worker running in vsphere. These are the logs from garden on the vsphere worker for that container:

root@4b738971-0ed1-48bc-93e3-402cb8dc9cb4:/var/vcap/sys/log# grep 7uv7ag0a28l garden/*
garden/garden.stdout.log:{"timestamp":"1451322764.911316395","source":"garden-linux","message":"garden-linux.pool.acquire.creating","log_level":1,"data":{"handle":"7uv7ag0a28l","session":"11.98"}}
garden/garden.stdout.log:{"timestamp":"1451322764.911461353","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.subnet-selecting","log_level":1,"data":{"allocated-subnets":["10.254.0.0/30"],"handle":"7uv7ag0a28l","session":"11.98.1.1"}}
garden/garden.stdout.log:{"timestamp":"1451322764.911491632","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.subnet-selected","log_level":1,"data":{"handle":"7uv7ag0a28l","session":"11.98.1.1","subnet":"10.254.0.4/30"}}
garden/garden.stdout.log:{"timestamp":"1451322764.911521673","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.ip-selecting","log_level":1,"data":{"allocated-ips":null,"handle":"7uv7ag0a28l","session":"11.98.1.1"}}
garden/garden.stdout.log:{"timestamp":"1451322764.911544085","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.ip-selected","log_level":1,"data":{"handle":"7uv7ag0a28l","ip":"10.254.0.6","session":"11.98.1.1"}}
garden/garden.stdout.log:{"timestamp":"1451322764.911564350","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.new-allocated","log_level":1,"data":{"allocated-ips":["10.254.0.6"],"handle":"7uv7ag0a28l","session":"11.98.1.1"}}
garden/garden.stdout.log:{"timestamp":"1451322764.911591053","source":"garden-linux","message":"garden-linux.pool.acquire.acquired-pool-resources","log_level":1,"data":{"handle":"7uv7ag0a28l","session":"11.98"}}
garden/garden.stdout.log:{"timestamp":"1451322766.814042568","source":"garden-linux","message":"garden-linux.pool.acquire.created","log_level":1,"data":{"handle":"7uv7ag0a28l","session":"11.98"}}
garden/garden.stdout.log:{"timestamp":"1451322767.248266459","source":"garden-linux","message":"garden-linux.garden-server.get-properties.got-properties","log_level":1,"data":{"handle":"7uv7ag0a28l","session":"13.49215"}}
garden/garden.stdout.log:{"timestamp":"1451322767.400682211","source":"garden-linux","message":"garden-linux.garden-server.stream-in.streamed-in","log_level":1,"data":{"destination":"/tmp/build/9e8bdca8-6fee-4bb0-5a0f-d638cf082547","handle":"7uv7ag0a28l","session":"13.49216","user":""}}
garden/garden.stdout.log:{"timestamp":"1451322770.576896667","source":"garden-linux","message":"garden-linux.garden-server.stream-in.streamed-in","log_level":1,"data":{"destination":"/tmp/build/9e8bdca8-6fee-4bb0-5a0f-d638cf082547/micropcf/.","handle":"7uv7ag0a28l","session":"13.49217","user":""}}
garden/garden.stdout.log:{"timestamp":"1451322770.770728111","source":"garden-linux","message":"garden-linux.garden-server.stream-in.streamed-in","log_level":1,"data":{"destination":"/tmp/build/9e8bdca8-6fee-4bb0-5a0f-d638cf082547/micropcf-ci/.","handle":"7uv7ag0a28l","session":"13.49218","user":""}}
garden/garden.stdout.log:{"timestamp":"1451322770.928391218","source":"garden-linux","message":"garden-linux.garden-server.stream-in.streamed-in","log_level":1,"data":{"destination":"/tmp/build/9e8bdca8-6fee-4bb0-5a0f-d638cf082547/current-base-box-version/.","handle":"7uv7ag0a28l","session":"13.49219","user":""}}
garden/garden.stdout.log:{"timestamp":"1451322771.087344170","source":"garden-linux","message":"garden-linux.garden-server.stream-in.streamed-in","log_level":1,"data":{"destination":"/tmp/build/9e8bdca8-6fee-4bb0-5a0f-d638cf082547/next-base-box-version/.","handle":"7uv7ag0a28l","session":"13.49220","user":""}}
garden/garden.stdout.log:{"timestamp":"1451322771.247172356","source":"garden-linux","message":"garden-linux.garden-server.stream-in.streamed-in","log_level":1,"data":{"destination":"/tmp/build/9e8bdca8-6fee-4bb0-5a0f-d638cf082547/base-box-commit/.","handle":"7uv7ag0a28l","session":"13.49221","user":""}}
garden/garden.stdout.log:{"timestamp":"1451322771.401214838","source":"garden-linux","message":"garden-linux.container.next pid","log_level":1,"data":{"handle":"7uv7ag0a28l","pid":1,"session":"262"}}
garden/garden.stdout.log:{"timestamp":"1451322771.407301426","source":"garden-linux","message":"garden-linux.garden-server.run.spawned","log_level":1,"data":{"handle":"7uv7ag0a28l","id":"1","session":"13.49223","spec":{"Path":"micropcf-ci/tasks/build-images/run.sh","Dir":"/tmp/build/9e8bdca8-6fee-4bb0-5a0f-d638cf082547","User":"root","Limits":{},"TTY":{}}}}
garden/garden.stdout.log:{"timestamp":"1451325767.289963245","source":"garden-linux","message":"garden-linux.garden-server.reaping","log_level":1,"data":{"grace-time":"5m0s","handle":"7uv7ag0a28l","session":"13"}}
garden/garden.stdout.log:{"timestamp":"1451325767.290051937","source":"garden-linux","message":"garden-linux.container.cleanup.done","log_level":1,"data":{"handle":"7uv7ag0a28l","session":"262.8"}}
garden/garden.stdout.log:{"timestamp":"1451325767.290072918","source":"garden-linux","message":"garden-linux.pool.release.releasing","log_level":1,"data":{"id":"7uv7ag0a28l","session":"11.100"}}
garden/garden.stdout.log:{"timestamp":"1451325767.374727011","source":"garden-linux","message":"garden-linux.garden-server.run.exited","log_level":1,"data":{"handle":"7uv7ag0a28l","id":"1","session":"13.49223","status":255}}
garden/garden.stdout.log:{"timestamp":"1451325767.459994793","source":"garden-linux","message":"garden-linux.oven-cleaner.remove.start","log_level":1,"data":{"ID":"7uv7ag0a28l","session":"5.50"}}
garden/garden.stdout.log:{"timestamp":"1451325767.492785931","source":"garden-linux","message":"garden-linux.pool.release.subnet-pool.release.changing-allocated-subnets","log_level":1,"data":{"allocated-subnets":null,"id":"7uv7ag0a28l","session":"11.100.2.1"}}
garden/garden.stdout.log:{"timestamp":"1451325767.492814064","source":"garden-linux","message":"garden-linux.pool.release.released","log_level":1,"data":{"id":"7uv7ag0a28l","session":"11.100"}}
garden/garden.stdout.log:{"timestamp":"1451326424.582373142","source":"garden-linux","message":"garden-linux.garden-server.attach.failed","log_level":2,"data":{"error":"unknown handle: 7uv7ag0a28l","handle":"7uv7ag0a28l","session":"13.49525"}}
garden/garden.stdout.log:{"timestamp":"1451326426.321076632","source":"garden-linux","message":"garden-linux.garden-server.set-grace-time.failed","log_level":2,"data":{"error":"unknown handle: 7uv7ag0a28l","handle":"7uv7ag0a28l","session":"13.49528"}}

We initially suspected that the issue was the network between AWS and vsphere, but this looks like a problem with the local processes on the worker.