concourse: /opt/resource/check failed to ping registry: 2 error(s) occurred

I’m trying to run “Hello world!” example with “Standalone Binaries”. It fails with strange errors: screenshot from 2016-04-01 18-56-38

Some logs that may help:

uname -a
Linux nexus 4.4.0-16-generic #32-Ubuntu SMP Thu Mar 24 22:38:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[Unit]
Description=Concourse worker
After=network.target

[Service]
ExecStart=/bin/concourse worker --tsa-public-key host_key.pub --tsa-worker-private-key worker_key --work-dir /opt/concourse/worker
Type=simple
RestartSec=3s
Restart=always
WorkingDirectory=/opt/concourse/worker

[Install]
WantedBy=multi-user.target

say-hello
resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to ping registry: 2 error(s) occurred:

* ping https: Get https://registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on 127.0.2.1:53: read udp 127.0.0.1:33435->127.0.2.1:53: read: connection refused
* ping http: Get http://registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on 127.0.2.1:53: read udp 127.0.0.1:40087->127.0.2.1:53: read: connection refused

-- The start-up result is done.
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.569760084","source":"baggageclaim","message":"baggageclaim.listening","log_level":1,"data":{"addr":"0.0.0.0:7788"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.588144064","source":"garden-linux","message":"garden-linux.failed-to-parse-pool-state","log_level":2,"data":{"error":"openning state file: open /opt/concourse/worker/linux/state/port_pool.json: no such file or directory"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.591316700","source":"tsa","message":"tsa.connection.forward-worker.register.start","log_level":1,"data":{"session":"4.1.4","worker-address":"127.0.0.1:35264","worker-platform":"linux","worker-tags":""}}
Apr 01 18:37:55 nexus concourse[29079]: 2016/04/01 18:37:55 failed to forward remote connection: dial tcp 0.0.0.0:7777: getsockopt: connection refused
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.591898680","source":"tsa","message":"tsa.connection.forward-worker.register.failed-to-fetch-containers","log_level":2,"data":{"error":"Get http://api/containers: EOF","session":"4.1.4"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.591985941","source":"garden-linux","message":"garden-linux.retain.starting","log_level":1,"data":{"session":"10"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.592156172","source":"garden-linux","message":"garden-linux.retain.retained","log_level":1,"data":{"session":"10"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.591971874","source":"tsa","message":"tsa.connection.forward-worker.register.done","log_level":1,"data":{"session":"4.1.4","worker-address":"127.0.0.1:35264","worker-platform":"linux","worker-tags":""}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.766176224","source":"garden-linux","message":"garden-linux.pool.restore.subnet-pool.release.new-allocated","log_level":1,"data":{"allocated-ips":["10.254.0.2"],"id":"ga33g4g3nd3","session":"8.1.1.1","subnet":"10.254.0.0/30"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.810664415","source":"garden-linux","message":"garden-linux.container.restore.restored","log_level":1,"data":{"handle":"ga33g4g3nd3","session":"14.1"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.811105728","source":"garden-linux","message":"garden-linux.pool.restore.subnet-pool.release.new-allocated","log_level":1,"data":{"allocated-ips":["10.254.0.6"],"id":"ga33g4g3nd4","session":"8.2.1.1","subnet":"10.254.0.4/30"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.853553295","source":"garden-linux","message":"garden-linux.container.restore.restored","log_level":1,"data":{"handle":"ga33g4g3nd4","session":"16.1"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.853964090","source":"garden-linux","message":"garden-linux.pool.restore.subnet-pool.release.new-allocated","log_level":1,"data":{"allocated-ips":["10.254.0.10"],"id":"ga5861ul42c","session":"8.3.1.1","subnet":"10.254.0.8/30"}}
Apr 01 18:37:55 nexus concourse[29079]: {"timestamp":"1459525075.900469542","source":"garden-linux","message":"garden-linux.container.restore.restored","log_level":1,"data": }
// ...
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.674717903","source":"garden-linux","message":"garden-linux.pool.acquire.creating","log_level":1,"data":{"handle":"ga6ab3n6gua","session":"8.4"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.674797535","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.subnet-selecting","log_level":1,"data":{"allocated-subnets":["10.254.0.0/30","10.254.0.4/30","10.254.0.8/30"],"handle":"ga6ab3n6gua","session":"8.4.1.1"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.674864531","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.subnet-selected","log_level":1,"data":{"handle":"ga6ab3n6gua","session":"8.4.1.1","subnet":"10.254.0.12/30"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.674902439","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.ip-selecting","log_level":1,"data":{"allocated-ips":null,"handle":"ga6ab3n6gua","session":"8.4.1.1"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.674956083","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.ip-selected","log_level":1,"data":{"handle":"ga6ab3n6gua","ip":"10.254.0.14","session":"8.4.1.1"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.674999714","source":"garden-linux","message":"garden-linux.pool.acquire.subnet-pool.acquire.new-allocated","log_level":1,"data":{"allocated-ips":["10.254.0.14"],"handle":"ga6ab3n6gua","session":"8.4.1.1"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.675029993","source":"garden-linux","message":"garden-linux.pool.acquire.acquired-pool-resources","log_level":1,"data":{"handle":"ga6ab3n6gua","session":"8.4"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.675055027","source":"garden-linux","message":"garden-linux.pool.acquire.running-graph-cleanup","log_level":1,"data":{"handle":"ga6ab3n6gua","session":"8.4"}}
Apr 01 18:48:28 nexus concourse[29079]: {"timestamp":"1459525708.675087929","source":"garden-linux","message":"garden-linux.pool.acquire.gc.start","log_level":1,"data":{"handle":"ga6ab3n6gua","session":"8.4.2"}}
Apr 01 18:48:29 nexus concourse[29079]: {"timestamp":"1459525709.341547012","source":"garden-linux","message":"garden-linux.pool.acquire.created","log_level":1,"data":{"handle":"ga6ab3n6gua","session":"8.4"}}
Apr 01 18:48:29 nexus concourse[29079]: {"timestamp":"1459525709.581398487","source":"garden-linux","message":"garden-linux.garden-server.create.created","log_level":1,"data":{"request":{"Handle":"","GraceTime":0,"RootFSPath":"/opt/concourse/worker/linux/resource-images/docker-image","BindMounts":null,"Network":"","Privileged":true,"Limits":{"bandwidth_limits":{},"cpu_limits":{},"disk_limits":{},"memory_limits":{}}},"session":"11.23"}}
Apr 01 18:48:29 nexus concourse[29079]: {"timestamp":"1459525709.596529722","source":"garden-linux","message":"garden-linux.garden-server.get-properties.got-properties","log_level":1,"data":{"handle":"ga6ab3n6gua","session":"11.25"}}
Apr 01 18:48:29 nexus concourse[29079]: {"timestamp":"1459525709.598648787","source":"garden-linux","message":"garden-linux.container.next pid","log_level":1,"data":{"handle":"ga6ab3n6gua","pid":1,"session":"22"}}
Apr 01 18:48:29 nexus concourse[29079]: {"timestamp":"1459525709.608063698","source":"garden-linux","message":"garden-linux.garden-server.run.spawned","log_level":1,"data":{"handle":"ga6ab3n6gua","id":"1","session":"11.26","spec":{"Path":"/opt/resource/check","Dir":"","User":"root","Limits":{},"TTY":null}}}
Apr 01 18:48:57 nexus concourse[29079]: {"timestamp":"1459525737.020558834","source":"tsa","message":"tsa.connection.forward-worker.heartbeat.start","log_level":1,"data":{"session":"4.1.27","worker-address":"127.0.0.1:35264","worker-platform":"linux","worker-tags":""}}
Apr 01 18:48:57 nexus concourse[29079]: {"timestamp":"1459525737.022392988","source":"tsa","message":"tsa.connection.forward-worker.heartbeat.reached-worker","log_level":0,"data":{"session":"4.1.27","took":"1.663044ms"}}
Apr 01 18:48:57 nexus concourse[29079]: {"timestamp":"1459525737.033593178","source":"tsa","message":"tsa.connection.forward-worker.heartbeat.done","log_level":1,"data":{"session":"4.1.27","worker-address":"127.0.0.1:35264","worker-platform":"linux","worker-tags":""}}
Apr 01 18:49:27 nexus concourse[29079]: {"timestamp":"1459525767.033954620","source":"tsa","message":"tsa.connection.forward-worker.heartbeat.start","log_level":1,"data":{"session":"4.1.28","worker-address":"127.0.0.1:35264","worker-platform":"linux","worker-tags":""}}
Apr 01 18:49:27 nexus concourse[29079]: {"timestamp":"1459525767.036112070","source":"tsa","message":"tsa.connection.forward-worker.heartbeat.reached-worker","log_level":0,"data":{"session":"4.1.28","took":"1.861335ms"}}
Apr 01 18:49:27 nexus concourse[29079]: {"timestamp":"1459525767.054392815","source":"tsa","message":"tsa.connection.forward-worker.heartbeat.done","log_level":1,"data":{"session":"4.1.28","worker-address":"127.0.0.1:35264","worker-platform":"linux","worker-tags":""}}
Apr 01 18:49:31 nexus concourse[29079]: {"timestamp":"1459525771.633763075","source":"garden-linux","message":"garden-linux.garden-server.run.exited","log_level":1,"data":{"handle":"ga6ab3n6gua","id":"1","session":"11.26","status":1}}

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 21 (4 by maintainers)

Commits related to this issue

Most upvoted comments

There were an issue with dns server in container. On host machine nslookup works, but in container it errors with “nslookup: can’t resolve ‘google.com’” I’ve started worker with -- -dnsServer 192.168.1.35, where 192.168.1.35 is host with bind dns server.

+4
ernado on Apr 1, 2016

I had to add --garden-dns-server 8.8.8.8 to get everything working.

+3
ergo on Aug 3, 2016

Finally! In case anyone is having trouble configuring UFW: The solution to the problem is to set UFW’s forward policy to “ACCEPT” (as described here: https://docs.docker.com/engine/installation/linux/ubuntulinux/#/enable-ufw-forwarding);

Cheers Korbi

+3
doejon on Jul 21, 2016

Get hints from this comments: https://github.com/concourse/concourse/issues/347#issuecomment-237354725

I restart iptables service.

service iptables stop
service iptables start

The problem is fixed.

If you try everything above and didn’t get the problem fixed, try this or reset iptables to default.

+1
ozbillwang on Jul 30, 2017

This issue is not likely to get resolved as it really depends on how each system is set up. I’ve opened an issue on Guardian for one fixable aspect, but I’m not sure what we can do about the rest (UFW/etc.). Seems like this issue has served its purpose as a Google reference at least, but it’s turning into a bit of a dumping ground, so I’m gonna close it.

For people who find this issue in the future, please open a fresh issue detailing your system setup (including any firewalls, IaaS setups like AWS VPC, and DNS config) and we’ll look into why it didn’t work out of the box, and update our docs if it’s a recurring theme.

+1
vito on Aug 8, 2016

Try passing --garden-dns-server 8.8.8.8 instead of -- -dnsServer 8.8.8.8.

+1
vito on Jul 15, 2016
Read more comments on GitHub
← concourse: Old workers stay in `stalled` state indefinitely after restart
concourse: Performance regression with 'overlay' driver and privileged containers →