vespa: Quick-start tutorial runs with internal timeouts

Gist with log files, ps output, netstat output, Kubernetes manifest.

I don’t know if this is an actual error or not, but I get lots of log entries about not being able to connect to the config server at port 19070, lines like this:

1507078186.060	vespa-0.vespa.default.svc.cluster.local	544/1	configproxy	configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient	info	Could not connect to config source at tcp/localhost:19070
1507078186.061	vespa-0.vespa.default.svc.cluster.local	544/1	configproxy	configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient	info	Could not connect to any config source in set [tcp/localhost:19070], please make sure config server(s) are running.

I’m starting Vespa under Kubernetes, using the exact commands here. One difference is that Kubernetes is setting the host name, but I don’t see why that matters. Using curl, I can confirm that connections to port 19070 hang and eventually time out.

From what I can tell, the config server has been started. The config sentinel (PID 833 in the ps output), however, keeps restarting.

Using netstat, I see that the config server process is indeed listening on port 19070.

Port 19070 is operational.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 1
  • Comments: 19 (2 by maintainers)

Most upvoted comments

D’oh. I just assumed that this would be the data folder and that Vespa would behave correctly if it was initially empty. I’ve never actually encountered an application before that puts non-volatile data under a folder named /var/db (var stands for variable, after all).

Does this mean that data in /opt/vespa/var/db (except for /opt/vespa/var/db/vespa/search) is transient and expendable? The reason I mount a volume is to get persistence with Kubernetes. But if Vespa writes important state outside of /opt/vespa/var/db/vespa/search, they’ll be lost if the pod is stopped/rescheduled.

Started up again, looks much better now.

Sorry, I think I got confused by the many different ports mentioned. While the errors about 19070 happen, the ones that keep coming over and over are about port 19090:

1507130648.340445	vespa-0.vespa.default.svc.cluster.local	2214/24672	config-sentinel	config-sentinel.config.frt.frtconnection	warning	Connection to tcp/localhost:19090 failed or timed out
1507130648.340520	vespa-0.vespa.default.svc.cluster.local	2214/24672	config-sentinel	config-sentinel.config.frt.frtconnection	warning	FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:24:58 GMT
1507130648.340540	vespa-0.vespa.default.svc.cluster.local	2214/24672	config-sentinel	config-sentinel.config.frt.frtconfigagent	info	Error response or no response from config server (key: name=sentinel,namespace=cloud.config,configId=hosts/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 10000 milliseconds
1507130649.790	vespa-0.vespa.default.svc.cluster.local	543/12	configproxy	configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient	info	Subscribe for 'name=sentinel,namespace=cloud.config,configId=hosts/vespa-0.vespa.default.svc.cluster.local,0944a8c189a502c0e2fe1930114897b7' failed, closing subscriber
1507130658.244562	vespa-0.vespa.default.svc.cluster.local	2214/54059	config-sentinel	config-sentinel	warning	Timout getting config, please check your setup. Will exit and restart: Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/vespa-0.vespa.default.svc.cluster.local'

They also keep coming after preparing a sample application with /opt/vespa/bin/vespa-deploy prepare /vespa-sample-apps/basic-search/src/main/application/ && /opt/vespa/bin/vespa-deploy activate, after which a lot of apparent failures happen. I can’t really make heads or tails out of the log since I don’t know what’s failing, but there are warnings about RPC errors and timeouts while activating the application. Here is a log from the moment I do the deploy.

Re Kubernetes: I’m running this in a single container locally in Minikube. I’m not yet trying out a multi-node setup.

(Protip: I recommend using triple backtics when pasting chunks of raw text, otherwise your lines will be wrapped and become unreadable.)