vespa: Quick-start tutorial runs with internal timeouts
Gist with log files, ps output, netstat output, Kubernetes manifest.
I don’t know if this is an actual error or not, but I get lots of log entries about not being able to connect to the config server at port 19070, lines like this:
1507078186.060 vespa-0.vespa.default.svc.cluster.local 544/1 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Could not connect to config source at tcp/localhost:19070
1507078186.061 vespa-0.vespa.default.svc.cluster.local 544/1 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Could not connect to any config source in set [tcp/localhost:19070], please make sure config server(s) are running.
I’m starting Vespa under Kubernetes, using the exact commands here. One difference is that Kubernetes is setting the host name, but I don’t see why that matters. Using curl, I can confirm that connections to port 19070 hang and eventually time out.
From what I can tell, the config server has been started. The config sentinel (PID 833 in the ps output), however, keeps restarting.
Using netstat, I see that the config server process is indeed listening on port 19070.
Port 19070 is operational.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 1
- Comments: 19 (2 by maintainers)
D’oh. I just assumed that this would be the data folder and that Vespa would behave correctly if it was initially empty. I’ve never actually encountered an application before that puts non-volatile data under a folder named
/var/db(varstands forvariable, after all).Does this mean that data in
/opt/vespa/var/db(except for/opt/vespa/var/db/vespa/search) is transient and expendable? The reason I mount a volume is to get persistence with Kubernetes. But if Vespa writes important state outside of/opt/vespa/var/db/vespa/search, they’ll be lost if the pod is stopped/rescheduled.Started up again, looks much better now.
Sorry, I think I got confused by the many different ports mentioned. While the errors about 19070 happen, the ones that keep coming over and over are about port 19090:
They also keep coming after preparing a sample application with
/opt/vespa/bin/vespa-deploy prepare /vespa-sample-apps/basic-search/src/main/application/ && /opt/vespa/bin/vespa-deploy activate, after which a lot of apparent failures happen. I can’t really make heads or tails out of the log since I don’t know what’s failing, but there are warnings about RPC errors and timeouts while activating the application. Here is a log from the moment I do the deploy.Re Kubernetes: I’m running this in a single container locally in Minikube. I’m not yet trying out a multi-node setup.
(Protip: I recommend using triple backtics when pasting chunks of raw text, otherwise your lines will be wrapped and become unreadable.)