rancher: Agent says: ERROR: http://10.1.3.4:8080/v1 is not accessible (but should say: Authentication failure)

I solved it 😃

I didn’t have to disable Access Control, my problem was with the certificate and NGinx 😒

The problem was with the chain of validation of the certificate. Since I was also configuring a Docker registry server, in their tutorial they explained that it’s important to bundle both certificates in case you face any problem.

It would be nice that this should be added to the documentation of rancher (SSL installation)

The solution:

• Concatenate the primary certificate and intermediate certificate:

You need to concatenate the primary certificate file (your_domain_name.crt) and the intermediate certificate file (IntermediateCA.crt) into a single pem or crt file by running the following command:

cat your_domain_name.crt IntermediateCA.crt >> bundle.crt

and use the bundle.crt in your nginx configuration:

...
server {
    listen 443 ssl;
    server_name my_server;
    ssl_certificate /etc/nginx/ssl/bundle.crt;
    ssl_certificate_key /etc/nginx/ssl/my_priv.key;

    location / {
...

Also (another clue to catch the problem) if you use curl, it should work properly without the -k parameter, and the validation of the certificates should return that everything is ok.

Have you tried to reset some elements of docker and rancher (on the host) in order to re-try adding it?

The elements I’m talking about are:

sudo rm -rf /var/lib/rancher/state; sudo docker rm -fv rancher-agent; sudo docker rm -fv rancher-agent-state

Edit: I just ran into it again, and after a lot of debugging it turned out (to my shame) to be the problem of the container not having DNS configured properly (I’m pretty sure this was not the reason when I initially reported this). However, I fixed this by setting DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4" in /etc/default/docker.

@munjalpatel , does this work for you too?

I never understood why this is necessary for docker, especially since things worked fine before (same docker service, different container).

@feliksik Let me explain how auth works. I’m not sure if I totally follow your sequence of events so if I just explain the details maybe that will help. In the README.md of Rancher we say to register a host with the following command

docker run --rm -it --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent http://MANAGEMENT_IP:8080

If you click the “Add Host” button in the UI we give you a command like (notice the long random strings at the end)

sudo docker run --rm -it --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent http://localhost:8080/v1/scripts/7AE22E31405688E533F4:1425056400000:MOGpr8ZsoQXpRstuwDfPOC5cI

Admittedly this is confusing and I think we may have to remove the instructions from the README.md and tell you just just follow the instructions on the UI.

The random string at the end is the registration token. If you use a URL that does not have the registration token then you can only register with a server that does not have auth on. If you have auth on you must use a URL with the registration token.

When an agent registers with or without the token, an account and corresponding access/secret key is created for the agent. This means that every host in rancher has a unique set of credentials assigned to it. The agent saves these credentials in the agent environment variables. The difference between auth on and off is that if auth is off any agent can create credentials, if auth is on you must have the registration token to create an account.

Now the different scenarios of turning auth on or off:

If you have already registered a host with auth on or off and then you switch auth to the opposite, the agents should still work. This is because they created a credential on registration and they continue to use that credential.

If you wipe out the server and bring up a new server with auth off, the existing agents that are still trying to communicate with that IP will reregister themselves with the new server and create a new credential.

If you wipe out the server and bring up a new server with auth on, the existing agents will get stuck because their credential is bad and they don’t have a registration token to register to the new server.

I hope that helps a bit.