the-littlest-jupyterhub: Installing https support via Let's Encrypt appears broken (instructions problematic)

Ok, so to me the error is “clear” :

From: https://certbot.eff.org/docs/using.html#webroot

The webroot plugin works by creating a temporary file for each of your requested domains in ${webroot-path}/.well-known/acme-challenge. Then the Let’s Encrypt validation server makes HTTP requests to validate that the DNS for each requested domain resolves to the server running certbot. An example request made to your web server would look like:

66.133.109.36 - - [05/Jan/2016:20:11:24 -0500] "GET /.well-known/acme-challenge/HGr8U1IeTW4kY_Z6UIyaakzOkyQgPr_7ArlLgtZE8SX HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Note that to use the webroot plugin, your server must be configured to serve files from hidden directories. If /.well-known is treated specially by your webserver configuration, you might need to modify the configuration to ensure that files inside /.well-known/acme-challenge are served by the webserver.

And tljh doesn’t allow to reach these files, thus, visibly, the challenge fails.

Thinking of it, I hadn’t set up the DNS redirection properly: I had set up a permanent web forwarding, not an A DNS (for foo.bar to ip) and CNAME DNS records (for www.foo.bar to foo.bar) Explanations here, and setup instructions if you’re on Gandi: https://docs.gandi.net/en/domain_names/common_operations/link_domain_to_website.html Other good Explanations : https://support.dnsimple.com/articles/a-record/

Now if I had done this properly from the start, it may have worked with the tljh’s default letsencrypt; When I find time, I’ll test 😉 (as I guess it works much better for the certificate renewal).

EDIT: This was indeed the problem, see my next post

Meanwhile, I finally got cerbot to work ( https://certbot.eff.org/lets-encrypt/ubuntubionic-other ) after quite a bit of trial-error, so I’m going to post what I’d been happy bumping on myself. However, it’s just what I did on my server, there may be shorter and simpler, but to be sure that would require a bit of testing that I don’t have time to do.

I was in root, all this will need extra sudo’s otherwise.

First, I undid all I had set up during my previous trials to setup https (we never know):

tljh-config unset https.enabled tljh-config unset OR remove-item AnyOtherStuffTested tljh-config reload

Then I tried the standalone certbot: sudo certbot certonly --standalone --preferred-challenges http -d foo.bar -d www.foo.bar But I had this error: Problem binding to port 80: Could not bind to IPv4 or IPv6.

So I started with: ufw allow 80 But it didn’t work yet

Actually, the reason why it wasn’t working is that tljh still had it’s frontend running on my address.

So to see if I could stop it, I tried (note: my jupyter instances/servers where all already shutdown, no idea if it’s important): systemctl | grep running systemctl stop jupyterhub.service Still not enough; by running: ss -tlnp | grep -E ":(80|443)" I saw that I still had traefik squatting the ports; so: systemctl stop traefik.service

And yay! Finally sudo certbot certonly --standalone --preferred-challenges http -d foo.bar -d www.foo.bar worked 😃

So

systemctl start traefik.service
systemctl start jupyterhub.service

I finally could load my key and certificat following the instructions in the second part of the tutorial: http://tljh.jupyter.org/en/latest/howto/admin/https.html

Now the problem I guess, is that for certificat renewal, I’ll have to shut down the server again; so I’ll definitely try the proper way anew later.

error msg="Unable to obtain ACME certificate for domains "a_domain.com" : unable to generate a certificate for the domains [a_domain.com]: acme: Error -> One or more domains had a problem. [a_domain.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://a_domain.com/.well-known/acme-challenge/ra01JKbw3Wv194BDVhjSeK_nkbFA-UVYqnhv08LUoM [2606:4700:3037::681b:a340]

Port 80 must be open for HTTP traffic over IPv4. I had mine restricted to IPv6 (by mistake) and allowing IPv4 traffic on 80 resolved it.

I have literally done the install dozens of times and it never worked. Which instructions did you follow?

Sorry for my late answer, I’m quite busy at the moment; Here is precisely all I did, from a clean Ubuntu server 18.04 install:

If your user hasn’t the sudo rights:

su
usermod -a -G sudo yourusername
exit

From now on, everything is run from the normal user “yourusername”:

sudo apt-get update
sudo apt-get upgrade  ## Enter on all dialogs if there are some
sudo dpkg-reconfigure locales ## to have locals set up properly and stop having LC errors; I chose EN-US utf8
sudo apt-get install linux-headers-generic ethtool libc-dev linux-libc-dev python3-dev
sudo reboot

Now all is ready, we can do:

sudo ls ## just to have the sudo password entered
curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py | sudo -E python3 - --admin myfirstadminuser ## that's precisely the command of the install instructions in the manual: http://tljh.jupyter.org/en/latest/install/custom-server.html

Then, get things going; I don’t know if it’s all needed:

export PATH=/opt/tljh/user/bin:${PATH}
nano ~/.bashrc && source ~/.bashrc  ## Added the export path from above; source: http://tljh.jupyter.org/en/latest/howto/env/user-environment.html
sudo env PATH=${PATH} conda update -n base conda ## do not forget the "env"; it's actually missing from the tutorial page above, I'll think about editing it.

At last, the normal SSL procedure from this page: http://tljh.jupyter.org/en/latest/howto/admin/https.html

sudo tljh-config set https.enabled true
sudo tljh-config set https.letsencrypt.email email@example.com ## more precisely, my email is hosted on mydomain.me, but I don't think it's important
sudo tljh-config add-item https.letsencrypt.domains mydomain.me
sudo tljh-config add-item https.letsencrypt.domains www.mydomain.me
sudo tljh-config show

When all is good: sudo tljh-config reload proxy

Now if you configured the DNS records properly (see my previous long post), all should go fine, and going to “mydomain.me” should bring you directly on the login secured with https 😉

Good luck testing 😉

Note that i already had a working https setup on the same domain using the universal letsencrypt procedure {my long post above) but I then wiped everything at started with a new ubuntu install, so it should not affect anything. Second, all this was part of quite a bit of trial and error, so you’re welcome to suggest improvements!

(By the way, it seems that the only reliable way of installing extra python modules is to use the command sudo -E pip intall module in the jupyter notebook terminal online! ‒and doing a sudo -E pip install --upgrade pip before‒. I didn’t manage any install of working modules any other way ‒for example through ssh‒. When I have time I’ll dig this, as it’s another issue. Linked help page, that details the steps: http://tljh.jupyter.org/en/latest/howto/env/user-environment.html)

Ok, so still fulfilling my noob role in this story, I ended up totally messing up my install. So I restarted from zero, and this time tested the proper tljh way of setting up a certificate. And guess what, it worked! So the issue was me not setting up the DNS records properly, confirmed.

By the way, having a look at sudo systemctl status traefik.service can help identify things a bit, if there is some network problem (I found it useful).