html-proofer: SSL link check fails, while curl succeeds

I’m getting the following error when running html-proofer in a Travis job:

- /home/travis/out/gnuarmeclipse/gnuarmeclipse.github.io/qemu/build-procedure/index.html
  *  External link https://developer.apple.com/xcode/downloads/ failed: 302 SSL connect error
- /home/travis/out/gnuarmeclipse/gnuarmeclipse.github.io/windows-build-tools/build-procedure/index.html
  *  External link http://developer.apple.com/xcode/downloads/ failed: 302 SSL connect error
htmlproofer 3.4.0 | Error:  HTML-Proofer found 2 failures!

However, in exactly the same environment, a curl to that address was ok:

$  curl -L --url http://developer.apple.com/xcode/downloads/ --verbose
* Hostname was NOT found in DNS cache
*   Trying 17.146.1.14...
* Connected to developer.apple.com (17.146.1.14) port 80 (#0)
> GET /xcode/downloads/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: developer.apple.com
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 04 Feb 2017 14:43:25 GMT
< Content-Type: text/html
< Content-Length: 179
< Connection: keep-alive
< Location: https://developer.apple.com/xcode/downloads/
* Server Shield is not blacklisted
...
< 
* Ignoring the response-body
* Connection #1 to host developer.apple.com left intact
* Issue another request to this URL: 'https://idmsa.apple.com/IDMSWebAuth/login?appIdKey=891bd3417a7776362562d2197f89480a8547b108fd934911bcbea0110d07f757&path=%2Fdownload%2F&rv=1'
* Hostname was NOT found in DNS cache
*   Trying 17.171.11.86...
* Connected to idmsa.apple.com (17.171.11.86) port 443 (#2)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES128-GCM-SHA256
* Server certificate:
* 	 subject: 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=California; businessCategory=Private Organization; serialNumber=C0806592; C=US; postalCode=95014; ST=California; L=Cupertino; street=1 Infinite Loop; O=Apple Inc.; OU=GNCS Traffic Management; CN=idmsa.apple.com
* 	 start date: 2017-01-20 00:00:00 GMT
* 	 expire date: 2019-01-20 23:59:59 GMT
* 	 subjectAltName: idmsa.apple.com matched
* 	 issuer: C=US; O=Symantec Corporation; OU=Symantec Trust Network; CN=Symantec Class 3 EV SSL CA - G3
* 	 SSL certificate verify ok.
> GET /IDMSWebAuth/login?appIdKey=891bd3417a7776362562d2197f89480a8547b108fd934911bcbea0110d07f757&path=%2Fdownload%2F&rv=1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: idmsa.apple.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Set-Cookie: JSESSIONID=E374FE13E2A29D92D8B47263B64ACFD7; Path=/; Secure; HttpOnly
< Set-Cookie: dslang=US-EN; Domain=.apple.com; Expires=Thu, 03-Aug-2017 14:43:26 GMT; Path=/; Secure; HttpOnly
< Set-Cookie: dslang=US-EN; Domain=.apple.com; Expires=Thu, 03-Aug-2017 14:43:26 GMT; Path=/; Secure; HttpOnly
< TOTAL_TIME: 105
< DS_TIME: default
< X-FRAME-OPTIONS: DENY
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Cache-Control: private, no-cache, no-store, must-revalidate, max-age=0, post-check=0, pre-check=0
< Pragma: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: text/html;charset=UTF-8
< Content-Language: en-US
< Transfer-Encoding: chunked
< Vary: Accept-Encoding
< Date: Sat, 04 Feb 2017 14:43:26 GMT
* Server APPSRV is not blacklisted
< Server: APPSRV
< Set-Cookie: X-SESS=28d4a3da9fb70633345a49ca2d42db4197df95593ba62c62d7c0d5cc6198624c97df0a3f;Version=1;Max-Age=1800;path=/;secure;httponly
< 
<!DOCTYPE html>
...

The entire log is available at: https://travis-ci.org/gnuarmeclipse/gnuarmeclipse.github.io-source/builds/198333357

For the moment I added the offending URLs to the --url-ignore list, but perhaps there is something you can do to accept these URLs too.

Any thoughts?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 44 (21 by maintainers)

Commits related to this issue

Most upvoted comments

YES. Adding this to my .travis.yml fixed the SSL problems:

addons:
  apt:
    packages:
    - libcurl4-openssl-dev

I’ll add this to the jekyll-test gem, and we should document it on the wiki somewhere too.

+12
Floppy on Sep 28, 2017

Try adding to your travis.yml;

sudo: required
before_install:
  - sudo update-ca-certificates

The default CAs shipped with Ubuntu 12.04 are somewhat out of date now. Running update-ca-certificates will get the latest certs (but you need sudo to run it).

+5
mattclegg on Mar 6, 2017

AHA! Got it!

Changing the libcurl to use openssl instead of gnutls fixes it. On the travis environment, if I do sudo apt-get install libcurl4-openssl-dev, it removes the gnutls version, then all the links work with 200 responses.

I’ve reported this upstream to typhoeus in the linked issue, but we can resolve it here with some instructions and suggested travis config. I’ll work out how to get travis to do that itself and report back.

+4
Floppy on Sep 28, 2017

@Floppy did an amazing job documenting the fix in the wiki, so I just provided links in the README to that document.

+3
gjtorikian on Oct 7, 2017

Thanks, yes, it is a parsing problem of the travis file, which is why travis does not even start (so there is no log). I found out that removing the space after the “:” works:

script: bundle exec jekyll build && bundle exec htmlproofer ./_site --check_html --typhoeus_config '{ "timeout":30 }'

Then travis runs and calls the htmlproofer. I still get several:

External link <!deleted!> failed: response code 0 means something’s wrong. It’s possible libcurl couldn’t connect to the server or perhaps the request timed out. Sometimes, making too many requests at once also breaks things. Either way, the return message (if any) from the server is: Server returned nothing (no headers, no data)

And it definitely does not wait 30 sec (command finishes in 5 sec). So either it does not take the timeout , or does not wait for some other reason. Passing additionally the verbose: true did have an effect.

+1
m-reuter on Apr 22, 2020

Added to .travis.yml samples in https://github.com/gjtorikian/html-proofer/wiki/Using-HTMLProofer-From-Ruby-and-Travis.

+1
Floppy on Sep 28, 2017

I wonder if anyone would try Travis’ instructions for Troubleshooting Locally in a Docker Image?

+1
gjtorikian on Sep 27, 2017

That’s a really great idea @Floppy. Wonder if I can convince @parkr to pull it in as an optional gem once it’s completed. 🙂

+1
gjtorikian on Aug 29, 2017

On the issue of getting SSL failures on travis specifically, I’m working on https://github.com/Floppy/jekyll-test which plans to automate all the required setup to make it work reliably, once I’ve found out how 😃

+1
Floppy on Aug 28, 2017

Here’s the error I get:

  *  External link https://talk.jekyllrb.com failed: response code 0 means something's wrong.
             It's possible libcurl couldn't connect to the server or perhaps the request timed out.
             Sometimes, making too many requests at once also breaks things.
             Either way, the return message (if any) from the server is: SSL connect error

I’ve tried sudo update-ca-certificates, but it didn’t work: https://travis-ci.org/cloud-tv/cloud-tv.github.io/builds/268911228

There is no error on CircleCI and on my local machine. Maybe the solution is just to abandon Travis in favour of CircleCI.

+1
mvasin on Aug 27, 2017
Read more comments on GitHub
← commonmarker: Problem with CommonMarker on an Azure VM
GCC-CL: script hook V error when generating people →