cargo: Cargo can't download crates: `SEC_E_ILLEGAL_MESSAGE`

Hi folks,

Sorry for the long wait. I’ve been working with our contacts at Fastly and they’ve enabled a more permissive TLS configuration with more cipher suites for us. I’ve redeployed our services and the new suites are now live.

Let me know if this fixes the issue with cargo and crates.io for you.

Let me know if this fixes the issue with cargo and crates.io for you.

It’s really all right now:

>set CURL_SSL_BACKEND=schannel
>curl -o argminmax.crate https://fastly-static.crates.io/crates/argminmax/argminmax-0.6.1.crate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 55395  100 55395    0     0   8589      0  0:00:06  0:00:06 --:--:-- 20343

Perhaps the root cause of the error is the incorrect configuration of the crates.io servers, which is why some versions of Windows schannel cannot work with them.

In the first post, I showed that Curl (the official distribution for Windows) normally downloads the required file. But it was with the default backend (openssl). Now I tested the curl with a non-standard backend (schannel) and got the same error:

>set CURL_SSL_BACKEND=schannel
>curl -L -o argminmax.crate https://crates.io/api/v1/crates/argminmax/0.6.1/download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.

So, if you still try to solve the problem from the side of Cargo, then two questions arise:

• Why does Cargo for Windows use Curl with a non-default ssl backend?
• Is it possible to make a configurable ssl backend in Cargo (as in Curl) or even just call an external Curl?

Here is an easier TLS mitm proxy in one line and does not need installation or anything else: download proxify_0.0.9_windows_386.zip then run:

proxify.exe -http-addr 127.0.0.1:18080

then add the proxify cert to windows cert manager:

certutil -addstore -f -user "Root" "%USERPROFILE%\.config\proxify\cacert.pem"

now before using Cargo do:

set "CARGO_HTTP_CHECK_REVOKE=false"
set HTTPS_PROXY=127.0.0.1:18080
set HTTP_PROXY=127.0.0.1:18080

that’s all

proxify is made in golang so it is faster than the python solutions, and have some very useful and interesting options… as @nokotto said: it is predictable that what is happening now will happen again with the time

let’s summarize

Previously the crates downloads were done by the cloudfront servers, they accept the TSL 1.1,1.2,1.3 protocols, and a number of certain ciphers under each of them. Recently, fastly servers have been included in the load distribution, which support the TSL 1.2, 1.3 protocols, and only six specific ciphers.

When cargo is launched, firstly does a query to crates.io under cloudfront servers, gets crates data, and uses static.crates.io for to download. The IP resolution of the load balancer under static.crates.io can take you to either a cloudfront or a fastly server.

Cargo uses libcurl for the downloads. The curl library has two options available to handle the encryptions of the queries, on the one hand it can use the windows encryption libraries, called SCHANNEL, and on the other hand it can use the OpenSSL library.

Currently cargo only uses schannel, which makes it dependent on the encryption libraries available in windows (TSL protocols + ciphers ).

Windows 7 SP1 users

When cargo is addressed to cloudfront servers , the download occurs normally, without issues, since two conditions are met:

• If win7 does not have TSL 1.2 protocol enabled, the query is performed by TSL 1.1, using their respective ciphers.

• If win7 has TSL 1.2 enabled [1], the query is performed by this protocol, using their respective ciphers.

This happens in both cases because in addition to the TSL protocols, the encryption algorithms (Cipher Suits) available in windows 7 are also available (enabled) on the servers.

[1] Update to enable TLS 1.1 and TLS 1.2 + Regedit edit, https://github.com/rust-lang/cargo/issues/12296#issuecomment-1632210190

When cargo is addressed to fastly servers, the download fails:

• If win7 does not have TSL 1.2 protocol usage enabled, the operation terminates there.

• If win7 has TSL 1.2 enabled (by the method discussed above), the ciphers available in the windows library are not available on the servers.

The accepted ciphers in fastly servers at this moment are

TLS 1.3 (suites in server-preferred order)

 TLS_AES_256_GCM_SHA384                      (0x1302)  ECDH x25519 (eq. 3072 bits RSA)   FS 256
 TLS_CHACHA20_POLY1305_SHA256                (0x1303)  ECDH x25519 (eq. 3072 bits RSA)   FS 256
 TLS_AES_128_GCM_SHA256                      (0x1301)  ECDH x25519 (eq. 3072 bits RSA)   FS 128
		
TLS 1.2 (suites in server-preferred order)
 
 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256       (0xc02f)  ECDH x25519 (eq. 3072 bits RSA)   FS 128
 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384       (0xc030)  ECDH x25519 (eq. 3072 bits RSA)   FS 256
 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)  ECDH x25519 (eq. 3072 bits RSA)   FS 256

Mean while, workarounds

Someones temporally mounted local proxies, https://github.com/rust-lang/cargo/issues/12296#issuecomment-1631520514 , https://github.com/rust-lang/cargo/issues/12296#issuecomment-1632968251 or manually downloaded https://github.com/rust-lang/cargo/issues/12296#issuecomment-1631094816

In my case, I did a ping to cloudfront-static.crates.io , copied the IP, edited Windows\System32\drivers\etc\host as Admin, and added a line with such IP as resolver for static.crates.io ( so IP_NUMBER static.crates.io).

I have to warn my workaround have a problem, when the host behind such IP changes, it stop working. And also have the danger of such IP being assigned to other host-client. I should create a script for checking and updating such host, or to install a DNS server local for making such IP overriding and check.

Anyway, it is clear all this workarounds are a pain.

How would it be solved ?

If Cargo had the option for to enable OpenSSL within libcurl, it would make the encryption system independent. What is more, almost all -if not all- the servers use such OpenSSL libraries, that sounds even ironic. Anyway, the important thing is this would include the currently needed ciphers, and even allow the use of the TSL 1.3 protocol, and its used ciphers, in all the windows versions (win7 and win10 specifically), because is predictable that what is happening now will happen again with the time.

Meanwhile, if fastly-static.crates.io servers could enable one of the ciphers that cloudfront-static also has enabled under the protocol TSL1.2, life would be easier for many users.

Copied from this https://github.com/rust-lang/cargo/issues/11344#issuecomment-1345637248:

…this is a duplicate of https://github.com/rust-lang/cargo/issues/8688 and https://github.com/rust-lang/cargo/issues/7104. Windows, unlike all other platforms, defaults to checking for TLS revocation certificates. We have considered disabling that on Windows, but have not followed through. This might also be resolved if crates.io leaves Heroku and switches to a service that supports OCSP stapling.

Also probably a duplicate of #12273, so I am going to close this. If you find anything interesting or different, please let us know. Thanks for the report anyway!

Hi folks,

The new configuration has been live for the past 10 days and so far we’ve seen no issues. I’m going to close the GitHub issue, but feel free to report back if you get the above error on a patched Windows version again.

Thanks 👋

Let me know if this fixes the issue with cargo and crates.io for you.

I also confirm it fixed it , thanks very much! ( I checked the Cargo’s downloads went through Fastly )

The Proxify solution works for me. Thanks, @badrelmers!

I have implemented it on AppVeyor CI. Below are the changes to my appveyor.yml. I had to replace the certutil command with Import-Certificate in PowerShell, because the certutil command would hang for some reason.

index 7e630d3..7c25286 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -8,9 +8,16 @@ cache:
   - '$(CARGO_PATH) -> appveyor.yml, Cargo.lock, Cargo.toml'
 
 install:
+  - cmd: appveyor-retry appveyor DownloadFile https://github.com/projectdiscovery/proxify/releases/download/v0.0.9/proxify_0.0.9_windows_386.zip -FileName proxify.zip
+  - cmd: unzip -n proxify.zip
+  - ps: Start-Process -FilePath proxify.exe -ArgumentList "-config","proxify-config","-http-addr","127.0.0.1:18080"
+  - cmd: timeout /t 5
+  - ps: Import-Certificate -FilePath "proxify-config\cacert.pem" -CertStoreLocation Cert:\LocalMachine\Root
   - cmd: appveyor-retry appveyor DownloadFile https://win.rustup.rs -FileName rustup-init.exe
-  - cmd: rustup-init.exe --default-toolchain %RUST_VERSION% -y
+  - cmd: set HTTP_PROXY=127.0.0.1:18080
+  - cmd: set HTTPS_PROXY=127.0.0.1:18080
   - cmd: set PATH=%CARGO_PATH%\bin;%PATH%
+  - cmd: rustup-init.exe --default-toolchain %RUST_VERSION% -y
 
 build_script:
   - cmd: cargo build

Sorry about the mention noise above. I didn’t remove the link to the issue from the commit message when debugging CI.

I’m almost certain that the SEC_E_ILLEGAL_MESSAGE error is caused by missing support for TLS 1.2 in the Secure Channel component in Windows 7. Sadly it only supports TLS 1.1 out-of-the-box, which was deprecated in June 2018.

The reason why the issues are appearing now is that we’ve started to serve traffic for crates (static.crates.io) and releases (static.rust-lang.org) through both CloudFront and Fastly, the latter of which only supports TLS 1.2 and higher. This is why it works sometimes (when traffic goes through CloudFront) and sometimes it doesn’t (when traffic goes through Fastly).

As far as I can tell, this needs to be fixed on the user’s machine. Microsoft provides a support article on how to Update to enable TLS 1.1 and TLS 1.2, which involves installing SP1 and adding a registry key. The article mentions an “easy fix” for the second step so that users don’t need to edit the registry manually. A restart might be required.

My bad! It was too careless of me. Repoened.

This kind of bug happened a couple years ago (https://github.com/rust-lang/cargo/issues/9788). Was fixed with a libcurl version bump. I will try to look into its changelog a bit to see if we can find something suspicious.

@jdno I already have this update installed Secure Channel update, it does not solve this problem

A discussion in the Factorio forum links to the same knowledge base article as above about enabling TLS 1.1 and 1.2, but also includes a link to a Secure Channel update that apparently adds more cipher suites.

I can’t confirm if this works, but if I have some time later I might try spinning up an old Windows 7 version to reproduce the issue locally.

@weihanglo I think that would only affect WIndows 11. AFAIK it is not possible to turn it on for earlier versions.

note that windows 7 support is intended to be removed soon: https://github.com/rust-lang/compiler-team/issues/651

Perhaps the root cause of the error is the incorrect configuration of the crates.io servers, which is why some versions of Windows schannel cannot work with them.

Do you know when this issue started to happen? That is, did downloading crates with Cargo on windows work at some point and then stop working? That could help us to narrow down configuration changes made to crates.io that could have caused this.

Do you have any thoughts on the kind of incorrect configuration of the servers that could cause this?