runtime: System.Net.Http.HttpRequestException: The server returned an invalid or unrecognized response

Similar to dotnet/runtime#26774 and potentially others, we’re getting System.Net.Http.HttpRequestException: The server returned an invalid or unrecognized response when making a HTTPS certificate authenticated request to Bloomberg’s Web API. We want to use dotnetcore but this is blocking us.

• Windows 10 Enterprise 2016 LTSB 1607 Build 14393.2368 64bit
• .NET Core 2.1
• VS 15.7.5 Microsoft WCF Web Service Provider generated proxy of https://service.bloomberg.com/assets/dl/dlws.wsdl
• Dependencies:

<ItemGroup>
    <PackageReference Include="log4net" Version="2.0.8" />
    <PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
    <PackageReference Include="System.ServiceModel.Duplex" Version="4.5.3" />
    <PackageReference Include="System.ServiceModel.Http" Version="4.5.3" />
    <PackageReference Include="System.ServiceModel.NetTcp" Version="4.5.3" />
    <PackageReference Include="System.ServiceModel.Security" Version="4.5.3" />
</ItemGroup>

Please let me know if I can provide any more detail since this will be hard for anyone to reproduce unless they mock a server to return the exact response Bloomberg’s servers do.

System.ServiceModel.CommunicationException: The server returned an invalid or unrecognized response. ---> System.Net.Http.HttpRequestException: The server returned an invalid or unrecognized response.
   at System.Net.Http.HttpConnection.ThrowInvalidHttpResponse()
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.SendRequestAsync(Message message, TimeoutHelper timeoutHelper)
   --- End of inner exception stack trace ---
   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult)
--- End of stack trace from previous location where exception was thrown ---

Below is the client setup code using NETCOREAPP;NETCOREAPP2_1. This was migrated from .NET 4.5 where the only changes are the 5 lines commented out as those properties are not supported in core 2.1.

public PerSecurityWSClient GetPerSecurityWsClient() {
	var binding = new BasicHttpBinding();
	binding.Name = "PerSecurityWSBinding";
	binding.CloseTimeout = TimeSpan.FromMinutes(1);
	binding.OpenTimeout = TimeSpan.FromMinutes(1);
	binding.ReceiveTimeout = TimeSpan.FromMinutes(10);
	binding.SendTimeout = TimeSpan.FromMinutes(1);
	binding.BypassProxyOnLocal = false;
	//binding.HostNameComparisonMode = HostNameComparisonMode.StrongWildcard;
	binding.MaxBufferPoolSize = 524288;
	binding.MaxBufferSize = 65536000;
	binding.MaxReceivedMessageSize = 65536000;
	//binding.MessageEncoding = WSMessageEncoding.Text;
	binding.TextEncoding = Encoding.UTF8;
	binding.UseDefaultWebProxy = true;
	binding.AllowCookies = false;
	binding.TransferMode = TransferMode.Buffered;
	binding.ReaderQuotas.MaxDepth = 32;
	binding.ReaderQuotas.MaxArrayLength = 16384;
	binding.ReaderQuotas.MaxStringContentLength = 8192;
	binding.ReaderQuotas.MaxBytesPerRead = 4096;
	binding.ReaderQuotas.MaxNameTableCharCount = 16384;
	binding.Security.Mode = BasicHttpSecurityMode.Transport;
	binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Certificate;
	binding.Security.Transport.ProxyCredentialType = HttpProxyCredentialType.None;            
	//binding.Security.Transport.Realm = "";            
	//binding.Security.Message.ClientCredentialType = BasicHttpMessageCredentialType.Certificate;
	//binding.Security.Message.AlgorithmSuite = SecurityAlgorithmSuite.Default;

	var endpoint = new EndpointAddress("https://dlws.bloomberg.com/dlps");
	var client = new PerSecurityWSClient(binding, endpoint);
	var clientCert = new X509Certificate2(Properties.Resources.DLWSCert, "<cert secret>");
	client.ClientCredentials.ClientCertificate.Certificate = clientCert;

	return client;
}

Fiddler traces show TLS handshake succeeding:

CONNECT dlws.bloomberg.com:443 HTTP/1.1
Host: dlws.bloomberg.com:443

A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

Version: 3.3 (TLS/1.2)
Random: 5B 5B 29 76 72 A8 37 E8 E2 4D AD 4A 8A FB 2B 55 53 61 83 63 78 98 C0 A1 7E 31 36 0E D1 42 7B 64
"Time": 10/26/2032 12:44:11 PM
SessionID: empty
Extensions: 
	server_name	dlws.bloomberg.com
	elliptic_curves	unknown [0x1D), secp256r1 [0x17], secp384r1 [0x18]
	ec_point_formats	uncompressed [0x0]
	signature_algs	sha256_rsa, sha384_rsa, sha1_rsa, sha256_ecdsa, sha384_ecdsa, sha1_ecdsa, sha1_dsa, sha512_rsa, sha512_ecdsa
	SessionTicket	empty
	extended_master_secret	empty
	renegotiation_info	00
Ciphers: 
	[C02C]	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
	[C02B]	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
	[C030]	TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	[C02F]	TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
	[009F]	TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
	[009E]	TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
	[C024]	TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
	[C023]	TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
	[C028]	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
	[C027]	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
	[C00A]	TLS1_CK_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
	[C009]	TLS1_CK_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
	[C014]	TLS1_CK_ECDHE_RSA_WITH_AES_256_CBC_SHA
	[C013]	TLS1_CK_ECDHE_RSA_WITH_AES_128_CBC_SHA
	[0039]	TLS_DHE_RSA_WITH_AES_256_SHA
	[0033]	TLS_DHE_RSA_WITH_AES_128_SHA
	[009D]	TLS_RSA_WITH_AES_256_GCM_SHA384
	[009C]	TLS_RSA_WITH_AES_128_GCM_SHA256
	[003D]	TLS_RSA_WITH_AES_256_CBC_SHA256
	[003C]	TLS_RSA_WITH_AES_128_CBC_SHA256
	[0035]	TLS_RSA_AES_256_SHA
	[002F]	TLS_RSA_AES_128_SHA
	[000A]	SSL_RSA_WITH_3DES_EDE_SHA
	[006A]	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
	[0040]	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
	[0038]	TLS_DHE_DSS_WITH_AES_256_SHA
	[0032]	TLS_DHE_DSS_WITH_AES_128_SHA
	[0013]	SSL_DHE_DSS_WITH_3DES_EDE_SHA

Compression: 
	[00]	NO_COMPRESSION

HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 10:17:26.127
Connection: close

Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.

Secure Protocol: Tls12
Cipher: Aes256 256bits
Hash Algorithm: Sha384 ?bits
Key Exchange: ECDHE_RSA (0xae06) 256bits

== Client Certificate ==========
[Subject]
  E=datalicense@bloomberg.net, CN=<redacted>:DLWS, OU=DataLicense, O=Bloomberg L.P, S=NY, C=US

[Issuer]
  CN=Data License SubCA

[Serial Number]
  <redacted>

[Not Before]
  6/2/2016 4:29:29 PM

[Not After]
  6/2/2021 4:39:29 PM

[Thumbprint]
  <redacted>


== Server Certificate ==========
[Subject]
  CN=dlws.bloomberg.com, O=Bloomberg LP, L=New York, S=New York, C=US, SERIALNUMBER=2110234, OID.1.3.6.1.4.1.311.60.2.1.2=Delaware, OID.1.3.6.1.4.1.311.60.2.1.3=US, OID.2.5.4.15=Private Organization

[Issuer]
  CN=DigiCert SHA2 Extended Validation Server CA, OU=www.digicert.com, O=DigiCert Inc, C=US

[Serial Number]
  0CA53FCF9C01626E39697942999764F6

[Not Before]
  12/12/2017 7:00:00 PM

[Not After]
  12/18/2019 7:00:00 AM

[Thumbprint]
  899E0AEE49C6A6BF5AD87A0AD1C780DF149F32F3

[SubjectAltNames]
dlws.bloomberg.com, services.bloomberg.com

After handshake, here is the proceeding request and response.

POST https://dlws.bloomberg.com/dlps HTTP/1.1
Cache-Control: no-cache, max-age=0
SOAPAction: "submitGetDataRequest"
Expect: 100-continue
Accept-Encoding: gzip, deflate
Content-Type: text/xml; charset=utf-8
Content-Length: 643
Host: dlws.bloomberg.com

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><submitGetDataRequest xmlns="http://services.bloomberg.com/datalicense/dlws/ps/20071001"><headers><programflag>oneshot</programflag><secid>BB_GLOBAL</secid><secmaster>true</secmaster><yellowkey>Equity</yellowkey><pricing>true</pricing></headers><fields><field>ID_BB_UNIQUE</field><field>PX_LAST</field></fields><instruments><instrument><id>MSFT.US</id><yellowkey>Equity</yellowkey><type>TICKER</type></instrument></instruments></submitGetDataRequest></s:Body></s:Envelope>

The error occurs immediately after digesting the response below.

HTTP/1.1 200 OK
Connection: Keep-Alive
Date: Fri, 27 Jul 2018 14:17:26+0000
Via: BAS/1.0 nybasv3
Content-Length: 625
Content-Type: text/xml; charset=utf-8
Server: Apache-Coyote/1.1
bb-request-id: d916550e-36a4-4be5-bc24-4cba127367d2
com.bloomberg.bas-recipientTask: machine=0; instance=0;
com.bloomberg.bas-serviceId: 125322
com.bloomberg.bas-serviceVersion: 3.1
Set-Cookie: machineNumber=0; path=/
X-Powered-By: Servlet 2.4; JBoss-4.3.0.GA_CP04 (build: SVNTag=JBPAPP_4_3_0_GA_CP04 date=200902200048)/JBossWeb-2.0

<?xml version="1.0" encoding="UTF-8" ?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><dlws:submitGetDataResponse xmlns='http://services.bloomberg.com/datalicense/dlws/ps/20071001' xmlns:dlws="http://services.bloomberg.com/datalicense/dlws/ps/20071001" xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"><dlws:statusCode><dlws:code>0</dlws:code><dlws:description>Success</dlws:description></dlws:statusCode><dlws:requestId>d916550e-36a4-4be5-bc24-4cba127367d2</dlws:requestId><dlws:responseId>1532701047-721515399</dlws:responseId></dlws:submitGetDataResponse></soap:Body></soap:Envelope>

I thought maybe it was related to dotnet/runtime#22404 with invalid headers but they appear to be OK according to these checks (valid-header-name.js and valid-header-value.js.)

Any help on the subject would be appreciated.