aspnetcore: Compression under HTTPS with `dotnet watch` and `launchBrowser: true` breaks some HTML responses

Describe the bug

Apologies if this is the wrong repo for this, took my best guess.

When using compression under HTTPS, with dotnet watch run and a certain controller response with launchBrowser: true, Kestrel responds with a badly encoded compressed response.

I’m not sure exactly which conditions need to be met for the controller response to trigger this bad behaviour but repro case seems to trigger it reliably on my machine. As far as I can tell the response must be chunked in a specific way to trigger this.

To Reproduce

~~Please use my zipped isolated bad case.~~ ~~badcase-isolate.zip~~

EDIT 2021-05-17 1:43 PM: You can now find the bad case example in this repo.

When it’s uncompressed simply run dotnet watch run NOTE: This doesn’t reproduce when running without watch.

Navigate to https://localhost:5001 and ignore SSL warning.

Exceptions (if any)

Chrome says: GET https://localhost:5001/ net::ERR_CONTENT_DECODING_FAILED 200 and displays nothing.

Looking at fiddler is slightly more informative.

Firstly Fiddler claims “Response body is encoded.” However attempting to decode in Fiddler results in the following error in the logs:

09:03:37:0882 !Cannot decode HTTP response using Encoding: br; error: System.Exception Brotli conversion failed
09:03:37:0882 !Cannot decode HTTP response using Content-Encoding: br

Secondly looking at the raw response in fiddler shows some interesting stuff:
- Specifically notice what looks like uncompressed plaintext injection of /_framework/aspnetcore-browser-refresh.js into the html body.

To me it looks like something in Kestrel or some built-in middleware in ASP.Net core, has decided to inject the browser link functionality, despite the fact that I never enabled or triggered this. It looks like whatever decides to do this is not aware or unable to resolve the fact that the response is compressed and therefore it cannot be simply injected as plaintext.

Further technical details

ASP.NET Core version 3.1.15
dotnet --info

.NET SDK (reflecting any global.json):
 Version:   5.0.201
 Commit:    a09bd5c86c

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19042
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.201\

Host (useful for support):
  Version: 5.0.4
  Commit:  f27d337295

.NET SDKs installed:
  2.1.812 [C:\Program Files\dotnet\sdk]
  3.1.202 [C:\Program Files\dotnet\sdk]
  3.1.300 [C:\Program Files\dotnet\sdk]
  3.1.409 [C:\Program Files\dotnet\sdk]
  5.0.102 [C:\Program Files\dotnet\sdk]
  5.0.201 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Using the latest release VS Code 1.56.2

Please let me know what else I can do to assist further in diagnosing and resolving this issue.

EDIT 2021-05-17 1:43 PM: I replaced the badcase-isolate.zip file with a link to this repo

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 31 (15 by maintainers)

Most upvoted comments

This was improved for .NET 7 as the script is no longer injected if the closing body tag cannot be located (e.g. because it’s compressed) or the content type is not UTF8, and the script-injection now detects that the failure is likely caused by the response compression and logs a warning. I’ve confirmed this using the .NET 7 SDK with the originally posted 3.1 repro app.

You can manually add the script with a block like the following:

@* Render the browser reload script *@
@if (Environment.GetEnvironmentVariable("__ASPNETCORE_BROWSER_TOOLS") != null)
{
<script src="/_framework/aspnetcore-browser-refresh.js"></script>
}

DamianEdwards on Oct 20, 2022

I don’t understand why is it so hard to let us do the work required manually, especially in cases like these. I had no idea why Hot Reload wasn’t working for me for weeks now, and at first glance (a deep glance, though) it was related to response compression. Once I removed that, there was an error for something trying to connect to an unsecure HTTP endpoint (not even my own).

Oh yes, I shouldn’t have used CSP either? That’s the wrong answer. Provide a TagHelper or docs for me to manually enable Hot Reload. I don’t want my HTTP responses messed with that inject script tags into what either is or isn’t HTML. What’s worse is that there is no reference to either: the middleware doesn’t log any errors, the script doesn’t get included, and there’s not much to go on. I had to literally make a new project and do bottom-up debugging what’s different.

The problem for me wasn’t that the response was jumbled. There seemingly were no errors whatsover in the browser or in Kestrel logs, just Hot Reload kept throwing a “General Exception” in Visual Studio.

I’m all for reducing boilerplate. What I’m not okay with is there being IDE-specific or ambient settings that can break the application or IDE that are not explicit in any way.

yugabe on Jun 23, 2022

I appreciate your feedback @yugabe but as you can imagine we are trying to balance the experiences of different types of users and sometimes that results in less-than-ideal side-effects for some subset. In this case, it seems we should still pursue the idea being able to disable just the script-injection so that apps that wish to manually configure the script can do so cleanly.

I propose we add a new launch profile property and/or environment variable that disables the script injection, that would be honored by dotnet watch and Visual Studio.

DamianEdwards on Oct 29, 2022

The changes are in the SDK, not the runtime, so the LTS consideration doesn’t apply in the same way. Visual Studio 2022 has already updated to use .NET 7 SDK in the latest previews.

DamianEdwards on Oct 25, 2022

So in my case was caused by enabling Projects and Solutions > ASP.NET Core > CSS Hot Reload

Upgrading to Visual Studio 2022 17.1.0 fixed the issue

jtkech on Mar 6, 2022

RE the launch profile setting, my local testing on an ASP.NET Core 6 project in latest VS 2022 shows that setting "hotReloadEnabled": false in a launch profile disables Hot Reload but preserves the traditional Edit and Continue experience. The big difference there of course is the ability to apply changes without launching under the debugger. We’re looking at improving the documentation around these options along with improving the VS UX to allow better discovery of issues and more control over the elements of the experience (e.g. ability to disable just the script injection).

As you point out, given the nature of the web, there are countless ways dynamic script injection can cause issues or fail to actually deliver the script intended. The way we’ve implemented this right now is indeed causing pain for more folks than perhaps we anticipated. Your suggestion of using a feature (and perhaps endpoint metadata) is a good one, and I imagine we’d auto-opt-in responses generated by Razor Pages/MVC. @pranavkm this is something we should consider for .NET 7.

DamianEdwards on Jan 25, 2022

@roa-nyx I have tried reproducing using other HTML, or HTML with an equivalent length. I’m not at all sure why but when I change the HTML significantly it starts working.

The middleware that injects the script tags to load the hot reload instrumentation performs a naive string find and replace/insert. If the response compression middleware produces a result which for whatever reason contains the token it is looking for (which iirc is a basic closing </body> tag) verbatim, which CAN easily ‘randomly’ happen considering it’s just a sequence of bytes that is being interpreted as, probably UTF-8, encoded text – then it does the swap and corrupts the compressed stream.

If it can’t, then it doesn’t – meaning the response will remain valid and work, but the hot reload features won’t.

@pranavkm But I’m not sure how we could corrupt the response.

Here’s your answer: https://github.com/dotnet/sdk/blob/36621631f5982fb9d23644aec68222015ba337a2/src/BuiltInTools/BrowserRefresh/WebSocketScriptInjection.cs#L28

Just treating any old response body for responses that are of Content-Type: text/html will break randomly on compressed response streams that happen to contain a series of bytes that matches with the UTF-8 byte-encoding of the string "</body>". And it will randomly break on any character encoding other than UTF-8 where the same series of bytes may mean something else entirely as well.

Nobody working on the development of this feature thought it was a bad idea to just treat ‘whatever bytes’ as always representing UTF-8 encoded text? Seriously? Nobody? My god…

rjgotten on Jan 24, 2022

The script tag for bundles/site.js doesn’t appear to be injected dynamically. It’s just here in the layout file code:

https://github.com/roa-nyx/dotnetcore-bad-compression-example/blob/main/Views/Shared/_Layout.cshtml#L192

SteveSandersonMS on May 17, 2021