aspnetcore: AspNetCore 2.1-preview2 - System.Text.DecoderFallbackException

Hi,

I recently upgraded my code from asp.net core 2.0 to asp.net core 2.1-preview2. Since we upgraded we are encountering a crash in the production (for some requests) which never occurred before for the same code:

System.Text.DecoderFallbackException · Unable to translate bytes [C7] at index 829 from specified code page to Unicode.
System.Text.DecoderExceptionFallbackBuffer.Throw(byte[] bytesUnknown, int index)	
System.Text.DecoderExceptionFallbackBuffer.Fallback(byte[] bytesUnknown, int index)	
System.Text.DecoderFallbackBuffer.InternalFallback(byte[] bytes, Byte* pBytes, ref Char* chars)	
System.Text.UTF8Encoding.GetChars(Byte* bytes, int byteCount, Char* chars, int charCount, DecoderNLS baseDecoder)	
System.Text.DecoderNLS.GetChars(byte[] bytes, int byteIndex, int byteCount, char[] chars, int charIndex, bool flush)	
System.Text.DecoderNLS.GetChars(byte[] bytes, int byteIndex, int byteCount, char[] chars, int charIndex)	
Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.ReadIntoBuffer()	
Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.Read(char[] buffer, int index, int count)	
Newtonsoft.Json.JsonTextReader.ReadData(bool append, int charsRequired)	
Newtonsoft.Json.JsonTextReader.ParseValue()	
Newtonsoft.Json.JsonTextReader.Read()	
Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, bool hasConverter)	
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, bool checkAdditionalContent)	
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()	
Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter+<ReadRequestBodyAsync>d__17.MoveNext()	
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()	
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)	
System.Runtime.CompilerServices.TaskAwaiter<TResult>.GetResult()	
Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder+<BindModelAsync>d__7.MoveNext()	
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()	
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)	
Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder+<BindModelAsync>d__11.MoveNext()	
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()	
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)	
System.Runtime.CompilerServices.TaskAwaiter<TResult>.GetResult()	
Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider+<>c__DisplayClass0_0+<<CreateBinderDelegate>g__Bind|0>d.MoveNext()	
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()	
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)	
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__13.MoveNext()	
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()	
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)	
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextExceptionFilterAsync>d__24.MoveNext()

It seems request parsing is failing which wasn’t happening before. Can’t even log the request model as crash occurs before the controller action is called.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 19 (14 by maintainers)

Most upvoted comments

Based on https://github.com/aspnet/Home/issues/3105#issuecomment-385252992, https://github.com/aspnet/Home/issues/3105#issuecomment-385492323 and the callstack, the scenario is that there’s a C++ client posting json data(having invalid characters) to asp.net core app and we failed to decode these characters in the server. AFAIK, we need not fix anything at our end.

So it sounds like everything here was behaving appropriately when presented with invalid bytes from the client.

@kichalla I can’t provide a wireshark trace because it’s happening on machines of our customers. However, I can share how I fixed the issue:

Our C++ HTTP client gets hostname of machine using GetComputerNameA function.

Some machines seem to have non-ASCII characters in the hostname, so I changed it to GetComputerName which is the utf-16 version of the function, and before http post I converted the utf-16 hostname string to utf-8 hostname string. This fixed the issue at my end.

E.g. a wireshark trace