runtime: New Asp.NET Core 3.0 Json doesn't serialize Dictionary

For the newcomers, the temporary solution is to revert back to Newtonsoft.Json.

1. Add package reference to Microsoft.AspNetCore.Mvc.NewtonsoftJson.
2. Add .AddNewtonsoftJson() just after .AddControllers() / .AddMvc() or any other combination.

Are there any plans to support this in .net core 3.1?

Setting milestone for 3.1 to remove any restrictions that prevents a custom converter from being created that can handle any TKey for Dictionary<TKey,TValue>.

Setting milestone to 5.0 for consideration (what if any of the above examples should work by default).

I too just hit this, and what a scary silent limitation it is, since as others have pointed out you won’t see this at compile time. In my case, I want to serialize Dictionary<int, List<string>>, which doesn’t strike me as particularly exotic.

@ahsonkhan I believe the key motivation is compatibility. The previous default serializer was Newtonsoft, so users may have written entire models relying on its ability to serialize and deserialize arbitrary classes. Migrating from 2.X to 3.0 will now cause a silent breaking change since we’ll only know the incompatibility at runtime.

I believe many scenarios involve using json just as transport across the wire, and in this case, the domain model might be Dictionary<int,TValue>. Your suggestion boils down to creating a separate DTO object Dictionary<string,TValue> and converting between the two, seems rather inefficient since now we need to allocate another object just to be compliant with the serializer. Looking strictly to the serializer, restricting dictionary to having string keys is logical as that’s the only possible json representation. But considering that the serializer plays a role in applications I believe it’s best to remove as much friction as possible.

@onionhammer .NET 5.0, you can also try out the feature in the next preview (5.0 preview8). There are no plans of porting this into 3.x.

I have implemented a converter which support both Serialization and Deserialization for IDictionary<TKey, TValue> where TKey has a static method TKey Parse(string):

using System;
using System.Collections;
using System.Collections.Generic;
using System.Globalization;
using System.Reflection;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace JsonDictionaryConverter
{
    sealed class JsonNonStringKeyDictionaryConverter<TKey, TValue> : JsonConverter<IDictionary<TKey, TValue>>
    {
        public override IDictionary<TKey, TValue> Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            var convertedType = typeof(Dictionary<,>)
                .MakeGenericType(typeof(string), typeToConvert.GenericTypeArguments[1]);
            var value = JsonSerializer.Deserialize(ref reader, convertedType, options);
            var instance = (Dictionary<TKey, TValue>)Activator.CreateInstance(
                typeToConvert,
                BindingFlags.Instance | BindingFlags.Public,
                null,
                null,
                CultureInfo.CurrentCulture);
            var enumerator = (IEnumerator)convertedType.GetMethod("GetEnumerator")!.Invoke(value, null);
            var parse = typeof(TKey).GetMethod("Parse", 0, BindingFlags.Public | BindingFlags.Static, null, CallingConventions.Any, new[] { typeof(string) }, null);
            if (parse == null) throw new NotSupportedException($"{typeof(TKey)} as TKey in IDictionary<TKey, TValue> is not supported.");
            while (enumerator.MoveNext())
            {
                var element = (KeyValuePair<string?, TValue>)enumerator.Current;
                instance.Add((TKey)parse.Invoke(null, new[] { element.Key }), element.Value);
            }
            return instance;
        }

        public override void Write(Utf8JsonWriter writer, IDictionary<TKey, TValue> value, JsonSerializerOptions options)
        {
            var convertedDictionary = new Dictionary<string?, TValue>(value.Count);
            foreach (var (k, v) in value) convertedDictionary[k?.ToString()] = v;
            JsonSerializer.Serialize(writer, convertedDictionary, options);
            convertedDictionary.Clear();
        }
    }

    sealed class JsonNonStringKeyDictionaryConverterFactory : JsonConverterFactory
    {
        public override bool CanConvert(Type typeToConvert)
        {
            if (!typeToConvert.IsGenericType) return false;
            if (typeToConvert.GenericTypeArguments[0] == typeof(string)) return false;
            return typeToConvert.GetInterface("IDictionary") != null;
        }

        public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions options)
        {
            var converterType = typeof(JsonNonStringKeyDictionaryConverter<,>)
                .MakeGenericType(typeToConvert.GenericTypeArguments[0], typeToConvert.GenericTypeArguments[1]);
            var converter = (JsonConverter)Activator.CreateInstance(
                converterType,
                BindingFlags.Instance | BindingFlags.Public,
                null,
                null,
                CultureInfo.CurrentCulture);
            return converter;
        }
    }
}

Test:

class Entity
{
    public string Value { get; set; }
}
class TestClass
{
    public Dictionary<int, Entity> IntKey { get; set; }
    public Dictionary<float, Entity> FloatKey { get; set; }
    public Dictionary<double, Entity> DoubleKey { get; set; }
    public Dictionary<DateTime, Entity> DateTimeKey { get; set; }
    public Dictionary<string, Entity> StringKey { get; set; }
}
class Program
{
    static void Main(string[] args)
    {
        var options = new JsonSerializerOptions();
        options.Converters.Add(new JsonNonStringKeyDictionaryConverterFactory());
        var x = new TestClass
        {
            IntKey = new Dictionary<int, Entity> { [1] = new Entity { Value = "test" } },
            FloatKey = new Dictionary<float, Entity> { [1.3f] = new Entity { Value = "test" } },
            DoubleKey = new Dictionary<double, Entity> { [1.35] = new Entity { Value = "test" } },
            DateTimeKey = new Dictionary<DateTime, Entity> { [DateTime.Now] = new Entity { Value = "test" } },
            StringKey = new Dictionary<string, Entity> { ["test"] = new Entity { Value = "test" } }
        };

        var value = JsonSerializer.Serialize(x, options);
        Console.WriteLine(value);
        var obj = JsonSerializer.Deserialize<TestClass>(value, options);
        Console.WriteLine(JsonSerializer.Serialize(obj, options));
    }
}

Result:

{"IntKey":{"1":{"Value":"test"}},"FloatKey":{"1.3":{"Value":"test"}},"DoubleKey":{"1.35":{"Value":"test"}},"DateTimeKey":{"8/25/2019 6:47:48 PM":{"Value":"test"}},"StringKey":{"test":{"Value":"test"}}}
{"IntKey":{"1":{"Value":"test"}},"FloatKey":{"1.3":{"Value":"test"}},"DoubleKey":{"1.35":{"Value":"test"}},"DateTimeKey":{"8/25/2019 6:47:48 PM":{"Value":"test"}},"StringKey":{"test":{"Value":"test"}}}

However it still cannot serialize a nested Dictionary such as Dictionary<int, Dictionary<int, int>> because System.Text.Json doesn’t accept the inner type Dictionary<int, int>. I think it’s a bug.

Deserialization also seems to map generic object types to JsonDocument rather than their normal (primitive?) types.

Yes that is by design. See https://github.com/dotnet/corefx/issues/38713

Here’s the proposal for adding support to non-string TKey types in dictionaries. https://github.com/dotnet/runtime/pull/32676

Please let us know any thoughts or concerns.

I just upgraded to 3.1 and got hit with this. Back to JSON.NET I go… (I use GUID keys)

Deserialization also seems to map generic object types to JsonDocument rather than their normal (primitive?) types.

Example:

string test = "[{\"id\":86,\"name\":\"test\"}]";
var SystemTextJson = System.Text.Json.JsonSerializer.Deserialize<List<Dictionary<string, object>>>(test);
var NewtonSoftJson = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Dictionary<string, object>>>(test);

SystemTextJson[0][“id”] shows as: ValueKind = Number : “86” NewtonSoftJson[0][“id”] shows as: 86

They should fix it but I see this time and time again even with the old formatter , binary formatter early newtsoft , dictionaries in dictionaries , dictionaries with interfaces . They should fix it but if you dont want trouble people really shouldn’t put complex objects like Dictionaries in serialization contracts your asking for trouble - newtsoft has spoiled people . Look at all the public properties on Dictionary count etc you are relying on something custom in the serializer to map this.

Unfortunate there is not a simple type for this in C# for property names so Dictionary is forced. So im just sad…

Update: I’ve added samples that work with 3.0. I didn’t notice any issues such as with nested dictionaries as reported above.

There are two JSON formats being used in the samples:

• JSON object with string properties: {"1":"val1","2":"val2"} even though TKey is not a string.
  • These rely on a TryParse method on the corresponding supported key type. For v3.0 it is not feasible to provide support for a generalized TKey since TryParse methods can be different for any TKey and because there are culture settings that need to be provided (typically Invariant). So the samples below are for a single TKey type.
  • Sample Dictionary<int, string>. This is a simple example for just TValue == string.
  • Sample Dictionary<Guid, TValue>. This can handle any TValue.
  • Sample Dictionary<TKey, TValue> where TKey is Enum. For Enums; this can handle any TValue.
• JSON array with KeyValuePair entries: [{"Key":1,"Value":"val1"},{"Key":2,"Value":"val2"}]
  • Sample Dictionary<int, string>. This is a simple example for just TValue == string.
  • Sample Dictionary<TKey, TValue> . This can handle any TKey and TValue because the types are natively represented in JSON and don’t require TryParse.

If these samples are satisfactory, I will change this issue to 5.0 in order to discuss whether we provide built-in support that don’t require custom converters.

@Jozkee so is this coming in .NET 5 only or will it be go into 3.*?

This won’t be backported to 3.x but you can add use the System.Text.Json NuGet package in your project to get all the new features in .NET 5.

Solution for asp net core 3.x:

var dic1 = new Dictionary<TKey, TValue>(); 
return Json(new { dic1 }); // does not work

var dic2 = from i in new Dictionary<TKey, TValue>() select new { i.Key, i.Value }
return Json(new { dic2 });  //works prety well

Here’s a workaround but by no means a complete solution:

   [JsonConverter(typeof(DictionaryConverter))]
   public Dictionary<string, object> ExtraProperties { get; set; } = new Dictionary<string, object>();

public class DictionaryConverter : JsonConverter<Dictionary<string, object>>
{
    public override Dictionary<string, object> Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        var dictionary = JsonSerializer.Deserialize<Dictionary<string, object>>(ref reader, options);

        foreach (string key in dictionary.Keys.ToList())
        {
            if (dictionary[key] is JsonElement je)
            {
                dictionary[key] = Unwrap(je);
            }
        }

        return dictionary;
    }

    public override void Write(Utf8JsonWriter writer, Dictionary<string, object> value, JsonSerializerOptions options)
        => JsonSerializer.Serialize(writer, value, options);

    private static object Unwrap(JsonElement je)
    {
        return je.ValueKind switch
        {
            JsonValueKind.String => je.ToString(),
            JsonValueKind.Number => je.TryGetInt64(out var l) ? l : je.GetDouble(),
            JsonValueKind.True => true,
            JsonValueKind.False => false,
            JsonValueKind.Array => je.EnumerateArray().Select(Unwrap).ToList(),
            JsonValueKind.Object => je.EnumerateObject().ToDictionary(x => x.Name, x => Unwrap(x.Value)),
            _ => null
        };
    }

@israellot, @unruledboy, and others on the thread, can you provide details on why your object model requires Dictionaries with integer keys in your scenarios and why changing it to be Dictionary<string, TValue> wouldn’t work? I would love to see some usages for gathering requirements and to help motivate the fix.

They would get serialized as strings anyway, so I don’t understand in what scenarios you’d want to your underlying dictionary to have int32 keys instead.

Deserialization also seems to map generic object types to JsonDocument rather than their normal (primitive?) types.

Example:

string test = "[{\"id\":86,\"name\":\"test\"}]";
var SystemTextJson = System.Text.Json.JsonSerializer.Deserialize<List<Dictionary<string, object>>>(test);
var NewtonSoftJson = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Dictionary<string, object>>>(test);

SystemTextJson[0][“id”] shows as: ValueKind = Number : “86” NewtonSoftJson[0][“id”] shows as: 86

Of all the mentioned issues, this bothers me the most. List<T> or T[] or Dictionary<string,object> should deserialize properly for any type that can be mapped directly from the few types Json has to CLR types.

The 3.0 release is intended to be a minimal viable product with the most common scenarios supported

I wonder how List<Dictionary<string,object>> isn’t one of the most common scenarios:

[// N objects
{"a":4},
{"b","Bla"},
]

As this does also deserialize into System.Text.JsonElement, where I would expect double (Number) and string (String)

Any ideas on if there’s a hoped for release date of Preview 8?

Some time later this month.

Thinking about this some more, removing the up-for-grabs for now since it may be something we don’t want to support by default.

@steveharter, @layomia - is there a potential workaround here in the meantime? What would it take to add support for non-string key’d dictionary within the serializer for 5.0?

@ahsonkhan @willyt150 the workaround for this is to use a custom converter that implements JsonConverter<T> where T is Dictionary<int, string>. See https://github.com/dotnet/corefx/issues/36639#issue-429928740 for some examples.

Presumably anything you can currently serialize as a standalone object?

Yes.

@roguecode here’s a Enum sample for Dictionary<TKey, TValue> where TKey is an Enum and uses the “property” JSON syntax instead of KeyValuePair. I also updated the list of samples above to include this new sample.

I’ve ran into this issue with a small program I’m writing where parts of a version label are provided through a json file. The label parts have a key that specifies the index where the label part may be inserted. This means the keys are numeric values, e.g

{
  "parts" : {
    "1" : "alpha",
    "3" : "beta"
  }
}

Using Newtonsoft, the json can be deserialized without issue to a Dictionary<int, string>. After converting to System.Text.Json serialization failed.

I’ve resolved this issue by creating my own DictionaryConverter, and DictionaryConverter<T,K>. I also created a simple converter that allows integers to be deserialized from a string

These are then registered through the serializer options: https://github.com/Kieranties/SimpleVersion/blob/master/src/SimpleVersion.Core/Serialization/Serializer.cs#L22

These changes allow the keys of a dictionary to be deserialized instead of directly read as a string. This further opens up support for keys to be arbitrary types that could have their own converters registered for serialization (e.g. enums/type/ types that may be serialized as unique keys etc)

I’ve not formally tested things but within the current development this seems to have resolved the issue.

From @israellot in https://github.com/dotnet/corefx/issues/41345

var dictionary = new Dictionary<int, int>()
            {
                [0] = 1
            };

 var serialized = System.Text.Json.JsonSerializer.Serialize(dictionary);

This simple serialization is handled well by the former default Newtonsoft json library by serializing the int key as string. On System.Text.Json is throws a not supported exception.

Oh boy, had this problem right after I upgraded to 3.0.

Had to install the newton package with AddNewtonsoftJson.

@steveharter At least you shouldn’t throw a notsupportexception when there’s a useable converter.

Yes that is a fair point. Perhaps we can remove this restriction for 3.1. cc @layomia

Also just to clarify that today dictionary elements are serialized like properties which is possible because the key is a string. Supporting non-string keys means elements will be serialized as a KeyValuePair.

Thank you very much for your quick replies @ahsonkhan!

The “limitation” of the key being a string actually makes sense when I think about it. I see now that Json.net actually generates json with the key being a string, when deserializing it would just get me an int back. It would definitely be nice having the support for non-string keys in the future, but not a show stopper.

Ok, glad to hear that the Mvc.SerializableError not being supported has been fixed. Any ideas on if there’s a hoped for release date of Preview 8? Tried to search and find something, but not seeing anything about that.

Once preview 8 comes out we’ll try giving the .net core 3 json serialization library a try again, but for now we’re needing to stick with Json.net

System.NotSupportedException: The collection type ‘Microsoft.AspNetCore.Mvc.SerializableError’ is not supported.

This was a known issue https://github.com/aspnet/AspNetCore/issues/11459 that was recently fixed (as part of preview 8): https://github.com/dotnet/corefx/pull/39001