runtime: Fail null deserialization on non-nullable reference types

Any improvements with .NET 7 and required properties? I see the JsonSerializer is failing with a JsonException when the JSON payload is missing a required property. However, it’s not failing when the property is there, explicitly set with a null value.

@Hottemax biggest problem is that (de)serializers in general have virtually infinite number of knobs and expectations how people want it to work. We have significant number of open issues and requests and we try to make sure we look at them holistically rather than per single request/issue. We cannot just remove APIs once added so we need to be conservative in that choice. This issue has accumulated enough upvotes for it to be considered for next release but note that it doesn’t guarantee we will do it. Soon we will look at all issues in 9.0 and Future milestone and make some initial choices but keep in mind that it’s overall almost 200 issues (12 being already considered for 9.0). There is a huge difference in how much time it takes to write simple prototype code like one above vs actual code which ships - latter requires much more time for testing and convincing people on the right design. We need to pick which of these issues are currently most important for product as a whole (i.e. our main scenario is seamless interaction with ASP.NET Core which might be invisible for most of the users) but upvoted issues like this one are also a big factor for us when picking. Certainly having someone from community doing some prototyping would help here. I.e. extending code above to figure out all corner cases, some test cases etc would decrease amount of work for us and make it more likely it would ship next release.

ASP.net Core can detect properties that are non-nullable reference types at runtime. It even rejects the request if one of the non-nullable property is null. I however don’t know if their code can detect non-nullable parameters as well.

For example, if I have this class:

   public class Dto
   {
      [JsonConstructor]
      public Dto(string value, string? optionalValue)
      {
         Value = value;
         OptionalValue = optionalValue;
      }

      public string Value { get; }

      public string? OptionalValue { get; }
   }

and the JSON payload is empty {}

ASP.net returns:

{
    "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
    "title": "One or more validation errors occurred.",
    "status": 400,
    "traceId": "00-5b5a8fd1b496464796a2992b194b5482-fb67217921735b4f-00",
    "errors": {
        "Value": [
            "The Value field is required."
        ]
    }
}

Note that only Value is required because OptionalValue is a nullable reference type.

The ASP.net code that detects non-nullable reference types is in DataAnnotationsMetadataProvider (link to the code) and it can be toggled on/off via MvcOptions.SuppressImplicitRequiredAttributeForNonNullableReferenceTypes

@ahsonkhan EF Core is actually doing this, when you have NRTs enabled, they’ll make columns NULL / NOT NULL based on how your model class is annotated, see https://docs.microsoft.com/en-us/ef/core/miscellaneous/nullable-reference-types

Any chance a fix for this is coming with .net 5?

This is currently possible to achieve with contract customization except for top level (we don’t have info on nullability for top level - for top level you can do null check separately). Here is example implementation:

using System.Reflection;
using System.Text.Json;
using System.Text.Json.Serialization.Metadata;

JsonSerializerOptions options = new()
{
    TypeInfoResolver = new DefaultJsonTypeInfoResolver()
    {
        Modifiers = { BlockNullForNonNullableReferenceTypesModifier }
    }
};

string json = """
    {
      "NonNullableString": null,
      "NullableString": null
    }
    """;

TestClass testClass = JsonSerializer.Deserialize<TestClass>(json, options); // JsonException: Null value not allowed for non-nullable property.


static void BlockNullForNonNullableReferenceTypesModifier(JsonTypeInfo jsonTypeInfo)
{
    if (jsonTypeInfo.Type.IsValueType)
        return;

    NullabilityInfoContext context = new();
    foreach (JsonPropertyInfo property in jsonTypeInfo.Properties)
    {
        Action<object, object?>? setter = property.Set;

        if (setter == null)
            continue;

        NullabilityInfo? nullabilityInfo = null;
        switch (property.AttributeProvider)
        {
            case FieldInfo fieldInfo:
                nullabilityInfo = context.Create(fieldInfo);
                break;
            case PropertyInfo propertyInfo:
                nullabilityInfo = context.Create(propertyInfo);
                break;
        }

        if (nullabilityInfo == null)
            continue;

        if (nullabilityInfo.WriteState == NullabilityState.NotNull)
        {
            property.Set = (obj, val) =>
            {
                if (val == null)
                    throw new JsonException("Null value not allowed for non-nullable property.");

                setter(obj, val);
            };
        }
    }
}

class TestClass
{
    public string NonNullableString { get; set; }
    public string? NullableString { get; set; }
}

there is extra work needed to cover all corner cases (array values etc) but this should hopefully cover most of the scenarios

I understand, but at the same time a type system is not the same thing as a serialization contract. In automatic serializers we use convention to map type system features to serialization contracts in a way that feels intuitive to the user. One good example is the required keyword, which is mapped from a C# type system concept to a similar (but not identical) feature in the serialization contract, and despite the fact that required isn’t enforced at run time.

Mapping non-nullable annotations to a “require non-null” deserialization contract seems useful to me for a couple of reasons:

1. It provides a language-integrated way for declaring intent, as opposed to needing to decorate the property with yet another attribute.
2. The deserialized result honours the nullability annotation of its model, meaning that users don’t need to take extra steps checking for nulls downstream.

I don’t believe it’s any different from:

I should clarify that the scope of this feature is limited to member and constructor parameter annotations. We can’t add this to generic parameters/collection elements due to constraints in their runtime representation, at least in the reflection serializer.

That’s a lotta code when your model has dozens or hundreds of properties that shouldn’t be null.

This ticket would be opt-in, no doubt. And I think it’s still a great idea if it’s implementable.

Far from final, but here’s a sketch: #1256 (comment). Will also need switches on the contract model for the source generator to be able to access.

Thanks. If it’s opt-in, my primary concerns are alleviated.

Supporting the feature would require OOBing the NullabilityInfoContext/NullabilityInfo APIs for these TFMs which to my knowledge hasn’t happened yet.

Those APIs are just using reflection to get at the relevant information. If those APIs aren’t exposed publicly downlevel, S.T.J could still achieve the same thing using reflection, even by building in a copy of those implementations as internal (and with whatever tweaks might be needed to work downlevel).

They also affect how Entity Framework creates columns - an NRT causes a column to be nullable.

This “key aspect” is so bad that not even Microsoft’s products stick to it. Kotlin handles the “Billion Dollar Mistake” way better, despite interoperability with the Java ecosystem.

I believe “nullable reference type” is a compile-time/static analysis feature. I don’t think we can leverage it to affect runtime behavior…

Would this help? “Expose top-level nullability information from reflection” (#29723) is slated for .NET 6 Preview 7.

So @ahsonkhan if I understand correctly you are suggesting a property attribute could be used to ensure NULL is ignored when de-serialising a non-nullable reference type. But don’t we want a way to trigger an exception to be equivalent to the non-nullable value type case?

And if we have an attribute or way to trigger an exception with NULL would that not also solve the compiler warning issue mentioned by @safern in that the compiler flow analysis can see that an incoming NULL will lead to an exception and hence the conversion to non-nullable is safe?

I think it would be really great to have a solution that is compatible with the compile-time checker if possible. 😀

NRTs convey intent as such it seems reasonable to me that automatic serializers generate schemas honoring that intent.

I don’t believe it’s any different from:

List<string> list =... ;
list.Add(null);

That will result in a compile-time warning if <Nullable>enabled</Nullable> is set, but it won’t impact the runtime behavior and result in Add throwing an exception, nor should it.

I think it is different because one of the most useful purposes of both reflection and source generators is to save the developer from writing a whole bunch of ‘handwritten’ code. It is from this perspective this feature request is thought about I believe.

Taking your example above, your point is (I think) that the compiler warning helps the author remember to insert a null check. In code generation, the ‘author’ is runtime code using reflection or a compile-time source generator. This automated author should have the benefit of the same information that a human author would. How else can it produce code of similar value to ‘handwritten’ code? The <nullable>enabled</nullable> is therefore analogous to a serializer flag.

Here’s a prototype showing how NRT metadata can be extracted in both reflection and source gen:

https://github.com/eiriktsarpalis/typeshape-csharp/commit/e96ef202263922c0d380554c02d929b16b2e999e

One thing to note is that this will most likely require an opt-in switch – it would otherwise be a breaking change:

namespace System.Text.Json;

public partial class JsonSerializerOptions
{
    public bool EnforceNonNullableReferenceTypes { get; set; }
}

public partial class JsonSourceGenerationOptionsAttribute
{
    public bool EnforceNonNullableReferenceTypes { get; set; }
}

Thanks for the detailed provided context @krwq ! 👍 Replies like these are so valuable, so us library users know what is going on behind the scenes.

This has been open for years, and many related requests are also closed as duplicates, pointing to this issue.

We are also facing this issue. What is the actual problem here to provide a solution for this?

If nullable annottations are turned on, fail/warn (potentially based on a JsonSerializer setting/annotation), when null is received as the value of the attribute in Json.

Currently it only fails when the attribute is not present at all (when I mark the property as required). Funnily enough, the deprecated IgnoreNullValues will work also for deserialization (as opposed to JsonIgnoreCondition.WhenWritingNull, which only handles the serialization direction).

Yes, the JSON gets deserialize but ASP.net knows that the value is null so it returns a validation error.

Sadly it does not 😦

It does, assuming you’re using controllers not minimal API.

Add this controller in an ASP.NET Core app

[ApiController]
[Route("/nullability")]
public class SomeController : ControllerBase
{
    [HttpPost]
    public Data Post([FromBody] Data body) => body;
}

public record Data(string NonNullableProp);

Posting { "nonNullableProp": null } to /nullability will get you a 400 response.

I found a temporary solution as follows. I don’t add the ThrowIfNull extension if it can be null.

public class User
{
   public User(string firstName, string? lastName)
   {
       FirstName = firstName.ThrowIfNull();
       LastName = lastName;
   }

   public string FirstName { get; set; }
   public string? LastName { get; set; }
}

public static class NullCheckParameter
{
    public static TKey ThrowIfNull<TKey>(this TKey value)
    {
        if (value == null)
            throw ArgumentNullException(typeof(TKey));

        return value;
    }
}

From @Eneuman in https://github.com/dotnet/runtime/issues/46431:

Using ReadFromJsonAsync in a nullable value type enabled .Net 5 ASP project has som strange behaviours.

For example: the following code return a MyClass? nullable object: var problemDetails = await response.Content.ReadFromJsonAsync<MyClass>();

This code returns a int (not nullable) (what happens if it is not a a numeric value in the response): var result = await response.Content.ReadFromJsonAsync<int>();

But this code returns a nullable string: var result = await response.Content.ReadFromJsonAsync<string>(); Shouldn’t it return a none nullable string and throw if it is null?