efcore: Result of left joining a nullable nominal type should be null (i.e. no instance) and not an instance with all null property values

We’ve just hit this after (attempting) to upgrade from 3.1 to 5.0. From what I can tell, this is a significant behavioural change.

Suggestion: Mention this is Breaking changes in EF Core 5.0?

Googling stack traces isn’t much fun 😃

This is definitely a breaking change

It’s too bad LINQ expressions don’t support the null conditional operator ?. Then we could indicate a possible null in the navigation to the EF query and quiet the VS Analyzers that complain about a possible null dereference (CS8602).

.Select(x => new {
    x.NavProperty1?.NavProperty2?.NonNullValueTypeProperty,
})

I can’t believe that after years, this error is still present. I have tons of LINQ queries that work correctly in .NET Framework 3.1, and I’m not sure how they might behave after an update. Has anyone found any workarounds? Is there at least one way to obtain a compile-time error instead of a runtime error?

This is very inconvenient, and has been ‘punted’ for 2 versions now…

@Mijalski This issue is in the 6.0 milestone, which means we plan to fix it for EF Core 6.0.

https://stackoverflow.com/a/65207398/1181624

You can also do a cast to a nullable type to make this issue go away. I posted that stack overflow answer before I found this post.

select new {
   ...
  optionalValue = (int?)leftJoinedType.someInt
}

@skclusive - Your query was giving incorrect results in 3.1, The bug fix for #20633 stopped generating incorrect results. In your case it throws exception because the query translation couldn’t figure out a way to generate right results. Your query working in LINQ is irrelevant as it worked in 3.1 also and EF Core generated results different than LINQ.

                var dbPosts = from p in db.Posts
                                  // select p;
                              select new Post
                              {
                                  PostId = p.PostId,
                                  BlogId = p.BlogId,
                                  Content = p.Content
                              };

                var query = from blog in db.Blogs
                            join post in dbPosts on blog.BlogId equals post.BlogId into posts
                            from xpost in posts.DefaultIfEmpty()
                            select new Blog
                            {
                                Url = blog.Url,
                                Post = xpost.BlogId == null ? null : xpost
                            };

Tested that above work-around gives correct result on 5.0 rc2 nightly build.

@hhhuut - There is no one answer to the question. Essentially, when you do a left join, right side can have null values so accessing any property over that will give you back null (EF Core will make property access null safe). But in C# there is no concept of left join or null-safe access. So in C# the property access will take CLR type of the property which can/cannot accommodate null value. e.g. entity.Property where entity is coming from right side of left join and Property is of type int will return int value. If entity turns out to be null then it throws null-ref exception. Since EF Core makes the access null safe it is same as writing entity == null ? (int?)null : entity.Property which will make the return value of type int? So when actual null value is encountered then it cannot be put into a int value throwing above exception. This only happens at runtime, when you actually encounter a null value.

So, as a user, you should carefully evaluate queries and see if you are expecting a null value appearing in left join and the property-accesses are able to accommodate that null value by being nullable type. In such cases you are required to put nullable type cast. It is not a “work-around”, it is required to materialize the results from the data you have. Though if your query will never get null from left join (be careful as data can change over the type in an app), or you are not accessing a property on an entity which can be nullable due to left join then you shouldn’t need it. The same exception can also arise when database contains null but client side property cannot take null values (even without left-joins) due to mismatch in model. There is also a possibility that there is a bug in EF Core somewhere. So if you have a query which has result types correct based on expected data and seeing this exception please file a new issue.

We just upgraded to .Net 6 and this is an embarrassing complete disaster. There’s no easy way to determine where this bug occurs other than scouring all 300+ LINQ expressions to see if we’re using a leftjoin, and selecting on the result. This isn’t even listed as a breaking change on the update page. The error message was completely useless and resulted in around 2 hours of debugging because I couldn’t figure out what was going wrong (why would I expect a simple SELECT to be the cause of the error?). Furthermore, this behaviour is completely inconsistent even within the LINQ query itself, and doesn’t follow expected C# paradigms. Consider the following query:

var userData = await (
    from u in db.user
    
    // left join
    join s in db.optional_user_settings_row
    on u.user_id equals s.user_id
    into leftjoin_u from u in leftjoin_u.DefaultIfEmpty()

    // s.suspended Doesn't error ???
    where u.user_id = UserIDToSearchFor && !s.suspended

    select new UserDataResult
    {
        name = u.display_name,
        // s.colourblind_enabled does error ???
        colourblind = s.colourblind_enabled,
    }
).ToArrayAsync();

This query errors because s is null, so it can’t cast the resulting bool? into the expected not-null bool. However, why doesn’t s.suspended in the where clause cause an error? Logically, the actual null value is s, but this seems to be behaving against expectations, as though we’ve just got a completely blank Object instead of a row, and everything inside of it is evaluating to null, but only sometimes. Not only that, but it’s not even evaluating to default as it should be based on the DefaultIfEmpty() name. This makes no sense at all, and honestly we’re seriously considering completely abandoning using Entity Framework LINQ because this isn’t the first time an update has broken queries across our application with no warning or logical way to fix it. I ended up coming to the same workaround conclusions of casting to (bool?) or using == true but again this is crazy behaviour to any user and causes confusion.

I know Microsoft doesn’t take suggestions, and this is a pipe dream, but the following syntax is what I would actually expect LINQ to work like.

var userData = await (
    from u in db.user
    
    // left join - new syntax
    // `s` is highlighted as POSSIBLY NULL
    // and will display a warning in Visual Studio if accessed directly below
    left join s in db.optional_user_settings_row
    on u.user_id equals s.user_id

    // Use null propagating operator since `s` is the actual null value
    // and we can EXPLICITLY HANDLE IT instead of automatically doing weird stuff
    where u.user_id = UserIDToSearchFor && (s?.suspended ?? false)

    select new UserDataResult
    {
        display_name = u.display_name,
        // Use coalescing to decide what the default should be
        colourblind_enabled = s?.colourblind_enabled ?? false,
    }
).ToArrayAsync();

Same problem in EF 7

@StrangeWill - This is what will happen in EF Core 6.0

• If the selector does conditional expression before construction of anonymous type, then we will not create instance of anonymous type
• If there is queryable of anonymous type which has DefaultIfEmpty applied over it then default value should be null rather than anonymous type with default property values (more complicated case of first one only)
• The selector you wrote above, if you have anonymous type construction without null check and if it ends up with null property value where expected is non-null value then it is going to throw the same error. I have explained in https://github.com/dotnet/efcore/issues/22517#issuecomment-758977488 things to consider when doing left join and why behavior is this way.
• None of above are going to be same behavior as EF Core 3.1 as EF Core 3.1 behavior was also wrong. It put wrong value rather than throwing exception when incorrect query results encountered.

Finally, this cannot be backported to 5.0 due to complexity of the fix.

Issues faced: When generating left join we need to add additional null check in selector so that we can generate correct default type if sequence is empty and not try to materialize result.

• This causes a complex selector which when used inside anything other than select, fails to translate since we don’t bind to column eventually. (it is ternary with client side types). On flip side non-select shouldn’t need check either since column would be null anyway.
• We cannot do when translating LeftJoin easily because projection is lifted after LeftJoin so we don’t have enough data in translation pipeline in that sense.
• For unexpanded navigations we try to add check which is not necessary. Requires #20291

Due to unintended side effect of breaking working queries due to having a complex expression for entity now, we should postpone this.

Confirmed this works in 3.1, but fails in latest daily, on SQL Server and SQLite.