efcore: DbContext Does Not Support Thread-Safe Local Additions

EF Core supports asynchronous operations and therefore can and will be used in asynchronous applications. My current workflow of loading data from the database, using that data to asynchronously generate other data, and then writing that data back to the database does not seem to be exceptionally strange; I have to imagine other developers will run into some of the same problems.

Asynchronicity really has nothing to do thread-safety, at least not in this context: the fact that you’re executing operations asynchronously does not mean executing them in parallel, only that you’re releasing your thread while waiting waiting for I/O responses. Operations can (and typically should) run asynchronously but in sequence, unlike your original code sample with Task.WhenAll. Consider the following code based on your original sample:

// The following works in parallel and will therefore fail work because context isn't thread-safe
await Task.WhenAll(CreateAndAddAsync(context), CreateAndAddAsync(context), ...);

// The following is completely asynchronous, but not parallel: each action starts only once the previous one has completed
await CreateAndAddAsync(context);
await CreateAndAddAsync(context);

I think it’s very unlikely we’ll consider making DbContext thread-safe in any way, but if you can provide a very concrete (and hopefully concise) description of why you think you need it, we can discuss further and possibly help. I suspect there may be a confusion between asynchronicity and parallelism which could explain why you think you need thread-safety.

If the lack of thread safety is by design, perhaps this would be better marked as a feature request. I would like to express again that I understand there are underlying issues with reading and writing concurrently with a single DbContext that make it impossible to do so. Rather, my problem is with in-memory manipulations of the context, which do not suffer from the same limitations as reading/writing, not running in a thread-safe manner.

EF Core supports asynchronous operations and therefore can and will be used in asynchronous applications. My current workflow of loading data from the database, using that data to asynchronously generate other data, and then writing that data back to the database does not seem to be exceptionally strange; I have to imagine other developers will run into some of the same problems.

While there are ways around the issues (I am now generating the values asynchronously before bubbling them back up to be added synchronously), the library doesn’t have any control over whether those workarounds are applied. And while the problems I am seeing with the 2.2.0 build are bad–namely, a whole lot of run-time exceptions–the possibility of data loss in the 3.1.0 build is unacceptable for any enterprise grade software library.

It would be nice to have the option to run with a thread-safe DbContext. I am personally not concerned about the added cycles required to run using an ImmutableDictionary with ImmutableInterlocked or a ConcurrentDictionary; those costs are insignificant when compared to the cost of operating on an external database.

For anyone else reading this issue, the EF Core documentation on thread-safety can be found here. It was not on the first page of results when searching ‘EF Core thread safety’ using Bing or Google

@conor-joplin none of these components are thread-safe, and that’s by design - DbContext is simply not meant to be manipulated concurrently. In general, components aren’t made thread-safe unless they specifically are meant to work in a concurrent scenario; thread-safety adds overhead (e.g. locking) and guaranteeing that a component is thread-safe is non-trivial. For example, .NET Dictionary isn’t thread-safe (and the alternative ConcurrentDictionary exists if needed).

Since at the end of the day, DbContexts are used to perform database operations and these cannot be thread-safe, it makes little sense to make other functionality of DbContext thread-safe.

@conor-joplin Assuming that IJobRepository is registered as a scoped service, and DbContext is also registered as a scoped service (the default for AddDbContext), then this should work because every ProcessJobAsync is creating its own context instance. I would suggest running in the debugger and double-checking that a different DbContext instance is used each time.

@conor-joplin I understand and I’m also continuing the conversation in order to make sure I understand user needs, and in case there’s something I’ve missed.

However, what you’re in effect saying is that a component should always be thread-safe simply because its users may not know whether it is or isn’t. That basically amounts to saying that everything, everywhere, should always be thread-safe; this simply isn’t how things work. Developers are expected to not assume that components are thread-safe, and to consult the documentation - this includes many other components where the risk of data corruption is high. Accessing a non-thread-safe component concurrently because the documentation hasn’t been consulted is a programmer bug - that’s just how things work.

Once again, there is a reason for this - thread-safety has a cost, both in terms of performance (unnecessary locking/synchronization overhead), and in terms of developer effort in providing that guarantee (making components thread-safe can be very complicated, especially if good performance is to be maintained).

Finally, note that our docs do contain a clear and explicit section documenting the fact that DbContext isn’t thread-safe. It’s always possible to further improve documentation, but I think this point should already be quite clear to developers.

I suppose I could have been more clear in my usage of the word ‘asynchronous’; in every day conversation I often use it to express concurrency. A more precise wording would have been ‘to concurrently generate other data’, with some other similar replacements throughout my comments.

I’m curious as to why you would say this:

Operations can (and typically should) run asynchronously but in sequence

One of the benefits of running asynchronously is to be able to run multiple unrelated operations in parallel. Your modification of my pseudo-code, for example, takes twice as long to run. Add a third invocation, three times. A fourth, four times. The operations graph I want is this:

        create
read -> create -> write
        create

Not this:

read -> create -> create -> create -> write

I have already provided concrete code samples in this thread and in this repo that are both clear and concise, and I have made my case for why the current behavior is not acceptable for a Microsoft backed product, so I don’t think there’s anything else that I can say. I would be interested in hearing why you think that it is unreasonable for anyone using your library to run parallel operations on returned data within a database context–perhaps I am missing something.

I am also happy to meet to discuss this further if you feel that doing so could help resolve this issue more quickly. I see that you, @roji, are located in Germany, but perhaps there are other PMs and Devs with whom I can meet in Redmond.

the context itself should be thread safe when manipulated locally.

DbContext is not thread-safe. The part of warning “EF Core does not support multiple parallel operations being run on the same context instance.” is applicable to every operation and not just “Saving” of records.

Firstly, Sorry for replying to this and reopening it, but i just want to get something clear in my head and get advice on how i can resolve my issue.

I have a hosted service that inherits a bunch of services from DI and selects a list of jobs that are scheduled to be executed and then passes it to a paralell.foreach like this

Parallel.ForEach(pendingJobIds,
                            new ParallelOptions { MaxDegreeOfParallelism = 10 },
                            async job => { await ProcessJobAsync(job); });

ProcessJobAsync has code like this

public async Task ProcessJobAsync(Guid pendingJobId)
{
    try
    {
      using (var serviceScope = _serviceProvider.GetRequiredService<IServiceScopeFactory>().CreateScope())
        {
           var jobRepository = serviceScope.ServiceProvider.GetService<IJobRepository>();
					
	    //.. attempt job business logic and then update it's status
	    await jobRepository.UpdateJobStatus(pendingJobId,JobStatus.Successfull);
					
	 }
	}
	catch(Exception ex)
	{
		//blah
	}
}

IJobRepository inherits DBContext from DI, it is registered using the .AddDbContext() extension

In .NET Core 2.1, i experienced no issues with this code but concurrency and parallelism dev is new to me so i don’t know if this the right design.

Anyway, I’ve upgraded the application to .net core 3.1 and i now get this error message, intermittently.

A second operation started on this context before a previous operation completed. This is usually caused by different threads using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. Secondly, is this safe, i don’t think it is having read this post.

Finally, to fix this issue should i not be relying on DI in the ProcessJobAsync method, and instead be creating a new db context and then injecting it into the services i need, like this:

public async Task ProcessJobAsync(Guid pendingJobId)
{
   using (var dbContext = new DbContext(options))
    {
		var jobRepository = new JobRepository(dbContext);
		
		//.. attempt job business logic and then update it's status
		await jobRepository.UpdateJobStatus(pendingJobId,JobStatus.Successfull);
	}
}

Again i do apologise for re-opening this, i just need a little bit of clarity to progress.

I am stating that I would expect components with asynchronous support to be able to also support parallelism

This is simply not the case in general. For example, sockets have async support, and by their nature cannot support parallelism. In addition, the APIs on DbContext which you’re asking to make threadsafe - adding entities - aren’t async in any case.

Regardless, I do agree we could improve documentation. Making thread-safety a top-level page seems a bit much, but we could do better in making sure that new users stumble upon the information. Opened https://github.com/aspnet/EntityFramework.Docs/issues/1832 to track.