ef6: Long running Code First Migration throws RemotingException: "Object has been disconnected from the server"

A code first migration of about 20 minutes causes the following exception when the Update-Database command is executed:

It appears to be something related to a timeout of the migration

In my case the migration was adding a column to a table with about 11 million records, the operation takes time but it does work when the command SQL is directly executed on the database manager

There are other people having the same issue here and here

System.Runtime.Remoting.RemotingException: Object '/7ba26d1b_81f4_4f58_8bf3_92a9b8a3a3fa/zq28jcy6udmgdh9278mbbwoy_163.rem' has been disconnected or does not exist at the server.
   at System.Data.Entity.Migrations.Design.ToolingFacade.ToolLogger.Verbose(String sql)
   at System.Data.Entity.Migrations.Infrastructure.MigratorLoggingDecorator.ExecuteSql(MigrationStatement migrationStatement, DbConnection connection, DbTransaction transaction, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsInternal(IEnumerable`1 migrationStatements, DbConnection connection, DbTransaction transaction, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsWithinTransaction(IEnumerable`1 migrationStatements, DbTransaction transaction, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsWithinNewTransaction(IEnumerable`1 migrationStatements, DbConnection connection, DbInterceptionContext interceptionContext)
  at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsInternal(IEnumerable`1 migrationStatements, DbConnection connection, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsInternal(IEnumerable`1 migrationStatements, DbConnection connection)
   at System.Data.Entity.Migrations.DbMigrator.<>c__DisplayClass30.<ExecuteStatements>b__2e()
   at System.Data.Entity.Infrastructure.DefaultExecutionStrategy.Execute(Action operation)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatements(IEnumerable`1 migrationStatements, DbTransaction existingTransaction)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatements(IEnumerable`1 migrationStatements)
   at System.Data.Entity.Migrations.Infrastructure.MigratorBase.ExecuteStatements(IEnumerable`1 migrationStatements)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteOperations(String migrationId, VersionedModel targetModel, IEnumerable`1 operations, IEnumerable`1 systemOperations, Boolean downgrading, Boolean auto)
   at System.Data.Entity.Migrations.DbMigrator.ApplyMigration(DbMigration migration, DbMigration lastMigration)
   at System.Data.Entity.Migrations.Infrastructure.MigratorLoggingDecorator.ApplyMigration(DbMigration migration, DbMigration lastMigration)
   at System.Data.Entity.Migrations.DbMigrator.Upgrade(IEnumerable`1 pendingMigrations, String targetMigrationId, String lastMigrationId)
   at System.Data.Entity.Migrations.Infrastructure.MigratorLoggingDecorator.Upgrade(IEnumerable`1 pendingMigrations, String targetMigrationId, String lastMigrationId)
   at System.Data.Entity.Migrations.DbMigrator.UpdateInternal(String targetMigration)
   at System.Data.Entity.Migrations.DbMigrator.<>c__DisplayClassc.<Update>b__b()
   at System.Data.Entity.Migrations.DbMigrator.EnsureDatabaseExists(Action mustSucceedToKeepDatabase)
   at System.Data.Entity.Migrations.Infrastructure.MigratorBase.EnsureDatabaseExists(Action mustSucceedToKeepDatabase)
   at System.Data.Entity.Migrations.DbMigrator.Update(String targetMigration)
   at System.Data.Entity.Migrations.Infrastructure.MigratorBase.Update(String targetMigration)
   at System.Data.Entity.Migrations.Design.ToolingFacade.UpdateRunner.Run()
   at System.AppDomain.DoCallBack(CrossAppDomainDelegate callBackDelegate)
   at System.AppDomain.DoCallBack(CrossAppDomainDelegate callBackDelegate)
   at System.Data.Entity.Migrations.Design.ToolingFacade.Run(BaseRunner runner)
   at System.Data.Entity.Migrations.Design.ToolingFacade.Update(String targetMigration, Boolean force)
   at System.Data.Entity.Migrations.UpdateDatabaseCommand.<>c__DisplayClass2.<.ctor>b__0()
   at System.Data.Entity.Migrations.MigrationsDomainCommand.Execute(Action command)
Object '/7ba26d1b_81f4_4f58_8bf3_92a9b8a3a3fa/zq28jcy6udmgdh9278mbbwoy_163.rem' has been disconnected or does not exist at the server.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 1
  • Comments: 40 (10 by maintainers)

Commits related to this issue

Most upvoted comments

The issue is that the ToolLogger lease lifetime (base class MigrationsLogger is a MarshalByRefObject) is at the default (5 minutes). The ToolingFacade creates the logger, which lives in the main program’s app domain. The migrations run in a different app domain. If a migration takes longer than 5 minutes, the attempt to log any further information results in this error. A solution would be to increase the lease lifetime in the main program. So… in the main program, prior to creating the ToolingFacade, set the lease lifetime to a longer time period:

using System.Runtime.Remoting.Lifetime;LifetimeServices.LeaseTime = TimeSpan.FromHours(1);

Migrations in this case can then take as long as 1 hour before this issue presents itself.

Spent half a day trying different fixes for this. I tried adding the config below to config file of ef6.exe.config, app.config of --config arg, and to the machine.config - it simply doesn’t read these settings.

 <system.runtime.remoting>
    <application>
      <lifetime leaseTime = "10M" sponsorshipTimeOut = "10M" renewOnCallTime = "15M" LeaseManagePollTime = "8s" />
    </application>
 </system.runtime.remoting>

The only solution is to download the source code of ef6 (https://github.com/dotnet/ef6.git), and mod program.cs;

image

We have also used this workaround in the past for several years by increasing the LeaseTime in our Configuration.cs file. But as @edswangren already mentioned, it no longer works.

However, i have other information for you: We have been using version 6.2 since its release. The workaround described (LeaseTime > 5min) has definitely worked with it in the past. We don’t have many long running migrations, but we have one this week and it suddenly stops working. So I do not think that the reason for this is an update to 6.3., because we still use 6.2 and it worked before! The reason must be something else.

Today I spent all day to get our migration running somehow.

My migration file to reproduce the issue:

public partial class Longrunningmigration : DbMigration
{
    public override void Up()
    {
        LifetimeServices.LeaseTime = TimeSpan.FromHours(1);
        System.Threading.Thread.Sleep(TimeSpan.FromMinutes(6));
    }
    
    public override void Down()
    {
    }
}

I have tried all versions from 6.1.3 to 6.4

  • EF 6.1.3
  • EF 6.2 Error Message from migrate.exe:
PM> Update-Database -ProjectName "XXX.Common" -StartUpProjectName "XXX.Common" -ConnectionStringName "XXXDbContext" -Verbose
Using StartUp project 'XXX.Common'.
Specify the '-Verbose' flag to view the SQL statements being applied to the target database.
Target database is: 'XXX' (DataSource: localhost,2433, Provider: System.Data.SqlClient, Origin: Explicit).
Applying explicit migrations: [202001161545394_Long running migration].
Applying explicit migration: 202001161545394_Long running migration.
after 5 minutes of waiting ....
System.Runtime.Remoting.RemotingException: Object '/9b12b632_a813_4f1f_ad7d_8965295def39/zfjvhghlcvw931bbghtfgxls_4774.rem' has been disconnected or does not exist at the server.
   at System.Data.Entity.Migrations.Design.ToolingFacade.ToolLogger.Verbose(String sql)
   at System.Data.Entity.Migrations.Infrastructure.MigratorLoggingDecorator.ExecuteSql(MigrationStatement migrationStatement, DbConnection connection, DbTransaction transaction, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsInternal(IEnumerable`1 migrationStatements, DbConnection connection, DbTransaction transaction, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsWithinTransaction(IEnumerable`1 migrationStatements, DbTransaction transaction, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsWithinNewTransaction(IEnumerable`1 migrationStatements, DbConnection connection, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsInternal(IEnumerable`1 migrationStatements, DbConnection connection, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatementsInternal(IEnumerable`1 migrationStatements, DbConnection connection)
   at System.Data.Entity.Migrations.DbMigrator.<>c__DisplayClass30.<ExecuteStatements>b__2e()
   at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.<>c__DisplayClass1.<Execute>b__0()
   at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation)
   at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute(Action operation)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatements(IEnumerable`1 migrationStatements, DbTransaction existingTransaction)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteStatements(IEnumerable`1 migrationStatements)
   at System.Data.Entity.Migrations.Infrastructure.MigratorBase.ExecuteStatements(IEnumerable`1 migrationStatements)
   at System.Data.Entity.Migrations.DbMigrator.ExecuteOperations(String migrationId, VersionedModel targetModel, IEnumerable`1 operations, IEnumerable`1 systemOperations, Boolean downgrading, Boolean auto)
   at System.Data.Entity.Migrations.DbMigrator.ApplyMigration(DbMigration migration, DbMigration lastMigration)
   at System.Data.Entity.Migrations.Infrastructure.MigratorLoggingDecorator.ApplyMigration(DbMigration migration, DbMigration lastMigration)
   at System.Data.Entity.Migrations.DbMigrator.Upgrade(IEnumerable`1 pendingMigrations, String targetMigrationId, String lastMigrationId)
   at System.Data.Entity.Migrations.Infrastructure.MigratorLoggingDecorator.Upgrade(IEnumerable`1 pendingMigrations, String targetMigrationId, String lastMigrationId)
   at System.Data.Entity.Migrations.DbMigrator.UpdateInternal(String targetMigration)
   at System.Data.Entity.Migrations.DbMigrator.<>c__DisplayClassc.<Update>b__b()
   at System.Data.Entity.Migrations.DbMigrator.EnsureDatabaseExists(Action mustSucceedToKeepDatabase)
   at System.Data.Entity.Migrations.Infrastructure.MigratorBase.EnsureDatabaseExists(Action mustSucceedToKeepDatabase)
   at System.Data.Entity.Migrations.DbMigrator.Update(String targetMigration)
   at System.Data.Entity.Migrations.Infrastructure.MigratorBase.Update(String targetMigration)
   at System.Data.Entity.Migrations.Design.ToolingFacade.UpdateRunner.Run()
   at System.AppDomain.DoCallBack(CrossAppDomainDelegate callBackDelegate)
   at System.AppDomain.DoCallBack(CrossAppDomainDelegate callBackDelegate)
   at System.Data.Entity.Migrations.Design.ToolingFacade.Run(BaseRunner runner)
   at System.Data.Entity.Migrations.Design.ToolingFacade.Update(String targetMigration, Boolean force)
   at System.Data.Entity.Migrations.UpdateDatabaseCommand.<>c__DisplayClass2.<.ctor>b__0()
   at System.Data.Entity.Migrations.MigrationsDomainCommand.Execute(Action command)
Object '/9b12b632_a813_4f1f_ad7d_8965295def39/zfjvhghlcvw931bbghtfgxls_4774.rem' has been disconnected or does not exist at the server.
  • EF 6.3
  • EF 6.4 Error Message from ef6.exe:
C:\Users\XXX\source\repos\XXX\packages\EntityFramework.6.4.0\tools\net45\any>ef6.exe database update --connection-string-name XXXDbContext --verbose --no-color --prefix-output --assembly C:\Users\XXX\source\repos\XXX\Common\src\XXX.Common\bin\Release\XXX.Common.dll --project-dir C:\Users\XXX\source\repos\XXX\Common\src\XXX.Common\ --language C# --root-namespace XXX.Common --config C:\Users\XXX\source\repos\XXX\Common\src\XXX.Common\App.config
info:    Specify the '-Verbose' flag to view the SQL statements being applied to the target database.
verbose: Target database is: 'XXX' (DataSource: XXX, Provider: System.Data.SqlClient, Origin: Explicit).
info:    Applying explicit migrations: [202001161545394_Long running migration].
info:    Applying explicit migration: 202001161545394_Long running migration.
after 5 minutes of waiting ....
info:    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Runtime.Remoting.RemotingException: Cannot load type 'System.Data.Entity.Infrastructure.Design.IResultHandler2, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
info:
info:    Server stack trace:
info:       at System.Runtime.Remoting.Messaging.MethodCall.ResolveMethod(Boolean bThrowIfNotResolved)
info:       at System.Runtime.Remoting.Messaging.MethodCall..ctor(SmuggledMethodCallMessage smuggledMsg, ArrayList deserializedArgs)
info:       at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(Byte[] reqStmBuff, SmuggledMethodCallMessage smuggledMcm, SmuggledMethodReturnMessage& smuggledMrm)
info:       at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCallback(Object[] args)
info:
info:    Exception rethrown at [0]:
info:       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
info:       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
info:       at System.Data.Entity.Infrastructure.Design.IResultHandler2.SetError(String type, String message, String stackTrace)
info:       at System.Data.Entity.Infrastructure.Design.WrappedResultHandler.SetError(String type, String message, String stackTrace)
info:       at System.Data.Entity.Infrastructure.Design.Executor.OperationBase.Execute(Action action)
info:       --- End of inner exception stack trace ---
info:       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
info:       at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
info:       at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
info:       at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
info:       at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
info:       at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
info:       at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
info:       at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
info:       at System.Data.Entity.Tools.AppDomainExecutor.Execute(String operation, Object resultHandler, IDictionary args)
info:       at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid4[T0,T1,T2,T3](CallSite site, T0 arg0, T1 arg1, T2 arg2, T3 arg3)
info:       at System.Data.Entity.Tools.ExecutorBase.InvokeImpl(String operation, IDictionary args)
info:       at System.Data.Entity.Tools.ExecutorBase.Update(String targetMigration, Boolean force, String connectionStringName, String connectionString, String connectionProviderName, String migrationsConfigurationName)
info:       at System.Data.Entity.Tools.Commands.DatabaseUpdateCommand.Execute()
info:       at Microsoft.DotNet.Cli.CommandLine.CommandLineApplication.Execute(String[] args)
info:       at System.Data.Entity.Tools.Program.Main(String[] args)
error:   Exception has been thrown by the target of an invocation.

Unfortunately I have no solution either.

@bricelam So, is it true to say from our discussion last week that the comment here is likely a bogus solution to this issue?