orleans: Orleans on Kubernetes, pod keeps crashing/restarting
Situation
I have a kubernetes cluster with 3 Pods
- Database (PostgreSQL)
- Silo Service 1 (AdoNET Clustering using PostgreSQL)
- Silo Service 2 (AdoNET Clustering using PostgreSQL
They work and communicate as expected when running for the first time with clean database, 2 rows for each Silo are created for the membership table.
I have a volume for my database Pod, so the data gets persisted.
If i then stop/delete my kube deployment/services/pods, change some configs and start again (normal development / experimentation flow), extra 2 rows for each Silo are created in the membership table, total of 4 rows, Orleans will perform around 5 or 6 connection re-tries to the older/dead silo row IPs and will timeout, services will run fine.
Now i stop/delete and start my deployment/services/pods again, now we have 6 rows, Orleans will not stop re-trying connection with the previous 4 dead/old silos until the pod crashes and restarts and this will keep going forever.
In summary, the steps to reproduce this problem are:
1. kubectl apply -f … -f … : start my deployments/pods/services with clean database 2. navigate, test stuff, working great 3. change config 4. kubectl delete services,pods,etc 5. kubectl apply -f…-f… start again 6. with 2 silos, now i have 4 rows in my membership table, my 2 silos will fail to connect to the older 2 rows for some time and run fine. 7. stop and start again, now i have 2 silos and 6 membership rows, my 2 silos will fail to connect to the older 4 rows until my pods crash and restart forever.
My pods after 2 crashes/restarts
NAME READY STATUS RESTARTS AGE
eventstoredb-75bfdf767-dt648 1/1 Running 0 4m2s
fhirserver-pod-6558c57dc4-pl8gr 1/1 Running 2 4m2s
patientservice-pod-788877d9f8-prs98 1/1 Running 2 4m2s
postgres-fhirdatabase-7dd856f8c4-gdg7m 1/1 Running 0 4m2s
ui-pod-85f768b7c6-cwq7l 1/1 Running 0 4m2s
fhirserver-pod Logs before crashing (both pod logs look the same before they crash/restart)
[24-Jul-2020 18:35:04 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Initializing PostgreOrleansStorage()
[24-Jul-2020 18:35:04 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - @ EnsureCreated()
ConnectionString: Server=fhirdatabase-service;Port=5108;Database=smartfhirserver.db;User ID=admin;Password=admin
[24-Jul-2020 18:35:06 FTL] Database error
Npgsql.NpgsqlException (0x80004005): Exception while connecting
---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (111): Connection refused 10.98.23.247:5108
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.<>c.<ConnectAsync>b__274_0(IAsyncResult iar)
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.TaskExtensions.WithCancellation(Task task, CancellationToken cancellationToken)
at Npgsql.TaskExtensions.WithTimeout(Task task, NpgsqlTimeout timeout)
at Npgsql.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 419
at SMARTPlatform.Messaging.Storage.PostgreOrleansStorage.DatabaseExists(String dbName, String connectionString)
at SMARTPlatform.Messaging.Storage.PostgreOrleansStorage.CreateDatabase(String databaseName, String connectionString)
at SMARTPlatform.Messaging.Storage.PostgreOrleansStorage.EnsureCreated(String dbName, String connectionString)
at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
[24-Jul-2020 18:35:06 INF] [1] Trying to make background database connections
[24-Jul-2020 18:35:16 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - @ EnsureCreated()
ConnectionString: Server=fhirdatabase-service;Port=5108;Database=smartfhirserver.db;User ID=admin;Password=admin
[24-Jul-2020 18:35:16 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Database smartfhirserverclustering.db already exists!
[24-Jul-2020 18:35:16 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Updated connection string!
[24-Jul-2020 18:35:17 INF] Entity Framework Core 3.1.6 initialized 'WriteApplicationDbContext' using provider 'Npgsql.EntityFrameworkCore.PostgreSQL' with options: MigrationsAssembly=Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
[24-Jul-2020 18:35:17 INF] Executed DbCommand (3ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT CASE WHEN COUNT(*) = 0 THEN FALSE ELSE TRUE END
FROM information_schema.tables
WHERE table_type = 'BASE TABLE' AND table_schema NOT IN ('pg_catalog', 'information_schema')
[24-Jul-2020 18:35:17 WRN] Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
[24-Jul-2020 18:35:17 INF] User profile is available. Using '/root/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
[24-Jul-2020 18:35:17 INF] Silo starting with GC settings: ServerGC=False GCLatencyMode=Interactive
[24-Jul-2020 18:35:17 WRN] Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
[24-Jul-2020 18:35:17 WRN] Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
[24-Jul-2020 18:35:17 INF] -------------- Initializing silo on host fhirserver-pod-6558c57dc4-pl8gr MachineName fhirserver-pod-6558c57dc4-pl8gr at 10.1.2.167:11111, gen 333311717 --------------
[24-Jul-2020 18:35:17 INF] Starting silo Silo_820bf
[24-Jul-2020 18:35:18 WRN] No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
[24-Jul-2020 18:35:18 INF] Starting VirtualBucketsRingProvider on silo S10.1.2.167:11111:333311717/xA66E0673.
[24-Jul-2020 18:35:18 INF] Added Server S10.1.2.167:11111:333311717/xA66E0673. Current view: [S10.1.2.167:11111:333311717 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
[24-Jul-2020 18:35:18 INF] -NotifyLocalRangeSubscribers about old <(0 0], Size=x100000000, %Ring=100.000%> new <MultiRange: Size=x100000000, %Ring=100.000%> increased? True
[24-Jul-2020 18:35:18 INF] -------------- Started silo S10.1.2.167:11111:333311717, ConsistentHashCode A66E0673 --------------
[24-Jul-2020 18:35:18 INF] Creating key {d8d191e5-450d-4b22-9ee5-9fb14ef90173} with creation date 2020-07-24 18:35:18Z, activation date 2020-07-24 18:35:18Z, and expiration date 2020-10-22 18:35:18Z.
[24-Jul-2020 18:35:18 WRN] No XML encryptor configured. Key {d8d191e5-450d-4b22-9ee5-9fb14ef90173} may be persisted to storage in unencrypted form.
[24-Jul-2020 18:35:18 INF] Writing data to file '/root/.aspnet/DataProtection-Keys/key-d8d191e5-450d-4b22-9ee5-9fb14ef90173.xml'.
[24-Jul-2020 18:35:18 INF] Now listening on: http://[::]:80
[24-Jul-2020 18:35:18 INF] Starting Orleans Silo.
[24-Jul-2020 18:35:18 INF] Stage First (-2147483648): Orleans.Runtime.SiloOptionsLogger
[24-Jul-2020 18:35:18 INF] Stage 1999: ConnectionManager, SiloConnectionListener, GatewayConnectionListener
[24-Jul-2020 18:35:18 INF] Stage RuntimeInitialize (2000): Orleans.Runtime.InsideRuntimeClient, SiloStatusListenerManager, ClusterMembershipService, SiloConnectionMaintainer, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage 2001: MembershipAgent
[24-Jul-2020 18:35:18 INF] Stage RuntimeServices (4000): Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage RuntimeGrainServices (8000): MembershipTableManager, HostedClient, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage AfterRuntimeGrainServices (8100): MembershipAgent, SiloStatusListenerManager
[24-Jul-2020 18:35:18 INF] Stage ApplicationServices (10000): Orleans.Runtime.Versions.GrainVersionStore
[24-Jul-2020 18:35:18 INF] Stage BecomeActive (19999): MembershipAgent, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage Active (20000): ClusterHealthMonitor, MembershipTableCleanupAgent, GatewayConnectionListener, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ActivationCountBasedPlacementOptions:
ChooseOutOf: 2
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ClusterMembershipOptions:
NumMissedTableIAmAliveLimit: 2
LivenessEnabled: True
ProbeTimeout: 00:00:05
TableRefreshTimeout: 00:01:00
DeathVoteExpirationTimeout: 00:02:00
IAmAliveTablePublishTimeout: 00:05:00
MaxJoinAttemptTime: 00:05:00
ValidateInitialConnectivity: True
UseLivenessGossip: True
NumProbedSilos: 3
NumMissedProbesLimit: 3
NumVotesForDeathDeclaration: 2
DefunctSiloExpiration: 7.00:00:00
DefunctSiloCleanupPeriod: 01:00:00
IsRunningAsUnitTest: False
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ClusterOptions:
ClusterId: smartplatform_silo_cluster
ServiceId: smartplatform_silo_service
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ConnectionOptions:
ProtocolVersion: Version1
ConnectionsPerEndpoint: 1
ConnectionRetryDelay: 00:00:01
OpenConnectionTimeout: 00:00:05
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ConsistentRingOptions:
NumVirtualBucketsConsistentRing: 30
UseVirtualBucketsConsistentRing: True
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.EndpointOptions:
AdvertisedIPAddress: 10.1.2.167
SiloPort: 11111
GatewayPort: 30000
SiloListeningEndpoint: 0.0.0.0:11111
GatewayListeningEndpoint: 0.0.0.0:30000
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.GrainCollectionOptions:
CollectionQuantum: 00:01:00
CollectionAge: 02:00:00
DeactivationTimeout: 00:00:30
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.GrainDirectoryOptions:
CachingStrategy: Adaptive
CacheSize: 1000000
InitialCacheTTL: 00:00:30
MaximumCacheTTL: 00:04:00
CacheTTLExtensionFactor: 2
LazyDeregistrationDelay: 00:01:00
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.GrainVersioningOptions:
DefaultCompatibilityStrategy: BackwardCompatible
DefaultVersionSelectorStrategy: AllCompatibleVersions
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.LoadSheddingOptions:
LoadSheddingEnabled: False
LoadSheddingLimit: 95
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.PerformanceTuningOptions:
DefaultConnectionLimit: 200
Expect100Continue: False
UseNagleAlgorithm: False
MinDotNetThreadPoolSize: 0
MinIOThreadPoolSize: 0
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SchedulingOptions:
PerformDeadlockDetection: False
AllowCallChainReentrancy: True
MaxActiveThreads: 4
DelayWarningThreshold: 00:00:10
ActivationSchedulingQuantum: 00:00:00.1000000
TurnWarningLengthThreshold: 00:00:00.2000000
MaxPendingWorkItemsSoftLimit: 0
EnableWorkerThreadInjection: False
StoppedActivationWarningInterval: 00:01:00
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SerializationProviderOptions:
FallbackSerializationProvider:
MaxSustainedSerializationContextCapacity: 64
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SiloMessagingOptions:
SiloSenderQueues: 1
GatewaySenderQueues: 1
MaxForwardCount: 2
ClientDropTimeout: 00:01:00
ClientRegistrationRefresh: 00:05:00
MaxEnqueuedRequestsSoftLimit: 0
MaxEnqueuedRequestsHardLimit: 0
MaxEnqueuedRequestsSoftLimit_StatelessWorker: 0
MaxEnqueuedRequestsHardLimit_StatelessWorker: 0
MaxRequestProcessingTime: 02:00:00
AssumeHomogenousSilosForTesting: False
DEFAULT_SHUTDOWN_REROUTE_TIMEOUT: 00:00:10
ShutdownRerouteTimeout: 00:00:10
SystemResponseTimeout: 00:00:30
ResponseTimeout: 00:00:30
ResponseTimeoutWithDebugger: 00:30:00
DropExpiredMessages: True
BufferPoolBufferSize: 4096
BufferPoolMaxSize: 10000
BufferPoolPreallocationSize: 250
PropagateActivityId: False
LargeMessageWarningThreshold: 85000
MaxMessageHeaderSize: 10485760
MaxMessageBodySize: 104857600
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SiloOptions:
SiloName: Silo_820bf
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.StatisticsOptions:
PerfCountersWriteInterval: 00:00:30
LogWriteInterval: 00:05:00
CollectionLevel: Info
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.TelemetryOptions:
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.TypeManagementOptions:
TypeMapRefreshInterval: 00:01:00
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.SiloOptionsLogger started in stage First (-2147483648) in 7.9077 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage First (-2147483648) took 8.8816 Milliseconds
[24-Jul-2020 18:35:18 INF] ConnectionManager started in stage 1999 in 0.0353 Milliseconds
[24-Jul-2020 18:35:18 INF] GatewayConnectionListener started in stage 1999 in 2.9368 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloConnectionListener started in stage 1999 in 4.5163 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage 1999 took 4.89 Milliseconds
[24-Jul-2020 18:35:18 INF] Loaded grain type summary for 7 types:
Grain class Infrastructure.Grains.Infrastructure.Grains.ResourceGrain [1059605650 (0x3F284C92)] from Infrastructure.dll implementing interfaces: SMARTPlatform.Messaging.GrainInterfaces.Fhir.IResourceGrain [-1455888765 (0xA938E683)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.LogConsistency.Orleans.LogConsistency.LogConsistentGrain<TView> [-319513858 (0xECF49AFE)] from Orleans.Core.dll implementing interfaces:
Grain class Orleans.Runtime.Development.Orleans.Runtime.Development.DevelopmentLeaseProviderGrain [940524075 (0x380F422B)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Development.IDevelopmentLeaseProviderGrain [-1234611069 (0xB6695483)]
Grain class Orleans.Runtime.Management.Orleans.Runtime.Management.ManagementGrain [1954798034 (0x7483D9D2)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.IManagementGrain [-1734666656 (0x989B1660)]
Grain class Orleans.Runtime.ReminderService.Orleans.Runtime.ReminderService.GrainBasedReminderTable [-55315191 (0xFCB3F509)] from Orleans.Runtime.dll implementing interfaces: Orleans.IReminderTableGrain [-1135060418 (0xBC585A3E)]
Grain class Orleans.Runtime.Versions.Orleans.Runtime.Versions.VersionStoreGrain [331003096 (0x13BAB4D8)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Versions.IVersionStoreGrain [-297600501 (0xEE42FA0B)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.Streams.Orleans.Streams.PubSubRendezvousGrain [1903070868 (0x716E8E94)] from Orleans.Runtime.dll implementing interfaces: Orleans.Streams.IPubSubRendezvousGrain [1746702088 (0x681C8F08)]
[24-Jul-2020 18:35:18 INF] Start InsideRuntimeClient took 5 Milliseconds
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.InsideRuntimeClient started in stage RuntimeInitialize (2000) in 6.3063 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloStatusListenerManager started in stage RuntimeInitialize (2000) in 0.0287 Milliseconds
[24-Jul-2020 18:35:18 INF] ClusterMembershipService started in stage RuntimeInitialize (2000) in 0.0944 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloConnectionMaintainer started in stage RuntimeInitialize (2000) in 63.9175 Milliseconds
[24-Jul-2020 18:35:18 INF] Silo Start()
[24-Jul-2020 18:35:18 INF] Configured .NET ServicePointManager to Expect100Continue=False, DefaultConnectionLimit=200, UseNagleAlgorithm=False to improve Azure storage performance.
[24-Jul-2020 18:35:18 INF] ConfigureThreadPoolAndServicePointSettings took 0 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.Silo started in stage RuntimeInitialize (2000) in 1.4288 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage RuntimeInitialize (2000) took 72.0834 Milliseconds
[24-Jul-2020 18:35:18 INF] MembershipAgent started in stage 2001 in 0.0314 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage 2001 took 0.0804 Milliseconds
[24-Jul-2020 18:35:18 INF] Start Message center took 0 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Start
[24-Jul-2020 18:35:18 INF] Starting AsyncAgent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer on managed thread 9
[24-Jul-2020 18:35:18 INF] Start local grain directory took 1 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Init implicit stream subscribe table took 2 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Create system targets and inject dependencies took 12 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.Silo started in stage RuntimeServices (4000) in 18.6763 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage RuntimeServices (4000) took 18.7516 Milliseconds
[24-Jul-2020 18:35:18 INF] HostedClient started in stage RuntimeGrainServices (8000) in 1.9141 Milliseconds
[24-Jul-2020 18:35:18 INF] MembershipOracle starting on host fhirserver-pod-6558c57dc4-pl8gr with SiloAddress S10.1.2.167:11111:333311717 at 2020-07-24 18:35:18.064 GMT
[24-Jul-2020 18:35:18 INF] Init grain services took 75 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Init type manager took 13 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Starting Windows perf counter stats collection with frequency=00:00:30
[24-Jul-2020 18:35:18 INF] Start silo statistics took 85 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Starting DeploymentLoadPublisher.
[24-Jul-2020 18:35:18 INF] Started DeploymentLoadPublisher.
[24-Jul-2020 18:35:18 INF] Start deployment load collector took 7 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Starting Silo Watchdog.
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.Silo started in stage RuntimeGrainServices (8000) in 184.4005 Milliseconds
[24-Jul-2020 18:35:18 INF] ProcessTableUpdate (called from Refresh) membership table: 5 silos, 1 are Active, 3 are Dead, 1 are Joining, Version=<123, 123>. All silos: [SiloAddress=S10.1.2.168:11112:333311706 SiloName=Silo_095e7 Status=Joining, SiloAddress=S10.1.2.162:11111:333311335 SiloName=Silo_e5ba2 Status=Active, SiloAddress=S10.1.2.157:11111:333300013 SiloName=Silo_44d47 Status=Dead, SiloAddress=S10.1.2.158:11112:333300010 SiloName=Silo_c316b Status=Dead, SiloAddress=S10.1.2.163:11112:333311335 SiloName=Silo_c383c Status=Dead]
[24-Jul-2020 18:35:18 INF] MembershipTableManager started in stage RuntimeGrainServices (8000) in 295.7757 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage RuntimeGrainServices (8000) took 295.9127 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloStatusListenerManager started in stage AfterRuntimeGrainServices (8100) in 0.0833 Milliseconds
[24-Jul-2020 18:35:18 INF] -Joining
[24-Jul-2020 18:35:18 INF] Added Server S10.1.2.162:11111:333311335/xF49D260A. Current view: [S10.1.2.167:11111:333311717 -> <MultiRange: Size=x65CCFB18, %Ring=39.766%>, S10.1.2.162:11111:333311335 -> <MultiRange: Size=x9A3304E8, %Ring=60.234%>]
[24-Jul-2020 18:35:18 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x100000000, %Ring=100.000%> new <MultiRange: Size=x65CCFB18, %Ring=39.766%> increased? True
[24-Jul-2020 18:35:18 INF] Expediting cluster type map refresh due to new silo, S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:19 INF] ProcessTableUpdate (called from TryUpdateMyStatusGlobalOnce) membership table: 6 silos, 1 are Active, 3 are Dead, 2 are Joining, Version=<124, 123>. All silos: [SiloAddress=S10.1.2.168:11112:333311706 SiloName=Silo_095e7 Status=Joining, SiloAddress=S10.1.2.167:11111:333311717 SiloName=Silo_820bf Status=Joining, SiloAddress=S10.1.2.162:11111:333311335 SiloName=Silo_e5ba2 Status=Active, SiloAddress=S10.1.2.157:11111:333300013 SiloName=Silo_44d47 Status=Dead, SiloAddress=S10.1.2.158:11112:333300010 SiloName=Silo_c316b Status=Dead, SiloAddress=S10.1.2.163:11112:333311335 SiloName=Silo_c383c Status=Dead]
[24-Jul-2020 18:35:19 INF] Gossiping S10.1.2.167:11111:333311717 status change to Joining to 1 partners
[24-Jul-2020 18:35:19 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:22 INF] MembershipAgent started in stage AfterRuntimeGrainServices (8100) in 3083.7579 Milliseconds
[24-Jul-2020 18:35:22 INF] Starting lifecycle stage AfterRuntimeGrainServices (8100) took 3084.4064 Milliseconds
[24-Jul-2020 18:35:22 INF] Orleans.Runtime.Versions.GrainVersionStore started in stage ApplicationServices (10000) in 0.2429 Milliseconds
[24-Jul-2020 18:35:22 INF] Starting lifecycle stage ApplicationServices (10000) took 0.3525 Milliseconds
[24-Jul-2020 18:35:22 INF] -BecomeActive
[24-Jul-2020 18:35:22 INF] Starting AsyncAgent Runtime.Messaging.GatewayClientCleanupAgent on managed thread 9
[24-Jul-2020 18:35:22 INF] Start gateway took 0 Milliseconds to finish
[24-Jul-2020 18:35:22 INF] Orleans.Runtime.Silo started in stage BecomeActive (19999) in 0.8027 Milliseconds
[24-Jul-2020 18:35:22 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:22 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:22 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:22 ERR] Exception while sending gossip notification to remote silo S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.MembershipService.MembershipSystemTarget.GossipToRemoteSilo(SiloAddress silo, MembershipTableSnapshot snapshot, SiloAddress updatedSilo, SiloStatus updatedStatus)
[24-Jul-2020 18:35:22 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:22 WRN] ProcessSiloAddEvent(S10.1.2.162:11111:333311335) failed, will be retried:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ProcessAddedSiloAsync(SiloAddress addedSilo, List`1 splitPartListSingle, List`1 splitPartListMulti)
at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ExecutePendingOperations().
[24-Jul-2020 18:35:22 WRN] Did not get ping response for ping #1/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:22 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #1
[24-Jul-2020 18:35:22 WRN] ProcessSiloAddEvent(S10.1.2.162:11111:333311335) failed, will NOT be retried:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.2.162:11111:333311335, will retry after 729.5555ms
at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ProcessAddedSiloAsync(SiloAddress addedSilo, List`1 splitPartListSingle, List`1 splitPartListMulti)
at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ExecutePendingOperations()
[24-Jul-2020 18:35:23 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:25 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:25 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:25 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:26 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:27 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:28 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:28 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:28 WRN] Did not get ping response for ping #2/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:28 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:28 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #2
[24-Jul-2020 18:35:28 INF] Closing connections to defunct silo S10.1.2.157:11111:333300013
[24-Jul-2020 18:35:28 INF] Closing connections to defunct silo S10.1.2.163:11112:333311335
[24-Jul-2020 18:35:29 INF] Closing connections to defunct silo S10.1.2.158:11112:333300010
[24-Jul-2020 18:35:29 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:31 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:31 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:31 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:32 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:33 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:34 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:34 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:34 WRN] Did not get ping response for ping #3/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:34 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #3
[24-Jul-2020 18:35:34 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:35 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:37 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:37 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:37 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:38 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:39 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:40 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:41 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:41 WRN] Did not get ping response for ping #4/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:41 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:41 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #4
[24-Jul-2020 18:35:42 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:44 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:44 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:44 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:45 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:46 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:47 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:47 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:47 WRN] Did not get ping response for ping #5/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:47 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:47 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #5
[24-Jul-2020 18:35:48 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:50 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:50 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:50 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:51 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:52 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:53 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:53 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:53 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:53 WRN] Did not get ping response for ping #6/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:53 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #6
[24-Jul-2020 18:35:54 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:56 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:56 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:56 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:57 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:58 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:59 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:59 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:59 WRN] Did not get ping response for ping #7/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:59 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:59 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #7
[24-Jul-2020 18:36:00 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:02 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:02 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:02 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:03 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:04 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:05 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:05 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:05 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:05 WRN] Did not get ping response for ping #8/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:05 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #8
[24-Jul-2020 18:36:06 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:09 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:09 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:09 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:10 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:10 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:12 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:12 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:12 WRN] Did not get ping response for ping #9/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:12 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:12 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #9
[24-Jul-2020 18:36:13 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:15 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:15 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:15 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:16 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:17 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:18 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:18 WRN] Did not get ping response for ping #10/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:18 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:18 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:18 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #10
[24-Jul-2020 18:36:18 INF] Before collection#1: memory=14MB, #activations=0, collector=<#Activations=0, #Buckets=0, buckets=[]>.
[24-Jul-2020 18:36:18 INF] After collection#1: memory=14MB, #activations=0, collected 0 activations, collector=<#Activations=0, #Buckets=0, buckets=[]>, collection time=00:00:00.0045574.
[24-Jul-2020 18:36:18 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.2.162:11111:333311335, will retry after 667.3114ms
at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:19 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:21 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:21 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:21 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:22 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:23 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:24 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:24 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:24 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:24 WRN] Did not get ping response for ping #11/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:24 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #11
[24-Jul-2020 18:36:25 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
My Silo
.UseOrleans(builder =>
{
builder
.UseAdoNetClustering(options =>
{
options.Invariant = "Npgsql";
options.ConnectionString = Environment.GetEnvironmentVariable("SILO_CLUSTERING_CS");
})
.ConfigureEndpoints(siloPort: 11111, gatewayPort: 30000, listenOnAnyHostAddress: true)
.Configure<ClusterOptions>(options =>
{
options.ClusterId = Environment.GetEnvironmentVariable("SILO_CLUSTER_ID");
options.ServiceId = Environment.GetEnvironmentVariable("SILO_SERVICE_ID");
})
.ConfigureApplicationParts(parts =>
{
parts.AddApplicationPart(typeof(ResourceGrain).Assembly).WithReferences();
});
})
It seems that the old Silos are staying in the membership table forever as i stop/start kubernetes pods and causing Orleans to get stuck performing multiple re-tries for nonexistent IP’s for around 5 or 6 times each, so the more rows there are in the table the longer it will take re-trying.
I noticed this can also be reproduced by using just normal docker containers in a docker-compose deployment
In this case, pods are crashing when the membership table contains 6 or more rows.
Is there any configuration i’m missing on my Orleans silo, maybe there is a special way of deploying Silos to a Kubernetes Pod ?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 23 (11 by maintainers)
The labels also need to be set as
orleans/serviceIdandorleans/clusterId, unlike in the PR descriptionIf you disable liveness probe you cannot recover from deadlocks. Stateful sets do not help afaik, because the IP is not stable, only the host name. I think I asked for this before, to use host names in the membership table.
You can’t use it that way: you have to set the labels on the pod spec, not by trying to set the env vars at startup. The feature queries Kubernetes for those labels in order to discover other pods in the cluster.
Hi @gabrielfreire, I’m glad to hear it, yes it is available in a preview release: https://www.nuget.org/packages/Microsoft.Orleans.Hosting.Kubernetes
The intention is to have it marked as -beta when 3.3.0 final is released