rancher: Failed to find instance to delegate to for agent

And again.

Rancher version : 1.4.0 / Cattle Docker version 1.12.6 OS: Ubuntu 16.04

2017-02-09 15:00:10,354 ERROR [:] [] [] [] [ecutorService-5] [i.c.p.agent.impl.AgentLocatorImpl   ] Failed to find instance to delegate to for agent [317] uri [delegate:///?vnetId=11&networkServiceProviderId=5] 
2017-02-09 15:00:10,355 ERROR [:] [] [] [] [ecutorService-5] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [services, agent-instance-scripts, monit, agent-instance-startup, node-services, ipsec, ipsec-hosts, hosts, iptables, healthcheck] on [agent:1919] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=21&networkServiceProviderId=8] has no instance associated
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812) ~[guava-15.0.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.lookupAgent(AgentLocatorImpl.java:85) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.call(ConfigUpdatePublisher.java:327) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publishUpdate(ConfigUpdatePublisher.java:231) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publish(ConfigUpdatePublisher.java:201) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.loopBody(ConfigUpdatePublisher.java:127) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.doRun(ConfigUpdatePublisher.java:100) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=21&networkServiceProviderId=8] has no instance associated
	at io.cattle.platform.agent.impl.AgentLocatorImpl.getAgentForDelegate(AgentLocatorImpl.java:132) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.buildDelegate(AgentLocatorImpl.java:104) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:52) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:47) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
	... 23 common frames omitted

@aemneina @sra

Thanks. That’s worked for me.

1. Find out active agent records which didn’t related to a host.

select id, kind, uuid, state, created, uri from agent where id not in (select agent_id from host) and state = 'active' and uri like 'delegate%';

1. Double check the uri with your rancher-server log. In my case I got 2 records which uri match the exception in server log.

2. Update the record state to purged

update agent set state = 'purged' where id = <ID_FROM_PREVIOUS_SELECT>

1. Restart rancher-server.

I made this quit by finding the row in the agent table in the rancher database (use the delegate message from your log messages):

select * from agent where uri like 'delegate:///?vnetId=19&networkServiceProviderId=3'

Then I marked the record as purged and restarted rancher server.

update agent set state = 'purged' where id = ID_FROM_PREVIOUS_SELECT

This stopped the spam, but I’m not sure if it had any other effect. So far so good.

@CLEMARCx I dont think you have the same issue as the others in this thread. It looks like there are orphaned agents, there might be an entry for the agent that doesnt match any existing host. Does this query return anything: select id from agent where id not in (select agent_id from host)