spring-cloud-netflix: EIP publicip association not correctly updated on fresh instance

I’ve been directed over here from the eureka folks, as they believe this should just ‘work’. Have the following issue running off spring-cloud-netflix:1.1.4.RELEASE. The issue I opened over there is: https://github.com/Netflix/eureka/issues/840

There seems to be a problem with public EIP address association not being correctly updated when a new AWS server starts and has a new Eureka server starting with it. When the server starts up, it correctly registers itself:

2016-09-06 15:55:29.040  WARN 3399 --- [Thread-10] com.netflix.eureka.aws.EIPManager        : The selected EIP 54.67.102.122 is associated with another instance i-0666b391 according to AWS, hence skipping this
2016-09-06 15:55:29.628  INFO 3399 --- [Thread-10] com.netflix.eureka.aws.EIPManager        :


Associated i-25f11391 running in zone: us-west-1c to elastic IP: X.X.X.X

But, every minute after that we get the following log entry:

2016-09-06 16:24:55.568  INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl    : Got 1 instances from neighboring DS node
2016-09-06 16:24:55.568  INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl    : Renew threshold is: 1
2016-09-06 16:24:55.568  INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl    : Priming AWS connections for all replicas..
2016-09-06 16:24:55.568  INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl    : No peers needed to prime.
2016-09-06 16:24:55.568  INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl    : Changing status to UP
2016-09-06 16:24:55.713  WARN 3399 --- [Eureka-EIPBinder] com.netflix.eureka.aws.EIPManager        : The selected EIP X.X.X.X is associated with another instance i-0666b391 according to AWS, hence skipping this
2016-09-06 16:24:55.804  INFO 3399 --- [Eureka-EIPBinder] com.netflix.eureka.aws.EIPManager        : My instance i-25f11391 seems to be already associated with the EIP X.X.X.X

Debugging this, the call to isEIPBound() is always failing, and this is because the following is always null:

String myPublicIP = ((AmazonInfo) myInfo.getDataCenterInfo()).get(MetaDataKey.publicIpv4);

It looks like there is stale datacenterinfo and it never gets refreshed (from what I can tell) and there there are no settings available to have it refreshed automatically.

The odd side affect of this, and we noticed, is that the registry continually gets wiped, and reset causing obvious potential issues down stream for our clients.

I have been trying to find where this datacenter info might be refreshed, but am unable to find anything that might actually do that.

The deployed app only has a single main class in it:

@SpringBootApplication
@EnableEurekaServer
@EnableAutoConfiguration
public class EurekaServer {

    @Value("${server.port}")
    private Integer nonSecurePort;
    @Autowired
    private InetUtils utils;

    public static void main(String[] args) {
        new SpringApplicationBuilder(EurekaServer.class).web(true).run(args);
    }

    @Bean
    @Profile("aws")
    public EurekaInstanceConfigBean awsEurekaConfig() {
        EurekaInstanceConfigBean b = new EurekaInstanceConfigBean(utils);
        b.setNonSecurePort(nonSecurePort);
        b.setSecurePortEnabled(false);
        AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
        b.setDataCenterInfo(info);
        return b;
    }

}

About this issue

Original URL
State: open
Created 8 years ago
Comments: 23 (17 by maintainers)

Most upvoted comments

@herder Netflix devs have moved the functionality to a shared class that we will be able to leverage. https://github.com/Netflix/eureka/pull/843

spencergibb on Sep 9, 2016

This works for us:

@Configuration
@Slf4j
@ConditionalOnAwsCloudEnvironment
@EnableContextInstanceData
@Import(UtilAutoConfiguration.class)
@AutoConfigureAfter(UtilAutoConfiguration.class)
public class AwsInstanceConfig {

    @Value("${server.port:${SERVER_PORT:${PORT:8080}}}")
    int nonSecurePort;

    @Value("${management.port:${MANAGEMENT_PORT:${server.port:${SERVER_PORT:${PORT:8080}}}}}")
    int managementPort;

    @Value("${eureka.instance.hostname:${EUREKA_INSTANCE_HOSTNAME:}}")
    String hostname;

    @Autowired
    ConfigurableEnvironment env;


    @Bean
    public EurekaInstanceConfigBean eurekaInstanceConfigBean(InetUtils utils) {
        log.info("Setting AmazonInfo on EurekaInstanceConfigBean");
        final EurekaInstanceConfigBean instance = new EurekaInstanceConfigBean(utils) {

            @Scheduled(initialDelay = 30000L, fixedRate = 30000L)
            public void refreshInfo() {
                log.debug("Checking datacenter info changes");
                AmazonInfo newInfo = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
                if (!this.getDataCenterInfo().equals(newInfo)) {
                    log.info("Updating datacenterInfo to {}", newInfo);
                    ((AmazonInfo) this.getDataCenterInfo()).setMetadata(newInfo.getMetadata());
                }
            }

            private AmazonInfo getAmazonInfo() {
                return (AmazonInfo) getDataCenterInfo();
            }

            @Override
            public String getHostname() {
                AmazonInfo info = getAmazonInfo();
                final String publicHostname = info.get(AmazonInfo.MetaDataKey.publicHostname);
                return this.isPreferIpAddress() ?
                    info.get(AmazonInfo.MetaDataKey.localIpv4) :
                    publicHostname == null ?
                        info.get(AmazonInfo.MetaDataKey.localHostname) : publicHostname;
            }

            @Override
            public String getHostName(final boolean refresh) {
                return getHostname();
            }

            @Override
            public String getHomePageUrl() {
                return super.getHomePageUrl();
            }

            @Override
            public String getStatusPageUrl() {
                String scheme = getSecurePortEnabled() ? "https" : "http";
                return scheme + "://" + getHostname() + ":"
                    + managementPort + getStatusPageUrlPath();
            }

            @Override
            public String getHealthCheckUrl() {
                String scheme = getSecurePortEnabled() ? "https" : "http";
                return scheme + "://" + getHostname() + ":"
                    + managementPort + getHealthCheckUrlPath();
            }
        };
        AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
        log.info("Info: {}", info);
        instance.setDataCenterInfo(info);
        instance.setNonSecurePort(this.nonSecurePort);
        instance.setInstanceId(getDefaultInstanceId(this.env));
        if (this.managementPort != this.nonSecurePort && this.managementPort != 0) {
            if (StringUtils.hasText(this.hostname)) {
                instance.setHostname(this.hostname);
            }
        }

        return instance;
    }

}

I.e. we do a scheduled check on whether the datacenterinfo has been updated, and reset it in that case. I’m sure there’s room for cleanup here, but maybe it’s a start?

herder on Sep 9, 2016

thanks @DickChesterwood. 1.6 is part of Dalston. See spring-cloud-release/milestones

spencergibb on Feb 9, 2017

Many thanks to @herder for the suggested auto-refresh hack; working great for me.

I can’t quite work out when the Eureka 1.6 upgrade will appear, will it be in the Dalston release train?

It’s far too long to read but I’ve documented my experiments here - let me know if I’ve made any blunders

Edit to add that the OP noticed that not doing this refresh causes the registry to be wiped; I had the opposite experience that instances never get expired (it’s not self preservation!). I can’t think how that could be the case, so I’d be interested if anyone has any insight.

DickChesterwood on Feb 9, 2017