micrometer: DatadogMeterRegistry StackOverflowError

Hi,

we receive endless Stackoverflow exceptions if the datadog endpoint is not available anymore for any reason or they have to restart the servers or something:

The message in DD lookes like this:

{"timestamp":"2019-03-18T23:47:21.187+00:00","message":"failed to send metrics to datadog","logger":"i.m.datadog.DatadogMeterRegistry","thread_name":"datadog-metrics-publisher","status":"WARN","stack_trace":"java.lang.StackOverflowError: null\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasN

I don’t see any logs other than this, no warning that metrics couldn’t be sent or any other exception.

If i look into DatadogMeterRegistry i just see only one line where “hasNext()” is used, maybe you can find the loop in your code.

We’r using “micrometer-registry-datadog-1.1.2.jar” which is shipped with spring-boot 2.1

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (6 by maintainers)

Commits related to this issue

Make Iterator.hasNext() of Collections.unmodifiableList() trigger StackOverflowError See https://github.com/micrometer-metrics/micrometer/issues/1292 — committed to izeye/samples-java-branches by izeye 5 years ago

Most upvoted comments

ok, we’ll wait for the next jdk-11 updates and see if we’ll have the problem again, we had the problem only 2 times in the past 4 months, maybe it will take a while until we see it again …

@olamelin thanks for you investigation @izeye you can close the ticket if you want to, i’ll open a new one if we see the error again after the jdk-11.0.3 release

jens-meiss on Apr 3, 2019

Since my last comment I created a parallel image of our code that was based on a openjdk12 image. In order to see if the JDK issue was the cause. The one using openjdk12 has not reported the stack overflow the last 6 days but the parallel image (which was restarted) has shown this error multiple times. In other words I think it is safe to say that this issue is caused by the bug described here https://bugs.openjdk.java.net/browse/JDK-8214129

olamelin on Apr 1, 2019

ok, thank you for your analysis, i hoped that it’s easy to find the loop, but as always with these kind of errors it’s not 😃 …

i’ll come back if i have a reproduceable example

jens-meiss on Mar 21, 2019