micrometer: Fails to push metrics to Stackdriver with short step interval and large batch size

Hi, I’m using micrometer version 1.1.3 and spring boot 2.1.0.RELEASE. StackdriverMeterRegistry is defined as a spring bean :

@Bean
  public static StackdriverMeterRegistry stackdriver() {
    return StackdriverMeterRegistry.builder(new StackdriverConfig() {
      @Override
      public String projectId() {
        return "test-project";
      }

      @Override
      public String get(String key) {
        return null;
      }

      @Override
      public Duration step() {
        return Duration.ofSeconds(5);
      }
    }).build();
  }

However, when I ran the app on GCP I keep seeing the following warning without any stacktrace : failed to send metrics to stackdriver: INTERNAL: http2 exception

It’s strange because I can push metrics from the same application and environment using just google-cloud-monitoring library: https://cloud.google.com/monitoring/docs/reference/libraries#client-libraries-usage-java

Could you please let me know if I’m missing something in configuration etc.

Thanks, Gleb

About this issue

Original URL
State: open
Created 5 years ago
Reactions: 5
Comments: 19 (6 by maintainers)

Commits related to this issue

Configure Stackdriver meter registry This doesn't work with the following error: failed to send metrics to stackdriver: INTERNAL: http2 exception See https://github.com/micrometer-metrics/micromete... — committed to izeye/sample-micrometer-spring-boot by izeye 5 years ago
Include stack trace when failed to send metrics to Stackdriver See gh-1268 — committed to izeye/micrometer by izeye 5 years ago

Most upvoted comments

I found the solution for me. I have to add custom tags to the metrics to ensure that these are unique across services/instances:

management:
  metrics:
    tags:
      app: my-unique-service-label
      instance: ${random.uuid}

See also:

AQS-DTheuke on Aug 17, 2022

Im also facing this issue, one of my buckets is negative. I did a little debugging like so:


 if (latencyNanos < 0) {
        print(s"got latency < 0 ${latencyNanos} ${name}" )
      }

      Timer
        .builder("akka.streams.upstreamlatency")
        .tags("step", name)
        .sla(Micrometer.timerPercentiles: _*)
        .description("Difference between a push and the last pull, gives a reading of how much time the upstream takes to produce an element after it's been requested")
        .register(micrometer)
        .record(latencyNanos, TimeUnit.NANOSECONDS)

And im definately not sending negative values in, somehow in the conversions the values are becoming negative

LukeDefeo on Apr 13, 2020

@izeye , thanks for reproducing it, the nearest I can try is Monday, I will let you know if it works.

GlebMendrul on Mar 9, 2019