micrometer: Micrometer logs error message when partial error happens due to Infinity value using Elasticsearch

Micrometer core and micrometer-elastic version: 1.1.3

Springboot version: 2.1.3.RELEASE

Hi, Using micrometer getting metrics from springboot application using micrometer-registry-elastic

Micrometer keeps logging error messages:

ERROR [market-fetcher,,,] 8106 --- [trics-publisher] i.m.elastic.ElasticMeterRegistry : failed to send metrics to elastic:

When checking elastic index it’s working, index get’s updated with new values but I noticed one of the entries in there reports an error:

{"index":{"_index":"market-fetcher-metrics-2019-04","_type":"doc","_id":"XyEp_WkB2oH6K2ScUlT5","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse [value]","caused_by":{"type":"illegal_argument_exception","reason":"[float] supports only finite values, but got [Infinity]"}}}}

which is probably what’s causing the error log entry being generated. I guess it should instead be able to wrap / handle Infinity? Also not sure where the error is exactly from as we don’t explicitly wrap any of the metrics in primitive float, still investigating that.

Find below a full entry of the log error message and pls let me know if it needs further details.

‘2019-04-08 15:35:37.761 ERROR [market-fetcher,] 8106 — [trics-publisher] i.m.elastic.ElasticMeterRegistry : failed to send metrics to elastic: {“took”:24,“errors”:true,“items”:[{“index”:{“_index”:“market-fetcher-metrics-2019-04”,“_type”:“doc”,“_id”:“CCEp_WkB2oH6K2ScUlT5”,“_version”:1,“result”:“created”,“_shards”:{“total”:2,“successful”:1,“failed”:0},“_seq_no”:322,“_primary_term”:1,“status”:201}},{“index”:{“_index”:“market-fetcher-metrics-2019-04”,“_type”:“doc”,“_id”:“CSEp_WkB2oH6K2ScUlT5”,“_version”:1,“result”:“created”,“_shards”:{“total”:2,“successful”:1,“failed”:0},“_seq_no”:328,“_primary_term”:1,“status”:201}}, … ,{“index”:{“_index”:“market-fetcher-metrics-2019-04”,“_type”:“doc”,“_id”:“XyEp_WkB2oH6K2ScUlT5”,“status”:400,“error”:{“type”:“mapper_parsing_exception”,“reason”:“failed to parse [value]”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:“[float] supports only finite values, but got [Infinity]”}}}},{“index”:{“_index”:“market-fetcher-metrics-2019-04”,“_type”:“doc”,“_id”:“YCEp_WkB2oH6K2ScUlT5”,“_version”:1,“result”:“created”,“_shards”:{“total”:2,“successful”:1,“failed”:0},“_seq_no”:345,“_primary_term”:1,“status”:201}},{“index”:{“_index”:“market-fetcher-metrics-2019-04”,“_type”:“doc”,“_id”:“YSEp_WkB2oH6K2ScUlT5”,“_version”:1,“result”:“created”,“_shards”:{“total”:2,“successful”:1,“failed”:0},“_seq_no”:341,“_primary_term”:1,“status”:201}}, … ,{“index”:{“_index”:“market-fetcher-metrics-2019-04”,“_type”:“doc”,“_id”:“EiEp_WkB2oH6K2ScUlX6”,“_version”:1,“result”:“created”,“_shards”:{“total”:2,“successful”:1,“failed”:0},“_seq_no”:383,“_primary_term”:1,“status”:201}}]}’

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 24 (13 by maintainers)

Commits related to this issue

Log request body on errors for Elasticsearch meter registry See gh-1370 — committed to izeye/micrometer by izeye 5 years ago
Log request body on errors for Elasticsearch meter registry (#1373) See gh-1370 — committed to micrometer-metrics/micrometer by izeye 5 years ago
Add a custom meter providing infinity to reproduce https://github.com/micrometer-metrics/micrometer/issues/1370 — committed to izeye/sample-micrometer-spring-boot by izeye 5 years ago
Drop custom meters having any non-finite measurement for Elasticsearch Closes gh-1370 — committed to izeye/micrometer by izeye 5 years ago
Drop custom meters' non-finite measurements for Elasticsearch (#1374) Closes gh-1370 — committed to micrometer-metrics/micrometer by izeye 5 years ago

Most upvoted comments

As @izeye’s sample shows, you need to add to your build configuration the repository that we publish snapshots to (https://repo.spring.io/libs-snapshot) and then use the snapshot version 1.1.5-SNAPSHOT. https://github.com/micrometer-metrics/micrometer#snapshot-builds

shakuzen on Apr 11, 2019

I created a sample to reproduce this and got the following error:

2019-04-10 15:08:39.904 ERROR 15826 --- [trics-publisher] i.m.elastic.ElasticMeterRegistry         : failed to send metrics to elastic: {"took":10,"errors":true,"items":[{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"P1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12831,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"QFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12832,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"QVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12901,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"QlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12902,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"Q1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12506,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"RFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12864,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"RVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12865,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"RlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12507,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"R1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12833,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"SFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12508,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"SVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12509,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"SlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12834,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"S1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12866,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"TFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12510,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"TVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12835,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"TlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12511,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"T1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12836,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"UFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12512,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"UVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12513,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"UlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12791,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"U1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12903,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"VFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12867,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"VVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12837,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"VlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12792,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"V1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12793,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"WFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12794,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"WVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12838,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"WlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12868,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"W1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12795,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"XFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12869,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"XVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12870,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"XlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12871,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"X1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12904,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"YFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12514,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"YVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12905,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"YlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12839,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"Y1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12840,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"ZFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12906,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"ZVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12907,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"ZlrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12872,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"Z1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12796,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"aFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12908,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"aVrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12515,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"alrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12841,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"a1rcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12873,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"bFrcBWoB8nnUV1ew1Wns","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12842,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"bVrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12874,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"blrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12909,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"b1rcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12843,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"cFrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12875,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"cVrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12844,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"clrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12845,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"c1rcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12516,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"dFrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12797,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"dVrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12876,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"dlrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12910,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"d1rcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12877,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"eFrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12798,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"eVrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12911,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"elrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12799,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"e1rcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12517,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"fFrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12878,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"fVrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12879,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"flrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12912,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"f1rcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12846,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"gFrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12913,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"gVrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12518,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"glrcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12880,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"g1rcBWoB8nnUV1ew1Wnt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":12519,"_primary_term":1,"status":201}},{"index":{"_index":"metrics-2019-04","_type":"doc","_id":"hFrcBWoB8nnUV1ew1Wnt","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse field [value] of type [float]","caused_by":{"type":"illegal_argument_exception","reason":"[float] supports only finite values, but got [Infinity]"}}}}]}

I’m not sure it’s the same cause as this issue but it’s worth fixing regardless of it.

This looks exactly like the original error I posted indeed.

mmaia on Apr 10, 2019

It seems to be possible for custom meters to supply infinity or NaN.

@mmaia Do you have any custom meter which could supply infinities? It’d be good to have a sample to reproduce this if possible. In the mean time I created #1373 to help track down the cause of this.

Yes, I get those from kafka client in producers and streams(which underlying wraps a consumer producer) when we get the metrics from them. It reports -Infinity and Infinity.

mmaia on Apr 10, 2019