prometheus: /api/v1/read is not documented
Two issues regarding remote_read feature:
-
Missing documentation about
/api/v1/readin the api documentation page here. -
Also in the configuration page here
# The labels to add to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
[ <labelname>: <labelvalue> ... ]
This does not give enough information that you cannot use remote_read with the conjunction with external_labels setting. Because if you do set in the remote_read prometheus a label but its different label then the prometheus instances that the remote_read prometheus is set - then the remote read does not work.
Consider to add some note in side the above link or maybe inside remote_read.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 9
- Comments: 24 (15 by maintainers)
Commits related to this issue
- documentation: Add Reference to the remote read API - For now this is relatively simplistic, but at least acknowledges some of the basics, and points out some parts that might not be obvious at fir... — committed to hdost/prometheus by hdost 4 years ago
- documentation: Add Reference to the remote read API - For now this is relatively simplistic, but at least acknowledges some of the basics, and points out some parts that might not be obvious at fir... — committed to hdost/prometheus by hdost 4 years ago
- documentation: Add Reference to the remote read API - For now this is relatively simplistic, but at least acknowledges some of the basics, and points out some parts that might not be obvious at fir... — committed to hdost/prometheus by hdost 4 years ago
- documentation: Add Reference to the remote read API - For now this is relatively simplistic, but at least acknowledges some of the basics, and points out some parts that might not be obvious at fir... — committed to hdost/prometheus by hdost 4 years ago
šš¼ ok on the Swagger part , and yes that project only supports JSON. The larger point was more about it might be best that the docs eventually get generated from sources in an effort to reduce drift. (Could be a separate issue)
As far as pointing to the example adapter Iām assuming this is what youāre meaning ādocumentation/examples/remote_storage/remote_storage_adapter/opentsdb/client.goā
I donāt think that would hurt to start, I think that would be a good supplement.
I can work on a PR for a new doc page inspired by the management api page, but about remote read.
From @skupjoe
I just went through the same thing using https://github.com/jacksontj/promxy and had
external_labelsset in my global config; removing that configuration made everything work.I spent days troubleshooting why remote read was not working (in this case, M3DB -> Promethesu), and it turned out to be because I was using
external_labelsin my global config. As described in the OP, this just flat-out breaks remote read from working (but Prometheus will still accept remote writes no problem) and there are no errors displayed in logs, nothing.I wish this was documented better- it would have saved me a lot of time.
Or I would appreciate some sort of error that displays when configuration is incompatible with
remote_read, or if the remote read endpoint canāt be reached. (I tested with using a garbage address and Prometheus carries on without warning, until you try to make a query in the WebUI. And Grafana does not display any error. )Also not mentioned is the fact that metrics browser (dropdown) in Prometheus/Grafana donāt display any remote read timeseries- whether it is working or not. This added additional confusion.
Usage questions like that are better suited to the prometheus-users mailing list.
Itās part of remote storage, which we havenāt documented generally and wouldnāt go on that page as itās not following our v1 API pattern.
You can use them together. This sounds more like a usage question.