alloy: Manually stored metrics do not have metadata

Samples which are manually written to the WAL (i.e., written outside of the normal scraping process of a metrics instance) do not support metadata being sent over remote write.

Metadata is used to display metric type and help info from Prometheus query frontends like Grafana.

Remote write currently expects that metadata comes from an instance of scrape.Manager, which is not always the case for code relying on remote write to send samples.

This impacts at least the following:

  • spanmetrics
  • integrations-next (which uses an external scrape manager whose metadata is not exposed directly to remote_write)
  • Metrics from integrations when using Grafana Agent Operator (since it uses integrations-next; see above)
  • Flow components

This went mostly unnoticed because most metrics still have metadata. I first became aware of this when prototyping grafana/agent#1261, where scraping and remote_write was initially fully decoupled, causing all metadata to disappear.

It’s not immediately obvious what the best solution here is. Some way of being able to customize what metadata gets collected by remote_write would be ideal though.

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 1
  • Comments: 15 (8 by maintainers)

Most upvoted comments

To add some context around the upstream status of this issue; we’ve had support for recording Metadata as WAL records and storing them from the scrape loop for a while now. The final piece, the way we’d communicate this over the remote write was still a point of contention.

The original design doc was agreed upon, but to achieve more efficient metadata delivery, it was decided to wait and group this feature together with an interning table implementation that was planned on remote write v2.0.

Currently metadata-over the remote write is already implemented on the remote-write-2.0 branch so we’re waiting for the feature to be completed to solve this issue on the Agent side as well.

As in, will Mimir also need a similar fix?

No, this is only relevant for the agent.

There was a Prometheus PR (prometheus/prometheus#7771) that adds metadata to the WAL. From what I understand, having metadata stored in the WAL is still of interest to the Prometheus team (or at least to @cstyan) and would resolve this issue.

(cc @mapno for the heads up wrt the spanmetrics part)