influxdb: Prometheus remote write missing measurements name
Bug report
Metrics send from prometheus to influxdb with native influx remote write have missing measurement name.
All metrics are saved by InfluxDB to _ measurement.
> select * from _ where __name__='up' and kubernetes_io_hostname='ctl03' limit 1;
name: _
time From To __name__ action addonmanager_kubernetes_io_mode alertmanager alertname alertstate annotation_io_kubernetes_container_hash annotation_io_kubernetes_container_ports annotation_io_kubernetes_container_restartCount annotation_io_kubernetes_container_terminationMessagePath annotation_io_kubernetes_container_terminationMessagePolicy annotation_io_kubernetes_pod_terminationGracePeriod annotation_kubernetes_io_config_seen annotation_kubernetes_io_config_source annotation_kubernetes_io_created_by annotation_prometheus_io_port annotation_prometheus_io_scrape annotation_scheduler_alpha_kubernetes_io_tolerations app backend beta_kubernetes_io_arch beta_kubernetes_io_os bind branch buildDate call client cluster_name code com_docker_stack_namespace com_docker_swarm_node_id com_docker_swarm_service_id com_docker_swarm_service_name com_docker_swarm_task_id com_docker_swarm_task_name com_mirantis_image_specs_calico_bird6Url com_mirantis_image_specs_calico_birdUrl com_mirantis_image_specs_calico_birdclUrl com_mirantis_image_specs_calico_confdUrl com_mirantis_image_specs_calico_felixImage com_mirantis_image_specs_docker_imgTag com_mirantis_image_specs_gerritChangeId com_mirantis_image_specs_gerritChangeNumber com_mirantis_image_specs_gerritPatchsetNumber com_mirantis_image_specs_gerritPatchsetRevision com_mirantis_image_specs_gerritProject com_mirantis_monitoring compiler container_image container_name container_version contentType controller controller_revision_hash cpu database device dialer_name direction dockerVersion endpoint engine engine_host event exe f64 fstype gitCommit gitTreeState gitVersion goVersion goversion grpc_code grpc_method grpc_service grpc_type handler host id image input instance integration interface interval io_kubernetes_container_logpath io_kubernetes_container_name io_kubernetes_docker_type io_kubernetes_pod_name io_kubernetes_pod_namespace io_kubernetes_pod_uid io_kubernetes_sandbox_id ip_version job k8s_app kernelVersion kubernetes_io_hostname le listener_name major method minor name namespace network node_attribute_master node_host node_id node_name node_role_kubernetes_io_master node_role_kubernetes_io_node operation_type osVersion output path pattern pid platform pod_name pod_template_generation pod_template_hash port process_name protocol_instance proxy quantile queue reason refid region remote resource retentionPolicy revision role route_type rule_type scope scrape_job server service severity slice socket state state_prefix status stratum subresource sv table type unit url username verb version walPath
---- ---- -- -------- ------ ------------------------------- ------------ --------- ---------- --------------------------------------- ---------------------------------------- ----------------------------------------------- --------------------------------------------------------- ----------------------------------------------------------- --------------------------------------------------- ------------------------------------ -------------------------------------- ----------------------------------- ----------------------------- ------------------------------- ---------------------------------------------------- --- ------- ----------------------- --------------------- ---- ------ --------- ---- ------ ------------ ---- -------------------------- ------------------------ --------------------------- ----------------------------- ------------------------ -------------------------- ---------------------------------------- --------------------------------------- ----------------------------------------- ---------------------------------------- ------------------------------------------ -------------------------------------- --------------------------------------- ------------------------------------------- --------------------------------------------- ----------------------------------------------- -------------------------------------- ----------------------- -------- --------------- -------------- ----------------- ----------- ---------- ------------------------ --- -------- ------ ----------- --------- ------------- -------- ------ ----------- ----- --- --- ------ --------- ------------ ---------- --------- --------- --------- ----------- ------------ --------- ------- ---- -- ----- ----- -------- ----------- --------- -------- ------------------------------- ---------------------------- ------------------------- ---------------------- --------------------------- --------------------- ------------------------ ---------- --- ------- ------------- ---------------------- -- ------------- ----- ------ ----- ---- --------- ------- --------------------- --------- ------- --------- ------------------------------ ---------------------------- -------------- --------- ------ ---- ------- --- -------- -------- ----------------------- ----------------- ---- ------------ ----------------- ----- -------- ----- ------ ----- ------ ------ -------- --------------- -------- ---- ---------- --------- ----- ---------- ------ ------- -------- ----- ------ ----- ------------ ------ ------- ----------- -- ----- ---- ---- --- -------- ---- ------- -------
1513343319131000000 up amd64 linux 0 ctl03 kubernetes-cadvisor ctl03 true region1
Looks like labels from ALL metrics are merged, so each metric have labels (set to null) from all other metrics…
When Prometheus Remote Storage Adapter is in place:
> select * from up limit 1;
name: up
time beta_kubernetes_io_arch beta_kubernetes_io_os instance job kubernetes_io_hostname node_role_kubernetes_io_master node_role_kubernetes_io_node region value
---- ----------------------- --------------------- -------- --- ---------------------- ------------------------------ ---------------------------- ------ -----
1513585977606000000 amd64 linux ctl03 kubernetes-node ctl03 true region1 0
IMHO native support for prometheus remote write should behave similar to remote storage adapter, and use label __name__ as measurement name
InfluxDB: 1.4.2-1 Ubuntu: 16.04 Prometheus: 2.0.0
How to reproduce:
- Configure remote write in prometheus:
remote_write:
# docker_remote_write
- url: http://INFLUXDB_HOST:8086/api/v1/prom/write?u=USER&p=PASS&db=DB
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 3
- Comments: 15 (3 by maintainers)
Ok, I think I’ve found a solution to this problem, but bear with me for a little bit of extra history/detail.
The original reason I pushed everything into a single measurement named
_was partly because of IFQL, but also because of a limitation in our query engine. Prometheus has matchers against__name__for!=and!~(which are standard tag matchers) which InfluxQL can’t support against measurements. For the first version of the Prometheus remote read endpoint I had to use the InfluxQL engine since that’s all that existed.That limitation kept me from being able to put the Prometheus metric name in addition to having
__name__as a tag. The experimental Prom -> Influx gateway opted to throw errors for those cases: https://github.com/prometheus/prometheus/blob/master/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go#L153-L164However, since I wrote the original remote adapter, we’ve had a bunch of work done on creating a lower level query layer that bypasses the InfluxQL engine to get to storage. This is work directly related to IFQL, but I realized it could probably be used for the remote adapter. I’ve pushed a branch here: https://github.com/influxdata/influxdb/tree/pd-prometheus-to-new-storage
The good news is that this will probably be more performant and use less memory since it doesn’t use the entirety of the InfluxQL query engine to process the request. I’ve tested it manually and it seems to work. That’s experimental (just like this whole feature). I need to review with @stuartcarnie to figure out a less “ghetto-hack” solution.
If you’re so inclined, I’d be interested in hearing if the branch works for you. This is a breaking change so you’d have to start with fresh data to really test it out.
Fixing this would also make the Influxdb prometheus backend backwards compatible with the remote_storage_adapter. This is essential to use the native backend as a drop-in replacement for the adapter.
Hi there!
Is someone taking care of this? I might give it a try, but I am totally new to Influxdb codebase and Golang. From my initial findings, the bug is localized in prometheus/converters.go. We should adapt the write function and probably also the read. I don’t know what the other fonctions do.
I am not the maintainer, but I guess any help is welcomed 😃
I’m fairly new to InfluxDB, so please excuse me if this is obvious, but how can I take advantage of this change @e-dard? In Influx v1.6.1 (in a new database, with data fresh out of Prometheus) I still only see one measurement (
_), and I cannot use Prometheus metric names as measurements in Influx queries. That is, I still have to write queries likeselect * from _ where __name__='up' and kubernetes_io_hostname='ctl03' limit 1;instead of the preferableselect * from up limit 1;.What am I missing?
Like @Jean-Daniel I would also like to know if this behaviour is on by default or if we need to opt-in with a setting.