prometheus: remote_read - read_recent does not work as expected

What did you do?

I tried to configure prometheus with a remote_read endpoint

What did you expect to see?

I expected prometheus to query the remote endpoint only if the metrics are not available in the local storage.

What did you see instead? Under which circumstances?

Prometheus always queries the remote endpoint, no matter its configuration.

Environment

• System information:

  k8s 1.7.8

• Prometheus version:

  both v2 and master

• Prometheus configuration file:

    global:
      evaluation_interval: 1m
      scrape_interval: 1m
    remote_write:
      - url: "http://influxdb:8086/api/v1/prom/write?db=test"
        remote_timeout: 30s
        queue_config:
          capacity: 100000
          max_shards: 1000
          max_samples_per_send: 100
          batch_send_deadline: 5s
          max_retries: 10
          min_backoff: 30ms
          max_backoff: 100ms
    remote_read:
      - url: "http://influxdb:8086/api/v1/prom/read?db=test"
        remote_timeout: 30s
        read_recent: false

I believe the default for read_recent should be set to false here: https://github.com/prometheus/prometheus/blob/master/config/config.go#L200

From the logs of influxdb I can see that, no matter what, prometheus always queries the remote endpoint. Help is appreciated