blackbox_exporter: HTTP Probe fails due to DNS resolution when used with proxy

Host operating system: Linux

blackbox_exporter version: 0.19.0

What is the blackbox.yml module config.

modules:
  http_2xx_probes:
    prober: http
    timeout: 90s
    http:
      valid_status_codes: []
      fail_if_body_matches_regexp:
        - ".*(ERROR).*"
      headers:
        Host: https://www.example.com:1234
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      proxy_url: http://proxy-username:proxy-password@internal-proxy-server:3120
      tls_config:
        insecure_skip_verify: true

What is the prometheus.yml scrape config.

- job_name: test
  honor_timestamps: true
  params:
    domain:
    - test-domain
    module:
    - http_2xx_probes
  scrape_interval: 5m
  scrape_timeout: 4m
  metrics_path: /probe
  scheme: http
  file_sd_configs:
  - files:
    - /etc/prometheus/testfile.yaml
    refresh_interval: 1m
  proxy_url: http://proxy-username:proxy-password@internal-proxy-server:3120
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: blackbox-exporter:9115
    action: replace

What logging output did you get from adding &debug=true to the probe URL?

target=https://www.example.com:1234 level=debug msg="Error for HTTP request" err="Get \"https://123.45.67.8:1234\": Forbidden"
ts=2022-01-27T06:38:39.363Z caller=main.go:180 module=http_2xx_probes 
target=https://www.example.com:1234 level=debug msg="Response timings for roundtrip" roundtrip=0 start=2022-01-27T06:38:39.348049772Z dnsDone=2022-01-27T06:38:39.355050347Z connectDone=2022-01-27T06:38:39.355463276Z gotConn=0001-01-01T00:00:00Z responseStart=0001-01-01T00:00:00Z tlsStart=0001-01-01T00:00:00Z tlsDone=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2022-01-27T06:38:39.363Z caller=main.go:180 module=http_2xx_probes 
target=https://www.example.com:1234 level=debug msg="Probe failed" duration_seconds=0.023786742

What did you do that produced an error?

I added the proxy in http_probe to hit that specific url.

What did you expect to see?

I expected a 200 response from the http_probe.

What did you see instead?

The probe fails with 403 Forbidden error due to DNS resolution. From the above error logs, I found out that blackbox exporter pod is trying to do dns resolution and using the resolved ip address and port to pass through the proxy-server.

Note: i have replaced the actual DNS Name, IP Address and Proxy server (squid) details for privacy purpose in these logs. Issue : The probe fails with 403 Forbidden error. From the above error logs, I found out that blackbox exporter pod is trying to do DNS resolution for the public target (https://www.example.com:1234) and using the resolved ip address and port (https://123.45.67.8:1234) to pass through the squid proxy-server (http://proxy-username:proxy-password@internal-proxy-server:3120). In our proxy server we only have the DNS url (https://www.example.com:1234) whitelisted and not it’s IP Address. Reason: IP Address keeps changing and the proxy server admins in any org will not add dynamic ip’s onto the whitelist due to various reasons.

We tried adding host header (host: https://www.example.com:1234) also as a flag on the blackbox exporter config yaml. However, proxy is not able to recognize that and we continue to get the same error

To resolve this issue, we would request blackbox exporter source code owners/maintainers to add a feature/flag (Eg:- use_proxy_dns: true) to not resolve the DNS into IP Address. It will help various organizations, who are having similar proxy setup and is facing similar issues, so that they dont need to look into any other exporter/monitoring solutions.