Re: @benyanke HTTPs doesn’t solve this. The service already allows HTTPs access (and insecure, because some legacy devices that use this service don’t have libssl), but that won’t help if a client is getting their X-Forward-For header modified.

Let’s say you make this curl request:

$ curl -H "X-Forwarded-For: 127.0.0.1" https://api.ipify.org

You will get back an IP of 127.0.0.1 from ipify. This is because of the way the load balancers handle the incoming request header (as I mentioned above).

Sure: SSL helps with MITM which is what you’re likely referring to: but if you’re connecting TO A PROXY they will almost definitely be overwriting the X-Forwarded-For header which is the problem.

And re: @jonashaag I’ve explained above the only way to solve this. What it basically boils down to is that we’ll consider moving to new infrastructure when possible. Right now there’s just not enough time to do so.

And… Warm DNS cache has nothing to do with that performance difference. The difference in performance is simply because the ident.me service isn’t designed for scale. Let me explain:

Check out the dig record for ident.me:

$ curl https://ident.me

; <<>> DiG 9.10.3-P4-Ubuntu <<>> ident.me
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46044
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;ident.me.			IN	A

;; ANSWER SECTION:
ident.me.		3	IN	A	176.58.123.25

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Oct 30 10:30:20 PDT 2017
;; MSG SIZE  rcvd: 53

See how there’s a single A record with a single public IP address? That means the service is running on a single server directly (with no redundancy).

This is fine for small services that don’t need to deal with high volumes of requests, but ipify doesn’t have that luxury. ipify was designed from day 1 to support billions of requests per day, and to be able to seamlessly scale as much as necessary.

Take a look at the dig info for ipify:

$ dig api.ipify.org

; <<>> DiG 9.10.3-P4-Ubuntu <<>> api.ipify.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4320
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;api.ipify.org.			IN	A

;; ANSWER SECTION:
api.ipify.org.		243	IN	CNAME	nagano-19599.herokussl.com.
nagano-19599.herokussl.com. 1541 IN	CNAME	elb097307-934924932.us-east-1.elb.amazonaws.com.
elb097307-934924932.us-east-1.elb.amazonaws.com. 14 IN A 184.73.220.206
elb097307-934924932.us-east-1.elb.amazonaws.com. 14 IN A 23.23.170.235
elb097307-934924932.us-east-1.elb.amazonaws.com. 14 IN A 174.129.241.106

;; Query time: 40 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Oct 30 10:32:59 PDT 2017
;; MSG SIZE  rcvd: 188

See the CNAMEs? We’re currently running on 3 separate load balancers. And that’s JUST the load balancer redundancy. These are massive load balancers that are not only highly redundant (as you can see), but are also distributed to multiple different availability zones in Amazon. This means in the event of massive outages, the service will still be running without problems.

When you’re only accepting requests to a single server, you can get away with a small amount of lower latency because the request is only hitting one server.

When you’re talking to scalable services like ipify, you are actually talking to multiple servers.

When a request goes to ipify, it first hits one of those load balancers, then is forwarded to a localized web server in the same availability zone. This is done on a round-robin basis to evenly distribute load.

The web server (running ipify’s source) is then going to process the request in parallel with millions of others, and return a response to the load balancer, which, in turn, forward the response to the user.

Finally: one additional thing to note: latency depends on where you’re connecting from in the world.

ipify is hosted on Amazon in the US-East region to be globally accommodating as much as possible. That location is split between EU/US West coast, and is statistically the shortest length to most parts of the world by cable (if we’re talking about JUST us-based zones.

If you perform a geoip lookup on ident.me’s service, you’ll see it’s running on a single Linode instance in their London region. This means that users who are closer to London will obviously have less latency than reaching the US east coast.

I’m on a fiber optic connection currently in San Francisco, CA, and here’s my results:

$ time curl https://ident.me
64.85.227.231curl https://ident.me  0.04s user 0.01s system 5% cpu 0.808 total
$ time curl https://api.ipify.org
64.85.227.231curl https://api.ipify.org  0.04s user 0.00s system 11% cpu 0.410 total

As you can see, there is no difference.

This is sort of a lengthy response, but I wanted to share it to explain the actual info behind all this, so hopefully you understand what’s happening: I want to be totally transparent.

Finally: if you want to host ipify yourself on your own servers (for instance, if you want to spin up a Linode instance), you can easily do this (that’s why it is open source!) – this way, you won’t need to worry about any load balacner issues.

-R

NOTE: ipify currently does nearly 30 billion requests per month. Moving infrastructure isn’t a super simple / quick task.

I work with reverse-proxies (and outgoing proxies) on a daily basis. This is simply a case of following X-Forwarded-For too much.

Typically, when chasing headers such as X-Forwarded-For, you would list your trusted proxies, and it would only chase up to those proxies, and then whatever is left over is your answer. In other words, you want what your outermost trusted reverse proxy saw as the client IP.

From the code:

	// We'll always grab the first IP address in the X-Forwarded-For header
	// list.  We do this because this is always the *origin* IP address, which
	// is the *true* IP of the user.  For more information on this, see the
	// Wikipedia page: https://en.wikipedia.org/wiki/X-Forwarded-For
	ip := net.ParseIP(strings.Split(r.Header.Get("X-Forwarded-For"), ",")[0]).String()

This is unfortunately incorrect behaviour… more precisely, it is prone to giving unexpected results when a user has to go out via a proxy.

Let’s say I’m a server in a datacentre, and to get out to the internet, I must go through a proxy. Assume that my server’s IP is 10.1.2.3. I want to know what IP services on the internet will see me coming from (so I know which IP needs to be added to their whitelist). Assume that my public ip is 123.1.2.3

Currently, I would get this:

http_proxy=http://proxy.example.com:3128 curl http://api.ipify.org
10.1.2.3

In this case, Squid (the outgoing proxy), can inject my IP (10.1.2.3) into the X-Forwarded-For header, because it is forwarding HTTP.

In in HTTPS case, squid just gets a CONNECT request and relays the TCP byte-stream (which is HTTPS) once the connection has been made by the proxy. Thus, Squid cannot inject X-Forwarded-For. This is equivalent to the case of NAT (or where an outgoing proxy has set some different header).

https_proxy=http://proxy.example.com:3128 curl https://api.ipify.org
123.1.2.3

Server-side

Let’s take a look at how we would enable a web-server running behind a reverse-proxy (or multiple reverse-proxies), so that it gets the ‘correct’ IP (‘correct’ meaning 'Internet-routeable IP the connection was seen to come from).

In nginx, we might have the following configuration, assuming that the connections from the reverse proxy to our server was seen to come from 10.10.11.12 or 10.10.11.13 (at the TCP layer).

set_real_ip_from 10.10.11.12/31;
real_ip_header    X-Forwarded-For;
real_ip_recursive off;

If Apache httpd (2.4) is more familiar, we might use mod_remoteip:

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.10.11.12/31

(and changing any LogFormat mentioning %h to %a)

In golang, there are multiple libraries (eg. https://github.com/sebest/xff, https://github.com/stanvit/go-forwarded and others – no idea which if any are good, those were just the first two hits)

If you’re looking for a relevant standard, then check RFC7239 and its Errata:

• https://tools.ietf.org/html/rfc7239
• https://www.rfc-editor.org/errata_search.php?rfc=7239

Thanks for providing api.ipify.org!

Cheers, Cameron