mitmproxy: Requesting complete URL over https proxy results in "Invalid request scheme: https"

Steps to reproduce the problem:

Set up an HTTPS mitmproxy
Run the following (minimized) code against it:

import httplib
conn = httplib.HTTPSConnection('<your proxy IP>',<your proxy port>)
conn.set_tunnel('www.google.com',443)
conn.request('GET','https://www.google.com:443/')
resp = conn.getresponse()
print resp.read()

What is the expected behavior?

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

What went wrong?

<html>
            <head>
                <title>400 Bad Request</title>
            </head>
            <body>HttpException('Invalid request scheme: https',)</body>
        </html>

Any other comments?

I believe it’s caused by a combination of https://github.com/mitmproxy/mitmproxy/blob/master/libmproxy/protocol/http.py#L544-L545 and https://github.com/mitmproxy/mitmproxy/blob/master/libmproxy/protocol/http.py#L502-L503. Indeed, if you modify my GET request above to ask for just / instead of https://www.google.com:443/, everything works fine.

I’m not sure if it’s sensible to support absolute-form requests over https tunnels, but I’m getting this behavior in real code (boto for python AWS) so I don’t think it’s very a contrived situation.

mitmproxy version: 0.14.0

About this issue

Original URL
State: closed
Created 9 years ago
Reactions: 7
Comments: 28 (9 by maintainers)

Commits related to this issue

First attempt to fix #848 by adding a option. — committed to MrNbaYoh/mitmproxy by MrNbaYoh 4 years ago
First step to fix #848 by adding a relax_http_form_validation option. — committed to MrNbaYoh/mitmproxy by MrNbaYoh 4 years ago
Merge pull request #3996 from MrNbaYoh/master First step to fix #848 by adding a relax_http_form_validation option. — committed to mitmproxy/mitmproxy by mhils 4 years ago
Began implementing JS API client using the axios http library. It is nearly functional and listing vehicles, it just requires a little fine-tuning. I'm currently trying to configure it to proxy throug... — committed to jeremy21212121/evo-re by jeremy21212121 3 years ago
warn about nonfunctioning options, permanently remove unused ones. If you are affected by this change, please do reach out by filing an issue. — committed to mitmproxy/mitmproxy by mhils 3 years ago

Most upvoted comments

--relax-http-form-validation whats the new version of this? This param doesnot work in the newest binary

abishekvashok on Jul 11, 2018

If someone needs a quick/hacky fix, change mitmproxy/net/http/http1/read.py

    try:
        import logging
        logging.error(line)
        method, path, http_version = line.split()

        import urllib.parse
        logging.error(urllib.parse.urlparse(path))

        if path == b"*" or path.startswith(b"/"):
            form = "relative"
            scheme, host, port = None, None, None
        elif method == b"CONNECT":
            form = "authority"
            host, port = _parse_authority_form(path)
            scheme, path = None, None
        else:
            form = "absolute" . # <<<<< Change this to `relative`
            scheme, host, port, path = url.parse(path)

        _check_http_version(http_version)

IMG_20200309_122625_exported_stabilized_2057254493992227421

drorwolmer on Feb 2, 2021

There are two things I’d love to see:

First, always accept requests in absolute-form going to the same host. For example, after CONNECT example.com, we should accept GET http://example.com/foo, but not GET http://example.org/foo. This would for example fix this issue.
Second, as you suggested, add a --relax-http-form-validation switch that disables the form enforcement.

mhils on Mar 3, 2017

@jeremy21212121: Corrent, and thanks for pointing out the circular link, I’ve just fixed it.

If you can’t wait and don’t want to build from source, we also have prebuilt development snapshots at https://mitmproxy.org/downloads/#branches/master/.

mhils on Jan 4, 2021

Same issue when requesting in emacs here, using mitmproxy 4.0.1.

QiBaobin on Jun 14, 2018