proxy.py: [Ubuntu] Cannot use TLS interception

Describe the bug Iam not able to use the TLS interception feature, as provided in the readme section. Even the basic

To Reproduce Steps to reproduce the behavior:

  1. [Host machine] Install proxy==2.1.2 pip install proxy.py
  2. [Host machine] create ssl files
export CA_KEY_FILE_PATH=ca-key.pem
export CA_CERT_FILE_PATH=ca-cert.pem
export CA_SIGNING_KEY_FILE_PATH=ca-signing-key.pem
python -m proxy.common.pki gen_private_key --private-key-path $CA_KEY_FILE_PATH
python -m proxy.common.pki remove_passphrase --private-key-path $CA_KEY_FILE_PATH
python -m proxy.common.pki gen_public_key --private-key-path $CA_KEY_FILE_PATH --public-key-path $CA_CERT_FILE_PATH
python -m proxy.common.pki gen_private_key --private-key-path $CA_SIGNING_KEY_FILE_PATH
python -m proxy.common.pki remove_passphrase --private-key-path
  1. [Host machine] Run proxy.py
$ proxy --plugins proxy.plugin.CacheResponsesPlugin --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem --host 0.0.0.0 --log-level d
2020-03-02 11:11:41,970 - pid:20523 [I] load_plugins:525 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2020-03-02 11:11:41,971 - pid:20523 [I] load_plugins:525 - Loaded plugin proxy.plugin.CacheResponsesPlugin
2020-03-02 11:11:41,972 - pid:20523 [I] listen:63 - Listening on 0.0.0.0:8899
2020-03-02 11:11:41,975 - pid:20523 [D] start_workers:81 - Started acceptor#0 process 20525
2020-03-02 11:11:41,976 - pid:20523 [I] start_workers:84 - Started 1 workers
  1. [Client machine] Send request from client system
$ curl --proxy http://host_ip:8899 --cacert ~/work/varnish_docker_virtual/squid_docker/upstream_haproxy_certs/haproxy-ca-cert.pem https://httpbin.org/ip -vvv
*   Trying host_ip...
* TCP_NODELAY set
* Connected to host_ip (host_ip) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /home/jitesh/work/varnish_docker_virtual/squid_docker/upstream_haproxy_certs/haproxy-ca-cert.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to httpbin.org:443 
* stopped the pause stream!
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to httpbin.org:443 
  1. [Host machine] Check error log at the host
2020-03-02 11:11:47,320 - pid:20525 [D] initialize:145 - Handling connection <socket.socket fd=9, family=AddressFamily.AF_INET, type=2049, proto=0, laddr=('host_ip', 8899), raddr=('gateway_ip/client_ip', 19673)>
2020-03-02 11:11:47,322 - pid:20525 [D] handle_readables:302 - Client is ready for reads, reading
2020-03-02 11:11:47,322 - pid:20525 [D] recv:65 - received 114 bytes from client
2020-03-02 11:11:47,324 - pid:20525 [D] connect_upstream:420 - Connecting to upstream httpbin.org:443
2020-03-02 11:11:47,363 - pid:20525 [D] connect_upstream:425 - Connected to upstream httpbin.org:443
2020-03-02 11:11:47,402 - pid:20525 [D] generate_upstream_certificate:362 - Generating certificates /home/ubuntu/.proxy.py/certificates/httpbin.org.pem
2020-03-02 11:11:47,422 - pid:20525 [D] flush:91 - flushed 39 bytes to client
2020-03-02 11:11:47,423 - pid:20525 [E] on_request_complete:278 - OSError when wrapping client
2020-03-02 11:11:47,423 - pid:20525 [I] access_log:332 - gateway_ip/client_ip:19673 - CONNECT httpbin.org:443 - 0 bytes - 105.42 ms
2020-03-02 11:11:47,424 - pid:20525 [I] close:48 - Cached response at /tmp/httpbin.org-89311d71dce24450b200947c9ef8ac1f.txt
2020-03-02 11:11:47,424 - pid:20525 [D] on_client_connection_close:189 - Closed server connection, has buffer False
2020-03-02 11:11:47,424 - pid:20525 [D] shutdown:217 - Closing client connection <socket.socket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('host_ip', 8899), raddr=('gateway_ip/client_ip', 19673)> at address ('gateway_ip/client_ip', 19673) has buffer False
2020-03-02 11:11:47,425 - pid:20525 [D] shutdown:225 - Client connection shutdown successful
2020-03-02 11:11:47,425 - pid:20525 [D] shutdown:230 - Client connection closed

Expected behavior The expected outcome of https://github.com/abhinavsingh/proxy.py#tls-interception

Version information

  • OS: Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-88-generic x86_64)
  • Curl: curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
  • proxy.py Version 2.1.2

Additional context I am able to get response of http://httpbin.org/ip without any issues. So the problem only occurs at the HTTPS section. Do let me know if there are any procedures that I missed.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 39 (17 by maintainers)

Most upvoted comments

@abhinavsingh can you please share your certificates for testing purposes please? Or maybe not for testing only… Many people here would have an opportunity to try the TLS interception feature 😃

After upgrading openssl version to OpenSSL 1.1.1d 10 Sep 2019 working on other destro’s. tested in ubuntu 16 and 18.04

Now I am here also with the same problem Just +1 for waiters 😃

superman@debian:~/proxy.py$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

Hi, I ve just try on, Debian, Python 3.7.3, Master Branch e827eaacfeb9359f6d522376aba70e7d246d40c5

  • with root -> RAS -> it’s working well.
  • with “user” -> RAS -> working well too…

superman@debian:~/proxy.py$ curl --proxy http://localhost:8899 --cacert ca-cert.pem https://httpbin.org/ip -vvv
* Expire in 0 ms for 6 (transfer 0x562fa295be80)
* Expire in 1 ms for 1 (transfer 0x562fa295be80)
* Expire in 0 ms for 1 (transfer 0x562fa295be80)
* Expire in 2 ms for 1 (transfer 0x562fa295be80)
* Expire in 0 ms for 1 (transfer 0x562fa295be80)
* Expire in 1 ms for 1 (transfer 0x562fa295be80)
* Expire in 4 ms for 1 (transfer 0x562fa295be80)
* Expire in 1 ms for 1 (transfer 0x562fa295be80)
* Expire in 1 ms for 1 (transfer 0x562fa295be80)
* Expire in 2 ms for 1 (transfer 0x562fa295be80)
*   Trying ::1...
* TCP_NODELAY set
* Expire in 149997 ms for 3 (transfer 0x562fa295be80)
* Expire in 200 ms for 4 (transfer 0x562fa295be80)
* Connected to localhost (::1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.64.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: ca-cert.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=httpbin.org
*  start date: Mar 26 17:14:05 2020 GMT
*  expire date: Mar 26 17:14:05 2021 GMT
*  common name: httpbin.org (matched)
*  issuer: CN=example.com
*  SSL certificate verify ok.
> GET /ip HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< Date: Thu, 26 Mar 2020 17:15:48 GMT
< Content-Type: application/json
< Content-Length: 32
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< 
{
  "origin": "83.194.64.161"
}
* Connection #0 to host localhost left intact

I double check, I ve deleted the .proxy/certificates/*.pem files, and it’s well generated.

@ja8zyjits @Benouare Hey folks, how are you all? It’s been tough times for everyone and at my end with all travel involved it became worse. Good news, I am finally at a stable place and will soon dig into pending items. Thank you for your patience 😃