podman: PODMAN not able to tunnel thru Proxy in an Enterprise Proxy Environment

We solved this in a corporate environment on MacOS that requires a secure proxy server to access all external resources. We use the podman managed machine via podman machine. All configuration takes place on the VM. We have been using this successfully now for almost two months now.

Requires: Podman 3.4.0 at a minimum on MacOS host and managed machine.

Configure Proxy Support Globally for All Systemd Services

sudo mkdir /etc/systemd/system.conf.d

Place in /etc/systemd/system.conf.d/10-default-env.conf:

[Manager]
DefaultEnvironment="HTTP_PROXY=<your-proxy-URL>"
DefaultEnvironment="HTTPS_PROXY=<your-proxy-URL>"
DefaultEnvironment="NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

DefaultEnvironment="http_proxy=<your-proxy-URL>"
DefaultEnvironment="https_proxy=<your-proxy-URL>"
DefaultEnvironment="no_proxy=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

Allow Interactive Login Sessions to VM to Use Proxy

Place in /etc/profile.d/proxy.sh:

PROXY_URL=<your-proxy-URL>

export http_proxy="$PROXY_URL"
export https_proxy="$PROXY_URL"
export ftp_proxy="$PROXY_URL"
export no_proxy=".<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

export HTTP_PROXY="$PROXY_URL"
export HTTPS_PROXY="$PROXY_URL"
export FTP_PROXY="$PROXY_URL"
export NO_PROXY=".<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1"

Configure Trusted Certificates

• Place each CA certificate into a separate PEM files in /etc/pki/ca-trust/source/anchors/ then update the certificate authority trust, for example:

install -o root -g root -m 0644 *.pem *.crt /etc/pki/ca-trust/source/anchors
sudo update-ca-trust

Restart Managed Machine

Unfortunately in order to have the Systemd configuration for proxy configuration to take effect you will have to restart the podman managed machine from the host:

podman machine stop
podman machine start

Works on both Catalina and Big Sur.

Some thoughts on how podman can discover the proxy settings:

1. Ingest the system proxy settings on Mac. You can get the settings by running: scutil --proxy or in Objective-C using the SystemConfiguration framework (I suspect the same is available in Go): https://developer.apple.com/documentation/systemconfiguration/1517088-scdynamicstorecopyproxies
2. Ingest proxy settings from ENV.
3. Allow the user to override by adding an argument to the podman commands or configuring a setting somewhere.

Also, PLEASE support no_proxy stuff. Most companies that use proxies need to access stuff that’s on the internal network and outside the network, but only outside requires proxy.

Ideally the HTTP_PROXY/HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE env vars should be picked up by the podman machine start command and automatically be passed into the ignition file. This is similar to how minikube operates (though I am unsure if it uses ignition):

$ minikube start
😄  minikube v1.23.0 on Darwin 11.6.1
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=4, Memory=16384MB, Disk=20000MB) ...
🎉  minikube 1.24.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.24.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🌐  Found network options:
    ▪ HTTP_PROXY=http://proxy.example.com:8080
    ▪ HTTPS_PROXY=http://proxy.example.com:8080
    ▪ NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
    ▪ http_proxy=http://proxy.example.com:8080
    ▪ https_proxy=http://proxy.example.com:8080
    ▪ no_proxy=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.22.1 on Docker 20.10.8 ...
    ▪ env HTTP_PROXY=http://proxy.example.com:8080
    ▪ env HTTPS_PROXY=http://proxy.example.com:8080
    ▪ env NO_PROXY=.<set-of-host-domain-exclusion(s)>,localhost,127.0.0.1,0.0.0.0,::1
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

I think you can pass the proxy variables if you set up a custom ignition file, though it would be nice if podman auto-detected proxy settings and passed them through.

Some enterprise proxies also do SSL inspection, which requires loading a custom trusted CA into a user’s workstation trust store. I believe tools commonly use the SSL_CERT_FILE env var to do this, similar to HTTP_PROXY/HTTPS_PROXY/etc variables to detect the proxy.

SSL_CERT_FILE is standard in openssl: https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_default_verify_paths.html

I’m assuming that @cpolizzi 's solution was to modify those files inside the podman VM, not the mac. (Since my Mac doesn’t have that first directory, because systemd is not installed.)

I note that by default my proxy env vars were already present in the virtual machine, but all in one line, which I’m guessing is fine.

proxy.sh did not exist. I added that. It didn’t resolve my problem.

@gclawes Can you point me to any readme that can help with the steps and syntax for creating custom ignition file?