kubernetes: Network policies are blocking allowed traffic

What happened: I have set up a network policy that’s blocking traffic that it shouldn’t What you expected to happen: Network policies are supposed to block everything that is not allowed. In my case I am allowing egress traffic to everywhere, except the cidr block 192.168.0.0/16. I can confirm that all traffic to that block is indeed restricted, and mostly anything else is allowed, except for some specific ips (might be affecting the entire 10.0.0.0/8 block as well even though it’s not declared as an “except” block, hence should be allowed).

How to reproduce it (as minimally and precisely as possible):

1. install eksctl to launch a cluster in EKS (this is where I am testing) (https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html)
2. deploy a simple cluster:

# test-eksctl.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: basic-cluster
  region: us-west-2

nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 3
    volumeSize: 80

eksctl create cluster -f test-eksctl.yaml

1. install calico network plugin: kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.5/config/v1.7/calico.yaml

2. create a new namespace: kubectl create ns netpol-test

3. deploy a test pod in this namespace:

# netutils.yaml
apiVersion: v1
kind: Pod
metadata:
  name: netutils-sd1
spec:
  containers:
    - name: my-pod
      image: amouat/network-utils
      command:
      - /bin/sh
      - -c
      - |
        tail -f /dev/null

then test basic networking

# Deploy the pod
kubectl -n netpol-test apply -f netutils.yaml
 # (This is to grab the cluster ip for the internal kubernetes service, in my case is `10.100.0.1`) 
kubectl get svc -A
# Get a shell in the pod
kubectl -n netpol-test  exec -it  netutils-sd1 bash
# This commands run inside the container
curl -k -I https://10.100.0.1:443
# Which should return 403 forbidden. This is expected, and it purpose is to demonstrate that we can at least hit this ip (we don't care about authentication here, only traffic)

1. deploy the following network policy to that namespace:

# test-netpol.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: test-netpol
spec:
  egress:
  - to:
    # This should allow communication between pods within this namespace
    - podSelector: {}
    - ipBlock:
        # We need this because we need pods to have connectivity to outside world, but not to pods living in the same cluster in other namespaces
        cidr: 0.0.0.0/0
       # Because all my pods for other namespace will live in the 192.169.0.0/16 block, 
       # I don't want pods from this namespace to reach other namespaces, 
       # this is what I found the easiest way to block connectivity across namespaces.
        except:
        - 192.168.0.0/16
  # This is to allow privileged namespaces like prometheus/kube-system to reach pods in this namespace.
  ingress:
  - from:
    - ipBlock:
        cidr: 0.0.0.0/0
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

kubectl -n netpol-test apply -f test-netpol.yaml Try to hit the ip again:

kubectl -n netpol-test  exec -it  netutils-sd1 bash
# This commands run inside the container
curl -k -I https://10.100.0.1:443
# This till it will hang until it times out. This is not expected, as we are not blocking this private ip space anywhere.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: EKS 1.17
• OS (e.g: cat /etc/os-release): macos
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):

• aws-cni
• calico network plugin

• Others: