kubernetes: HostPort seemingly not working

@thockin @microadam - Correct me if I’m misunderstanding something, but I think that hostPort just isn’t going to work with CNI based integrations at the moment (I see you’re using Calico).

hostPort currently relies on Docker to configure the port mapping, but in the CNI case Docker doesn’t have the knowledge to do this, since pods are started with net=none.

I think this could be resolved by sending hostPort the way of NodePort, and writing the iptables rules from k8s instead of docker. I don’t think the kube-proxy watches pods at the moment, so probably not a trivial change.

I have the same issue with kube-registry-proxy on 1.4 bare metal with kube-weave cni.

just a question - is this issue being worked on or not? Is it not that important? I think it should be quite important, because we have to use NodePorts as workaround for all external traffic, which is not optimal.

@philips yep. Kubernetes now has support for plugin chaining and the portmap capability, and an upstream port mapping plugin exists which can be chained to provide the port mapping.

@gunjan5 has tested this out with Calico and it works using k8s v1.7.0-alpha.4

FYI CNI PR is merged

Setting NET_PLUGIN=cni to use cni network will cause the hostport not to work. Setting NET_PLUGIN=kubenet to use kubenet, then the hostport works fine.

We’ve also seen this issue with Deis. We use a hostPort with a container called registry-proxy to bypass the Docker --insecure-registry requirement for internal networks. We’ve seen this occur only on CoreOS-specific installations such as with kube-aws. Vagrant (Fedora), GKE (Debian), AWS (Ubuntu) and Minikube (custom ISO) all work without issue.

To reproduce:

• install Workflow v2.7.0 on a kube-aws cluster
• observe registry-proxy isn’t listening on the host’s port 5555 with netstat -tan | grep 5555

A bit of history/debugging is available on both https://github.com/deis/registry/issues/64 and https://github.com/deis/workflow/issues/442

Just to say, I can get this to work if I set hostNetwork: true on the pod, but that then exposes all of the ports in the pod to the host network, where as I only want one specific port (80) to be exposed

@nailgun You need to use hostNetwork in your router pod i think, example with an nginx ingress controller :

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ingress-lb
spec:
  template:
    metadata:
      labels:
        name: nginx-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3
        name: nginx-ingress-lb
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        # use downward API
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend

@gunjan5 oh geez! Don’t I feel like an idiot. That was 100% my problem. Well that’s several hours of my life I won’t get back. Thanks a ton to you and @caseydavenport for the help to get me up and running! Port mapping is now working for me. Great!

@Rastusik the hostNetwork pod can reach other (CNI) pods directly by IP. But it can’t resolve services by their name, because it does not use kube-dns as the resolver.

E.g. the Nginx ingress controller works with hostNetwork.

@linhdsv: I’ve solved issue by using custom Calico CNI configuration to enable portmap — https://github.com/seletskiy/kubernetes-bootstrap

@pichouk you’ll need to add the portmap plugin to your CNI config.

Theres an example using Calico here - should be similar for Weave.

/assign

Solution for Weave CNI is described here

@moserke what is the config file called? It has to end with .conflist extension (not .conf)

@bacongobbler, looks like it’s not just CoreOS.

Just followed kubeadm tutorial to install Kubernetes on single CentOS 7 bare-metal host. Then I’ve deployed Deis Workflow by following this guide. Now when I’m trying to deploy example app using Deis, I’m experiencing the same issue.

I can confirm this problem and solution renders all the Ingress examples useless. Most of those nginx RC and DS examples contain the use of hostPort but no use of hostNetwork. Took me a while to stumble on this issue and figure out how to make it work.

The above is true to v1.4, bare metal