consul-k8s: connect-init command breaks workaround for supporting multiple ports in sidecar proxy

As of consul-helm chart v0.32.0 which runs consul 1.10 and consul-k8s 0.26.0 we can no longer use our workaround to support multiple ports in sidecar proxy as suggested in open issue https://github.com/hashicorp/consul/issues/5388#issuecomment-854061709. Our workaround was functional as of Consul 1.9.4 (consul-helm 0.31.1).

The consul upgrade to 1.10 is integrated with Kubernetes objects. Kubernetes services are now part of the standard deployment of any pod that is ran on the mesh.

Each pod is now tied to a Kubernetes service and only one Kubernetes service is expected since a pod can only expose one port due to Consul’s limitation to supporting one port per pod. In consul-k8s, the new connect-init function makes that assumption and breaks the current workaround we have in place to support multiple ports.

It is made clear in the code that no more than two services are expected (https://github.com/hashicorp/consul-k8s/blob/v0.26.0/subcommand/connect-init/command.go#L166). Since it doesn’t log the services that are returned, I presume that it’s returning all the services associated with that pod.

filter := fmt.Sprintf("Meta[%q] == %q and Meta[%q] == %q", connectinject.MetaKeyPodName, c.flagPodName, connectinject.MetaKeyKubeNS, c.flagPodNamespace) (https://github.com/hashicorp/consul-k8s/blob/v0.26.0/subcommand/connect-init/command.go#L159)

I think adding the service name to the filter will bring back exactly the Kubernetes services tied to the service name.

Our workaround is tied to manually adding sidecars to the main application container. Here is an example:

POD A —>service 1 (port 9888) with associated Kubernetes Service 1 —>service 2 (port 9889) with associated Kubernetes Service 2

We have POD A which exposes two ports each representing a different service on POD A. We created the corresponding Kubernetes service for each of the services exposed in POD A.

In this example I think the filter for Pod A for service 1 is returning Kubernetes Services 1 and 2 since the filter is only applied to the pod/namespace and not pod/namespace/service.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 8
Comments: 22 (11 by maintainers)

Most upvoted comments

Do you have an individual connect-init init container per Kubernetes headless service that points to the same Pod? So you only need the one service-name flag to the connect-init container?

We could modify the filter based on the flag, something like:

filter := fmt.Sprintf("Meta[%q] == %q and Meta[%q] == %q", connectinject.MetaKeyPodName, c.flagPodName, connectinject.MetaKeyKubeNS, c.flagPodNamespace)
if c.flagServiceName != "" {
  filter = filter + fmt.Sprintf(" and Meta[%q] == %q", connectinject.MetaKeyKubeServiceName, c.flagServiceName)
}

ndhanushkodi on Jul 29, 2021

Closing as our initial release on multi-port was merged in https://github.com/hashicorp/consul-k8s/pull/1012, docs will shortly follow this or next week. Stay tuned!

david-yu on Feb 23, 2022

Hi @fouadsemaan you can also pull that image by using

docker pull hashicorpdev/consul-k8s:2d2b303

david-yu on Jul 30, 2021

Hi @fouadsemaan I still have a few unit test tweaks to make, but in the meanwhile you can pull the image hashicorpdev/consul-k8s@sha256:0de6573eacb0aa1782f14044d13cddcdb8ddbf258d01a103a8b95869bdbf0257 and run the step @david-yu mentioned above to grab the binary from it. The new flag is called -k8s-service-name, rather than -service-name because the latter flag already exists and is used for setting a Consul service name that’s different from the K8s service name. You can use the -k8s-service-name flag as you described in this comment. Can you let us know if this works for you as expected in your setup?

ndhanushkodi on Jul 30, 2021

The dev container image should include the binary which you can copy to your local filesystem like so:

docker cp <containerId>:/bin/consul-k8s /host/path/target

david-yu on Jul 30, 2021

@david-yu

As I mentioned to @ndhanushkodi and @ishustava we manually setup the sidecars. We manually replicate what the connect-inject does for multiple ports. We use the consul helm chart for everything else (servers, agents, controllers, service intentions et al.)

fouadsemaan on Jul 29, 2021