I’d be happy to work on this

Propositions

At a high level, make hubble cli more “intuitive” by behaving like kubectl with respect to namespace and pod selector flags.

1. cli: --namespace along with --pod

hubble observe --from-namespace cilium --from-pod hubble-relay

Doesn’t work at the moment, we get:

invalid argument "hubble-relay" for "--from-pod" flag: filters --from-namespace and --from-pod cannot be combined

proposition

Translate this into a [source_pod:"cilium/hubble-relay"] FlowFilter. This is analog to

kubectl --namespace cilium --pod hubble-relay

Fail with an error when we get an inconsistency between the namespace arg and pod arg, e.g. --from-namespace cilium --from-pod default/hubble-relay.

Also apply for destinations and services.

2. cli: handle the namespace label specifically

hubble observe --from-label io.kubernetes.pod.namespace=cilium --from-pod hubble-relay

This is translated as [source_pod:"hubble-relay" source_label:"io.kubernetes.pod.namespace=cilium"] which won’t ever yield any flow as the source_pod part is equivalent to default/hubble-relay.

proposition

translate this into either:

• [source_pod:"cilium/hubble-relay"]
• [source_pod:"cilium/hubble-relay", source_label:"io.kubernetes.pod.namespace=cilium"]

Additionally, lookout for inconsistencies between --from-label io.kubernetes.pod.namespace=cilium, --from-namespace, and --from-pod.

Also apply for destinations and services.

3. cli: namespace consistency between --from-pod and --from-service

hubble observe --from-pod kube-system/hubble-relay --from-service default/hubble-relay

currently translate into [source_pod:"kube-system/hubble-relay" source_service:"default/hubble-relay"] which won’t ever yield any flow.

proposition

Fail with an error when we get an inconsistency between the service arg and pod arg

Also apply for destinations.

4. FlowFilter

unlike the other proposition, this one will require some changes on the Hubble server side.

Force clients to always use the form namespace/pod for source_pod, destination_pod. This mean that we’ll add the default/ part on some queries, like hubble observe --from-pod hubble-relay. The form without slash (implicit default/) can be deprecated and kept for a while for backward compatibility’s sake.

An empty name will act as wildcard. We already use this:

hubble observe --from-namespace cilium

Is currently translated into [source_pod:"cilium/"] which mean “every pod under the cilium namespace”.

With that scheme it is easy to design a --all-namespace flag akin to kubectl as @glibsm suggested:

hubble observe --from-all-namespace --from-pod hubble-relay

Will be translated into [source_pod:"/hubble-relay"]

To query all pods in all namespaces, we would have [source_pod:"/"]. As for the implicit default/ we can keep the current FlowFilter [] compatible for a while.

Also apply for destinations and services.

I started looking into this and started an MVP for option 4. So far I think everything seems to mostly work as expected. At the moment I believe the behavior is completely backwards compatible, because the new behavior is only in effect when using the pod and namespace filters together, which was previously not allowed via CLI.

Branches:

• Cilium: https://github.com/cilium/cilium/tree/pr/chancez/flow_filter_namespace
• Hubble CLI: https://github.com/cilium/hubble/tree/pr/chancez/remove_namespace_pod_filter_conflict

The main thing I think is worth noting is that if you only specify the pod filters, then it uses the existing behavior, and if you specify namespace filters it will ignore the namespace portion of the pod filters. Additionally, when you use the namespace flags with the Hubble CLI, it sets both pod filters, and namespace filters. Combined, this means that the server uses the new behavior only when both pod and namespace filters are specified together. The server never supported namespace flow filters prior to this change, so the new filters will be ignored on old versions of Cilium, meaning the old behavior will be retained. If you try using both --pod and --namespace together on an older version of Hubble, then it will basically ignore the --namespace and act as --pod default/<pod_name>.

As a future step, the client and server could negotiate to determine which behavior to use, or the server could simply deprecate the use of namespace in pod filters, and eventually remove it, leaving only the new behavior.

Some example raw filter output:

$ ./hubble observe --to-namespace kube-system --to-pod hubble-relay --print-raw-filters
allowlist:
- '{"destination_pod":["kube-system/","hubble-relay"],"destination_namespace":["kube-system"]}'
./hubble observe --to-namespace kube-system --print-raw-filters                                                                                                                                          pr/chancez/remove_namespace_pod_filter_conflict ✚
allowlist:
- '{"destination_pod":["kube-system/"],"destination_namespace":["kube-system"]}'
$ ./hubble observe --to-pod hubble-relay --print-raw-filters
allowlist:
- '{"destination_pod":["hubble-relay"]}'
$ ./hubble observe --to-pod kube-system/hubble-relay --print-raw-filters
allowlist:
- '{"destination_pod":["kube-system/hubble-relay"]}'

And agent logs for each with some extra printlns indicating “how” it’s filtering, using the some commands as above:

2023-01-14T00:59:06.001687308Z filter by podName [hubble-relay]
2023-01-14T00:59:06.001703933Z filter by namespace [kube-system]
2023-01-14T00:59:06.002010102Z level=debug msg="GetFlows finished" blacklist="{}" buffer_size=4095 number_of_flows=20 subsys=hubble took="313.211µs" whitelist="{destination_pod:\"kube-system/\" destination_pod:\"hubble-relay\" destination_namespace:\"kube-system\"}"
2023-01-14T00:59:15.442809199Z filter by namespace [kube-system]
2023-01-14T00:59:15.442984076Z level=debug msg="GetFlows finished" blacklist="{}" buffer_size=4095 number_of_flows=20 subsys=hubble took="159.501µs" whitelist="{destination_pod:\"kube-system/\" destination_namespace:\"kube-system\"}"
2023-01-14T00:59:22.823465864Z filter by NamespacedName: [{default hubble-relay}]
2023-01-14T00:59:22.824444413Z level=debug msg="GetFlows finished" blacklist="{}" buffer_size=4095 number_of_flows=0 subsys=hubble took=1.131509ms whitelist="{destination_pod:\"hubble-relay\"}"
2023-01-14T00:59:44.588980112Z filter by NamespacedName: [{kube-system hubble-relay}]
2023-01-14T00:59:44.589282490Z level=debug msg="GetFlows finished" blacklist="{}" buffer_size=4095 number_of_flows=20 subsys=hubble took="216.043µs" whitelist="{destination_pod:\"kube-system/hubble-relay\"}"

@kAworu All of this makes a lot of sense to me. I especially like that you came up with a backward-compatible solution for --all-namespaces in 4!

Minor comment: I strongly believe that hubble observe --from-label io.kubernetes.pod.namespace=cilium --from-pod hubble-relay should be translated to [source_pod:"cilium/hubble-relay", source_label:"io.kubernetes.pod.namespace=cilium"], i.e. the label filter should be preserved it the user specified it to avoid any surprises.