istio: istio-init container exit(status 2) when kernel lacks 'iptables-legacy' modules

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

The current implementation of the iptables detect logic looks like below:

  1. Execute iptables-nft-save/ip6tables-nft-save. If there are more than three rules and iptables-nft-save/ip6tables-nft-save and iptables-nft --version/ip6tables-nft --version do not return errors, use the iptables-nft series binaries (iptables-nft, iptables-nft-save, iptables-nft-restore).
  2. Execute iptables-legacy-save/ip6tables-legacy-save. If there are more than three rules and iptables-legacy-save/ip6tables-legacy-save and iptables-legacy --version/ip6tables-legacy --version do not return errors, use the iptables-legacy series binaries (iptables-legacy, iptables-legacy-save, iptables-legacy-restore).
  3. Execute iptables-save and iptables --version. If there are no errors, use the iptables series binaries.

In my environment, the host does not have iptables-legacy installed, only iptables-nft. According to the logic above, istio-init will erroneously select iptables series binaries(instead of iptables-nft), because the corresponding kernel modules are not installed in my environment, as a result, istio-init encounters error when executing iptables-restore:

image

And here’s the reason: In the istio-init container, steps 1 and 2 get skipped because the Rules Dump check (>3 rules) are not passed. This is expected since the newly created Pod and it has no existing rules in pod network ns. Therefore, step 3 is executed, which uses the iptables commands (iptables, iptables-save, iptables-restore). However, in the istio-init context, the iptables commands correspond to the iptables-legacy commands. Since my host lacks the iptables-legacy kernel modules, this triggers the error: unable to initialize table 'nat'.

I noticed that the current implementation for selecting the iptables binary was submitted by @bleggett in this Pull Request (https://github.com/istio/istio/pull/49703). The initial discussion of this issue took place in this Issue (https://github.com/istio/istio/issues/49279), where @bleggett described his scenario: in the Ambient mode, the CNI Daemon misplaces rules into the iptables-legacy chains when configuring traffic rules in the node network namespace, while k8s adds its rules to the iptables-nft chains:

Seeing an issue where the single ambient iptables rule istio-cni injects into the host-level iptables POSTROUTING chain (for host healthchecks) ends up in the iptables-legacy chain, even though all the other rules (kubelet, CNI, etc) are in the iptables-nft chain.

@bleggett pinpointed the root cause in the old inspection logic:

This is because our iptables (legacy OR nft) selection logic only checks the container-local binary default, which may or may not actually match what was used or is present on the host node.

He resolved the bug in this scenario by introducing a new detection logic. However, the current solution still faces issues when the host does not support iptables-legacy (as described earlier). Essentially, when neither iptables-legacy nor iptables-nft meet the criteria, it falls back to the iptables command. In Istio 1.22’s four use-cases, the iptables command could imply different implementations:

Usage Using iptables Binaries From Networking context to select correct binary against ‘iptables’ binary corresponding to
from CNI plugin (sidecar) host $PATH pod netns context depends on host environment, iptables-legacy or iptables-nft are both possible
from init container (sidecar) init container $PATH pod netns context iptables-legacy implementation
from CNI agent (ambient) CNI container $PATH pod netns context iptables-legacy implementation
from CNI agent (ambient) CNI container $PATH host netns context iptables-legacy implementation

As you can see, as long as a binary inside the container $PATH executes in the Pod network ns, an error will arise if the host does not have iptables-legacy kernel modules installed. Based on my understanding, the current fallback to iptables aims to adhere to the environment settings (assuming iptables pointed to the default implementation), but this implementation may deviate from the expected behavior due to the different implications of the iptables binary under various conditions. I believe we need a different approach to resolve this issue. Here is a proposed solution inspired by @bleggett 's current implementation.

First, we need to check what iptables implementations the current kernel supports. This could be achieved by inspecting kernel modules. The /proc/modules file lists all the kernel modules loaded by the current kernel. This file remains consistent across the host and all containers, indicating the kernel modules loaded by the host’s OS.

After determining the supported implementations, we should confirm the binaries corresponding to each supported implementation in the current context. Then, we use these binaries to retrieve the Rule Dump for each implementation. Here’s how to determine the binaries for each implementation:

  • For legacy, try executing iptables-legacy --version. If no error occurs, use iptables-legacy. If it fails, try executing iptables --version. If the output does not contains nf_tables, use iptables. Otherwise, the current environment does not support legacy.
  • For nft, try executing iptables-nft --version. If no error occurs, use iptables-nft. If it fails, try executing iptables --version. If the output contains nf_tables, use iptables. Otherwise, the current environment does not support nft.

By this point, we should have enough information to determine which binary set to use. (the final judgment process is mainly inspired by @bleggett 's approach. Thanks again to him.)

  • If both legacy and nft are supported by the current kernel, use the implementation with more than 3 rules for the Rule Dump. (If both have more than 3 rules, may be should report an error or support configuration to specify which one to prioritize for this case?)
  • If only one implementation is supported, use the currently supported implementation directly.

I will submit a PR to trying improve the iptables detect implementation, feedback and suggestions are welcome!

Version

istio 1.22.2(both controlplane and dataplane)

Additional Information

none

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 17 (11 by maintainers)

Commits related to this issue

Most upvoted comments

I would prefer you do the following to fix this specific issue:

  1. Change the existing code to check the rulecounts for both iptables-legacy and iptables-nft, if both binaries are present.
  2. If both have rulecounts >3 (e.g. both are in use) or neither have rulecounts > 3 -> prefer nft.
  3. A unit test to assert the above behavior.

Then we just need a docs PR with the one-sentence clarification I mentioned above and you should have a tested fix for this issue without needing to check /proc/modules. Checking for kmods (as I mentioned) is not the best detection heuristic to use here, as it will not produce consistent results in all the scenarios the current unified logic needs to operate in.

Following that, if you would like to propose checking /proc/modules then I agree with @zhlsunshine that it would be good to write that up and present it to the community.

The reason why I think a new implementation is needed is that I understand that the current implementation does not work as expected in the istio-init scenario for the following reasons:

  • iptables-nft/iptables-legacy MUST EXIST in istio-init image
  • rule-dump by iptables-legacy-save/iptables-nft-save MUST 0 in pod network ns

As a result, the ‘tie’ situation is a inevitable in istio-init, so the bottom-line logic(pick iptable) will definitely be executed, but, whatever we pick unconditionally for this situation(legacy/nft), there will always be situations that cannot be handled expectedly:

  • if we pick iptables-legacy, then if the host only has nft, an error will occur.
  • if we pick iptables-nft, then if the host only has legacy, an error will occur.

To resolve this, i prefer the way blow:

Determine what implementation(legacy/nft) has been installed in the host kernel (only installed implementation should being considered to use, check /proc/modules is the easiest way to do so) Determine which binary each implementaion corresponding to (we need to get version and dump rules by executing the binary) If both installed, use corresponding binary to dump their rules, and choose one which have more than 3 rules. (we prefer to choose implementation already have rules). else, use the installed one.

I agree with this:

Checking for kmods (as I mentioned) is not the best detection heuristic to use here, as it will not produce consistent results in all the scenarios the current unified logic needs to operate in.

but maybe we can have a better way to achieve this.

Hi @bleggett and @xocodergit, it really makes sense to have a unified iptables detect instructions for all scenarios.(istio-init, CNI agent, CNI plugin), rather than how to fix this problem easily.

Also thanks to @zhlsunshine , I think we can discuss this in two parts:

1. A unified iptables detect instructions

yes, we definitely need a unified instructions to detect which iptables binary should be used in all scenarios.(istio-init, CNI agent, CNI plugin), and i have described my opinion on this:

  • Determine what implementation(legacy/nft) has been installed in the host kernel (only installed implementation should being considered to use, check /proc/modules is the easiest way to do so)
  • Determine which binary each implementaion corresponding to (we need to get version and dump rules by executing the binary)
  • If both installed, use corresponding binary to dump their rules, and choose one which have more than 3 rules. (we prefer to choose implementation already have rules). else, use the installed one.

I agree with your concern about the OS dependency of checking /proc/modules, but I think we can discuss this separately in the part 2.

2. How to detect kernel modules

well, maybe check /proc/modules would introduce some OS level dependencies(although i think it seems ok because iptables it self also depending on the OS). we can check it in other ways if there is a better one.

Hi all, some ideas about above discussion: Firstly, there should be unified instructions for CNI plugin (sidecar), init container (sidecar) and CNI agent (ambient), the unified instructions means that they may need to follow the same logic to adjust the iptables rules. Secondly, there are considerations on both the installation of the appropriate series iptables binaries and rulecount of corresponding iptables binaries. And this second point may causes the differences for CNI plugin (sidecar), init container (sidecar) and CNI agent (ambient), the unified instructions At last, make a reasonable and executable logics for choosing the proper iptable binaries. So I suggest that @xocodergit can write a proposal in the community for this. BTW, I believe that @bleggett and I are disagree to check /proc/modules for inspecting kernel modules due to it would import the OS level dependency of Istio. It’s unacceptable. Please correct me if any incorrect.

In the istio-init container, steps 1 and 2 get skipped because the Rules Dump check (>3 rules) are not passed.

The rules count check is meant to be used as a ‘tiebreaker’ when both iptables-nft and iptables-legacy are present.

And we can’t just prefer nft in that case because even tho both are available, the host may have preferred to inject rules into legacy - and we want to respect that if we can.

But I think we just don’t correctly handle the case where both are present and neither have any rules.

If this is the problem, the simpler fix is probably to

  • Identify if both nft and legacy are present
  • If both present, check the rulecount for both.
  • If neither have any rules, unconditionally pick nft

That should fix your scenario without requiring kmod introspection.