istio: istio-init container exit(status 2) when kernel lacks 'iptables-legacy' modules
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
The current implementation of the iptables detect logic looks like below:
- Execute
iptables-nft-save/ip6tables-nft-save. If there are more than three rules andiptables-nft-save/ip6tables-nft-saveandiptables-nft --version/ip6tables-nft --versiondo not return errors, use theiptables-nftseries binaries (iptables-nft,iptables-nft-save,iptables-nft-restore). - Execute
iptables-legacy-save/ip6tables-legacy-save. If there are more than three rules andiptables-legacy-save/ip6tables-legacy-saveandiptables-legacy --version/ip6tables-legacy --versiondo not return errors, use theiptables-legacyseries binaries (iptables-legacy,iptables-legacy-save,iptables-legacy-restore). - Execute
iptables-saveandiptables --version. If there are no errors, use theiptablesseries binaries.
In my environment, the host does not have iptables-legacy installed, only iptables-nft. According to the logic above, istio-init will erroneously select iptables series binaries(instead of iptables-nft), because the corresponding kernel modules are not installed in my environment, as a result, istio-init encounters error when executing iptables-restore:
And here’s the reason:
In the istio-init container, steps 1 and 2 get skipped because the Rules Dump check (>3 rules) are not passed. This is expected since the newly created Pod and it has no existing rules in pod network ns. Therefore, step 3 is executed, which uses the iptables commands (iptables, iptables-save, iptables-restore). However, in the istio-init context, the iptables commands correspond to the iptables-legacy commands. Since my host lacks the iptables-legacy kernel modules, this triggers the error: unable to initialize table 'nat'.
I noticed that the current implementation for selecting the iptables binary was submitted by @bleggett in this Pull Request (https://github.com/istio/istio/pull/49703). The initial discussion of this issue took place in this Issue (https://github.com/istio/istio/issues/49279), where @bleggett described his scenario: in the Ambient mode, the CNI Daemon misplaces rules into the iptables-legacy chains when configuring traffic rules in the node network namespace, while k8s adds its rules to the iptables-nft chains:
Seeing an issue where the single ambient iptables rule istio-cni injects into the host-level iptables POSTROUTING chain (for host healthchecks) ends up in the iptables-legacy chain, even though all the other rules (kubelet, CNI, etc) are in the iptables-nft chain.
@bleggett pinpointed the root cause in the old inspection logic:
This is because our iptables (legacy OR nft) selection logic only checks the container-local binary default, which may or may not actually match what was used or is present on the host node.
He resolved the bug in this scenario by introducing a new detection logic. However, the current solution still faces issues when the host does not support iptables-legacy (as described earlier). Essentially, when neither iptables-legacy nor iptables-nft meet the criteria, it falls back to the iptables command. In Istio 1.22’s four use-cases, the iptables command could imply different implementations:
| Usage | Using iptables Binaries From | Networking context to select correct binary against | ‘iptables’ binary corresponding to |
|---|---|---|---|
| from CNI plugin (sidecar) | host $PATH | pod netns context | depends on host environment, iptables-legacy or iptables-nft are both possible |
| from init container (sidecar) | init container $PATH | pod netns context | iptables-legacy implementation |
| from CNI agent (ambient) | CNI container $PATH | pod netns context | iptables-legacy implementation |
| from CNI agent (ambient) | CNI container $PATH | host netns context | iptables-legacy implementation |
As you can see, as long as a binary inside the container $PATH executes in the Pod network ns, an error will arise if the host does not have iptables-legacy kernel modules installed. Based on my understanding, the current fallback to iptables aims to adhere to the environment settings (assuming iptables pointed to the default implementation), but this implementation may deviate from the expected behavior due to the different implications of the iptables binary under various conditions. I believe we need a different approach to resolve this issue. Here is a proposed solution inspired by @bleggett 's current implementation.
First, we need to check what iptables implementations the current kernel supports. This could be achieved by inspecting kernel modules. The /proc/modules file lists all the kernel modules loaded by the current kernel. This file remains consistent across the host and all containers, indicating the kernel modules loaded by the host’s OS.
After determining the supported implementations, we should confirm the binaries corresponding to each supported implementation in the current context. Then, we use these binaries to retrieve the Rule Dump for each implementation. Here’s how to determine the binaries for each implementation:
- For legacy, try executing
iptables-legacy --version. If no error occurs, useiptables-legacy. If it fails, try executingiptables --version. If the output does not containsnf_tables, useiptables. Otherwise, the current environment does not support legacy. - For nft, try executing
iptables-nft --version. If no error occurs, useiptables-nft. If it fails, try executingiptables --version. If the output containsnf_tables, useiptables. Otherwise, the current environment does not support nft.
By this point, we should have enough information to determine which binary set to use. (the final judgment process is mainly inspired by @bleggett 's approach. Thanks again to him.)
- If both legacy and nft are supported by the current kernel, use the implementation with more than 3 rules for the Rule Dump. (If both have more than 3 rules, may be should report an error or support configuration to specify which one to prioritize for this case?)
- If only one implementation is supported, use the currently supported implementation directly.
I will submit a PR to trying improve the iptables detect implementation, feedback and suggestions are welcome!
Version
istio 1.22.2(both controlplane and dataplane)
Additional Information
none
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 17 (11 by maintainers)
Commits related to this issue
- improve iptables detect implementation(#52357) — committed to ZarrianShi/istio by deleted user 2 years ago
- improve iptables detect implementation(#52357) — committed to ZarrianShi/istio by ZarrianShi 2 years ago
- imporve implementation of iptables detection (#52357) — committed to ZarrianShi/istio by ZarrianShi 2 years ago
The reason why I think a new implementation is needed is that I understand that the current implementation does not work as expected in the istio-init scenario for the following reasons:
istio-initimageAs a result, the ‘tie’ situation is a inevitable in
istio-init, so the bottom-line logic(pickiptable) will definitely be executed, but, whatever we pick unconditionally for this situation(legacy/nft), there will always be situations that cannot be handled expectedly:iptables-legacy, then if the host only has nft, an error will occur.iptables-nft, then if the host only has legacy, an error will occur.To resolve this, i prefer the way blow:
I agree with this:
but maybe we can have a better way to achieve this.
Hi @bleggett and @xocodergit, it really makes sense to have a unified iptables detect instructions for all scenarios.(
istio-init,CNI agent,CNI plugin), rather than how to fix this problem easily.Also thanks to @zhlsunshine , I think we can discuss this in two parts:
1. A unified iptables detect instructions
yes, we definitely need a unified instructions to detect which iptables binary should be used in all scenarios.(
istio-init,CNI agent,CNI plugin), and i have described my opinion on this:I agree with your concern about the OS dependency of checking
/proc/modules, but I think we can discuss this separately in the part 2.2. How to detect kernel modules
well, maybe check
/proc/moduleswould introduce some OS level dependencies(although i think it seems ok because iptables it self also depending on the OS). we can check it in other ways if there is a better one.Hi all, some ideas about above discussion: Firstly, there should be unified instructions for
CNI plugin (sidecar),init container (sidecar)andCNI agent (ambient), theunified instructionsmeans that they may need to follow the same logic to adjust theiptables rules. Secondly, there are considerations on both theinstallation of the appropriate series iptables binariesandrulecount of corresponding iptables binaries. And this second point may causes the differences forCNI plugin (sidecar),init container (sidecar)andCNI agent (ambient), theunified instructionsAt last, make a reasonable and executable logics for choosing the proper iptable binaries. So I suggest that @xocodergit can write a proposal in the community for this. BTW, I believe that @bleggett and I are disagree to check/proc/modulesfor inspecting kernel modules due to it would import the OS level dependency of Istio. It’s unacceptable. Please correct me if any incorrect.The rules count check is meant to be used as a ‘tiebreaker’ when both
iptables-nftandiptables-legacyare present.And we can’t just prefer
nftin that case because even tho both are available, the host may have preferred to inject rules intolegacy- and we want to respect that if we can.But I think we just don’t correctly handle the case where both are present and neither have any rules.
If this is the problem, the simpler fix is probably to
nftandlegacyare presentnftThat should fix your scenario without requiring kmod introspection.