istio: istio-cni: failed to remove pod from mesh

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

Namespaces previously added to ambient mesh are not deleted

After removing the label (istio.io/dataplane-mode) from the tools namespace:

kubectl label ns tools istio.io/dataplane-mode-

the tools namespace continues to be in the ambient mesh:

./istioctl ztunnel-config workload

2024-11-08 09 56 26

istio-cni logs: 2024-11-08 10 09 58

Version

control plane version: 1.24.0
k8s: v1.30.4
CNI Plugin: cilium v1.16.3

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 37 (30 by maintainers)

Most upvoted comments

@ppalucki I suspect that the error you have in the logs about inpod rules not being deleted correctly should be addressed by my commit, that error led to an early exit in the RemovePodFromMesh() when certain iptables cfgs are used.

I noticed the same problem while working on the test cases for the idempotency/reconciliation PR so I tried to fix it. From the description of my PR:

  • Resolved in-pod deletion errors caused by the deletion logic incorrectly assuming that certain commands would work, even when the relevant rules were not present. The creation of these rules depends on the values of tproxy and ingressMode. Therefore, certain failed deletion commands cannot reliably indicate successful deletion. See https://github.com/istio/istio/commit/1b03f56e1476cf850afe11f70abdb15ed810e686.

On the other end, I cannot say for sure why sometimes you experienced the issue and sometimes you did not as I’m not well versed in the cni-agent logic.

If it’s feasible to have a smaller fix for the patch only id love to do it. Unfortunately I don’t have any ideas though

On Thu, Nov 14, 2024, 1:17 PM Leonardo Sarra @.***> wrote:

My commit is a prereq to fix it as it adds the reconciliation feature but it is not enough on its own. @bleggett https://github.com/bleggett is working on a follow-up to that commit to trigger reconciliation on all existing pods.

— Reply to this email directly, view it on GitHub https://github.com/istio/istio/issues/53829#issuecomment-2477328765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXKO7BNXETE426W5O532AUANXAVCNFSM6AAAAABRMLWEZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZXGMZDQNZWGU . You are receiving this because you were mentioned.Message ID: @.***>

Yeah folks said unconstained doesn’t work. I wonder if the issue is this ptrace role in the appArmor config: https://github.com/containerd/containerd/blob/2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41/contrib/apparmor/template.go#L94. I have zero context here, but maybe allowing ptracing containers within the pod disallows ptracing containers/processes outside the pod (similar to how Istio AuthZ policies work)

Yeah https://gitlab.com/apparmor/apparmor/-/wikis/TechnicalDoc_Proc_and_ptrace#apparmor-3-with-ptrace-rules implies that a bit.

On slack someone said privileged=true fixed it

On Mon, Nov 11, 2024, 10:29 AM Ben Leggett @.***> wrote:

Edited Daemonset by adding the Unconfined profile for AppArmor:

[image: 2024-11-08 09 56 26] https://private-user-images.githubusercontent.com/155072573/384786496-8bdb2a82-a3be-4313-ad0f-00bc0a9dffd8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzEzNDk3MzksIm5iZiI6MTczMTM0OTQzOSwicGF0aCI6Ii8xNTUwNzI1NzMvMzg0Nzg2NDk2LThiZGIyYTgyLWEzYmUtNDMxMy1hZDBmLTAwYmMwYTlkZmZkOC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMTExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTExMVQxODIzNTlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iYzU2NDY0NDc0YmE1ODQzOWQ1YzM1MTViZTdjZTY5MWY1ZmEzNmU2M2Y2NTNiNTdiMTA1ZDc1MTRmY2I1MzM5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.oFpCfVJDWzvgzlnNKDgtIYmv4GOfZ6tqwuO893MF7Lo

QQ - does setting privileged: true change any behavior here? It may elide/bypass AppArmor.

I find it odd, if this is a default Ubuntu, that no one would have hit this in 1.23.

— Reply to this email directly, view it on GitHub https://github.com/istio/istio/issues/53829#issuecomment-2468792633, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXP3V6BRWMOYYRSMDJL2ADZO7AVCNFSM6AAAAABRMLWEZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRYG44TENRTGM . You are receiving this because you were mentioned.Message ID: @.***>