kubernetes: DELETECOLLECTION doesn't always

What happened:

Deleting a large collection times out (504) and leaks a goroutine that’s busy deleting the collection. Calling multiple times in a row makes it worse. There’s no indication that apiserver is currently deleting the collection.

What you expected to happen:

a) Change delete collection so that it deletes just the first page of items in the collection, rather than loading the whole collection into memory. Callers must call repeatedly to delete the whole collection. b) Change delete collection to use a single-flight approach, so that if you call delete collection while one is in progress, you join it rather than start a new one. (This actually doesn’t work because there’s multiple apiservers.)

Environment:

Kubernetes version (use kubectl version): 1.14.x (but I don’t think it matters)

/sig api-machinery

About this issue

Original URL
State: open
Created 4 years ago
Comments: 33 (27 by maintainers)

Most upvoted comments

I opened https://github.com/kubernetes/kubernetes/pull/117971 to address my comment above - still requires adding appropriate tests, but it seems to work fine in a way of passing all existing tests. Will try to finish that soon.

wojtek-t on May 12, 2023

Yup - the stop-the-bleeding part: https://github.com/kubernetes/kubernetes/pull/117971

We should think if we want to evolve that further more medium/longer term.

wojtek-t on Aug 25, 2023

It will be available in 1.28 - we can potentially consider backporting it , but backports weren’t opened at this point.

wojtek-t on Jul 18, 2023

Getting back to it in the context of https://github.com/kubernetes/kubernetes/issues/115090

With https://github.com/kubernetes/kubernetes/pull/105606 and https://github.com/kubernetes/kubernetes/pull/107950 we actually got the point, where DeleteCollection is respecting context cancellation, and thus is finishing soon after the API call times out.

So by just introducing a pagination into DeleteCollection implementation (which should be relatively straighforward) we should actually improve its state to the point where:

it can make some progress even if the collection of objects is huge (and currently listing all those objects times out)
it will not hit into the problem of deleting same objects multiple times in case when we retry the previously failed DeleteCollection request

I’m going to take a look at implementing it in 1.27 timeframe (unless someone objects in the meantime).

wojtek-t on Jan 26, 2023