LdapRecord: [Bug] Getting an exception for "paged results cookie is invalid" when chunking a model

@stevebauman just wanted to let you know the fix works in production, with the new named ‘isolate’ argument. My live project is now updated and running fine.

OK, got it working. Had to clone both repos locally, set up symlinks for both in my composer.json with versions of “*”, branch ldaprecord-laravel locally so I could set the composer.json dependency for ldaprecord in that to “*”, then switch to the chunk-query-isolation branch on ldaprecord.

And it works. 😃

As in, the query isolation works, and I can now happily query members and groups inside my chunk code.

Great work, thank you!

Unfortunately that seems to conflict with the requirement in ldaprecord-laravel …

        "directorytree/ldaprecord-laravel": "^2.7",
        "directorytree/ldaprecord": "dev-chunk-query-isolation",

//

Problem 1
    - directorytree/ldaprecord-laravel[v2.7.0, ..., v2.7.3] require directorytree/ldaprecord ^2.4.4 -> found directorytree/ldaprecord[v2.4.4, ..., v2.19.4] but it conflicts with your root composer.json require (dev-chunk-query-isolation).
    - Root composer.json requires directorytree/ldaprecord-laravel ^2.7 -> satisfiable by directorytree/ldaprecord-laravel[v2.7.0, v2.7.1, v2.7.2, v2.7.3].

I’ll try the symlink method.

BTW, here’s where Microsoft document their cookie handling, and the MaxResultSetsPerConnection limits.

I haven’t managed to definitely confirm it yet, but it seems like OpenLDAP (at least the server implementation I have) simply doesn’t support this, and just has one cookie per connection. But documentation on specific server implementations of RFC 2696 are essentially non existent, apart from Active Directory.

https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/how-ldap-server-cookies-are-handled

Yeah, that might work. Some of the discussion I’ve seen in googling this topic talks about connection pooling on the client side. I’m not sure how much overhead it is setting up and tearing down connections, so a pool might be the way to go. Some mechanism where a connection is unavailable to use if it has a cookie, and only open a new connection and add it to the pool if there aren’t any in the pool with no cookie, rather than setting up and tearing down every time a paginated request is made. With a configurable max pool size, which throws an exception if you try and open too many concurrent paginated requests.

I’d also be interested to see if this works as-is against Active Directory, but I don’t have an AD server handy. I definitely get the impression AD handles cookies per-search, up to a maximum number of concurrent searches, rather than per-connection.

Here’s an example of where I think someone ran in to this same issue, where the code works against AD but not against OpenLDAP.

https://stackoverflow.com/questions/57632585/paginating-openldap-with-apache-directory-ldap-api-over-multiple-connections

I’ve spent a couple of hours debugging this, and I don’t see LdapRecord doing anything wrong. It is storing the returned cookie from the chunk() in the LazyPaginator’s query, and correctly setting it again in applyServerControls() when fetching the next chunk. I had thought that maybe the cookie was being overwritten by the AbstractPaginator doing the paginate() within my chunk() processing when I fetch the user’s groups.

I’m beginning to think that maybe OpenLDAP doesn’t implement the same “cookie pool” that Active Directory does, and that any given client can only have one search cookie active at a time - I’ve tried reducing my chunk size down to 3, no luck. And even if it did, it I suspect I would overrun the limits, as Microsoft talk about have a MaxResultSetsPerConn of 10. Whereas I’m iterating over hundreds of users in my chunk, fetching their groups, so would generate hundreds of search sets requiring cookies (although they should run to completion and get cleared, one would have thought).

I think I’m just going to have to refactor how I do this, and avoid doing anything that generates potentially paginated LDAP searches within the chunking.

No desperate hurry on this one, as the project is in limited internal testing atm, and we don’t expect significant growth till later this year. So right now I only have 300 or so users, and 50 or so groups. But the expectation is that this will potentially grow within a year or so to tens of thousands of users, so I’m trying to build in chunking where it may be necessary now.

BTW, my interpretation of the RFC for simple pagination is that the intent is to allow multiple concurrent paginated search requests, and that the server associates each cookie with a specific searchRequest. But the language uses “may” rather than “must”, so who knows.

https://www.rfc-editor.org/rfc/rfc2696

A client may have any number of outstanding search requests pending, any of which may have used the pagedResultsControl. A server implementation which requires a limit on the number of outstanding paged search requests from a given client MAY either return unwillingToPerform when the client attempts to create a new paged search request, or age out an older result set. If the server implementation ages out an older paged search request, it SHOULD return “unwilling to perform” if the client attempts to resume the paged search that was aged out.

I’m stepping thru code, see if I can find anything. As an experiment, I tried setting $this->paginated to true in chunk(), at which point the LDAP server does return a cookie, but things break when it tries to hydrate the response. So looks like something somewhere isn’t setting the control to ask for a cookie when chunking.