meilisearch: Unable to add documents and endpoints timeout/unresponsive

I tried another change this morning during meetings. I simply set my resources for meilisearch to be:

          resources:
            limits:
              cpu: '3'
              memory: 4Gi
            requests:
              cpu: '3'
              memory: 4Gi

After that, meilisearch probably performed for the longest it has since updating to v0.21 as shown below:

Then it died all of a sudden, so I do think it’s most likely related to some kind of consumption issue (before the resource scaled as “Burstable”). This is only for around 8000 documents on the index. If I inspect the usage on the node, it’s nothing close to the 3 CPU + 4GB RAM:

I close this issue due to lack of feedback regarding it FYI v0.27.0 will be release next week including indexation performance improvement. Feel free to ask to reopen if needed

Oops! I indeed miss read, sorry 😅

If you can share (in private) your dataset and the steps @Jure-BB it would be awesome. We desperately try to reproduce this bug

I’ve encountered (probably) the same bug.

Here’s a sample that tries to reproduce the bug. It doesn’t always fail, but if you try to run it a few times, it should fail at least once.

public class Book
{
    public string  Id    { get; set; }
    public string  Title { get; set; }
    public string? Description  { get; set; }
    List<string>?  Tags  { get; set; }
}

[Test]
public async Task Add_docs_bug_test()
{
    var client  = new MeilisearchClient("http://localhost:7700");
    var indexes = await client.GetAllIndexes();
    var indexId = "books";
    if (indexes.Select(x => x.Uid).Contains(indexId))
    {
        await client.DeleteIndex(indexId);
        await Task.Delay(1000);
    }

    var newIndex = await client.CreateIndex(indexId);

    for (int batch = 0; batch < 10; batch++)
    {
        var index = await client.GetIndex(indexId);
        for (int i = 0; i < 1000; i++)
        {
            var documents = Enumerable.Range(0, 200)
               .Select(j => new Book { Id = $"{i}_{j}", Title = $"Le Book {i}_{j}" }).ToArray();
            var result = await index.AddDocuments<Book>(documents);
        }
    }
}

I tested it using meilisearch-dotnet & NUnit, running inside WSL2 docker on windows

meilisearch:
  image: getmeili/meilisearch:v0.22.0
  restart: always
  ports:
    - 7700:7700
  environment: 
    MEILI_ENV: development
    MEILI_NO_ANALYTICS: 1
  volumes:
    - ./meilisearch/data:/data.ms

Exception:
System.Threading.Tasks.TaskCanceledException : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
  ----> System.TimeoutException : The operation was canceled.
  ----> System.Threading.Tasks.TaskCanceledException : The operation was canceled.
  ----> System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
  ----> System.Net.Sockets.SocketException : The I/O operation has been aborted because of either a thread exit or an application request.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Meilisearch.Index.AddDocuments[T](IEnumerable`1 documents, String primaryKey) in ...\_libs\meilisearch-dotnet\src\Meilisearch\Index.cs:line 103

Hey @MarinPostma, thanks for the effort! I am still sorry that I couldn’t seem to create something reproducible and do hope that someone does figure it out, also for my own ease of mind 😄

Hello @carlreid ! I have unsuccessfully tried to reproduce your bug with minikube. I have setup a cluster with the same configuration as you did, and repeatedly sent the documents you provided, 1 by 1 about every 200ms, and then waited multiple hours querying the stats route every now and then. We’ll put this issue in standby for now, until someone encounters the same problem as you and can give us additional context.

Thank you @carlreid for your investigation! I’ll keep you informed on the issue!