kaminari: paginate calls slow COUNT query for large sets (when it might not need to)

I’m experiencing slow performance issues on my app using the kaminari paginate command, and wondering if there might be a way to avoid it.

Here’s the relevant trace:

/kaminari-0.15.1/lib/kaminari/models/active_record_relation_methods.rb:24:in `total_count'
/kaminari-0.15.1/lib/kaminari/models/page_scope_methods.rb:22:in `total_pages'
/kaminari-0.15.1/lib/kaminari/helpers/action_view_extension.rb:18:in `paginate'

And here’s the SQL query, which in my case is related to a Spree backend view:

(209.9ms)  SELECT DISTINCT COUNT(DISTINCT `spree_orders`.`id`) FROM `spree_orders` WHERE (`spree_orders`.`completed_at` IS NOT NULL)

It seems that kaminari needs to know the full count from this table (which in my case is circa 250,000) in order to generate the “LAST” link.

What I’m thinking is, perhaps there’s a way to pass ?page=last into kaminari and have the last page’s number computed only then. If that were the case, we would only need to know whether the count exceeded the number of items per page * the number of pages. Presumably that would be a much faster SQL query than getting the total number each time we run the query.

I’d be interested to know what the community thinks. This would be a huge performance boost for my application, and likely for anyone else working with large data sets.

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 28 (5 by maintainers)

Commits related to this issue

Most upvoted comments

Solution!

class BillsController < ApplicationController
  def index
    @bills = Bill.where(group_name: 'teste') # must be a relation
    @bills.instance_variable_set(:@total_count, Bill.default_per_page * 20)
  end
end

Pagination without count is now available: https://github.com/amatsuda/kaminari/pull/681#issuecomment-258727907 Give it a shot!

There is solution/workaround suggested in another issue: https://github.com/amatsuda/kaminari/pull/818#issuecomment-252788488

Optimize paginate COUNT query for large sets

  1. The cause is the database dose not use the corrent index smartly.
  2. The solution is making the database to use a specified index.
  3. So I have modified the count code, as my pr.
  4. This is useful for me when I add a useful index for a large sets and make paginate COUNT with it.
  5. Good luck for everyone.

the example code as below:

Find

@model_names = ModelName.use_index('index_for_find').page(params[:page])

Search

class ModelName < ActiveRecord::Base
  ...
  scope :search, -> (type, search) do
    self.total_count_index = 'index_for_search'
    where(ModelName.arel_table[type].matches("#{search}%"))
  end
  ...
end

+1 or just saying “it doesn’t work” don’t solve the issue. Any contribution is appreciated.

+1 for this, it doesn’t seem to work