aos-AVP: [Bug]: AutoScrapeService crash with large collection

Problem description

Encounter SQLiteQuery: exception: Row too big to fit into CursorWindow.

Steps to reproduce the issue

Scrap a large collection.

Expected behavior

No response

Your phone/tablet/androidTV model

nvidia shield

Operating system version

Android 11

Application version and app store

6.0.xx

Additional system information

No response

Debug logs

01-26 20:43:55.140 11027 11078 E SQLiteLog: (9) statement aborts at 50: [SELECT _id, poster_id, _data, COALESCE(scraper_name,title) AS name, bookmark, Archos_bookmark, Archos_lastTimePlayed, COALESCE(cover,'') AS cover, duration, _size, Archos_traktSeen, Ar
01-26 20:43:55.204  3705  3775 D WindowManager: handleComboKeys keyCode: 22, keyAction: 1
01-26 20:43:55.220 11027 11269 E SQLiteConnection: startPos 8292 > actual rows 7684
01-26 20:43:55.221 11027 11269 E SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=10853, totalRows=7684; query: SELECT _id, _data, title, m_id, e_id, ArchosMediaScraper_type, video_online_id, e_season FROM video WHERE (ArchosMediaScraper_id=0 AND Archos_hideFil
--------- beginning of crash
01-26 20:43:55.233 11027 11269 E AndroidRuntime: FATAL EXCEPTION: Thread-22
01-26 20:43:55.233 11027 11269 E AndroidRuntime: Process: org.courville.nova, PID: 11027
01-26 20:43:55.233 11027 11269 E AndroidRuntime: android.database.sqlite.SQLiteBlobTooBigException: Row too big to fit into CursorWindow requiredPos=10853, totalRows=7684
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.sqlite.SQLiteConnection.nativeExecuteForCursorWindow(Native Method)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.sqlite.SQLiteConnection.executeForCursorWindow(SQLiteConnection.java:1001)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.sqlite.SQLiteSession.executeForCursorWindow(SQLiteSession.java:838)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.sqlite.SQLiteQuery.fillWindow(SQLiteQuery.java:62)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.sqlite.SQLiteCursor.fillWindow(SQLiteCursor.java:161)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.sqlite.SQLiteCursor.onMove(SQLiteCursor.java:131)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.AbstractCursor.moveToPosition(AbstractCursor.java:248)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.AbstractCursor.moveToNext(AbstractCursor.java:280)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.CursorWrapper.moveToNext(CursorWrapper.java:206)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at com.archos.mediaprovider.CustomCursorFactory$CustomCursor.moveToNext(CustomCursorFactory.java:82)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at android.database.CursorWrapper.moveToNext(CursorWrapper.java:206)
01-26 20:43:55.233 11027 11269 E AndroidRuntime:        at com.archos.mediascraper.AutoScrapeService$3.run(AutoScrapeService.java:387)

About this issue

Original URL
State: open
Created 2 years ago
Comments: 28 (13 by maintainers)

Commits related to this issue

ContentResolver.QUERY_ARG_LIMIT trick does not work use limit appendQueryParameter("limit", offset + "," + limit) on uri instead since it is processed in VideoProvider note that QUERY_ARG_LIMIT is v... — committed to nova-video-player/aos-MediaLib by courville 2 years ago

Most upvoted comments

If it doesn’t crash with let’s say 10-100 or 500 rows per query that means 1000 is too much or my guess would be an “about movie” text might be taking too much space so maybe saving compressed version of this text would solve your issue and only decompress it when it is used.

As I do something like this in an api work. Compressing text is really fast and saves from much space.

Also is it possible for you to share full query that is thrown in exception, I am really curious what kind of query it is.

Another thing I just thought now bcoz of the log you showed even though you set window size 10.

Maybe if you set window size 1 and see this cursor error output again, that definitely means the data you are pulling has toooo much text in it. This kind of trial will definitely give you an idea about what is wrong. If this is the case you will need to have somehow less text.

okan35 on Jan 30, 2022

By the way I see there is a blob exception, you dont store an image or something like that in table as a blob right ? Because if you are doing that the solution below wont work for that.

So basically main outcome with this function is to limit the amount of rows you get from your table, limit the amount for memory to process and the cursor will always bring your set limit or what is left in the last 1000 or whatever your limit is.

https://gist.github.com/okan35/1c57e4f545a8ddabf9976e0699ba2bc4 I roughly put it here you will need to adapt to your code

If I am seeing correctly, why your code is not working is because of this line;

Cursor cursor = getFileListCursor(PARAM_ALL, null); final int numberOfRows = cursor.getCount(); //THIS LINE cursor.close(); Reason is you basically want to get count of that huge table this will cause you issues, your cursor should not know how big is your table.

okan35 on Jan 29, 2022

If you wish I can help you with this as I made a recursive function that gets data from database into a json array with limit and offset, such as 1-1000, 1001-2000 and so on till it gets all the rows.

I made that function because I personally had this exact issue when query returns a very big result, the app would use too much memory and cursor would get full of course.

okan35 on Jan 28, 2022