mopidy-tidal: Better caching to avoid timeouts on large collections

Initializing large collections (or accounts with many playlists) often results in timeouts, even when the collection has already been opened once and the application has simply been restarted.

The current lru_cache implementation only stores cached items in memory, but it’d be good if we could store them to disk (at least the playlist IDs -> names) in order to speed up the initialization time.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 53 (12 by maintainers)

Commits related to this issue

Most upvoted comments

@BlackLight It was added a long time ago by one of the previous maintainers. Probably just as a simple way to display favourites.

No objections from me - let’s delete it 😃

FYI, I have not yet had time to test your latest fixes but I will test it again ASAP.

@BlackLight @fmarzocca I have merged the PR and made some adjustments. Please feel free to test, if there are no issues, I will make a new PyPi release in a few days time.

Found this bug/issue after a playlist refresh:

ERROR [Core-13] mopidy.core.library TidalBackend backend returned bad data: Expected a list of Track, not [Playlist(last_modified=0, name='Your Time Capsule', tracks=[Track(album=Album(artists=[Artist(name='Sting', uri='tidal:artist:17356')], name='...Nothing Like The Sun', ...

This is for ALL Tidal’s playlists

This should be fixed in the latest version of the better-cache branch.

I am wondering how does mopidy-spotify work

Details are in lookup.py and playlists.py. In short:

  1. It uses libspotify whenever it’s available to retrieve things (and it’s usually faster than the web API)
  2. It’s quite granular in retrieving either only the playlist info, or only the URIs of its tracks, or the full content of the tracks. I’m not sure if this level of granularity can be achieved with the Tidal API

the first time ever you load the playlist it’s always a pain waiting for playing.

That’s because the API lookup is still sequential. I think that we should probably have a pool of lookup workers (5 of them should suffice to get a noticeable speedup without overworking the CPU on a humble RPi), but keep in mind that background workers can speed things up only when lookup requests are done in batches. If the client still does the lookup one item at the time there won’t be much of an improvement.

I don’t have installed python-tidal’s branch yet. Should it improve this?

My fork of python-tidal adds support for pagination - it retrieves all the items in case you have playlists/collections with >999 items. And it exposes the lastUpdated field on the playlist objects - without that field mopidy-tidal doesn’t know if a playlist has been updated upstream, so it won’t refresh the cache in case of updates. So it doesn’t really speed things up, but it’s still good to have it if you want a more bug-free experience.