spotipy: API requests suddenly stopped working and hang forever

Hi,

I recently started using Spotipy to get information about artists for a dataviz project.

As you can see, one of my apps makes a lot of requests, but I make sure to wait between requests so I don’t get limited by the 30 second time-window limit.

Sometimes, the API just stops working for 1 to 24 hours. I don’t get any error message, it just keeps running without responding.

Here is how I initialize the client

SP = spotipy.Spotify(
    auth_manager=SpotifyClientCredentials(
        client_id=os.environ.get("CLIENT_ID"),
        client_secret=os.environ.get("CLIENT_SECRET"),
    ),
    requests_timeout=10,
    retries=10,
)

Do you know why I have this error ? I try every 12h to run my code, sometimes it works and sometimes it doesn’t.

I can’t find any information about the rate limite of Spotify, but I think I’m not reaching any limit otherwise I would get an error as a response?

Thanks for your help!

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 20 (7 by maintainers)

Most upvoted comments

I was having this issue occur when trying to call sp.search() or sp.playlist_items(). Using a debugger I found that eventually in the retry.py file in urllib3, a function called sleep_for_retry() was being called:

def sleep_for_retry(self, response=None):
        retry_after = self.get_retry_after(response)
        if retry_after:
            time.sleep(retry_after)
            return True

        return False

It seems that time.sleep(retry_after) is used to wait instead of sending more requests and being rate limited. Since no message is printed, it seems like it just freezes. I couldn’t find a consistent fix, as using different credentials when initializing the client didn’t fix it. Waiting out the retry_after timer appears to work, but that could take many hours.

BenAM63 on Jun 16, 2022

Hi @Peter-Schorn, I managed to debug by doing this :

Go on the Spotify doc
Use the API playground with my ID and secret_key to try the API
Check the result

When I tried it, the API sent an error message because I was rate-limited and asked me to try after x minutes. The issue was the rate-limitation but the behaviour Spotipy was not what I was expected, it just hanged forever without returning an error.

I think the Spotipy package should expose this information so we could automate the process of retrying after x seconds.

After looking at the code, I think it should work, so I don’t know what the issue was: https://github.com/plamere/spotipy/blob/master/spotipy/exceptions.py#L8

julienbeisel on May 4, 2022

I was having this issue occur when trying to call sp.search() or sp.playlist_items(). Using a debugger I found that eventually in the retry.py file in urllib3, a function called sleep_for_retry() was being called:
def sleep_for_retry(self, response=None):
        retry_after = self.get_retry_after(response)
        if retry_after:
            time.sleep(retry_after)
            return True

        return False
It seems that time.sleep(retry_after) is used to wait instead of sending more requests and being rate limited. Since no message is printed, it seems like it just freezes. I couldn’t find a consistent fix, as using different credentials when initializing the client didn’t fix it. Waiting out the retry_after timer appears to work, but that could take many hours.

Apparently this happened to me too. It would be better if at least we know how long we need to wait before the next retry.

Terima on Jul 10, 2022

Still getting this after running the code for about a day, then this happens for about 24h where I have to manually enter the token generated in https://developer.spotify.com/console/

The weird thing is, it works for some functions even with the token updated from Spotipy (which seems to freeze no matter where it’s used in some functions). So to clarify: When I generate a token with:

token = util.prompt_for_user_token(username='XYZ', scope=scope, client_id=client_id, client_secret=client_secret, redirect_uri=redirect_uri)
sp = spotipy.Spotify(auth=token)

The code works when running: sp.available_markets() But freezes (likely infinitely looping) when running for example: sp.search('Test', limit=50, offset=0, type='playlist', market=None) Or as previously mentioned for example: sp.current_user_playlists()

This makes me think there’s some try: except: somewhere that’s depressing whatever error this is about. Why? I don’t get any error, the token from the console is working (thus, my account does not rate limited), and seeing that OAuth is required for Get Markets as described here, yet works while some other OAuth functionality does not work: https://developer.spotify.com/console/get-available-markets/

Could it maybe be some functionality built-in for automatically avoiding rate limits? I have implemented time.sleep(0.3), but it seems like that’s not helping prevent this from happening.

I have yet to get a debugger going that actually shows me valuable information. From what I’ve achieved so far, the debugger hasn’t shown me anything of value, but I will look into this in more detail in the next weeks. Also, if I can’t figure out the debugger to identify exactly where, and if it’s not already found by someone else by then, I will just try all the functionalities to see what works or not and go through the Spotipy library code manually to see what causes this.

dasbts on May 21, 2022

That’s not spotipy related. Well, the only thing we could do is add a debug line to let the users know how long they can expect to wait. See this thread where some element of answers were provided in https://github.com/spotipy-dev/spotipy/issues/937 and https://github.com/spotipy-dev/spotipy/issues/913#issuecomment-1474314058

I imagine that switching to a new Spotipfy app would solve the issue, for some time at least, depending on how much the Spotify API rate limits your app.

Let’s continue this discussion in https://github.com/spotipy-dev/spotipy/issues/913

stephanebruckert on Mar 30, 2023