SoundScrape: Utilize rate limit information from error messages

~/Music $ soundscrape https://soundcloud.com/grrrreatdane/roll-in-peace-bootleg
Downloading: roll in peace (bootleg)
Problem downloading roll in peace (bootleg)
Traceback (most recent call last):
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/soundscrape/soundscrape.py", line 437, in download_tracks
    stream = client.get(track['stream_url'], allow_redirects=False, limit=200)
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/soundcloud/client.py", line 133, in _request
    return wrapped_resource(make_request(method, url, kwargs))
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/soundcloud/request.py", line 148, in make_request
    result.raise_for_status()
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Unknown for url: https://api.soundcloud.com/tracks/377559152/stream?limit=200&client_id=175c043157ffae2c6d5fed16c3d95a4c

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rjones/anaconda/bin/soundscrape", line 11, in <module>
    sys.exit(main())
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/soundscrape/soundscrape.py", line 119, in main
    process_soundcloud(vargs)
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/soundscrape/soundscrape.py", line 292, in process_soundcloud
    id3_extras=id3_extras)
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/soundscrape/soundscrape.py", line 460, in download_tracks
    puts_safe(e)
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/soundscrape/soundscrape.py", line 1315, in puts_safe
    puts(text)
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/clint/textui/core.py", line 57, in puts
    s = tsplit(s, NEWLINES)
  File "/Users/rjones/anaconda/lib/python3.6/site-packages/clint/utils.py", line 69, in tsplit
    string = string.replace(i, final_delimiter)
AttributeError: 'HTTPError' object has no attribute 'replace'

About this issue

Original URL
State: open
Created 6 years ago
Comments: 17 (1 by maintainers)

Most upvoted comments

Hey there.

Issues #206 & #204 are related.

I think Sounscrape cannot keep going with this client_id & secret key or even with many of them as it will always reach Soundcloud API limits at some point.

As Soundcloud closed new app registering I think it would be better to just ask the user to login and then scrap the DOM (with Selenium + Chrome/Firefox in --headless) to get the token and then download the tracks. If you go to urls like:

https://api.soundcloud.com/i1/tracks/387417257/streams?client_id=MgT8dvRJVcFR9fI5Szar82usLfSQdg3n

You then get a response like:

{
  "http_mp3_128_url": "https://cf-media.sndcdn.com/XFlrBjPMUKHI.128.mp3?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLW1lZGlhLnNuZGNkbi5jb20vWEZsckJqUE1VS0hJLjEyOC5tcDMiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1MTc3MzY3NTB9fX1dfQ__&Signature=wfLUQ5-w9NxFP7EOWqB0LN9junfC-DDb4ZNJ8rRJ0MNI0YorEGiCy13V4-nwatJ9G1TX8osBMtfzD~UfEyC-oRifpYWT~0sEnRQ19S9QQpYVg8QoDPCaCrfxMRxNHGpH1WQvGCdgYR5mI6mdj9gwj10ML~hTBbt7AE0~2jOKKy1nvZftydMjTt3cYGdR1gtUP2-J741be4TGzO~pSonV~rVgqbhntatlyTTo9uWj9CCwvGvX4sexZBXS3KPA-76XbqW1wXLbZoDKqtrLk2I9rQnWHyK~OvqUfoJE53HOE6eSS4Ql4JwutQ59sX6w8gao~yqwJFW988Y-MtEtS7zb4A__&Key-Pair-Id=APKAJAGZ7VMH2PFPW6UQ",
  "hls_mp3_128_url": "https://cf-hls-media.sndcdn.com/playlist/XFlrBjPMUKHI.128.mp3/playlist.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLWhscy1tZWRpYS5zbmRjZG4uY29tL3BsYXlsaXN0L1hGbHJCalBNVUtISS4xMjgubXAzL3BsYXlsaXN0Lm0zdTgiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1MTc3MzY3NTB9fX1dfQ__&Signature=lpHe8hejzHGlLtOiuF1b2esSGUu8mgSCa1Y6wAHb0fioBJV5DLzWy~7XGvaSsxxzlJVSu~X2bGCmmQ0kdU0xwP7dLQX9enl2QJwhm3kkggfAfsCFtFToMmA6BxEBaeMtwwC0ePLRzvSaw7mTLBV2vURUxky7P2RpJD87MURx0n8-mGpsaf1rwMKM9dRLKW6kMFqbkppjl4~geuA1SRC12lWHRV8socCEwfu-evCU~Ds~pa8aX2bSj~BK1Erai0E7ht7~jQImxqVae2gyiqU60QofsYIjyWLbyEcLmdElGtdw3NUEP1TtEnAfTK8-zW6z0DifKmoLV-~jn8QstiCgQg__&Key-Pair-Id=APKAJAGZ7VMH2PFPW6UQ",
  "hls_opus_64_url": "https://cf-hls-opus-media.sndcdn.com/playlist/XFlrBjPMUKHI.64.opus/playlist.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLWhscy1vcHVzLW1lZGlhLnNuZGNkbi5jb20vcGxheWxpc3QvWEZsckJqUE1VS0hJLjY0Lm9wdXMvcGxheWxpc3QubTN1OCIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTUxNzczNjc1MH19fV19&Signature=nlLbtT5xpScnlENCznAyPlX0aRvMHA-Y1AXQieVjQg~dWWskwO7b2AB1LDydy7~fzmOkdu6GLoQyK174GLD1fcjy02FD4UQql799CEBtQ4Ker7YzNy4l78F3kbrU03KqcULWot2DvZpuUvNV3nGfUDobwCkC6JLPsx0dkmek8XyigeEemAsbQbHNWPissM10C4LgGzbekQhLwRrOVEQp9ixV7y8z6DghuOcrRg0RTbz~R~NKKdLP3A5tEnLcPPjv1dsyfK~B0dq~ddWFEbH7bPlcB0qLM7TsmGCEHtyTjfeFiYKtKpYZrXegKyUg-nTZcdenIHKsLXAELy5HiUXuLw__&Key-Pair-Id=APKAJAGZ7VMH2PFPW6UQ",
  "preview_mp3_128_url": "https://cf-preview-media.sndcdn.com/preview/0/30/XFlrBjPMUKHI.128.mp3?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLXByZXZpZXctbWVkaWEuc25kY2RuLmNvbS9wcmV2aWV3LzAvMzAvWEZsckJqUE1VS0hJLjEyOC5tcDMiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1MTc3MjkzMzF9fX1dfQ__&Signature=GiyBE4RM-ed6uOHn283mYHiRZVh-4v9vJ3KRjFX2pHZE-G~eC5CZBWN4nlqc18E5KJKpVI3UInRTnloIscatUuAtRKtKjiDR0kn5MxhQA7k2dLGq-2V0KvVCIm1eoSXRDkwFOomg15l62d5b7wWoL-1XJomC7JiEb8ayxPPEr5FRmip9cP05dk57OvqziIwjMIfCv7ubbkSxJ7s-lh9nUvojagQWQ2H~GT-50R0yYoYcFLvG~QpW8HiT2SBIOPT07M9wbavRbF7dqcW1xyStL2QHSWMcESBZBG-ea47oEVuJaYP57FTVCCSGjzbjgKpbWwNup1OSJ53vro50PnKoZw__&Key-Pair-Id=APKAJAGZ7VMH2PFPW6UQ"
}

You can then simply download with “http_mp3_128_url”.

The idea would be :

add login & password in argument parser
pass it to Selenium client
submit form with Selenium
get “Policy” key token
then download stuff

What do you think ?

I can do it if you like but as I don’t know Sounscrape lib I’ve no idea how much refactoring I would have to do to make that fit.

bourdeau on Feb 4, 2018