librespot: Suddenly unable to play any tracks

I just tried to force the ap-gew4.spotify.com domain to point to the IP of ap-gew1.spotify.com, by setting this in /etc/hosts:

104.199.65.124  ap-gew4.spotify.com

That works!! Sooo… Is Spotify having an issue? Or are Spotify changing something?

Just in case you are gathering stats/data: Same issue for me (in Denmark).

See @morphar’s comment further up this report:

I just tried to force the ap-gew4.spotify.com domain to point to the IP of ap-gew1.spotify.com, by setting this in /etc/hosts:

104.199.65.124  ap-gew4.spotify.com

That works!! Sooo… Is Spotify having an issue? Or are Spotify changing something?

@kingosticks’s suggestion to offer an ABI to manually set the access point isn’t a bad idea in this particular case. I am however hesitant to offer it as a command-line option because I don’t want to get it misused.

What you can also try is blocking apresolve.spotify.com altogether, which will cause librespot to default to pick ap.spotify.com as a fallback access point. I assume that this is geobalanced and might offer some other, in working order, access point.

I consider all of these as band-aids, ideally librespot would reconnect and drop the erroneous access point. Like I said I can investigate this as part of new-api. It’s not on top of my list – I’d rather have Spotify to fix their infrastructure – but I should get around to it sometime. In the meantime PR’s are more than welcome of course.

I don’t think it’s a common problem. But it’s probably still worth trying to report to Spotify?

And should librespot be smarter and not only rely on the first element in the list?

Anyone got any answers from Spotify? This is lazy work at their end.

We have no lines of communication to Spotify.

I’ve now had reports from Austria and Switzerland, too… are they shutting down old infrastructure?

It would surprise me because many legacy devices would suffer. I don’t think Spotify announced the EOL for such hardware.

On the other hand I don’t understand as this is just their own resolver, which last I checked was also used by their own clients. One difference is that I saw so when I was working on new-api with HTTP. Who knows the Mercury is phased out in that server cluster, and legacy clients fall back?

Seeing how this issue is only becoming worse we could consider blacklisting this host name.

This problem started for me a few days ago in Norway. After finding this thread, I blocked apresolve.spotify.com, and now librespot works OK again. For now.

+1 from The Netherlands.

Fixed by adding 104.199.65.124 ap-gew4.spotify.com to /etc/hosts

+1 from Sweden Used the “fallback ap hack”. Set --ap-port=[some random port]. Log then said Using fallback “ap.spotify.com:443” and it works!

@radiorasmus

Type “sudo nano /etc/hosts”, and paste in “104.199.65.124 ap-gew4.spotify.com” exit and save and you should be up and running again. You might have to restart the system, not sure.

Ha, you beat me to it! I wanted to test the opposite (using ap-gew4 instead of what I receive).

Yes, I believe there’s a problem with that one host.

v0.4.2 has been released: https://github.com/librespot-org/librespot/releases/tag/v0.4.2

librespot also did stop working for me since this morning due to the problem described in this issue. I am located in Bavaria, Germany. my apresolve.spotify.com also contains the ap-gew4 as first entry.

root@al:~# curl apresolve.spotify.com
{"ap_list":["ap-gew4.spotify.com:4070","ap-gew4.spotify.com:443","ap-gew4.spotify.com:80","ap-gew1.spotify.com:4070","ap-guc3.spotify.com:443","ap-gae2.spotify.com:80"]}

Can confirm that adding

0.0.0.0                apresolve.spotify.com

OR

# IP of ap-guc3.spotify.com
104.154.127.126         ap-gew4.spotify.com

to /etc/hosts both work for me.

I was hoping to make some time to do a quick update release to blacklist this “broken” (to us) access point, but chances of me completely upgrading to the HTTP-based API before September are next to zero:

1. the current state of new-api is functional but needs bugs ironed out and dev merged in. I don’t realistically have time for all of that before September. I have called multiple times for people to take on that chore but no one has answered the call.

2. new-api now retrieves files over audio files over HTTP but not encryption keys or commands.

Now I don’t know what Spotify will be deprecating and what not, but:

• the encryption keys over HTTP have not been reversed engineered yet (with decompilation attempts from some having been taken down).

• the HTTP-based network commands (websockets) require implementing the dealer and this will take a lot of work as well as some advanced Rust skills and knowledge of librespot internals. librespot-java has the reverse engineering mostly down but it’s much more than just a matter of Java-2-Rust as the code architecture has diverged a lot.

Again I don’t know what that Spotify SDK deprecation may bring but if this project wants to stay up to date then we really need more contributors.

i know it is not a beautiful solution but i can confirm that i works. as morphar suggested. 😃 thanks for the tip.

But instead of doing in all the pi’s /etc/hosts we have implemented the redirect in our dns - 104.199.65.124 ponting to ap-gew4.spotify.com

You will also want to lookup those host names and report their IP. Same names could result in different IPs depending on geography, I’m not sure.

This backend is known to panic on several platforms.
Consider using some other backend, or better yet, contributing a fix.
Country: "US"
The application panicked (crashed).
Message:  could not find device
Location: /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/github.com-1ecc6299db9ec823/librespot-playback-0.2.0/src/audio_backend/portaudio.rs:62

what is workaround? macOS 11.6

I don’t think this is related to the subject of this issue.

But on Mac, just use Rodio instead of PortAudio. I don’t know about this Homebrew recipe but Rodio should be the default backend anyway. Or specify --device with whatever PortAudio needs, because that’s what it’s complaining about.

Thanks @miguelprates and @marciogranzotto, I will add ap-gue1 to the blacklist and do a release. Hope this will cover most of the cases and otherwise people can use any of the workarounds mentioned in this issue.

Will thereafter focus on development of new-api which is where the real fix is – that new API doesn’t run into trouble with any of these access points because it downloads files over HTTP instead of Mercury.

Hello, noob here – I added following line to /etc/hosts of Spotify service (via Balena dashboard -> Terminal -> Spotify target -> then editedd with vi, saved with :x), but the changes are not kept after rebooting.

Could you please explain step by step how to make this fix?

Context: unable to play songs since this morning, I’m also located in Germany. I’m using BalenaSound, not Librespot, but this issue has been seen across many platforms (also Volumio afaic)

I just temporary fixed the issue for BalenaSound by adding a command in the start.sh file of the Spotify Docker container:

1. locate the file in plugins/spotify/start.sh
2. Add the following line before the “set” command for librespot (almost last command): echo "104.154.127.126 ap-gew4.spotify.com" >> /etc/hosts
3. Push the new code to BalenaCloud via balena push or git

@roderickvd also having the same problem with ap-gue1

Here in Brazil ap-gue1 is the problem. I had to redirect dns from ap-gue1 to ap-gew1 on my router and now it’s working.

@roderickvd also having the same problem with ap-gue1

I merged #1026 with a fix for ap-gew4. If nothing else pops up I will do a 0.4.2 release the coming days.

@kirenida:

why should they care about unofficial clients?

Because they deprecated the official libspotify so there isn’t an “official client” anymore.

Awesome workarounds! However, I am a bit curious what the difference is between defining

Have you tried opening apresolve.spotify.com in your browser?

If you do that, you will get something like:

{"ap_list":["ap-gew1.spotify.com:4070","ap-gew1.spotify.com:443","ap-gew1.spotify.com:80","ap-gew1.spotify.com:4070","ap-guc3.spotify.com:443","ap-gae2.spotify.com:80"]}

which is a list of Spotify content delivery servers most suitable for your location (as determined from Spotify’s side)

As stated previously in this thread, it looks like Spotify has started rolling out some new method of serving content through some of these servers. That new method works with the official Spotify client, but not with librespot. (Which I don’t consider “lazy” as stated in a previous comment - why should they care about unofficial clients?)

With apresolve.spotify.com blocked, librespot defaults to some server that doesn’t have that new method enabled (yet).

If you just define a single server, there is a chance that the response from apresolve could change at some point in time, and then you would need to enter a new combination of IP address + URL.

As I’m not experienced with the inner workings of librespot or Spotify, I’m just going with my basic knowledge and logic, so what I’ve just written may be wrong in some way. If that is so, somebody please correct me 😃

+1 from NL. Adding 104.199.65.124 ap-gew4.spotify.com to /etc/hosts fixed it for me

I had the same issue, I’m from Italy 🤌. I’m using a Raspberry Pi 3B with Raspberry Pi OS and Raspotify.

I added the following line to /etc/hosts and now everything works fine:

0.0.0.0		apresolve.spotify.com

Thank you @EdoardoLaGreca 😃 Same country same fix 😉

+1 from NL. Adding 104.199.65.124 ap-gew4.spotify.com to /etc/hosts fixed it. Also blocked apresolve.spotify.com in pihole to be on the safe side 😉.

Try blocking apresolve.spotify.com in pihole, that worked for me.

This actually worked. I am astounded. Thank you very much!

@agramner I did not find any recommended ways to add parameters to containers in Balena. Since Balena-sound is a custom build of Balena OS with many containers, without any excessive customizability. None the less, I thank you for your reply!

I tried adding it to pi.hole local DNS records. I can ping ap-gew4.spotify.com from Spotify container and Host OS with reply. With no luck tracking down the real issue.

Try blocking apresolve.spotify.com in pihole, that worked for me.

+1 from Switzerland With “104.199.65.124 ap-gew4.spotify.com” in the /etc/host file it works, thanks! I am not sure what is causing the problem and how to properly fix it.

@michaelherger No, I didn’t even look for a setting. I falsely assumed it “just did it” It works now! Thanks!

Did you check the new “fallback” option in the Spotty settings? No need to fiddle with DNS.

Yes. It was a misunderstanding. The person was already using a workaround I implemented in my Spotty plugin (forcing fallback by providing an invalid ap-port value). That seems to be working for people with the failing ap-gew4.spotify.com.

One way to enforce this behavior actually already is built in: set ap-port to some random, invalid port. If librespot fails to connect to that port, it would use the fallback.

Channel errors are not new. If you use GitHub search on this project you will find a number of reports them. How long they’ll last, only Spotify could say.

Programmatically detecting channel errors is not difficult, you literally see channel error: 2 1 in the log so we could act on that. But architecturally it’s more difficult to let librespot reconnect to another access point. In new-api it’s a bit easier to implement but still needs work.