node-yahoo-finance2: Invalid Crumb / Yahoo disables API access to quotes data
Bug Report
Describe the bug
Quote and Quote summary stopped working recently due to invalid crumbs. First encountered it on my NextJS server and was able to reproduce the bug in CLI as well.
Minimal Reproduction
npx yahoo-finance2 quote AAPL
Environment
Node
Node version (if applicable):
v21.6.1
Npm version:
9.5.1
Additional Context
About this issue
- Original URL
- State: closed
- Created 3 months ago
- Reactions: 13
- Comments: 32 (13 by maintainers)
Commits related to this issue
- fix(getCrumb): use getcrumb endpoint to get the crumb (fixes #764) Thanks to @AbhigyanBafna who first reported and the issue and @nocodehummel, @programmishka, @Yhprum who helped to investigate and w... — committed to gadicc/node-yahoo-finance2 by gadicc 2 months ago
- chore(release): 2.11.2 [skip ci] ## [2.11.2](https://github.com/gadicc/node-yahoo-finance2/compare/v2.11.1...v2.11.2) (2024-04-17) ### Bug Fixes * **getCrumb:** use getcrumb endpoint to get the cru... — committed to gadicc/node-yahoo-finance2 by semantic-release-bot 2 months ago
Hi all, a fix is building now and there will be an automated message here when the release is published.
Thanks to @AbhigyanBafna who first reported the issue and @nocodehummel, @programmishka, @Yhprum who helped to investigate and work towards a fix.
@nocodehummel, this is a great find! And a great compromise that I’m happy to publish. I wasn’t willing to ship something that falsified the user-agent, and had intended to provide some explanations and general directions on manually overriding the user-agent by hand. So, on behalf of all our users, thanks for figuring out something that works out the box!
Thanks again to everyone who contributed above. The magic of open source remains awesome and keeps helping everyone 🙏 ❤️
I would strongly encourage everyone to go to that link and fill out the form. https://bit.ly/yahoo-finance-api-feedback
This project has always done its best to respect Yahoo’s policies and while many efforts have been made to keep up-to-date with their technical requirements, we will never blatantly help to side-step their policies. Both for legal reasons but also out of great respect and appreciation for the data they have been providing us at no cost (!) until now.
The above form is an amazing step by Yahoo to try better understand developers who are using their API. Many serious users of this project (myself included) have looked at alternative (paid) options and found them inferior in many areas. Whereas in the past, Yahoo has mostly just ignored developers using their platform, it now looks like they’re looking to seriously engage us and (perhaps) better address our needs.
The possibility of having a official API support from Yahoo is a big deal, and would result in much more stable use without us having to constantly ship updates for the frequent changes they make to their existing and basically internal API. I think having a small free plan for personal use with sensible limits would be amazing, as would a paid option which opens up the possibility for developers to monetize their hard work (alongside any local regulatory requirements).
So, again, I really encourage everyone to complete the form and see where this process leads. As users of this library, you may have enjoyed a reasonably stable experience (if you keep the library up-to-date), but as developers, keeping things working (and working correctly) against a constantly changing service is a huge effort and massive energy drain. Something official from Yahoo would be absolutely incredibly for all involved, so again, I encourage you take advantage of this opportunity and keep your fingers crossed.
🎉 This issue has been resolved in version 2.11.2 🎉
The release is available on:
Your semantic-release bot 📦🚀
I’ve pushed a commit to my fork that fixed the issue for me - the only thing to consider is your stance on the user-agent thing. Everyone (and this library) using yahoo finance APIs has been using the yahoo finance APIs long after they had been unsupported and after they introduced this cookie+crumb thing last year so I have no qualms with this change, but ultimately it’s up to you
https://github.com/Yhprum/yahoo-finance/commit/51b1e9ec6faa0b967e3166bcf4b8e6954cb57525
Ah yeah sorry, I thought you meant for new fetches (non-cached). Thanks.
Quick update: had a quick chance to look at this now. Using their
/v1/test/getcrumb
endpoint we can actually get a different crumb that is valid. It’s a slightly different flow than what we used before. I’m on track but ran out of time, presumably I can get something working later this week, BUT, it requires changing theUser-Agent
, which goes against the guiding policies of this library, so, I need to think more about this.As mentioned above, the library works from the console. I’ve modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.
Thanks also to everyone above for reporting the issue, and for your input and feedback. I know this is a huge pain and inconvenience, but I really believe that Yahoo asking for developer feedback rather than just blocking us is a very positive step that could lead somewhere good for all of us.
Hey all. Firstly, thanks everyone for chiming in. And wow, I personally had never seen the form until now, so thanks for letting me know it’s been around for some time.
A few comments:
~/.yf2-cookies.json
, works great, but if I delete that file, I get the invalid crumb error for the newly retrieved cookie / crumb.I guess that’s encouraging that probably this issue could be worked around, but a) I unfortunately don’t have any time to look at this now, and b) we need to be very careful of any legal implications (I’m thinking at least, requiring an extra step from the developer to acknowledge the situation with a link to yahoo’s form).
FETCH_DEVEL="nocache"
it will instead make fresh network requests (link to docs). Note, this is very slow, can sometimes trigger a yahoo rate limiting ban (although I think we added our own rate-limiting code to avoid such bans), and also, a lot of tests that pass in the cache can sometimes fail when they added/removed fields we didn’t accommodate yet.Anyway, let’s see what we can do as a community. As I said, I won’t have a chance to investigate this now, but maybe others in the community might have some luck in the meantime. Thanks again to everyone above who has posted their findings so far - you guys are awesome! - and also to all users for your understanding of the situation and our stance.
P.S. Probably quickest workaround would be to access and accept all terms from your web browser, and then for us to have a good way to import the cookie/crumb for yf2. With the appropriate warnings, acknowledgements and links I mentioned above. Lastly, one big thanks again to this awesome community! 🙏 ❤️
The library (v.2.11.1) works from my console:
npx yahoo-finance2 quote AAPL Storing cookies in /Users/xxxxxx/.yf2-cookies.json Retrieved crumb from cookie store: ELG7IOJ9t.1 { language: 'en-US', region: 'US', ...
For my case I need to find out, why the error “Invalid Crumb” occurs when I use the library (v.2.9.0 or v.2.11.1) in my Google Cloud Function. This problem I have after last Tuesday. But this is not the problem of the library, I think.@gadicc I was on the same path. When calling that crumb endpoint with a non-browser user-agent it responds with
Too Many Requests
. The endpoint does work withMozilla/5.0 (compatible; yahoo-finance2/0.0.1)
.Using the URL https://query2.finance.yahoo.com/v1/test/getcrumb You need your cookie for YF, named A1 or A3 for both - getcrumb and quote requests (they expire in 6 months, so need to update them sometimes). For the quote request you need to set the crumb as an URL parameter: https://query2.finance.yahoo.com/v7/finance/quote?crumb=XXXXX
I’ve just analyzed the network activity when searching for AAPL on the YF site and made the same requests. The API works.
UPD: for getcrumb request you also need to set the User-Agent header, otherwise it will respond with “too many requests”.
This form first appeared last year around May or so, so it’s definitely not a new thing they just made and set out - but who knows about any new API since the new Yahoo Finance interface is public now.
I assume this is just a different cookie/crumb/useragent thing to fix - for me it’s working fine on my deployed site but not in dev (maybe because prod has an older crumb?) so I’ll be looking into the issue on my end as well.
Welp, I filled out the form… Back to Selenium I go 😔
If anybody has any other simple stock/crypto price feeds you know about, let me know
Confirmed not working on my side too. Hope Yahoo reopens as a paid service.
Just started getting this error today. Sad to see this library not working anymore due to Yahoo disabling their API. Hopefully they open their API up even if its not free. I’d pay for it if needed.
I am able to get a positive response from the API https://query2.finance.yahoo.com/v7/finance/quote?symbols=AAPL&crumb=XXXXXXX Using Postman, with my A1 cookie set.
This message from yahoo has been provided since many months. The API was still accessible using the node-yahoo-finance2 library, because it deals with cookies. However, the “Invalid Crumb” error is new, as of last week. The API responded last Tuesday, using the version 2.9.0 of the library.