yahoofinance-api: Intermittent HTTP 401 returned from Yahoo API

Isn’t this valid?

use https://query1.finance.yahoo.com/v6/finance/quote?symbols=…

it works for me…v7 -> v6

This means the following may be a workaround in the client-code without changes to the yahoofinance-api code itself

System.setProperty("yahoofinance.baseurl.quotesquery1v7", "https://query1.finance.yahoo.com/v6/finance/quote");

At least it seems to make it work for me.

It seems the “v6” API was now shutdown by Yahoo, likely because everyone switched over from “v7”

The problem is that crumb is not generated in the function.

If I manually use URL https://query1.finance.yahoo.com/v1/test/getcrumb in browser, than I got a crumb. The code below does not return crumb. It is always empty.

    URL crumbRequest = new URL(YahooFinance.HISTQUOTES2_CRUMB_URL);
    RedirectableRequest redirectableCrumbRequest = new RedirectableRequest(crumbRequest, 5);
    redirectableCrumbRequest.setConnectTimeout(YahooFinance.CONNECTION_TIMEOUT);
    redirectableCrumbRequest.setReadTimeout(YahooFinance.CONNECTION_TIMEOUT);

    Map<String, String> requestProperties = new HashMap<String, String>();
    requestProperties.put("Cookie", cookie);
    log.info(cookie);

    URLConnection crumbConnection = redirectableCrumbRequest.openConnection(requestProperties);
    InputStreamReader is = new InputStreamReader(crumbConnection.getInputStream());
    BufferedReader br = new BufferedReader(is);        
    String crumbResult = br.readLine();
    log.info("crumbResult", crumbResult);

I have the same problem - Crumb is indeed generated on the browser, but not in the app where the same URL is used. Like I’ve mentioned before, the patterns are not matched against the HTML the app receives, so scraping the crumb and cookie isn’t working for me also.

use https://query1.finance.yahoo.com/v6/finance/quote?symbols=… it works for me…v7 -> v6

This means the following may be a workaround in the client-code without changes to the yahoofinance-api code itself

System.setProperty("yahoofinance.baseurl.quotesquery1v7", "https://query1.finance.yahoo.com/v6/finance/quote");

At least it seems to make it work for me.

Changing the URL does seem to work. Apparently that one hasn’t changed how it’s accessed and is working exactly as it was before (for now, anyway!)

@centic9 @emidesy Same here. Apparently you need to pass the crumb as an argument now?

If you go to https://query1.finance.yahoo.com/v1/test/getcrumb on a browser, you should get a string result. If you pass it to the URL, you should be able to get the info like so:

https://query1.finance.yahoo.com/v7/finance/quote?symbols=ORCL&crumb=<YOUR_CRUMB_RESULT>

However, this is not working in my Android App. I’ve debugged the app a little and Cookie and Crumb are returning empty because the CrumbManager is not matching the following patterns that are used:

Pattern patternPostForm = Pattern.compile("action=\"/consent\"");
Pattern patternInput = Pattern.compile("(<input type=\"hidden\" name=\")(.*?)(\" value=\")(.*?)(\">)");

I’m assuming something changed related to how the API is accessed and the content returned from HISTQUOTES2_SCRAPE_URL also changed.

When I tried to load https://query1.finance.yahoo.com/v7/finance/quote?symbols=ORCL in the Browser, it mostly works, but sometimes I get

{"finance":{"result":null,"error":{"code":"Unauthorized","description":"Invalid Crumb"}}}

So it seems this may be some problem at Yahoo itself.

@code-monkey-101 where do the following packages come from?

com.ib.client com.ib.contracts com.ib.controller

That’s the Interactive Brokers API. You don’t need that.

I have the api as a dependency in my Maven project. How can I modify the QuotesRequest class as you have above? I do not know where to find or how to modify the files? I am using Eclipse IDE

Just download the source from github, unpack it into a subdirectory and add it to the build path

Is there a reason that this still hasn’t been added to the repo?

You need to use the v7 quotes query URL as the v6 one has been disabled.

Looking at the the 3.17.0 code base you need to change the getResult() method in the abstract class QuotesRequest in package yahoofinance.quotes.query1v7 and append the crumb to the URL.

    String url = YahooFinance.QUOTES_QUERY1V7_BASE_URL + "?" + Utils.getURLParameters(params);
    if (!CrumbManager.getCrumb().isEmpty()) {
        url = url + "&crumb=" + CrumbManager.getCrumb();
    }

@kallavikas

I’ve adapted the code to my use case, and I can confirm this solutions works (for now). Let’s hope it stays that way this time.

For me, the crumb is not null and adding it as a url parameter is sufficient. Don’t need to go back to v6.

I’m assuming that the content from which the crumb is extracted is regionalized then. I’m from Europe and the content from the base URL scrape does not get any pattern matches. Let’s hope v6 doesn’t suffer the same fate!

Hi,

I tried to use windscribe vpn. I found out that it did not work from Europe(Vienna, Oslo, Zagreb) but it worked from Tokyo and US. Then I tried also Frankfurt Germany and it worked. After that also other EU cities worked (Vienna, Oslo, Zagreb).

Change in QuotesRequest works for me. String crumb = CrumbManager.getCrumb(); params.put(“crumb”, crumb);

Thanks

Thank you @lumism! You are right, adding the crumb to the URL fixed it for me indeed on PC.

The pattern doesn’t match for me either but for some reason I can get the crumb without the cookie.