yfinance: Exception: yfinance failed to decrypt Yahoo data response with hardcoded keys, contact developers

@ValueRaider I maintain YahooFinancials. I fixed this earlier, had to dynamically regex the keys from the pages main.js file.

The function that gets the keys “_get_decryption_keys()” starts on line 147 of etl.py. https://github.com/JECSand/yahoofinancials/pull/125/files

Hopefully it helps yfinance:

import re
import time
import requests
from json import loads


# Class used to get data from urls (used by _get_decryption_keys())
class UrlOpener:
    user_agent_headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
    }

    def __init__(self, session=None):
        self._session = session or requests

    def open(self, url, user_agent_headers=None, params=None, proxy=None, timeout=30):
        response = self._session.get(
            url=url,
            params=params,
            proxies=proxy,
            timeout=timeout,
            headers=user_agent_headers or self.user_agent_headers
        )
        return response


# returns correct decryption {key:value} from BeautifulSoup(response.content, "html.parser")
def _get_decryption_keys(soup):
     key_count = 4
     re_script = soup.find("script", string=re.compile("root.App.main")).text
     re_data = loads(re.search("root.App.main\s+=\s+(\{.*\})", re_script).group(1))
     re_data.pop("context", None)
     key_list = list(re_data.keys())
     if re_data.get("plugins"):  # 1) attempt to get last 4 keys after plugins
         ind = key_list.index("plugins")
         if len(key_list) > ind+1:
             sub_keys = key_list[ind+1:]
             if len(sub_keys) == key_count:
                 re_obj = {}
                 missing_val = False
                 for k in sub_keys:
                     if not re_data.get(k):
                         missing_val = True
                         break
                     re_obj.update({k: re_data.get(k)})
                 if not missing_val:
                     return re_obj
     re_keys = []    # 2) attempt scan main.js file approach to get keys
     prefix = "https://s.yimg.com/uc/finance/dd-site/js/main."
     tags = [tag['src'] for tag in soup.find_all('script') if prefix in tag.get('src', '')]
     for t in tags:
         urlopener_js = UrlOpener()
         response_js = urlopener_js.open(t)
         if response_js.status_code != 200:
             time.sleep(random.randrange(10, 20))
             response_js.close()
         else:
             r_data = response_js.content.decode("utf8")
             re_list = [
                 x.group() for x in re.finditer(r"context.dispatcher.stores=JSON.parse((?:.*?\r?\n?)*)toString", r_data)
             ]
             for rl in re_list:
                 re_sublist = [x.group() for x in re.finditer(r"t\[\"((?:.*?\r?\n?)*)\"\]", rl)]
                 if len(re_sublist) == key_count:
                     re_keys = [sl.replace('t["', '').replace('"]', '') for sl in re_sublist]
                     break
             response_js.close()
         if len(re_keys) == key_count:
             break
     re_obj = {}
     missing_val = False
     for k in re_keys:
         if not re_data.get(k):
             missing_val = True
             break
         re_obj.update({k: re_data.get(k)})
     if not missing_val:
         return re_obj
     return None

Thanks @JECSand, I’ll code this up. Btw if you think there’s scope for merging the two projects, drop me an email. Seems inefficient to have multiple scrapers basically doing same thing.

EDIT: Code works, I just need to tidy etc.

Using the first key isn’t working anymore either. And I’m getting 10004 keys to pick from. They really are trying to make this hard. (And are probably watching this conversation).

PIP release out

Pull request @ #1353 , test it please

It is in and was working, for a whole 7 hours it seems before Yahoo changed yet again. Apparently there’s a better way to handle this, just needs someone to implement - #1338

When will this be fixed? When someone fixes it. Welcome to open-source software! If you need financials data then check out yahooquery module.

Finally - keep posts constructive. Posting “I have issue too” helps no one and pollutes the thread.

PIP release out.

waiting for him to fix, you can try to get the information differently

    ticker = yf.Ticker('AAPL')
    hist  = ticker.history(period="1d")
    df = hist.reset_index()
    
    for index, row in df.iterrows():
        index = row['Close']

Problem seems to have got worse overnight, affecting me now. Please keep thread posts constructive from now.

@dschaefer I’ve tried ALL the keys, none work.

@Sebastvin Impossible to know chance. Depends on community.

For those that need the data, look into yahooquery Python module. Accesses non-price data different way so not affected by encryption, but their documentation sucks.

Any chance this bug will be fixed today? I’m using yfinance for a project I’m handing in for graduation tomorrow. Previously, the error occurred sometimes and restarting the code solved the problem, but today the error appears every time when I want to run the code

Is last_price equivalent to regularMarketPrice ?

Yes, and to info["currentPrice"], I never saw a difference.

can’t find “beta”

I haven’t moved beta. The relocated keys are defined at top of quote.py.

Thanks @mmm0. Wow that is a lot of keys. I’ve made two more changes to branch, please run again:

1. discard new_keys that point to same value - potential fix
2. improve error message to print the values too

In case not obvious: feedback wanted

I have a delay between pulls, so mine is now fine. Thank you, @ValueRaider!

edit: I got one: /lib/python3.10/site-packages/yfinance/data.py", line 188, in decrypt_cryptojs_aes_stores raise Exception(“yfinance failed to decrypt Yahoo data response with hardcoded keys, contact developers”)

We think we’ve fixed in branch hotfix/decryption, please test and feedback. #1336

@SamZhang02 I’m fast-tracking #1300 now, see #1317 (merged into dev today).

Okay. In case it helps, the issue seems to occur with paginated/large? requests.

I’m getting this issue when I run a number of requests, but it seems OK if I just do a single request, by hand

I wonder if they’ve added some kind of frequency of request checks?

Just edit yfinance/data.py with the correct AES password would work:

def decrypt_cryptojs_aes_stores(data):
    encrypted_stores = data['context']['dispatcher']['stores']

    password = None

The password can be easily retrieved with a script.

msft.basic_info still working

Is the hotfix mentioned above not available in the updated 0.2.6 release? If not, will it be soon? I’m getting the following error message:

raise Exception(“yfinance failed to decrypt Yahoo data response with hardcoded keys, contact developers”)

Just curious as I can’t run the git branch (hotfix) on a cloud compute resource I’m using. Thanks for ya’ll’s hard work on this btw

@idan6767 #1080

In addition to @ValueRaider’s comment above, I tried out a few other functionalities locally, there are some alternatives for people who need to access some data but still want to stick to yfinance for now: Ticker.history() is still usable for historical price data Ticker.get_history_metadata() for non-price data such as exchange, timezone, currency, current price, etc.

I would agree with @ValueRaider’s proposal on #1300, as it might provide a more stable user-experience for most users of yfinance, while also be less likely to make Yahoo mad by sending expensive requests like .info

Not sure if this is significant or not, but I was able to get my routines working again by using Ticker() and pulling info individually instead of the Tickers() list call.

I was using this code to get tickers one at a time (including reading all tickers returned each time), with random delays between each get and retrying ones that failed - that was OK for most of yesterday, but is now broken - the JSON returned by get_json_data_stores() is now just empty.

from yfinance import data as yfdata

ask = sys.argv[1].upper()
ticker = yfdata.TickerData(ask)
js = ticker.get_json_data_stores()
store = js["StreamDataStore"]["quoteData"]

Yesterday the problem def seemed to be caused by getting too frequently - like they were throttling the query rate

Not sure if this is significant or not, but I was able to get my routines working again by using Ticker() and pulling info individually instead of the Tickers() list call.

Doesn’t work for me. I’m using Ticker() since the beginning. it was working intermittently yesterday(had to restart kernel on jupyter) but now its always this exception. msg.txt

@mm0 That new file has 2x more keys than the previous. And I’m not seeing the usual traffic spike that accompanies Yahoo breaking something (GitHub, Twitter, etc). I don’t understand …

Anyway I’ve pushed another change to hardcode the key (thanks @ifel), try that.

Seems like the first key works anyway. I simply removed the check for l == 1. Not sure that’s the right answer though.

I never said it would fix the decryption.

I updated my code to handle the exception and retry the request. That has fixed the issue for me.

I’ve quickly tested and working for me.

Error message tells us there are 8 fields when yfinance expected 1. But there is a bug in generating that error message, that’s why all print same - can you try again with Git branch hotfix/decryption.