yfinance: Scraper error "TypeError: string indices must be integers" - Yahoo decrypt fail

Updates

2023 January 13

By the time of posting the issue (2023 January 12), the issue only occured sometimes. The library is now (2023 January 13) completely broken and I am unable to retrieve any stock informatio

2023 January 14

Fix has been merged to the branch dev

Info about your system:

yfinance version: 0.2.3 Operating system: macOS Monteray 12.0.1

Snippet that can recreate the error

stock = yf.Ticker("^GSPC")
info = stock.info

Error

Message:TypeError: string indices must be integers It seems to be a problem where the scraper is not scraping the correct information, leading to a crash.

Traceback:

Traceback (most recent call last):
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/apscheduler/executors/base_py3.py", line 30, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
  File "/home/2022/szhang139/repos/STONK/src/main.py", line 61, in notify
    market = get_major_index(f'Market Close - {daytime.today_date()}')
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 63, in get_major_index
    sp500 = get_stock('^GSPC')
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 41, in get_stock
    stock_info = get_stock_info(stock_name)
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 8, in get_stock_info
    info = stock.info
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/ticker.py", line 138, in info
    return self.get_info()
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/base.py", line 894, in get_info
    data = self._quote.info
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 27, in info
    self._scrape(self.proxy)
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 58, in _scrape
    quote_summary_store = json_data['QuoteSummaryStore']

Frequency

The error occurs in no apparent pattern. Every time it occurs, it seem to persist for some range of time before it recovers back to normal. n.

About this issue

Original URL
State: closed
Created a year ago
Reactions: 56
Comments: 36 (1 by maintainers)

Most upvoted comments

PIP release out

+20

ValueRaider on Jan 14, 2023

We know the decryption is failing. Please keep future comments related to debugging or fixing.

+14

ValueRaider on Jan 13, 2023

I believe the cause of this relates to the underlying data structure this library scrapes from Yahoo Finance has changed. They introduced encryption on the ticker data before the turn of the year, which was fixed here: https://github.com/ranaroussi/yfinance/pull/1253/commits/8e5f0984af347afda6be74b27a989422e49a975b. This time around they changed the parameters on the encryption.

+13

chris-hmmr on Jan 13, 2023

I think 93634354fa08860f003e key/value will change with time…

         encrypted_stores = data['context']['dispatcher']['stores']
-        _cs = data["_cs"]
-        _cr = data["_cr"]
-
-        _cr = b"".join(int.to_bytes(i, length=4, byteorder="big", signed=True) for i in json.loads(_cr)["words"])
-        password = hashlib.pbkdf2_hmac("sha1", _cs.encode("utf8"), _cr, 1, dklen=32).hex()
+        password_key = next(key for key in data.keys() if key not in ["context", "plugins"])
+        password = data[password_key]

         encrypted_stores = b64decode(encrypted_stores)

and it seems this

-    plaintext = unpad(plaintext, 16, style="pkcs7")
+    plaintext = unpad(plaintext, 128, style="pkcs7")

+12

alexa-infra on Jan 13, 2023

PR #1297 seems to work. But before I merge & release, I want to make other changes to reduce Yahoo server load - see #1300 for discussion.

ValueRaider on Jan 14, 2023

PR #1297 seems to work. But before I merge & release, I want to make other changes to reduce Yahoo server load - see #1300 for discussion.

I’m not sure it makes sense to conflate the release of a critical fix with a potential efficiency improvement. 100% of the yfinance community will benefit from #1297 urgently being released.

cmjordan42 on Jan 14, 2023

Thank you very much - your time and expertise is appreciated by the community. I uninstalled and reinstalled latest…everything working perfectly! Again - many thanks!

interstormtech on Jan 14, 2023

Where Yahoo used to provide the decryption parameters, there is this new key-value. It appears to be constant.

'93634354fa08860f003e': '4f566bf243f93d68718581c573aee0ff86456eef74bf9c33244c765b6ae2470059db233bf70a4fb3c16c24047befc4e993e717df7cbd4f1bebddb9649abdf5e3'

Any ideas?

ValueRaider on Jan 13, 2023

I would appreciate an expedited roll out. I use Yfinance pretty extensively and all my code is now defunct. Sent from my iPhoneOn Jan 14, 2023, at 11:52 AM, ValueRaider @.***> wrote: I think it does make sense to conflate. Why is Yahoo encrypting this data? Is it to stop scraping because of the high server load? Then it’s in our benefit to reduce load of yfinance. What will Yahoo do next? Plus this will not delay much, most groundwork was done weeks-months ago.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

bruce1095 on Jan 14, 2023

Download latest PIP release to get the fix. That’s how bug fixes work.

ValueRaider on Jan 15, 2023

I think it does make sense to conflate. Why is Yahoo encrypting this data? Is it to stop scraping because of the high server load? Then it’s in our benefit to reduce load of yfinance. What will Yahoo do next?

Plus this will not delay much, most groundwork was done weeks-months ago.

EDIT: Actually, makes more sense to release the info change afterwards, because will require a new major release 0.3. So fix will be released first asap.

ValueRaider on Jan 14, 2023

That is working for me though I had to leave the unpad block_size arg at 16 when using usePycryptodome. But also works without usePycryptodome.

dschaefer on Jan 14, 2023