client-python: list_aggs hangs sometimes hangs indefinitely

Describe the bug The list_aggs method in the Python Client will sometimes hang indefinitely without any response code or errors. It is not consistent about when it hangs and when it does not. There are times everything works correctly.

To Reproduce Run this script using your Polygon API credentials (I assume in this script they are stored as an environment variable)

Expected behavior This script runs successfully sometimes and assembles the dataframe of all stock prices. Often times, however, it instead proceeds through the dictionary of ticker/title pairs for about an hour before hanging indefinitely on a polygon_client.list_aggs call for a particular ticker. The ticker where the loop hangs is never the same. When I look in the Polygon dashboard online, I don’t see any errors in my API calls. This whole thing works and works and works until it stops working, at which point polygon_client.list_aggs doesn’t throw an error, it just hangs.

About this issue

Original URL
State: closed
Created 9 months ago
Comments: 21 (8 by maintainers)

Most upvoted comments

it does hold it in memory, but I have plenty to spare on my local and I’ll be sizing my VM’s appropriately.

Please give me a day or two to circle back here while I do some more testing myself. I’m not abandoning this, just need a bit of time to poke at it some more.

evanvolgas on Oct 11, 2023

Hi @evanvolgas, I was just looking through the SEC json file that you’re using to seed tickers and for some there isn’t a 1-1 mapping to polygon. We have the Tickers API where you can query all the ticker symbols https://polygon.io/docs/stocks/get_v3_reference_tickers. So, this might be a cause of some issues too in they you’re likely asking for data that doesn’t exist because the ticker is wrong.

For example from the SEC file:

    {
        "cik_str": 1067983,
        "ticker": "BRK-B",
        "title": "BERKSHIRE HATHAWAY INC"
    },

Then on the polygon side these would be:

{
    "results": [
        {
            "ticker": "BRK.A",
            "name": "Berkshire Hathaway Inc.",
            "market": "stocks",
            "locale": "us",
            "primary_exchange": "XNYS",
            "type": "CS",
            "active": true,
            "currency_name": "usd",
            "cik": "0001067983",
            "composite_figi": "BBG000DWCFL4",
            "share_class_figi": "BBG001S902J2",
            "last_updated_utc": "2023-10-09T00:00:00Z"
        },
        {
            "ticker": "BRK.B",
            "name": "BERKSHIRE HATHAWAY Class B",
            "market": "stocks",
            "locale": "us",
            "primary_exchange": "XNYS",
            "type": "CS",
            "active": true,
            "currency_name": "usd",
            "cik": "0001067983",
            "composite_figi": "BBG000DWG505",
            "share_class_figi": "BBG001S90346",
            "last_updated_utc": "2023-10-09T00:00:00Z"
        }
    ],
    "status": "OK",
    "request_id": "c010cbb661516cec616e0c2564b19501",
    "count": 2
}

You might have better luck using the https://polygon.io/docs/stocks/get_v3_reference_tickers API.

justinpolygon on Oct 10, 2023

google.api_core.exceptions.ServiceUnavailable: 503 recvmsg:Operation timed out google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 recvmsg:Operation timed out

Hi @evanvolgas, I’ve read through all the comments and these looks like google API 503 errors now and nothing to do with Polygon. Have you looked into your GCP logs? It’s totally reasonable to fetch dividends from Polygon. After you make a request to polygon and you print the len(results) or something to make sure you’re actually getting a result. Just since you seem to have a few things going on I’d like to make sure it’s actually a polygon issue and not after you received the results and then the next step in the process is hanging.

How are you calling polygon from within GCP? Does the google container have a 60 second limit or something that might be killing the requests? I don’t know enough about what’s happening to know for sure. But, maybe something to explore.

Are you able to test this locally or provide me with an example I can reproduce?

These all seem to be google libs related now:

Traceback (most recent call last): File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/api_core/retry.py”, line 204, in retry_target return target() File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/api_core/timeout.py”, line 120, in func_with_timeout return func(*args, **kwargs) File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/api_core/grpc_helpers.py”, line 77, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.ServiceUnavailable: 503 recvmsg:Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “/Users/evan/Documents/new/gh/arapbi/new-life/polygon_scraper_dividends.py”, line 38, in <module> logger.log_text(f’Retrieving dividend data for {ticker} // {title}', ) File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/cloud/logging_v2/logger.py”, line 199, in log_text self._do_log(client, TextEntry, text, **kw) File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/cloud/logging_v2/logger.py”, line 168, in _do_log client.logging_api.write_entries(entries, partial_success=True) File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/cloud/logging_v2/_gapic.py”, line 163, in write_entries self._gapic_api.write_log_entries(request=request) File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/cloud/logging_v2/services/logging_service_v2/client.py”, line 727, in write_log_entries response = rpc( File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py”, line 131, in call return wrapped_func(*args, **kwargs) File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/api_core/retry.py”, line 366, in retry_wrapped_func return retry_target( File “/Users/evan/.pyenv/versions/ds/lib/python3.10/site-packages/google/api_core/retry.py”, line 220, in retry_target raise exceptions.RetryError( google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 recvmsg:Operation timed out

justinpolygon on Oct 9, 2023

Okay, thanks for letting me know. Yeah, I was wondering about that since the error message seemed to imply it already retried on the 504. Using a higher retry count like 10 might help. Let me know how it works out. Ideally, you’ll never see these 504 type errors and we can dig into it on the support side if you have the request id’s for the ones that fail.

justinpolygon on Oct 9, 2023

RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 504 Deadline Exceeded

Great, thank you. So, it look like you get a 504 http error at some point during the request run and then entire loop aborts. What client version are you running by change? The reason I ask, is that in version 1.12.8 we added logic to retry on those types of error and that’ll likely solve your problem. Just run pip install -U polygon-api-client.

Also, you can update the client to retry X number of times and wait for things to get better if there is an error. You can do this by adding something like client = RESTClient(retries=10) to the client. Ideally, if you update the client and set retries the client will hopefully just retry and all this will be seamless on your end.

justinpolygon on Oct 9, 2023

Hi @evanvolgas, I’m sorry to hear you’re experiencing an inconsistency with the list_aggs method and hopefully we work together to get to the root of the issue. For a more in-depth look at what might be causing the hang, it might be beneficial to enable request debugging. By doing this, you can trace the underlying API calls and gather more insight. You can turn on debugging as outlined in our official documentation. In short:

client = RESTClient(trace=True)

Once enabled, this will output the request and response details, including a request-id which can be useful for us to trace the request on our side. In addition to this, it would be helpful if you could provide some context:

How frequently do you encounter this issue? (Is it a matter of days, weeks?)
Roughly, how many requests precede the hang?

If possible, please include some additional debugging logs immediately before and after the API calls to Polygon, so we can pinpoint exactly where the program gets stuck. I imagine you’re doing something with the API results after you fetch them so I just wanted to make sure there wasn’t a hang there too.

justinpolygon on Oct 9, 2023