ibeam: 503 Service Unavailable (IBKR/Gateway issue)

Describe the bug IBKR/Gateway sometimes runs into returning a 503 Service Unavailable error. In this issue we try to understand what can be done to overcome it.

Environment IBeam-independent, it is the Gateway’s issue

Additional context So far it has been observed when calling:

  • /iserver/account/{accountId}/orders

In https://github.com/Voyz/ibeam/issues/7#issuecomment-945684378 @Wenuka outlines:

when I tried to place an order (using https://localhost:{port_num}/v1/api/iserver/account/{accountId}/orders API) I am getting a 503 response as below,

{'_content': b'', '_content_consumed': True, '_next': None, 'status_code': 503, 'headers': {'x-response-time': '10000ms', 'X-Cnection': 'close', 'Expires': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Pragma': 'no-cache', 'Date': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Connection': 'close', 'Server-Timing': 'cdn-cache; desc=MISS, edge; dur=4, origin; dur=10019', 'Vary': 'Origin', 'Transfer-Encoding': 'chunked'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f15b168d520>, 'url': 'https://localhost:5000/v1/api/iserver/account/XXXXXX/orders', 'encoding': None, 'history': [], 'reason': 'Service Unavailable', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(seconds=10, microseconds=68901), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f15b166aeb0>}

Then I killed the docker session and re-run using the changed proxyRemoteHost value and it worked as expected.


In https://github.com/Voyz/ibeam/issues/19#issuecomment-946766857 @isque03 outlines:

Gateway logs show this:

GET /v1/api/tickle,200|61ms 02:43:34.032 DEBUG vert.x-eventloop-thread-0 ClientPortalService : tickle > {“session”:“asdasdadasd”,“ssoExpires”:469642,“collission”:false,“userId”:123456,“iserver”:{“authStatus”:{“authenticated”:true,“competing”:true,“connected”:true,“message”:“Another session with the same user name has disconnected your session.”,“MAC”:“00:00:00:00:00:00”}}}

The only workaround I have found so far is to manually log into IBKR on the mobile app (or I suppose web/desktop). It will say “another session is in use, disconnect?” I click “yes” to disconnect, then I explicitly click the logout in the mobile app. After this I’m able to get a properly working brokerage session from the logout/reauthenticate flow.

My suspicion is that it’s the call for brokerage session ssodh/init that the gateway makes. See this document again: https://www.tradersinsight.news/ibkr-quant-news/tutorial-web-api-connect-to-brokerage-session/

Specifically the compete param in here:

`SSODH Init Request

POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init

The body, of type x-www-form-urlencoded, must have the following parameters:

      machineId (String)       8 char alphanumberic string e.g. CCCCCC01-99
      mac (String)                   six 2-char alphanumerical pairs separated by a hyphen
      **compete (Boolean)      whether or not the session should compete, usually set to false**
      locale (String)                set to “en_US
      username (String)        set to dash “-“`

The compete param here recommends setting it to false. What I think this does is trigger any other brokerage session to be invalidated, and allow a new one to be created. Setting compete “true” I beleive means that it will not kick out the other session. (Even though that seems sort of backwards from what you’d expect) The gateway passes complete:true. (From gateway logs) post={“username”:“XXXX”,“machineId”:“12349”,“compete”:true}

Programmatic workarounds here? I think the only choices are:

  1. File a ticket with IBKR and see if they’ll add the ability to control the compete param in the gateway
  2. Have iBeam actually implement the ssodh/init /complete flows to explicitly trigger brokerage session creatation (this would replace the call to /reauthenticate). I have not proven this compete true|false theory, so I wouldn’t recommend going down this path yet.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 4
  • Comments: 16 (7 by maintainers)

Most upvoted comments

@Voyz The issue with no accounts being returned was one of the biggest problems I had after switching the proxyRemoteHost. Sometimes it only went away the next day. Since using docker-compose I haven’t this issue.

@gikeberyfinn thanks for your kind words and sorry you’re running into 503. Other than reading other users’ suggestions, I’d say getting in touch with IBKR support and talking to them about it would be the best choice, as this is unrelated to IBeam itself

I am using voyz/ibeam:0.4.0-rc1. I observed that the gateway is running and authenticated. But the /iserver/accounts returns an empty account with 200 status. And iserver/marketdata/snapshot returns 500. I am not sure if this is also related to the issue. Any suggestions?

Ibeam log 2021-11-11 17:27:31,747|I| Gateway running and authenticated 2021-11-11 17:28:31,751|I| Gateway running and authenticated 2021-11-11 17:29:31,721|I| Gateway running and authenticated 2021-11-11 17:30:31,715|I| Gateway running and authenticated

IBKR Gateway log 11-11-2021 17:28:02 UTC: DEBUG : _make_request : https://localhost:5000 "GET /v1/api/iserver/accounts HTTP/1.1" 200 None 11-11-2021 17:28:02 UTC: DEBUG : _make_request : https://localhost:5000 "GET /v1/api/iserver/marketdata/snapshot?conids=199169591&fields=31%2C55%2C70%2C71%2C74 HTTP/1.1" 500 None 11-11-2021 17:28:02 UTC: ERROR : invest : There are errors when investing 500 Server Error: Internal Server Error for url: https://localhost:5000/v1/api/iserver/marketdata/snapshot?conids=199169591&fields=31%2C55%2C70%2C71%2C74

I’ve been using voyz/ibeam:0.4.0-rc1 with docker-compose now for two weeks. I’ve also added cronjobs to stop ibeam outside of trading hours and restart it before trading opens for the region I trade in. Since those two weeks I didn’t have any troubles with getting errors on the trading endpoints. I request order data frequently and before the newest rc1 and docker-compose changes if had to change proxyRemoteHost atleast once a week. Previously I didn’t use docker-compose and had the ibeam-container running indefinitely.

Since I don’t persist the docker logs in any way, I loose all logs, since docker-compose cleans up the containers daily. So I can’t check if the logout happend at some point before. What I also think helps alot now, is that each day ibeam and the IB gateway get a complete clean new container without any previous caches or whatnot. This might be the main reason, that I’ve not had the issue anymore.

I can also only reccomend everyone to implement incremental retry logic for api calls in their trading bots. I’ve set the ibeam maintance interval to 15s so if the gateway connections gets lost or anything it will be re-established asap. Here’s a small outtake of my logs showing how the incremental retries gives ibeam some time to login again and after a few retries everything will continue again: image