ibeam: Ibeam unstable - disconnect and never reconnects

Describe the bug A clear and concise description of what the bug is.

There seems to be a problem with the “keep alive” feature. It does not happen always and I can’t reproduce it, but it has been happening consistently after a while of the gateway running, ever since the login procedure of the gateway changed and you upgraded ibeam to 0.5.7

The problem is when calling tickle, you expect the response to be present, which is not, and it throws

│ ----------------------------------------                                                                                                                                                                                                                                                                                                                                 │
│ Exception occurred during processing of request from ('10.0.40.47', 33210)                                                                                                                                                                                                                                                                                               │
│ Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                       │
│   File "/usr/local/lib/python3.11/socketserver.py", line 317, in _handle_request_noblock                                                                                                                                                                                                                                                                                 │
│     self.process_request(request, client_address)                                                                                                                                                                                                                                                                                                                        │
│   File "/usr/local/lib/python3.11/socketserver.py", line 348, in process_request                                                                                                                                                                                                                                                                                         │
│     self.finish_request(request, client_address)                                                                                                                                                                                                                                                                                                                         │
│   File "/usr/local/lib/python3.11/socketserver.py", line 361, in finish_request                                                                                                                                                                                                                                                                                          │
│     self.RequestHandlerClass(request, client_address, self)                                                                                                                                                                                                                                                                                                              │
│   File "/usr/local/lib/python3.11/socketserver.py", line 755, in __init__                                                                                                                                                                                                                                                                                                │
│     self.handle()                                                                                                                                                                                                                                                                                                                                                        │
│   File "/usr/local/lib/python3.11/http/server.py", line 432, in handle                                                                                                                                                                                                                                                                                                   │
│     self.handle_one_request()                                                                                                                                                                                                                                                                                                                                            │
│   File "/usr/local/lib/python3.11/http/server.py", line 420, in handle_one_request                                                                                                                                                                                                                                                                                       │
│     method()                                                                                                                                                                                                                                                                                                                                                             │
│   File "/srv/ibeam/src/health_server.py", line 17, in do_GET                                                                                                                                                                                                                                                                                                             │
│     return self._ready()                                                                                                                                                                                                                                                                                                                                                 │
│            ^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                 │
│   File "/srv/ibeam/src/health_server.py", line 31, in _ready                                                                                                                                                                                                                                                                                                             │
│     status = check_status()                                                                                                                                                                                                                                                                                                                                              │
│              ^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                              │
│   File "/srv/ibeam/src/handlers/http_handler.py", line 248, in get_status                                                                                                                                                                                                                                                                                                │
│     status.authenticated = status.response['iserver']['authStatus']['authenticated']                                                                                                                                                                                                                                                                                     │
│                            ~~~~~~~~~~~~~~~^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                    │
│ KeyError: 'iserver'                                                                                                                                                                                                                                                                                                                                                      │
│ ----------------------------------------

Environment IBeam version: 0.5.7 Docker image or standalone: docker OS: intel, on aws linux

Suggest a Fix

I’m trying to look into the code and I have a feeling that the issue might be that IB changed something in their return code on failure to tickle

What I find maybe weird is that in try_request, you do not set the Status.session to false on either 401 or unknown error code

                if e.code == 401:
                    # the gateway is running but there is no active session
                    pass

                elif e.code == 500 and 'Internal Server Error' in str(e):
                    _LOGGER.error(f'IBKR server error: "{
                                  e}". One of reasons for this error is IBKR server restart.')
                    status.session = False  # ensure we reauthenticate

                elif e.code == 503 and 'Service Unavailable' in str(e):
                    _LOGGER.error(f'IBKR service unavailable: "{
                                  e}". It seems IBKR servers are not ready to handle requests. We may need to wait until the servers are ready.')
                    status.session = False  # ensure we reauthenticate

                else:  # todo: possibly other codes could appear when not authenticated, fix when necessary
                    try:
                        raise RuntimeError('Unrecognised HTTPError') from e
                    except Exception as ee:
                        _LOGGER.exception(ee)

that means that the get_status method then tries to read the response body, which is not there, and throws

Is it the place to look into? I’m trying to add some debug in a local image and make it crash to give you more info

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (7 by maintainers)

Most upvoted comments

awesome.

I have been running 0.5.7 and 0.5.8-rc1 in parallel for 1 hour. 0.5.7 failed 0.5.8-rc1 did show the output you mentioned and reauthenticated

looks good!

2025-03-19 11:11:06,969|I| Maintenance
2025-03-19 11:11:07,079|I| Gateway running and authenticated, session id: ca12d5c419c713b84ad33a87b6fe4d0d, server name: JifZ10088
2025-03-19 11:12:06,971|I| Maintenance
2025-03-19 11:12:07,138|I| Gateway running and authenticated, session id: ca12d5c419c713b84ad33a87b6fe4d0d, server name: JifZ10088
2025-03-19 11:13:06,967|I| Maintenance
2025-03-19 11:13:08,680|E| Error: "no session" returned.
2025-03-19 11:13:08,680|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)
2025-03-19 11:13:08,680|I| Authentication strategy: "B"
2025-03-19 11:13:08,680|I| No active sessions, logging in...
2025-03-19 11:13:08,681|I| Loading auth webpage at http://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2025-03-19 11:13:15,681|I| Gateway auth webpage loaded
2025-03-19 11:13:15,681|I| Login attempt number 1
2025-03-19 11:13:20,842|I| Submitting the form
2025-03-19 11:13:21,543|I| Switching to paper mode and reattempting to submit the form
2025-03-19 11:13:27,672|I| Webpage displayed "Client login succeeds"
2025-03-19 11:13:28,677|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0xffffbbd2a690> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="07830f82f3320ccf4cb67cafc0404794")>
2025-03-19 11:13:28,719|I| Logging in succeeded
2025-03-19 11:13:28,851|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='ca12d5c419c713b84ad33a87b6fe4d0d', server_name='JifZ10088', server_version='Build 10.34.1b, Mar 17, 2025 7:04:02 PM', expires=596640)
2025-03-19 11:13:28,930|I| Gateway running and authenticated, session id: ca12d5c419c713b84ad33a87b6fe4d0d, server name: JifZ10088
2025-03-19 11:14:06,970|I| Maintenance

@Voyz no worries at all!

@lth98 is this good to close?

@weklund my bad! just updated latest, thanks for pointing it out 🙈

No worries, @Voyz. The system with 0.5.8 looks stable whole day today. If anything pops up, I’ll raise a new ticket. Thanks.

hey guys, I’ve just published 0.5.8 with this patch as it seems things are working well with it 👍

@lth98 sorry, I only just saw your updates 🙈 Could you create a new issue and post them there? Please include your full IBeam log. Also, do note that we don’t get notifications for comment updates, so in the future try creating new comments instead of updating existing one so that we can see them right away

Hey, thanks for sending these over. I’ve updated IBeam to handle this response better in try_request. Can you try image voyz/ibeam:0.5.8-rc1 and let me know if it handles the issue for you? It should print Error: "no session" returned. when that response is found, but handle it gracefully by logging in once again.