ibind: Disconnect / Errors from stream

Hey,

I tend to have this problem, where the ´md´ subscription works for a while but then crashes in the background and is not handled properly.

2024-09-05 01:24:40,317|E| IbkrWsClient: Status unauthenticated: {'authenticated': False}
2024-09-05 01:24:40.317 [ERROR] [ws_client_thread] - IbkrWsClient: Status unauthenticated: {'authenticated': False}
2024-09-05 01:24:40,459|I| IbkrWsClient: Unknown status response: {'topic': 'sts', 'args': {'competing': False}}
2024-09-05 01:24:40.459 [INFO] [ws_client_thread] - IbkrWsClient: Unknown status response: {'topic': 'sts', 'args': {'competing': False}}
2024-09-05 01:24:40,460|I| IbkrWsClient: Unknown status response: {'topic': 'sts', 'args': {'message': ''}}
2024-09-05 01:24:40.460 [INFO] [ws_client_thread] - IbkrWsClient: Unknown status response: {'topic': 'sts', 'args': {'message': ''}}
2024-09-05 01:24:40,460|E| IbkrWsClient: Status unauthenticated: {'authenticated': False}
2024-09-05 01:24:40.460 [ERROR] [ws_client_thread] - IbkrWsClient: Status unauthenticated: {'authenticated': False}

Are there any best practices for reconnecting to WS gracefully?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (14 by maintainers)

Most upvoted comments

Fantastic! Many thanks for your patience and help with this issue. I’m glad we’ve managed to catch it 👏👏👏

Music to my ears! Keep me posted, hopefully we got it this time 👍

Heh 😃 It seems solved for now. I will monitor until Friday, and close this issue, if success keeps repeating 👍

Updated to 0.1.8rc3 will get results tmrw 💯

Okay, I think I’ve got some good news. I found the core issue here.

The IbkrWsClient has a lot of code in place to handle drops in connectivity, but it wasn’t handling authentication drop as gracefully. While IBKR WS API sends a message when authentication is dropped, it doesn’t send a message when authentication is restored, leaving us thinking that authentication continues to be false. I’ve spent a while to add a bunch of code for handling authentication drops.

In short, it will close the WS connection completely, as this will get it into a reauthentication loop, attempting to reopen and resubscribe every few seconds. Once the authentication is restored, this should eventually succeed.

I’ve tested it with Market Data channel and can report that it is restored successfully.

Please try the ibind-0.1.8rc3 and let me know if it helps.

2024-11-04 16:32:09,794|I| IbkrWsClient: Starting
2024-11-04 16:32:09,795|I| IbkrWsClient: Trying to connect
2024-11-04 16:32:09,795|D| IbkrWsClient: Creating new WebSocketApp
2024-11-04 16:32:10,044|D| IbkrWsClient: Thread started (ws_client_thread-25700)
2024-11-04 16:32:10,067|I| IbkrWsClient: Connection open
2024-11-04 16:32:12,152|I| IbkrWsClient: Subscribing reattempt (2/5) smd+265598+{"fields": ["55", "71", "84", "86", "88", "85", "87", "7295", "7296", "70"]}
2024-11-04 16:32:12,253|I| IbkrWsClient: Subscribed: smd+265598+{"fields": ["55", "71", "84", "86", "88", "85", "87", "7295", "7296", "70"]}
data: [RETURNED CHANNEL DATA]
data: [RETURNED CHANNEL DATA]

[NOTE - THIS IS WHERE I FORCE A DROP IN AUTHENTICATION]

2024-11-04 16:32:19,944|E| IbkrWsClient: Status unauthenticated: {'authenticated': False}
2024-11-04 16:32:19,944|W| IbkrWsClient: Not authenticated, closing WebSocketApp
2024-11-04 16:32:19,947|I| IbkrWsClient: on_close
2024-11-04 16:32:19,947|I| IbkrWsClient: Invalidated subscription: md+265598
2024-11-04 16:32:19,947|I| IbkrWsClient: Connection closed
2024-11-04 16:32:19,947|D| IbkrWsClient: Thread stopped (ws_client_thread-25700)
2024-11-04 16:32:19,947|I| IbkrWsClient: Reconnecting
2024-11-04 16:32:19,948|I| IbkrWsClient: Trying to connect
2024-11-04 16:32:19,948|D| IbkrWsClient: Creating new WebSocketApp
2024-11-04 16:32:20,090|D| IbkrWsClient: Thread started (ws_client_thread-26364)
2024-11-04 16:32:20,111|I| IbkrWsClient: Connection open
2024-11-04 16:32:20,361|E| IbkrWsClient: Status unauthenticated: {'authenticated': False, 'competing': False, 'message': '', 'fail': '', 'serverName': 'JifZ37090', 'serverVersion': 'Build 10.30.1y, Nov 1, 2024 1:41:18 PM', 'username': 'ancov851'}
2024-11-04 16:32:20,361|W| IbkrWsClient: Not authenticated, closing WebSocketApp
2024-11-04 16:32:20,364|I| IbkrWsClient: on_close
2024-11-04 16:32:20,364|I| IbkrWsClient: Connection closed
2024-11-04 16:32:20,364|D| IbkrWsClient: Thread stopped (ws_client_thread-26364)
data: [RETURNED CHANNEL DATA - NO IDEA WHY THIS STILL GOT RETURNED HERE AFTER LOSING AUTHENTICATION, BUT OH WELL]
2024-11-04 16:32:30,273|E| IbkrWsClient: Reconnecting and recreating subscriptions stopped due to lack of authentication.

[THE FOLLOWING BLOCK REPEATS INDICATING THAT THE CONNECTION GETS REESTABLISHED PERIODICALLY]
2024-11-04 16:32:30,304|I| IbkrWsClient: Reconnecting
2024-11-04 16:32:30,304|I| IbkrWsClient: Trying to connect
2024-11-04 16:32:30,304|D| IbkrWsClient: Creating new WebSocketApp
2024-11-04 16:32:30,868|D| IbkrWsClient: Thread started (ws_client_thread-26444)
2024-11-04 16:32:31,240|I| IbkrWsClient: Connection open
2024-11-04 16:32:31,302|E| IbkrWsClient: Status unauthenticated: {'authenticated': False, 'competing': False, 'message': '', 'fail': '', 'serverName': 'JifZ37090', 'serverVersion': 'Build 10.30.1y, Nov 1, 2024 1:41:18 PM', 'username': 'ancov851'}
2024-11-04 16:32:31,302|W| IbkrWsClient: Not authenticated, closing WebSocketApp
2024-11-04 16:32:31,409|I| IbkrWsClient: on_close
2024-11-04 16:32:31,409|I| IbkrWsClient: Connection closed
2024-11-04 16:32:31,409|D| IbkrWsClient: Thread stopped (ws_client_thread-26444)
2024-11-04 16:32:41,272|E| IbkrWsClient: Reconnecting and recreating subscriptions stopped due to lack of authentication.
[THE ABOVE BLOCK REPEATS INDICATING THAT THE CONNECTION GETS REESTABLISHED PERIODICALLY]

2024-11-04 16:32:41,317|I| IbkrWsClient: Reconnecting
2024-11-04 16:32:41,317|I| IbkrWsClient: Trying to connect
2024-11-04 16:32:41,317|D| IbkrWsClient: Creating new WebSocketApp
2024-11-04 16:32:41,752|D| IbkrWsClient: Thread started (ws_client_thread-13028)
2024-11-04 16:32:42,047|I| IbkrWsClient: Connection open
2024-11-04 16:32:42,140|E| IbkrWsClient: Status unauthenticated: {'authenticated': False, 'competing': False, 'message': '', 'fail': '', 'serverName': 'JifZ37090', 'serverVersion': 'Build 10.30.1y, Nov 1, 2024 1:41:18 PM', 'username': 'ancov851'}
2024-11-04 16:32:42,140|W| IbkrWsClient: Not authenticated, closing WebSocketApp
2024-11-04 16:32:42,202|I| IbkrWsClient: on_close
2024-11-04 16:32:42,202|I| IbkrWsClient: Connection closed
2024-11-04 16:32:42,202|D| IbkrWsClient: Thread stopped (ws_client_thread-13028)
2024-11-04 16:32:52,110|E| IbkrWsClient: Reconnecting and recreating subscriptions stopped due to lack of authentication.
2024-11-04 16:32:52,141|I| IbkrWsClient: Reconnecting
2024-11-04 16:32:52,141|I| IbkrWsClient: Trying to connect
2024-11-04 16:32:52,141|D| IbkrWsClient: Creating new WebSocketApp
2024-11-04 16:32:52,716|D| IbkrWsClient: Thread started (ws_client_thread-26112)
2024-11-04 16:32:52,932|I| IbkrWsClient: Connection open
2024-11-04 16:32:53,023|E| IbkrWsClient: Status unauthenticated: {'authenticated': False, 'competing': False, 'message': '', 'fail': '', 'serverName': 'JifZ37090', 'serverVersion': 'Build 10.30.1y, Nov 1, 2024 1:41:18 PM', 'username': 'ancov851'}
2024-11-04 16:32:53,038|W| IbkrWsClient: Not authenticated, closing WebSocketApp
2024-11-04 16:32:53,147|I| IbkrWsClient: on_close
2024-11-04 16:32:53,147|I| IbkrWsClient: Connection closed
2024-11-04 16:32:53,147|D| IbkrWsClient: Thread stopped (ws_client_thread-26112)
2024-11-04 16:33:03,051|E| IbkrWsClient: Reconnecting and recreating subscriptions stopped due to lack of authentication.
2024-11-04 16:33:03,083|I| IbkrWsClient: Reconnecting
2024-11-04 16:33:03,083|I| IbkrWsClient: Trying to connect
2024-11-04 16:33:03,099|D| IbkrWsClient: Creating new WebSocketApp
2024-11-04 16:33:03,655|D| IbkrWsClient: Thread started (ws_client_thread-6896)
2024-11-04 16:33:03,966|I| IbkrWsClient: Connection open
2024-11-04 16:33:07,158|I| IbkrWsClient: Unknown status response: {'topic': 'sts', 'args': {}}
2024-11-04 16:33:07,314|E| IbkrWsClient: Status unauthenticated: {'authenticated': False}
2024-11-04 16:33:07,314|W| IbkrWsClient: Not authenticated, closing WebSocketApp
2024-11-04 16:33:07,515|I| IbkrWsClient: on_close
2024-11-04 16:33:07,515|I| IbkrWsClient: Connection closed
2024-11-04 16:33:07,515|D| IbkrWsClient: Thread stopped (ws_client_thread-6896)
2024-11-04 16:33:14,018|E| IbkrWsClient: Reconnecting and recreating subscriptions stopped due to lack of authentication.

[FINAL RECONNECITON, THIS TIME SUCCESSFUL]
2024-11-04 16:33:14,049|I| IbkrWsClient: Reconnecting
2024-11-04 16:33:14,049|I| IbkrWsClient: Trying to connect
2024-11-04 16:33:14,049|D| IbkrWsClient: Creating new WebSocketApp
2024-11-04 16:33:14,717|D| IbkrWsClient: Thread started (ws_client_thread-26384)
2024-11-04 16:33:15,058|I| IbkrWsClient: Connection open
2024-11-04 16:33:15,167|I| IbkrWsClient: Recreating 1/1 subscriptions: {'md+265598': {'status': False, 'data': {'fields': ['55', '71', '84', '86', '88', '85', '87', '7295', '7296', '70']}, 'needs_confirmation': True, 'subscription_processor': None}}
data: [RETURNED CHANNEL DATA]
2024-11-04 16:33:15,422|I| IbkrWsClient: Subscribed: smd+265598+{"fields": ["55", "71", "84", "86", "88", "85", "87", "7295", "7296", "70"]}
data: [RETURNED CHANNEL DATA]

Where can I track the changes made to the code for 0.1.8rc2?

On the master branch.

Trying the ibind==0.1.8rc1

Will let know, in few days

Hey @Tsopic thanks for your patience 🙏

I’ve spent a while investigating the reason behind the 'Topic unknown \'{"s\'' and I’ve concluded that IBind was not responding correctly to authenticated=False in status message. Quite frankly, in the old documentation the way the connection guide was laid out was unclear to me. I’ve just reviewed the current docs and updated the code accordingly.

In short:

  • We no longer take action upon seeing authenticated=False. This indicates that the session is not authenticated, and IBind has no means of resolving it. Previously I was incorrectly lead to believe that sending a {"session"="[SESSION_ID]"} payload would solve this, but it doesn’t. This is where 'Topic unknown \'{"s\'' was coming from. In reality, it needs to be solved externally, either by IBeam (if used) or by the user.
  • Now when creating the WebSocketApp, we acquire the session_id and use it as the cookie. This is along with what I can see in the docs at the moment: https://www.interactivebrokers.com/campus/ibkr-api-page/cpapi-v1/#ws-connection-guide

The rest of your log tells me that where you’re calling hard_reset(True) you should actually call it with hard_reset(False). The restart happens automatically when the WebSocketApp is closed, and doesn’t need to be initiated from hard_reset separately.

Try the ibind==0.1.8rc1 and let me know if it helps. Thanks for helping me catch this issue, this has been a great contribution 👍

I’m sorry @Tsopic I’m travelling at the moment. I should be able to investigate within the next couple of weeks. Thanks for bearing with me in the meantime and for keeping this thread active 👍

hey @Tsopic I’d appreciate if you could share the logs here for consistency and for other users who may run into this issue in the future. If you feel it’s too long you can make it collapsible in the following fashion:

.<details>
```
[YOUR CODE HERE]
```
</details>

Which will look like this: .

[YOUR CODE HERE]

Note the dot . character before <details>.

Thanks!