ibeam: Problem: 2-Factor Authentication using IBEAM_TWO_FA_HANDLER=EXTERNAL_REQUEST

Hi All! I am attempting to set up my 2FA automated authentication within IBeam and have hit a roadblock I have been frantically trying to debug before asking for your help.

To be quick - here’s the situation.

  1. I have set up a server endpoint http://{cloud_ip:cloud_port}/sms/code that returns the IBRK authentication code. NOTE The response is NOT wrapped in JSON - just the string of the authenticaton code. Per the wiki, I assume that is what IBeam is expecting. Is that correct?

  2. I have thoroughly tested my ‘auth_code_server’ and it works as expected: a. When IBeam triggers an authentication SMS from IBRK- the message is caught by the server and correctly parsed and is ready to provide the desired auth code to IBeam via a GET request to the http://{cloud_ip:cloud_port}/sms/code endpoint. b. the needed firewall port for the ‘auth_code_server’ is open - and tests have accessed it outside being on localhost. (I have manually logged into my live IBRK account from which the external server catches and saves the authentication code correctly) c. How long does IBeam wait to get that authentication code? And which IBEAM environment variable controls that?

  3. I dumped all my IBeam variables in an ‘env.list’ file which is the following:

$ cat /root/ibeam_files/env.list
IBEAM_ACCOUNT={my_account_name}
IBEAM_PASSWORD={my_password}
IBEAM_OUTPUTS_DIR=/srv/outputs
IBEAM_REQUEST_RETRIES=5
IBEAM_OAUTH_TIMEOUT=30
IBEAM_REQUEST_TIMEOUT=30
IBEAM_PAGE_LOAD_TIMEOUT=500
IBEAM_MAX_FAILED_AUTH=100
IBEAM_TWO_FA_HANDLER=EXTERNAL_REQUEST
IBEAM_EXTERNAL_REQUEST_URL=http://{cloud_ip:cloud_port}/sms/code
IBEAM_LOG_LEVEL=DEBUG
  1. When I spin up m IBeam container - the log looks like this (see below): Specifically: 1. Why doesn’t my IBEAM_EXTERNAL_REQUEST_URL show up in my ‘Configuration’ dictionary? (Oddly- IBEAM_TWO_FA_HANDLER=EXTERNAL_REQUEST does appear in the dictionary) 2. Should I define each of the environment variables needed for IBEAM_TWO_FA_HANDLER=EXTERNAL_REQUEST explicitly? I was assumimg the defaults for IBEAM_TWO_FA_HANDLER=EXTERNAL_REQUEST to be set.

My IBeam launch:

$ docker run --rm --name ibeam --env-file /root/ibeam_files/env.list --network tradenet -p 5000:5000 -v /root/ibeam_files/inputs_directory/:/srv/inputs -v /root/ibeam_files/outputs:/srv/outputs:rw voyz/ibeam:0.5.3

The Log:

2024-05-12 01:22:39,029|I| ############ Starting IBeam version 0.5.3 ############
2024-05-12 01:22:39,031|I| Custom conf.yaml found and will be used by the Gateway
2024-05-12 01:22:39,034|I| Secrets source: env
2024-05-12 01:22:39,035|I| Health server started at port=5001
2024-05-12 01:22:39,035|I| Configuration:
{'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 5, 'REQUEST_TIMEOUT': 30, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'HEALTH_SERVER_PORT': 5001, 'SECRETS_SOURCE': 'env', 'GCP_SECRETS_URL': None, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_INITIALISE': '/v1/api/iserver/auth/ssodh/init', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'NAME@@password', 'SUBMIT_EL': 'CSS_SELECTOR@@.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'TAG_NAME@@Client login succeeds', 'OAUTH_TIMEOUT': 30, 'PAGE_LOAD_TIMEOUT': 500, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 100, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'CLASS_NAME@@ibkey-promo-skip', 'AUTHENTICATION_STRATEGY': 'B', 'MAX_STATUS_CHECK_RETRIES': 120, 'MAX_REAUTHENTICATE_RETRIES': 3, 'UI_SCALING': 1.0, 'TWO_FA_EL_ID': 'ID@@twofactbase', 'TWO_FA_NOTIFICATION_EL': 'CLASS_NAME@@login-step-notification', 'TWO_FA_INPUT_EL_ID': 'ID@@chlginput', 'TWO_FA_HANDLER': 'EXTERNAL_REQUEST', 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'ID@@sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key', 'CUSTOM_TWO_FA_HANDLER': 'custom_two_fa_handler.CustomTwoFaHandler'}
2024-05-12 01:22:39,036|I| Gateway not found, starting new one...
2024-05-12 01:22:39,036|I| Note that the Gateway log below may display "Open https://localhost:[PORT] to login" - ignore this command.
2024-05-12 01:22:39,037|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 config file  : root/conf.yaml
2024-05-12 01:22:39,046|I| Gateway started with pids: [12]
2024-05-12 01:22:39,047|D| GET https://localhost:5000 (unverified)
2024-05-12 01:22:39,049|I| Cannot ping Gateway. Retrying for another 20 seconds
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-05-12 01:22:40,049|D| GET https://localhost:5000 (unverified)
2024-05-12 01:22:40,052|I| Cannot ping Gateway. Retrying for another 19 seconds
 -> mount demo on /demo
Java Version: 11.0.20
****************************************************
version: a27ed42161ad96c53e715ca5c5e3e3fa4cff5262 Mon, 24 Apr 2023 15:41:53 -0400
****************************************************
This is the Client Portal Gateway
for any issues, please contact api@ibkr.com
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo#/
2024-05-12 01:22:41,052|D| GET https://localhost:5000 (unverified)
2024-05-12 01:22:41,699|I| Gateway connection established
2024-05-12 01:22:41,700|D| POST https://localhost:5000/v1/api/tickle (unverified)
2024-05-12 01:22:41,834|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)
2024-05-12 01:22:41,835|I| Authentication strategy: "B"
2024-05-12 01:22:41,835|I| No active sessions, logging in...
2024-05-12 01:22:41,836|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2024-05-12 01:22:50,482|D| Targets: {'PASSWORD': Target(NAME@@password), 'SUBMIT': Target(CSS_SELECTOR@@.btn.btn-lg.btn-primary), 'SUCCESS': Target(TAG_NAME@@Client login succeeds), 'IBKEY_PROMO': Target(CLASS_NAME@@ibkey-promo-skip), 'TWO_FA': Target(ID@@twofactbase), 'TWO_FA_NOTIFICATION': Target(CLASS_NAME@@login-step-notification), 'TWO_FA_INPUT': Target(ID@@chlginput), 'TWO_FA_SELECT': Target(ID@@sf_select), 'USER_NAME': Target(NAME@@username), 'ERROR': Target(CSS_SELECTOR@@.xyz-errormessage)}
2024-05-12 01:22:50,532|I| Gateway auth webpage loaded
2024-05-12 01:22:50,532|I| Login attempt number 1
2024-05-12 01:22:50,738|D| target: Target(NAME@@username)
2024-05-12 01:22:56,036|I| Submitting the form
2024-05-12 01:23:26,744|E| Timeout reached when waiting for authentication. The website seems to not be loaded correctly. Consider increasing IBEAM_PAGE_LOAD_TIMEOUT.
Website URL: https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on

Exception:
  File "/srv/ibeam/ibeam_starter.py", line 168, in <module>
    success, shutdown, status = client.start_and_authenticate()
  File "/srv/ibeam/src/gateway_client.py", line 53, in start_and_authenticate
    success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 85, in try_authenticating
    return self._authentication_strategy_B(status, request_retries)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 140, in _authentication_strategy_B
    return self._log_in(status)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 151, in _log_in
    success, shutdown = self.login_handler.login()
  File "/srv/ibeam/src/handlers/login_handler.py", line 462, in login
    self.handle_timeout_exception(e, targets, driver, website_version, self.route_auth, self.base_url, self.outputs_dir)
  File "/srv/ibeam/src/handlers/login_handler.py", line 443, in login
    self.attempt(targets, wait_and_identify_trigger, driver)
  File "/srv/ibeam/src/handlers/login_handler.py", line 367, in attempt
    trigger, target = self.step_login(targets, wait_and_identify_trigger, driver, self.secrets_handler.account, self.secrets_handler.password, self.secrets_handler.key, self.presubmit_buffer)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/ibeam/src/handlers/login_handler.py", line 188, in step_login
    trigger, target = wait_and_identify_trigger(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/ibeam/src/handlers/login_handler.py", line 67, in _wait_and_identify_trigger
    trigger = WebDriverWait(driver, timeout).until(any_of(*expected_conditions))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until
    raise TimeoutException(message, screen, stacktrace)

  <class 'selenium.common.exceptions.TimeoutException'> Message:

2024-05-12 01:23:26,745|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f58aa2ed1d0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="d14bc5c4713c8fb18d8c0e4df48617f2")>
2024-05-12 01:23:26,814|I| Logging in failed
2024-05-12 01:23:26,819|I| Starting maintenance with interval 60 seconds
2024-05-12 01:24:26,819|I| Maintenance

Here’s my auth_server_code:

from flask import Flask, request, jsonify
import threading
import time
import logging
import json

## all messages at INFO level and above will be captured
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

app = Flask(__name__)

sms_token = None
lock = threading.Lock()

@app.route('/sms/code', methods=['GET'])
def get_sms_code():
    global sms_token
    try:

        logger.info('request to /sms/code')

        wait_count_max = 120  # seconds
        wait_count = 0
        while wait_count < wait_count_max:
            with lock:
                if sms_token is not None:
                    break
            time.sleep(1)
            wait_count += 1
            if wait_count % 20 == 0:
                logger.info(f'polling for verification code: {wait_count} of {wait_count_max} secs')
        if sms_token is None:
            logger.info(f'SMS polling interval expired.')

        with lock:
            if sms_token:
                rsp = f"Responding with verificationCode: {sms_token['verificationCode']} obtained at {sms_token['dateTime']}"
                logger.info(f'Response:\n {rsp}')
                response = sms_token['verificationCode']
                sms_token = None
                return response
            else:
                logger.info('No SMS token received.')
                return jsonify({'error': 'No SMS token received.'}), 404
    except Exception as e:
        logger.info(e)
        return jsonify({'error': 'Internal Error'}), 500

@app.route('/sms/incoming', methods=['POST'])
def post_sms_incoming():
    global sms_token

    VERFICATION_CODE_TAG = 'Your requested authentication code'
    try:
        logger.info(f'posted data:\n{json.dumps(request.json, indent=4)}')
        body = request.json.get('content', '')
        parts = body.split(':')

        if len(parts) == 2:
            message_tag, code_part = parts[0].strip(), parts[1].strip()
            if message_tag == VERFICATION_CODE_TAG and len(code_part) == 6:
                logger.info(f'Parsed Verification Code: {code_part}')
                with lock:
                    sms_token = {
                        'verificationCode': code_part,
                        'dateTime': time.strftime('%Y-%m-%dT%H:%M:%S')
                    }
            else:
                logger.info(f'Invalid verification code= {body}.')
        else:
            logger.info(f'Non-verification code SMS detected= {body}.')
    except Exception as e:
        logger.info(e)
    finally:
        response = {}
        if sms_token: response = sms_token
        return jsonify(response), 200



if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5051)

Thanks again for all your time, patience and help!

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Very helpful. Thanks!

On Thu, May 23, 2024 at 1:54 AM voyz @.***> wrote:

Ahh understood. IBeam interacts with the webpage using Selenium. If you’re not familiar with it - it is a library used for browser automation, ie. being able to programmatically interact with a browser. In fact, we can even interact with it without displaying anything, just “pretending there is a display”. Sort of like solving a Rubics cube blind folded. And that’s what IBeam does: it loads the login page without a display, finds the right input fields, types in our credentials and hits the ‘Login’ button. The complexity comes in due to the fact that there are a bunch of versions of that webpage that IBKR servers - as you’ve unfortunately noticed in this Issue - and there are various control flows - eg. there are 4 ways to complete 2FA. Hence, we need to recognise all of these and act accordingly. That’s what login_handler.py is doing, along with a bunch of functionalities that improve the experience, such as automatic shutdown on critical errors, error handling, screenshots, 2FA acquisition, etc. Hope that helps 👍

— Reply to this email directly, view it on GitHub https://github.com/Voyz/ibeam/issues/185#issuecomment-2126290405, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMDCZ6VW4IVKQGEVYNCSKLZDWABLAVCNFSM6AAAAABHULOYK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRWGI4TANBQGU . You are receiving this because you were mentioned.Message ID: @.***>