degiro-connector: Is there a way to access fault data on API requests, instead of returning `None`?

My bad, will respect the process.

That’s a great work @funnel20 ! Will check that in detail the following days and give you my feedback.

I’ve been prototyping today and discovered that the exceptions that are raised by the requests library already have the request and response objects as can be seen in the source code of the RequestException. All other exceptions, such as HTTPError, ConnectTimeout, ConnectionError and JSONDecodeError, inherit from RequestException. So no need for custom exception(s) 👍

Proposed changes

Based on the existing code of version 2.0.14 (for example for update_order()) I made a generic request handler for fictitious call get_data of class API:

class API:
    def get_data(self, raise_exception:bool=False) -> bool:
        try:
            # Make GET request to server and use JSON response body:
            response_raw = requests.request("GET", "https://api.github.com/users/google")
            response_raw.raise_for_status()
            response_dict = response_raw.json()
        except requests.exceptions.HTTPError as exception:
            self.handle_exception(exception, raise_exception, True)
            return None
        except Exception as exception:
            self.handle_exception(exception, raise_exception)
            return None

        # Do something with `response_dict`...
        
        return response_raw.status_code == 200

Notes

1. The new optional input parameter raise_exception that defaults to False for a seamless migration of existing users.
2. For the simplicity of this proposal I’ve left out the input parameter raw.
3. The current code handles specific HTTPError (to print details to terminal) and the generic Exception.

These 2 exceptions are handled generically in method handle_exception():

• The existing behaviour is performed when raise_exception is False. The only new feature is the Tip in the terminal about this new input parameter.
• The new behaviour when raise_exception is True, is as simple as forwarding the existing exception.

    def handle_exception(self, exception:Exception, raise_exception:bool, is_http_error:bool=False):
        # Show details in terminal.
        # This is the current behaviour of version 2.0.14, see: https://github.com/b-aeby/degiro-connector/blob/c9a9a236eb89a50f0141a16603d028eaef3e8c58/degiro_connector/trading/actions/action_update_order.py#L126-L129
        if is_http_error:
            status_code = getattr(exception.response, "status_code", "No status_code found.")
            text = getattr(exception.response, "text", "No text found.")
            logging.error(status_code)
            logging.error(text)
        else:
            logging.error(exception)

        if raise_exception:
            # Forward standard exceptions, such as `HTTPError`, `ConnectTimeout`, `ConnectionError` and `JSONDecodeError`:
            raise(exception)
        else:
            # Migration idea: Teach user about new feature:
            logging.info("Tip: set property `raise_exception` to `True` when calling this function and catch these fault details as exception `{0}`.".format(ExceptionUtils.get_full_class_name(exception)))

Notes

1. The line to show the tip contains a reference to ExceptionUtils.get_full_class_name(). It will return a string like “requests.exceptions.ConnectTimeout”. The source code can be found in the executable example at the bottom of this post.

Existing usage

Existing users will still use the function call without argument:

    # Init API:
    api = API()

    # Make an API call:
    my_data = api.get_data()    
    logging.info("Received data from server:\n{0}".format(my_data))

This will either:

• return True for HTTP 200
• return None and show logs in the terminal in case of an error. For example for HTTPError:

  ERROR:root:404
  ERROR:root:{"message":"Not Found","documentation_url":"https://docs.github.com/rest"}
  INFO:root:Tip: set property `raise_exception` to `True` when calling this function and catch these fault details as exception `requests.exceptions.HTTPError`.
  INFO:root:Received data from server:
  None
  

New usage

Add parameter raise_exception to the function call and set it to True. Embed the call in a try/except condition and handle whatever specific or generic exceptions:

    # Init API:
    api = API()

    # Make an API call:
    try:
        my_data = api.get_data(raise_exception=True)
        logging.info("Received data from server:\n{0}".format(my_data))
    except requests.exceptions.HTTPError as exception:
        logging.error("HTTPError: {0}".format(exception))
        logging.error("Server responded with HTTP Status code {0} and {1} on {2} request.".format(ExceptionUtils.http_status_code(exception), exception.response, exception.request))
    except requests.exceptions.ConnectTimeout as exception:
        logging.error("ConnectTimeout: {0}".format(exception))
    except requests.exceptions.ConnectionError as exception:
        logging.error("ConnectionError: {0}".format(exception))
    except Exception as exception:
        logging.error("Exception: Failed to connect to server with exception: {0}".format(exception))

This will either:

• return True for HTTP 200
• return None and show the same logs as the existing code (except the tip) in the terminal in case of an error. For example for HTTPError, the first 2 log lines come from the API. The other 2 demonstrate the existence of the request and response objects in the returned exception:

  ERROR:root:404
  ERROR:root:{"message":"Not Found","documentation_url":"https://docs.github.com/rest"}
  ERROR:root:HTTPError: 404 Client Error: Not Found for url: https://api.github.com/users/google
  ERROR:root:Server responded with HTTP Status code 404 and <Response [404]> on <PreparedRequest [POST]> request.
  

Notes

1. The line to show the HTTPError details contains a reference to ExceptionUtils.http_status_code(). It’s a convenient method to return the HTTP Status Code as integer (e.g. 404). The source code can be found in the executable example at the bottom of this post. Of course this is also possible by exception.response.status_code, but that might raise a AttributeError in case the exception has no response attribute.

Working example code

Here is the code for a working example. It shows the current output in the Terminal, and the new output when using raise_exception. Just copy/paste everything in a file error-handling.py and run. Note that the comments in get_data() contain instructions to force different exceptions.

import logging
import requests

class ExceptionUtils:
    def get_full_class_name(obj) -> str:
        """
        Returns the full class name of an object.

        Example
        -------
        Get the full class name of a specific `Exception`, for instance from the `requests` library:
        ```
        try:
            response = requests.request("GET", "https://wwz.google.com")
        except Exception as exception:
            print(get_full_class_name(exception))
        ```
        This will print `requests.exceptions.ConnectionError`.
        """
        # See: https://stackoverflow.com/a/58045927/2439941
        module = obj.__class__.__module__
        if module is None or module == str.__class__.__module__:
            return obj.__class__.__name__
        return module + '.' + obj.__class__.__name__

    def http_status_code(exception:Exception) -> int:
        """
        Convenience method to return the HTTP reponse status code from an `Exception`.

        When there is no `response` object or it has no information about the status code, `0` is returned.

        For the description of the HTTP status codes, see: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes.
        """
        if hasattr(exception, 'response') and hasattr(exception.response, 'status_code'):
            return exception.response.status_code
        else:
            return 0


class API:
    def get_data(self, raise_exception:bool=False) -> bool:
        try:
            # To force the following exceptions:
            # - HTTPError       : change method from "GET" to "POST"
            # - ConnectionError : change "github" in `url` to "git44hub"
            # - ConnectTimeout  : change `timeout` from 30.001 to 0.001
            # - JSONDecodeError : change `url` from "https://api.github.com/users/google" to "https://api.github.com/zen"

            # Make GET request to server and use JSON response body:
            response_raw = requests.request("GET", url="https://api.github.com/users/google", timeout=30.001)
            response_raw.raise_for_status()
            response_dict = response_raw.json()
        except requests.exceptions.HTTPError as exception:
            self.handle_exception(exception, raise_exception, True)
            return None
        except Exception as exception:
            self.handle_exception(exception, raise_exception)
            return None

        # Do something with `response_dict`...

        return response_raw.status_code == 200

    def handle_exception(self, exception:Exception, raise_exception:bool, is_http_error:bool=False):
        # Show details in terminal.
        # This is the current behaviour of version 2.0.14, see: https://github.com/b-aeby/degiro-connector/blob/c9a9a236eb89a50f0141a16603d028eaef3e8c58/degiro_connector/trading/actions/action_update_order.py#L126-L129
        if is_http_error:
            status_code = getattr(exception.response, "status_code", "No status_code found.")
            text = getattr(exception.response, "text", "No text found.")
            logging.error(status_code)
            logging.error(text)
        else:
            logging.error(exception)

        if raise_exception:
            # Forward standard exceptions, such as `HTTPError`, `ConnectTimeout`, `ConnectionError` and `JSONDecodeError`:
            raise(exception)
        else:
            # Migration idea: Teach user about new feature:
            logging.info("Tip: set property `raise_exception` to `True` when calling this function and catch these fault details as exception `{0}`.".format(ExceptionUtils.get_full_class_name(exception)))


def main():
    # SETUP LOGGING LEVEL
    logging.basicConfig(level=logging.INFO)

    # Init API:
    api = API()

    # Make an API call:
    # 1) Seamless migration for existing usage without the new optional `raise_exception`
    my_data = api.get_data()    
    logging.info("Received data from server:\n{0}".format(my_data))

    # 2) New usage with the new optional `raise_exception` to parse all kind of faults:
    logging.info("Use `raise_exception`")
    try:
        my_data = api.get_data(raise_exception=True)
        logging.info("Received data from server:\n{0}".format(my_data))
    except requests.exceptions.HTTPError as exception:
        logging.error("HTTPError: {0}".format(exception))
        logging.error("Server responded with HTTP Status code {0} and {1} on {2} request.".format(ExceptionUtils.http_status_code(exception), exception.response, exception.request))
    except requests.exceptions.ConnectTimeout as exception:
        logging.error("ConnectTimeout: {0}".format(exception))
    except requests.exceptions.ConnectionError as exception:
        logging.error("ConnectionError: {0}".format(exception))
    except Exception as exception:
        logging.error("Exception: Failed to connect to server with exception: {0}".format(exception))


# Run main
if __name__ == "__main__":
    main()

@Chavithra Please let me know what you think about this.

1. Good one. As also indicated by @e9henrik we can make a custom Exception to handle our dedicated set of return parameters. But when for example a ConnectionException occurs (e.g. no internet connection), there is no http response. So we can either populate that return parameter as None, but then the exception parser still has no idea about the root cause and what kind of action should take place. So probably it’s better to return a dedicated custom Exception for each possible Exception that the API connection and parsing can raise.
2. That’s often not the best rational for a requirement. But if you insist

Hi @Chavithra , I don’t have (or want) Discord. Since we’re no longer talking about breaking changes, is it an idea that we can formulate the requirements here. Once we have mutual understanding and an agreement, I’m happy to code a PR for just 1 function, e.g. Trading API connect(). We can do some iterations, or accept it as a solution. Then it will probably be more or less copy/paste to the other functions, to have the exact same behaviour.

Hello @funnel20, alright will try to formulate a specification tonight (or during the week if I can’t tonight).