Zappa: remote exception logging with flask

_Continuing from https://github.com/Miserlou/Zappa/pull/257_

It looks like flask on zappa is currently unable to log to remote exception loggers/aggregators. I’ve tried Sentry, Opbeat and Raygun.

It has probably something to do with the following exception that is raised during exception handling.

[1472050994820] Traceback (most recent call last):
[1472050994820] File "/usr/lib64/python2.7/logging/__init__.py", line 880, in emit
[1472050994821] stream.write(fs % msg)
[1472050994821] File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-9rkaZs/Werkzeug/werkzeug/local.py", line 343, in __getattr__
[1472050994821] AttributeError: 'str' object has no attribute 'write'
[1472050994821] Logged from file app.py, line 1587

From what it looks like, flasks log_exception function tries to write to something in werkzeug that is a string but should be some kind of stream.

I believe the error get’s swallowed somewhere by Zappa, continuing with handling the original exception.

Workaround

Use logging via mail for now:

if not app.debug:
    import logging
    from logging.handlers import SMTPHandler
    mail_handler = SMTPHandler(
        mailhost='your.host.com',
        fromaddr='postmaster@your.host.com',
        toaddrs=['you@gmail.com'],
        subject='Error on your app',
        credentials=('username', 'password')
    )
    mail_handler.setLevel(logging.ERROR)
    app.logger.addHandler(mail_handler)

Reproducing the issue

requirements (sentry)

zappa # latest, installed from source
flask==0.11.1

Settings

{
    "dev":{
        "app_function": "app.app", 
        "s3_bucket": "foo",
        "exception_handler": "app.unhandled_exceptions"
    }
}

Minimal flask app (app.py)

from flask import Flask
from raven import Client

app = Flask(__name__)
SENTRY_DSN = 'https://user:pass@app.getsentry.com/id' # free account at getsentry.com
from raven.contrib.flask import Sentry
sentry = Sentry(app, dsn='YOUR_DSN_HERE')

@app.route('/', methods=['GET'])
def index():
    # this throws an exception that should be sent to sentry
    open("/non-existent.txt")
    return "Hi", 200

# We only need this for local development.
if __name__ == '__main__':
    app.run(debug=True)

Full cloudwatch log

[1472050994816] [INFO]  2016-08-24T15:03:14.816Z        e144417b-6a0b-11e6-89cb-931b5b898cc6        Detected environment to be AWS Lambda. Using synchronous HTTP transport.
[1472050994820] Traceback (most recent call last):
[1472050994820] File "/usr/lib64/python2.7/logging/__init__.py", line 880, in emit
[1472050994821] stream.write(fs % msg)
[1472050994821] File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-9rkaZs/Werkzeug/werkzeug/local.py", line 343, in __getattr__
[1472050994821] AttributeError: 'str' object has no attribute 'write'
[1472050994821] Logged from file app.py, line 1587
[1472050994821] [ERROR] 2016-08-24T15:03:14.820Z        e144417b-6a0b-11e6-89cb-931b5b898cc6        Exception on / [GET]
Traceback (most recent call last):
  File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1988, in wsgi_app
  File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1641, in full_dispatch_request
  File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1544, in handle_user_exception
  File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1639, in full_dispatch_request
  File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1625, in dispatch_request
  File "/var/task/app.py", line 16, in index
    open("/non-existent.txt")
IOError: [Errno 2] No such file or directory: '/non-existent.txt'
[1472050994822] [INFO]  2016-08-24T15:03:14.821Z        e144417b-6a0b-11e6-89cb-931b5b898cc6        93.232.91.154 - - [24/Aug/2016:15:03:14 +0000] "GET / HTTP/1.1" 500 291 "" "HTTPie/0.9.3" 0/234.269
[1472050994822] {"http_status": 500, "content": "PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDMuMiBGaW5hbC8vRU4iPgo8dGl0bGU+NTAwIEludGVybmFsIFNlcnZlciBFcnJvcjwvdGl0bGU+CjxoMT5JbnRlcm5hbCBTZXJ2ZXIgRXJyb3I8L2gxPgo8cD5UaGUgc2VydmVyIGVuY291bnRlcmVkIGFuIGludGVybmFsIGVycm9yIGFuZCB3YXMgdW5hYmxlIHRvIGNvbXBsZXRlIHlvdXIgcmVxdWVzdC4gIEVpdGhlciB0aGUgc2VydmVyIGlzIG92ZXJsb2FkZWQgb3IgdGhlcmUgaXMgYW4gZXJyb3IgaW4gdGhlIGFwcGxpY2F0aW9uLjwvcD4K"}: LambdaException
Traceback (most recent call last):
  File "/var/task/handler.py", line 396, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 143, in lambda_handler
    raise lex
LambdaException: {"http_status": 500, "content": "PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDMuMiBGaW5hbC8vRU4iPgo8dGl0bGU+NTAwIEludGVybmFsIFNlcnZlciBFcnJvcjwvdGl0bGU+CjxoMT5JbnRlcm5hbCBTZXJ2ZXIgRXJyb3I8L2gxPgo8cD5UaGUgc2VydmVyIGVuY291bnRlcmVkIGFuIGludGVybmFsIGVycm9yIGFuZCB3YXMgdW5hYmxlIHRvIGNvbXBsZXRlIHlvdXIgcmVxdWVzdC4gIEVpdGhlciB0aGUgc2VydmVyIGlzIG92ZXJsb2FkZWQgb3IgdGhlcmUgaXMgYW4gZXJyb3IgaW4gdGhlIGFwcGxpY2F0aW9uLjwvcD4K"}
[1472051018205] Zappa Event: {u'account': u'747357196988', u'region': u'us-east-1', u'detail': {}, u'detail-type': u'Scheduled Event', u'source': u'aws.events', u'version': u'0', u'time': u'2016-08-24T15:03:10Z', u'id': u'7135a8fb-e8bc-4f80-9a5a-8c5846ba3b7b', u'resources': [u'arn:aws:events:us-east-1:747357196988:rule/test-dev-zappa-keep-warm-handler.keep_warm_callback']}
[1472051018205] Zappa Event: {}
[1472051018205] [DEBUG] 2016-08-24T15:03:38.205Z        efbca15e-6a0b-11e6-af26-2bf427140012        Zappa Event: {}

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 1
  • Comments: 22 (7 by maintainers)

Most upvoted comments

with below config, its working fine for us, we tested a vpc enabled deployment

INSTALLED_APPS += (
    'raven.contrib.django.raven_compat',
)

RAVEN_CONFIG = {
    'dsn': 'https://client_id:secret_key@sentry.io/project_id',
    'transport': 'raven.transport.requests.RequestsHTTPTransport'
}

I have the same problem with django. does anyone know how bypass this issue? My settings:

INSTALLED_APPS += (
    'raven.contrib.django.raven_compat',
)

RAVEN_CONFIG = {
    'dsn': 'requests+https://key1:key2@sentry.io/project_id',
    'transport': 'raven.transport.requests.RequestsHTTPTransport'
}

I believe that issue is not related particularly to Flask, but to lambda function lifetime. Flask example sends errors with synchronous transports (schemes like sync+https, requests+https) but fails to send with asynchronous (schemes: async+https(default transport), gevent+https, threaded+requests+https). I haven’t tried with Pyramid or Django, but with Bottle I get the same behavior.

Bottle example for the reference

from bottle import route, run, request
import bottle


@route('/')
def index():
    request.app.sentry.captureMessage("Sentry won't get it with async transport")
    return 'Hello name!'

app = bottle.app()
app.catchall = False

from raven import Client
from raven.contrib.bottle import Sentry
client = Client('threaded+requests+https://<YOURDSN>')
app = Sentry(app, client)

if __name__ == '__main__':
    run(app, host='localhost', port=8080)