Zappa: remote exception logging with flask
_Continuing from https://github.com/Miserlou/Zappa/pull/257_…
It looks like flask on zappa is currently unable to log to remote exception loggers/aggregators. I’ve tried Sentry, Opbeat and Raygun.
It has probably something to do with the following exception that is raised during exception handling.
[1472050994820] Traceback (most recent call last):
[1472050994820] File "/usr/lib64/python2.7/logging/__init__.py", line 880, in emit
[1472050994821] stream.write(fs % msg)
[1472050994821] File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-9rkaZs/Werkzeug/werkzeug/local.py", line 343, in __getattr__
[1472050994821] AttributeError: 'str' object has no attribute 'write'
[1472050994821] Logged from file app.py, line 1587
From what it looks like, flasks log_exception function tries to write to something in werkzeug that is a string but should be some kind of stream.
I believe the error get’s swallowed somewhere by Zappa, continuing with handling the original exception.
Workaround
Use logging via mail for now:
if not app.debug:
import logging
from logging.handlers import SMTPHandler
mail_handler = SMTPHandler(
mailhost='your.host.com',
fromaddr='postmaster@your.host.com',
toaddrs=['you@gmail.com'],
subject='Error on your app',
credentials=('username', 'password')
)
mail_handler.setLevel(logging.ERROR)
app.logger.addHandler(mail_handler)
Reproducing the issue
requirements (sentry)
zappa # latest, installed from source
flask==0.11.1
Settings
{
"dev":{
"app_function": "app.app",
"s3_bucket": "foo",
"exception_handler": "app.unhandled_exceptions"
}
}
Minimal flask app (app.py)
from flask import Flask
from raven import Client
app = Flask(__name__)
SENTRY_DSN = 'https://user:pass@app.getsentry.com/id' # free account at getsentry.com
from raven.contrib.flask import Sentry
sentry = Sentry(app, dsn='YOUR_DSN_HERE')
@app.route('/', methods=['GET'])
def index():
# this throws an exception that should be sent to sentry
open("/non-existent.txt")
return "Hi", 200
# We only need this for local development.
if __name__ == '__main__':
app.run(debug=True)
Full cloudwatch log
[1472050994816] [INFO] 2016-08-24T15:03:14.816Z e144417b-6a0b-11e6-89cb-931b5b898cc6 Detected environment to be AWS Lambda. Using synchronous HTTP transport.
[1472050994820] Traceback (most recent call last):
[1472050994820] File "/usr/lib64/python2.7/logging/__init__.py", line 880, in emit
[1472050994821] stream.write(fs % msg)
[1472050994821] File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-9rkaZs/Werkzeug/werkzeug/local.py", line 343, in __getattr__
[1472050994821] AttributeError: 'str' object has no attribute 'write'
[1472050994821] Logged from file app.py, line 1587
[1472050994821] [ERROR] 2016-08-24T15:03:14.820Z e144417b-6a0b-11e6-89cb-931b5b898cc6 Exception on / [GET]
Traceback (most recent call last):
File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1988, in wsgi_app
File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1641, in full_dispatch_request
File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1544, in handle_user_exception
File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1639, in full_dispatch_request
File "/private/var/folders/82/bhs05x6n72s3bl5xdpm_nzsw0000gn/T/pip-build-I1t8YY/flask/flask/app.py", line 1625, in dispatch_request
File "/var/task/app.py", line 16, in index
open("/non-existent.txt")
IOError: [Errno 2] No such file or directory: '/non-existent.txt'
[1472050994822] [INFO] 2016-08-24T15:03:14.821Z e144417b-6a0b-11e6-89cb-931b5b898cc6 93.232.91.154 - - [24/Aug/2016:15:03:14 +0000] "GET / HTTP/1.1" 500 291 "" "HTTPie/0.9.3" 0/234.269
[1472050994822] {"http_status": 500, "content": "PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDMuMiBGaW5hbC8vRU4iPgo8dGl0bGU+NTAwIEludGVybmFsIFNlcnZlciBFcnJvcjwvdGl0bGU+CjxoMT5JbnRlcm5hbCBTZXJ2ZXIgRXJyb3I8L2gxPgo8cD5UaGUgc2VydmVyIGVuY291bnRlcmVkIGFuIGludGVybmFsIGVycm9yIGFuZCB3YXMgdW5hYmxlIHRvIGNvbXBsZXRlIHlvdXIgcmVxdWVzdC4gIEVpdGhlciB0aGUgc2VydmVyIGlzIG92ZXJsb2FkZWQgb3IgdGhlcmUgaXMgYW4gZXJyb3IgaW4gdGhlIGFwcGxpY2F0aW9uLjwvcD4K"}: LambdaException
Traceback (most recent call last):
File "/var/task/handler.py", line 396, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 143, in lambda_handler
raise lex
LambdaException: {"http_status": 500, "content": "PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDMuMiBGaW5hbC8vRU4iPgo8dGl0bGU+NTAwIEludGVybmFsIFNlcnZlciBFcnJvcjwvdGl0bGU+CjxoMT5JbnRlcm5hbCBTZXJ2ZXIgRXJyb3I8L2gxPgo8cD5UaGUgc2VydmVyIGVuY291bnRlcmVkIGFuIGludGVybmFsIGVycm9yIGFuZCB3YXMgdW5hYmxlIHRvIGNvbXBsZXRlIHlvdXIgcmVxdWVzdC4gIEVpdGhlciB0aGUgc2VydmVyIGlzIG92ZXJsb2FkZWQgb3IgdGhlcmUgaXMgYW4gZXJyb3IgaW4gdGhlIGFwcGxpY2F0aW9uLjwvcD4K"}
[1472051018205] Zappa Event: {u'account': u'747357196988', u'region': u'us-east-1', u'detail': {}, u'detail-type': u'Scheduled Event', u'source': u'aws.events', u'version': u'0', u'time': u'2016-08-24T15:03:10Z', u'id': u'7135a8fb-e8bc-4f80-9a5a-8c5846ba3b7b', u'resources': [u'arn:aws:events:us-east-1:747357196988:rule/test-dev-zappa-keep-warm-handler.keep_warm_callback']}
[1472051018205] Zappa Event: {}
[1472051018205] [DEBUG] 2016-08-24T15:03:38.205Z efbca15e-6a0b-11e6-af26-2bf427140012 Zappa Event: {}
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 1
- Comments: 22 (7 by maintainers)
with below config, its working fine for us, we tested a vpc enabled deployment
I have the same problem with django. does anyone know how bypass this issue? My settings:
I believe that issue is not related particularly to Flask, but to lambda function lifetime. Flask example sends errors with synchronous transports (schemes like
sync+https,requests+https) but fails to send with asynchronous (schemes:async+https(default transport),gevent+https,threaded+requests+https). I haven’t tried with Pyramid or Django, but with Bottle I get the same behavior.Bottle example for the reference