prometheus-fastapi-instrumentator: the middleware can't be installed for the actual FastAPI version

Hi! Since FastAPI version 0.91.0, I started getting the following error when trying to run the application:

INFO:     Will watch for changes in these directories: ['/home/my_name/PycharmProjects/my-app']
INFO:     Uvicorn running on http://127.0.0.1:5000 (Press CTRL+C to quit)
INFO:     Started reloader process [15443] using WatchFiles
INFO:     Started server process [15472]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 648, in startup
    await handler()
  File "/home/my_name/PycharmProjects/my-app/core/app/service.py", line 75, in integrate_monitoring_plugin
    Instrumentator(excluded_handlers=["/health", "/metrics"]).instrument(app).expose(app, include_in_schema=False)
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 121, in instrument
    app.add_middleware(
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 135, in add_middleware
    raise RuntimeError("Cannot add middleware after an application has started")
RuntimeError: Cannot add middleware after an application has started

ERROR:    Application startup failed. Exiting.

I initiate the application this way:

def create_app() -> FastAPI:
    app = FastAPI()

    #blah-blah-blah

    @app.on_event("startup")
    async def integrate_monitoring_plugin():
        Instrumentator(excluded_handlers=["/health", "/metrics"]).instrument(app).expose(app, include_in_schema=False)

    return app

About this issue

Original URL
State: closed
Created a year ago
Reactions: 11
Comments: 23 (5 by maintainers)

Most upvoted comments

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

trallnag on Mar 8, 2023

can confirm, this works with the new Fastapi

instrumentator = Instrumentator().instrument(app)
@app.on_event("startup")
async def _startup():
    instrumentator.expose(app, endpoint='/metrics', tags=['metrics'])

harochau on Feb 23, 2023

Sure, just a quick note, I’m using start_http_server because I run my prometheus server on a diferent port, but I asume you could save the Instrumentator variable and just call .expose on the startup_event function

from fastapi import FastAPI
from prometheus_client import start_http_server
from prometheus_fastapi_instrumentator import Instrumentator

async def startup_event():
    start_http_server(8080)
    
app = FastAPI(
    on_startup=[startup_event]
)

Instrumentator(
    excluded_handlers=["/metrics"],
).instrument(app)

alfaro28 on Feb 21, 2023

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

Thanks. NameError is gone.

AliSayyah on Mar 8, 2023

@trallnag thanks for the release. with the new version I started to get another error when running unit tests:


tests/v1_1/advertisers/test_get_expenses.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1757: in get
    return await self.request(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1533: in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1620: in send
    response = await self._send_handling_auth(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1648: in _send_handling_auth
    response = await self._send_handling_redirects(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1685: in _send_handling_redirects
    response = await self._send_single_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1722: in _send_single_request
    response = await transport.handle_async_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_transports/asgi.py:162: in handle_async_request
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/fastapi/applications.py:271: in __call__
    await super().__call__(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/applications.py:118: in __call__
    await self.middleware_stack(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:184: in __call__
    raise exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:162: in __call__
    await self.app(scope, receive, _send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py:84: in __call__
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:108: in __call__
    response = await self.dispatch_func(request, call_next)
src/utils/middlewares/log_requests.py:19: in log_requests
    response_body = [chunk async for chunk in response.body_iterator]
src/utils/middlewares/log_requests.py:19: in <listcomp>
    response_body = [chunk async for chunk in response.body_iterator]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/middleware.py:181: in __call__
    instrumentation(info)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

info = <prometheus_fastapi_instrumentator.metrics.Info object at 0x7feb23667fd0>

    def instrumentation(info: Info) -> None:
>       TOTAL.labels(info.method, info.modified_status, info.modified_handler).inc()
E       NameError: free variable 'TOTAL' referenced before assignment in enclosing scope

../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/metrics.py:625: NameError

alexted on Mar 4, 2023

Is there any reason to implement the instrumentator on the “startup” event? Just moving the instrumentator outside of the event makes things work

According to the maintainer:

Prometheus expects a fully initialized ASGI client, where uvicorn now delays execution in newer versions. Closing.

jonathanloske on Feb 21, 2023