gradio: 404 Session Not Found error When accessing gradio via a proxy

Describe the bug

I am running a Gradio application locally, where there’s a requests request to a remote server in the click event function of a button, and the result is returned to the component. Everything works fine, but if I turn on a proxy (Shadowsocks) to access the Gradio application, requests with short response time return normally, while requests that take longer return exceptions.

Have you searched existing issues? 🔎

  • I have searched and found no existing issues

Reproduction

#!/usr/bin/env python

import gradio as gr
import os
import json
import requests
import base64

URL = os.environ.get("URL")

def image_to_base64(image_path):
    with open(image_path, "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read())
        return encoded_string.decode('utf-8')

def post(
        input_text,
        image_prompt,
        ):
    headers = {
        "Content-Type": "application/json; charset=UTF-8",
        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36",
    }
    if image_prompt:
        encoded_img = image_to_base64(image_prompt)
    else:
        return "", []

    data = json.dumps({
        'text': input_text,
        'history': [],
        'image': encoded_img
    })
    try:
        response = requests.request("POST", URL, headers=headers, data=data, timeout=(60, 100)).json()
    except Exception as e:
        return "", []
    answer = str(response['result'])
    return "", [[input_text, answer]]


def main():
    gr.close_all()
    with gr.Blocks() as demo:
        with gr.Row():
            with gr.Column(scale=4.5):
                with gr.Group():
                    input_text = gr.Textbox(label='Input Text', placeholder='Please enter text prompt below and press ENTER.')
                    with gr.Row():
                        run_button = gr.Button('Generate')
                    image_prompt = gr.Image(type="filepath", label="Image Prompt", value=None)

            with gr.Column(scale=5.5):
                result_text = gr.components.Chatbot(label='Multi-round conversation History', value=[("", "Hi, What do you want to know about this image?")], height=550)
                
        run_button.click(fn=post,inputs=[input_text, image_prompt],
                         outputs=[input_text, result_text])
    demo.launch(server_port=7862)

if __name__ == '__main__':
    main()

Screenshot

image

Logs

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/routing.py", line 73, in app
    await response(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/responses.py", line 259, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/responses.py", line 255, in wrap
    await func()
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/responses.py", line 244, in stream_response
    async for chunk in self.body_iterator:
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/gradio/routes.py", line 660, in sse_stream
    raise e
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/gradio/routes.py", line 601, in sse_stream
    raise HTTPException(
fastapi.exceptions.HTTPException: 404: Session not found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/user/anaconda3/envs/py3.8/lib/python3.8/site-packages/starlette/_exception_handler.py", line 59, in wrapped_app
    raise RuntimeError(msg) from exc
RuntimeError: Caught handled exception, but response already started.

System Info

Python 3.8.16
Gradio 4.12.0
requests 2.31.0
fastapi 0.108.0

System: Ubuntu
Browser: Chrome 120.0.6099.129, Safari 16.1
Proxy: Shadowsocks

Severity

Blocking usage of gradio

About this issue

  • Original URL
  • State: open
  • Created 6 months ago
  • Reactions: 4
  • Comments: 55 (31 by maintainers)

Commits related to this issue

Most upvoted comments

@abidlabs For what it’s worth @edsna’s suggestion of rolling back to 3.50.2 works, and that’s what I have deployed in stage and production. Not sure exactly what breaking changes were made since then though. I do want to update to a newer version to resolve the security vulernabilities listed by dependabot prior to version 4.19.2, but I can’t do that until this issue is resolved.

+2
mgirard772 on Mar 13, 2024

Looking into this!

+2
abidlabs on Feb 12, 2024

@abidlabs Note that this is a regression, 3.50.2 worked fine. Should be fixed I’d hope. I’m unable to upgrade to gradio 4 due to his, event though all non-networking things are wonderful with gradio4.

+2
pseudotensor on Feb 12, 2024

Hi @abidlabs thank you so much for the prompt response.

It’s me again, I had to change machines but I’ve sth that may help reproduce the issue, please follow the steps below:

  1. I created a simple app using gr.TabbedInterface introduced in gradio 4.0 I believe. here’s how the app looks like:
import gradio as gr

# Function that echoes the text input
def echo_text(text):
    return text

# Function that displays the uploaded image
def show_image(image):
    return image

# Create the text interface
text_interface = gr.Interface(fn=echo_text, inputs="text", outputs="text")

# Create the image interface
image_interface = gr.Interface(fn=show_image, inputs="image", outputs="image")

# Combine both interfaces into a tabbed interface
tabbed_interface = gr.TabbedInterface([text_interface, image_interface], ["Text Echo", "Image Display"])

# Launch the app
tabbed_interface.launch(server_name="0.0.0.0", server_port=7860)

  1. I ran this locally in an isolated env with gradio==4.19.2 gradio_client==0.10.1 using python app.py and it loaded just fine, see image below, I’ve also attached the gradio==4.19.2ANDgradio_client==0.10.1.txt file used to run this locally: gradio==4.19.2ANDgradio_client==0.10.1.txt local-run at gradio==4 19 2ANDgradio_client==0 10 1

  2. Next, containerize the app using a Docker with a simple image such as this:

# Use an official Python runtime as a parent image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy the current directory contents into the container at /usr/src/app
COPY . .

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Make port 7860 available to the world outside this container
EXPOSE 7860

# Define environment variable
ENV NAME World

# Run app.py when the container launches
CMD ["python", "app.py"]

You can run the docker image to test as well and it will work. Then tag and push that docker image to whichever platform you’d like, - I used ECR.

  1. Next create a simple deployment, service and ingress file to deploy the app in a k8s cluster:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whatever-you'd-like
  namespace: whatever-you'd-like
  labels:
    app: whatever-you'd-like
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whatever-you'd-like
  template:
    metadata:
      labels:
        app: whatever-you'd-like
    spec:
      containers:
      - name: whatever-you'd-like
        image: Please use the image deployed and pushed to some containers registry as explained earlier.
        ports:
        - containerPort: 7860
-----
apiVersion: v1
kind: Service
metadata:
  name: whatever-you'd-like
  namespace: whatever-you'd-like
  labels:
      app: whatever-you'd-like
spec:
  selector:
    app: whatever-you'd-like
  ports:
    - protocol: TCP
      port: 80
      targetPort: 7860

------
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whatever-you'd-like
  namespace: whatever-you'd-like
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: test.exampledomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: whatever-you'd-like
            port:
              number: 80

Make sure to configure DNS for the domain you use eg: test.exampledomain.com

  1. After deploying these, the app should be accessible through test.exampledomain.com and that’s when you get the error message: Screenshot 2024-03-01 at 2 18 32 PM

  2. The temporary solution at the moment is to downgrade gradio and gradio_client to this:

gradio==3.50.2
gradio_client==0.6.1

You can do that directly in the requirements.txt file, build the image again, push it and update the deployment.yml file with the new image URI/… whatever you use.

Everything will work well after that. Screenshot 2024-03-01 at 2 29 02 PM

I have to say that in different stages of the deployment you find the different errors many mentioned above - this is already long so I can’t reproduce them all step by step but I’ve attached screenshots of some of them below. @abidlabs I hope this helps you get a picture of what might have changed to cause the problem.

Screenshot 2024-03-01 at 2 37 35 PM

+1
edsna on Mar 1, 2024

I’m working on it @edson-arcaea but progress has been a little slow since this error has been hard to reproduce. Anything you can share that will help us reproduce the issue will be much appreciated

+1
abidlabs on Mar 1, 2024

FYI Still have this issue for h2oGPT for latest gradio for k8 issue, have to revert to gradio 3 for k8.

+1
pseudotensor on Feb 27, 2024

@abidlabs Do you have any ideas on what might be the issue and how to work around it, as it’s blocking the major version upgrade for us? Is it related to the switch to using SSE by default in v4? Is there a way to disable it?

I think this may be an important issue as more “production” Gradio apps being served on internal infra try upgrading to v4.

+1
shimizust on Jan 12, 2024

I am also experiencing this issue after bumping gradio from 3.50.2 to 4.12.0.

I basically have a Gradio app deployed on a k8s cluster. Port-forwarding directly to the pod works as expected, but accessing it externally via emissary causes this same error.

+1
shimizust on Jan 5, 2024
Read more comments on GitHub
← gradio: 3.42.0 breaks automatic reload after a change is detect
gradio: Bad handling of \n in chatbot →