openai-python: Stream error after 5 minutes

Describe the bug

Streaming output from gpt4 consistently fails with Connection broken: InvalidChunkLength(got length b'', 0 bytes read) for prompts that trigger long-form responses. I’ve been able to reproduce this by prompting gpt to rewrite a long document.

A few observations

This failure occurs exactly at the 5 minute mark every time, which leads me to believe there’s a timeout somewhere.
The python implementation of stream reader assumes that an empty string will never be returned by the server.
I sent the same request using a non-streaming client with a timeout at 10min. The timeout wasn’t triggered until the 10min mark,so this problem seems isolated to streaming

Code snippet from urllib3/response.py:

def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()  # self._fp.fp.readline() returns b''
        line = line.split(b";", 1)[0]
        try:
            self.chunk_left = int(line, 16)  # this raises a value error
        except ValueError:
            # Invalid chunked protocol response, abort.
            self.close()
            raise InvalidChunkLength(self, line)

It seems like a problem on the server side but Im not confident

To Reproduce

Create gpt4 streaming client
Prompt it to rewrite a very long text. For example, verses 1:1 -> 4:10 in the book of genesis

Code snippets

No response

OS

macOs

Python version

Python 3.10.9

Library version

v.0.26.4

About this issue

Original URL
State: closed
Created a year ago
Reactions: 12
Comments: 29

Commits related to this issue

[Eval] JSON Patch on nested JSON objects (#1047) While this Eval does not specifically test large context windows directly, see "Eval details" for some context on work this humble developer is doin... — committed to openai/evals by elh a year ago
[Eval] JSON Patch on nested JSON objects (#1047) While this Eval does not specifically test large context windows directly, see "Eval details" for some context on work this humble developer is doin... — committed to arbreton/evals by elh a year ago
[Eval] JSON Patch on nested JSON objects (#1047) While this Eval does not specifically test large context windows directly, see "Eval details" for some context on work this humble developer is doin... — committed to TablewareBox/evals by elh a year ago

Most upvoted comments

Doublechecked and tested with various settings - timeout=600, stream=true - connection closed in 5 minutes timeout=600, stream=false - connection closed in 5 minutes(so won’t receive a response)

Also tested with the official web UI - completes only 87 lines and closes response after(there might be other limitations there)

I think the large token count was designed with a larger input in mind and a large sized output was not considered.

oomti on Apr 20, 2023

Thanks @karim-attia

Tested with a 600seconds timeout on my part.

After 5 minutes the stream has ended with the following error message:

openai-chatgpt\openai\Lib\site-packages\aiohttp\streams.py", line 304, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

This indicates that the socket from the serverside is disconnected after 5 minutes, which is probably a default setting.

oomti on Apr 20, 2023

I’ll add my voice to the chorus . I’ve been struggling with this error for a couple of weeks now. I’ve come up with some workarounds (asking the model to do less so that it completes in less time), and I’m even working on a code-based workaround, but I’m hoping that this gets fixed soon since any form of workaround comes at a cost both literally (in terms of API usage) and user experience.

(For those just discovering this thread, it’s worth noting that this is not a bug with the OpenAI library. I’m using the REST APIs directly; the bug is with OpenAI’s servers. Also, at least in the JavaScript world, the resulting network error cannot be caught and handled gracefully.)

cantrell on Jun 20, 2023

I have the same issue when I generate long text using GPT-4.

takatost on May 22, 2023

I believe there was an issue in the client (no longer there) and a limitation on the server side which has also been improved since then (other users have reported they hit a timeout at 10min recently).

rattrayalex on Nov 10, 2023

anyone has possible solutions?

wdeng on May 16, 2023

We find out this is not a bug of client, but a bug of the openai server

jizs on May 16, 2023

Same bug here

yannoth on Apr 27, 2023