prefect: Excessive amount of AssertionError: feed_data before feed_eof in the log

First check

I added a descriptive title to this issue.
I used the GitHub search to find a similar issue and didn’t find it.
I searched the Prefect documentation for this issue.
I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Everything seems to be working fine, but when I looked in the orion.log file (SERVER_LEVEL:WARNING) I see the exception as below. Unfortunately I get the impression I can’t dynamically change the logging level using SIGHUP/re-read configuration so I can’t add any more context and I don’t want to stop the Orion server etc.

If I could get some background info on why this might be happening over and over, I can debug it further. Perhaps my code is doing something incorrectly?

Reproduction

Need more info before I can provide code

Error

17:35:38.216 | ERROR   | asyncio - Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'INFO:     1...1.1" 200 OK\n')
handle: <Handle SubprocessStreamProtocol.pipe_data_received(1, b'INFO:     1...1.1" 200 OK\n')>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/asyncio/subprocess.py", line 73, in pipe_data_received
    reader.feed_data(data)
  File "/usr/lib/python3.8/asyncio/streams.py", line 472, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof

Versions

Version:             2.0.3
API version:         0.8.0
Python version:      3.8.10
Git commit:          2f1cf4ac
Built:               Fri, Aug 5, 2022 3:57 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         hosted

Additional context

No response

About this issue

Original URL
State: open
Created 2 years ago
Comments: 21 (7 by maintainers)

Most upvoted comments

Leaving a comment here for any unfortunate soul who runs into the same issue as me.

I was seeing this AssertionError: feed_data after feed_eof error, as the reporter described, though neither the linked PR or https://github.com/PrefectHQ/prefect/issues/6446 fixed my error.

I was running prefect in a django command (which i do not recommend unless you have to), and this command was running in a container. I was seeing this error, and eventually discovered that the output stream was being wrapped by colorama.ansitowin32.StreamWrapper, but only when the container was not attached to a tty.

I found out django was initializing colorama for me, and the behavior from Django only makes this an issue when not attached to a tty, and uninstalling colorama fixed the issue.

acookin on Sep 1, 2022

I was able to solve those errors by enabling UTF-8 codecs globally for my agent’s Python process on Windows. This was apparently not the case by default, since Windows still uses legacy encodings, see this paragraph in the Python docs. I guess anyIO’s reported decoder errors were prompted by this implicit usage of ANSI encoding in Windows Python.

So, in short, setting PYTHONUTF8=1 in my Prefect agent’s Powershell instance, as recommended in the docs, makes everything work smoothly. Maybe this also works for you @drfraser? (Even though you are on Linux, perhaps there is some other encoding problem present)

nicholasjng on Aug 18, 2022