python-engineio: ValueError: Too many packets in payload after update to v3.10.0

I am sending a file using fixed size chunks (binary data, with size=1024 in example) using socket.io from a client to a flask_socketio server emit:

sio.emit("chunk", data=content, namespace="audio")

The audio source is just a wave file, the snippet of the generator is:

chunk_size = 1024
delay = 0.0

while data != b'':
    yield data
    data = wave_file.readframes(chunk_size)
    time.sleep(delay)
    break

Until now, the data arrived without problem. After update python-engineio to v3.10.0, if delay = 0.0, sometime server print the following error:

2019-10-24T07:47:42.708635Z [error    ] post request handler error     event_id=146cb3c88 host=vgonisanz-linux source=engineio.server user=vgonisanz version=dl:0.1.0
Traceback (most recent call last):
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/server.py", line 394, in handle_request
    socket.handle_post_request(environ)
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/socket.py", line 123, in handle_post_request
    p = payload.Payload(encoded_payload=body)
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/payload.py", line 15, in __init__
    self.decode(encoded_payload)
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/payload.py", line 61, in decode
    raise ValueError('Too many packets in payload')
ValueError: Too many packets in payload

This error didn’t appear before updating. Also, If I set up delay as 0.01, it don’t appear. It seems the change comes from commit c8407ae97821bb00c33a91114f425b8454f5e50e, and the issue is related with the time between requests.

Why is this happening now?
It is possible to continue use delay = 0 (for testing purpose I want to run my tests faster as I can)?

Thanks.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (5 by maintainers)

Commits related to this issue

Fix chat_server_start.py - 도커 운영 시에 ValueError: Too many packets in payload 해결 - 참조 링크 : https://github.com/miguelgrinberg/python-engineio/issues/142 — committed to YoungchanChang/ES_BERT_CHAT by YoungchanChang 2 years ago
Try fix for "Too many packets in payload" from engineio https://github.com/miguelgrinberg/python-engineio/issues/142#issuecomment-545822543 — committed to SUNET/cnaas-nms by indy-independence 10 months ago
Try fix for "Too many packets in payload" from engineio https://github.com/miguelgrinberg/python-engineio/issues/142#issuecomment-545822543 — committed to SUNET/cnaas-nms by indy-independence 10 months ago

Most upvoted comments

Oh no, sorry I did not make myself clear on this. This was not about modifying the dependency. Just add that to your application before your create your socketio object. Something like this:

from flask_socketio import SocketIO
from engineio.payload import Payload

Payload.max_decode_packets = cfg.service.ENGINEIO_MAX_DECODE_PACKETS
socketio = SocketIO(async_mode='gevent', ping_timeout=cfg.service.PING_TIMEOUT, ping_interval=cfg.service.PING_INTERVAL)

+14

miguelgrinberg on Oct 24, 2019

This change that I’ve made is for security purposes. Decoding packets from payloads is somewhat expensive, and I have rarely seen payloads with lots of small packets, except when someone is trying to get the server to break. If you want to bypass this measure, you can do something like this:

from engineio.payload import Payload

Payload.max_decode_packets = 50

You can raise the 50 to the number that you need. The value that I selected prevent high CPU usage is 16, which I thought was more than enough, but I’m open to review it and raise if a lot of people get tripped by this.

miguelgrinberg on Oct 24, 2019