rosbridge_suite: Increasing delays when subscriber callback takes too long

I’m encountering increasing delays on messages that arrive in a callback that runs longer than the topic’s period time. See https://answers.ros.org/question/334898/weird-increasing-delay-in-rosbridge-despite-queue_size1-and-buff_size224 for further details. I can reproduce this behavior in roslibpy and roslibjs, and infer from that that it is somehow caused by rosbridge. I can also reproduce it in pure rospy without the bridge in between when I leave the rospy.Subscriber settings regarding queue_size and buff_size on default. If I set them like queue_size=1 and buff_size=2**24 this behavior does not occur.

Expected Behavior

If roslibpy.Topic(..., queue_length = 1) or queue_length = 0 is set, messages should always arrive on time. If they cannot be processed because the callback is still processing the previous message, they should be dropped. The callback should always process the newest (!) message. No buffering should take place.

Actual Behavior

It seems like the messages are still buffered/queued somewhere. See my Update 2 in the linked question for the places that I’ve found possible buffers/queues so far.

Steps to Reproduce the Problem

There is a test script in the linked question under Update 1 that has proven to reproduce the behavior.

Specifications

ROS Version (echo $ROS_DISTRO): Kinetic Kame
OS Version (grep DISTRIB_CODENAME /etc/lsb-release): Ubuntu 16.04 LTS
Rosbridge Version (roscat rosbridge_server package.xml | grep '<version>'): 0.11.3
Twisted Version (python -c 'import twisted; print twisted.version'): [Twisted, version 19.7.0]

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 18 (9 by maintainers)

Most upvoted comments

This may be the intrinsics of the client libraries and environments at work.

In Chrome (and probably other browsers?) a WebSocket callback will block all the way back to the server until it finishes. A good pattern to work around this when doing visualizations is to, instead of doing all your drawing in the subscription handler, just store the most recent data and do the drawing for that data in a requestAnimationFrame cycle. This way you never block the socket, you never draw the same thing more than once per frame, and data you aren’t fast enough to render is dropped. Maybe this pattern can help your use case too.

I can’t speak for roslibpy or UDP server though.

mvollrath on Oct 10, 2019

The rosbridge protocol and implementation leaves a lot to be desired when it comes to queues, until recently setting queue_length could have no effect whatsoever, and even now there is an undefined transport library queue on the server side and an undefined queue on the client side for probably all of the client libraries. Sorry that this is throwing a wrench in your application. Rosbridge is not rospy.

You can solve this problem quickly with a deque, e.g.

d = collections.deque(maxlen=1)

def consumer_thread():
    while True:
        try:
            msg = d.pop()
        except IndexError:
            time.sleep(N)  # Sleep for 0-N seconds here to yield thread.
            continue
        """Process msg here."""

def sub_callback(msg):
    d.appendleft(msg)  # Will push existing msg out of the deque.

mvollrath on Oct 14, 2019