rmw_fastrtps: SHM transport: The longer it runs, The more latency increases. (Buffer Full Exception)

Description

tf / tf_static topics start generating the dropping message warning, 'the timestamp on the message is earlier than all the data in the transform cache`. once it starts happening, the frequency just increases linearly as time goes. this actually increases the delay and latency between publisher and subscribers significantly.

the reproduction rate is approximately 30%.

System Information

  • ros2: Humble, patch release 5
  • ros-humble-fastrtps: v2.6.7 with + https://github.com/eProsima/Fast-DDS/pull/4708
  • Number of Nodes: - 120
  • Number of Topics: - 500
  • tf / tf_static topic: Publisher -15 and Subscriber 30 for each topic.

XML configuration

<?xml version="1.0" encoding="UTF-8"?>
<dds xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <profiles>
        <transport_descriptors>
            <!-- Create a descriptor for the new transport -->
            <transport_descriptor>
                <transport_id>shm_transport</transport_id>
                <type>SHM</type>
                <segment_size>10485760</segment_size> 
                <port_queue_capacity>256</port_queue_capacity>
            </transport_descriptor>
        </transport_descriptors>
        <participant profile_name="SHMParticipant" is_default_profile="true">
            <rtps>
                <!-- Link the Transport Layer to the Participant -->
                <userTransports>
                  <transport_id>shm_transport</transport_id>
                </userTransports>
                <!-- <useBuiltinTransports>false</useBuiltinTransports> -->
            </rtps>
        </participant>
        <publisher profile_name="service">
            <qos>
                <reliability>
                    <max_blocking_time>
                        <sec>10</sec>
                    </max_blocking_time>
                </reliability>
            </qos>
        </publisher>
    </profiles>
</dds>

Additional Information

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 39 (4 by maintainers)

Commits related to this issue

Most upvoted comments

@fujitatomoya One thing you could try is reducing the heartbeat period on the publishers. You can do that by XML, see this section of the documentation.

Reducing the heartbeat_period would make the reliability mechanism more responsive, thus (hopefully) reducing the possibility of old and new data to be sent at the same time. I would start with something like heartbeat_period = 0.7 * publication_period.

@MiguelCompany

  • UDP Transport (w/o SHM transport)

    • the problem (tf dropping message) can be reproduced, that tells us this problem is not only with SHM transport.
    • when the problem starts happening eprosima::fastrtps::rtps::StatefulReader::processGapMsg consumes CPU time a lot compared to normal situation. This indicates that publisher skips or drops the certain messages on the HistoryCache, and prevent subscriber from request the retransmission for the efficiency.
  • patch applied with SHM transport

    • can mitigate situation until the spike. w/o patch it was 10 hours, but with patch it can be more than 35 hours.
    • even before the spike, Buffer Full situation are appeared in order of 10~100 constantly. this is different from without patch condition.​
    • once it hits the spike, there are hundreds thousands Buffer Full situation are appeared… almost same without patch situation.

i believe that Buffer Full situation should not be true in the 1st place…