rmw_fastrtps: SHM transport: The longer it runs, The more latency increases. (Buffer Full Exception)
Description
tf / tf_static topics start generating the dropping message warning, 'the timestamp on the message is earlier than all the data in the transform cache`.
once it starts happening, the frequency just increases linearly as time goes.
this actually increases the delay and latency between publisher and subscribers significantly.
the reproduction rate is approximately 30%.
System Information
- ros2: Humble, patch release 5
- ros-humble-fastrtps: v2.6.7 with + https://github.com/eProsima/Fast-DDS/pull/4708
- Number of Nodes: - 120
- Number of Topics: - 500
tf/tf_statictopic: Publisher -15 and Subscriber 30 for each topic.
XML configuration
<?xml version="1.0" encoding="UTF-8"?>
<dds xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
<profiles>
<transport_descriptors>
<!-- Create a descriptor for the new transport -->
<transport_descriptor>
<transport_id>shm_transport</transport_id>
<type>SHM</type>
<segment_size>10485760</segment_size>
<port_queue_capacity>256</port_queue_capacity>
</transport_descriptor>
</transport_descriptors>
<participant profile_name="SHMParticipant" is_default_profile="true">
<rtps>
<!-- Link the Transport Layer to the Participant -->
<userTransports>
<transport_id>shm_transport</transport_id>
</userTransports>
<!-- <useBuiltinTransports>false</useBuiltinTransports> -->
</rtps>
</participant>
<publisher profile_name="service">
<qos>
<reliability>
<max_blocking_time>
<sec>10</sec>
</max_blocking_time>
</reliability>
</qos>
</publisher>
</profiles>
</dds>
Additional Information
- When this warning happens, https://github.com/eProsima/Fast-DDS/blob/b1e4707ad3cfe4cad7e5100318402206e0bd5e78/src/cpp/rtps/transport/shared_mem/MultiProducerConsumerRingBuffer.hpp#L228-L231 this if statement comes True.
- if we increase
port_queue_capacityincreases 4 times bigger, the 4 times longer it takes until the problem starts. (almost the same situation forsegment_size) this does not avoid this original problem, it only can buy some time based on the configuration.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 39 (4 by maintainers)
Commits related to this issue
- self-containeed simple reproducible test. https://github.com/ros2/rmw_fastrtps/issues/789 Signed-off-by: Tomoya Fujita <Tomoya.Fujita@sony.com> — committed to fujitatomoya/examples by fujitatomoya 2 years ago
- self-containeed simple reproducible test. https://github.com/ros2/rmw_fastrtps/issues/789 Signed-off-by: Tomoya Fujita <Tomoya.Fujita@sony.com> — committed to fujitatomoya/examples by fujitatomoya 2 years ago
@fujitatomoya One thing you could try is reducing the heartbeat period on the publishers. You can do that by XML, see this section of the documentation.
Reducing the
heartbeat_periodwould make the reliability mechanism more responsive, thus (hopefully) reducing the possibility of old and new data to be sent at the same time. I would start with something like heartbeat_period = 0.7 * publication_period.CC: @Barry-Xu-2018
@MiguelCompany
UDP Transport (w/o SHM transport)
patch applied with SHM transport
i believe that Buffer Full situation should not be true in the 1st place…