Universal_Robots_ROS_Driver: Reading data Streams - Could not get fresh data package from robot

Summary

While reading from the topics ur_hardware_interface/rtde_data/target_q and ur_hardware_interface/rtde_data/temperature sometimes connection is brakes and we get log messages as follow:

[INFO] [1589967979.606725]: Loading controller: joint_state_controller [INFO] [1589967979.607874]: Controller Spawner: Waiting for service controller_manager/unload_controller [INFO] [1589967979.611608]: Loading controller: pos_traj_controller [INFO] [1589967979.618885]: Loading controller: scaled_pos_traj_controller [INFO] [1589967979.672892]: Loading controller: joint_group_vel_controller [INFO] [1589967979.720808]: Loading controller: speed_scaling_state_controller [INFO] [1589967979.728871]: Controller Spawner: Loaded controllers: pos_traj_controller, joint_group_vel_controller [INFO] [1589967979.732789]: Loading controller: force_torque_sensor_controller [ INFO] [1589967979.737161355]: Robot's safety mode is now NORMAL [ INFO] [1589967979.738849418]: Robot mode is now RUNNING [INFO] [1589967979.742886]: Controller Spawner: Loaded controllers: joint_state_controller, scaled_pos_traj_controller, speed_scaling_state_controller, force_torque_sensor_controller [INFO] [1589967979.748990]: Started controllers: joint_state_controller, scaled_pos_traj_controller, speed_scaling_state_controller, force_torque_sensor_controller [ INFO] [1589992656.595420159]: Robot mode is now POWER_ON [ INFO] [1589992656.909132755]: Robot mode is now POWER_OFF [ INFO] [1589992657.524765450]: Robot's safety mode is now FAULT [ERROR] [1589992659.718762803]: Could not get fresh data package from robot [ERROR] [1589992659.818972283]: Could not get fresh data package from robot [ERROR] [1589992659.919288408]: Could not get fresh data package from robot [ERROR] [1589992660.019491602]: Could not get fresh data package from robot [ERROR] [1589992660.119762006]: Could not get fresh data package from robot [ WARN] [1589992660.204704932]: Failed to read from stream, reconnecting in 1 seconds... [ERROR] [1589992660.219980952]: Could not get fresh data package from robot

It looks like that the error “Could not get fresh data package from robot” always occures in combination with the error “Robot’s safety mode is now FAULT”.

Introduction to the issue

Versions

  • ROS Driver version: forked at 19.4.2020
  • ROS Meldic, Ubuntu 18.4, no-realtime adjustments
  • UR5e CB3

Impact

The arbitraray behavoir of this error blocks completly our measurements campaign. We need to collect temperatures and joint angles during a complete day.

Issue details

The error occures also when nothing else is running in the ROS system. There is no much payload. the UR is connected with a switch directly connected on the PC running ROS.

Project status at point of discovered

When did you first observe the issue?

  • Before I started? No
  • In first couple of tries? No
  • In normal use? Yes

We need to know in which conditions principally the error message is loged to understand more about the conditions.

Steps to Reproduce

Simple listen to the topics: ur_hardware_interface/rtde_data/target_q or ur_hardware_interface/rtde_data/temperature

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Comments: 17 (4 by maintainers)

Most upvoted comments

@orat

If you are still looking to install kernel with PREEMPT-RT compatibility w/ Nvidia drivers, try the following method. We were successful in doing so, although we had to use a much older kernel (5.4.78). Although FYI we still face the ‘Could not get fresh data package from robot’ error.

Setting up Linux Kernel with Real-Time Capabilities

Building and Installing Real-Time Kernel

## Set up source lists and install required dependencies
echo "deb-src http://archive.ubuntu.com/ubuntu focal main" | sudo tee -a /etc/apt/sources.list.d/focal.list
sudo apt-get update
sudo apt-get build-dep linux
sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf fakeroot dwarves

## Create installation and build directories. Can be deleted after the kernel is installed and functional
mkdir ~/kernel && cd ~/kernel

## Installing and patching Linux Ubuntu 5.4 kernel with real-time patch
## Instructions copied from (https://docs.ros.org/en/foxy/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2.html)
wget https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.4.78.tar.gz
tar -xzf linux-5.4.78.tar.gz
wget http://cdn.kernel.org/pub/linux/kernel/projects/rt/5.4/older/patch-5.4.78-rt44.patch.gz
gunzip patch-5.4.78-rt44.patch.gz
cd linux-5.4.78/
patch -p1 < ../patch-5.4.78-rt44.patch

## Editing configs for build
cp /boot/config${MOST RECENT KERNEL} .config
yes '' | make oldconfig
make menuconfig

In the terminal GUI, do:

# Enable CONFIG_PREEMPT_RT
 -> General Setup
  -> Preemption Model (Fully Preemptible Kernel (Real-Time))
   (X) Fully Preemptible Kernel (Real-Time)

# Enable CONFIG_HIGH_RES_TIMERS
 -> General setup
  -> Timers subsystem
   [*] High Resolution Timer Support

# Enable CONFIG_NO_HZ_FULL
 -> General setup
  -> Timers subsystem
   -> Timer tick handling (Full dynticks system (tickless))
    (X) Full dynticks system (tickless)

# Set CONFIG_HZ_1000 (note: this is no longer in the General Setup menu, go back twice)
 -> Processor type and features
  -> Timer frequency (1000 HZ)
   (X) 1000 HZ

# Set CPU_FREQ_DEFAULT_GOV_PERFORMANCE [=y]
 ->  Power management and ACPI options
  -> CPU Frequency scaling
   -> CPU Frequency scaling (CPU_FREQ [=y])
    -> Default CPUFreq governor (<choice> [=y])
     (X) performance

in .config remove the KEYS

CONFIG_SYSTEM_TRUSTED_KEYS="debian/canonical-certs.pem" > CONFIG_SYSTEM_TRUSTED_KEYS=""

run the make:

## Making .deb packages will take a lot of time (~10-30 mins)
make -j `nproc` deb-pkg

## Check build there should be 4 .deb files
ls ../*deb

## Install .deb. This will also take some time
sudo dpkg -i ../*.deb

Preparing Real-Time Kernel for UR Integration

sudo groupadd realtime
sudo usermod -aG realtime $(whoami)

Then, make sure /etc/security/limits.conf contains. If it does not, then open it as superuser and copy in the below lines into the file:

@realtime soft rtprio 99
@realtime soft priority 99
@realtime soft memlock 102400
@realtime hard rtprio 99
@realtime hard priority 99
@realtime hard memlock 102400

Check for RT capabilities

## Update grup and reboot
sudo update-grub
sudo reboot

## Once returned to the rt kernel, check if PREEMPT-RT is functional
uname -v | cut -d" " -f1-4
## Outpu should read >> #1 SMP PREEMPT_RT

Installing Real-Time Compatible Graphic Drivers

Download .run file for a compatible NVIDIA Driver (version 455) using following website:

https://www.nvidia.com/Download/driverResults.aspx/166177/en-us

Create/Edit following file:

sudo nano /etc/modprobe.d/blacklist-nouveau.conf

If the lines do not already exist, add the following lines:

blacklist nouveau
options nouveau modeset=0

Update kernel initramfs

sudo update-initramfs -u
sudo reboot

Upon reboot, login to the real-time kernel through Recovery Mode and access kernel though root. Then, navigate to the location where the .run file is downloaded and run the following command:

sudo IGNORE_PREEMPT_RT_PRESENCE=1 bash NVIDIA-Linux-x86_64-455.38.run

Move through installation with default option (i.e. select Yes for everything). Then exit root and boot normally.

Once this is done, check the installation of NVIDIA driver with command:

nvidia-smi

It should show something like:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 00000000:01:00.0  On |                  N/A |
| 30%   35C    P0    N/A /  75W |    215MiB /  4031MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+