ros2 topic hz perform worse than c++ code, also worse than result in another docker container

[Chenxintang]

Bug report

Required Info:

• Operating System:
  • Ubuntu18.04
• Installation type:
  • source
• Version or commit hash:
  • humble
• DDS implementation:
  • FastDDS

Steps to reproduce issue

• run two docker container using the following docker-compose.yml

container1:
    image: image_name
    network_mode: "host"
    cpuset: 2-9
    cpu_shares: 2048
    shm_size: 4G
    container_name: container1
    runtime: nvidia
    privileged: true
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    logging:
      options:
        max-size: "${DOCKER_LOGS_MAX_SIZE:-100M}"

container2:
    image: image_name
    network_mode: "host"
    privileged: true
    cpuset: 12-17
    cpu_shares: 2048
    shm_size: 2G
    container_name: container2
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    runtime: nvidia
    logging:
      options:
        max-size: "${DOCKER_LOGS_MAX_SIZE:-100M}"

• publish sensor_msgs::msg::PointCloud2 type message with frequency 10hz in container1
• run subscriber and process data c++ code in container2, record the frequency subscription callback called
• use ros2 topic hz tool follow the frequency of topic in both containers

Expected behavior

• In both containers, ros2 topic hz shows the same frequency which is around 10hz
• Log of c++ code in container2 shows that pointcloud2 messages arrive each 100ms

Actual behavior

• In container1 terminal, ros2 topic hz shows frequency slightly under 10hz
  
• In container2 terminal, ros2 topic hz shows frequency highly under 10hz, verity from 7hz to 9hz
  
• In log of container2, c++ code receive message each 100ms stably
  

Additional information

Since code in both container are quite complicate, I can't paste the code here.
As discussed in the blow question, ros2 topic hz result is lower than c++ code because the qos used is different. But why the hz results in different containers turn out to be different?

https://answers.ros.org/question/350753/ros2-topic-hz-provides-wrong-rate-for-larger-msgs/
Also, since ros2 topic hz result is not reliable, is therea a better way to track the transportaion stability between many nodes?

[fujitatomoya]

Ubuntu 18.04

Humble supports Ubuntu 22.04 as tier 1 platform.
I am not sure if Humble can built with Ubuntu 18.04. (btw, Ubuntu 20.04 is community support.)
see more details for https://docs.ros.org/en/humble/Releases/Release-Humble-Hawksbill.html#supported-platforms.

In container1 terminal, ros2 topic hz shows frequency slightly under 10hz

i believe that in this case, shared memory transport (not zero copy via LoanedMessage) can be enabled as localhost communication. this could be one of the benefits for the performance.

In container2 terminal, ros2 topic hz shows frequency highly under 10hz, verity from 7hz to 9hz

inter-process communication via network interface (in the case of docker, via bridge) would be one of the reason for this.

In log of container2, c++ code receive message each 100ms stably

there could be several differences, like language advantage for performance, rclpy pybind performance overhead.

above is my assumption, i did not test and analyze the performance difference. hopefully this helps.

[fujitatomoya]

There are a few issues here, I suggest the following.

• Performance issue of rclpy. This is well known issue that rclpy is not well-performative compared to rclcpp. (this is out of scope this issue.)
• No need to deserialize the message via hz and bw unless necessary, minor enhancement to make it more efficient for hz and bw sub-commands.
• add more description about how ros2 topic hz/bw (or anything else like these based on subscription) works. It might not show the expected frequency set on publisher, it just show what subscription/receiver actually get. and this is affected by QoS configuration and platform computation resource. (ros2 documentation can be also updated with quick note like this.)