Tracking issue: Build a performance test-suite for various real-time operating systems

[shannmu]

Tracking issue

• Add a github bot to track the issues/PRs and report the performance. (enable github bot && gitmaya  #23)

[shannmu]

The RROS community is preparing to develop a test suite compatible with common RTOS, aimed at comparing the performance of RROS with other mainstream RTOS. This thread mainly discusses the following aspects (other potential topics are also welcome for discussion):

• Which RTOS to support
• Which common functionalities of RTOS to perform performance testing on, and whether to test specific features of certain key RTOS
• How to minimize the testing difficulty caused by the differences in interfaces between different RTOS
• What techniques to use for profiling, and a discussion on the pros and cons of different techniques

[Richardhongyu]

IMO, performance test construction is divided into three parts:

1. A CI system, Github-bot and optional triggers for performance testing;
2. A Test framework: provides optional performance test options, as well as optional pressurization content, runs in the CI system, this should be a user-mode program;
3. Test code and Pressurized code: Test code is our core. The first version may use existing standards, but we need to design our own for dual-core and robot/satellite scenarios later. It also needs to be compatible with the POSIX interface, and exist as user-mode code or kernel driver; Pressurized code: This can be used off-the-shelf, but the variety should be complete, including CPU/GPU/DISK.

[Richardhongyu]

• Which RTOS to support

Not only RTOS but also other real-time kernels. Like Preempt-RT Linux, Xenomai/EVL, one typical RTOS(maybe zephyr?), RROS.

• Which common functionalities of RTOS to perform performance testing on, and whether to test specific features of certain key RTOS

We need a draft to discuss. Our previous testbed for RROS could be placed here.

• How to minimize the testing difficulty caused by the differences in interfaces between different RTOS

The test code should be written with a POSIX interface or standard Linux ABI.

• What techniques to use for profiling, and a discussion on the pros and cons of different techniques

Good point! Currently, our logging mechanism sometimes influences the profiling. We need to investigate this.