XCMetrics: Measure performance metrics using XCTest

XCMetrics is a reference implementation to accurately measure performance metrics of your code using Unit and UI tests. It provides detailed examples covering both old( i.e. XCTPerformanceMetric) & the new(i.e. XCTMetric) performance-testing system. It also fixes bugs/peculiarities in the XCTest performance APIs with runtime hooks(i.e. Monkey Patching), making it more stable & consistent.

Purpose

Performance testing using XCTest framework(i.e. XCTMetric confirming types) have some rough edges i.e. bugs/peculiarities. This project aims to fix all those bugs in the performance APIs using Swizzling, making a consistent experience on both iOS 13 & 14. Here's the list of open issues and radars:

1. Default Launch Performance Test Crashes Occasionally
2. Getting error measuring signpost metrics in XCUITests
3. XCTests crash randomly in parallel testing
4. XCTOSSignpostMetric doesn’t detect signposts logged by static library
5. Lost connection to test manager service
6. Lastly, Lack of performance tests documentation

Identifying Performance Bottlenecks & Regressions

Performance tests helps in identifying the impact of your code in terms of various metrics like CPU, Clock, Memory, Storage, Signpost, AppLaunch and more. Capturing performance metrics in your CI pipeline helps in identifying performance bottlenecks and regressions produced by your code over-time. If your performance metrics are going off, then Instruments can help in analysing the root cause.

What's Inside

Detailed examples of performance testing covering the usage of following metrics:

1. XCTCPUMetric to record information about CPU activity during a performance test.
2. XCTClockMetric to record the time that elapses during a performance test.
3. XCTMemoryMetric to record the physical memory that a performance test uses.
4. XCTOSSignpostMetric to record the time that a performance test spends executing a signposted region of code.
5. XCTStorageMetric to record the amount of data that a performance test logically writes to storage.
6. XCTApplicationLaunchMetric to record the application launch duration for a performance test.
7. MonotonicClockMetric to record the time(in nanoseconds) that elapses during a performance test, implemented as a custom XCTMetric.
8. scrollDecelerationMetric to record scroll deceleration animations.
9. scrollDraggingMetric to record scroll-dragging animations.
10. navigationTransitionMetric to record the duration of navigation transitions between views.
11. wallClockTime to record the time in seconds to execute a block of code.
12. And, all private metrics of XCTPerformanceMetric system.

Tools & Support

1. Xcode 12.0 Beta 6, Build version 12A8189n.
2. Swift 5.3 and later.
3. iOS 13 and later.

Dependencies

XCMetrics uses InterposeKit for adding runtime hooks(i.e. swizzling).

Contributing

XCMetrics welcomes contributions in the form of GitHub issues and pull-requests:

1. For PRs, please add the purpose and summary of your changes in the PR description.
   
2. For issues, please add the steps to reproduce and tools/OS version.
   
3. Make sure you test your contributions.
   
4. Ideas and suggestions are very welcome!

By submitting a pull request, you represent that you have the right to license your contribution to Soaurabh Kakkar and the community, and agree by submitting the patch that your contributions are licensed under the XCMetrics project license.

License

XCMetrics is licensed under the MIT License

References

https://developer.apple.com/documentation/os/logging/recording_performance_data
https://developer.apple.com/documentation/xctest/performance_tests
https://developer.apple.com/documentation/xctest/xctestcase
https://indiestack.com/2018/02/xcodes-secret-performance-tests
https://developer.apple.com/videos/play/wwdc2018/405
https://developer.apple.com/videos/play/wwdc2019/417
https://developer.apple.com/videos/play/wwdc2020/10077

Building an end to end performance tests CI pipeline? Give XCTool a try!