Commit for review of ATF Selenium Support proposal#2
Commit for review of ATF Selenium Support proposal#2
Conversation
|
|
||
| ## Introduction | ||
|
|
||
| This proposal is to integrate the Generic HMI into ATF tests scripts by using Selenium to control a web browser programmatically. |
There was a problem hiding this comment.
As I know Generic HMI is not a test tool. It is an HMI implementation one.
So integration Generic HMI into ATF is not a good idea such as Generic HMI can not perform all HMI behavior which is needed to testing of SDL and even testing of HMI.
There was a problem hiding this comment.
It seems it was mentioned integrate testing of the Generic HMI
|
|
||
| ## Motivation | ||
|
|
||
| Currently ATF is used to test SDL Core as a black box. ATF uses simulated mobile applications and a simulated HMI to test SDL Core functionality. While ATF is great at testing SDL Core, the author aims to create a solution for having an automated test suite that tests both SDL Core and the Generic HMI together. |
There was a problem hiding this comment.
Tests aimed to test SDL is not really good to test HMI, because they cover HMI functionality really week.
However, it can be used to do some scope of HMI checks.
Another concern to motivation of this proposal is aiming to test two program systems at the same time. Such approach may lead to hard defect localization and fake negative test results from point of view of each of tested systems (SDL and HMI).
Examples:
- Test report has 001 - 004 failed test cases (001 and 002 are HMI defects, 003, 004 are SDL defects). It is all failed test cases need to be analyzed to consider that part of them are failed because HMI defects and another part because of SDL ones. (Additional time to fails investigation from test team)
- Test report has 001 - 004 failed ( all cases are HMI defects) SDL works as expected but test report is not confirms that. (Fake negative test results from point of view of SDL testing)
|
|
||
| #### Life Cycle | ||
|
|
||
| ATF will launch the python app using the os.execute() method. The python application will then launch a physical instance of the Generic HMI in the chromium browser. |
There was a problem hiding this comment.
I think a launch of python app can be performed during test environment preparation (by CI server for example).
So I would propose not to obligate ATF to lauch the app.
|
|
||
| #### Communication with ATF | ||
|
|
||
| This python application will have a websocket server that will receive messages from ATF for specific test actions and content validation. |
There was a problem hiding this comment.
A receiving of messages from ATF for specific test actions and content validation require Generic HMI implementation to be known to ATF and to be specified in test scripts. This is really bad practice because:
- any one change in Generic implementation lead to multiple changes in test scripts
- such information is redundant for test scripts in case of running of tests in "emulated HMI" mode, but still to be handled in ATF to ignore it
|
|
||
| #### Selenium Websocket Client | ||
|
|
||
| ATF will need to create a websocket client that relays messages to the python Selenium application. These messages will tell the WebDriver to click buttons and verify content displayed on the screen. |
There was a problem hiding this comment.
According to described above causes it is proposed to send to python HMI testing app commands (send data or expect data) with the same JSON data that is used into communication with SDL in "emulated HMI" mode. Python HMI testing app's responsibility to perform all needed actions and checks on Generic HMI (or other HMI implementation) on base of command from ATF and its JSON data.
Described approach allows:
- to avoid changes in test scripts on changes in Generic HMI
- implement another HMI testing app (for HMI implementation from a car manufacturer or using another programming language) without any changes in ATF
|
|
||
| #### HMI Websocket Server | ||
|
|
||
| ATF will need to create a websocket server that will receive messages from the HMI. This server will then be routed through the ATF test logic to verify the contents of the HMI messages. After verifying that the messages are valid, ATF will pass the HMI message to SDL Core. |
There was a problem hiding this comment.
This server will then be routed through the ATF test logic to verify the contents of the HMI messages. After verifying that the messages are valid, ATF will pass the HMI message to SDL Core.
This functionality has significantl influence on all timings in test scripts.
As was proposed above ATF sends to python HMI testing app a JSON to perform all checks on HMI testing app side and perform all needed actions on HMI to send such data to SDL directly after that report a result to ATF.
|
|
||
| #### Core Websocket Client | ||
|
|
||
| ATF already has a websocket client that connects to SDL Core but this piece will need to be extended to allow for validation of messages before they are passed to the HMI. |
There was a problem hiding this comment.
It is proposed to have next chain of connections
[ATF] (c)--(s) [HMI testing application] (c)--(s) [SDL]
| { | ||
| "UI.Show" : { | ||
| "ExpectRequest": { | ||
| "validate": [{ |
There was a problem hiding this comment.
All next data to validate HMI are HMI implementation specific and should not be in test scripts.
|
|
||
| ## Alternatives considered | ||
|
|
||
| An alternative solution would be to first create automated Selenium tests that only interact with the Generic HMI. This simplified solution wouldn't involve SDL Core or ATF. |
There was a problem hiding this comment.
An another alternative is "HMI testing application" approach:
- HMI testing application is specific application which can perform actions and verifications on specific HMI implementation
- ATF can connect and conduct of test run on specific HMI testing application via web socket connection
- ATF can perform testing using emulated HMI testing application (current approach)
- ATF test scripts are independent from details of HMI implementation
2 participants
Review of ATF Selenium Support proposal