Skip to content

Conversation

@hofi1
Copy link
Owner

@hofi1 hofi1 commented Oct 11, 2023

This PR fixes two flaky tests:

  • org.json.junit.JSONObjectTest#valueToString
  • org.json.junit.JSONMLTest#testToJSONObject_reversibility

The flaky test have been found by using the NonDex tool.

org.json.junit.JSONObjectTest#valueToString

Problem:

The string in the assertion can change, because the Map which is used to store the data in the JSONObject returns the data in non-deterministic order (the order of child elements in JSON strings do not matter, and JSON strings are equal regardless of the ordering of the elements on the same hierarchy level)

The flakiness was discovered on the following lines

assertTrue("jsonObject valueToString() incorrect",
JSONObject.valueToString(jsonObject).equals(jsonObject.toString()));

as well as

assertTrue("map valueToString() incorrect",
jsonObject.toString().equals(JSONObject.valueToString(map)));

Solution:

Changed the assert statement from a JUnit Assertion to a JSON Assertion, so the strings are not compared charwise but are compared the way as specified for JSON strings (not taking care of the order of the elements on the same level).
The JSONAssertion library was chosen because it provides general assertions to compare JSON strings without the need of writing complex JUnit Assertions, what would need a lot of work to convert the strings in a different data structure, a lot of boilerplate code as well as the chance of adding more errors than fixing.

Result:

The test is deterministic and not flaky. This improves the quality of the test and reduces the time to search for the bug during future development.

Reproduce:

mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.JSONObjectTest#valueToString

org.json.junit.JSONMLTest#testToJSONObject_reversibility

Problem:

The string in the assertion can change, because the Map which is used to store the data in the JSONObject returns the data in non-deterministic order.

Solution:

Change the assertion is to a JSONAssertion (e.g. the order of child elements in an element do not matter)

Result:

The test is deterministic and not flaky. This improves the quality of the test and reduces the time to search for the bug during future development.

Reproduce:

mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.json.junit.JSONMLTest#testToJSONObject_reversibility

@hofi1 hofi1 force-pushed the bugfix/fix-JSON-flakiness branch from 732155b to 468f4b2 Compare October 14, 2023 02:10
@hofi1 hofi1 force-pushed the bugfix/fix-JSON-flakiness branch from 468f4b2 to 29a7f46 Compare October 14, 2023 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants