Fixed Bean and Encryption Issues Causing Order-Dependent Flakiness in Multiple Tests

[LucasEby]

Fixes #1465

Motivation

Multiple tests show order-dependent behavior because they rely on shared static encryption state that is mutated by other tests. In particular, the polluter test EncryptorConfigTest.testEncryptionWithoutEncryptorImplementation removes the encryption bean from the Spring context and leaves two global static variables in a polluted state:

• EncryptionProvider.encryptor (cached encryptor instance)
• SlangEntitiesSpringConfig.applicationContext (shared application context reference)

When EncryptorConfigTest.testEncryptionWithoutEncryptorImplementation runs before the following victim tests:

• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValue
• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted
• io.cloudslang.fortest.SensitiveValueTest.testSensitiveValueEncryptDecrypt
• io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive

these victim tests fail because they attempt to construct SensitiveValue instances which require a valid encryption bean and a correctly initialized Spring context. They fail because they either use a cached encryptor with the wrong context, the encryption bean cannot be resolved because the application context is misconfigured, or throw "Application context bean is missing" errors.

A similar dependency exists for the victim test DeserializeTest.testDeserializeInput, which only passes when a prior test (such as SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted) has populated the correct Spring context (SlangEntitiesSpringConfig.applicationContext) and reset the cached encryptor. If it runs in isolation or before the necessary state is established, it fails due to missing DummyEncryptor bean and stale static state.

Order dependent tests like these are a cause of test flakiness which is particularly harmful in CI: they break the assumption that the tests can be reordered or run in parallel without changing their outcome. As a result, each of the 5 mentioned victim tests can fail due to external factors despite the test and the code that it is testing remaining unchanged, causing unreliable results from CI and eroding developer trust in the test suite.

Modifications

A new reset() method was added to EncryptionProvider to provide a clean API for clearing the static encryptor cache without requiring reflection. This method sets the cached AtomicReference<Encryption> back to null, allowing the encryptor to be lazily re-initialized with the correct Spring context.

A new abstract base class SpringEncryptionTestBase was introduced to ensure static state is properly initialized before each test method. Before each test, it:

• Calls EncryptionProvider.reset() to clear the cached encryptor instance
• Re-injects the test's ApplicationContext into SlangEntitiesSpringConfig.applicationContext using the existing setApplicationContext() method

Each of the affected test classes (ValueFactoryTest, SensitiveValueTest, and DeserializeTest) now extend this base class. Additionally, SlangEntitiesSpringConfig bean configurations were added to ValueFactoryTest to ensure the base class can properly inject the application context.

The cleanup method in EncryptorConfigTest was also updated to use EncryptionProvider.reset() instead of reflection-based field manipulation. This makes the cleanup more maintainable and resilient to future changes in the EncryptionProvider implementation, as it no longer depends on the internal field name or structure.

Affected Tests

This change affects the existing tests:

• io.cloudslang.lang.entities.bindings.values.ValueFactoryTest.testCreatePyObjectValueFromValueSensitive
• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValue
• io.cloudslang.fortest.SensitiveValueTest.testEncryptedStringSensitiveValuePreEncrypted
• io.cloudslang.fortest.SensitiveValueTest.testSensitiveValueEncryptDecrypt
• io.cloudslang.lang.entities.DeserializeTest.testDeserializeInput

Tools Used

iDFlakies was utilized to identify the order dependent polluter(s), victim(s), and cleaner(s) in the test suite(s). The tool functions by repeatedly running a project's JUnit tests under many different deterministic test orders to detect flaky tests which pass and fail inconsistently, and then re-executes the failing orders to classify each flaky test as either order-dependent or non-order-dependent. iFixFlakies was executed on the output to further diagnose the source of the problems and additionally generate test helper patches from existing cleaners that were found. With the combination of these tools, I was able to more easily diagnose the source(s) of the problem(s) and create effective solution(s) for them.

[LucasEby]

I had to check out an older version of the repo to utilize the above mentioned tools since they do not support the current JDK version. After opening the PR I discovered that some of the tests had already been fixed so the proposed parent class is no longer needed.

[LucasEby]

This PR was re-opened after investigating the previously accepted PR 1427 and determining that it did not entirely fix the order dependent issues that were identified in the SensitiveValueTest's methods. While the previous solution correctly identified one of the polluters and mitigated it, it did not mitigate all of the order dependent sources of pollution which cause the SensitiveValueTest's methods to fail. This is because there are other polluters that exist in the test suite such as ValueFactoryTest.testCreatePyObjectValueFromValueSensitive which still pollute the tests, causing them to fail.