Fix: Unit test CTE failures not being captured

[VaggelisD]

Fixes #5067

In unittest internals TestCase::addSubTest appends the result of that subtest directly to self.errors or self.failures:

    def addSubTest(self, test, subtest, err):
        """Called at the end of a subtest.
        'err' is None if the subtest ended successfully, otherwise it's a
        tuple of values as returned by sys.exc_info().
        """
        # By default, we don't do anything with successful subtests, but
        # more sophisticated test results might want to record them.
        if err is not None:
            if getattr(self, 'failfast', False):
                self.stop()
            if issubclass(err[0], test.failureException):
                errors = self.failures
            else:
                errors = self.errors
            errors.append((subtest, self._exc_info_to_string(err, test)))
            self._mirrorOutput = True

However, our custom classes expected that this'd happen through addError(...) or addFailure(...) so that we can intercept these exceptions (since they get turned into strings before being stored in their parent classes).

To solve this regression, ModelTestTextResult::addSubTest now follows the parent implementation and instead calls the respective APIs which do fill our own original_failures & original_error structures.

The output for that looks like:

❯ sqlmesh test
F
======================================================================

----------------------------------------------------------------------
FAIL: test_example_full_model (/Users/vaggelisd/Desktop/tobiko/test_dir/tests/test.yaml)
----------------------------------------------------------------------
         Data mismatch (CTE "filtered_orders_cte")          
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃   Row    ┃      id: Expected       ┃     id: Actual      ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│    0     │            5            │          1          │
└──────────┴─────────────────────────┴─────────────────────┘

----------------------------------------------------------------------
Test Failure Summary
======================================================================
Ran 1 tests against duckdb in 0.09 seconds. 

Failed tests (1):
 •<path>::test_example_full_model
======================================================================

Docs

unittest.TestResult | unittest.TestCase | unittest.runner

[VaggelisD]

This is to differentiate the CTE mismatch from the output one, e.g if both subtests fail in a single test that'd be the output:

FF
======================================================================

----------------------------------------------------------------------
FAIL: test_example_full_model (/Users/vaggelisd/Desktop/tobiko/test_dir/tests/test.yaml)
----------------------------------------------------------------------
         Data mismatch (CTE "filtered_orders_cte")          
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃   Row    ┃      id: Expected       ┃     id: Actual      ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│    0     │            5            │          1          │
└──────────┴─────────────────────────┴─────────────────────┘

----------------------------------------------------------------------
FAIL: test_example_full_model (/Users/vaggelisd/Desktop/tobiko/test_dir/tests/test.yaml)
----------------------------------------------------------------------
                       Data mismatch                        
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃   Row    ┃      id: Expected       ┃     id: Actual      ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│    0     │            2            │          1          │
└──────────┴─────────────────────────┴─────────────────────┘

----------------------------------------------------------------------
Test Failure Summary
======================================================================
Ran 1 tests against duckdb in 0.14 seconds. 

Failed tests (1):
 • /Users/vaggelisd/Desktop/tobiko/test_dir/tests/test.yaml::test_example_full_model
======================================================================

[VaggelisD]

Simple abstraction to simplify console.py

[VaggelisD]

Previously we copied these files directly but unit tests don't work well if the name is duplicated, I'll follow up with a UX fix

[VaggelisD]

dead code, infos is no longer used

[izeigerman]

I wonder if it even makes sense at this point to rely on subtests rather than write our own context manager which sets the current CTE and then resets it.

[izeigerman]

Are there any other benefits that subtests offer us here?

[VaggelisD]

I had the same thought as I was working on this, I think it's merit is that it triggers the callback addSubTest when it finishes, otherwise we'd have to schedule that ourselves.

Not really a strong argument for it, but I'm not sure if there's a strong argument against it either. Happy to reconsider if you had a better thought with the custom CM