Testing non-determinism (ruby and threads)

[goobber-gawber]

Hi!

I'm in the process of packaging klayout for GNU Guix. I think by now I've got almost everything sorted out - except for 1 unit test failure.

Running rbaTests:basic
    Special chars: <&>
    Loaded suite Basic_TestClass
    Started
    ...................F
    ==============================================
    Failure: test_27(Basic_TestClass)
    /tmp/guix-build-klayout-0.30.5.drv-0/source/testdata/ruby/basic_testcore.rb:1684:in `test_27'
         1681:     end
         1682:
         1683:     GC.start
      => 1684:     assert_equal(RBA::B.has_inst, false)
         1685:
         1686:   end
         1687:
    <true> expected but was
    <false>

    diff:
    ? tru e
    ? fals
    ==============================================
    ....................ANYTHING
    ANYTHING
    .....Special chars: <&>
    ..........

    Finished in 0.516415394 seconds.
    ----------------------------------------------
    55 tests, 1151 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
    98.1818% passed
    ----------------------------------------------
    106.50 tests/s, 2228.83 assertions/s
    Test rbaTests:basic failed:
    (last checkpoint: ../../../src/rba/unit_tests/rbaTests.cc, line 65): Exception caught: RuntimeError: Tests failed (1 Errors + Failures)
      /tmp/guix-build-klayout-0.30.5.drv-0/source/testdata/ruby/test_epilogue.rb:36:in `<top (required)>'
      /tmp/guix-build-klayout-0.30.5.drv-0/source/testdata/ruby/basic.rb:32:in `load'
      /tmp/guix-build-klayout-0.30.5.drv-0/source/testdata/ruby/basic.rb:32:in `<top (required)>'

Since this is the only failure I somewhat doubt that it's a packaging issue - but I'm way out of my ball-park here. Any ideas? I guess klayout works fine otherwise (but I have no means to check that for I am not an EDA designer).

[klayoutmatthias]

Hi @goobber-gawber,

That is definitely not a packaging issue.

Ruby has some difficulties with the garbage collector - sometimes you need to call GC twice to get your objects garbage collected for sure. Maybe there is also a bug on my side with mark-and-sweep, but that is horribly difficult to debug. They are also tuning the GC, so new Ruby versions may offer new surprises. I also suspect that Ruby intentionally does not clean all objects and keeps some objects "in stock" for speeding up new object creation.

I am using Ruby up to 3.2 which comes with Ubuntu 24 and so far, this test passes for me. If you are using a more recent version, that may explain the fail. It may help to duplicate the "GC.start" line, but if it does not, I'd file that under "weird Ruby behavior" and ignore that fail.

Python BTW is much better in terms of consistent memory management as they use reference counting. The same issue in Python I'd take more seriously.

Best regards,

Matthias

[goobber-gawber]

not sure if i can trust this resource [SO] but it suggests that GC can only be advised to start, and not forced by the language front end. even though i can not confirm by reading the relevant passage in the ruby reference manual the reasoning sounds solid.

are the GC tests in klayout essential for klayout to work? if not - and if you also are having some issues already - i'd suggest to omit these, as their lack of determinism or dependence on specific builds/backends could hinder building klayout.

btw: in Guix we're currently building it with ruby 3.3.9

[goobber-gawber]

is it also known that tests in tlThreadedWorkersTests fail in non-deterministic ways? maybe depending on the testing machine's perfomance?

[klayoutmatthias]

The Ruby tests never fail for me with Ruby <= 3.2 on all my supported platforms, so I guess it's something related to the new Ruby version. It's not ultimately important to have objects garbage collected, but if you use very heavy objects, you would like them to be freed when they are no longer needed. Hence, I consider the GC tests important, yet they may suffer from GC peculiarities and a fail is not necessarily a bad sign, but needs investigation. I don't want to delete them as on my platforms they provide an essential regression. Right now, I don't have the bandwidth to debug another Ruby version.

The ThreadedWorkersTests sometimes fail under heavy load. As with the Ruby GC test, they are useful to me and without excessive CPU overbooking they do not fail within my private CI infrastructure. You can exclude the failing tests from the unit tests (ut_runner -x ...) if you consider them too unstable. Empirically they fail more often on older Linux versions (such as CentOS 7) and typically the failing one is "tlThreadedWorkersTests:10".

I have no indicating that this is due to a catastrophic failure. What the test does is to perform 10 million locked "add +1" on four parallel threads. I give it 2 seconds for that, which is more than enough under normal conditions, but under load this may not be sufficient.

Best regards,

Matthias