Patch seems to break (or is ignored by) testcoverage tooling?

[DGollings]

First off, I really liked the API/implementation of Patch. Really enjoy using it so thanks for making it

But whilst experimenting with some code coverage tooling (mix test --cover) noticed that code that uses Patch.patch disappears from the reports.

So decided to clone this repo and noticed the same. Picked a test at random that uses patch(), refute_any_call_test.exs

❯ grep 'patch(' ./test/user/refute_any_call_test.exs
      patch(RefuteAnyCall, :function_with_multiple_arities, :patched)
      (..)
      patch(RefuteAnyCall, :other_function, :patched)
      patch(RefuteAnyCall, :function_with_multiple_arities, :patched)

but then:

mix test --cover 2>&1 | grep RefuteAny

outputs nothing, same for

grep RefuteAnyCall cover -r

which, seeing all the generated html files in cover/ seems unlikely

I assume this is known behaviour? I can imagine its a side-effect of how the 'super-powers' work but are you aware of a workaround or fix? Because its a bit ironic that the most well tested code in my codebase shows up as 0% or not at all :)

And a ps. Running the test directly mix test --cover ./test/user/refute_any_call_test.exs does show the User.Patch modules nearby but almost all of them as 0% coverage

[ihumanable]

@DGollings This is a known issue, I've got it on the roadmap to fix this. I'll see if I can figure it out soon ™️

[DGollings]

thanks @ihumanable , but you think its fixable? That's good to hear then, thanks

[ihumanable]

I think so, I've actually got some time today to take a look so 🤞

[JonasGruenwald]

This is pretty tricky, when running the mix coverage task, it just cover-compiles all the modules once before running all the tests, which of course gets broken immediately when patch recompiles and reloads a cover-compiled module

https://www.erlang.org/doc/apps/tools/cover_chapter.html#code-loading-mechanism

When a module is cover-compiled, it is also loaded using the normal code loading mechanism of Erlang. This means that if a cover-compiled module is re-loaded during a Cover session, for example using c(Module), it will no longer be cover-compiled.

I was poking around a bit but I can't really see a way to fix this in patch without changing how it works, even if you were to restore a module to its original after mocking it, and then cover-compile it again, that would wipe any coverage data collected already:

https://www.erlang.org/doc/apps/tools/cover.html#compile_module/2

Note that the internal database is initialized during the compilation, which means that any previously collected coverage data for the module is lost.

I think I'll switch back to Mox as I don't see how this can be solved with the architecture of patch at the moment.