[plugin] add workaround for recursive typedef overflow

[yuxiaomao]

Repro with -dce no.
Stack overflow when comparing types, e.g. haxe.ds.StringMap hvirtual - hmethod. This PR suggest to workaround the problem by stop checking when recursion is too deep.

However there's still another plugin bug that, excluded functions calls (at least some of them in fun$init) are not replaced by the base version of the function. Repro with --macro exclude("Type") (Edit: any static call that are excluded). I didn't find how to fix yet ^^'

[Simn]

It might be good to have this failsafe here, but this should really be fixed in the code generation. See HaxeFoundation/haxe#12539

[yuxiaomao]

I remember well that one, but I still don't know what to do ^^'
This bug isn't literally the same because this only occurs on hl.Api.loadPlugin, but the origin of both problem is probably related to HVirtual doesn't have a hash id or something

[Simn]

I somehow thought this was already fixed but either way, let's discuss that further over there.

As for this, isn't 10 a bit low?

[yuxiaomao]

I tried in haxe but never successed :< Maybe if we fix there, this problem will no longer exists.

I'll probably not merge this yet, until I find a fix or workaround for my other bug. As this is plugin-only, I didn't really think about how deep should I check (I can set it to 20 or more if you prefer, but I hope that most people doesn't write a very deep and non-recursive typedef)

[ncannasse]

That does not seem like a good fix :) could we instead mark temporarily the ->type value with -1 while we recurse and restore it after so we check for strict equality ?

[yuxiaomao]

Note:

• HL ttype infinity haxe#12682 uses a stack to protect recursive type compare
• I can't repro assert with "static call that are excluded" anymore ^^'
• @:virtual seems to have side effect on non-plugin build, e.g. when added to hrt.prefab.Prefab it cause game to null access

[ncannasse]

I don't think that @:virtual should have such side effect. To be investigated at it's really weird.

[ncannasse]

@yuxiaomao for the recursive tanon check, both tanon we are trying to check should be mutated with a single unique tag for the pair, and when found it should return true as we already have them on the stack. This is more efficient and safe than using a depth counter.

Since the assert is gone, I guess the remaining bug is the virtual one. It should not affect the program as its only impact should be to force using virtual calls for all methods instead of doing static calls when we know that the method has no override (we can't tell when it's a plugin). To investigate further.