Question about strongly skewed in-re results on DomainNet (Table 5)

[LeeYuanzhuo]

Hi and thanks a lot for releasing the PLeaS code and the paper!

I have a question about the DomainNet results in Table 5 of the paper (“Detailed Results on DomainNet, ResNet-50”).

For the in-re pair under Budget 1.2, Original data, the numbers look very skewed between the two domains:

• PLeaS-Act (1.2, Original)

  • in: 26.3 ± 1.2
  • re: 69.8 ± 0.5

• ZipIt! (1.2, Original)

  • in: 60.0 ± 1.2
  • re: 21.9 ± 1.1

So PLeaS seems to heavily favor the re domain (real), while ZipIt! seems to heavily favor the in domain (infograph). This strong asymmetry also seems to persist, to some extent, at higher budgets.

I’m trying to understand whether this is mainly:

1. A consequence of the optimization objective / heuristics of the two methods (e.g., PLeaS’s LS fitting on mixed-domain activations tending to preserve the model that is more stable on the union of in+re, versus ZipIt! preserving more domain-specific features for one side),
2. An artifact of the evaluation protocol (e.g., how the two per-domain models are sampled / averaged over random seeds and permutations), or
3. A more fundamental property of this particular domain pair (Infograph vs Real being extremely different in DomainNet), meaning that any single merged model under such a tight compute budget (1.2×) will almost inevitably sacrifice one domain heavily.

Concretely, my questions are:

• Do you have any additional diagnostics or ablations (e.g., cross-domain performance of the individual in-only and re-only models before merging, or per-pair performance at different budgets) that help explain why in-re ends up so skewed in opposite directions for PLeaS and ZipIt!?
• When you looked at per-pair results internally, did you see similar extreme trade-offs for other domain pairs, or is in-re a special / particularly difficult pair?
• If we wanted to bias PLeaS towards a more “balanced” trade-off between the two domains (rather than optimizing for overall mixed-domain LS fit), do you have recommendations? For example:
  • changing the sampling ratio between the two domains when computing activations,
  • weighting the LS objective differently per domain,
  • or modifying the permutation matching stage to encourage fairness across domains?

Any clarification or intuition you can share about this in-re behavior would be really helpful for understanding how to use PLeaS in practice on strongly shifted domain pairs.

Thanks again for the great work and for open-sourcing the code!