We used the ESM++ model for protein property prediction. We observed two undesired aspects of current protein embeddings, as described here.
While we have ad-hoc internal fixes for 2nd point (sequence pair representation) it would be great if you were prepared to provide proper implementation in your repository.