The MathJax JS library is quite large, so ideally it would be loaded only on pages that actually contain equations -- else it's just wasted bandwidth. Currently the Simple-MathJax plugin loads the library globally.
I appreciate there's a tradeoff here with the simplicity of the plugin, so no worries if this is a "won't fix" issue.