Ch/psestimate memory

[cahofer]

A quadratic estimator that is distributed over power spectrum bands.

feat(QuadraticPSEstimationXLarge):

• a pipeline task to perform the m-mode quadratic powerspectrum estimation distributed over power spectrum bands. This solves the memory error problem when performing power spectrum estimation for large telescope classes. Depends on driftscan PR #91

[cahofer]

This PR goes along with the one in driftscan. Asking also @tristpinsm and @sjforeman additionally to @jrs65 for review. Thanks!

[sjforeman]

Looks good to me! Perhaps @jrs65 or @tristpinsm should also take a look before the final approval.

[jrs65]

Why does this need to be a separate class at all? Is there any reason this can't just replace the existing implementation? Or if not, merge them and change the behaviour to a config option, or even better, something that gets automatically determined based on the input data size.

[cahofer]

I mean, I could put them into one class and add a config option, but the two classes are fundamentally very different for the following reasons:

• the QuadraticPSEstimationLarge uses from driftscan the PSMontelCarloLarge class in which the clarray and the qestimator are MPIarrays parallelized over power spectrum bands. This is very different from the original code, in which clarray is gathered on all ranks and qestimators are calculated locally.
• This class respects the fact that clarray and qestimator are parallelized over bands, therefore the sky data need to be pre-computed for all m first, as each rank needs all m's. This is also very different from the original code, where the sky data were handled parallel in m.

If we were to use a config option - then we would have one class but and if/else statement in the process method handling the two different calculations.

I personally thought a separate class is cleaner, but after all - you are the boss ;)

[jrs65]

This looks like left over debug statements. Either remove it, or at least improve the formatting.

[sjforeman]

Would be good to keep, but agree about improving formatting

[sjforeman]

Would be good to keep, but agree about improving formatting

[jrs65]

Suggested change

	et = time.time()
	if mpiutil.rank0:
	print("m, time needed for quadratic pse", m, et - st)
	et = time.time()
	self.log.debug(f"{m=}, time needed for quadratic pse {et - st}")