[Feature] Two-way python bridge with limited MAPL support in Python

[FlorianDeconinck]

Types of change(s)

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Trivial change (affects only documentation or cleanup)
• Refactor (no functional changes, no api changes)

Checklist

• Tested this change with a run of GEOSgcm
• Ran the Unit Tests (make tests)

Description

This PR aims at solving the "90%" case of Python integration has it as been seen lately at GMAO. The main features are:

• One liner to drop into python, with dynamic package load (with a simple structure for user to hook in)
• MAPL/ESMF states pass-through with a limited API to pull memory down in Python
• Tooling for common Python operations (ongoing, more to come)

The entire API is heavily #ifdef by a flag at compile time BUILD_PYTHONBRIDGE - by default the functions are passthroughs.

Review points

@tclune / @atrayano: it was non-trivial for me to find a spot where I could initialize the system with the atmospheric grid. I ended up calling in GEOSAgcm_GridComp (see sister PR). This is only needed because right now the bridge back to MAPL gives me the pointer "blind", without sizes and rank. So I am giving the GRID to users so we can tool down.
A better version would be to augment the "MAPLpy" bridge to get shape and ranks, but I'd like to see that in a subsequent version so we can stress test this bridge first.

@mathomp4: I ended up not feeding the add_definitions upstream like we talked about. I kept it within the MAPL.python_bridge cmakelist and forwarded the API there. This looks cleaner and achieve the "opt-in by default" behavior we wanted.

@pchakraborty: You should find the design quite familiar. It's a CFFI bridge, like we've done before. The trick relies in the _get_code_object_from_package_name where I dynamically load the user given module and search for a CODE object. The README spells it out with some code. The rest is marshalling up & down memory via void*.

[tclune]

We will have a meeting this afternoon to take a quick look. A deeper dive will wait until next week.

[FlorianDeconinck]

Python API updated to simplify retrieving memory. Now you can do

MAPLPy = get_MAPLPy()
T = MAPLPy.get_pointer("T", state=import_state, dtype=np.float32, dims=MAPLPy.grid_dims)

and you get Fortran memory mapping in T, while T is a numpy array.

[mathomp4]

Well, that coupled CI failure is weird. I'm re-running it.

[FlorianDeconinck]

With the latest round of Python work I believe we have reached MVP - review away!

[mathomp4]

I tested with NAG by hand and it's happy too.

[tclune]

There are a few naming convention changes that I think would be good. (There will be more for MAPL3 issues, but we can handle those later.)

Biggest question is where the bridge is initialized?

[FlorianDeconinck]

@tclune aptly pointed out the file naming is a bit random, I am going to a pass to clarify it all

[FlorianDeconinck]

@pchakraborty : renaming and file move to Python has been done, ready for re-review

@mathomp4 : installed into Python/ seems to be a-ok

[tclune]

Can we resolve the remaining comments and get this merged?

[FlorianDeconinck]

Can we resolve the remaining comments and get this merged?

Ready for merge on my side.

[tclune]

Yay - done for now!

[FlorianDeconinck]

Thank you everyone