Generate reproduction script from GUI input

[jokasimr]

A problem when working with the GUI is that you cannot easily reproduce the results.
If you worked in a jupyter notebook it describes the code you ran to produce the output.
Same if you work with a basic script.

To fix this we could let the user to export the GUI state to be able to import it at a later time.
This is probably a useful feature we should have, but it's not really a good way to enable reproducibility, because the user still has to know how to do some more steps in order to actually produce the results (for example, press the "reduce button").
Another problem is that it's very easy to create the results using the GUI, then change some field, forget that you did, and then export the gui state which now does not match the results produced.

Another option is to let the user export a script (probably bash or python) that does everything you need to do to reproduce a set of output files.
The script would contain the gui state and when executed it would

• create an environment
• install the same dependencies as in the environment the gui runs in
• run the workflow to produce the results

The script could then be stored in scicat and would enable anyone to reproduce the results without running the gui.

To illustrate a bit more how I imagine this could work here is a code snippet:

import sys
import json
import subprocess
import pandas


def create_reproduction_script(
    parameters,
    template,
):
    requirements = subprocess.check_output([sys.executable, "-m", "pip", "freeze"]).decode()
    # or maybe?
    requirements_conda = subprocess.check_output(['conda', 'list', '--export']).decode()

    parameters = json.dumps({
        name: value.to_csv() if isinstance(value, pandas.DataFrame) else value
        for name, value in parameters.items()
    })
    return template.substitute(
        requirements=requirements,
        requirements_conda=requirements_conda,
        parameters=parameters
    ) 

# reproduce_gui.py.template
import sys
import json
import subprocess
import tempfile


requirements = '''$requirements'''
requirements_conda = '''$requirements_conda'''

# Install dependencies
with tempfile.NamedTemporaryFile() as f:
    f.write(requirements.encode())
    subprocess.check_output([sys.executable, '-m', 'pip', 'install', '-r', f.name])


with tempfile.NamedTemporaryFile() as f:
    f.write(requirements_conda.encode())
    subprocess.check_output(['conda', 'install', '-y', '--file', f.name])


import pandas
# Data definitions
parameters = json.loads('''$parameters''')
parameters = {name: pandas.read_csv(value) for name, value in parameters.items()}

# Import workflow
from ess.reflectometry.gui import AmorBatchReductionGUI
run = AmorBatchReductionGUI.workflow


if __name__ == '__main__':
   run(parameters)

[jokasimr]

This works, with some caveats, for example it can't install packages installed with pip install -e ..
But that's a case I don't think we need to handle in practice.

I'm sure it doesn't use the capabilities of conda and pip in the best way, how can this be made better / more robust ?
What do you think about the approach? @scipp/ess-maintainers

[jl-wynen]

I'm sure it doesn't use the capabilities of conda and pip in the best way, how can this be made better / more robust ?

Given that users are free to install, update, and remove packages, I think getting a full recording of the current env using pip freeze and conda list is best. Not sure if this is portable to other platforms, though. (A small thing, but pip is a Python package. So you should be able to import it.)

Your approach for saving parameters is quite limited. We need a way to, at least, handle Scipp objects.

Please see scipp/sciline#92. It discusses this question. As long as a GUi builds a sciline pipeline, we can already save it. We can also replay a graph, see scipp/sciline#147. But handling parameters is the real difficulty in general. In practice, inputs from GUIs are small and we should be able to serialise them to JSON alongside the graph.

Saving a task graph has the advantage that we get the exact version of the graph that was used to make some data. It doesn't depend on user-controllable state of a notebook.
Do we really need to recreate a GUI? Is it not enough to have a human readable (albeit complicated) JSON repr and a script to run the workflow?

[jokasimr]

(A small thing, but pip is a Python package. So you should be able to import it.)

As I understand pip recommends the above way of doing it, oddly enough.

Your approach for saving parameters is quite limited. We need a way to, at least, handle Scipp objects.

It was a bit unclear in the description The parameters here is what I called the "GUI state" (and you called the GUI input). It's just widget values. It's the job of the run function to take the gui input and use it to populate a sciline workflow with scipp datastructures.

(This is really what I see as the big advantage of this approach, it only needs to serialize a controlled set of small input values.)

Do we really need to recreate a GUI? Is it not enough to have a human readable (albeit complicated) JSON repr and a script to run the workflow?

No I don't think we should recreate a GUI. ☺️ The script should only run a workflow to produce files.

Importing/exporting the GUI state is a feature I think we should also have, because it's convenient for users, but it should not be the mechanism that we rely on for reproducibility.

Saving a task graph has the advantage that we get the exact version of the graph that was used to make some data. It doesn't depend on user-controllable state of a notebook.

I don't see the advantage of having a serialized graph if we have a script. I mean, if we use GUI.workflow(parameters) to generate results in the GUI, and we use GUI.workflow(parameters) to generate the results in the reproducing script, we are going to get the same results in both cases. What is the advantage of the serialized graph?

[jokasimr]

(fat fingered the close button)

[jl-wynen]

I don't see the advantage of having a serialized graph if we have a script. I mean, if we use GUI.workflow(parameters) to generate results in the GUI, and we use GUI.workflow(parameters) to generate the results in the reproducing script, we are going to get the same results in both cases. What is the advantage of the serialized graph?

The advantage is that we would not be limited to reproducing results of using the GUI.