docs: add remote computing tutorial by leahaeusel · Pull Request #265 · queens-py/queens

leahaeusel · 2025-12-16T09:04:29Z

Description and Context:
What and Why?

This PR adds a tutorial on remote computing with 4C. It combines our old wiki entry on GitLab with an updated version of tutorial 4 on GitLab.

Beyond that, I have cleaned up the TestCluster class to clearly separate tests, cluster configurations, fixtures, and functions while avoiding duplicate code.

I have tested the changes in the cluster tests with our pipeline on GitLab.

Any additional input on how you manage remote QUEENS runs is warmly welcome!

Related Issues and Pull Requests

Closes Add a Cluster-Usage Tutorial #264
Related to Tutorials #58

Interested Parties

@BishrMaradni

Note: More information on the merge request procedure in QUEENS can be found in the Submit a pull request section in the CONTRIBUTING.md file.

leahaeusel · 2025-12-16T09:06:51Z

tutorials/3_grid_iterator_fourc_remote.ipynb

+    "> **Subclassing**:\n",
+    "> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",
+    "> You can do this in one of the following ways:\n",
+    "> 1. Include the custom classes in the QUEENS repository. They will be automatically copied to the remote QUEENS repository at the start of the QUEENS runs.\n",
+    "> 1. Manually synchronize the custom classes between your local and your remote machine, e.g., via an additional repository and install this repository in the remote QUEENS environment."


This section was not part of the old files, and I would welcome your thoughts on it.

I have only used the first option in my cluster runs, and I would assume it is the most reliable option

Thanks, that's good to know! I agree that it's the more reliable option and have therefore added that it's the recommended way.

Copilot

Pull request overview

This PR adds a comprehensive tutorial on remote computing with 4C that demonstrates how to run QUEENS experiments on a remote cluster using SSH port forwarding. It consolidates and updates existing documentation from GitLab.

Key changes:

New Jupyter notebook tutorial showing remote 4C simulations with a Grid iterator
4C input template file for the hex8 stress extrapolation example
Integration test framework for the remote computing tutorial

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tutorials/3_grid_iterator_fourc_remote.ipynb	Main tutorial notebook with step-by-step instructions for setting up SSH, configuring remote connections, and running 4C simulations on a cluster
tutorials/solid_runtime_hex8.4C.yaml	4C input template defining a structural mechanics problem with templated Neumann boundary conditions
tests/tutorial_tests/test_3_grid_iterator_fourc_remote.py	Placeholder file to exclude tutorial from standard test execution
tests/integration_tests/cluster/test_dask_cluster.py	Integration test that executes the tutorial with injected credentials and validates results
src/queens/drivers/jobscript.py	Enhanced documentation clarifying local vs. remote paths for input templates and executables

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tutorials/3_grid_iterator_fourc_remote.ipynb

BishrMaradni

Thank you for contributing in enriching the ever-growing tutorial library!

BishrMaradni · 2025-12-17T12:36:01Z

tests/tutorial_tests/test_3_grid_iterator_fourc_remote.py

+"""Placeholder for integration test of 3-grid-iterator-4C-remote tutorial.
+
+The actual test is in tests/integration_tests/cluster/test_dask_cluster.py.
+This file ensures that the tutorial is not run by tests/tutorial_tests/test_tutorials.py.
+"""


Is there a reason we are putting the testbook in the cluster test instead of the tutorial_tests?

Yes, I am testing the new notebook in the cluster tests because this notebook can only execute successfully when a remote computing resource is available. This is not the case for the normal tutorial tests, which are part of the local tests on the GitHub runner. The cluster tests are not part of this pipeline but are run once per night on GitLab and have access to thought / charon.

BishrMaradni · 2025-12-17T13:00:22Z

tutorials/5_grid_iterator_4c_remote/5_grid_iterator_4c_remote.ipynb

The notebook in the documentation has no output due to the placeholders not being replaced. Would it make sense to provide a placeholder replacement + experiment_dir injection to the docu the same way it is done for the tests, thus ensuring stable output and no experiment_dir overwriting issues?

The problem here, like for the tests, is that even if we replace the placeholders while building the documentation, we would still need access to a remote computing resource to execute the notebook and generate the output. Since the documentation is built here on GitHub, this is currently not possible, as far as I know.

BishrMaradni · 2025-12-17T13:00:25Z

tutorials/5_grid_iterator_4c_remote/5_grid_iterator_4c_remote.ipynb

+    "# 3. Remote Computing with 4C\n",
+    "\n",
+    "In this tutorial, you will use the `Grid` iterator to probe the response surface of a 4C model.\n",
+    "The 4C simulations will run remotely on a cluster.  \n",


A small describtion of the 4C model in question would be appreciated 😄

To be honest, I have no clue what this 4C model does. I just copied it from the old tutorial 😁 Maybe somebody can help us out here and explain?

BishrMaradni · 2025-12-17T13:16:32Z

tutorials/3_grid_iterator_fourc_remote.ipynb

+    "5. Install the same QUEENS environment that you are using on your local machine also on the remote machine. You will later have to pass the path to its python executable via `remote_python`.\n",
+    "\n",
+    "> **Subclassing**:\n",
+    "> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",


Suggested change

"> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",

"> If you run QUEENS locally with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",

I am not sure if this doesn't make the sentence more confusing 🫣 Strictly speaking, if you run QUEENS locally, the classes don't need to be available remotely. Maybe this is better?

Suggested change

"> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",

"> If you want to run QUEENS remotely with custom classes that inherit from QUEENS objects, you need to ensure that these classes are available in the remote QUEENS environment.\n",

tutorials/3_grid_iterator_fourc_remote.ipynb

BishrMaradni · 2025-12-17T13:26:42Z

tutorials/3_grid_iterator_fourc_remote.ipynb

+    "\n",
+    "In the following, we will set up the QUEENS experiment for cluster usage:\n",
+    "\n",
+    "1. To evaluate 4C on a cluster, use a `Jobscript` driver instead of a `Fourc` driver and adjust the paths:\n",


Suggested change

"1. To evaluate 4C on a cluster, use a `Jobscript` driver instead of a `Fourc` driver and adjust the paths:\n",

"1. To evaluate 4C remotely on a cluster, use a `Jobscript` driver instead of a `Fourc` driver, and adjust the paths:\n",

I have added the remotely, but I believe the comma is grammatically incorrect here (it's not a list).

tutorials/3_grid_iterator_fourc_remote.ipynb

leahaeusel

Thank you for the feedback @BishrMaradni 🙏

leahaeusel · 2025-12-29T06:54:57Z

tests/tutorial_tests/test_3_grid_iterator_fourc_remote.py

+"""Placeholder for integration test of 3-grid-iterator-4C-remote tutorial.
+
+The actual test is in tests/integration_tests/cluster/test_dask_cluster.py.
+This file ensures that the tutorial is not run by tests/tutorial_tests/test_tutorials.py.
+"""


Yes, I am testing the new notebook in the cluster tests because this notebook can only execute successfully when a remote computing resource is available. This is not the case for the normal tutorial tests, which are part of the local tests on the GitHub runner. The cluster tests are not part of this pipeline but are run once per night on GitLab and have access to thought / charon.

leahaeusel · 2025-12-29T06:57:23Z

tutorials/5_grid_iterator_4c_remote/5_grid_iterator_4c_remote.ipynb

+    "# 3. Remote Computing with 4C\n",
+    "\n",
+    "In this tutorial, you will use the `Grid` iterator to probe the response surface of a 4C model.\n",
+    "The 4C simulations will run remotely on a cluster.  \n",


To be honest, I have no clue what this 4C model does. I just copied it from the old tutorial 😁 Maybe somebody can help us out here and explain?

leahaeusel · 2025-12-29T07:01:34Z

tutorials/5_grid_iterator_4c_remote/5_grid_iterator_4c_remote.ipynb

The problem here, like for the tests, is that even if we replace the placeholders while building the documentation, we would still need access to a remote computing resource to execute the notebook and generate the output. Since the documentation is built here on GitHub, this is currently not possible, as far as I know.

leahaeusel · 2025-12-29T07:13:02Z

tutorials/3_grid_iterator_fourc_remote.ipynb

+    "5. Install the same QUEENS environment that you are using on your local machine also on the remote machine. You will later have to pass the path to its python executable via `remote_python`.\n",
+    "\n",
+    "> **Subclassing**:\n",
+    "> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",


I am not sure if this doesn't make the sentence more confusing 🫣 Strictly speaking, if you run QUEENS locally, the classes don't need to be available remotely. Maybe this is better?

Suggested change

"> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",

"> If you want to run QUEENS remotely with custom classes that inherit from QUEENS objects, you need to ensure that these classes are available in the remote QUEENS environment.\n",

leahaeusel · 2025-12-29T07:19:17Z

tutorials/3_grid_iterator_fourc_remote.ipynb

+    "\n",
+    "In the following, we will set up the QUEENS experiment for cluster usage:\n",
+    "\n",
+    "1. To evaluate 4C on a cluster, use a `Jobscript` driver instead of a `Fourc` driver and adjust the paths:\n",


I have added the remotely, but I believe the comma is grammatically incorrect here (it's not a list).

…is dir for all other cluster tests

leahaeusel · 2026-02-25T15:40:41Z

I have rebased all commits onto the newest main and have changed the tutorial number from 3 to 5 in accordance with what @reginabuehler and I have agreed on. I have also checked that the cluster tests still run successfully on Gitlab.

leahaeusel requested a review from Copilot December 16, 2025 09:04

Copilot started reviewing on behalf of leahaeusel December 16, 2025 09:05 View session

leahaeusel commented Dec 16, 2025

View reviewed changes

Copilot AI reviewed Dec 16, 2025

View reviewed changes

tutorials/3_grid_iterator_fourc_remote.ipynb Outdated Show resolved Hide resolved

tutorials/3_grid_iterator_fourc_remote.ipynb Outdated Show resolved Hide resolved

queens-py deleted a comment from Copilot AI Dec 16, 2025

leahaeusel force-pushed the add-remote-computing-tutorial branch 5 times, most recently from 75c6f2e to 62277ba Compare December 17, 2025 10:16

leahaeusel marked this pull request as ready for review December 17, 2025 11:58

BishrMaradni reviewed Dec 17, 2025

View reviewed changes

leahaeusel commented Dec 29, 2025

View reviewed changes

leahaeusel force-pushed the add-remote-computing-tutorial branch from 27e0e99 to fb74a07 Compare December 29, 2025 09:54

leahaeusel added 8 commits February 24, 2026 14:57

docs: add remote computing tutorial

b428b1f

fix: typos, don't execute notebook with sphinx, move input file

d25b7bd

fix: change experiment base dir for tutorial test & remove mocking th…

dfb81c2

…is dir for all other cluster tests

fix: fix tutorial markdown such that it's rendered correctly by Sphinx

808cf50

test: avoid duplicate code & clean up cluster tests

a88d442

docs: include Bishr's feedback

d4406dd

docs: include student's feedback

668ff2b

docs: rename tutorial from 3 to 5

0d86770

leahaeusel force-pushed the add-remote-computing-tutorial branch from fb74a07 to 0d86770 Compare February 25, 2026 15:35

test: make new tutorial test comply with pylint

b2043a3

leahaeusel force-pushed the add-remote-computing-tutorial branch from cf70345 to b2043a3 Compare February 25, 2026 18:17

leahaeusel mentioned this pull request Feb 25, 2026

Add UKACM & GACM tutorial: Orchestrating 4C simulations with QUEENS #287

Open

	"> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",
	"> If you run QUEENS locally with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",

	"> If you run QUEENS with custom classes that inherit from QUEENS objects, you need to ensure that these classes are also available in the remote QUEENS environment.\n",
	"> If you want to run QUEENS remotely with custom classes that inherit from QUEENS objects, you need to ensure that these classes are available in the remote QUEENS environment.\n",

	"1. To evaluate 4C on a cluster, use a `Jobscript` driver instead of a `Fourc` driver and adjust the paths:\n",
	"1. To evaluate 4C remotely on a cluster, use a `Jobscript` driver instead of a `Fourc` driver, and adjust the paths:\n",

Conversation

leahaeusel commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description and Context: What and Why?

Related Issues and Pull Requests

Interested Parties

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

BishrMaradni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leahaeusel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leahaeusel commented Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leahaeusel commented Dec 16, 2025 •

edited

Loading

Description and Context:
What and Why?