Remove empty experiment dirs

[leahaeusel]

Description and Context:
What and Why?

This PR introduces the deletion of empty experiment directories when exiting the global settings context.

Related Issues and Pull Requests

• Closes
• Related to Introduce logger on dask workers #170
• Follows feat: add flag & user prompt to overwrite existing experiment directory #230

Interested Parties

@sbrandstaeter

Note: More information on the merge request procedure in QUEENS can be found in the Submit a pull request section in the CONTRIBUTING.md file.

[Copilot]

Pull Request Overview

This PR refactors the experiment directory management to prevent accidental overwrites and automatically clean up empty directories. The experiment_directory() function now returns a tuple (Path, bool) indicating both the directory path and whether it already exists. Schedulers gain the ability to prompt users for confirmation before overwriting existing experiments and automatically delete empty experiment directories when the GlobalSettings context exits.

• Modified experiment_directory() to return whether the directory exists without creating it
• Added user confirmation prompts to prevent accidental overwrites of existing experiments
• Implemented automatic cleanup of empty experiment directories via a scheduler registry

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/queens/utils/config_directories.py	Changed experiment_directory() return type to tuple including existence flag
src/queens/schedulers/_scheduler.py	Added user confirmation prompt, cleanup registry, and local experiment directory management
src/queens/schedulers/local.py	Added overwrite_existing_experiment parameter and uses new experiment directory logic
src/queens/schedulers/pool.py	Added overwrite_existing_experiment parameter and uses new experiment directory logic
src/queens/schedulers/cluster.py	Added remote/local experiment directory methods and overwrite handling
src/queens/schedulers/_dask.py	Refactored to use cleanup registry instead of shutdown client list
src/queens/global_settings.py	Changed to call scheduler cleanup functions instead of client shutdown
src/queens/drivers/jobscript.py	Updated to use create_folder_if_not_existent() helper
tests/unit_tests/tutorials/test_tutorials.py	Mocks base_directory to prevent writing to user's home during tests
tests/unit_tests/schedulers/test_experiment_dir.py	New test file for experiment directory creation and overwrite behavior
tests/unit_tests/drivers/test_jobscript.py	Replaced local experiment_name fixture with shared test_name fixture
tests/integration_tests/fourc/test_fourc_mc.py	Updated to unpack tuple from experiment_directory()
tests/integration_tests/fourc/test_fourc_mc_random_material_field.py	Updated to unpack tuple from experiment_directory()
tests/integration_tests/cluster/test_dask_cluster.py	Updated mock and added comprehensive experiment directory tests

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[maxdinkel]

Would it be possible to delay the creation of the experiment directory by removing self.start_cluster_and_connect_client() from the init of the Dask scheduler and adding the experiment_directory() function to the self.start_cluster_and_connect_client() method? Then the experiment directory is only created once an evaluation is called, I believe.

[leahaeusel]

Would it be possible to delay the creation of the experiment directory by removing self.start_cluster_and_connect_client() from the init of the Dask scheduler and adding the experiment_directory() function to the self.start_cluster_and_connect_client() method? Then the experiment directory is only created once an evaluation is called, I believe.

You are right that we could delay the creation of the experiment directory until evaluate() is called. However, this does not get rid of empty experiment directories in cases where the function passed to evaluate() does not actually write anything to the experiment directory, as is currently the case with the function driver. Therefore, I haven't been able to come up with a prettier solution than checking for an empty experiment directory at the end of the run.

[maxdinkel]

You are right that we could delay the creation of the experiment directory until evaluate() is called. However, this does not get rid of empty experiment directories in cases where the function passed to evaluate() does not actually write anything to the experiment directory, as is currently the case with the function driver. Therefore, I haven't been able to come up with a prettier solution than checking for an empty experiment directory at the end of the run.

Aren't the log files still written by the scheduler in this case?

[leahaeusel]

Aren't the log files still written by the scheduler in this case?

Only for the Cluster scheduler. If you use either the Local or the Pool scheduler, no log files or any other outputs are written to the experiment directory by the scheduler itself.

[maxdinkel]

The changes look good to me. Thank you for your efforts :)