New config #1342

clessig · 2025-11-24T11:44:39Z

Description

Updated config for new model developments, as well as overall improvement of structure.

Issue Number

Is this PR a draft? Mark it as draft.

Checklist before asking for review

I have performed a self-review of my code
My changes comply with basic sanity checks:
- I have fixed formatting issues with ./scripts/actions.sh lint
- I have run unit tests with ./scripts/actions.sh unit-test
- I have documented my code and I have updated the docstrings.
- I have added unit tests, if relevant
I have tried my changes with data and code:
- I have run the integration tests with ./scripts/actions.sh integration-test
- (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
- (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
I have informed and aligned with people impacted by my change:
- for config changes: the MatterMost channels and/or a design doc
- for changes of dependencies: the MatterMost software development channel

… training still

kacpnowak · 2025-11-25T14:46:11Z

config/config_default.yml

+    # start_date: 197901010000
+    start_date: 201401010000
+    end_date: 202012310000
+    start_date_val: 202101010000
+    end_date_val: 202201010000


Since we support the ISO datetime format we can use it the default config

tjhunter · 2025-11-25T08:33:19Z

config/config_default.yml

@@ -0,0 +1,386 @@
+# streams_directory: "./config/streams/era5_1deg/"
+streams_directory: "./config/streams/era5_nppatms_synop/"


noting it could also be stream:...

tjhunter · 2025-11-25T08:34:11Z

config/config_default.yml

+### Model parameters ###
+
+model :
+  embedding :


consistency: should it be assimilation_engine or embedding? the code mostly talks about the assimilation engine.

We will have an Embedding module very soon.

tjhunter · 2025-11-25T09:08:14Z

config/config_default.yml

+        mlp_hidden_factor: 2
+
+  forecast_engine:
+    pass


you could just put forecast_engine: {} or even forecast_engine:

tjhunter · 2025-11-25T09:09:37Z

config/config_default.yml

+    # blocks: 6
+    # dropout_rate : 0.1
+
+  decoder :


should it be split across streams?

decoder: ERA5: type: ...

This is in the stream config (where it belongs). At the moment it's a combination between some global params specified here and local params in the stream configs

tjhunter · 2025-11-25T09:11:50Z

config/config_default.yml

+  # a regex that needs to fully match the name of the modules you want to freeze
+  # e.g. ".*ERA5" will match any module whose name ends in ERA5\
+  # encoders and decoders that exist per stream have the stream name attached at the end
+  freeze_modules: ""


eventually, it makes more sense to me to move that into the description of the model:

model: frozen: True forecast_engine: frozen: False

The current regex will not handle code refactoring with new names or packages. Longer term question

I think it's for the moment still better to specify a list of modules to be frozen

tjhunter · 2025-11-25T09:13:30Z

config/config_default.yml

+  freeze_modules: ""
+
+
+forecast :


why is it separate and not part of the model?

Will be in the model

tjhunter · 2025-11-25T09:25:28Z

config/config_default.yml

+
+### Learning rate params ###
+
+learning_rate :


why not under training or learning? we have other keys such as the descent algo etc.

tjhunter · 2025-11-25T13:19:50Z

config/config_default.yml

+### Shared model+training parameters ###
+# TODO: rename
+
+shared_params :


this name is vague and the content is not coherent. to me, most of these params would go to the model.

tjhunter · 2025-11-25T13:23:34Z

config/config_default.yml

+  mode : "student-teacher"
+  # 
+  source : 
+    - masking_params : 


this indentation is not the format you mean. it should be:

source: - masking_params: strategy: healpix num_samples: 4 rate: 0.4 hl_mask: 4 same_strategy_per_batch: false teacher_relationship: subset - masking_params: strategy: random num_samples: 4 rate: 0.4 hl_mask: 4 same_strategy_per_batch: false teacher_relationship: subset

or equivalently in json:

"source": [ { "masking_params": null, "strategy": "healpix", "num_samples": 4, "rate": 0.4, "hl_mask": 4, "same_strategy_per_batch": false, "teacher_relationship": "subset" }, { "masking_params": null, "strategy": "random", "num_samples": 4, "rate": 0.4, "hl_mask": 4, "same_strategy_per_batch": false, "teacher_relationship": "subset" } ]

tjhunter · 2025-11-25T15:11:07Z

config/config_default.yml

+  checkpoint: 250
+  log_validation: 0
+
+


missing the latest tags section

clessig · 2025-11-25T17:07:44Z

config/config_default.yml

+
+### Latent noising parameters ###
+
+latent_noise :


This needs to go to training_strategy

clessig · 2026-01-05T13:00:48Z

Will be merged with #1541

Initial sketch for new config

583803e

github-project-automation bot added this to WeatherGen-dev Nov 24, 2025

move around config, new section for eg lr, data. some invalid yaml in…

5cd960b

… training still

kacpnowak reviewed Nov 25, 2025

View reviewed changes

tjhunter reviewed Nov 25, 2025

View reviewed changes

shmh40 added 5 commits November 25, 2025 17:23

valid congfig

e7d4a0f

restored old model params

78c9091

updated older parts

0f50087

removed extra student-teacher specs

21a4da0

loader_num_wrkers 8

bba8d9c

clessig commented Nov 25, 2025

View reviewed changes

config/config_default.yml

### Latent noising parameters ###

latent_noise :

Copy link

Collaborator Author

clessig Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to go to training_strategy

tjhunter marked this pull request as draft December 15, 2025 08:46

clessig closed this Jan 5, 2026

github-project-automation bot moved this to Done in WeatherGen-dev Jan 5, 2026

		@@ -0,0 +1,386 @@
		# streams_directory: "./config/streams/era5_1deg/"
		streams_directory: "./config/streams/era5_nppatms_synop/"

		freeze_modules: ""


		forecast :

New config #1342

New config #1342

Uh oh!

Conversation

clessig commented Nov 24, 2025

Description

Issue Number

Checklist before asking for review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clessig commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants