Problem with the create-dataset.py

[zion0210]

Hello,

I tried to set up CloudCast locally for a custom region and followed the preprocessing steps. I successfully completed Step 1 ("Creating our own training dataset from NWCSAF data"). There are my output files in the nwcsaf directory and the files have normal name format (e.g. 20230905T090000.grib2).

However, I'm encountering an problem with the next step, running tools/create-dataset.py. The script does not find my input files.

Input directory nwcsaf exists and contains GRIB2 files (e.g. 20230905T090000.grib2).

I run the following command:

python tools/create-dataset.py --start_date="2023-09-01 00:00:00" --stop_date="2023-10-01 00:00:00" --preprocess=test --producer=nwcsaf --param=CloudProbability ODIR

Result:

Input directory:  nwcsaf
Filter matched 0 files

What is the expected filename pattern for the --producer=nwcsaf option? My files are named YYYYMMDDThhmmss.grib2. Should they follow a different convention?

Could you clarify the purpose and other required values for these options?

--preprocess
--packing_type
--dtype

The producer argument can be set to mnwc, meps, DEM, LSM, clim, cloudcast. Are they intended for fetching additional ancillary data (like orography or land-sea masks) required for the main preprocessing?
If yes, where can I get these files?

Any guidance on resolving the file matching issue or clarifying the parameters would be greatly appreciated.

Best regards,
zion0210

[mpartio]

Hi, and nice to hear you are trying out Cloudcast!

The expected filename pattern is: f"{INPUT_DIR}/{producer}/**/*{param}*.grib2" (link), ie. in your case you should rename the files to YYYYMMDDThhmmss_CloudProbability.grib2.

The three options you mention control the output data shape and how its stored as file:

• --preprocess: apply any preprocessing to the data. For example in our case we downsample the data from original resolution to 512x512 pixels, so we set --preprocess img_size=512x512. There are some other preprocessing methods also in the code but they are not typically used.
• --packing_type: defines if the data is written as npz (default) or npy; only in case data needs to be run memory-mapped the latter should be used.
• --dtype: datatype of the numpy array, the default f32 is used in our case but for some other data reduced datatypes might save space

The other producers you mention (mnwc, meps, dem, lsm, clim) are auxiliary datasets for verification (meps, mnwc, clim) OR static environmental data used for training (LSM=land sea mask, DEM=digital elevation map). None of them are hard requirements for Cloudcast. We can provide LSM and/or DEM for Scandinavia, but the data is also available in the internet as open source datasets. You can also use z and lsm from a suitable high-resolution NWP simulation if you one those at hand.

Hope this helps,

Mikko

[zion0210]

Hello, Mikko!

Thank you for your previous help — it resolved my preprocessing issue. I have successfully created a new dataset and completed model training.

Now I have a question regarding the loss_function parameter. I noticed a difference between the documentation and the research paper, and would appreciate some clarification.

The CloudCast paper states that MAE (Mean Absolute Error) yielded the best results in experiments. However, the GitHub documentation recommends using bcl1.

When I was studying the code, I found the following available options for --loss_function:

• ssim
• msssim
• bcl1
• fss
• ks
• coss

Could you briefly explain the purpose and use case for each of the available loss functions listed above?

Understanding these options would help me (and likely others) make more informed choices when training models for different use cases.

Thank you in advance for your guidance.

Best regards,
zion0210

[mpartio]

MAE was found to perform best as described in the paper. Afterwards we experimented with bcl1 which is just binary crossentropy combined with MAE (L1). It has very similar verification scores than just MAE but subjectively we decided that bcl1 fields look slightly better. It doesn't matter much which loss function you choose.

The other loss functions were used in development and prototyping and were found to be inferior in this specific use case.