web.load incompatibility with latest tidy 3.9

[heydaridlm]

Describe the bug
When upgrading to the latest tidy version (3.9), we find that web.load fails and gives the following error:
sim_data = web.load('<HASH>', verbose=False)

Error:

WARNING: updating Simulation from 2.9 to 2.8                       
WARNING: Could not execute validator                               
'_source_homogeneous_isotropic' because field 'structures' failed  
validation.                                                        
WARNING: Could not execute validator 'plane_wave_boundaries'       
because field 'structures' failed validation.                      
WARNING: Could not execute validator 'tfsf_boundaries' because     
field 'structures' failed validation.                              
WARNING: Could not execute validator '_structures_not_close_pml'   
because field 'structures' failed validation.                      
WARNING: Could not execute validator 'bloch_boundaries_diff_mnt'   
because field 'structures' failed validation.                      
WARNING: Could not execute validator                               
'_warn_monitor_mediums_frequency_range' because field 'structures' 
failed validation.                                                 
WARNING: Could not execute validator                               
'_projection_monitors_homogeneous' because field 'structures'      
failed validation.                                                 
WARNING: Could not execute validator                               
'diffraction_and_directivity_monitor_medium' because field         
'structures' failed validation.                                    
WARNING: Could not execute validator '_warn_grid_size_too_small'   
because field 'structures' failed validation.                      
WARNING: Could not execute validator                               
'_validate_num_lumped_elements' because field 'structures' failed  
validation.                                                        
WARNING: Could not execute validator 'check_fixed_angle_components'
because field 'structures' failed validation.                      
WARNING: Could not execute validator 'data_monitors_match_sim'     
because field 'simulation' failed validation.                      
WARNING: Could not execute validator 'validate_no_ambiguity'       
because field 'simulation' failed validation.             


ValidationError: 10 validation errors for SimulationData
simulation -> structures -> 0 -> priority
  extra fields not permitted (type=value_error.extra)
simulation -> structures -> 1 -> priority
  extra fields not permitted (type=value_error.extra)
simulation -> structures -> 2 -> priority
  extra fields not permitted (type=value_error.extra)
simulation -> structures -> 3 -> priority
  extra fields not permitted (type=value_error.extra)
simulation -> subpixel
  value could not be parsed to a boolean (type=type_error.bool)
simulation -> subpixel -> pec -> PECConformal -> edge_singularity_correction
  extra fields not permitted (type=value_error.extra)
simulation -> subpixel -> lossy_metal -> SurfaceImpedance -> edge_singularity_correction
  extra fields not permitted (type=value_error.extra)
simulation -> subpixel -> pmc
  extra fields not permitted (type=value_error.extra)
simulation -> precision
  extra fields not permitted (type=value_error.extra)
simulation -> structure_priority_mode
  extra fields not permitted (type=value_error.extra)

Expected behavior
We expect to successfully load the data post sim as it always did in the past. From my understanding, tidy is performing validations of the sim and somehow in the new version these validation checks have changed and are incompatible with our version (3.8) of web.load.

Suggested fix
It may require discussion with flexcompute.

Versions

[flaport]

@joamatab

[momchil-flex]

So a couple of things:

• Just to be clear 3.8/3.9 is a typo, you guys mean 2.8/2.9 (as also visible in the log and the screenshot)
• Tidy3D is actually on 2.10 already, but not urgency to upgrade

Regarding the issue at hand, the problem is that you seem to be loading a simulation that was run with 2.9 in an environment that runs 2.8 (we are backwards compatible, but not forwards compatible). Look at this:

WARNING: updating Simulation from 2.9 to 2.8    

and

If you run a new simulation with 2.9 you need to load it in 2.9 too. 2.9 can load all previous simulations though.