[Bug] ExportSWC: exceptions are wrong

[sanjayankur31]

Using the ExportSWC class to export NeuroML2 to SWC works, but it has some checks that aren't necessarily correct for all cell models.

For example, on valid cells like the HL23PYR cell, it throws this exception:

$ pynml -swc HL23PYR.cell.nml 
pyNeuroML >>> INFO - Trying to convert HL23PYR.cell.nml to swc format...
pyNeuroML >>> INFO - Loading NeuroML2 file: HL23PYR.cell.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/CaDynamics_E2_NML2.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/pas.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Ih.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/SK.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/K_T.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/K_P.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Kv3_1.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Im.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/NaTg/NaTg.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Ca_HVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Ca_LVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Nap.channel.nml
Traceback (most recent call last):
  File "/home/asinha/.local/share/virtualenvs/neuroml-311-dev/bin/pynml", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 971, in main
    _evaluate_arguments(args)
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 866, in _evaluate_arguments
    convert_count += 1 if convert_to_swc(f) else 0
                          ^^^^^^^^^^^^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/swc/ExportSWC.py", line 174, in convert_to_swc
    raise Exception(
Exception: The numbers of the segments in groups: soma_group+dendrite_group+axon_group (5818), is not the same as total number of segments (5840)! All bets are off!

I need to check why these numbers don't add up---where are the 22 segments? But I also see it with other cells, like this one from the Allen cells: https://github.com/OpenSourceBrain/AllenInstituteNeuroML/blob/master/CellTypesDatabase/models/NeuroML2/Cell_497232312.cell.nml

$ pynml -swc Cell_497232312.cell.nml 
pyNeuroML >>> INFO - Trying to convert Cell_497232312.cell.nml to swc format...
pyNeuroML >>> INFO - Loading NeuroML2 file: Cell_497232312.cell.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/pas.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/NaV.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/K_T.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Kd.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Kv2like.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Kv3_1.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/SK.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Ih.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Im_v2.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Ca_HVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Ca_LVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/CaDynamics_all.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/CaDynamics.nml
Traceback (most recent call last):
  File "/home/asinha/.local/share/virtualenvs/neuroml-311-dev/bin/pynml", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 971, in main
    _evaluate_arguments(args)
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 866, in _evaluate_arguments
    convert_count += 1 if convert_to_swc(f) else 0
                          ^^^^^^^^^^^^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/swc/ExportSWC.py", line 174, in convert_to_swc
    raise Exception(
Exception: The numbers of the segments in groups: soma_group+dendrite_group+axon_group (1161), is not the same as total number of segments (3489)! All bets are off!

[sanjayankur31]

In the allen inst cell, the apical dendrites aren't included in the "dendrite_group":

a = cell.get_ordered_segments_in_groups(["all", "soma_group", "dendrite_group", "axon_group", "dend", "apic"])

Checking:

In [39]: len(a["apic"]) + len(a["dendrite_group"]) + len(a["soma_group"]) + len(a["axon_group"])
Out[39]: 3489

In [40]: len(a["all"])
Out[40]: 3489

In [41]: len(a["dendrite_group"]) + len(a["soma_group"]) + len(a["axon_group"])
Out[41]: 1161

@pgleeson : I guess the apical dendrites segment group apic should also be included in the dendrite_group here?

In the HL23PYR cell, there are 22 segments in the myelin_group that aren't in the conventional groups:

a = cell.get_ordered_segments_in_groups(["all", "soma_group", "dendrite_group", "axon_group", "basal_dendrite_group", "apical_dendrite_group", "myelin_group"])

here's verification:

(a["myelin_group"])
Out[12]: 
[<Segment|5819|Seg0_myelin_0>,
 <Segment|5820|Seg1_myelin_0>,
 <Segment|5821|Seg2_myelin_0>,
 <Segment|5822|Seg3_myelin_0>,
 <Segment|5823|Seg4_myelin_0>,
 <Segment|5824|Seg5_myelin_0>,
 <Segment|5825|Seg6_myelin_0>,
 <Segment|5826|Seg7_myelin_0>,
 <Segment|5827|Seg8_myelin_0>,
 <Segment|5828|Seg9_myelin_0>,
 <Segment|5829|Seg10_myelin_0>,
 <Segment|5830|Seg11_myelin_0>,
 <Segment|5831|Seg12_myelin_0>,
 <Segment|5832|Seg13_myelin_0>,
 <Segment|5833|Seg14_myelin_0>,
 <Segment|5834|Seg15_myelin_0>,
 <Segment|5835|Seg16_myelin_0>,
 <Segment|5836|Seg17_myelin_0>,
 <Segment|5837|Seg18_myelin_0>,
 <Segment|5838|Seg19_myelin_0>,
 <Segment|5839|Seg20_myelin_0>,
 <Segment|5840|Seg21_myelin_0>]

In [7]: len(a["dendrite_group"]) + len(a["soma_group"]) + len(a["axon_group"])
Out[7]: 5818

In [8]: len(a["myelin_group"])
Out[8]: 22

In [9]: len(a["all"])
Out[9]: 5840

I don't think the myelin segments are meant to be in any of the standard groups, so this seems correct?

[sanjayankur31]

This is not actually an exception bug. The cells we tested here are valid but not completely. We just don't have a test to check that the sum of dendrite_group + axon_group + soma_group = all. For correctness, this should hold, so we'll add another validation test for this: if one is using the default segment groups, this check should hold.