allow newlines as whitespace in formulas

[phrrngtn]

It would be nice to allow newlines in formulas: we try to use multi-line formulas for particularly lengthy formulas such as when we are creating a dictionary from many key value pairs.

My home Windows laptop has developed a hardware fault and I need to get it repaired/replaced before I can contribute a fix myself. I really should purchase a Windows development VM in the cloud.

ValueError: Unexpected token token=Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>)

Not sure why the last

In [34]: list(filter(lambda t: t.value == '\n', tokenize(formula)))
Out[34]:
[Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Unknown: 0>, sub_type=<SubType._None: 0>),
 Token(value='\n', type=<Type.Operand: 1>, sub_type=<SubType.Range: 7>)]

[tonyroberts]

Agreed, would be nice to have a token type for newlines (or maybe a subtype of whitespace). In most cases, simply using formula.replace("\n", "") is probably a suitable solution (but would be good to have those tokens if you want to convert back to a string preserving newlines and indentation).