Adds a Dylan lexer

[jneen]

Extends and supersedes #1706.
Fixes #25.

Changes made:

• Explicitly support Dylan Interchange Format (the frontmatter at the beginning of Dylan modules)
• Use keyword sets instead of joined regexes
• Fix string parsing and operator parsing (=> parsed as two separate tokens previously)
• Inline most regexes

[jneen]

May be of interest to @pedro-w - please let me know if I've missed any syntax!

[pedro-w]

Great, thank you @jneen . I can give it quick try on some other dylan files if that would help?

[jneen]

For sure! If you have more detailed stress-test files I'd love to see them as well.

[jneen]

I'm going to merge this for now, @pedro-w please let me know if you have more dylan code to test against.

[pedro-w]

Sorry, it took me a while to remember how to work ruby and then build rouge from github !!

It looks great apart from handling backslash - it's used to escape names that might be reserved words or operators.

From the DRM

name:
    word
    \ word
    operator-name

unreserved-name:
    any word that is not also a reserved-word
    \ word
    operator-name

operator-name:
    \ unary-operator
    \ binary-operator

Is that trivial for you to add? - please say if not and I will roll up my sleeves and send a PR, might take a while...

Thanks!

[jneen]

Can you provide some sample code that uses this?

[pedro-w]

Sure,

https://github.com/dylan-lang/opendylan/blob/85fc782bf2b1390929bdb9fc98f88e5e55d7a742/sources/collections/collectors.dylan#L160-L180

define inline method collector-protocol
    (class :: subclass(<number>), #key from = 0, by = \+)
  => (new-collector      :: <box>,
      add-first          :: <function>,
      add-last           :: <function>,
      add-sequence-first :: <function>,
      add-sequence-last  :: <function>,
      collection         :: <function>)
  values(box(from),
         method (collector, value)
           collector.object := by(value, collector.object);
         end,
         method (collector, value)
           collector.object := by(collector.object, value);
         end,
         sequence-collection-not-yet-implemented,
         sequence-collection-not-yet-implemented,
         method (collector)
           collector.object
         end)
end method;

In the part (class :: subclass(<number>), #key from = 0, by = \+) , \+ is an "operator name" in the DRM grammar. Rougify show it as a a white char on a red background and rougify debug says it's an error, not matched (I haven't got the exact text in front of me sorry)