I seem to recall when profiling that memory allocation was one of the
bottlenecks of the code. In Rust 1.50 Vec will support custom allocators, which
is something we can look into.
I understand that @hoodmane has experimented with some such ideas back then.
