Reduce allocations of conversion #23
Description
We currently perform a lot of allocations and do a lot of cloning during the conversion process, which is likely not necessary.
We should try to remove as many .clones as we can and only use references where possible.
Should be tackled after #22 so we verify that the changes yield an improvement.