Add rough version of an autodiff refactor

[aseyboldt]

Completely redo the implementation of the Rop/Lop/grad functions and replace them by pull_back, push_forward. grad can just call pull_back then. They are currently not really working, but only a sketch of how they could look like.

Add new alternative methods pull_back and push_forward and linear_transpose to Op. In the long-term they should replace the grad, Rop and Lop methods, and give more control and context to the Op when implementing those operations.

Invesigate a bit more how we can derive Rop from Lop and the other way round. We can go both ways: Given an implementation of push_forward we can use the fact that the mapping between the tangents is linear, and if all involved linear ops implement linear_transpose, we can use that to derive pull_back. Given pull_back we can use the fact that pull_back is a push_forward and transpose. And since the push_forward of a linear function is itself, we can apply it twice to effectively remove the transpose. (see pull_back_through_transpose and push_forward_through_pull_back).

We can possibly provide hints to the graph optimizer that the values can sometimes be computed in a smarter way if we also want gradients. Ricardo convinced me that we probably don't want to produce new output values in the push_forward and pull_back functions, but maybe if the gradients are computed lazily we could collect those possible
rewrites and apply them during the rewrite that computes the gradients?

Hopefully delete 2000 lines of complicated gradient code once the new stuff is working well.

cc @ricardoV94

[ricardoV94]

No, but we should

[ricardoV94]

@aseyboldt let's try to replace the L_op / R_op to see what needs to be added. I would also like a non-trivial example of "alternative outputs" to test ways we can achieve that.