Use arrow + shared memory to pass data between Go and Python

[tebeka]

We'd like to use Arrow + shared memory to pass data between Go & Python. We assume client and server will be on the same machine.

[tebeka]

Talking to the arrow team, they confirm Go is lacking arrow IPC support.

[tebeka]

I've started a small project to test this approach, it is here

[tebeka]

After trying for 1 more day and going over the Go arrow bindings, I've reached the conclusion that the current state of Go arrow does not provide us what we want and will require a lot of work to get there.

IMO the faster (devleopment wise) approach will be to use the C/C++ arrow from Go. It's much more mature and since Python uses the same code we can easilty share memory, probably using the plasma store

[tebeka]

The other option is to use IPC which uses flatbuffers implementation. We can encode/decode to the arrow flatbuffers representation.

[tebeka]

Started minimal implemtation of using C++ arrow from Go here