Google cast support

[Merovius]

I don't know if this is feasible, but we might consider streaming directly via Google cast. The advantage is that we save a round-trip in latency when screen-sharing - the device can directly stream into a hangout or to a cast-capable device, without bouncing via the browser.

However, this would probably require doing actual video-encoding, it's not clear the reMarkable can handle that in software.

[mlgarchery]

@Merovius Would you mind explaining to me where is the bottleneck in terms of the streaming performance ? If so, would you be able to quantify it for the moment ? For example if it is the transmission flow of the raw data from the remarkable to another machine that is limited or something else, so that I can have realistic expectation of what is achievable. Thanks in advance

[Merovius]

It's CPU, especially if you want the UI to continue to work smoothly. With enough CPU, bandwidth would actually be super easy, given how compressible the screen content is.

One uncompressed raw frame is ~5MB. When running over USB and just shoveling them out, I got ~25 FPS, which is fine. But we can't do that if we don't control the receiving end (e.g. the browser), because we need to send an actual graphic format. Even writing uncompressed PNG made the framerate tank - you have to interleave pixel-data with compression-headers, even if they're only saying "no compression". As a result, it's no longer a direct memcpy or write, but you actually have to spend CPU time ranging over the rows etc. Furthermore, with uncompressed frames, while USB works fine, the wifi apparently was too slow.
Currently we write compressed (with a low compression setting, which is fine for our black-and-white images) PNGs, which offered a decent tradeoff between bandwidth- and CPU-usage.

Unfortunately, I haven't run any actual A/B-tests and real experiments and I haven't profiled it. So I'm neither totally sure how much of the CPU time is avoidable (by allocating and/or pipelining more cleverly and/or optimizing the encoding) and I don't really have hard numbers. All I can say is that the things I did test out left me with the impression that doing too much algorithmic work on the device is probably not feasible. Especially as video-formats tend to be fixed-FPS AIUI, so we'd have to have pretty consistent encoding speeds for real-time.

Also, while it hasn't been a problem so far, if we do video-encoding, we have to do inter-frame-compression, I believe. On the one hand that's good, because it will make the stream very small (there are ~no changes between frames), on the other hand, we might have to keep several frames (10? 30? more? Dunno) in RAM and given that each frame is >5MB and the device only has 512 MB RAM (which has to be enough for the kernel, any daemons, xochitl and srvfb)… Maybe memory would become a bottleneck at some point. I'm just guessing there, though. If we're being clever, we might not have to do any of that and instead just write out keyframes and pretend that the image doesn't change until the next keyframe?

I honestly have no idea :) I never looked at video formats so I don't know how feasible any of this would be to implement and where the bottlenecks are :) Currently it's mainly CPU.

FWIW, if you ever tried uploading e.g. a PDF to the reMarkable via the web UI, you will have noticed that it takes forever. That's, AFAICT, for exactly the same kinds of reasons. The reMarkable pre-renders the PDF pages as PNGs for previews and it takes ages to do that. It's also, I guess, why the new handwriting-recognition happens in the cloud, instead of on-device - it's just not feasible to do otherwise.

[Evidlo]

Would it be possible to only update the parts of the screen that have changed? Something like this: https://www.linuxquestions.org/questions/linux-software-2/how-to-identify-the-frame-buffer-updated-region-574123/