General optimization idea's and findings

[NickHu]

Given that ffmpeg is in entware, has anyone tried to use a real video codec to grab from /dev/fb0 instead of using lz4 on the raw bytes? I think this should in principle implement @rien's bsdiff idea (changes between frames are small, so this will reduce IO throttle) that I saw on the reddit post. I was able to get a stream to show by doing

ssh root@192.168.0.25 -- /opt/bin/ffmpeg -f fbdev -framerate 1 -i /dev/fb0 -c:v libx264 -preset ultrafast -pix_fmt yuv420p -f rawvideo - | ffplay -i -

but it seems heavily laggy. It does seem to encode at a framerate of just over 1 per second, so there's clearly a long way to go. It also definitely seems like ffplay is waiting for a buffer to accumulate before playing anything. I'm really curious as to whether a more ingenious choice of codecs/ffmpeg flags would be availing.

Here's some sample output of ffmpeg if I set the loglevel of ffplay to quiet:

ffmpeg version 3.4.7 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.3.0 (OpenWrt GCC 8.3.0 r1148-dbf3d603)
  configuration: --enable-cross-compile --cross-prefix=arm-openwrt-linux-gnueabi- --arch=arm --cpu=cortex-a9 --target-os=linux --prefix=/opt --pkg-config=pkg-config --enable-shared --enable-static --enable-pthreads --enable-zlib --disable-doc --disable-debug --disable-lzma --disable-vaapi --disable-vdpau --disable-outdevs --disable-altivec --disable-vsx --disable-power8 --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-inline-asm --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-mmi --disable-fast-unaligned --disable-runtime-cpudetect --enable-lto --disable-vfp --disable-neon --enable-avresample --enable-libopus --enable-small --enable-libshine --enable-gpl --enable-libx264
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[fbdev @ 0xb80400] w:1404 h:1872 bpp:16 pixfmt:rgb565le fps:1/1 bit_rate:42052608
[fbdev @ 0xb80400] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, fbdev, from '/dev/fb0':
  Duration: N/A, start: 1586221544.651449, bitrate: 42052 kb/s
    Stream #0:0: Video: rawvideo (RGB[16] / 0x10424752), rgb565le, 1404x1872, 42052 kb/s, 1 fps, 1000k tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0xb84590] using cpu capabilities: none!
[libx264 @ 0xb84590] profile Constrained Baseline, level 5.0, 4:2:0, 8-bit
Output #0, rawvideo, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264), yuv420p, 1404x1872, q=-1--1, 1 fps, 1 tbn, 1 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=   10 fps=1.0 q=2.0 size=     216kB time=00:00:09.00 bitrate= 196.8kbits/s speed=0.919x

I think one of the big slowdowns here is it's taking the input stream as [fbdev @ 0xb80400] w:1404 h:1872 bpp:16 pixfmt:rgb565le fps:1/1 bit_rate:42052608, rather than the 2 bytes-per-pixel gray16le stream that reStream is using - I can't seem to configure this though.

Also, is lz4 really faster than zstd?

[levincoolxyz]

I just (enjoyably) wasted an evening testing different options. My findings (all thru usb):

1. tried zstd from entware with different --fast=# parameter (ranging from 1 to 200), lz4 still seems faster (and it takes only a single core process ~30% cpu on rM comparing to 15-60% on two cores by zstd). [But maybe zstd can improve overall speed with bad internet connection since it can give a higher compression ratio?]
2. using ffmpeg to simply cut frame size in half (704x936) (I/O both -f rawvideo) before compression with lz4 still significantly slowed the pipeline down.
3. lossy image compression (per frame) without invoking ffmpeg also seems to do a worse job comparing to lz4 (both in compression ratio and cpu time). [played with gm convert from entware, for reference gm convert to .jpg compresses to about 5% size of raw frame, while lz4 gives a bit over 6%. (zstd with default option gives a bit below 4%). and gm convert is much much slower.]

So my conclusion is that unless a custom video encoding with streaming in mind is written in a single compiled program, there's probably not much more chance for improvement speed-wise.

---edit---
forgot to mention that without any knowledge in these areas I dared to naively use bsdiff to get a delta encoding... now that I know what bsdiff is used for I realized how dumb that idea was. probably would have fared better compiling a simple for loop in c for this usage.

[rien]

Hi guys! Great you've been experimenting with different encodings. It definitely seems possible to achieve near zero lag with the right tools.

In writing this script I did some of the same experimentation. But the reMarkable processor is just too weak handle something as heavy as video encoding.

The clue in making this as fast as possible is getting the framebuffer data out of the reMarkable as quickly as possible. I've tried using no compression, but then the kernel (read: TCP or USB IO) seems to be the bottleneck.

Writing something in C (or Rust 🎉) will probably the long-term solution. But in the meanwhile I think experimenting with bsdiff could give some nice results.

@levincoolxyz what did you exactly try with bsdiff? What I had in mind, was keeping a temporary framebuffer to store the previous frame and use bsdiff to compute the difference between that frame and the current one, send the diff, and then reconstruct the image it at the receiving end in the same manner.

[levincoolxyz]

that is what I attempted to code with bsdiff. but I quit before fully debugging as I saw the cpu usage shot up like crazy on reMarkable. from the man page etc., i think it is written to generate a small patch of very large files (e.g. software updates) without real time applications in mind.

For reference, this took 1.6s on my laptop (doing virtually nothing): time ( bsdiff fb_old fb_old fb_patch )

[NickHu]

I have a couple of observations to make. I think lots of the potential of h264 lie in the colourspace, and also having specialised instructions to get hardware acceleration. Seeing as the video stream coming out of the reMarkable should be entirely gray-scale, this suggests that it's not the right codec to use. I don't know if any codecs which are better suited to gray-scale exist, but to me it's really surprising that a general purpose compressor (lz4) is so much faster than a specialised video codec. It might be worth trying to write a naïve video codec, but I don't really know anything about graphics.

Secondly, according to https://github.com/thoughtpolice/minibsdiff, the bsdiff binary is incorporating gzip. My guess is that's where it's spending most of its time, and replacing that with, say, lz4 ought to give you something faster than what we have right now. I still feel like the 'morally correct' solution is some sort of video compressor though.

[rien]

What a normal video encoder would do is throwing away information (e.g. colours, image complexity, ...) in order to create a smaller video.

The reason why a general purpose compressor works is because there is a lot of repeated information (a bunch of white FF bytes in the background and some darker bytes which will be more to the darker end).

There are probably codecs which support gray-scale images, but I doubt they will be effective because of the performance constraints we have. Our 'codec' should be as simple as possible.

Maybe bsdiff has an option to not compress its result? If want to do more research what is slowing bsdiff down, you could try to use a profiler like perf.

As an alternative to bsdiff we could xor two images byte-by-byte. The xor-ed image will contain only changes and thus be even more compressible then it is now.

[NickHu]

If you link against https://github.com/thoughtpolice/minibsdiff, you can change the compressor that bsdiff uses.

XOR is an interesting idea too - I would guess it also has less complexity, seeing as the byte buffer is the same size all the time - bsdiff probably has some overhead as it has to account for file size increases/decreases
(accidentally hit close, whoops)

[rien]

I currently don't have the time to dive deeper into this, so feel free to experiment with it! I'm open to PR's.

[levincoolxyz]

I tested the xor and then compress with lz4 idea. To my surprise, even with my rusty c programming skills I actually did not slow the pipeline down, which should mean that someone better at coding these low level stuff should bring down streaming latency (especially by choosing a better buffer size and merging xor operation into lz4 binary).
Moreover if the xor buffer is kept for a longer period of time instead of being replaced at every read, I think the performance will increase b/c of using less fread/fwrite (this part maybe possible with some shell script also?). But then we are really approaching the point of writing an efficient embedded device video encoder from scratch ...

(I just put some codes I used for testing in my forked project (https://github.com/levincoolxyz/reStream) for reference. If I get more time I might fiddle with it more...)

[rien]

That looks great. It will indeed be faster to make your xorstream really streaming by constantly reading stdin and writing the XOR-ed output to stdout.

[levincoolxyz]

It is already doing that, currently the file read and write are for (updating) the reference file. Maybe I should rewrite it to keep the reference in memory since the supplied stdin is already continuously giving out the new buffer... will try that next.

[jwittke]

Maybe the the JBIG1 data compression standard is something to look at...

https://www.cl.cam.ac.uk/~mgk25/jbigkit/

[NickHu]

@levincoolxyz I coded up a similar experiment, using a double buffer for input, xor'ing it together, and then compressing the result with lz4 - unfortunately, it doesn't seem any faster. I think there might be too much memory latency in doing two memcpys, but I didn't profile it.
https://gist.github.com/NickHu/8eb7ead78a5489d6a95ad5c7473994f5

I also tried to code up a minimal example using lz4's streaming compression, in the hopes that this would be faster (as it uses a dictionary-based approach), but again it was slightly slower.
https://gist.github.com/NickHu/95e8e5e1b8b326d2cb46ce461d3ec701

I'm not sure how the bash script outperformed me on this one, but I guess the naive C implementation I did is no good!

[levincoolxyz]

I just fixed a decoding bug and moved my fread out of the while loop. Now it (with xorstream) does perform better than the master branch over my wifi connection, and is on par through usb. I also found that since the data transmission would be slightly laggy anyway, it sometimes improve performance (esp. via wifi) by adding something like `sleep 0.03' into the read loop. This reduces load on reMarkable as well obviously.