reStream: Does not work on reMarkable 2

@darthoctopus - i just tried something similar to what you put together but created a mountpoint in ram first:

mount -t tmpfs -o size=32m tmpfs ram/

…i’m not sure whats up with the pipe, but:

Thanks to @ddvk and @raisjn we have xofb to expose the framebuffer on /dev/shm/xofb and a way to stream the framebuffer in color (check this patch)!

This is a screenshot taken while streaming my rm2 with the above patch and library.

Best way to start 2021!

@harrylepotter @thomsten If you increase the block size, you have to divide skip and count by that size.

Also, increasing the block size past 4096 will not increase performance as that’s the page size for i.MX7.

@tadfisher great you were able to dump the image!

If someone could write a script which is able to fetch the memory address and then dump that memory, I can fix it in the reStream script to work with the reMarkable 2. The following things should be done:

1. pid=$(pidof xochitl)
2. Interpret the 4 bytes in /dev/$pid/mem at 0x003ee1c0 (4121024 in decimal) as little-endian integer.
3. Use the address of the previous step to read 24893440 bytes from /dev/$pid/maps.

To check whether de data is correct, you can use @ddvk’s convert command. I haven’t tested it so I am not sure the pixel format is correct.

If I had a rM2 I would do it myself, but that’s unfortunately not the case.

@rien I have to be honest, I am A little bit proud that my work was not completely dropped. 😉 If reStream is normal working fine you should post it on Reddit. There are a bunch of people waiting for this. Just post a short Exclamation and a gif. I read tons of complains about the live view feature of remarkable.

If you have any problem in the future just drop me a line. I am down about any research about the rM2.

I think it would be really cool if we can use LD_PRELOAD (or the systemwide LD_PRELOAD) to modify xochitl and swap out the pointer location from where it’s pointing now to shared memory. this would then expose the framebuffer as /dev/shm/<name of shared mem> and reStream and others can just dump from there.

PS: unrelated (and sorry to interrupt) but we have a channel on discord (#rm2 on https://discord.gg/JSSGnFY) that we are going to try to use to coordinate rM2 framebuffer reversing, if you want to chat more there 😄

On my reMarkable 2 I get

reMarkable: ~/ grep 'REMARKABLE_RELEASE_VERSION' /usr/share/remarkable/update.conf
REMARKABLE_RELEASE_VERSION=2.3.1.27

and

reMarkable: ~/ grep 'deviceid' /etc/remarkable.conf
deviceid=RM110-042-07857

So it doesn’t start with RM2, if that was your idea. However, I get

reMarkable: ~/  cat /sys/devices/soc0/machine
reMarkable 2.0

I also did install rm2fb … https://github.com/ddvk/remarkable2-framebuffer maybe that has something to do with it?

Yes, this is not compatible with rm2fb yet. See #41

If you redirected your frame buffer to /dev/fb0 you should use the old rM1 way of streaming. Hard code the rM verison to 1.0 and the image sizes to 1872px times 1404px with the corresponding number of bytes per pixel you are outputting. This should do the trick.

Hello @1ykos, the reason why we are not reading from /dev/fb0 directly is that the rM2 does not provide the image in this way anymore. 😢 We have to locate the frame buffer from the QtApplication itself and the dump the memory from there.

I’ve just pulled the latest commit on this branch (cf6b1050a7f2c46593c3f61977684ef8378626e3 in case you force push again).

Can confirm that it works on the rm2 too.

I’ve only stumbled onto this project recently, and wow, great work!

It works perfectly now. I haven’t measured, but it feels great (with regards to fps and latency).

I know the rM1 could handle color because the framebuffer was a the ‘normal’ one built into Linux. The rM2 uses special hardware and drives the display from user-space, so I don’t think there is any color information present.

From what I remember from the cloud stream, it sends file updates (e.g. the lines you draw) instead of an actual “video stream” of the screen. If the file contains images in color, it will be rendered by your browser.

@1dotd4 If you take the way with entware and install the head command, you will get around 2-3 FPS (the same you are getting). The current idea is to replace the dd, head and tail combination entirely. I have a working prototype that gives around 12-15 FPS but there ist some one else who is doing the same think. For more information take a look at the pull request in this Repo.

What is your idea vor the colored output? I took a close look at the memory In /proc/…/mem and for what I found, there is only the grayscale image. How would you realize this without patching?

I give my 2 cents here.

thanks to all the information in this thread (thank you all!), I successfully grabbed a picture from my RM2 updated with firmware 2.5.

Here is the recap (please correct me if I something is wrong):

Remarkable version

reMarkable: ~/ cat /usr/share/remarkable/update.conf
[General]
#REMARKABLE_RELEASE_APPID={98DA7DF2-4E3E-4744-9DE6-EC931886ABAB}
#SERVER=https://get-updates.cloud.remarkable.engineering/service/update2
#GROUP=Prod
#PLATFORM=reMarkable2
REMARKABLE_RELEASE_VERSION=2.5.0.27

Locating the global framebuffer

reMarkable: ~/ strace xochitl
...
563 openat(AT_FDCWD, "/dev/fb0", O_RDWR)    = 5
564 ioctl(5, FBIOGET_FSCREENINFO, 0x7ee9d5f4) = 0
565 ioctl(5, FBIOGET_VSCREENINFO, 0x42f0ec) = 0
566 ioctl(5, FBIOPUT_VSCREENINFO, 0x42f0ec) = 0

Global framebuffer is located at 0x42f0ec-4 =0x42f0e8 (4387048 in decimal)

Now get a picture:

#!/bin/sh
pid=`pidof xochitl`
addr=`dd if=/proc/$pid/mem bs=1 count=4 skip=4387048  2>/dev/null | hexdump | awk '{print $3$2}'`
skipbytes=`printf "%d" $((16#$addr))`
dd if=/proc/$pid/mem bs=1 count=2628288 skip=$skipbytes > out.data

Note: 2628288 = 1404*1872 (the number of pixel in the screen of the remarkable)

Get the picture back on my laptop and convert it:

convert -depth 8 -size 1872x1404+0 gray:out.data out.png

which gives:

Edit: I’ve uploaded a gist in Go that takes a screenshot on the remarkable

I am not sure if feedback is wanted at this point of time and if it’s helpful.

You guys are simply awesome. Managed to get my Remarkable 2 to work with your streaming script (feature branch) by downloading “head”, scp-ing it over to the Remarkable2 and name it “shead” (there is no /usr/local/bin on my device). I then replaced “head -c” with “shead -c” in reStream.sh. Now runing reStreamsh gives me the image:

Image is still upside-down, but performance is great. Almost no delay while sharing the screen of the device.

Edit: I switched into landscape view, screen was still turned. I removed the added filter in reStream.sh so no “transpose=2” is added. The screen then shows up in the correct direction.

Looking at the line from @harrylepotter’s comment

71432000-72bf0000 rw-s a8100000 00:06 248 /dev/fb0

I also looked at /proc/$pid/maps for the 2.4.0.27 version of xochitl to find:

713a4000-72b62000 rw-s a8100000 00:06 275        /dev/fb0
72b62000-732e8000 rw-p 00000000 00:00 0 

When using dd if=/proc/242/mem bs=1 count=2628288 skip=1924538376 > out.data (0x72b62008=1924538376) analogous to the above comment this results in the right image. Would it not be simpler to use the information in /proc/$pid/maps which is actually meant to list the different memory sections of the program than to search for a signature in the hexdump of the memory itself (of course this only holds if it is no coincidence that the right adress is right after the adress of fb0 but it seems that we have two data points that suggest this).

On my remarkable I get:

$ grep 'REMARKABLE_RELEASE_VERSION' /usr/share/remarkable/update.conf
REMARKABLE_RELEASE_VERSION=2.4.0.27

now the interesting part: there is no deviceid in /etc/remarkable.conf but a large devicetoken.

$ cat /sys/devices/soc0/machine
reMarkable 2.0

here you go:

reMarkable: ~/ grep 'REMARKABLE_RELEASE_VERSION' /usr/share/remarkable/update.conf
REMARKABLE_RELEASE_VERSION=2.3.1.27
reMarkable: ~/ grep 'deviceid' /etc/remarkable.conf
deviceid=RM110-039-45166

After some trial and error, I’ve gotten it to work with the following definition of head_fb0: dd if=/proc/$pid/mem bs=3060 count=920 skip=628906 2>/dev/null > temp && dd if=temp bs=2628288 count=1 2>/dev/null

Unfortunately, the latency in this case is pretty high. I’m also not sure if this will continue to work across e.g. restarts of xochitl (e.g. if the location of the framebuffer gets updated). The address supplied by @tadfisher appears not to work for me (possibly because I am on version 2.4 of the software).

edit: as expected; it did not work after restarting xochitl. it also seems unwieldy to me to have to recompute some integer factorisation when confronted with the updated address every time the script is run, in any case.

@tadfisher also tried re-piping to dd with

dd if=/proc/237/mem bs=3647 count=721 skip=527864 2>/dev/null | dd bs=2628288 count=1 2>/dev/null

but no dice. Can’t figure out if i’m doing something wrong on the DD side, or on the ffmpeg side at this stage…

We could make a mapping between the xochitl sha1 and the memory address of the framebuffer