ISO: Live mode boot slow, needs 4GB+ RAM

let me know whether it boots on your machine. My results are as follows:

• Attempted in a VirtualBox VM with 2 GB RAM - boots
• Attempted in a VirtualBox VM with 3 GB RAM - boots
• Attempted in a QEMU VM with 2 GB RAM - boots

Screenshot with htop output on a 2GB VBox VM (which also shows a bug where the dock does not listen to resolution changes - has that been reported yet?):

I’d say it’s a resounding success.

Thanks to the help of @pkgdemon we have an experimental Live build that should address this issue:

https://github.com/helloSystem/ISO/releases/tag/continuous-12.2-unionfs

Note that this build contains fluxbox rather than helloSystem to make the ISO smaller (and because the 12.2-unionfs branch has diverged a lot from the current experimental branch of helloSystem), but it should be possible to port the changes over to the helloSystem builds in case everything turns out to work well.

So let’s test, especially

• Does it run on VMs with <4 GB RAM (akik on IRC confirmed that it boots with -m 512 in qemu)
• How stable is it
• autoloading of ums, if_iwm, if_iwn (see below)

Username: liveuser, no password

@pkgdemon noted on IRC:

There may be other catches for unionfs such as devmatch not supporting kldload properly that I ran into. One thing that should be testing is to make sure autoloading of ums, if_iwm, if_iwn work with devmatch. In the past with ghostbsd it could also be observed that mate would lock up. I never ran into the lock up issues myself with FuryBSD. I think it just to be evaluated how well it works with Hello. (…) With unionfs you cannot reoot into the memory file system. You chroot. So what might end up happening is that certain low level utilities like kldload if_iwn might not be able to load a firmware like iwm8000Cfw because trying to use the real upper layer kldload /sysroot/boot/kernel/iwm8000Cfw.ko but because kldload is chrooted it needs to do kldload /boot/kernel/iwm8000Cfw.ko is what i ran into. (…) Hopefully those low level issues have been resolved since I’ve last tried and maybe it would be a good solution for hellosystem.

Pinned the issue @Tunas1337

This seems like an issue that, if possible, really ought to be referenced in a little “echo” message in the bootloader, before the system attempts to load further. It’s a source of confusion for a lot of people, and I’m sure there are people who have tested helloSystem, seen this, thought it was fundamentally broken and given up on it (myself included, had I not checked the GitHub issues!) EDIT: alternatively, this issue could be pinned for easier discovery, as opposed to searching closed issues to find a reference to this. It’d also stop people from making issues related to it, thus saving time.

We are mostly interested in supporting real (“bare metal”) hardware though.

I’m aware; sadly I have no machines with <4GB RAM right now. PS: Somehow, in my previous comment I neglected to mention the most important thing, that the implementation of geom_rowr reduced boot times to 30-45 seconds from several minutes as seen before. That was the thing I wanted to report in the first place, and that was what I was testing with the VMs. D’oh!

I am excited to share the news that a new FreeBSD kernel module is in the works that will allow us to do what we need, combine a read-only with a read-write disk to get a read-write disk: geom_rowr by Jordan Gordeev. I am preparing an ISO for testing that will use it for booting in the geom_rowr branch.

Edit: We now have a working ISO for the geom_rowr tree. Known issue: Cannot be installed yet because the installer has not yet been reworked.

This is the diff: https://github.com/helloSystem/ISO/compare/helloSystem:6139ff2...3aa68d9?expand=1

Maybe we have been thinking way too complicated all along.

So, another approach: Setting Up FreeBSD for Read-Only Booting makes it seem like it is possible to boot FreeBSD from read-only media. Looks rather simple!

You want to disable any programs that have to write to the filesystem and provide a memory-backed filesystem for the one area that does need to be written (/var/run). You could also make all of /var an md(4) filesystem but this is not necessary unless you want to run syslogd(8), sendmail(8), etc.

The article suggests that FreeBSD can be booted if /var/run the the only place that is writable.

I suspect that this is not entirely correct, at least not for a modern desktop system with Xorg:

• /var/log also regularly gets written to
• Xorg seems to need to write to $HOME, e.g., to ~/.Xauthority

We could probably work around these by symlinking these directories into /var/run/.

Then there is initgfx, which wants to write to, e.g., /usr/local/etc/X11/xorg.conf.d/. So the question is, can we trick Xorg into also using e.g., /var/run/X11/xorg.conf.d/?

• We could symlink /usr/local/etc/X11/xorg.conf.d/ to /var/run/X11/xorg.conf.d/
• Put the non-initgfx-related files into /usr/local/share/X11/xorg.conf.d/ instead of /usr/local/etc/X11/xorg.conf.d/

New plan of attack: Install helloSystem to hard disk, then find a way to make it work with root_rw_mount="NO" while still having Xorg initgfx working. Once we have accomplished that, let the Live ISO boot into a read-only system, and only once the system is fully booted start adding union mounts if so desired by the user.

Main issue: initgfx seems to require a writable /etc and /usr/local/etc since it is doing things like cp -a "${path_tmp_rc_conf}" /etc/rc.conf, rm -f "/usr/local/etc/libmap.d/nvidia.conf", mkdir /usr/local/etc/libmap.d. So the main issue with getting a read-only system running may actually be initgfx?

@probonopd This is basically what ghostbsd did for years. Except that it would tar specific directories at boot like /etc that had to be writable, mount this as tmpfs, and untar the small archive over the top. The rc.initdiskless script if used will already make /var and /tmp writable. If you need additional directories tar is probably the way to go. I don’t know why you would want rw_root_mount="NO" after install though? Yes the main issue with a read only system would be things like initgfx unless you can punch through whatever directories it needs. The next issue would be not allowing the user to instal packages or configure other parts of the system for example if you didn’t ship a driver they needed. Depending on your desired goals I think readonly sounds like a great solution for you.

Just a thought, but we can probably get around the chroot/module issues by nullfs mounting /boot with all of the modules etc onto the chroot’s /sysroot/boot, or wherever necessary. Obviously this needs doing before the chroot call.

I was privately contacted by a helloSystem user who wishes to remain unnamed, and has summarized the issues as follows:

1. Splash screen doesn’t always appear.
2. The whole system image needs to be decompressed before the user can start using the system. CPU utilization when decompressing system.uzip is pretty high, which can turn it into a bottleneck on computers with slower CPU’s and/or can cause the CPU fan to spin at high RPM thus generating unpleasant noise.
3. The whole system image needs to be read into memory before the user can start using the system which takes time and depends on I/O speed.
4. The whole system image is kept in memory, which raises the minimum required RAM to 4 GB and reduces the amount of RAM available for normal operation of the system.

The user presented two solutions to problem 2 and noted that 3 and 4 need to be addessed separately and noted:

Both solutions drop uzip, which apart from solving problem 2 above also saves us from kernel panics and hangs caused by or related to uzip (see issue #102). It also makes building helloSystem ISO’s quicker. For problems 3 and 4 above using unionfs is not the solution. See the BUGS section in mount_unionfs(8) if you wonder why.

Solution 1 Use as system.img the byte stream generated by ‘zfs send’. It contains only the data from the source zpool, without any free disk space. When booting, you create a 3 GB memory disk, then create a zpool on that memory disk and then you use ‘zfs recv’ to fill in the data. This solution is already in use by GhostBSD so you can take a look at their scripts. The increase in size measured on one recent helloSystem .iso is 10.5% or 174 MB. With Zstandard compression you will likely get even better results.

Solution 2 Have system.img be a disk image of the sole disk of a zpool which is 1.6 GB in size and is full of 1.6 GB of useful data. On boot, copy system.img to a memory disk (using simple dd; there is no need to treat it as a device and to use zfs send | zfs recv) to make it read-write. Then add a second memory disk to the pool which will contain all the free disk space in the pool. Care may need to be taken to avoid ZFS seeing two zpools with the same GUID’s.

For problem 4 the use of swap space was suggested, which unfortunately does not work on truly read-only media such as DVD.

So the solutions suggested by the unnamed user appear to be (partial) alternatives to what @pkgdemon has implemented and what seems to address at least 3 and 4.

Probably all solutions come with specific pros and cons which I currently do not fully understand yet. So some experimentation will be needed to see what works best for helloSystem.

Thank you everyone sharing your ideas and code! 👍

Been experimenting in the 12.2-unionfs branch but it is not working.

https://github.com/helloSystem/ISO/blob/dea799d632438cf2d065f03a5a8b62ac595a1359/overlays/ramdisk/init.sh#L33

fails.

Any help appreciated.