In our work with automotive we have run into issues where we would like to store /etc and /var on a different filesystem than the regular /ostree directory. This would allow us to use different types of protections (encryption, integrity checks, IMA policy, etc) for the writable files in the system compared to the read-only system files.
Currently it seems that /etc is part of the deploy directory for the currently booted commit (/ostree/deploy/$os/deploy/$CSUM), and /var is in a shared per-distro dir (/ostree/deploy/$os/var). Is there any chance we could have some kind of configuration options to store these in a custom location?
The /var one is easy, as it is already a bind mount. We just need some option to have a different base directory for all bind mount. We could probably also use this as-is to use custom filesystems for subsets of /var (like say /var/log) by setting up the right mounts before ostree-prepare-root is run.
Handling /etc is trickier as it is stored separately for each commit, and it does the 3-way merging. However, I wonder here if what we want isn’t actually a read-only etc, potentially with an overlayfs mount on top. Then the overlayfs, or custom bind-mounts can be used to apply runtime changes where and how it is needed.
Making /var a mount point is definitely supported; the ostree-system-generator goes to some effort to support that.
/etc is a different case. However, if we don’t want a persistent/etcthen we also don’t need a 3 way merge. I think having some config option to enable a transient /etc would make total sense. We already make /etc a bind mount in the readonly /sysroot case. I guess right now this might make sense as a repo option, similar to sysroot.readonly? We’d have sysroot.etc = transient perhaps?
So, in ostree-prepare-root, where would I put the upper layer of the transient etc? I don’t think any filesystem outside /sysroot survives across the pivot_root? So, is this even possible? Maybe i can mount a tmpfs, use it as a lower and then unmount it (the overlay will get a private copy of the mount).
Another option is to have ostree just set up a straight read-only etc, and then have the rootfs set up the overlay, using a systemd mount unit.
systemd-volatile-root.service seems related to this. It seems to do a bit much, but clearly something similar could be done that just does a transient tmpfs for /etc
I don’t think any filesystem outside /sysroot survives across the pivot_root?
systemd’s switchroot preserves /run from the initramfs (this is how data passing in general from the initramfs works), which is the right place for the upper.