GSettings in Flatpak crashes session via inotify

[adrianvovk]

In xdg-desktop-portal#241, it was proposed to expose GSettings (i.e. dconf) via a portal. Instead, GSetting’s “keyfile” backend was used to store an INI file in the Flatpak’s private directory and that’s that. inotify is used to monitor the file for changes.

Quick look at KDE’s kconfig seems to indicate that they use something very similar, but I’m unsure how notifications work. Seems like whenever a client changes a config file, it anonymously broadcasts onto the session bus that “I just changed these settings paths: …” and that’s about it. I didn’t even know dbus supports something like that, and it’s unclear to me whether or not that would work through the Flatpak sandbox.

The problem with our approach is inotify. The kernel, by hard-coded default, only allows 128 individual inotify connections per user. Ultimately, this means that having 128 processes inside of some Flatpak containers that use GSettings, will exhasut the user’s inotify resources.

You can actually hit this 128 process limit quite easily by opening many tabs in Epiphany. Each web worker process (and thus tab) makes a GSettings connection, which of course inside of a Flatpak means using inotify, which means if you have ~128 tabs open in Epiphany the resource is exhausted. Related: inotify usage is way too high, Epiphany crashes after opening ~75 tabs (#1454) · Issues · GNOME / Epiphany · GitLab.

This eventually leads to full session crashes: gnome-software will install a system update, which causes dbus-broker to try and relaunch itself, which fails because it cannot set up inotify watches it needs, which means that there’s no session bus anymore, which rapidly brings down GNOME-shell and the entire rest of the session.

So, ideas on how to best solve this would be appreciated. I’ve yet to come up with anything better than just bumping the limit in the kernel (or maybe make systemd tune the value by default? Or make it a per-distro change?), but I don’t know the repercussions of that. If anyone has alternative ideas I’m all ears!

[ebassi (Emmanuele Bassi)]

The usual limit for open fds on recent OSes is 1024 per user, so 128 processes seems to me a misconfiguration.

In any case, yes: this is a well known issue, and it’s not really solvable; back when we had a single settings storage per session we could get a user daemon to multiplex the write operations, and notify any reader of changes in order to reload the database: sandboxing put an end to that.

The problem with exposing dconf inside the sandbox is that dconf watches the whole database, which means having some kind of a proxy to allow applications to only see their own schemas; see, for reference: wip: Dconf by matthiasclasen · Pull Request #2295 · flatpak/flatpak · GitHub

In practice, exposing dconf is a complex issue and nobody ever figured out how to make that particular bit of infrastructure work. It would require:

• a D-Bus proxy set up by flatpak to let dconf’s user daemon receive write commands and notify of changes caused by a write
• some mechanism inside dconf’s user daemon to limit writes to a specific set of schemas and keys
• some mechanism inside libdconf’s GSettingsBackend to limit the ability to enumerate and read a specific set of schemas and keys

All of those have to be programmatically available so that Flatpak can set them up using the appropriate application id. In theory, the GSettingsBackend could check if dconf is being used inside a sandbox and only match the read and write operations using the application id inside the sandbox metadata. Of course, that entirely hinges on dconf’s user daemon to become a security boundary, because it has to stay outside of the sandbox in order to function—applications can’t run their own dconf daemon inside the sandbox itself.

In reality, though, nothing like this is strictly necessary, and I mentioned it only to provide context to the decision to not block on a dconf-based solution; since settings are not shared across sandboxed applications, there’s no need whatsoever to monitor the settings storage file: all changes happen within the sandbox, and are caused by the application itself. This means that the current keyfile settings backend could check if the code is currently running inside a sandbox, and skip monitoring the file and/or directory for changes.

As a side note: I would recommend dropping the keyfile settings backend for sandboxed applications, in favour of a GVDB-based solution specific for sandboxes; storing complex data inside a key/value file leads to weird escaping rules, and it’s not really as efficient as it should, considering that applications may end up shoving a lot of structured data inside their settings.

[adrianvovk]

The usual limit for open fds on recent OSes is 1024 per user, so 128 processes seems to me a misconfiguration

The kernel has a special limit on inotify FDs specifically. The default value is hard-coded in the kernel sources, but can be changed via a sysctl at runtime

[dconf background]

Yeah I’m not convinced doing all that work in dconf is necessary or a good idea.

Also, I eventually want to leverage homed to make each Flatpak’s indivudal .var directory encrypted with a unique per-app key. Using a centralized settings database is a bit antithetical to that - ideally settings belong in the sandbox.

there’s no need whatsoever to monitor the settings storage file

Hmm… what about cases where two different parts of the same app are sharing access to the settings store? For instance, the Epiphany UI process might be changing the settings, and the Epiphany web worker process

I would recommend dropping the keyfile settings backend for sandboxed applications, in favour of a GVDB-based solution specific for sandboxes

This sounds like a great idea.

I suppose we will still have the issue of multiple threads/processes/instances within the same app trying to write to the same settings database. But we can probably ensure that writes are synchronized (could be as simple as flock!) and call it a day.

Frankly this could allow us to get rid of dconf entirely… the only thing missing is change notifications, but we could probably get away with copying KDE’s solution where we make anonymous signal broadcasts without a centralized daemon. It doesn’t have to cross the sandbox barrier, just work inside the sandbox and outside the sandbox.

[ebassi (Emmanuele Bassi)]

@adrianvovk

Hmm… what about cases where two different parts of the same app are sharing access to the settings store? For instance, the Epiphany UI process might be changing the settings, and the Epiphany web worker process

That’s complicated. It probably can be fixed by using some other notification mechanism like D-Bus, or a pipe, but of course GSettings knows nothing about sub-processes, so it cannot reliably do anything about it, and would have to fall back to inotify.

Unless, of course, WebKitGTK/Epiphany uses a different GSettingsBackend instead of the sandboxed one, and then it can deal with notifying different processes within the same scope.

@adrianvovk

Frankly this could allow us to get rid of dconf entirely…

No, it can’t. The whole reason why dconf was written was to ensure that the session would start more quickly by reducing contention on reads when multiple session components—Shell, session services, etc—started and read session-wide settings at the same time. Do not try to re-invent a setting storage mechanism starting from first principles: it’ll end badly for everyone involved.

[adrianvovk]

reducing contention on reads when multiple session components—Shell, session services, etc—started and read session-wide settings at the same time

Is the dconf README inaccurate then when it says:

The server is only involved in writes (and is not activated in the user session until the user modifies a preference).

I mean, if that README is to believed, dconf is essentially the same thing as this GVFS settings storage backend you’re proposing with the addition of a dbus service to manage synchronizing writes to the database.

Clients already read & parse the database in-process - very similar code would be there for the GVFS-based backend. So why can’t the clients themselves ensure that writes to the database are atomic? They’d do the same thing the daemon is doing to ensure a consistent view of the settings when reading, only in-process and using flock (or some other mechanism) to ensure that two different processes aren’t trying to assert control over the database at the same time.

Then we can have all the exact same benefits of dconf when it comes to reading settings (since the code is pretty much the same), without a dbus service being involved at all, which lets all of this work inside of a Flatpak and probably makes the implementation simpler (we’re cutting out a whole IPC layer!).

And we don’t particularly care for the performance of writes, but I’d imagine that ultimately the cost of IPC versus the cost of waiting for write locks on the DB would even itself out given that writes don’t happen that often, and sandboxing reduces overlap between the settings store files that would lead to lock contention, etc.