The nested mode was dependent on the X11 backend, so if the X11 backend is disabled, which is now the default, the nested mode is no longer available.
The replacement for the nested mode is called “development kit”, so now instead run dbus-run-session -- gnome-shell --devkitto get a nested shell instance.
You might see that command complain that mutter-devkit binary is missing
(gnome-shell:740984): libmutter-WARNING **: 04:14:29.485: Failed to launch devkit: Failed to execute child process “/usr/libexec/mutter-devkit” (No such file or directory)
For fellow Ubuntu users: you need to install the package mutter-dev-bin. For other distros, you’ll need to search for the package that provides it.