The default setting (i.e. the one that doesn’t work with fractional scaling) is, to simplify things a bit, compatible with the way X11 has always done HiDPI. This is also how GNOME Shell itself worked since it learned how to do HiDPI on X11 many years ago, and it is by simply drawing things twice as large using a single global integer scaling factor. This has some annoying consequences such as inability to conveniently draw things with the right size per monitor, use fractional scaling factors, or automatically scale HiDPI-unaware applications to make them usable.
In other words, in the default setting, X11 applications implement their HiDPI just as they always have, and display server just goes with it the way it always has.
When you enable the experimental setting, things change a bit; the coordinate space for each monitor is scaled according to their individual scaling factor, and instead of having GNOME Shell manually drawing itself twice as large, it draws itself with an unchanged logical size, but uses higher resolution images or font caches. The same applies to Wayland clients; for example HiDPI aware Wayland surface with the size 100x100 on a HiDPI monitor will actually provide 200x200 sized buffers containing the window content. This is roughly how most Wayland compositors work these days. In other words, if you have two physically similar monitors, one 2K and one 4K, to the compositor, in this mode, they will all have the logical size equivalent to the 2K monitor, which happens to more correctly mimic their physical real world sizes.
The problem with the new way is that it makes X11 applications believe they are always on a LoDPI monitor, even if that is not the case. To make them adequately sized on the screen, compositors will scale them up, and this is why they are blurry.