When using GtkGridView and GtkColumnView (and other list-based widgets), the focus moves between internal children of the GtkListItemWidget class. So that is what orca sees then the focus moves around. But GtkListItemWidget doesn’t propagate any accessible information from its (application-provided) child.
See https://gitlab.gnome.org/GNOME/nautilus/-/issues/2460 for an example and test script.
Currently, as a workaround in nautilus, we poke at the GtkListItemWidget and set it as being labelled-by its child. Which is all kinds of wrong, but the only solution we were able to come up with this close to release.
It would be nice to have a proper, not hackish solution. Maybe this needs some improvements or new API in GTK? Or is it just me being a newbie at a11y in GTK?