Yes, you need an installed desktop file with the same name as your application id—so, in this case, you need an org.example.mytestapp.desktop file.
The GNotification API will use the org.gtk.Notification D-Bus interface when running under GNOME; this interface requires a desktop file for your application, to allow the shell to match the notification to the application that sent it, and to allow the notification to re-launch the application if the application instance goes away before the user has acknowledged the notification.
When running outside of GNOME, GNotification will use the old org.freedesktop.Notification D-Bus interface, which does not have these capabilities.
Likely because they either use the old org.freedesktop.Notification interface directly, or because they have a desktop file.
notify-send uses the old org.freedesktop.Notification D-Bus interface.
org.gtk.Notifications and org.freedesktop.Notifications are two different D-Bus interfaces that provide desktop notification functionality. The former is not an implementation of the latter or vice-versa.
Is there a simple way to find out what notification backend GLib is using when my application is running?
I looked a bit at the GLib code, but i found no other way than to reimplement the same logic GLib uses.