I had assumed that it would permanently prevent my callback from being called by the signal but it does just stop the default for that instance of the signal (when executed in the callback).
You can replace the default handler with an empty callback function that does nothing by using g_signal_override_class_handler. However, this would have effect for all instances of GtkTextView in your application.