I think that putting it in the manifest is “the right way”. It uses pip to install the model and as far as I know pip is only present in the Sdk (i.e during build time).
Have you seen this section of the spacy docs? It suggests to install the model’s package directly. Maybe you could just install it similar to all the other python packages?
For dealing with dependency hells I strongly recommend to use a package manager, e.g. poetry, pipenv, etc. The flatpak-builder-tools also has a script to generate the flatpak modules from a poetry lockfile.