GNOME Builder: import spacy to a python-project through flatpak manifest fails: help appreciated

Platform
, , ,
1

Hello there!

I’m having trouble importing spacy to my project, using python. I used the Flatpak PIP Generator to create the JSON-File. On the first run, I got the error message that numpy was missing (although it’s in the dependencies), and on generating more JSON for numpy, finally got a lengthy error message from the compiler:

[ 2/46] Cythonizing spacy/kb/candidate.pyx
  Traceback (most recent call last):
    File "/usr/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
      main()
    File "/usr/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/usr/lib/python3.10/site-packages/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/usr/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
      exec(code, locals())
    File "<string>", line 245, in <module>
    File "<string>", line 232, in setup_package
    File "/usr/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1112, in cythonize
      cythonize_one(*args)
    File "/usr/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1235, in cythonize_one
      raise CompileError(None, pyx_file)
  Cython.Compiler.Errors.CompileError: spacy/kb/candidate.pyx
 error: subprocess-exited-with-error

I fear, that it’s not a thing I can do to fix this problem, but would be glad and thankful to receive any hints what to do. (Just to get to trust the process, I imported fpdf in another project, and that seemed to work fine. I hope I didn’t overlook anything obvious – I’m more of a newbie when it comes to the gnome platform.)

The flatpak manifest looks like this:

{
    "app-id" : "org.jzqk.gapgenerator",
    "runtime" : "org.gnome.Platform",
    "runtime-version" : "43",
    "sdk" : "org.gnome.Sdk",
    "command" : "gap_generator",
    "finish-args" : [
        "--share=network",
        "--share=ipc",
        "--socket=fallback-x11",
        "--device=dri",
        "--socket=wayland"
    ],
    "cleanup" : [
        "/include",
        "/lib/pkgconfig",
        "/man",
        "/share/doc",
        "/share/gtk-doc",
        "/share/man",
        "/share/pkgconfig",
        "*.la",
        "*.a"
    ],
    "modules" : [
	{
	    "name": "python3-numpy",
	    "buildsystem": "simple",
	    "build-commands": [
		"pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"numpy\" --no-build-isolation"
	    ],
	    "sources": [
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/e4/a9/6704bb5e1d1d778d3a6ee1278a8d8134f0db160e09d52863a24edb58eab5/numpy-1.24.2.tar.gz",
		    "sha256": "003a9f530e880cb2cd177cba1af7220b9aa42def9c4afc2a2fc3ee6be7eb2b22"
		}
	    ]
	},
    	{
	    "name": "python3-spacy",
	    "buildsystem": "simple",
	    "build-commands": [
		"pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"spacy\" --no-build-isolation"
	    ],
	    "sources": [
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl",
		    "sha256": "6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/95/7e/68018b70268fb4a2a605e2be44ab7b4dd7ce7808adae6c5ef32e34f4b55a/MarkupSafe-2.1.2.tar.gz",
		    "sha256": "abcabc8c2b26036d62d4c746381a6f7cf60aafcc653198ad678306986b09450d"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/ff/a1/0ece7b0cceda6ca764ecba21e32f099cebff3448d88aa33ff0fce708d34b/blis-0.7.9.tar.gz",
		    "sha256": "29ef4c25007785a90ffc2f0ab3d3bd3b75cd2d7856a9a482b7d0dac8d511a09d"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/dc/28/a2b0cc4bfa7916ef9caf08475be5810fc564411c5a801f225a1f40a458c5/catalogue-2.0.8-py3-none-any.whl",
		    "sha256": "2d786e229d8d202b4f8a2a059858e45a2331201d831e39746732daa704b99f69"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/71/4c/3db2b8021bd6f2f0ceb0e088d6b2d49147671f25832fb17970e9b583d742/certifi-2022.12.7-py3-none-any.whl",
		    "sha256": "4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/96/d7/1675d9089a1f4677df5eb29c3f8b064aa1e70c1251a0a8a127803158942d/charset-normalizer-3.0.1.tar.gz",
		    "sha256": "ebea339af930f8ca5d7a699b921106c6e29c617fe9606fa7baa043c1cdae326f"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/c2/f1/df59e28c642d583f7dacffb1e0965d0e00b218e0186d7858ac5233dce840/click-8.1.3-py3-none-any.whl",
		    "sha256": "bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/b1/c4/07291f4947d20f545eee76ef20d1eacfb1f80d1d6ab4792bfa6797e0578c/confection-0.0.4-py3-none-any.whl",
		    "sha256": "aeac5919ba770c7b281aa5863bb6b0efed42568a7ad8ea26b6cb632154503fb2"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/b0/d3/50c90dd821e44ad289653f0029d1690fca82f90df293e77e21443f5ea4fc/cymem-2.0.7.tar.gz",
		    "sha256": "e6034badb5dd4e10344211c81f16505a55553a7164adc314c75bd80cf07e57a8"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl",
		    "sha256": "90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/fe/c3/0d04d248624a181e57c2870127dfa8d371973561caf54333c85e8f9133a2/langcodes-3.3.0-py3-none-any.whl",
		    "sha256": "4d89fc9acb6e9c8fdef70bcdf376113a3db09b67285d9e1d534de6d8818e7e69"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/f6/24/30e74dc5970f357a897b3ec0fd44e1330c3f5fabefcfa0bc54c80b7ee0a5/murmurhash-1.0.9.tar.gz",
		    "sha256": "fe7a38cb0d3d87c14ec9dddc4932ffe2dbc77d75469ab80fd5014689b0e07b58"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/e4/a9/6704bb5e1d1d778d3a6ee1278a8d8134f0db160e09d52863a24edb58eab5/numpy-1.24.2.tar.gz",
		    "sha256": "003a9f530e880cb2cd177cba1af7220b9aa42def9c4afc2a2fc3ee6be7eb2b22"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/ed/35/a31aed2993e398f6b09a790a181a7927eb14610ee8bbf02dc14d31677f1c/packaging-23.0-py3-none-any.whl",
		    "sha256": "714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/82/c6/683e3955de9a13b14dfa3ea358cd58f3914057e8064a2dcbfd450958e72e/pathy-0.10.1-py3-none-any.whl",
		    "sha256": "a7613ee2d99a0a3300e1d836322e2d947c85449fde59f52906f995dbff67dad4"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/f1/78/c28a568317088b9cef21f8516bed2f41210492ae569c5842ad80251631a0/preshed-3.0.8.tar.gz",
		    "sha256": "6c74c70078809bfddda17be96483c41d06d717934b07cab7921011d81758b357"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/28/59/5d2fc3499d9ce8ce48ee7e00f043d5cc429a9198bd96c3512809428ade15/pydantic-1.10.5.tar.gz",
		    "sha256": "9e337ac83686645a46db0e825acceea8e02fca4062483f40e9ae178e8bd1103a"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/d2/f4/274d1dbe96b41cf4e0efb70cbced278ffd61b5c7bb70338b62af94ccb25b/requests-2.28.2-py3-none-any.whl",
		    "sha256": "64299f4909223da747622c030b781c0d7811e359c37124b4bd368fb8c6518baa"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/47/80/c2d1bdd36c6b64ae566d9a29724291510e4f3796ce99639d3c2999286284/smart_open-6.3.0-py3-none-any.whl",
		    "sha256": "b4c9ae193ad6d3e7add50944b86afa0d150bd821ab8ec21edb26d9a06b66f6a8"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/5a/52/30975d56f3fb068a72b1ce7b0d4c59e78247020d90c52a6cdf4420df8482/spacy-3.5.0.tar.gz",
		    "sha256": "fe20127012992778804d93f75ce9370d588573072639c3832cec38f54bf7e4a5"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/c3/55/12e842c70ff8828e34e543a2c7176dac4da006ca6901c9e8b43efab8bc6b/spacy_legacy-3.0.12-py2.py3-none-any.whl",
		    "sha256": "476e3bd0d05f8c339ed60f40986c07387c0a71479245d6d0f4298dbd52cda55f"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/62/8c/814e0bd139a8c94b50298be3a4e640d90cdce78efe0099e373a767b7d854/spacy_loggers-1.0.4-py3-none-any.whl",
		    "sha256": "e050bf2e63208b2f096b777e494971c962ad7c1dc997641c8f95c622550044ae"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/1c/9e/6497a75e4d72f21a56c8b1ddd21a5be3962850ff52e6c309ce6f2fcad881/srsly-2.4.5.tar.gz",
		    "sha256": "c842258967baa527cea9367986e42b8143a1a890e7d4a18d25a36edc3c7a33c7"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/7b/3a/f6672ba25ca15e50754d32101f65560964db26ea943fb7581e1edfd1d718/thinc-8.1.7.tar.gz",
		    "sha256": "0f08f6d1fc50e28bf18814ca2b1c807cd4d59a930d713459a675e086c4779af9"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/47/bb/849011636c4da2e44f1253cd927cfb20ada4374d8b3a4e425416e84900cc/tqdm-4.64.1-py2.py3-none-any.whl",
		    "sha256": "6fee160d6ffcd1b1c68c65f14c829c22832bc401726335ce92c52d395944a6a1"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/0d/44/56c3f48d2bb83d76f5c970aef8e2c3ebd6a832f09e3621c5395371fe6999/typer-0.7.0-py3-none-any.whl",
		    "sha256": "b5e704f4e48ec263de1c0b3a2387cd405a13767d2f907f44c1a08cbad96f606d"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/31/25/5abcd82372d3d4a3932e1fa8c3dbf9efac10cc7c0d16e78467460571b404/typing_extensions-4.5.0-py3-none-any.whl",
		    "sha256": "fb33085c39dd998ac16d1431ebc293a8b3eedd00fd4a32de0ff79002c19511b4"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/fe/ca/466766e20b767ddb9b951202542310cba37ea5f2d792dae7589f1741af58/urllib3-1.26.14-py2.py3-none-any.whl",
		    "sha256": "75edcdc2f7d85b137124a6c3c9fc3933cdeaa12ecb9a6a959f22797a0feca7e1"
		},
		{
		    "type": "file",
		    "url": "https://files.pythonhosted.org/packages/58/5f/1f2833d59abf53e24dbadc21b0565fe10c64545da8705faed8eff3b14745/wasabi-1.1.1-py3-none-any.whl",
		    "sha256": "32e44649d99a64e08e40c1c96cddb69fad460bd0cc33802a53cab6714dfb73f8"
		}
	    ]
	},
    	{
            "name" : "gap_generator",
            "builddir" : true,
            "buildsystem" : "meson",
            "sources" : [
                {
                    "type" : "git",
                    "url" : "file:///home/jan/git-Projekte"
                }
            ]
        }
    ]
}
2

spacy’s pyproject.toml lists more build dependencies than just numpy. Maybe you should install these before the python3-spacy module?

3

Thanks a lot! That brought me on the right path!

By integrating those dependencies (and some more, as those were dependent on others), I was able to build the project. (I have heard the term »dependency hell« before, and now I have a better understanding of it.)

I’m still struggling with the language model I would like to download; for just using a spacy model, I would type

python -m spacy download de_core_news_sm

in my virtual environment, resulting (I believe) in two directories which then hold those files. Is the flatpak manifest the right place for this line, and I just didn’t succed in putting it right (I already tried), or does it have to go somewhere else?

4

I think that putting it in the manifest is “the right way”. It uses pip to install the model and as far as I know pip is only present in the Sdk (i.e during build time).

Have you seen this section of the spacy docs? It suggests to install the model’s package directly. Maybe you could just install it similar to all the other python packages?

For dealing with dependency hells I strongly recommend to use a package manager, e.g. poetry, pipenv, etc. The flatpak-builder-tools also has a script to generate the flatpak modules from a poetry lockfile.

1 Like
5

Thank you so much! It took me a while to figure out the right way to put it, but now I suceeded and am able to use these spacy language models. Thank you very much for your hints, they pointed me to the exactly right places!

In case anyone stumbles over the same problem, here is the section of my flatpak manifest:
(I generated the sha256-checksum after downloading the file with sha256sum in the terminal.)

	{
	    "name": "download-german-language-model",
	    "buildsystem": "simple",
	    "build-commands": [
		"pip install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"de_core_news_sm\" --no-build-isolation"
	    ],
	    "sources": [
		{
		    "type": "file",
		    "url": "https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.5.0/de_core_news_sm-3.5.0-py3-none-any.whl",
		    "sha256": "c9c06808be2f1e7f448b37f94391a50dab2e2f72f9b5c2e979be18aebac4a06c"
		}
		]
	}

Just in case you wondered: I use the language model to generate gapped texts for learners:

grafik
grafik719×493 14.1 KB

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.