Compiling GTK from scratch - compilation of gobject-introspection fails

(Prelude: I’m trying to find the proper forum with my problem. I have tried stackoverflow with the same question, but it seems no one is interested in GTK details there, so I try my luck here.)

I’m trying to compile and install the GTK 3 library from source packages using a custom built GCC into a separate installation directory on linux (mint 18.3). Since I couldn’t find any definitive guide for this, I started with common sense approach. (Is there anything to read? I couldn’t find detailed instructions by using google.)

First I have compiled and installed GCC (10.2.0, tested and working), then I started to install all the tools and libraries needed by GTK. (cmake, ninja, meson, gettext, glib, pixman, cairo, gobject-introspection, etc). Environmental variables (CC, CXX, PATH, PKG_CONFIG_PATH, etc) are carefully set to point to the new tools and installation directories.

After spending significant effort on combing the details, cmake, ninja, meson, gettext, glib, pixman, and cairo have been successfully compiled. However, I can’t compile the gobject-introspection library. After successful configuration by meson it fails with the following mystique error message:

[78/168] Generating gir-glib with a custom command
FAILED: gir/GLib-2.0.gir 
/home/username/temp/gtk-install/gtk-3.24.23/source/native-builddir/gobject-introspection/tools/g-ir-scanner --output=gir/GLib-2.0.gir --no-libtool --quiet --reparse-validate --add-include-path /home/username/temp/gtk-install/gtk-3.24.23/source/native-builddir/gobject-introspection/gir --add-include-path /home/username/temp/gtk-install/gtk-3.24.23/source/gobject-introspection-1.66.0/gir --identifier-prefix=G --symbol-prefix=g --symbol-prefix=glib --c-include=glib.h --namespace=GLib --nsversion=2.0 --library=glib-2.0 --library=gobject-2.0 --external-library --pkg=glib-2.0 --cflags-begin -I/home/username/temp/gtk-install/gtk-3.24.23/native/include/glib-2.0 -I/home/username/temp/gtk-install/gtk-3.24.23/native/lib/x86_64-linux-gnu/glib-2.0/include -DGLIB_COMPILATION -D__G_I18N_LIB_H__ -DGETTEXT_PACKAGE=Dummy --cflags-end /home/username/temp/gtk-install/gtk-3.24.23/native/include/glib-2.0/gobject/glib-types.h /home/username/temp/gtk-install/gtk-3.24.23/native/lib/x86_64-linux-gnu/glib-2.0/include/glibconfig.h /home/username/temp/gtk-install/gtk-3.24.23/native/include/glib-2.0/glib-unix.h 
__Several similar items removed to shorter the message__
/home/username/temp/gtk-install/gtk-3.24.23/native/include/glib-2.0/glib/gwin32.h ../../gobject-introspection-1.66.0/gir/glib-2.0.c
/home/username/temp/gtk-install/gtk-3.24.23/source/native-builddir/gobject-introspection/tmp-introspectb7f8nxow/GLib-2.0: symbol lookup error: /home/username/temp/gtk-install/gtk-3.24.23/source/native-builddir/gobject-introspection/tmp-introspectb7f8nxow/GLib-2.0: undefined symbol: g_number_parser_error_quark
Command '['/home/username/temp/gtk-install/gtk-3.24.23/source/native-builddir/gobject-introspection/tmp-introspectb7f8nxow/GLib-2.0', '--introspect-dump=/home/username/temp/gtk-install/gtk-3.24.23/source/native-builddir/gobject-introspection/tmp-introspectb7f8nxow/functions.txt,/home/username/temp/gtk-install/gtk-3.24.23/source/native-builddir/gobject-introspection/tmp-introspectb7f8nxow/dump.xml']' returned non-zero exit status 127
ninja: build stopped: subcommand failed.

Apparently, this error message isn’t originating from the C compiler but from a tool provided by the gobject-introspection library itself.

Among other things I’ve tried various combinations of library versions: gobject-interspection 1.30.0, 1.36.0, 1.49.2, 1.58.3, 1.60.2, 1.64.1, 1.66.0, glib 2.66.0, 2.58.3, 2.56.4, 2.50.3, 2.46.2, etc. All of the plentyful combinations I have tried breaks down with some error message, similar to the one shown above, but referring to some other undefined symbols. Some of these library version come with a configure script instead of meson. These break down too, therefore I guess the problem is not directly related to meson.

I have spent the entire weekend with this problem and tried to dig up relevant webpages, forums, stackoverflow, etc using google, without any success. Now I feel there is nothing else left than resorting to black magic. Before committing myself to such devices, I’m asking the kind help of the members of the stackoverflow community.

Questions:

  1. What to do now?
  2. Is there any detailed instructions for setting up GTK from scratch (native, and possibly cross compiled from linux to win using mingw)?
  3. How can I determine the required versions of the dependency/dependent libraries of components of GTK? (E.g. What version of library yy satisfies the dependency of libxx version a.b.c?)

(I have to emphasise that I do need to install everything from scratch, since I need to retain some control over the choice of compiler/library versions, versioning of the binaries built with regard to the system .so libraries, etc. So I can not simply install every dev library from the system repository or use precompiled packages. Moreover, I have to install some auxiliary tools as well, since some of the tools/libraries to be installed need higher version of them than available from the system repository. For example cmake 3.5.1 is available form the repo but some libraries need 3.18.)

1 Like

Hi,

Can you explain why you’re doing what you’re doing ?

Thanks!

Hi,

Here is some background information.

I’m a physicist working on computational physics problems, partly in academic environment, partly doing some consultancy. Now I’m trying to put together a development environment for some future projects I’m dreaming about, i.e. I’m trying to learn and test possible tools to decide which one with to go forward. I need a really good compiler, the ability to develop graphics user interface, and some kind of stability and portability.

I want to be able to choose the exact version of the compiler, since in HPC extreme optimisation is unavoidable. (In the relevant, typically tiny part of the program, implementing the core of the numerical method.) Preferably a recent version of the compiler, so the one coming from the repository will likely not be fine. My current installation (mint 18.3) offers GCC 5.4, for example, while version 10.x is available from the GNU webpage. Heavy optimisation is often revealing bugs in the compilers, I have seen various version of both GCC and Int*l C/Fortran to fail with “Internal compiler error” or generate buggy code. (This is not a criticism. Compilers are very complicated creatures and we are pushing them to the edge, so there are some battlefield losses. :slight_smile: ) In this case there is nothing else I could do than moving to a different compiler version. At the same time I would be happy to use fairly modern language features, like c++14/17, etc. Therefore the need to be able to control the exact compiler version.

I would like to develop graphical user interfaces for these programs, so I’m assessing the possible environments, one of them is GTK. I prefer GTK over Qt for its elegance and simplicity. (No need to preprocess, ready for c++11 by libgtkmm and similar libraries, etc.) I would prefer implementing my programs in an “integrated” way, that is, the computational part of the code would go into the same executable as the user interface. I believe this is a healthy strategy for some less complicated projects. I would definitely like to avoid using one compiler for the computational part and an other one for the UI.

Some kind of portability and stability is a must. Linux systems are upgraded regularly, thus both the compiler and the UI libraries change significantly. Readapting a considerable source code for every such change is a nightmare, so I would like to nail down one development environment for a longer time period. Binary portability is a particularly hard question on Linux, but I hope I can solve it as well, at least partially.

Finally, I have a distant dream to be able to compile a MS Win version of my programs on my Linux computer, using GCC as a cross compiler.

Overall, my plan is the following: I install the compiler in a separate directory (say /opt/gcc-x.y.z) and I use the usual environment variables to activate it. Also, I can prepare some “module load …” configuration files. Then I compile all necessary libraries into an other directory (say /opt/workenv-a.b.c) mainly for the graphical user interface. Then I could compile my programs against this “working environment”. I’m trying to create a shell script to go through all the steps needed to compile and install all of the tools.

This could be fairly stable, since I could keep a given version of the compiler (and libraries, etc) for longer times, hopefully for years. It could be fairly easily to transfer to an other computer, since it would be enough to run the setup script, so I could expect reasonable degree of source code portability. For example, this could help a PhD student a lot, since he/she could immediately focus on the physics part of the code, instead of fighting with intricate details of Linux/UI internals. Finally, I would be able to simply copy my resulting executables to other computers, where we want to run them but development is not needed. (Again, I know that binary portability is serious problem in Linux, but let’s solve problems step by step, and I’m starting with the compiler/UI question, since this is far more important.)

Now I’m spending some time to study if this whole idea is feasible or not. Maybe GTK will do the job, if not, I can (re)turn to Qt, or, if everything fails, I would have to discard my dreams of using Linux and I have to go to MS Win and VS Studio. Actually, I would be very unhappy to return to MS Win, but this is a different topic that shall not be discussed here, I think.

I think what you are indirectly working on is a linux distro specialized for HPC. Did you check, if there is one already there, solving this problem.

Few questions:

  1. Did you try running a classic physics HPC program in a non-customized vanilla Linux environment, to quantify if there is an unacceptable performance drop, which can be fixed only by what you’re proposing ?

  2. The main issue I see here is that, the more you customize, the more you need to be sure, you got everything right. When some combination of x + y + z software fails or gives an error, you’re mostly on you’re own, as very few users do what you’re trying to achieve.

  3. What happens when a new compiler comes with new and better / faster code generation ? You will need to repeat the entire process again.

Thanks

As for the error message, it appears that 'g_number_parser_error_quark()' is a new symbol introduced in glib-2.54

https://developer.gnome.org/glib/stable/glib-String-Utility-Functions.html#G-NUMBER-PARSER-ERROR:CAPS

So, you might want to check that you’re running glib >=2.54 in your workenv.

Hi,

Thank you for your kind comments.

Trying to put together any kind of linux distro for myself would be a huge job and it’s futile effort. I do not do that, obviously. My goal is quite the opposite: using a plain vanilla distro, like ubuntu or mint, which have huge user base, thus they are well tested and reliable, and using the version of my choice of a few key components, like GCC and the UI libraries. There are some HPC distros available, but these are usually specialised for cluster computing, providing job submission software, support for efficient interconnect of the cluster nodes, etc, and they usually have little support for user interface development. This is not something I need now. (For other project I had been using such systems.)

Regarding your question 1: The choice of distribution has very little effect on the performance. The important factor is the choice of the compiler producing the machine code of the relatively small part of the program responsible for the mass computations. This is the reason I would like to have the freedom to use an arbitrary version of the compiler on a vanilla distro. And this is one of the reasons motivating me to put together my own toolset, containing GCC, GTK, etc.

Regarding question 2: Yes, doing something on my own way is always a danger. But it is also the source of success, since there is no improvement without deviation from the usual. And I think my aim is a rather limited goal. I’m just trying to compile a UI library and its support elements with a newer version of the compiler than the ancient one provided by the system. Now I feel it’s like trying to conquer the north and south pole at the same time, without warm clothing. :slight_smile:

Regarding your question 3: I hope once I’m done with the first version and I have learnt the tricks, there will be fairly limited work needed to update it, and since in this case I am in the position to decide when and if I want to upgrade my toolset, it could save a lot of headache.

Regarding your comment about the version of glib: I have tried countless combinations, but now I gave a chance to the newest ones: glib 2.66.0 and gobject-introspection 1.66.0. They are producing the same error.

I think I’m not the only one fighting with this trouble. The same or similar problems have been showing up for several years for other people on a variety of distros and library versions:




There is usually no solution other than going for the precompiled version of the introspection library from the distro’s repository.

At this point my goal is very simple: I would like to set up the whole GTK from source, compile a simple hello world program against it, and see everything working. I thought that compiling a recent version of GTK and its base libraries will not be a very serious adventure but it seems I was wrong. Detailed planning of any kind of serious project based on GTK is way too early before I can do this.

1 Like

The typical way to do this is to use a tool like JHBuild. Of course, since you went down the rabbit hole of building your own toolchain, you’re going to need something more complex than this; you will also need a lot more knowledge of how a “from scratch” build is assembled, something that it’s not really recommended for anybody who is not already an expert.

My suggestion would be to use something like BuildStream or the Yocto Project; both of them have the goal to build a whole system from scratch, including a toolchain.

Alternatively, you could use Flatpak to develop your application, since it comes with a run time that contains its own toolchain; you won’t need to build anything at all, and you will be able to use the existing copy of GTK3. The latest freedesktop run time comes with GCC 10, for instance.

1 Like

Dear Emmanuele,

Thank you, these look very promising options, I will explore them.

The reason I started to investigate these topics (compiling from scratch, etc) is exactly that I want to gain expertise before starting relying on these tools and I think the only way to gain expertise is to actually try to do the things myself. Otherwise I will never know anything about it.

Is this expertise documented somewhere in written form, at least partially?

Building systems from scratch is already fairly complicated; that’s why projects like Yocto, BuildStream, and LinuxFromScratch exist, and why each one of them encompasses the equivalent of tens of thousands of man years of work. It’s not an easy knowledge to learn from first principles, and it’s definitely out of scope for the GNOME Discourse.

You need to learn what each build system does; what are the effects of the environment variables; how to set up search paths for tools, compilers, and linkers; the various default configuration files for building a system; how to bootstrap a build environment. Basically, all the stuff that is required to go from a pile of tar archives and turn them, for instance, into a Linux distribution.

I’d start looking at the documentation of those projects I mentioned.

2 Likes

For BuildStream, there are instructions for building GTK (which I didn’t see mentioned above).

I found that JHBuild no longer worked on my fairly old system (for reasons I can’t remember) so I switched to using BuildStream. Eventually I succeeded but there were a few delays due to my inexperience which I’ll share, although I can no longer remember the exact details.

  1. Just following the instructions, the build failed. I seem to recall this was due to a dependency problem in the elements for freedesktop-sdk. My guess at the time was that the master branch of gnome-build-meta may have some instability due to ongoing development so I checked out a tag (3.36.1) and then the build succeeded. If the master branch is not expected to be stable, the wiki could usefully mention that.

  2. The Buildstream shell for the application I was working on didn’t have certain tools available. I think one was gdb. In the end, I found that creating my own element file that depended on both the application and the tools needed worked, e.g. I created the file
    elements/test-<some-application>.bst
    containing

    kind: stack
    depends:
    - sdk.bst
    - world/<the-application>.bst
    

    and opened a shell for that.

  3. Once in a Buildstream shell, I couldn’t see my local filesystem. I ended up using scp to transfer files between the shell and my local machine. Maybe there’s a better way.

Building for the first time will involve a large download (several GBs). Also at least 20G of spare disk space is needed. I had > 120G available according to df but builds kept running out of memory. This was actually due to my lack of understanding about how BTRFS works. Although not a Buildstream matter, I mention it because you may want to keep an eye on space usage when using Buildstream. The amount of space actually used on a BTRFS filesystem is shown by

btrfs fi show

My filesystem was using 214G to store just 90G. I just needed to rebalance the chunks, which is easily and safely done by, e.g.

btrfs balance start / -dusage=40

which rebalances chunks with less than 40% usage. There are other ways a BTRFS filesystem may need to be adjusted though, as noted in the FAQ.

2 Likes

Regular df should report correctly in the Btrfs single device case, or it might be a bug. Same for any manual balance as a work around, probably a bug.

There are probably some (unknown) edge cases still. Suggestions:

  • Report the problem to your distro, include: kernel version and output from:
    • df
    • btrfs filesystem usage /mnt
    • grep -R . /sys/fs/btrfs/<UUID>/allocation/

*Use lsblk -f to discover the mountpoint and UUID for your file system.

  • btrfs maintenance scripts is maintained by the upstream btrfs-progs maintainer, and contains some useful systemd services and timers, including a balance timer. This basically papers over rare issues related to block group allocation, and avoids problems without the user having to babysit the file system.

The first option is ideal. The second option is a workaround. But they are mutually exclusive, the choice is yours.

At least in Fedora, the expectation is to report this against the kernel (and ideally assign to, or cc, fedora-kernel-btrfs@fedoraproject.org so we can figure out what’s going on. I personally don’t ever balance any of my file systems, except in cases where I’m adding/removing drives or changing raid profiles.

Just to clarify - I’m not claiming that there is an issue with a file system tool or BTRFS. The issue was my lack of understanding about what free space means for BTRFS and, from reading, it looks like other Linux users may be unaware. As Buildstream uses a lot of disk space and may very well trigger space issues, it seemed relevant to mention this, to improve the Buildstream experience for those using BTRFS.

By the way, would df report free space correctly even for a single device where e.g. only 30% of each chunk is used? I can see that the figure from df would be indicative for a BTRFS file system that is ‘well balanced’. Or are you saying that there should never be a need to manually rebalance if BTRFS is working properly? (I thought the need to rebalance was inherent in the design of BTRFS.) Good to know about the maintenance scripts.

My suggestion is anyone seeing behavior on Btrfs that’s different from ext4/XFS in the same situation, start asking questions. #btrfs on irc.freenode is a good resource.

As Buildstream uses a lot of disk space and may very well trigger space issues, it seemed relevant to mention this, to improve the Buildstream experience for those using BTRFS.

For the most part it should behave like ext4 or XFS. Sometimes folks are surprised to discover Btrfs uses space when deleting files. Since metadata writes are always copy on write, updates to indicate file deletions are written to free space. Space is initially consumed, and only once the transaction fully commits, are the deleted blocks actually freed and free space goes up. Btrfs commit interval is 30 seconds, so fairly substantial delayed allocation could result in what seems like delayed free space reporting.

Snapshots and reflink copies, which create shared extents, and transparent compression, can all result in multiple truths when it comes to du style reporting.

There could be some Btrfs specific optimizations worth taking advantage of if the workload involves creating and destroying many files or resetting state.

By the way, would df report free space correctly even for a single device where e.g. only 30% of each chunk is used?

Yes.

Or are you saying that there should never be a need to manually rebalance if BTRFS is working properly?

In theory it shouldn’t be necessary. In practice there are some known difficult edge cases still being worked on, and there could always be unknown edge cases not yet discovered.

I tend to take the view that we’re better off not doing any routine balancing, in order to expose and report these real world problems, so upstream can fix them. And I know they want to fix them. But they also know folks have work to do, and may not want to be interrupted with filing Btrfs bug reports - hence the user space maintenance scripts (specifically the balance service unit). That script has becomes less aggressive as time goes on, as improvements are made in btrfs kernel code, so I do expect it will become unnecessary eventually.