Too many files open error when using GLib.Process.spawn_async_with_pipes multiple times

[mibi88]

I’m getting a “Too many files open” when running commands asynchronously with GLib.Process.spawn_async_with_pipes in my Markdown editor, here: https://github.com/mibi88/MibiMdEditor/blob/main/src/ui/window.vala#L200-L293 . How can I fix that?

I did something similar with the sample code of valadoc:

private static bool process_line (IOChannel channel, IOCondition condition,
                                  string stream_name) {
    if (condition == IOCondition.HUP) {
        print ("%s: The fd has been closed.\n", stream_name);
        return false;
    }

    try {
        string line;
        channel.read_line (out line, null, null);
        print ("%s: %s", stream_name, line);
    } catch (IOChannelError e) {
        print ("%s: IOChannelError: %s\n", stream_name, e.message);
        return false;
    } catch (ConvertError e) {
        print ("%s: ConvertError: %s\n", stream_name, e.message);
        return false;
    }

    return true;
}

public static int main (string[] args) {
    bool finished = true;
    MainLoop loop = new MainLoop ();
    for (int i=0;i<4000;i++) {
        if (finished) {
            finished = false;
            try {
                string[] spawn_args = {"ls", "-l", "-h"};
                string[] spawn_env = Environ.get ();
                Pid child_pid;

                int standard_input;
                int standard_output;
                int standard_error;

                Process.spawn_async_with_pipes ("/",
                    spawn_args,
                    spawn_env,
                    SpawnFlags.SEARCH_PATH | SpawnFlags.DO_NOT_REAP_CHILD,
                    null,
                    out child_pid,
                    out standard_input,
                    out standard_output,
                    out standard_error);

                // stdout:
                IOChannel output = new IOChannel.unix_new (standard_output);
                output.add_watch (IOCondition.IN | IOCondition.HUP, (channel,
                                  condition) => {
                    return process_line (channel, condition, "stdout");
                });

                // stderr:
                IOChannel error = new IOChannel.unix_new (standard_error);
                error.add_watch (IOCondition.IN | IOCondition.HUP, (channel,
                                 condition) => {
                    return process_line (channel, condition, "stderr");
                });

                ChildWatch.add (child_pid, (pid, status) => {
                    // Triggered when the child indicated by child_pid exits
                    Process.close_pid (pid);
                    finished = true;
                    loop.quit ();
                });
                loop.run ();
            } catch (SpawnError e) {
                print ("Error: %s\n", e.message);
            }
        }
    }
    return 0;
}

I think it is due to the IOChannels, and I tried using shutdown with set_close_on_unref (true) and without, but nothing fixed this problem!

[jfrancis (Jason Francis)]

Just my guess, probably the sources need to be removed:

@mibi88

guint out_id = output.add_watch ( /* ... */ );
guint error_id = error.add_watch ( /* ... */ );
/*  ... */
ChildWatch.add (child_pid, (pid, status) => {
    Process.close_pid (pid);
    GLib.Source.remove(out_id);
    GLib.Source.remove(error_id);
});

[mibi88]

Thanks
I tried, but it didn’t fixed the bug, and said that they are already removed (because I return false in some cases, or at least that’s what I think does it).

[jfrancis (Jason Francis)]

I don’t see how that could be, the source will be holding a reference to the channel until you remove it. You could also try without the IO channels and just set up the source yourself using Source.add_unix_fd.

[mibi88]

I’m trying that, I’ll say you if it works tomorrow.

[lb90]

You might also inspect all files opened in your program: lsof -p [PID]. On Linux you can also use ls -la /proc/[PID]/fd

[mibi88]

I checked that, and I was right, the problem was due to the pipes. So I’m currently trying @jfrancis’s solution.

[mibi88]

@jfrancis

Source.add_unix_fd.

But I don’t really understand how I should do it, what will be called if the IOConditions happen? I’m new to GLib (I know I should have said that before, sorry).

[mibi88]

I finally made it using a Subprocess, and it works fine, even better than it worked before, but it’s slow, really slow, and I still need to fix that.