How to get pixel data from a Cogl.Texture in GJS

[mityax]

In an extension I’m working on, I’m trying to get the raw pixel data of an area of the screen in order to calculate the average brightness in that area. However, I can’t get Cogl.SubTexture.get_data(...) to work as documented.

const shooter = new Shell.Screenshot();
const [content]: [Clutter.TextureContent] = await shooter.screenshot_stage_to_content();
const wholeScreenTexture = content.get_texture();

const area = {
    x: this.pill.x - 20 * this.scaleFactor,
    y: this.y,
    w: this.pill.width + 40 * this.scaleFactor,
    h: this.height,
};

const ctx = Clutter.get_default_backend().get_cogl_context();
const subtex = Cogl.SubTexture.new(ctx, wholeScreenTexture, area.x, area.y, area.w, area.h);

const size = subtex.get_data(PixelFormat.ARGB_8888, 0, null);  // query size of pixel data (works fine)
const buf = new Uint8Array(size);
subtex.get_data(PixelFormat.ARGB_8888, 0, buf);  // doesn't seem write anything to `buf`

console.log("Buf length: ", buf.length, " - max: ", Math.max(...buf.values()));

After this code, buf still contains only zeros, this is the output:

Buf length:  32560 - max:  0

What am I doing wrong here?

Also, as alternative, I tried global.stage.read_pixels(area.x, area.y, area.w, area.h), which would be a perfect fit for my needs, but when called the Shell crashes with this output:

Gjs:ERROR:../gi/arg.cpp:2134:bool gjs_array_from_fixed_size_array(JSContext*, JS::MutableHandleValue, GITypeInfo*, GITransfer, void*): assertion failed: (length != -1)
Bail out! Gjs:ERROR:../gi/arg.cpp:2134:bool gjs_array_from_fixed_size_array(JSContext*, JS::MutableHandleValue, GITypeInfo*, GITransfer, void*): assertion failed: (length != -1)
#0   55667b029b60 i   file:///home/x/.local/share/gnome-shell/extensions/gnometouch@mityax.github.com/extension.js:979 (1aca1a5bce0 @ 597)

If there’s some way to fix this, this would be even better than the cogl/get_data way.

–
For context: This is a follow-up to Memory leak in GJS code, since that topic has been closed automatically.

[andyholmes]

For function like this, typically you would be doing this:

let [byteArray, size] = subtex.get_data(PixelFormat.ARGB_8888, 0);

However, it looks like the source for cogl_texture_get_data() does not have type annotations, so GJS probably can’t interpret the function signature properly.

You may want to open an issue for that, but will likely need to find another route to go until a fix lands.

EDIT:

@mityax

Also, as alternative, I tried global.stage.read_pixels(area.x, area.y, area.w, area.h), which would be a perfect fit for my needs, but when called the Shell crashes with this output:

Sorry, missed this the first time. I would open a bug in GNOME Shell for this, though it may get bumped to GJS.

[mityax]

Okay, thanks for your help. I opened the issues here and here.

I cannot think of any other routes I can take at the moment sadly. I’ve also tried global.stage.paint_to_buffer and paint_to_framebuffer but both gave other distinct errors that go beyond GJS scope.

Is there any other place to ask how to do this? I don’t know much about the organizational side here

[andyholmes]

Probably the best place is the extensions’ matrix channel: https://matrix.to/#/#extensions:gnome.org

I’d guess with a bit of waiting, someone will know a relevant answer. I recall people doing similar-ish things.