Proposal: Transition GNOME Shell JS & Extensions To TypeScript + Guide For Extensions Today

I just saw that there is a GNOME Extensions Reboot Initiative, and I think this would be an excellent opportunity to pave the way for solving a vast majority of issues with extension development, including issues in the JS portion of GNOME Shell itself, by having plans to transition to TypeScript, and have official TypeScript bindings for all GJS APIs.

As TypeScript is a superset of JavaScript that generates JavaScript as its output, very little effort is required to start the transition, and no effort is required to consume it. Linux distributions could continue to package GNOME Shell as they always do if we included the generated JavaScript sources in the source tarballs, and made TypeScript is merely a part of the chain that generates the source tarballs.

So as a disclaimer, I’m a Pop!_OS developer that has been working on the Pop Shell tiling window manager GNOME extension over the last 9 months, which is written entirely in TypeScript, so this is based on my own experiences developing and maintaining a large GJS project in TypeScript.

I have been thanked by contributors since the release of the extension for my decision to use TypeScript, because it’s made it easy to get familiar with and working with GNOME JS and the extension without needing prior experience with either. Even a lack of JavaScript experience has been easy to overcome because IDE support is first class for Pop Shell development. All GNOME type information that I’ve manually added to a declaration is immediately available in their IDE as they type, as well as all the APIs throughout the extension.

I think we could bring these benefits to the whole of GNOME Shell and its ecosystem as well, and I would be willing to contribute to such a cause to make it happen.

Why Switch

GNOME Shell development could focus more time and effort into feature development and optimizations, and less time on fixing bugs and documenting code. The self-documenting nature of static typing would enable anyone to contribute with reduced barrier to entry, and first class IDE support would finally be available to anyone who wants to write for or against GNOME Shell.

Many extensions I’ve come across are riddled with errors that would have never been an issue if TypeScript were the standard. It’s always painful when getting logs from a person having issues with their GNOME Shell session, to find that the majority of the logs are littered with basic type errors. Lets fix this once and for all.

Self-Documenting APIs

Currently, documentation for GNOME Shell APIs are based entirely on good will and effort. As we’ve all experienced when maintaining projects, although documentation is rarely kept in sync with the actual state of the code, code never lies. However, unless you leave the safety of the API documentation and begin digging into the actual code being written about, you’ll never know for sure if the code actually does what the documentation claims it will do, or if there are certain behaviors that are possible which hasn’t been documented. There may even be some features that aren’t documented at all, even though it exists in the code.

This is especially so for software written in dynamic languages such as JavaScript, where it is perilous to interact with any library that you’re not already familiar with, and even the ones you already know. Code comments are the only way to find out what kinds of inputs are accepted by a function, or what kind of types are returned.

The worst is when the code has behaviors that weren’t intended by the programmer, but because there isn’t a static type system validating their code assumptions, they have no idea that the code is broken. The code could be broken for years before someone finally realizes it, and by then it’s already been too late for everyone who’s suffered bugs as a result of it.

First class IDE support in all major JavaScript IDEs

One of the most critical barriers to entry to GNOME Shell development is the lack of IDE support. It’s simply not possible to source type information from GNOME Shell or its GJS bindings to be consumed by an IDE. Today, it’s completely up to the programmer to reference the DevDocs API as their guide, and hope that they’re using it correctly.

TypeScript could solve this because every TypeScript file generates a detailed source map containing all known information about the state of all code in the file. These source maps are used by all JavaScript IDEs supporting TypeScript to perform advanced in-depth auto-completions, in-line code linting, and even provide helpful documentation in tooltips when hovering your mouse over a known definition. It would be entirely possible to distribute these detailed source maps for developers to use when they’re writing extensions.

Static Type System

It goes without saying that a static type system makes APIs self-documenting, and validates code written by the programmer. One of the major benefits of the type system in TypeScript is that it makes writing complex and generic classes trivial. The generic hop slot arena allocator at the bottom of this post is one such example of a type benefiting from having generics with a type system. Take this sum type that replicates a familiar Rust type as another:

export const OK = 1;
export const ERR = 2;

export type Result<T, E> = Ok<T> | Err<E>;

export interface Ok<T> {
    kind: 1,
    value: T
}

export interface Err<T> {
    kind: 2,
    value: T
}

export function Ok<T, E>(value: T): Result<T, E> {
    return { kind: 1, value: value };
}

export function Err<T, E>(value: E): Result<T, E> {
    return { kind: 2, value: value }
}

let result = Err("failed"); // Err<string>

if (result.kind === ERR) {
    global.log(`error: ${result.value}`);
}

Support for newer JavaScript syntax and editions

GJS supports a syntax that’s almost ES2015, but not quite. However, because TypeScript is a transpiler, you’re completely capable of using modern JavaScript syntax available in the latest editions today, and TypeScript will happily rewrite your code to be compatible with an older edition. You can then potentially continue to support both old and new versions of GJS alike with a single codebase.

How a TypeScript Conversion Would Work

The TypeScript compiler supports incomplete and partial conversions in order to assist transitions from JavaScript to TypeScript, so the process of converting GNOME Shell to TypeScript is actually completely painless in practice. Although strict rules can be enforced in a tsconfig.json to keep the quality of code at a high bar in later stages of development — once the transition is complete — TypeScript is as permissible as JavaScript by default.

The TypeScript compiler fully supports bundling JS alongside TS sources, so it’s possible to perform a gradual migration where both JS and TS sources peacefully co-exist in the same directories. TypeScript will simply copy the JS files to the target directory after generating JS files from TS files. GJS itself needs not know how we generated the JS, and pre-compiled JS sources could be cached so that Linux distributions need not have TypeScript packaged. This is purely for the benefit of the developer.

Additionally, because TypeScript is a superset of JavaScript, renaming a .js file to .ts is enough to make JS sources compatible with the TypeScript compiler. TypeScript-specific syntax can be added on at a later time as you are documenting your code with code.

Real World TypeScript Extension

I’ve gained a lot of experience using TypeScript to generate JavaScript for GJS over the last ~9 months as I’ve been working on Pop Shell, which is an i3-like tiling window manager extension for Pop!_OS’s GNOME session. I originally started with JavaScript, as there is no official support for TypeScript, but after a month or two into the extension’s development, the struggle became real, and a type system was the answer.

In addition to using it to protect myself from misusing (and forgetting) APIs as I develop with them, it has been instrumental in logic-checking all the code that I write which interacts with GNOME’s JS APIs. Even though GNOME does not officially support TypeScript, I have a declaration file which declares all of the GNOME JS APIs that I use throughout the extension, and I update it as I learn new undocumented behaviors along the way. Misuse of types immediately highlights the code in my IDE as I type to tell me that it’s wrong.

Doing so also opened the door to making it practical to develop complex architectures easily. In particular, Pop Shell is written around a custom entity-component system architecture, which uses generational IDs to represent windows in the main world, and forks in the tiling world. The end result is that the vast majority of my time is spent implementing features and fixing bugs that are unrelated to types.

How to Write TypeScript Extensions Today

  1. Installing the tsc TypeScript compiler is all that’s necessary to begin
  2. At the root of your project, create a tsconfig.json like so:
    {
        "compilerOptions": {
            "target": "es2015",
            "strict": true,
            "outDir": "./target",
            "forceConsistentCasingInFileNames": true,
            "downlevelIteration": true,
            "lib": [ "es2015" ],
            "pretty": true,
            "sourceMap": true,
            "declaration": true,
            "removeComments": true,
            "noUnusedLocals": true,
            "noUnusedParameters": true
        },
        "include": [
            "src/*.ts"
        ]
    }
    
  3. After running tsc, the .ts files in src/ are transpiled into ES2015-style JavaScript, and output to a target/ directory.
  4. However, because GJS is not fully compliant with ES2015, create a Makefile rule to use sed to fix every generated JS file to replace the incompatible syntax with compatible ones
    for file in target/*.js; do \
    	sed -i \
    		-e 's#export function#function#g' \
    		-e 's#export var#var#g' \
    		-e 's#export const#var#g' \
    		-e 's#Object.defineProperty(exports, "__esModule", { value: true });#var exports = {};#g' \
    		"$${file}"; \
    	sed -i -E 's/export class (\w+)/var \1 = class \1/g' "$${file}"; \
    	sed -i -E "s/import \* as (\w+) from '(\w+)'/const \1 = Me.imports.\2/g" "$${file}"; \
    done
    
  5. To get started with working with GJS and GNOME APIs, you’ll need a declaration module to declare the types and global variables that GJS exposes. Any file that ends with .d.ts will instruct the compiler to acknowledge the existence of these types.
    declare const global: Global,
        imports: any,
        _: (args: string) => string;
    
    interface Global {
        log(msg: string): void;
        display: Meta.Display;
    }
    
    declare namespace Meta {
        interface Display extends GObject.Object {
            get_focus_window(): Meta.Window | null;
        }
        
        interface Window extends Clutter.Actor {
            minimized: Readonly<boolean>;
            activate(time: number): void;
        }
    }
    
  6. The only thing to be wary of is to call this at the top of any module that imports from other modules:
    // @ts-ignore
    const Me = imports.misc.extensionUtils.getCurrentExtension();
    
  7. And then you have free reign to import modules like so:
    import * as Config from 'config';
    
  8. Where a module that exports a class might look like so:
    /** Hop slot arena allocator */
    export class Arena<T> {
        private slots: Array<null | T> = new Array();
        private unused: Array<number> = new Array()
    
        get(n: number): null | T {
            return this.slots[n];
        }
    
        insert(v: T): number {
            let n;
            const slot = this.unused.pop();
            if (slot !== undefined) {
                n = slot;
                this.slots[n] = v;
            } else {
                n = this.slots.length;
                this.slots.push(v);
            }
    
            return n;
        }
    
        remove(n: number): null | T {
            if (this.slots[n] === null) return  null;
            const v = this.slots[n];
            this.slots[n] = null;
            this.unused.push(n);
            return v;
        }
    
        * values(): IterableIterator<T> {
            for (const v of this.slots) {
                if (v !== null) yield v;
            }
        }
    }
    
  9. The entry point for any extension will be in src/extension.ts, where you will start like so:
    const Me = imports
    	.misc
    	.extensionUtils
    	.getCurrentExtension();
    
    import * as arena from 'arena';
    
    const { Arena } = arena;
    
    // @ts-ignore
    function init() {}
    
    // @ts-ignore
    function enable() {
    	let arena = new Arena<string>();
    	
    	let id1 = arena.insert("hello");
    	let id2 = arena.insert("world");
    	
    	global.log(`${arena.get(id1)} ${arena.get(id2)}`);
    }
    
    // @ts-ignore
    function disable() {}
    

I’m sure that with some additional work, we could make it significantly easier to get started with developing GNOME extensions in TypeScript, potentially even with a template to bootstrap this process, and official tooling for generating TypeScript bindings for GJS APIs.

8 Likes

Thanks for this, I think it could be a great idea to lower the entry-barrier for extension authors. Can you provide an example of the transpiled JavaScript for reference?

When we review extensions, type errors and syntax errors are not really the primary concern. What we are concerned with is malicious or unstable code being distributed from extensions.gnome.org, and if the resulting code is obfuscated that will get harder.

While GJS is executed in a JavaScript engine it’s ultimately running in a GLib environment, so it’s entirely possible to write invalid, leaky or unsafe code that is valid, well-typed JavaScript.

2 Likes

It’s worth hanging out with us here on matrix - https://gnome.element.io/#/room/#extensions:gnome.org - to discuss further. We have a number of extension writers, reviewers, and others. You can also join from irc as well.

In general, I’m worried about sustainability. Secondly, I think if we want gnome shell developers to accept official bindings then the extensions community as a whole should come up with the proposal and support amongst ourselves. What I don’t want is to have the GNOME Shell developers on the hook for adding bindings they might be forced to support.

So my advice here is to join with us and lets hammer out something that looks sustainable that doesn’t include gnome shell developers in the mix and then propose something that can be sustainable by a larger number of extension writers - I think that would be looked on more favorably.

Thanks for bringing this forward, I like what you are selling but a sustainable plan is desired. I think we have some leeway to experiment since we can’t get any worse than where we are today. :slight_smile:

4 Likes

There are configuration options for the generated JavaScript, so how it looks depends on what settings you defined for the formatter. I use a pretty-printer that strips comments for Pop Shell’s generated sources, so it’s easier to debug issues since everything’s universally formatted the same, and extra developer-only stuff isn’t included.

This would require development on GNOME Shell, so GNOME Shell developers would need to be involved, since the source maps are generated from source code. While you could manually create declarations to provide TypeScript details for JavaScript sources, the information could never be trusted. It would need the developer to update it as they make changes to the code, and would be virtually worthless if the code itself can’t live up to the standards of type assumptions in the declarations. There’s a lot of room for human error in that approach.

The human that maintains a TypeScript declaration would more than likely not be the same people writing the code, so they would have a very narrow and limited understanding of the code they’re trying to document. The other problem is the same as the C vs Rust debate: we can’t trust anything written in JavaScript to be held up to the same discipline as a static analysis engine that enforces the code to adhere to its constraints.

Part of the problem with extension development right now is that GNOME Shell’s API is very fickle. It’s easy to trigger internal JavaScript exceptions because of simple human errors in GNOME Shell. Many of these would actually be fixed by TypeScript because the compiler wouldn’t allow you to make those mistakes. For example, if a field in a class is optional or nullable, TypeScript would require you to check for its existence before you can call it. Sometimes GNOME Shell’s JS API tries to access fields that don’t exist, and we get an exception that crashes the entire desktop (back to the login screen if in Wayland), or certain features start behaving weirdly (like applications remaining in the dash after they’ve been closed).

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.