Flowgraphs in GTK+

At GUADEC 2016 in Karlsruhe, Germany, Daniel "grindhold" Brendle presented his work developing a new library and widget set that will allow GTK+ applications to implement flowgraphs in a standard manner. The widget set would enable applications to provide interactive widgets for linking filters and other block-oriented components—a type of interface many applications currently need to reinvent on their own.

Flowgraphs, Brendle explained, are a general-purpose diagramming technique that many people will recognize from textbooks and other printed matter. They show how objects, information, and signals flow through some sort of process. Biology textbooks use them to illustrate circulation in the body, technical manuals use them to show how a manufacturing process runs, and so on. In software, he said, they are most familiar as the node-and-pipe diagrams that illustrate signal processing or data filtering.

But every open-source project that includes a flowgraph component in its interface seems to reinvent it from scratch, resulting in considerable duplication of effort. He showed several examples: GNU Radio, which uses flowgraphs to connect audio-processing components, Conduit, which uses them to link data sources to data sinks for synchronization, and the computer-aided design (CAD) program Antimony, which uses them to model the construction of objects.

The most familiar implementation, he said, is Blender's node editor, so he took an in-depth look at how it is implemented and what features it offers. The node editor lets Blender users construct a rendering pipeline, specifying various lighting, shading, and reflection filters that are executed on the objects in a rendering job. He identified six key features of Blender's node editor that he wanted to define as the basis for a GTK+ flowgraph library:

1. Each node has input points and output points that can be connected to link nodes together (in a directed acyclic graph).
2. The I/O points and pipe connections are displayed in different colors to indicate the data type used.
3. The connections between nodes are created by the user on the canvas.
4. Each node can optionally display editable parameters for the user to adjust.
5. The program's UI provides warnings for various problem states (such as trying to connect type-incompatible I/O points or creating circular connections between nodes).
6. Special null values are supported for parameters.

Brendle decided to implement his own flowgraph library, since an examination of the existing options revealed that most of them would be unsuitable for adaptation into a general-purpose tool. They are often tied directly into the logic of the application in question, he said, and there were none that hooked into other GTK+ services like the theming engine. Consequently, the flowgraph components always look different from the rest of the user interface.

His first attempt at a flowgraph implementation was a library called libgtkflow, which he started in May 2015. It is written in Vala, GNOME's high-level object-oriented language that compiles to standard C. Subsequently, Daniel Espinosa joined him in the project, reworking the build system and simplifying the API. They eventually decided that the project should be split into two libraries—one to handle the abstract flowgraph model and one to provide the onscreen widgets (although both are still hosted at the same GitHub repository as the original monolithic implementation).

The abstraction library is called GFlow. It provides classes to represent the nodes and the I/O points (which are called "sink" and "source" docks, names borrowed from the similar concepts already used in GStreamer).

The UI sub-library is called GtkFlow. At present, it provides screen-ready implementations of NodeRenderer and DockRenderer elements. The node elements can contain other GTK+ widgets to display parameters, labels, and so on. There is also a NodeView widget designed to display a complete graph. Both GFlow and GtkFlow support GObject Introspection, so they are accessible from any language with GTK+ bindings, including Python, Perl, Lua, and JavaScript.

He added that one important feature planned for future releases is to convert the current NodeRenderer and DockRenderer implementations into generic interfaces that developers can use to create their own customized nodes. For example, someone writing an Arduino or Raspberry Pi simulator might want the Node widget to visually resemble the wiring headers of those devices, rather than a generic GTK+ rectangle.

Looking to the future, Brendle discussed where the library needs further work. The code on GitHub currently runs only in GTK+ 3.20, so fixing compatibility with other recent releases is vital. The NodeView widget only supports selecting and dragging one Node at a time, which needs changing, and the pipe connectors do not yet have all of the features in the original list culled from Blender's implementation. Brendle said that the pipe connectors will likely need to become their own widget type in order to support more "intelligent" features like detecting cycles or other errors.

He also speculated that the GFlow nodes might need to support recursion (so that, for example, an entire flowgraph could be encapsulated into a node, simplifying the construction and display of multi-level graphs), and that the widget set might need to be extended to support right-to-left languages. As it stands today, input docks are hard coded to always be on the left and output docks on the right, which mirrors left-to-right writing systems.

He closed the talk by speculating on what applications might benefit from having a flowgraph implementation in GTK+. The obvious choice is media-processing programs like the video editor Pitivi and the PulseAudio utility Pavucontrol. But there are surely others, he said. "I want you to be creative and do better than me" at making something interesting with the library, he concluded.

It is hard to say when or even if libgtkflow might make it into GTK+ proper. There seemed to be interest from others in the work done by Brendle and Espinosa, though, which is a good sign. If progress continues, application developers could soon have an important new tool available to express complex interactions in their GTK+ interfaces.

[The author would like to thank the GNOME Foundation for travel assistance to attend GUADEC 2016.]