GStreamer Conf: The road ahead

The development cycle for GStreamer 1.0 took longer than many (including some in the project itself) had originally anticipated. A big part of the reason was the GStreamer team's desire to deliver a stable and well-rounded 1.0 release — but that does not mean that the 1.0 milestone designated a "completed" product with no room for improvement. Several sessions at the 2012 GStreamer Conference in San Diego explored what is yet to come for the multimedia framework, including technical improvements and secure playback for untrusted content.

Shiny new features

GStreamer release manager Tim-Philipp Müller gave the annual status report, which both recapped the previous year's developments and set the stage for what lies ahead. Several new content formats need support, including the various flavors of Digital Video Broadcasting (DVB) television (all of which are based on MPEG-2), the new streaming standard Dynamic Adaptive Streaming over HTTP (MPEG-DASH) in both server and client implementations, and the 3D Multiview Video Coding (MVC) format. There is also room for improving the MPEG Transport Stream (MPEG TS) demultiplexer, where Müller said "lots of stuff is happening", to the point where it can be confusing to follow. GStreamer also still lacks support for playlists, which is a very common feature that ends up being re-implemented by applications.

In addition to the media formats, there are several additional subtitle formats that the framework needs to support. But subtitle support requires extension in other areas as well, such as porting all of the subtitle elements to the new overlay compositor API, which allows an application to offload compositing to the video hardware. A related feature request is for a way to overlay subtitles at the native resolution of the display hardware, rather than at the video content's resolution. The two resolutions can be different, and to be readable subtitles should be rendered at the sharpness provided by the display resolution. The project also wants to expose more control over subtitle rendering options to the application, again to provide smarter choices and clearer rendering.

Hardware accelerated rendering has taken major steps forward in recent releases, but it, too, has room for improvement. Müller mentioned NVIDIA's Video Decode and Presentation API for Unix (VDPAU) as needing work, and said the libva plugin that implements Video Acceleration API (VA-API) support needed to be moved to the "good" plugin module and be used for playing back more content. He also said more work was required on GStreamer's OpenGL support. Although OpenGL output is possible, a lot could be done to improve it and make integration more natural. For starters, all OpenGL-based GStreamer elements must currently be routed through special glupload or gldownload elements; being able to directly connect OpenGL elements to other video elements would simplify coding for application developers. Second, OpenGL coding is easier when operations remain in a single thread, which conflicts with GStreamer's heavy use of multi-threading. There is a long list of other proposed OpenGL improvements, including numerous changes to the OpenGL structures.

Refactoring, streamlining, and such

The project is also intent on playing better with other device form factors, including set-top boxes and in-vehicle systems. In some cases, there is already outside work that simply needs to be tracked more closely — for example, the MeeGo project's in-vehicle infotainment (IVI) platform wrote its own metadata extraction plugin that reportedly has excellent performance, but has not been merged into GStreamer. In other cases, the project will need to implement entirely new features, such as Digital Living Network Alliance (DLNA) functionality and other "smart TV" standards from the consumer electronics industry.

GStreamer developers have plenty of room to improve the framework's day-to-day functionality, too. Müller noted that the new GStreamer 1.0 API introduces reworked memory management features (to reduce overhead by cutting down costly buffer copy operations), but that many plugins still need optimization in order to fully take advantage of the improvements. It is also possible that the project could speed up the process of probing elements for their features by removing the current GstPropertyProbe and relying on D-Bus discovery. There is also room for improvement in stream switching. As he explained, you certainly do not want to decode all eight audio tracks of a DVD video when you are only listening to one of them, but when users switch audio tracks, they expect the new one to start playing immediately and without hiccups.

Some refactoring work may take place as well. A big target is the gstreamer-plugins-bad module, which is huge in comparison to the gstreamer-plugins-good and gstreamer-plugins-ugly. Historically, the "good" module includes plugins that are high-quality and freely redistributable, the "ugly" module includes plugins with distributability problems, and the "bad" module contains everything else not up to par. But plugins can end up in -bad for a variety of reasons, he said — some because they do not work well, others because they are missing documentation or are simply in development. Splitting the module up (perhaps adding a gstreamer-plugins-staging) would simplify maintenance. The project is also considering moving its Bluetooth plugins out of the BlueZ code base and into GStreamer itself, again for maintenance reasons. Post-1.0 development will also allow the project to push forward on some of its add-on layers, such as the GStreamer Streaming Server (GSS) and GStreamer Editing Services (GES) libraries.

Finally, there have been several improvements to the GStreamer developer tools in the past year, including the just-launched SDK and Rene Stadler's log visualizer. Collecting those utilities into some sort of "GStreamer tools" package could make life easier for developers. The project is committed to accelerating its development cycle for the same reason: faster releases mean improvements get pushed out to application authors sooner, and code does not stagnate relying on old releases. Müller announced that the project was switching to a more mainstream N.odd unstable, N.even stable numbering scheme, with the addendum that the framework will stick to 1.x numbers until there is the need to make an ABI break for 2.0.

Sandboxed streams

On a different note, Guillaume Emont presented a session about his ongoing experiments with sandboxing GStreamer media playback. The principal use case is playing web-delivered content inside a browser. The Internet may have been invented to watch videos of cute animals, he said, but that does not mean you should trust arbitrary data found online. In particular, untrusted data is dangerous when used in combination with complex pieces of software like media decoders.

For media playback, the security risk stems from the fact that although the decoder itself should not be considered evil and untrustworthy (as one might regard a Java applet), the process becomes untrustworthy when it must handle untrusted data. Thus, GStreamer should be able to use the same decoder plugin on untrusted and trusted content, but when handling untrusted content the framework must have a way to initialize the player, then drop its privilege level to isolate it.

Emont's work with sandboxing GStreamer playback started with setuid-sandbox, a standalone version of the sandbox from Google's Chromium browser. Setuid-sandbox creates a separate PID namespace and chroot for the sandboxed process. Although it is not very fine-grained, Emont thought it a good place to start and produced a working implementation of a sandboxed GStreamer playback pipeline.

The pipeline takes the downloaded content as usual and writes it to a file descriptor sink (fdsink element). When the fdsink element reaches the READY state, it is opened in an fdsrc element inside a setuid-sandbox, where it is then decoded, loaded into a GStreamer buffer, and loaded into a shmsink shared memory sink. The shmsink is the last stage in the sandboxed process; outside the sandbox, the pipeline accesses the shared memory and plays back the contents within. This design sandboxes the demultiplexing and decoding steps in the pipeline, which Emont said were the most likely to contain exploitable bugs.

The playback pipeline worked, he said, but there were several issues. First, he discovered that many GStreamer elements do not acquire all of their resources by the time they reach the READY state, though they do by the time they reach the PAUSED state that follows. It might be possible to modify these elements to get their resources earlier, he said, or to add an ALL_RESOURCES_ACQUIRED signal. Next, he noted that the memory created by the shmsink inside the sandbox could not be cleaned up by the sandboxed process, but only by the "broker" portion of the pipeline outside the sandbox. A more noticeable problem was that sandboxing the decoder made it impossible to seek within the file. Finally, the sandboxing process as a whole adds significant overhead; Emont reported that a 720p Theora video would consume 30-40% of the CPU inside the sandboxed pipeline, compared to 20-30% under normal circumstances.

Some of the problems (such as the READY/PAUSED state issue and the lack of seekability) might be solvable by sandboxing the entire pipeline, he said, or by adding proxy elements to allow for remote pipeline control. Either way, going forward there is still a lot of work to do.

It is also possible that setuid-sandbox is simply not the best sandboxing solution. There are others that Emont said he was interested in trying out for comparison. He outlined the options and their various pros and cons. Seccomp, for example, is even less flexible, which probably makes it a poor replacement. On the other hand, seccomp's new mode that combines with Berkeley Packet Filters (BPF) provides a much greater degree of control. It also has the advantage of being usable without end-user intervention. SELinux, in contrast, could be used to define a strict playback policy, but it is under the control of the machine's administrators. GStreamer and application developers could make suggestions for users, but ultimately SELinux is not under the developers' control. Finally, Emont did his experiments on Linux, but in the long term GStreamer really needs a sandboxing framework that is cross-platform, and perhaps provides some sort of fallback mechanism between different sandboxing options.

Emont's work is still experimental, and more to the point he is not conducting it as part of GStreamer's core development. But he did make a good case for its eventual inclusion. Certainly any part of a large framework like GStreamer has bugs and therefore the potential to be exploited by an attacker. But isolating the un-decoded media payload from the rest of the system already goes a long way toward protecting the user. As did Müller's talk, Emont's presentation shows that GStreamer may reach 1.0 soon, but it is still far from "complete."