Igalia's Brian Kardell and Nikolas Zimmermann talk about the history of SVG and Web browsers, and how WPE and embedded systems are bringing new developments
Brian Kardell: Ok, so… I'm Brian Kardell from Igalia and I'm here with my coworker
Nikolas Zimmermann: Yeah hi, this is Nilolas Zimmermann from Igalia
Brian Kardell: ...and we are gonna talk about a whole bunch of fun stuff today: SVG, WebKit, embedded browsers and standards and history -- and it's because we plug into a lot of that. But, I'd kind of start in a little bit of a weird place because the first time I met you I learned this really interesting origin story which actually ties into it really well. And I think it is also just super compelling and interesting. So, let's go way back and talk about origins. How did you get into programming?
Nikolas Zimmermann: Well, I got into programming quite early. I think at the age of 10? So my mom owned a bookstore, and from the previous owner she got a computer. But, It was not a computer as you think of it in today's terms - it was basically a machine that you would plug into a TV, and it had a runtime BASIC interpreter. So, when you started it up it just told you" 100 READY, listening". There was a manual in German and I had a lot of free time, so I started to read the manual and try lots of things… and that's how I got into programming. Hard drives, at the time this was made weren't really a thing: Not at all. So if you wanted to store your stuff you would need to plug in a cassette recorder to save your work, and so it was really cumbersome.
Brian Kardell: I also started with a TI-99/4a. That was a computer that was just like that. You would plug it into your TV kind of a… well, not even coax cable, if you can believe it. There was like, kind of a -- you would kind of screw into your antenna a coax adapter. And they also had a cassette recorder and everything, and that's also how I started learning BASIC. But,there's a really interesting spin to this: People who don't know us won't catch this -- but, we're actually quite different ages. And so, for me that was in like, 1981 - or roughly something about like that - but I think you weren't even born then?
Nikolas Zimmermann: (chuckles) that's correct.
Brian Kardell: And so, I think when you started there were considerably more advanced machines available - there were like Windows and Macs and PCs and even the Web existed, I think when you were -- like, ten, right?
Nikolas Zimmermann: That's right. And I feel really fortunate to have had this, let's say, old school education by accident. It really helps to learn things from really the basic concepts onwards.
Brian Kardell: But then you shifted and you started using those more modern computers. Like, what happened?
Nikolas Zimmermann: Right, so well… At some point my father discovered that there was something called Q-Basic available on his machine. Back then I think he had a DOS machine with windows 3, or something? I discovered this and it was really perfect since I could finally save my work and I could work on multiple projects and toy around. My father, back then, was a tax accountant. And I managed to crash his machine two times, basically erasing everything so he had to redo a lot of work for his clients - which took him months, I think, since he wasn't used to backups.
Brian Kardell: (laughter)
Nikolas Zimmermann: But this finally gave me my own machine, and from then on I really got into this: Computing and Linux/Free software.
Brian Kardell: Wow. Linux is really interesting. Like, how did you…. You were very young - I mean... How did you even learn about Linux? Because… I don't know how many people today would realize this, but at the time, Linux was considerably more obscure than it is today.
Nikolas Zimmermann: Yeah, so in the first year or so I think, I programmed exclusively on Windows - like using visual basic stuff like this. But, since my mother owned a store where there were lots of computer magazines and everything: I had access, free access, to all of this and basically read something line 10 magazines a week, and there were a lot of topics regarding Linux at this time - magazines covering...featuring Linux. This really hooked me as it was an opportunity to learn about the details, or inner workings of an operating system by simply looking at the source code.
Brian Kardell: Ok… so… like I said: The web already existed at this point. But how we do and think about all of this has dramatically evolved.
Nikolas Zimmermann: Right.
Brian Kardell: When Tim conceived of the Web, Windows and Mac we're already - like advanced. There were hypermedia systems and almost browsers kind of exhausted already in a lot of ways. But all of this was very proprietary and a shrink wrap thing. I have this great image of Tim Berners-Lee, like, literally trying to give away the idea of a browser to Ian Richie from OWL software. So, the Web was almost born as a shrink-wrapped product. But, because he had to do it himself. Linux was, at that time, very fringe - but there's this new idea - relatively new idea - of free software… It is not really the same as open source, but we can come back to that… But, this is where you come into this really - and you're at the ripe old age of... 13?
Nikolas Zimmermann: That's correct, yes. When I started using Linux I cannot even remember what distribution it was, but it came pre-installed with the KDE desktop environment. So, I found that a really interesting and cool project and I immediately wanted to contribute. So I first started by helping translate stuff from English to German. And I started to hang around on IRC and talk to other people in this community. Due to my background, I was quite confident with BASIC programming but I wanted to learn C and C++ to help contribute to KDE. So that's what I did, and well… I was quite successful in the end with the help of many people in the community.
Brian Kardell: I just love this story because it's just such a cool story of communities and history and everything.. So, you're in your early teens or something and you get a homework assignment, right? And you think "I would really like to include some diagrams in this homework assignment"
Nikolas Zimmermann: Right (chuckles)..That's correct. I was given a homework assignment and I wanted to put in some flow charts. And, of course, I was a Linux user at this time and I wanted to use some existing programs. One of these was Kavio - which was a program included in the office suite for KDE. It was really a nice piece of software, but the problem was you would need some stencils to draw these diagrams. So I was looking at where could I get some free stencils - and the answer was there are none for office, for Kivio, but there are some in an alternative Gnome project called Dia.. They had a huge set of stencils and they were all written in SVG - scalable vector graphics, which was new to me at this time. So that's how I learned about scalable vector graphics.
Brian Kardell: In 1999, SVG was like really new.
Nikolas Zimmermann: That's correct, yeah. In order to fulfill my homework assignment, I thought "Ok, I need to write an import filter for SVG" so that Kavio can read and parse these SVGs and I can finally finish my homework. So this was the basic idea: Add a simple subset of SVG to Kavio to be able to render these stencils. That was the beginning.
Brian Kardell: Ok, so... Here's my question: Did you ship the homework? With the diagrams, I mean?
Nikolas Zimmermann: (chuckles) uhhhhhhhhhhhhh… You know, I think I did. But, honestly I only put in as much as I needed to render the first basic stencils so I could finish it - finish it - finish the homework… (chuckles)
Brian Kardell: A few months go by and you actually find someone else who is interested, who has similar interest in SVG…
Nikolas Zimmermann: That's correct! That's my good old friend Rob Buis. He is also interested in SVG, but from a different context. He was writing an import filter for SVGs for a vector graphics drawing program called Karbon - it used to be KIllustrator back then. So I was working on this import filter for Kavio and we got together on IRC and talked about it, and we thought maybe we should join forces and write one common library that we could reuse within the KDE project for all of the programs that want to import and render SVGs. So, we got together and created a library called lib-svg -- which was renamed, I think only a few days later to just "ksvg". And that's how it all started.
Brian Kardell: Ok, so… you're about 14 and Rob is in his twenties. You're both working on this kind of part-time. You're geographically distributed - which is possible because of the internet. And you start merging your code bases - parsing SVG, stuff like that. Then… How long does it take you to extract the common stuff, so that your existing filters and stuff can work?
Nikolas Zimmermann: I think it was... Maybe a few weeks? I cannot I cannot remember, it's almost two decades ago,the actual time frame - but I think it was a few weeks only. Then we had a common base for our Kavio and Karbon import filters.
Brian Kardell: Ok, so then - you have this common library -- like what's the idea? Like.. What can you do now that you have this common library?
Nikolas Zimmermann: Actually, we wanted to use SVG not only in KOffice, but also in this browser called Konqueror back then. So, we researched: How could we use this ksvg to render KSVGs within the Konqueror browser? After a while we found there was this software project called KParts and you could use it to write a so-called component and embed it into other KDE projects. So by simply asking for a plugin that can render SVG+xml mime type, you could embed the library into a running program and use it to display content. So you could simply give it a URL and here is an SVG and please render it in this kind of canvas. And this week we both thought it was interesting so we got together, and we implemented this.
Brian Kardell: okay so - you and Rob Buis inevitably wind up putting svg's in the browser for basically the first time - right? I mean - that was pretty novel at the time…
Nikolas Zimmermann: Right but I think we didn't really notice at this time how significant this was. The only thing was to regard Konqueror as a quick testing shell for our SVG work so that we could easily open SVGs - zoom and pan them, and simply have a look at them.
Brian Kardell: Yeah, that's so interesting that so many things wind up being - like so much more widely useful than you originally imagined…
Nikolas Zimmermann: Right…
Brian Kardell: So, at this time - I think these were basically... like… whole documents, right? Like you could open a whole SVG with a URL - that's what you could do. I mean… giving them a URL is also just a really interesting idea and being able to open them in your browser - but I don't think -- you couldn't embed them, right?
Nikolas Zimmermann: Right… well… you could embed them in theory in an object - using an HTML object tag, since the the browser would simply sniff them from the MIME type from the KParts framework and then ask for a handler for this kind of MIME type… but there was no integration between the host document and the embedded document. That was not possible, technically.
Brian Kardell: The web is actually in a very weird time here because W3C had sort of decided the HTML, as we think about it, was - like... done. That was just the first pancake, but it probably wasn't suitable to the needs of all these other things that we imagined we really needed.. and so SVG kind of came from that world, right? And so does MathML.
Nikolas Zimmermann: Right, exactly. So we spent really the first one and a half or two years of ksvg in implementing this SVG 1.1 specification. stand-alone. So, really we implemented all the features that SVG needed like gradients, patterns, clipping, masking and stuff like this - . Also JavaScripting was a big part - manipulating the DOM and CSS and so on... But we quickly realized - I think in 2003 - It was clear that we would never be able to support - like compound documents, which is really interesting thing: like being able to to mix in line SVG into HTML documents using CSS stylesheets in one place to really style the whole compound document and shared event handling, etc. It is clear that this is not possible with the solution that we are working on.
Brian Kardell: And so then… there's this really wild turn in when history - about like, almost exactly halfway between the creation and us recording this - the Web 2.0 is sort of here, and it's defined by developers, and not the W3C. There's this split within the W3C and there's this special Workshop called that is like "what do we do with this?!". And there's a proposal that maybe we should do another HTML with like.. the web that we have. And so in 2004 there's this break away and the WHAT working group is created. It's Apple and Mozilla and Opera - and by 2006 the first spec arrives and it's… the HTML Parser. (Giggling astonishingly) I feel like this is a story that doesn't get old enough... So…. we're halfway and the web basically doesn't have tests - the W3C has no certification mechanism. It's not the same as some other bodies. But, so, we get HTML5 and web platform tests as a thing, and they've decided they are going to pull over a whole bunch of these interesting ideas. And, two languages kind of come over in whole - like they're just like "yeah we're just going to put that right in… You can embed them." - and they're in the HTML specification SVG and MathML.
Nikolas Zimmermann: Right. I know that back in 2004 I was really in the XML camp -- like many of the people -- so for me the future was XHTML, XQuery, XForms and stuff like this. All these XML derived dialects. Until this point, I had not even been thinking about the semantic web or HTML5 -- for me, the main concern was really prototyping what is possible. Like: how can we embed SVG in HTML. And it seemed really simple to me - in the beginning, before I tried to implement it. And then, after a while working on it, both Rob and me realized that the whole situation was completely under-specified. So, there's no specification which tells us how to really integrate SVG or MathML really into an HTML document. Like, there's so many details to consider. CSS Has this box model, layout systems - SVG works completely differently. How do transformed SVG elements behave in HTML? How does event handling, text selection -- all the things you would expect them to work: how do they actually work in detail. This was uncharted territory. So when we tested all these things - we simply opted for a solution. We simply tried something. And then we'd see: How far we can come with this solution? But this tight integration between SVG and CSS and HTML is really difficult if you have many, many degrees of freedom and no specification enlightening you how to do it properly .
Brian Kardell: There's this weird evolutionary thing - where the thing that we have is probably not the thing that an engineer would sit down and be like "This is the good design. this is the one that's going to make it."
Nikolas Zimmermann: (laughing) Right.
Brian Kardell: It's strange in retrospect how technically better things often don't win.
Nikolas Zimmermann: Right. I didn't really think about the HTML parser so much back then since SVG was really a relatively new language dialect. So, how to parse an SVG document was really straightforward - and there was really no content in the wild. Like, when you build an HTML engine you have to deal with a lot of quirks - a lot of existing content which you simply have to deal with - so there's lots of legacy and history. And, SVG didn't have any of that, so that's why I was not very concerned about all these things - simply because SVG was a new language.
Brian Kardell: I think everybody was like on board the XML train initially. It's really interesting to see, in retrospect, how much that turned for practical reasons - and how there were those things that were like... strangely hard to appreciate somehow. Ok, so - the next few years but what happens?
Nikolas Zimmermann: When I joined university in 2005, I began to study physics - not computer science as many people would think, simply because this is a personal passion of mine and I was always interested in it - especially in particle physics. Besides this I kept working on the open-source stuff: working and contributing to KSVG - which was in 2006 merged into WebKit. And that's how I joined the WebKit project - simply because there was a person called Eric Seidel who recognized the importance of KSVG - and he worked for Apple back then. He imported all of the libraries that we invented within the KDE project. So this was: kdom, ksvg2, khtml2 and kcanvas into WebKit, because WebKit was missing an SVG implementation and we simply filled the gap with our implementation. Of course, for Rob and me it was exciting since there were so many people in the WebKit project and we were used to working on our own - and the pace of progress within WebKit was tremendous. So, we simply thought "okay that's a perfect time to join this project in and work on our SVG stuff there. It has a promising future and we want to participate in this." So, this went on for maybe 5 years, while I was working… or, maybe 4 years when I was working in my spare time on all of this and … At some point I joined a company called Torch Mobile - also founded by a friend of mine who was in the KDE project - George Staikos... and so I was working there, upstream on WebKit and also some internal porting of WebKit to some embedded devices. Torch mobile sold this IRIS browser. Then in 2009, I think it was, Torch Mobile was acquired by Research in Motion - so, the makers of the Blackberry Phone. Since then I used to work for Blackberry, also upstream on SVG - and I think this continued until 2012? Then I really needed to graduate so I needed to concentrate on my studies, and then I had to stop contributing to WebKit for a while.
Brian Kardell: ...And then, very recently you graduated with your doctorate, right?
Nikolas Zimmermann: Really recently, yes.
Brian Kardell: Congratulations. But also really recently, you came back and you came to work at Igalia, where Rob Buis also works - and so we have like: The original team ksvg! And also interestingly: Just like when you worked for Torch and you were working on this idea of embedding WebKit - the project that you are working on here is embedded. Because, here at Igalia we work on WPE, which is the official WebKit port for embedded. If you go to webkit.org/downloads - it's right there. Maybe you can explain to people: Why would you want to embed a web browser?
Nikolas Zimmermann: Well… People wouldn't believe how many devices that they have at home where there is actually a WebKit Web browser running on it. There are many, many billions of devices out there which run WebKit. So, there are for instance, all the PlayStation consoles. There are many, many set top boxes, phones, smart Home systems, cooking stations in your kitchen - many of them which also show some kind of user interface a nowadays are using a web browser since it's much more convenient for the companies that design these hardware products to use the standard web browser for the user interface. And then they have many creative designers at hand which can build their user interfaces. So, you don't need a special developer who knows a certain commercial product which was used in the past for building user interfaces. No, you simply use the web.
Brian Kardell: For me at least, this was a little bit.. Funny? Because we have WebKit, which I think most people think about as like.. Apple and Safari. I think for most people when you think about a browser you think like… the button for the internet. We know that there are rendering engines and projects that are embedded in - like Electron, for example, to make distributable applications that also use the same rendering engine, but… So, help explain like… What are the important things to take away about WPE?
Nikolas Zimmermann: So… WPE is one of the WebKit ports, specifically designed to be optimized for embedded devices. So what is a WebKit port? WebKit itself is a project that consists of several components. One of them is WebCore which implements the actual browser engine. So it's responsible for parsing and interpreting CSS, building the DOM tree, etc. But then, in order to, let's say paint something on the screen on your device or react to input events - such as keyboard touch events Etc - you need to call out to the actual operating system or the underlying windowing system. And so, each of the ports in WebKit is tied to a certain architecture. For instance the WebKit GTK port uses the GTK toolkit on Linux desktop environments to handle the drawing and the event handling. There are other ports, such as the MacOS port which uses the MacOS APIs to achieve the same goal. But for embedded devices this might not be a suitable solution since you do not want to be tied to a certain UI toolkit or a certain mechanism for painting, event handling, etc. So, WPE is basically the bare minimum of a port which is highly customizable so that you can adapt WebKit to your custom needs.
Brian Kardell: think that's really awesome, and it's just growing and growing. Our WebKit port has been around for a long time and it's in really a lot of those devices. We are kind of close partners on WebKit and, like last year we were the number to contribute to WebKit: I think 11% of all the commits to WebKit were from Igalians.
Nikolas Zimmermann: Yeah, quite impressive.
Brian Kardell: Really interesting in this: Part of how we met was that you had this idea… and there was this sort of legacy thing that you sort of left sitting there when you had to leave for college, that feels like kind of a gap in web standards/in browser implementations. You wrote a post on this but, can you maybe...
Nikolas Zimmermann: Yes… I think the bottom line is that we have to finally unify these coevolutionary branches of HTML and SVG. What I mean by this is that when SVG started, it had transformations, from the beginning - so that you can add any kind of affine transformation to an element, rotate it, scale it and so on. And all of these features came to HTML or CSS, but much later. CSS transforms were born much later. By much later, I mean long after SVG was published and implemented in the browsers, especially in WebKit. So in WebKit there was always some special rendering path for SVG which had all of these fancy things - stuff like transforms, clipping, masking and stuff which was not available in HTML - at least not to the same extent as in SVG. But then, HTML, CSS and all these specifications - they caught up. CSS got all these features: CSS Transformations, masking, composition, 3d transforms. And the implementation within WebKit is completely separated for SVG and HTML and CSS. This is a bad thing because it leads to interoperability issues. Many of these issues within SVG 2.0, by like, mapping certain SVG special attributes - like the transform attribute - to a CSS property. This was finally specified: which takes precedence? If you specify both an attribute and a property in CSS property? And so, when I was absent -- when I returned it was really happy to find out that all of these specification issues -- well -- not all, but many of them are solved and now. And so now it's the right time to unify these rendering engines, finally.
Brian Kardell: CSS is sort of, I think, in the core triad that people think about when they think about the web and as a result CSS. As a result, CSS got all these critical boosts and all these investments.
Nikolas Zimmermann: Exactly.
Brian Kardell: One of the boosts it got was hardware acceleration. SVG 2.0, by mapping those things to CSS, provides a path toward hardware acceleration in SVGs.
Nikolas Zimmermann: Correct. In WebKit-based browsers, say, for example, when you apply a transformation on a div element and rotate it on the screen, in WebKit, you can offload all of these to the GPU, so the composition of all of the graphics layers happens only on the GPU. So, there's no need for the CPU to be involved in this and performance when offloading all of this work to the GPU is much higher and you get fluent animations and a good user experience. For SVG, none of this is currently hardware accelerated. So this leads to severe degradations when applying animations, and you don't really get a fluent response, which people expect nowadays. So, the clear path forward for WebKit is to move SVG to the same rendering quality that HTML and CSS already offers.
Brian Kardell: ...but... one thing that I think it's worth maybe being clear about is that the lack of ability to hardware accelerate things isn't exclusive to WebKit.
Nikolas Zimmermann: Right - I mean, Chromium has the same legacy, so it has the same origin. So, blink forked WebKit and to the best of my knowledge the SVG implementation basically works the same as for WebKit, still. So it's not hardware accelerated. The blink guys, they have a clear plan toward hardware acceleration as well, but, currently it's just a plan, so it's not available yet.
Brian Kardell: Because tho: this is even more important on embedded devices, right?
Nikolas Zimmermann: Right… On embedded devices… currently many of the embedded devices, they have, let's say "under powered CPUs" - but they have a GPU. So when you want to build you the interfaces that are really nicely looking on let's say your cooking machine, you want to use SVG. But, at the moment it would be slow. The animations, when you make a nicely animated user interface - it looks nice on your desktop machine, but when you run it on an embedded device… It looks, well… stuttering, the animations not fluent and it's not something you want to sell to customers.
Brian Kardell: It's a little bit central to Igalia's whole ethos, in a way… which is...what we're talking about here is sort of the commons. And how we build out the commons has all kinds of implications. A rising tide lifts all boats, right?
Nikolas Zimmermann: Yes, and most importantly we can resolve much of the frustration that currently developers are faced with since they expect things to be the same for SVG as in CSS. We have to unify and resolve these issues .
Brian Kardell: Yeah… Sarah Drasner, she's been on for a long time about this: "can we not just hardware accelerate these, please?". And, for whatever reason, it's been just really hard to prioritize the investment in these things, right? So, it's interesting that here the first mover is driven by a group that isn't Google or Apple - or sort of the traditional pressures that you think.
Nikolas Zimmermann: Yes, I'm very thankful for Igalia who allowed me to prototype all this unification and stuff in WebKit, since it involves a lot of research. Since the implementation is so divergent internally. It's really two completely different implementations: How SVG works and how the rest works. It was like this from the beginning, but from a different perspective. In the beginning HTML was lacking all these fancy stuff that SVG needs, and now it's the other way around. HTML and CSS offer way more way more, like 3D Transformations, stuff that SVG traditionally never supported -- and now to unify them you basically need to rewrite the whole SVG rendering engine, which is a huge task. And it's not clear from the beginning how can this be split into smaller atomic chunks. And it's also not clear how to integrate HTML and CSS and SVG internally in terms of the implementation - so it needs a lot of research into how to deal with all these different requirements that SVG still has. But, finally, I think we are on a good way in 2020 to have an initial patch for this discussion, and I'm thankful that Igalia allowed me to work on this.
Brian Kardell: We can include the link to your animated tiger demo which is really cool and I think really exciting and it's been shared around a bunch since you wrote it. We've had feedback from a lot of different venues that this is really interesting and really exciting for a lot of people. Our work, because it is being funded by embedded, then, also everybody gets tha advantage: It's not just embedded WebKit that's going to get that.
Nikolas Zimmermann: Exactly.
Brian Kardell: The, that also plays into this kind of like a first-mover pressure that exists in Web standards. Because WebKit then has a competitive advantage because we hardware accelerate, that sort of lights a little fire under other people to do it as well. And, this is only like one illustration. It's a nice one because it's like - very tied to WebKit and things that we're talking about, but there are other things where these pressures are similar, and different people participating at the implementation and prioritization level, that we at Igalia enable. That can be really healthy for the ecosystem. I think this was like really, really interesting.. Full of interesting history about the web and WebKit, and your work. It is really exciting, the stuff that I think you're talking about doing with SVG. I learned a lot actually about SVG history as well myself. Are there any other closing thoughts that you wanted to share?
Nikolas Zimmermann: Well, I will just say: Stay tuned for 2020, hopefully this will be the year of hardware accelerated SVG. Well… I will do my best to try to get this into WebKit and open the discussion on how to get it into other browsers.
Brian Kardell: awesome... thanks a lot for sitting and talking with me.