Solomon Hykes Discusses Dagger, DevOps, and Docker

Welcome to the InfoQ podcast

Daniel Bryant: Hello, it's Daniel Bryant here. Before we start today's podcast, I wanted to tell you about QCon London 2024, our flagship conference that takes place in the heart of London next April 8th to 10th. Learn about senior practitioners' experiences and explore their points of view on emerging trends and best practices across topics, like software architectures, generative AI, platform engineering, observability, and the secure software supply chain. Discover what your peers have learned, explore the techniques they're using and learn about the pitfalls to avoid. I'll be there hosting the platform engineering track. Learn more at qconlondon.com. I hope to see you there.

Hello, and welcome to the InfoQ podcast. I'm Daniel Bryant, and today I've had the pleasure of chatting with Solomon Hykes, co-creator of the Dagger Project, and of course, the original Docker Project. We touched on all things continuous integration and continuous delivery, and covered everything from why there is currently a need for new abstractions for building software, all the way through to the impact of licensing, search tooling and donating, or not, such projects to foundations like CNCF and the CDF. I've been following along with the Dagger Project for quite some time and experimented with the latest quick start just before chatting to Solomon.

I could see the connections with existing tech, such as Dockerfiles and buildpacks, and it was fun to be defining application build steps using a node-based fluent API. However, the conversation helped me join the dots for the bigger picture of where modern build tooling like Dagger fits in and how the ecosystem could develop to support a set of reusable build modules and components. Welcome to the InfoQ podcast, Solomon. I'm sure many of our listeners will know you from the dot cloud days, and of course, we're kick-starting the docker and container revolutions. For those folks that haven't bumped into you, could you introduce yourself to the listeners please?

Solomon Hykes: Sure. Hi, everyone. I'm Solomon. I'm the co-founder of a startup called Dagger. Before that, I was the co-founder of Docker. I'm an undercover French, I sound very American, but I grew up in France. Now I live in San Francisco and having a great time working on new tech and learning from this great DevOps community.

Can you define the problem space that tools like Dagger aim to address? [01:55]

Daniel Bryant: Fantastic. Much innovation going on in this space at the moment. We're definitely going to dive more into that, Solomon. I'd love to dive more into Dagger in particular. I wanted to set up the context for the listeners like, what's the problem space you are working in? I know it's CI/CD-scoped, but what's the actual problem space you're working in and why now to spin up a tool, a company around this space?

Solomon Hykes: Well, there's a personal answer and I think an industry answer. The personal answer is that I spent 10 years of my life on Docker and then I took a break because I was tired. Then we brought the band back together and wanted to build more things. We just set sail again and went out there and talked to people about their problems and honed in on the pain to try and help fix it.

Daniel Bryant: I like it.

Solomon Hykes: Discovered that a lot of the pain was still related to application delivery, build, test, deploy. We can help with that. We just took our experience and skills and transposed them into today's problems, because some things are the same. 10 years later, some things have changed. I went from personal to industry answer. It feels like a continuation of what we started, of what we set out to do. I started on this whole container thing in 2007, 2008. It is just one big arc. The arc is, it should be easy to ship your code into a live application in the cloud, but it just isn't. We're still trying to climb that mountain together as an industry. For us personally, that started with trying to build a path, Platform as a Service. That was the big hope.

Then we discovered collectively that, that had flaws. Then we pivoted to a much more modular model where the first step was let's build this foundational ecosystem around containers. We need that. Then we just spent five years doing that. Now, I think we get to build the next layer on top of that and bring back the best of the original ideas of Paz and of this new thriving modular container ecosystem. We're still trying to solve delivery of applications to the cloud. I think we may actually solve it. Actually, it's been so long I think people are used to the problems as being not solved and the experience being overall not great. We just got used to the duct tape. There are people like us and many others that are still solving it, making it better. We got to do it. There's too much duct tape.

How has the DevOps landscape evolved over the last ten years? [04:24]

Daniel Bryant: Fantastic. I was chatting to Adam Jacob from System Initiative a few weeks ago, we had him on the podcast. He was like, "If you look at the 10-year gap, we've actually got the same process but different tools." I wonder if you've got any thoughts on that as in granted, you are building a new tool and so he, to be fair as well, but do you think we're going to need to fundamentally change the way we approach the delivery of software to make these big changes?

Solomon Hykes: I think it's been a continuous change ever since this whole DevOps and cloud thing started. I think now we're steeped in it and so we forget how much things have changed already. Speaking, for example, for one little area of this huge stack, docker and containers, I mean the concept of a Dockerfile of building a container around the application. That was completely new and weird when we came up with it and we ourselves were like, "Wow, this is an interesting hack. Let's see if it goes anywhere." It was very unnatural and we had to explain it a lot and defend it. Now I talk to new grads who are starting their journey now and it's just, they don't even know how it was before.

Daniel Bryant: Yeah, makes sense.

Solomon Hykes: I think each year as new people enter the industry and we, old-timers, learn and adapt as much as we can, everything shifts. I just think that there's a lot more of that coming.

What does Dagger’s pitch of “CI/CD as code that runs anywhere” mean? [05:39]

Daniel Bryant: Dagger has pitched on the website as CI/CD as code that runs anywhere. Now, I like that pitch. It was a nice marketing pitch. Could you break it down for the engineers in the room, for the folks that want to be using this, developers, operators, platform engineers, what does that actually mean?

Solomon Hykes: Right now, our main audience is the DevOps engineer, the platform engineer. The person whose job is to make the developers on the team more productive. Over time, we think Dagger will become more and more accessible to the point where developers can use it directly. Right now, at this time, that's not really the case, unless you're a very DevOps-minded developer. The problem we solve for that designated DevOps person, whoever they may be, is that the experience of creating and maintaining these pipelines that ship the application is just a pain. The reason it's a pain is it's made of duct tape. The duct tape involves a lot of shell scripts and YAML. The duct tape straddles these two worlds of the development environment where you have to build and test and sometimes deploy from the dev environment directly, pre-push.

Then there's CI, post-push, where typically pushing and you get committal trigger actions. Things will happen, tests will run, et cetera, et cetera. That's what I call the pipelines, the delivery pipelines. Over time, those got more and more complex. As the team, the application grows, the complexity grows exponentially. Someone has to wrangle these and keep them working and improve them and keep up with the application development. For example, if the application team adds a new component, new database, new framework, new anything, new AI tool, someone has to add those into the pipeline. Otherwise, it's like saying, oh, we have this new feature in the car that we're building, but it's only available in the prototype. Therefore, it's done. Well, no, it's not done. You've got to manufacture the thing. Someone is going to have to incorporate that into the manufacturing of the software.

This is really, it's a software factory problem. It's just that as things scale and applications become bigger and they move faster, and the stakes are just generally higher, these factories have to go from little artisanal workshops, which let's face it, that's the reality of 90% of software shops to an actual factory. That transition is just painful. We're proposing a way to do that, that is not ridiculously over-engineered and actually solves specific problems. The specific problems we're focusing on are a number one, something we call push and pray. I'm changing the pipeline, I'm adding something to the CI pipeline. How do I know it works? Well, I got to commit it, I got to push it. I got to wait for the CI server to pick it up. Then I get a red thing that says, oh, you forgot a space and then I have to do it again.

You have these 40 git commits, please work. Okay, this time, I think I got it. If you think of these pipelines as software, it's like a specific software that ships your application, then that's just a terrible development experience for that software. We're telling that designated DevOps person, your pipeline should be code, it should be software. We call it pipeline is code, for example. Here is a reasonable tool to do that, an engine that will run it locally or in CI, it'll run it the same on both. You can actually develop locally and when it works locally, you can run it in your CI and there's a reasonable chance it'll work the same and it's not locked into this proprietary black box. Because that's another problem DevOps people have. There's different CI platforms and they all have a different proprietary YAML language.

If God forbid you have to migrate from one to another, or maybe you work in a larger organization and you acquire a company, now you have to deal with two. It's okay to have two different infrastructure services, two compute providers, that's fine. Two completely different platforms to express the logic of your pipelines, that's really damaging. Because now, you can't compose different pipelines together into a bigger pipeline. You have silos forming. Anyway, there's a whole list of problems, some small and subtle, others very painful and obvious that we think can be solved in a major way by changing how you think about the pipeline from a collection of shell scripts in YAML to proper codes that you can develop properly. That's the core focus for us.

What are the components of Dagger? [10:11]

Daniel Bryant: I like it, Solomon. I think definitely myself, and wearing my Jenkins hat back in the day, I recognize the pain of some of the things you mentioned there. I'm sure many of our listeners coming from that era of the Jenkins and the Rundeck, and as you've said, it's moved through different tools now. It totally makes sense. You mentioned a bunch of things there. Could we break down the components within Dagger? Because I've noticed Dagger Engine, I played around a little bit with that, and then there's Dagger Cloud and there's Dagger SDKs. Could we walk through what all those things are and what they do and when I'd use them?

Solomon Hykes: Yeah, absolutely. There's really two big parts. There's an open-source engine called a Dagger Engine. That comes with, there's a core engine, really, a core component, and then there's a command line tool that lets you interact with it. Then there are SDKs, as you mentioned, that let you program it because the whole point of this engine is to run your pipeline logic written in your code. The SDKs are really important because they're available for major programming languages that are familiar to teams already. Python, Go, JavaScript, TypeScript, and there are experimental SDKs being contributed now by the Dagger community. There's a Java-

Daniel Bryant: Interesting. I was going to ask about that in a minute, actually. Interesting.

Solomon Hykes: Java, Rust, Elixir.

Daniel Bryant: Wow, Elixir. Interesting.

Solomon Hykes: They're experimental, but they do work. The whole stack is really neat. The SDKs are generated from a common API reference.

Daniel Bryant: Interesting. Makes sense.

Solomon Hykes: The engine has in fact a GraphQL API.

Daniel Bryant: That makes sense.

Solomon Hykes: That's really just a kernel. Think of it as the operating system for your pipelines. You can have it run all these operations that are based on containers. Then these SDKs are just generated bindings in the language of your choice to do that.

Solomon Hykes: It's relatively easy to create new SDKs and it'll become easier. Anyway, that's the engine, all of that, we can just call it the engine. Then on the other side, there's Dagger Cloud, which is an optional collection of services that are run in a centralized way by us and that are designed to remove obstacles to adopting the Dagger Engine in production. What happens is you start your journey with Dagger. Designated DevOps person hears about Dagger and they realize, this could help me solve these problems that I have. Pipeline is too slow, push and pray, et cetera, et cetera. Let me go and try it out on my laptop. They develop their first pipeline. They'll go through the quick start, like you did. Then eventually, you get to something that you like that's useful. Now you want to get your team to use it. Part of that is getting them to use it in their dev environment locally. At some point, you need it to run in your CI. You can retrofit it in your existing workflow.

Daniel Bryant: Makes sense.

Solomon Hykes: Because we don't replace your existing CI, we don't replace circle CI, grid of actions. We give you something to run on top of it. That's the beauty of it. When you do that and you start running this in production on your CI, it works really easily. Operationally, you realize, there's some things I need now to hook this up to upgrade my CI stack, to Daggerize it. One of them is caching and the other is visualization of the pipeline.

Solomon Hykes: I can go into those. We just add these solutions and then you subscribe to our service and then you connect it to your engine. It's a bring your own engine solution. We don't run the pipelines for you. You run them anywhere you want.

Daniel Bryant: Makes sense.

Solomon Hykes: That's an important part of it because there're so many ways to run a CI pipeline already, so many compute providers. We think really CI compute infrastructure should converge with general compute infrastructure. The fact that it's so siloed and bundled together, a lot of these CI platforms, they bundle proprietary software for configuring your pipelines and proprietary hardware. We run your pipeline. Lately, there's been so much pressure that now reluctantly some of these providers are adding a way to self-host, but it's still a weird category of compute. We think it should just be regular compute. We really are unbundling CI as a category and splitting it into a pure software layer and infrastructure. We're clearly on the software side. I think everyone in this market is going to have to choose, what is their focus? Are they the best compute provider, the best possible machines to run your pipelines? There's a new generation of providers out there focusing on that, or are you the best software platform for it? That's what we're going for.

What does a migration path towards Dagger look like? [14:36]

Daniel Bryant: It makes sense. That makes sense. Just to double down to make sure it's super clear for the listeners, because I think a lot of folks are used to using things like Jenkins, and you mentioned GitHub Actions. They are very different, but very similar in terms of their goals. Something I heard from you there, and I also was lucky enough to watch the talk at QCon SF where the presenter, Connor [Barber], also mentioned this as well. The migration path is quite nice as well. You don't have to rip out everything in one go. You can, to your point, if I'm using Jenkins, I can literally call your binary. I could call Dagger dot or Dagger whatever, and then gradually migrate things over. Because I'm always super cautious of the big bang.

Solomon Hykes: Absolutely.

Daniel Bryant: Big bangs generally result in big bangs, which is not a good look. The migration path I think would be super interesting to listeners as well.

Solomon Hykes: You're right. That is a very important part of how Dagger works, and we were very deliberate about it. You can adopt Dagger incrementally by retrofitting it into the stack you have. That includes the CI platform you have, but also, the build tools you have, the dev environment you have. It's really designed for minimal friction. By the way, so is Docker in a different way. Two things are true at the same time. We want it to be incredibly easy and seamless to start using Dagger. You can do it in a very small way. Typically, what happens, what we're seeing is 90% of your CI is fine, and then 10% is a complete dumpster fire.

Daniel Bryant: Makes sense.

Solomon Hykes: It's slow. It's costing tons of money. There's an urgent change to it that's needed, but no one knows how. The person who built it just went nuts and then left the company. It's written in a language no one understands. It could be any reason. Typically, what happens is, you use Dagger as a fire extinguisher. You Daggerize that 10%, you replace that mess into 40 lines of perfectly clean Python or Go or type script. Then like you said, you just remove 400 lines of shell and YAML, and then you replace them with two lines in your existing configuration that say run Dagger with this thing. It's very incremental. Then hopefully, that really helps and you get the hang of it. Your dev team grows familiar with Dagger. What happens is over time, there's no big bang replatforming, like you said, those always fail. Developers aren't forced to use the new thing, oh God, what is it this time?

It's just like, oh, serendipity sets in. This is the pipeline now. I know this. This is Python. I know this. I need it. I see, this is very straightforward. It's going to run my BUILD tool at this version in this container with my source code mounted here. Now I want to use this flag. Let me go open a plot request. All of a sudden, this big black box that is the CI pipeline, that is usually a bottleneck because only the designated DevOps person understands how it works. They're always overworked. Now all of a sudden, you're starting to spread the work around. Before you know it, all these big principles of platform engineering and digital transformation that everyone's talking about, but they're hard to implement because they're implemented via big bang replatform that fails, like we said. All of a sudden, no one has to talk about it. It's just very gradually happening. That's our focus.

How do things like Dockerfiles and CNCF Buildpacks relate to Dagger? [17:47]

Daniel Bryant: I like it. I like it, Solomon. There's something called my interest there as well. You mentioned that everything is built or done within a container. I'm thinking in part of the migration, I might already have Dockerfiles or I might already have buildpacks. Personally, I love the CNCF buildpack projects coming back from Heroku days. I still long for my buildpack. Can I migrate from then? Perhaps even, what is the relationship between say, the Dagger SDK and things like the Dockerfile and a buildpack?

Solomon Hykes: The goal of Dagger is to integrate all the pieces into a pipeline. If you have a Dockerfile, the goal is that you can integrate a Docker build with that Dockerfile into the pipeline. If you have a buildpack step, the goal is that we can integrate that into the pipeline. Modularity and the ability to customize is really, that's the killer feature. It is an integration tool. Pipelines are about integration.

Daniel Bryant: Totally.

Solomon Hykes: Whoever can integrate more things better in a more robust way, wins. We think of it as replacing, maybe you can say replacing duct tape with Lego.

Solomon Hykes: It's still modular. It's still all about changing and customizing, but it's robust. It's sane and it could scale better. That's really the core of it. I could give you specific examples around buildpacks specifically or Dockerfiles. Dockerfile, that one's easy because of the tech we're using. We're building on tech that was made in Docker BuildKit, among others.

Solomon Hykes: This team basically started and was involved back at Docker. We're very familiar with it. We know where the good parts are. We get native compatibility with Dockerfiles. It's a native call, and the Dagger engine API, you just say, Dockerfile over there, give me a container. You're actually using the same implementation as Docker Builds. It's not like these Red Hat reimplementations where they try to get as close as possible, but it's hard being compatible. Dockerfile is a really solid compatibility. Then buildpacks and all these custom tools, there's a whole community of Daggernauts, we call them.

Daniel Bryant: Very nice.

Solomon Hykes: That are basically packaging them up into reusable modules.

Daniel Bryant: Interesting. I've heard about this. Adrian Mouat tweeted about this.

Solomon Hykes: It's a feature that's in development. It's actually in there in the latest release, but it's hidden because we're not quite finished. It adds cross-language reusable modules.

Daniel Bryant: Very interesting. Cross-language.

Solomon Hykes: Someone can say, I know buildpacks, I know the CNCF buildpacks tool. I'm going to package it into a set of functions that are completely encapsulated. They're running on containers. They'll run the same way anywhere. They have an actual API with types in types out, and there's a build function that takes a directory as inputs and it produces a container as outputs. These are fundamental types in the Dagger API. Then you write them in whatever language, Go, Python. Really, under the hood, it's going to orchestrate running the right buildpack tool and the right container with all the inputs and outputs hooked up. It wraps all this container machinery into an API. That API can be called from any other Dagger pipeline across languages. You can write yours in no JS. That module can be written in Python and they can call each other.

That's the missing piece for all this to make sense, because then it unlocks an actual software ecosystem where all these bits of DevOps knowledge can actually be implemented once properly and then reused, which is so obvious for an application development ecosystem. In DevOps, that's not how things work. You read a blog post that talks about another blog posts about how someone did it once, and then you find a piece of a helm chart in a Dockerfile, and then you copy paste some parts. Then you start over again. Everyone is just reinventing the wheel again and again. That's the reward of all this stuff we talked about, doing it all in code with a clean foundation, making it Lego-like. Eventually, the reward is I get to reuse the work of other people in the DevOps community.

Does Dagger use Wasm to implement cross-language calls, or is this simply calling containerized functions? [21:53]

Daniel Bryant: That's awesome. I'm not sure if this is even related, but it sounds almost WASM-like. A lot of guys doing the WASM talks at the moment, I'm reading the WASM blogs, and they're the same promise. I can write and go and rust, and we can do eventually cross-function calling, happy days. Are you doing any of that under the hood, or is this just more a case of you're running it in containers so the language doesn't really matter?

Solomon Hykes: Well, so the short answer is we're doing it all in containers. We don't use WebAssembly. I'm very familiar with what you're talking about. In fact, back at Docker around 2016 or so, things were already taking off. For a short while, we mistakenly at Docker thought, smooth sailing from here on the core platform, let's start thinking about the future at the next layer. A lot of the stuff we're doing now, 2016, '17 is when ICE shifted my focus towards trying to build an ad-docker. Really, let's make all this a platform. Let's make all this programmable. At the time, WebAssembly was an obvious choice.

Daniel Bryant: I remember your tweet now, so I know you said it, the famous tweet. If WASM was available. Yeah, of course.

Solomon Hykes: That tweet was the only output from these. We actually set out to build a WebAssembly team in 2016 and make a big investment there. It ended up just not shipping anything because it turns out we were not done with the core work on containers. It was, I think we set out to try and do too much. It turns out it wasn't Docker's role to do that. At the time, we did look at WebAssembly as the vehicle for almost a Trojan horse.

Daniel Bryant: Interesting.

Solomon Hykes: The thinking at the time was everyone's got a Docker CLI, and everyone thinks it as just a regular client. What if that client could be scripted, programmed even? You just sneak a little bit of programming in this tool. Over time, people learn to use it for more and more powerful things. Before they know it, they're scripting their whole deployments and boom, a new platform appears. That was the vehicle that seemed to make sense at the time. Then you definitely needed something lightweight enough to embed in a Docker CLI. Something like WebAssembly or Lua really were the only two options. That was very exciting to me. Now, I think we're just in a different place where for our purposes, making application delivery pipelines, programmable containers themselves are so ubiquitous and the ecosystem is so amazing. The problems in pipelines today are so terrible and basic. It's not rocket science, it's just that there's a missing standard and a missing upgrade to the state-of-the-art. WebAssembly is just too much.

Solomon Hykes: You can solve most of the problem with just good old containers, but you just got to get people to adopt an API and make all this duct tape into code and containers are actually good enough, at least for now.

So, in the future there could be a portal or marketplace for Dagger modules? [24:44]

Daniel Bryant: That's fantastic. I love your thinking points. That's really helpful. Just to wrap up at least that part for the listeners, there'll be a future where you'd have some maybe marketplace or library or portal or something where I can go and pull in all my different modules. Then really, I'm only writing a little bit of glue code. To your point, when I was playing around with a node SDK, I was like, it's just promises and it's really obvious. It's all fluent APIs. I'm just chaining stuff together. The idea was, I wondered at the time, I was like, I'm going to have to be writing a bit of code here. The future could be the fact that I get a module and then I only do the code that's necessary to join altogether.

Solomon Hykes: Exactly.

Daniel Bryant: Very cool.

Solomon Hykes: To be clear, this is a feature that's going to layer on top of what's already there. It doesn't change the APIs. It makes it much more accessible. Because like you said, instead of having to write a complete software project on your own in order to leverage Dagger, which is very powerful, but it sets a higher bar. It means in practice today, most of the successful deployments of Dagger are in medium to large DevOps teams. Everybody needs this, not just medium to large teams. We just need to make it a little more accessible. The way to make it accessible is, 99% of what I need is already in a module.

There's a high quality, easy to consume, a marketplace, we call it the Daggerverse. You go and search and you find the module you want. Then depending on what you need, you can either write a little bit of your own module and make it 10 lines, 20 lines, 30 lines of code. Just little functions, not a huge project. Then you just call these powerful existing, or another thing you can do is sometimes the module actually has exactly what you need. It's like a Swiss Army knife, and really all you need is you want to call that function in that module from the command line, from a shell script directly. You don't even need to write additional code. We're also adding that feature. A lot of bigger shops have what they call the paved roads.

Daniel Bryant: Golden paths.

Solomon Hykes: 80% of the work, you just use this template.

Daniel Bryant: I like that.

Solomon Hykes: Then for the ring, 20%, you can go and customize. The way everyone implements these 80% right now is boilerplate, a generator that just blah, dumps something on your repo, and then you modify it from there. Paz actually had a better model there where you just provided your code and then the platform took care of building it the right way, setting up the load balancer the right way, all of that stuff. That can be a module. Part of the Dagger community that's emerging from this dagger versus project are higher-level modules that are basically a paved road in a module. Because it's Lego and these things stack really well on top of each other, the reward is just like code. It's not shocking to use a Go library that calls another Go library. It works.

You can have a module that you call from the command line and it's simple app, and you look at the function doc and it says, you need to give it a source code directory, an API key for the versatile account or whatever. A name for an app and a host name, whatever. Then you just provide these from the command line, boom, boom, and then there you go. It's deployed. They're useful for ephemeral staging deployments. All these complex motions that are usually out of reach for most teams. You barely have CI/CD at all. If you want on-demand environments for anyone, forget it. You can't do that in a custom way. Now, it's just a module. The goal is to make this really accessible to everyone. You need any DevOps automation, it should be one command line away.

Daniel Bryant: I love it. I didn't quite get the full, bigger picture there. I'd gone up a couple layers of abstraction, I think from the quick start. I like your vision of the golden path, because everyone, pretty much I'm chatting at the moment, is striving towards that. People are like, what is the abstraction? Is it copy, paste YAML? Which I've got a gut feel like it's not, to your point, with Pulumi, for example, and Terraform. Seeing the values of libraries and modules and even those abstractions was really powerful. I like what you're pitching. It's almost world domination in the CI space.

Solomon Hykes: Well, we'll start with CI.

How do you build a successful community around open source software? How does licensing and donating technology to a foundation play into this? [28:44]

Daniel Bryant: There you go. I like it. One thing that didn't really come through, Solomon, is the strength of the community. Something actually, I shared to Adam Jacob, he had said the same thing. You've got to get this critical mass. It sounds like you are doing very well with that, with the Daggerverse and the Daggernaut. I love the brand in there. I'm guessing, to build this critical mass and to touch maybe the third rail for a second, you've got to think about licensing. You've got to think about community. It's a hot topic at the moment, maybe a controversial topic. Would you think about donating Dagger to a foundation like CNCF, like the Continuous Delivery Foundation? What do you think about the licensing model here? Because there's quite a lot of play for folks to want to invest in the community.

Solomon Hykes: Those two really important things here. First, community is everything. First, something like Dagger. We are a startup. We have investors. We have a team that is hoping to have joined a successful business. We have a strategy for growing and succeeding. That strategy we call community-led growth. That's the actual name of our strategy. The whole point is to create a thriving, fast-growing community in the DevOps space. The community is about people. People who are excited about a shared objective. They're excited about learning from each other, helping each other. It's building around a brand and a product, but it's bigger than the brand and the product.

Daniel Bryant: Yes, totally.

Solomon Hykes: It's a group of people that just love building things together, and it's really hard to get it right. It takes a lot of effort and dedication because you have to take a leap of faith that it will pay off later. That's basically, we've made a decision to go all in on that. Now, it's starting to pay off. It took several years. Now, like you said, we have the Daggernauts, we have a Discord server that is the beating heart of our community, and it's just a lot of fun. Our team hangs out there all day long. We don't actually have a separate private server. It's the same Discord server with private channels. We're literally in there all the time. That's the first part of the answer. Community, super important. It's got to be authentic.

Daniel Bryant: It has to be. Has to be.

Solomon Hykes: You have to actually believe it. The founders have to be involved. Me and my co-founders, Sam and Andrea, we're always interacting with the community. I think we've succeeded in creating a culture where everyone in the team, it's 25 of us, actually cares deeply. You'll see if you go on Discord, which I hope people do.

Daniel Bryant: I must join afterwards. Totally, yes.

Solomon Hykes: Let's say you come in and ask a question, a basic question because you're getting started, I saw your tweet. FYI, I had a missing JavaScript dependency, and I didn't see it right away and standard outputs. Let's say you come in and ask about that. You may get an answer from one of the core engineers who's doing their magic in the core, doing something impossibly hard. They're not out there in an ivory tower saying, I'm not going to go talk to users. Everyone's there just having a good time. That's really valuable and precious.

Solomon Hykes: We're going to protect it at all costs, and we think it'll lead to world domination. Because that's what you need to create a new standard and just change how people work. That's the first part. The second part, the CNCF thing, to me, that's completely different.

Solomon Hykes: That's not a community. To me, that's an ecosystem. It's an interconnected network of projects and companies like ours that will sometimes compete, sometimes collaborate. In the end, that's our area of the industry. We're solving a problem in tech together. Institutions like CNCF and these foundations, and even the norms of open-source development, the rules of engagement, that's usually valuable. In my opinion, you don't want to mess with that. You want to be a good citizen. I think of it differently as community, because collaborating properly as a good citizen with your competitors and future partners, collaborating and creating technical standards, whatever.

Daniel Bryant: Open standards. I love that kind of thing.

Solomon Hykes: Deciding if you want your license to be OSI compliant or not. All the important decisions, but I think of them as different from community because the people you interact there, they don't really care about Dagger. It's not a bad thing.

Daniel Bryant: No. I'm with you.

Solomon Hykes: It's just, it's different. Community versus ecosystem is how we think about it. There's a connection, of course. To answer your question, we will never donate Dagger to CNCF.

Daniel Bryant: That's a bold, obvious, great statement. I was curious.

Solomon Hykes: We're comfortable explaining why. It could be open source and have a real community that is undeniably there and true and also say, this is our community built on our product. It's open source, so you can fork it and go and build a competitor. By the way, good luck to you because this is going to move fast if we do a good job. That's part of the game in the open source ecosystem. We're not going to mess with that. We don't have any plans for weird licenses or anything like that. In fact, Adam and I are very aligned on the playbook there on how to connect open source and business. It's basically the Red Hat playbook, which is extreme openness on copyright. In other words, the codes, the software, which is really your license, and extreme control over your trademark and how it's used. Because that's your brand, that's your identity, that's not free for use. That's something, by the way, we did not do well at Docker. We were naive about it.

Daniel Bryant : The whole Moby Project and things like that.

Solomon Hykes: Well, Moby was an attempt at fixing it, but it came too late. It was too little too late. Basically, we did not protect the Docker brand enough. We allowed too many participants in the broader ecosystem to do whatever they wanted with it. We didn't feel great about it, but we didn't realize how strategic it was to be decisive and clear. Ironically, we were too worried about everyone loving us. We were afraid to make clear statements about what is okay, what is not okay, what is our position? We just wanted everyone to love us.

Daniel Bryant: That's understandable.

Solomon Hykes: As a result, the opposite happened. Everyone got increasingly frustrated because everyone projects their ambitions, their hopes, their dreams, and then they're not all compatible, so someone's going to be disappointed. Then that led to some trauma at the time, I think mostly forgotten now. The lesson for us is be clear, be straightforward, just say what you're going to do and then do it. If someone is going to disagree with it, it's better if they hear it early.

Daniel Bryant: I hear you on that. Totally.

Solomon Hykes: We're saying the engine is open source, it's real open source. You can fork it, it's there. No plan to change it. I can't guarantee that if we get bots, the next owner will change that. There's no good mechanism for that today.

Solomon Hykes: We've set up our business model so that we don't need to change the license to succeed because we're selling this cloud service on the side.

Daniel Bryant: >That's where you're making your money

Solomon Hykes: The trademark, you can't mess with that. You can't redistribute Dagger in any modified way and call it Dagger. We're not going to let that happen. We won't donate it to a foundation because it's not a neutral piece of technology. It's a complete platform.

Daniel Bryant: I was curious where the connection was there in terms of not the donation, but then you say you want to be super clear on you own this bit and then the rest of it, open source in the community.

Solomon Hykes: I think foundations are great for low-level components and technical standards.

Solomon Hykes (36:02): Where product opinion and subjective design and user experience are not important.

Daniel Bryant: Interesting way to look at it.

Solomon Hykes: That was really a lot of the misunderstandings. I think in Docker, I've always been very forceful and cared a lot about user experience.

Daniel Bryant: It came through in the product for sure, in Docker.

Solomon Hykes: When things align, then the results are great. That's our secret sauce. That's how we achieve things that others don't. It does require focus and ownership, so we can say no to things.

Daniel Bryant: Yes.

Solomon Hykes: That's a big part of good design is you have 10 good ideas, there's 100 bad ideas, but then there's 10 awesome ideas, and you can only do one. Because if you do all 10, the result is going to be bad. That's painful and tedious. That's part of the work, and it's hard. It's hard even if you're in full control. If you come in and let your competitors or partners, just random participants dictate what your user experience will be, it's not going to be a good user experience. That's why anything that requires that kind of product-minded focus, product excellence, design excellence, it cannot be donated to a foundation because it won't be a good product. Over time, you can spin out well-scoped, low-level components

Daniel Bryant: Like BuildKit with Docker as an example kind of analogy.

Solomon Hykes: Yeah. Well, actually, BuildKit was not spun out to a foundation. It was at least spun out into a separate repo. It is a good example. That actually, there's an open question that we're interested in flipping the tables here as a participant in Docker's BuildKit ecosystem. We want BuildKit to succeed, but we want BuildKit to be open and neutral. It would be a logical conclusion for BuildKit to be donated to a foundation so that Docker can build their product on it. We can build our product on it. There are others. There's a good dozen.

Daniel Bryant: I see a lot.

Solomon Hykes: Gitpod uses BuildKit. Okteto uses BuildKit. Depot uses BuildKit. There's a lot of really interesting things being built on BuildKit. That's a great candidate actually for a neutral component, but it's not fully realized. The canonical example would be runC, OCI. Containerd is a great example. ContainerD is, I think the perfect example. It's a very successful implementation of taking something that was nested deep into a product. We, at Docker, decided to spin that out. No one forced us, but people asked for it and we realized, this is a good idea. We will be able to share the burden of developing this low-level component. It'll be more reliable, et cetera, and we still get to control our user experience. Everybody's happy. Dagger at the moment just isn't large enough to have its own components inside of it, except BuildKit. We're building on top of BuildKit.

Daniel Bryant: Fantastic. That's a super clear definition. I've never really heard about the product and the UX aspects of it in that way before. That totally makes sense. Unlike Moby, I did think BuildKit was already in the foundation, to be honest. I see like, I bump into it so much. I do take your point about those low-level components being perfect candidates for extraction.

Solomon Hykes: BuildKit is in the Moby Project, and the Moby Project was our attempt at Docker to open up more. I think in the end, CNCF ended up being the right vehicle for it. Because as open as Moby was, it was still Docker projects only. If you take the analogy of Red Hat, Moby would've been our fedora.

Daniel Bryant: Yes.

Solomon Hykes: It's open, it's community-based. There's a lot more room for building your own products on it, but it's still the Docker stack with a bunch of Docker opinions in it. Most of the projects in there, we ended up spinning out into CNCF F instead. I'm not sure why BuildKit actually is not there yet. I think it just got caught in the, I don't know, everyone's busy.

Daniel Bryant: Everyone is busy doing stuff. Again, just one of my things, I see it so much in many projects.

Solomon Hykes: You just assume.

Daniel Bryant: I chatted up Justin Cormack, I just assume it was CNCF.

Solomon Hykes: We're talking about it with them. I've shared with Docker that I think it's something that is in Docker's interest to do, because BuildKit is a really powerful ecosystem in the making. It does require some nurturing. A 25-people startup can only nurture so much. Docker now is much more successful in the larger company. Tonis, the creative BuildKit works there. I think we'll see some gradual steps towards it. Justin Cormack is keenly aware of this, and he's got a lot on his plate.

Is there anything else you want to share with the listeners? [40:36]

Daniel Bryant: Fantastic. I'm conscious of your time. Really, appreciate the insight you provided today. Just a wrapping up point, is there anything you want to share with the listeners, or particularly around the future directions, I think? I know we've covered that pretty nicely, to be honest. Anything you want to leave the listeners with would be much appreciated.

Solomon Hykes: I think it's interesting because sometimes I talk to people about DevOps and application delivery and CICD, all those words that have been around for quite some time now. There is sometimes an assumption that it's a solve problem, it's this thing that's there that's not really moving, and sometimes it gets in the way. That's just how it is. Let's just focus on more exciting things in the future. I think what people don't realize is that you can't ship the future without fixing the factory that's going to build it and get it out there. You can't really, as much as everyone would love to just forget about Jenkins and YAML and shell scripts and just step into a bright future that's completely ready for us, pipelines are just a reality of life and it's going to get fixed. When you look at something like AI and all those amazing capabilities that are coming up now, we're in the really early days of discovering these capabilities. Prototypes, labs, a few products here and there. 99% of AI products that will exist are not shipped yet.

Daniel Bryant: Point taken.

Solomon Hykes: It's really a universe of experimentation and tinkering. All of those products will need to somehow be shipped and built and tested and monitored. DevOps is here to stay, and it's about to go through the massive transformation. I think this excitement around AI will be the catalyst for it. It's the excuse that we DevOps and cloud people now have to go get our respective management to finally invest in upgrading the factory.

Daniel Bryant (42:33): Good analogy.

Solomon Hykes: I think that there is an incredible interdependency and there's an opportunity for the AI community and the DevOps community to really help each other achieve their goals. I think it's just going to energize both parts of the industry. I think we're about to enter a really exciting phase.

Daniel Bryant: I'm here for it. Fantastic. As a long-term Jenkins survivor.

Solomon Hykes: Jenkins survivor.

Daniel Bryant (42:55): Way too many YAMLs in the Kubernetes world. I'm totally sold, Solomon. I like what you and several folks in the community are doing with this bringing it all together. I think it totally makes sense to me. Fantastic. Solomon, it's been amazing chatting to you. Thank you so much for your time today.

Solomon Hykes: My pleasure. Thanks so much for having me.