Charity Majors on Intentional Culture, Dual Track Careers and AIOPS

Transcript

Introductions [00:05]

Shane Hastie: Good day, folks. This is Shane Hastie, with the InfoQ Engineering Culture Podcast. Today I have the privilege and pleasure of sitting down, across many miles with Charity Majors. Charity is the CTO and co-founder of Honeycomb.io. Listeners to the Architecture Podcast on InfoQ will be very familiar with her voice, and people who come to the QCon conferences will have met Charity many times, but this is the first time we've got Charity on the Culture Podcast.

So, Charity, welcome. Thanks so much for taking the time to talk to us.

Charity Majors: How is that the first time? Oh my goodness. Thank you so much for having me.

Shane Hastie: There's a few people out there who might not have heard of you. Who's Charity?

Charity Majors: Well, I am an infrastructure engineer, is how I think of myself. I am co-founder, as you said, of honeycomb.io. I was the CEO for the first three years, CTO now. I live in San Francisco. I took up hand lettering and art over the pandemic as my hobby, and I'm currently in London, so I just gave a talk at WTF is SRE conference. It was super fun.

Shane Hastie: This is the Culture Podcast. You made a beautiful statement when we were chatting earlier. Culture is meaningless when somebody says your culture is broken. So how do we tease out what a good "culture" looks like?

Culture is both formal and designed and informal and emergent [01:32]

Charity Majors: Yes, it bugs me whenever people are like, culture is this, culture is that, because it can mean anything and everything. If you look it up in Google it's everything from the rules and regulations to the practices and the habits, and the informal and the formal, and it's just like you're saying nothing at all when you say culture. And so I think it's really important to get more specific.

One of the ways of differentiating here is, I think, when you're talking about companies, it's important to differentiate between the formal culture and the informal culture. The formal culture of the company is everything that managers are responsible for. It's everything from how we do payroll is part of your culture, your vacation policy is part of your culture, whether or not you train your managers as part of your culture. The balance of acceptable behavior that people can do and not get fired is culture.

And then there's the informal culture, which, I think, is much more bottoms up. It's informal, it's not in the employee handbook. It is often playful, and fun, and anarchic, and chaotic. It's the in-jokes and the writing your release notes in Limerick form, or it's all the things that you bring about your character and your personality that you bring to work with you and you play off each other. When we talk about culture at work, I think, we often think about the informal culture because it's what jumps to mind, but in fact they're both, I think, really important.

Shane Hastie: As a leader, how do I create the frame, perhaps, for that informal culture to be generative?

Leadership actions create culture irrespective of your intent [03:06]

Charity Majors: Yes, the thing that I think that sometimes escapes people's minds is that if you are, I don't really like to use the term leader as a synonym for manager because they're not, but as managers you are being paid by the org to do certain things on behalf of the organization, and whether you like it or not, whether you think about it or not, your actions are creating culture. It's just like how the President of the United States, everything the president says is policy for the executive branch. You can't get around it. And whether you're intentionally creating culture or not, you are creating culture because people look to you and they see what you accept or what you don't.

My friend Emily Nakashima, who's our VP of Engineering, wrote this amazing blog post, called Power Bends Light. It's about her experience going from being an engineer to a manager, and how suddenly she started noticing, she has a weird sense of humor, suddenly as a manager people started laughing at her jokes. They had never thought she was funny before and now suddenly they thought she was funny. She was like, at first this really bugged me and then I just realized it's how we experience power. It's subconscious, it just is. We're monkeys, and our power dynamics play out in these really subtle ways.

So, I think that the culture that we, as managers and formal leaders of the company, are tasked with is creating a healthy organization. Pat Lencioni, who's the author of Five Dysfunctions of a Team and a bunch of other books, I feel like his book, called The Advantage, is the single best book I've ever read about this. It's all about how we, as companies, we are so obsessed with becoming smart companies, with our strategy and our tactics and all this stuff, and we're not nearly focused enough on becoming healthy companies. He makes the point that health trumps and often begets success, because most organizations are using just a tiny fraction of the overall intelligence and wisdom of their org, but healthy ones can tap into almost all of it.

It's like families. You think of the super dysfunctional family. It doesn't matter how smart they are, the kids are going to be fucked up. But a healthy family, they don't have to be the top 10% of IQs because they learn from their mistakes. They communicate in healthy ways, they have affection and compassion for each other, and so they're just able to be much more highly functional, and they're likelier to be successful.

And so, I feel like the job of leadership is, number one, we have fiduciary responsibilities to make the organization successful, which means that we need to make it healthy. And an organization that is unhealthy is maybe easier to spot. It's the ones where projects are getting canceled and people are doing political stuff, or they're unsure where they're going, or they're not on the same page as the teams that are next to them, or they're spending a lot of resources trying to argue about something. These are all super obvious visible ways that the unhealthiness of our culture manifests.

I've been talking for a long time so I'm going to pause there.

Shane Hastie: And I'm thoroughly enjoying listening to what you say. So, creating that space for the culture to emerge.

Charity Majors: Yes.

Shane Hastie: Actually, let's delve into one statement you made, that is, I think, quite important. Leader manager, not synonyms. How do they show up in the workplace?

Leader and manager are not synonyms [06:27]

Charity Majors: The first blog post I ever wrote that really took off was the one called The Engineer/ Manager Pendulum, and it was in 2017, I think. I wrote it for a friend, actually, who was the director of engineering at Slack at the time, and hating his job, but really reluctant to go back to being an engineer because he liked to have the impact, and he liked to have the power and the control just to shape the technology and the team, and it felt like going back to being an individual contributor would be a huge step back in his career. The piece that I wrote, the argument that I was making, is that the most powerful technologists in our industry tend to be people who have gone back and forth, not just once or twice, but multiple times, because you accrue these skill sets of both the tech itself but also the people skills, and the merging of the two, and the navigating the organization, and you just get stronger and stronger as you go back and forth.

I feel like lots of managers who go back for the first time are shocked to see that when they're an engineer after having been a manager, they're treated differently and they operate differently in the workplace because they have these skills. People still look to them for this people leadership, even though they're technically responsible only for the technical outcomes. I feel so, so strongly about this topic that your leaders are not just your managers, and in fact that management is not a promotion. Management should not be a promotion, it should just be a change of career. You're peers with the people that you're "managing". Even though there is a hierarchy, there is a need for decisions to be made in an organized manner. I feel like technical contributors should be responsible for technical outcomes. Managers should be responsible for organizational outcomes. Managers are responsible for making sure that a decision gets made, but they don't sit there making all the decisions. Good ones don't, at least, and the ones who do rapidly find themselves without anybody who wants to be on their team.

I feel like this is a concept that is starting to really catch on and even go beyond the balance of just engineering teams, because you really want people doing their best work to be in the part of the organization where they feel most engaged and challenged and excited. And honestly, people who just do the management track for too long really lose touch with a lot of that, with the hands-on brilliance, and the genius of making things and making things work, and how your system is built, and how it's structured. And in engineering at least, I think, the best line managers I have ever known have never been more than five years away from writing code and production. You really either need to move up the ladder as a manager, or you need to go back to the well periodically to refresh your skillset.

Shane Hastie: That has to be a very deliberate and conscious choice, doesn't it?

Shifting between engineer and manager and back needs to be a very deliberate choice [09:19]

Charity Majors: It does. I feel like we're just starting to see the first generation of technologists who have intentionally done this, and the results are phenomenal because you want to retain the talent who feels compelled to be good at their jobs, who feels compelled to stay on the sharp edge, who feels compelled. And so you don't want your managers to be like, "Well, I've been doing this for two or three years, I'm bored. I guess I have to leave in order to be challenged again, or in order to be hands-on again." You don't want that. And you also don't want people to join the manager track because they feel like they have no power otherwise, they don't get to make decisions otherwise, they don't get responsibility or accountability otherwise.

On both sides of the coin, as an organization, I feel like we are just starting to acknowledge that we need to restructure the entire way we think about compensation, job ladders, the career advice that we're giving to our top performers, or any performers. It goes along with this that you also, I think, want to lower the barriers to becoming a manager. I think anybody who's interested in being a manager should get a chance to at least build those skills. There might not be a seat in the org for them to join, but management is not an if then, it's not an either or. It's just a bunch of skills. If you're interested in management let's hook you up with an intern. Maybe you can run our intern program, maybe you can lead some meetings, maybe you can take over while this manager goes out on a four-month maternity break. There is never any shortage of work for managers to do, and if we can just level the playing field and make it...

I feel like a lot of times it's been like if you want to go into management you sit here crossing your fingers and hoping to be tapped from above, and just be like, "You have been chosen to be our manager." That's bullshit. That's creating some artificial scarcity that just doesn't need to exist. You should be asking everyone on your team if they're interested in being a manager, and if they're interested hook them up with some practice skills. Sometimes they'll be like, "No. Whoa. Tried that. Definitely not for me." Sometimes they'll be interested. I feel like everybody wins when we demystify management, and then we suck the hierarchy out of it as much as we can.

Shane Hastie: For the person who's at that three to five year cuss, they've maybe done the pendulum a couple of times and now they are thinking, do I want to actually go deep in that management and start to climb up the hierarchy? What do they need to do to position themselves, and to move into that space?

Progressing through organisational hierarchy [11:51]

Charity Majors: A lot of it comes down to the randomness of fate and opportunities. You need to be working at a place. I mean, if you just look at the number ratios. If you need one manager for every, let's say, seven engineers, and if you need one director for every, say, two to five managers, and you need one VP for N directors. You can see that the opportunities get scarcer and scarcer as it goes up, so you might need to change companies. And this is something where I feel like there's this taboo about expressing that you're interested in it because it's seen as a blessing, or something, and it's almost uncouth to express too much excitement for going up the ladder. Which is again, I feel like we drain the hierarchy out of this stuff, we make it more acceptable to talk about this stuff openly.

If you're interested, I think that one of the first steps is to ask people around you if they think you'd be good at it. One of the things that I feel like if you're an engineer and you're interested in being a manager, you should know if people feel like they'd like to report to you or not, because that's a pretty decent signal whether you'll be any good at it or not. And maybe people don't think you will, in which case you've got some work to do right there. There's that. Do people naturally want to follow your lead? There is the opportunity aspect. And if you're, say, a director at a mid-level company and the company's not growing and your boss doesn't seem like they want to go anywhere, you might want to try joining a smaller company, or a different company that's growing. Typically, if you go from a mid-level company to a small company or big to a mid-level, you can typically jump up one level. You can go from a director to a VP, or a manager to a director, so that's a way to get it on your resume.

Managing managers is an entirely different role to managing engineers [13:32]

I know I'm just jumping all over the place here. But one thing that I think is a little mysterious to some people is that going from being an individual contributor, as I see, to a manager versus going from being a manager to a manager of managers, they're almost as big of jumps as each other. Managing managers is an entirely different job than managing engineers. This is underappreciated, because, as managers, each of you who's managing people has a way, it's usually through intuition, of interacting with the world that makes people want to follow your lead. And usually this is very intuitive, and usually you don't even know how it works really, but once you start managing managers, they all have their own unique way of doing things that makes people want to follow their lead. And so now that you're trying to help them figure out how to debug problems you need to think about management and leadership in entirely new and different ways. It's up a level of abstraction, and it means having to relearn the job from first principles.

Another thing that I would say about this, and then I think I'm done, is that almost everybody who becomes a manager, who starts on this ladder, almost everybody assumes that they want to go to the top. Everybody is like, "Yes, I want to be a CTO someday. Yes, I want to be VP someday. Of course I want the opportunity to be a director." There's nothing wrong with that. It's very natural. We see a ladder, we want to climb it. We're primates. But, in my experience, the overwhelming majority of people get to a point and they realize that they hated it, they didn't want it, they don't want to go any higher. And that's fine too. My only advice would be to people to be sure and check in with yourself. It takes a year or two, maybe three, to really settle into the new gig. But really, after that, check in with yourself. What brings you joy? Does something bring you joy? Does anything bring you joy? What are the threads you want to pull on for your next steps that will help you lean harder into that joy?

Because I know way too many people, and so many of my friends are these ambitious types, I'm a dropout, so I don't really get this, but the people who are fucken ladder climbers from birth to grave, and they're just like climb, climb, climb, climb. They get too something resembling the top and then they realize that they're miserable. While it is easy to go back and forth between manager and engineer, it is real hard to go back from, say, a VP to engineer if you haven't done it in 10 years. And I know so many people who have literally done just that.

My friend, I wrote a blog post about her. My friend, Molly, came in as our VP of Customer Success or whatever, then her husband struck it rich when OCTI POd, and she suddenly had this real come to Jesus moment and realized, "I'm jealous of the software engineers. All I really want to do is sit there and write code. I hate what I do." It took her a few years. We moved her to support. She wrote code on the side and everything. She finally managed to move back to software engineering, and she is happy as a frog in the pond. But it was rough, and it's really hard to find those routes back. So I think it's really important to check yourself, and remember that ladder climbing is in and of itself not actually fulfillment.

Shane Hastie: Great points there. If we can bounce topics a little bit, AI. Generative AI versus AIOps. I know you've got opinions, so here's a platform.

The challenges in and around AIOPS [16:53]

Charity Majors: One or two. I'm on record as being pretty spicy about AIOps in the past, and I think a lot of people think that I'm just an AI hater. And I thought maybe I was too, but then generative AI came along and I'm like, oh no, there's some real stuff here. This is going to change our industry in the next couple of years. It's going to change our industry really fast. And so I've had to stop and think about, okay, what was it that I hated about AIOps, and what is different about this? For those who missed it, this is Friday, May 5th, and on Wednesday we shipped our own generative AI product that lets you generate and execute queries using natural language against your observability data. What's slow about the thing that I just deployed, or where are the errors? It's great. It's such a democratizing leveling feature.

AIOps, okay, so I think the way that I'm thinking about it is this, the last 20, 30 years of software development have been about the difficulty of writing and building software. It's been hard. And so it's been hard enough that I think that it's obfuscated or hidden from us the reality that it has actually always been harder to run, and maintain, and extend, and understand our systems, than it has been to build them. But because the upfront cost of building was so high we got to stuff that onto the ops team, or amortize those costs down the road a bit. We could stuff it under the quilt and forget about it. But now that building is getting so easy, I feel very strongly that the next five to 10 years or so are going to be all about understanding software, all about understanding what you've just merged, understanding what you're writing, understanding what your tests are doing, understanding while you write, after the fact, after you write, and everything in between, understanding what your users are doing.

And so I think that my beef with AIOps has been about the fact that they so often do things or claim to be removing the need to understand. Making it so that they're like, you don't need to understand your software. This AI is going to understand your software, and it's going to take action, or it's going to do the right thing, or it's going to alert you, or whatever. I feel like that is not only wrong, but it's harmful. It's harming you, not even the long term, in the midterm. It is harming you in your ability to understand your software, or explain your software, or migrate, or use, or extend your software. And I feel like there are lots of great uses for AI, but they come in form of helping us understand our software.

Liz Fong-Jones uses this example of we don't want to build robots to do things for us. When AI goes off and does something, it is no notoriously difficult, bordering on impossible, to go back and understand what it did or why. This is not a path that's going to lead us to great things. But there are paths where, like Liz says, we're not building a robot, we're building a mecca suit. Like a transformer suit that you can get into, with big ass limbs and everything, but you are still the brain of the thing. You're still making decisions, because machines are great at crunching numbers and any machine can tell you if there's a spike in this data or not, but only people can attach meaning to things. Only you can tell me if that was a good spike, or a bad spike, or a scary spike, or a unexpected spike, or what. And so giving people tools with generative AI to help them, like ours does, you know what you want to ask, but it was really complicated to use the query browser.

We help you ask the question so that you can understand it better. And here's where a lot of CTOs out there, if you get them a little drunk, they will be like, "Yes, actually I am willing to buy things, or I want to buy things that tell me that my people don't have to understand the systems, because," and this freaked me out when I first heard it from someone, "because people come and go, but vendors are forever." What the fuck? They're literally saying, no, we don't want to invest in making our people smarter and more informed and everything, because we know we can sign a multimillion dollar contact and that's going to be more reliable to us than our actual people are. And while I get the logic, come on, this is not leading us down a path to success or happiness. We have to invest in making our people smarter, and better informed, and better able to make decisions and judgments, because at the end of the day, someone is going to have to understand your system at some point. They just are.

Shane Hastie: I would go back even further and say that there's another thing we've not been good at. Fred Brooks said it a while back, "The hardest thing about building software is figuring out what to build."

Charity Majors: Yes. Oh, my goodness. That is so true.

Shane Hastie: How does generative AI help us with that, or does it?

The limits and potential of generative AI in software engineering [21:46]

Charity Majors: That's a great question. I don't think we know that yet. I don't think it does. I really don't think it does, and I would be really dubious of any products they claimed to. Yes, I mean, these tools are powerful. They're really freaking powerful. People who haven't used them, they're incredibly powerful. But at the end of the day, cleverness has never been the most important thing. In fact, it's often overrated. At the end of the day, I believe that we need to build systems that are comprehensible. We need to understand who we're building them for, why we're building them, how we're going to make money off of them, and how to fix them. And there's a lot of really powerful use cases in there for generative AI and other super powerful tools, but I don't want a machine telling me any of those things.

"A computer can never be held accountable, therefore a computer must never make a management decision." [22:38]

Maybe 10 years from now, but I think that as far as we can see down the pipe, this is not a thing that we should look to them for. If for no other reason then ultimately, there was this quote that Fred Hebert dug up that came from an IBM slide deck in 1979. It was, "A computer can never be held accountable, therefore a computer must never make a management decision." I think that applies in a lot of different ways. A computer cannot be held accountable for the rise and fall of your stock, therefore a computer can't tell you what to build. I think that we're a long way from holding computers accountable. So, I think, that capability and accountability should go hand in hand. And so, for the foreseeable future, I think that we need to not just make the decisions, but ensure that we have the detail that we need to make good decisions, which again, is where AI can definitely help us. But being clear on the why's, I think, at the root of everything,

Shane Hastie: And just to delve into a topic while I've got you, platform engineering. You made the statement, "We need a new kind of engineer." Please expand.

Platform engineering is the grand reunification of DevOps [23:48]

Charity Majors: Yes. I'm walking a very narrow line here because there's a company in particular out there who's been making very inflammatory statements about how DevOps is dead, and screw those people. That's not true. I mean, oh my God. Ugh, wait, clickbait. But I would go so far as to say that the DevOps split never really should have happened, and I understand why it did. There was too much complexity, too much surface area, et cetera. But fundamentally, and this does go back to accountability and responsibility again, because the people who are writing the code need to also run the code. It's that fundamental. And the more complicated things get the more we're running back to this, because the people who don't build the system have no hope of understanding and debugging the system, and the people who don't run the system have no hope of building a runable system.

Along those lines, I feel like what we are seeing is a grand reunification, and I think that the platform engineers are at the tip of the spear here. Every engineer should be writing code, and every engineer should be running the code that they write. And where platform engineering is, I think, really exciting, is number one, these tend to be engineers with deep background in both operations and software engineering. Number two, I think it's an engineering team that when done correctly, when done well, isn't actually owning reliability. It's like the first ops team that doesn't actually own reliability. Instead, your customers are not the customers, your customers are your internal developers. And your job is not to keep their code running at four 9s, or whatever, your job is to see how quickly and easily they can write code and own their own code in production. And then it's their jobs to be responsible for however many 9 in the SLOs, and the SLIs and everything. And I think this is awesome.

I also think that right now we're at a stage where you can't have junior platform engineers. You really have to have experienced platform engineers. But it's also, I think, the highest leverage engineering that anyone's ever been able to do. Because the platform engineer, you are sitting here leveraging the work of vendors who have tens, hundreds, even thousands of engineers working on this product, and you write a very few lines of code or Shell script or whatever, and make it accessible to your entire organization, which is, I jokingly call it vendor engineering, but it's incredibly powerful. And it involves taste, I think, as much as it involves raw engineering. You need to know how to build a thin layer, or an API, or an SDK, or something that empowers everyone internally, while providing this consistent look and feel, the right conventions and everything, to leverage everything this vendor has to offer. Then I just think it's a really exciting place to be right now.

Shane Hastie: That's a huge cognitive load.

One of the most important jobs of platform engineering is to manage cognitive load [26:38]

Charity Majors: It is. This is why, I think, that one of the number one jobs of platform engineers is to manage that load and to manage it very consciously. I think that you have to be constantly shedding responsibilities, because otherwise you're going to be constantly adding responsibilities. Anytime it turns into you have to build a software product it's time to offload it to another team, because platform engineers do not have time to write and own products. They can spec them out, they can make dummies, they can prototype, but they do not have the cycles to do that.

Yes, I think it's a very interesting, it's a very new space, and it's really exciting. I tweeted jokingly a couple of weeks ago, something like, "A team that you respect has just announced that they're building a startup for platform engineering. What are they building?" And the responses were all over the map. I really think that platform engineering is probably DevOps, in that you can't build a platform engineering product because it's a philosophy, it's a way of operating. It's a social invention, not a technical invention. I think there are a lot of technical challenges in conventions being hammered out in the ground right now, as we speak.

Shane Hastie: Charity, great conversation, and I wish we had plenty more time, but I know it's late for you, and we have limits on how long the podcast can be. If people want to continue the conversation where can they find you?

Charity Majors: Well, I am still on Twitter, @mipsytipsy. I plan on checking out Blue Sky this weekend, but we'll see. There's also my blog at charity.wtf, and of course there is the Honeycomb blog, honeycomb.io/blog, where I and others write quite a lot about things, not just concerning Honeycomb, but also observability, and platform engineering, and stuff all over the map.

Shane Hastie: And we'll make sure to include all of those links in the show notes.

Charity Majors: Awesome.

Shane Hastie: Thank you so much for taking the time to talk to us today.

Charity Majors: Thank you for having me. This is really fun.

Mentioned

• Honeycomb
• Emily Nakashima
• Power Bends Light blog post
• Patrick Lencioni book The Advantage
• The Engineer/Manager Pendulum blog post
• Charity Majors on Twitter
• Charity.wtf blog