The Engineering Manager

I recently finished Poor Charlie’s Almanack, a collection of eleven talks by Charlie Munger. When Stripe Press published a brand new edition of it with their usual beautiful type setting and cover design, I couldn’t resist.

For those unfamiliar to Charlie, he is worth getting to know. While his notoriety stemmed from being one of the greatest investors of his generation, he was also a prolific speaker and writer, and was an advocate of cross-disciplinary thinking and application of mental models. He excelled in taking ideas from mathematics, philosophy, and psychology in order to think about the world in a new way.

In this month’s article, we’re going to be looking at one of the mental models I learned from Charlie and how it can help us as engineering leaders think, plan, and execute better by avoiding failure.

Death by optimism

Software engineering is tough. No matter how hard we try and how hard we plan, we always end up missing simple things which cause a whole bunch of problems down the line.

This can range from missing features and functionality (“how did we not think of that?”), to edge cases that we haven’t thought about (“how did we not see that coming?”), to poorly conceived rollouts and launches (“why didn’t we check this didn’t work in Italy?”).

There seems to be an ability to repeatedly stumble on the same simple mistakes that suggests this is just part of human nature. We spend so much time thinking about the big picture, focusing only on the happy path that we’re traveling down, that we tend to overlook stupid mistakes that then shoot us in the foot.

The question, therefore, is why?

Why is it that we make the same mistakes over and over again? After reading Poor Charlie’s Almanack, I’ve come to think that it’s because we apply mental models that are far too optimistic to our planning. As a result, because we typically rely on an optimistic outlook, we fail to consider what could go wrong.

Countless software projects in the last twenty years have significantly underestimated time and complexity, have not done enough QA, have not considered their rollouts carefully enough, and haven’t scrutinized their scope to ensure that key features were missing.

There’s clearly a core bug in our thinking, because humans would have solved these planning problems a very long time ago if it didn’t exist.

What is inversion?

If it is the case that thinking too optimistically is one of the reasons that we keep getting things wrong in software engineering, can we instead use a pessimistic mental model? The answer, I believe, is yes, and the solution lies in one of Charlie Munger’s models called inversion.

Inversion is one of the cross-disciplinary mental models that I mentioned above that Charlie mentioned often. He would use the inversion mental model in order to scrutinize investments before he made them. When it comes to planning projects, estimating scope, and especially rolling out changes, inverting the problem can expose the blind spots optimism leaves behind. The principle of inversion is highly applicable to us as engineers.

As Charlie says: “Invert, always invert.” This is the way that you can save yourself from disaster.

The origin of inversion comes from the 19th-century German mathematician Carl Gustav Jacob Jacobi, who advocated for solving problems by approaching them backwards. Rather than trying to work something out directly, you assume the opposite and work toward a contradiction.

Munger adapts this into practical situations: to succeed at an outcome, you should invert it by thinking about what would have to happen for you to fail, and then completely avoid all of those things in order to succeed.

For example, if your goal is to keep your home clean, instead of thinking about what it means for it to be spotless, you can invert the problem by thinking about what it would mean for it to be disgusting, and then make sure you do everything possible to make it not disgusting.

It follows that for your home not to be disgusting, you would need to:

• Clean your dishes within 24 hours.
• Take out the bin when it’s full.
• Vacuum the floor once a week.
• Dust surfaces regularly.
• Do your laundry when the laundry bin is full.
• …and so on.

And it just turns out that by inverting the problem and then doing all the items above to avoid it being disgusting you arrive at the same outcome: you have a really clean house.

This is the beauty of inversion. Instead of asking yourself, “How do I succeed?” you ask, “How do I fail?” and then systematically avoid those failure modes.

Inversion for engineering teams

Engineering teams can greatly benefit from using the inversion approach when thinking about larger initiatives such as estimation, planning, and rollouts.

As we saw above, whenever we do these activities with default optimism—thinking about what would be nice to have and what we must try to achieve—we often forget the things we must avoid as part of that process, which is where the mistakes creep in.

For example, if we are thinking of rolling out a new feature gradually to our clients, instead of leading our thinking with cohorting and the desire to launch to everyone as quickly as possible (i.e. goal oriented), we should think about what it would mean for the rollout to be a complete disaster, identify those factors, and completely avoid them. Doing so will ensure that we avoid edge cases in our thinking that we may have missed previously.

This is best explained by example.

Let’s imagine that you’ve just made a significant upgrade to part of your application and you’re thinking about how to roll it out. Instead of asking yourself how the rollout could be a success, you could invert the problem and ask yourself how the rollout could disastrously fail.

You identify that it would fail if:

• Bugs are present.
• Customers are unable to opt out if they don’t like the experience.
• Enterprise customers are caught off guard by the change and are not given adequate advance notice.
• Workflows take more clicks or scrolls or text inputs than the previous workflow that was upgraded.
• It looks worse, or is less intuitive, than the previous functionality that it replaced.
• It is slower than the previous workflow to render.

I’m sure there’s other examples that you can think of here too.

If you can take that list of reasons that the rollout could fail, and then systematically work to put protections in place so you avoid them, then it would follow that you would have a successful rollout.

Doing an inversion pass

I’d like to propose that the next time you do a significant piece of work, you do an inversion pass. It will help you systematically identify failure modes and build your defenses before you embark on whatever you’re about to do.

Below is a template that you can copy and edit for your own needs.

Setup

Before we get going, we need to work out who’s doing what.

• Begin by assigning roles. One person should be the facilitator who keeps time and the discussion flowing. There should also be somebody acting as a scribe that is able to capture the conversation.
• Then define the scope. Decide what it is that we’re trying to invert, which could be a feature rollout, an infrastructure change, or an architectural decision, etc.
• Make it clear that no idea is too pessimistic, and that today we are being paid to be cynics.

Inversion questions

With roles defined, work through these questions as a group.

The facilitator keeps the group moving and on time, and the scribe ensures that everything is captured. Questions that are in quotes are intended for the facilitator to ask the group.

1. Catastrophic failure. “What would make this an absolute disaster?” If this was to cause a P1 incident, what could some of the likely root causes be? What kind of scenario would have happened in order for that to trigger? Which single component failure would cascade most dramatically?
2. Silent degradation. “How could this fail without us knowing?” What metrics are we not monitoring that we should be? Which failure modes wouldn’t trigger our existing alerts? Where might we have blind spots in our observability and logging? What could degrade slowly enough that we wouldn’t notice until customers complained?
3. Rollback. “What if we need to roll this back at 3am?” Can this change be reversed? How long would rollback take? Is there anything that’s irreversible? At what point does rolling back become more dangerous than rolling forward? What happens if none of this works?
4. Load and scale. “What happens when real load exceeds our assumptions?” If you’ve currently estimated a certain load, what would break at 10x that load? Are there any resources that you have assumed will work properly that could go wrong? What kind of behavior exists in high traffic scenarios with extreme contention or queuing?
5. Dependency failures. “What if everything that we depend on breaks?” List out all of the external services that you rely on, such as databases and APIs. For each of them, think about what could go wrong if they became slow or unavailable. Think about whether you should have retries or circuit breakers.
6. Human error. “How could we break this ourselves?” Are there any operational steps that could be prone to human error? Do we have everything written down in playbooks in case whoever is on call doesn’t understand what to do, or are we missing documentation?
7. Data integrity and security. “Is it possible for us to corrupt or lose data?” Have we thought about race conditions that could happen? Or have we assumed transactionality that doesn’t actually exist? What happens if we process the same event twice, or if we skip one event? How do we know if data becomes inconsistent? Are there any attack vectors that we need to think about? Which data are we exposing and to whom?

You may want to add or remove inversion questions depending on the kind of project that you’re doing.

Once you’ve captured your list, go through and mark each item as to whether it’s:

• A showstopper (which must be addressed before launch)
• A mitigation (which will need monitoring, fallbacks, or workarounds)
• An accepted situation of which we are understanding the risk and moving forward regardless.

When you’ve got to this point, you should have list of actions captured by your scribe that you go and work on, plus a documented inversion pass outcome that proves you have done this exercise. You can use this to generate a risk register, update your design docs, and expand your documentation.

Try it yourself

Sometimes being pessimistic is good.

Using the principle of inversion, you can identify gaps in your planning and thinking, which can make your projects better, safer, and more resilient.

In your next project, try out an inversion pass. Run the exercise on your own or do it with your team and see whether it helps you feel more confident about what you’re going to be doing next.

Additionally, think about inversion in your own life. If you were to apply the inversion principle to how you manage your finances or what you want to achieve next year, could it potentially help you to think about these goals in a new light? Perhaps it could increase your confidence in getting them done to a high standard.

Remember: invert, always invert. If it worked for Charlie, it works for me.

Introduction

It’s been two months since I finished a sequence of LLM-based posts which were intended to think of unique ways that you could improve your leadership skills by leaning into AI as a coach, a contrarian thinker, and a way in which to expand and accelerate your decision-making.

If you’re interested in reviewing those posts, then you can find them here, in reverse chronological order:

• Leadership co-processing with LLMs, which introduces a number of prompting and usage ideas that could help you develop your thinking.
• A bag of worries: tackling overwhelm with LLMs, which is a technique I’ve been using to help me manage my own never-ending to-do list by offloading some of the cognitive load of prioritization to an LLM.
• A weekly mind meld, which uses some LLM assistance to communicate weekly with my department.
• LLMs: an operator’s view, the original post in this series, which covers some of the cultural change addressed in the first post in this list, and how code review and hiring are also changing.

Building on the leadership co-processing article, we’re going to go further this week and think about how we can expand our usage of a thinking partner into multiple thinking partners by using LLM agents to create your own councils that you can use to accelerate and supercharge your thinking and also simulate situations where many actors may not have default consensus on issues.

Inspiration

At work, I’ve mostly been using Claude Code as my go-to interface. Generally speaking, I like the Claude models, which have become particularly good since Sonnet 4.5 and Haiku 4.5 were released. However, I also enjoy the terminal interface, which I always keep open at the side of the screen as it is space efficient. I can mix regular prompting along with the generation of code, and it looks cool as well.

One neat feature of Claude Code, which is the backbone of this article, is that it makes it extremely easy to build agents that you can delegate to. For example, you just type /agents like below:

From here, you can create a new agent.

In order to create an agent, all you have to do is give a rough sketch of what the role of the agent is. For example, here is my initial prompt for a QA Engineer agent.

Claude Code is able to expand that into something far more detailed which you can edit and tweak however you wish.

If you’d like to see more examples of how a number of agents could be defined, then there’s a great public GitHub repository where people have been collecting together their examples. Each of the folders has a different type of agent (or agents) combined with a collection of commands that you can use in order to work with them.

This is especially useful when you’re programming since you can delegate specific types of work to these agents, such as refactoring, security testing, initial code reviews, and so on. Now, none of these are meant to be substitutions for the real thing, however, they become great partners when you’re coding because you can call upon specific functionality when you need it, which is especially useful when you know that you have blind spots. For example, a security engineer agent can continually make sure that you’re aware of these kinds of issues as you code.

One particularly interesting observation, highlighted in Simon Willison’s recent post, is that Claude Code could be considered a general purpose agent, rather than just a coding tool.

Following my own usage, I agree with this viewpoint. Claude Code is not just a tool for coding, but is also a tool to think. And agents let you get really creative with your thinking.

In previous posts, we looked at how we could use simple prompts to “pair think” with LLMs, from organising our time, to cross-checking decisions, to enabling mob sessions where architecture or code is designed together, and so on.

The more senior you get as a leader, the more nuanced and complex some of your decisions can be: not necessarily only because of the impact of those decisions, but also because of the challenge of finding consensus within large groups with differing opinions, biases, and experiences.

Sometimes the act of maintaining fast, synchronous connections with groups of people in order to debate, discuss, and forward your thinking can be blocked by others’ busyness or timezone. That’s where agents come in.

Council of agents

In the same way that there were examples above of how to use agents in Claude Code for specific technical functions (such as refactoring and testing), you can use agents to form specific thinking councils that you can use to accelerate your thinking faster than you could by working either on your own or by requiring synchronous time with others.

This is most easily shown by example, so we’ll go through two of them in this section. The first will be a technical council, and the second will be an executive council.

Creating a technical council

Now that we’ve seen how to build one agent, we can now think about building multiple agents that can make up a council that you can work with and think with. For this first example, let’s consider a technical council that can help you.

Firstly, you’ll need to think about which kind of roles you’d like to have in your technical council of agents. For example, you might want to have the following:

• Principal Engineer, covering broad technical vision, architectural patterns, and cross-cutting concerns.
• Platform Engineer, covering infrastructure, deployment pipelines, developer experience, and tooling.
• Security Engineer, with an interest in threat modeling, secure architecture, compliance, and vulnerability management.
• QA Lead, covering testing strategies, quality metrics, automation, and release confidence.
• AI/ML Engineer, responsible for machine learning systems, model deployment, and intelligent feature design.

Your own mileage may vary on the above depending on what kind of role and company you’re in.

With a decision made about what kind of roles you want to have in your council of agents, you then create them using the same methodology as we did in the example above.

For each of those roles, you create an agent in Claude Code, and you use your initial prompt to outline what they’re responsible for. Then, Claude Code will expand those into more detailed role definitions for the agents, which you can edit if you wish. And then, you continue doing this until you’ve defined your whole council.

Once this is done, it means that you can then ask questions to your whole council of agents. Here’s an example of what that could look like.

Here’s the output that was achieved from that particular query.

For this example, I specified in the query that I wanted just one paragraph of output per agent. However, if you wanted it to generate a detailed report and use one of the more expensive thinking models to do so, and then output that to a file, then the choice is yours.

And that’s it: with a small upfront effort, you now have a model of a technical council at your fingertips that can help you quickly interrogate ideas and decisions taking into account the diverse strengths and biases of each of those particular roles. It’s an incredibly useful tool to have.

Creating an executive council

Here’s one that helps me a lot.

As a CTO, my team is probably the most multi-disciplinary of them all, given that everybody in my executive team runs one of the other functions of the business. I am the only engineer!

Therefore, in order to help with my own thinking, especially about company-wide issues, having an executive agent council is highly valuable, and improves the kinds of proposals and decisions that I can bring to the group. Essentially, I almost get one or two meetings of iteration on my ideas completely for free (token costs aside) by doing it locally with my agent council.

In order to create this executive council, I go through the exact same steps but change the role definitions. For example:

• CEO, owning the overall company vision, board relations, strategic direction, and having final decision authority.
• Chief Product Officer, covering product strategy, roadmap prioritization, user experience, and feature decisions.
• Chief Operating Officer, responsible for operational execution, cross-functional coordination, process efficiency, and delivery.
• Chief Revenue Officer, owning sales strategy, pipeline, revenue targets, and go-to-market execution.
• Chief Marketing Officer, covering brand, demand generation, market positioning, and customer acquisition
• Chief Customer Officer, responsible for customer success, retention, NPS, implementation, and the voice of customer.

Again, depending on what kind of company you work for and who you’d like to have in your executive council, you may want to switch these roles out for whatever is relevant for you.

Here’s an example of the kind of query that you could ask. Let’s assume that the chat functionality that we were previously thinking about with the technical council is now ready to launch: how do we bring it to market?

You can see in the query that I accidentally wrote “rollout” twice, which I guess proves to you that these articles are still written by a human.

What’s fascinating about our query this time is that Claude Code even asked for input from the user to model the company that we’re working at. I did not explicitly ask to be given these choices in the setup.

After thinking for a while, I received output that looks like this. There was far more output than could fit in the screenshot, but for each of the executive council members, there are at least three to five interesting points that I know that we need to think through when we discuss it as a real human team.

Try it yourself

I’ve been finding this council of agents approach really useful for thinking through larger, more strategic plans and also in helping me develop my ideas before I bring them to the human version of the groups they represent. They allow me to do much more asynchronously. And then, when I do connect with my team synchronously, I find that the discussion and ideas that I am bring are more nuanced, considered, and researched.

So why not try it yourself?

Experiment with creating your own councils of agents using Claude Code or other tools and frameworks that support them. Get creative and think about ways in which you could improve your programming, your thinking, and your strategic decision-making.

It’s also a whole lot of fun as well. What councils of agents will you create?