Code Yellow, Code Red

Leave a comment
Growth

Towards the end of 2025, it seemed that we couldn’t get through a week at Provet without some kind of infrastructure degradation. Outages were piling up, they were hard to diagnose, and customers were losing patience.

When a major AWS outage hit in October, our customers didn’t distinguish between Amazon’s problem and ours: months of flakiness had already eroded their trust.

So we declared a Code Yellow, the first time we’d done it: a company-wide written announcement explaining the concept, the structure, the exit criteria, and the reasoning. The target was simple: tighten our SLAs and achieve zero downtime for eight consecutive weeks.

The good news was that we hit it, with only one brief degradation incident in the entire period, and the contrast with where we’d been just weeks earlier was extraordinary.

This article is about that experience, and about the broader practice of Code Yellows and Code Reds: what they are, how to run one well, when to call one, and when to end one.

Here’s what we’re going to cover:

  • What Code Yellow and Code Red actually mean, and where the terms come from.
  • How we ran ours at Provet: the structure, the work, and what we learned.
  • A generalised template you can adapt for your own organisation.
  • When to call one, when to escalate, and the failure modes to watch for.

If you find this topic interesting, here are some complementary articles from the archive:

  • One bottleneck at a time explores why fixing one binding constraint compounds faster than optimising everything at once, which is what a Code Yellow forces you to do.
  • The beauty of constraints covers how deliberate limitation unlocks unconventional thinking.
  • Going direct is about communicating without intermediaries, which becomes essential during an escalation.
  • Invert, always invert explores defining failure first, which connects directly to setting exit criteria before you begin.

Let’s get going.

What Code Yellow and Code Red actually mean

Before I share our experience, it’s worth defining the vocabulary.

The terms Code Yellow and Code Red originated at Google. As Steven Levy describes in In The Plex, the name came from a yellow tank top owned by engineering director Wayne Rosing: whoever wore the shirt became the designated leader and could tap anyone at Google on the shoulder, pulling them off their current project to help.

The practice has since spread across the industry: LinkedIn, Meta, Shopify, Instacart, and OpenAI all use variations of it. And us of course, and maybe even you too.

The core idea is simple. A Code Yellow is a formal declaration that something is seriously wrong and needs concentrated, cross-functional attention now, before it becomes catastrophic, while a Code Red is the next level up: an existential threat that demands everything stops until it is fixed.

Here’s the distinction:

DimensionCode YellowCode Red
SeveritySerious but not existentialExistential threat or critical failure
UrgencyPreventative interventionEmergency response
DurationWeeks to monthsDays to weeks
Work hoursPrimary focus during business hoursAll hands, around the clock
Normal workDeprioritised but not fully stoppedFully paused

The real power lies in having a shared language at the company, not in the table’s definitions. When everyone in your organisation understands what “Code Yellow” means, you don’t need to explain the severity, negotiate priority, or convince people to shift their focus, because the concept itself does the work for you.

Compare that to “we’re really concerned about uptime” or “this needs to be a top priority,” both of which are vague enough to be ignored, or for other functions like product to argue that their own priorities are more important.

As LinkedIn’s SRE team described it, the goal is to get a team “out of a reactive mode where they are running from crisis to crisis and into a proactive state.” That’s precisely what it felt like for us, and what the next section is about.

How we ran ours

Let me dig into the Provet Code Yellow in more detail.

The trigger was a pattern, not a single incident. As described in the opening of this article, throughout the back end of 2025, we’d had multiple outages that were frustratingly difficult to diagnose, and that difficulty itself was a signal: it pointed to deeper infrastructure problems rather than isolated bugs.

When the October AWS outage hit, our customers perceived it as our problem because months of instability had already conditioned them to expect that this again was all our fault.

As an exec team, we agreed that something bolder was needed, and the CEO signed off. I announced the initiative to the entire company in writing, and since it was our first Code Yellow, the announcement explained the concept itself alongside the specifics: what we were targeting, how long it would take, what the exit criteria were, and why we were doing it now.

The framing was deliberate. Uptime metrics are getting worse, we are losing trust with customers, and we need to act now before this becomes a Code Red.

The first week was intense and, honestly, chaotic: daily stand-ups, daily war rooms, and a full audit of monitoring, speed measurements, logs, and alerting.

The scope of what needed fixing was larger than anyone had anticipated, and prioritising within the Code Yellow itself generated real debate, though we found a surprising amount of low-hanging fruit (e.g. implementing additional timeouts and circuit breakers) and those quick wins built early momentum.

As the immediate fixes gave way to longer-term infrastructure work (migrating services to better technologies, rearchitecting for resilience), the cadence naturally slowed from daily to bi-weekly, then weekly.

Our approach was to walk the software stack from the bottom up: starting at the databases and what was connected to them, analysing access patterns and API calls, then moving further up the stack to look at rate limiting. Who was calling our API? What slow queries existed? What should the timeouts be across the stack as a result?

We also walked the stack from the outside in, looking at monitoring and alerting and identifying gaps that needed fixing.

Throughout, we maintained async updates in a public Slack channel called #code-yellow-uptime, deliberately over-sharing so anyone in the company could see exactly what was happening. Written shares followed a similar cadence to the above.

That transparency, the proximity and visibility it created to the actual work, was one of the most valuable parts of the whole exercise.

One powerful structural decision of Code Yellows is what is called “tap on the shoulder” authority. If an endpoint was particularly slow for our biggest customers, anyone working on the Code Yellow could tap the domain team that owned it and tell them to reprioritise to speed it up immediately. That kind of authority only works when the entire company understands and supports the escalation, which is why the concept and upfront communication mattered so much.

Getting people to deprioritise their roadmap work was surprisingly easy in our case, though I recognise that’s partly a function of context. At a company of our size, the instability was felt by everyone, and after the AWS outage, everyone saw fixing it as the number one priority. After all, a direct correlation exists between the stability of our product and the happiness of our customers.

I suspect deprioritisation is harder at larger organisations where the pain isn’t evenly distributed. If that’s your world, the communication piece becomes even more critical: the problem statement needs to make the impact concrete and company-wide, and getting exec buy-in and sign off is critical.

If you’re a manager without the authority to deprioritise on your own, bring the data to your leadership and let them make the call publicly. The worst outcome is a quiet, unofficial Code Yellow where your team bears the cost without organisational support.

To give you a sense of the breadth, we improved and tightened rate limiting, fixed database timeouts, added circuit breakers, migrated away from Redis in areas where it was a bottleneck, and found and fixed memory leaks in background tasks. We added measurement of all of our API calls, partitioned by team, and set minimum speed thresholds that teams had to meet, lowering that threshold as the weeks went on.

We migrated our incident management tooling to incident.io and retrained everybody on a tighter incident process. And that’s just the tip of the iceberg.

When we hit our target, eight weeks with effectively 100% uptime as measured by our monitoring, it felt awesome. The communications rolled out in stages: engineering first, then company-wide, then a customer-facing announcement that was deliberately non-technical.

If I were to do it again, I’d push even harder in the first two to three weeks, because that initial burst was where the bulk of the work happened and increasing the intensity early, when momentum is high, would have compounded the gains faster.

The template: how to run your own

So how do you take something like our experience and generalise it? Whether you’re dealing with reliability problems, scaling challenges, or technical debt that’s reached crisis levels, the underlying framework is the same.

You need four components before you start your own Code Yellow:

  • Problem statement. This needs to be simple enough that everyone in the organisation can repeat it. “Our infrastructure is unreliable and customers are losing trust” is clear. “We need to improve our SLA posture across the P0 service tier” is not.
  • Exit criteria. These need to be specific, measurable, and time-bound, and you should define them before you enter the Code Yellow (ours was eight weeks of zero downtime). Without clear exit criteria, you’ll never know when you’re done, and the Code Yellow will either drag on forever or fizzle out.
  • Timeframe. Set a clear start date and expected end date, and don’t plan for longer than a quarter: if your Code Yellow needs more than three months, you’ve either scoped it too broadly or you’re dealing with a Code Red.
  • Authority structure. You need to define who can pull people in, who reports progress and to whom, and who decides if the scope needs to change. In smaller organisations, this might mean direct “tap on the shoulder” authority, while in larger ones, it might mean escalation through a VP chain or a designated incident commander with cross-team authority. The mechanism matters less than having one that’s explicit and understood.

Communication is the next piece, and it’s worth working hard on getting right: announce the Code Yellow to the full organisation with the problem statement, exit criteria, and timeline, so that surrounding teams know what’s happening, can expect delays on other work, and are ready to help if asked.

Transparency is what gives the Code Yellow its power. There’s nothing optional about it. This is a #announcements channel level communication. A Code Yellow that’s invisible to the rest of the company is just a stressed team working harder, which is exactly the opposite of what a Code Yellow should do; it should rally everyone around the team instead.

There are also cultural prerequisites that matter more than any process. Don’t think of declaring a Code Yellow as an admission of failure. It’s a sign that the organisation is mature enough to recognise a problem early and respond decisively. If early escalation gets punished rather than celebrated, people will avoid it, and by the time someone finally raises the alarm, you’ll be in Code Red territory.

When to call one, when to end one

So you have the vocabulary and the template, but the hardest part isn’t running a Code Yellow: it’s knowing when to start one and, harder still, when to stop.

Common triggers include alert fatigue overwhelming your teams, technical debt reaching crisis levels, key metrics degrading consistently, and incidents piling up faster than you can resolve them. The pattern to look for is a slow boil: as LinkedIn’s SRE team noted, Code Yellows typically result from “increasing technical debt, many small issues or breakdowns in a process” rather than a single dramatic failure.

If you’re waiting for the dramatic failure, you’ve already waited too long.

Escalation to Code Red should happen when something threatens the core business: severe and widening customer impact, existential competitive threats, or fundamental system failures. Google famously declared a Code Red when ChatGPT launched in December 2022, reassigning teams across the company to develop their AI capabilities, which became Gemini. That’s the severity level we’re talking about: existential, not merely inconvenient.

Ending a Code Yellow requires discipline, because exit criteria must be met for it to finish: “we’re exhausted” is not a reason to de-escalate, and neither is “things seem better.” You defined measurable criteria at the start for exactly this reason.

Once you’ve hit them, conduct a blameless retrospective, communicate the conclusion to the full organisation, restart paused work deliberately, and check in on team wellbeing, because the push took a toll and acknowledging that matters.

Crucially, the retrospective of the Code Yellow should produce systemic changes, not just a summary of what happened. If you ran a Code Yellow but nothing changes about how work is prioritised, how monitoring is maintained, or how early warnings are escalated, you’ve treated the symptom without addressing what caused it.

The goal is to make the next Code Yellow less likely, not just to survive this one. For example, earlier in the article we outlined how we pretty much overhauled resiliency across our entire stack.

Finally, watch for these failure modes:

  • Overuse. If you call Code Yellows too often, the signal loses its value: every escalation becomes background noise, and people stop taking them seriously.
  • Performative escalation. This is when war rooms become theatre, with lots of visible activity and impressive dashboards but no real change in priority or approach: if the work during the Code Yellow is the same as before, you haven’t escalated anything.
  • Failure to de-escalate. A Code Yellow that never ends becomes permanent urgency, which is indistinguishable from normal operations, and if you can’t exit, your criteria were wrong.
  • Shortcut temptation. Adding technical debt to exit a Code Yellow defeats the purpose entirely, because you’ll be back in another one within months, this time with even more debt to dig out of.

Your turn

Here are three things you can do with this:

  • Draft your own escalation protocol. Even if you don’t need a Code Yellow right now, having the framework ready means you won’t be designing it during a crisis. Define what triggers a yellow vs. a red in your organisation, who has authority to declare one, and what the communication plan looks like.
  • Reflect on your last crisis. Think about the most recent time your team dropped everything to fix a problem, and ask yourself whether it had a clear problem statement, exit criteria, and a defined timeline. What would have been different if it had been formalised as a Code Yellow?
  • Have the vocabulary conversation. Introduce the concept to your team or your leadership group, because the value of Code Yellow and Code Red as terms is that they compress a lot of meaning into two words. Even if you never formally use them, having a shared understanding of escalation levels makes the next crisis easier to navigate.

Wrapping up

Code Yellows work when they’re structured, have clear exit criteria, and give teams both the autonomy and the permission to focus. They’re a blunt instrument. As one engineering leader at Instacart put it, “Code yellows suck, drain team morale, and they leave a lingering distaste amongst all those involved. Yet they were the most effective and consistent weapon in ensuring we made meaningful progress on our hairiest problems.”

That matches my experience. Running our Code Yellow at Provet was exhausting, but the outcome (real confidence in our infrastructure after months of anxiety) was worth every minute of it. The key is treating them as the exception that proves the rule: a structured, temporary departure from normal operations, not a permanent state of crisis.

Until next time.

The right kind of AI sceptic

Leave a comment
Uncategorized

Read about AI opinion online and you get two camps: those who believe it will fundamentally reshape how software gets built, and those who see it as useful tooling but doubt the transformative claims being made around it. Neither side has a monopoly on insight, and both can point to real evidence (or lack thereof).

I’ve watched this split play out everywhere over the last six months: in conversations with other engineering leaders, in how candidates talk about AI during interviews, and within my own teams. The hostility has been personal too. Over the past year I’ve written several very pro-AI articles, and some of the private messages and emails I’ve received in response have been borderline insulting(!), dismissing me as naive, a shill, or someone who’s drunk too much of the Kool-Aid. It’s been my first time blocking people on Substack.

But the hostility made me think more carefully about what’s actually going on, because the perceived professional risk doesn’t sit where most people think it does. It’s not about whether you’re optimistic or sceptical about AI, it’s about how you hold that view: whether you’ve arrived at it through hands-on experience, or whether it’s become part of your identity, something you defend rather than inquire into.

This article is about finding the right mixture of optimism and scepticism, and learning to hold a flexible, informed opinion that benefits both you and how you’re perceived by others. We’ll look at why there are two very different kinds of sceptic, why ungrounded enthusiasm is just as risky as ungrounded dismissal, and what it looks like to hold your view well, whatever that view happens to be.

If you’d like to dig deeper, here are some related articles from the archive:

  • LLMs: an operator’s view was my first attempt to lay out an AI-positive but grounded position: what’s genuinely useful, what’s not, and where to focus your attention.
  • Use it or lose it covered the opposite risk: what happens when you lean too far into AI and start outsourcing the thinking that makes you valuable.
  • Invert, always invert explores the habit of seriously considering the opposing view, which is central to what we’ll cover here.

Let’s dig in.

The two kinds of sceptic

If scepticism isn’t the problem, what is? The distinction that matters is between scepticism as a conclusion and scepticism as an identity.

You might have used the tools extensively, kept on top of the latest research, and landed somewhere genuinely unconvinced. That’s totally fine. Or you might have absorbed your position through proximity, adopting the talking points of people around you without firsthand exposure, and stopped there. That’s not so fine.

You can often hear the difference: the first person says something like “I’ve been using AI for six months and I don’t think it’s made me measurably faster on the kind of work I do, here’s why,” while the second says “it’s just autocomplete” or “stochastic parrots” or some other meme and leaves it there. Both are sceptical views, but only one has done the real work, and the other is repeating another’s position, not deriving one.

I’ve seen this firsthand, in both directions. Some of the engineers and leaders I’ve spoken with continually dismissed AI as gimmicky, only to watch them completely change their position once they actually got hands-on with tools like Claude Code or Cursor as the models improved. Their scepticism had been inherited, not earned, and the experience exposed that.

But I’ve also seen people use the tools extensively and remain genuinely unconvinced, particularly those working in significantly large codebases or rarer languages where the models still aren’t as good. Their scepticism is specific, grounded, and hard to argue with. It’s the difference between “I looked and I’m not impressed” and “I haven’t looked but I already know.”

In some developer communities, dismissing AI has become a way of signalling belonging, and the position reinforces itself through repetition rather than evidence. Anyone who breaks ranks by actually engaging with the tools risks exclusion from their tribe, or a loss of identity (or both).

There’s a name for this pattern: the crab bucket describes a group where no individual is allowed to escape, because the others keep pulling them down. The problem isn’t that the crabs are wrong about the bucket, it’s that none of them have climbed out to check.

To be clear, climbing out doesn’t mean changing your mind. There are well-documented examples of developers who went from sceptic to enthusiast after engaging deeply and having breakthroughs with the tools, but the goal of engagement isn’t to make you an enthusiast. It’s to give you an informed position, whatever that position turns out to be.

Some of the crabs that climb out will come back down again, and that’s fine. Maybe the bucket is good enough. The view from outside the bucket might confirm everything they already thought, but at least they’d know.

The enthusiasm mirror

If ungrounded scepticism is one failure mode, ungrounded enthusiasm is the other, and it’s arguably done more damage while feeding the very scepticism it dismisses. Sceptics who disengage mostly hurt themselves: they miss opportunities; they look out of touch. But enthusiasts with budget authority can hurt entire organisations, because when they overcommit and underdeliver, they hand sceptics exactly the evidence they were looking for.

Think back over the last few years and the examples aren’t hard to find: Klarna announced AI would do the work of 700 support agents, only to quietly start rehiring months later when quality collapsed; CEOs from Microsoft to Google continually stated the percentage of code written by AI without any clear methodology for measuring whether it was actually any good; and startups promised autonomous AI agents that quietly turned into vaporware.

Each bold claim that didn’t land gave sceptics another reason to disengage, and you can’t entirely blame them given their position.

I’ve heard versions of this closer to home too. Friends and acquaintances have told me about receiving AI usage mandates from leadership at their own companies with zero guidance, no clear examples of leaders using the tools hands-on themselves, and access to only a limited set of enterprise tools chosen by people who clearly hadn’t done any real evaluation or had a single technical bone in their body. This breeds exactly the kind of cynicism this article is about.

But, if we look closely, the pattern is the same on both sides: a position adopted without sufficient evidence, defended as identity, and insulated from feedback. The enthusiast who dismisses every failure as “early days” is doing the same thing as the sceptic who dismisses every success as “cherry-picked.” Neither is thinking clearly, and both are optimising for being right over being accurate. And, if you think about it, AI just happens to be the catalyst for seeing this behaviour, and it’s a big human bug that we need to fix if we want to be clear and rational thinkers.

So if how you hold your position matters more than what the position is, what does holding it well actually look like? Whether you lean sceptical or optimistic, there are a handful of practices that separate a considered view from a reactive one.

How to hold your view well about AI

The first practice is the most obvious, and the one most often skipped: actually use the tools for yourself. Not a five-minute demo, not other people’s opinions on a Hacker News thread (did you read the article, or just the comments?), or a thirty-second YouTube clip of someone building an app. Sit down with the best available model, bring it a real problem from your actual work, and spend enough time to form a genuine impression for yourself.

If you come away unimpressed, that’s a legitimate data point. If you come away surprised, that’s a legitimate data point too. Either way, you’ve earned your opinion in a way that reading about it never provides, because the only truth comes from firsthand exposure.

I went through this journey myself. Years ago, I started with Copilot in Visual Studio Code and ChatGPT as a web-based prompt, then moved to daily usage of Cursor at Shopify, but to me they felt somewhat bounded and only applicable to individual contributors spending all of their time coding. The real shift came when Claude Code let me work entirely in the terminal and interact with the file system directly: suddenly the use cases, especially for leadership and general organisation and productivity, multiplied, and I ended up building an entire daily driver around it.

That progression took months, not minutes. Forming a real opinion takes sustained engagement over a long period of time.

The second is to separate capabilities from claims. “AI can generate working code from a natural language prompt” is an observable fact: you can verify it immediately. “AI will replace most software engineers within five years” is a prediction, and a speculative one at that.

Much of the AI discourse collapses these two categories, treating a genuine capability as proof of a sweeping conclusion. Arguments on both sides conflate these categories in surprisingly lazy ways, and it’s worth training yourself to spot when it’s happening.

The third is to make your scepticism specific. “AI is overhyped” is a slogan. “I’m sceptical that current LLMs can reliably handle complex, multi-step reasoning in large legacy codebases, and here’s what I’ve seen when I’ve tried” is a position worth engaging with. Specificity forces you to think about what exactly you believe, and it gives other people something concrete to respond to rather than a vibe to agree or disagree with.

The fourth, and perhaps the most revealing, is to ask yourself a simple question: what would change your mind? If you can answer that clearly, you’re holding a view, which is good. If you can’t, or if your honest answer is “nothing,” you’re holding an identity, which is bad. This is true whether you’re sceptical or enthusiastic, and it’s a useful check to run on yourself every few months as the tools and the evidence evolve.

Honestly, it’s hard for me to imagine changing my broadly enthusiastic position right now. I use AI every day at work and at home, and I have enough practical examples and workflows to talk about for hours. But that’s precisely why the question matters. If my answer is “nothing would change my mind,” I’ve crossed from having a view into having an identity, and I should worry about that as much as anyone.

The fifth is to engage seriously with the strongest version of the opposing view. If you’re sceptical, don’t just dismiss the most breathless hype: find someone thoughtful who’s genuinely optimistic and try to understand why. If you’re enthusiastic, seek out the most articulate critics, not the ones shouting at each other on social media, but the ones who’ve used the tools extensively and still have reservations alongside their praise.

This takes effort, because the most reasoned opinions rarely get the most attention: you’ll have to look past the hot takes to find them. But you’ll either sharpen your own position or discover a blind spot, and both of those are wins.

For me, the strongest opposing argument is the one I worry about most: that outsourcing cognitive work to AI gradually erodes the very thinking that makes you good at your job. I wrote about this in Use it or lose it, and it’s a concern I haven’t resolved. That’s what good scepticism feels like from the inside: not a slogan, but an open question you keep coming back to in order to refine your thinking and engagement with the matter at hand.

What this means for your team

Everything above applies to individuals, but if you’re managing a team, you have an additional responsibility: creating an environment where people can get the most out of these tools. That means resisting the temptation to mandate enthusiasm or punish scepticism, because both of those shortcuts produce compliance rather than genuine engagement. What you want is a team that’s actively experimenting, sharing what’s working and what isn’t, and building on each other’s discoveries.

I try to do this at Nordhealth by constantly finding new ways to use AI and then sharing what I discover with my team.

Take one recent-ish example. I took a recording of a video of someone using our app, had Claude Code split it into screenshots, and then ran UX analysis on each frame to generate a list of improvements, essentially creating an AI-powered friction log. It took minutes, and the output was genuinely useful and found numerous issues I wouldn’t have spotted myself.

Sharing experiments like that, including the ones that produce nothing interesting, signals to your team that this is about curiosity and outcomes, not about picking a side.

Creating an environment where people can be curious means framing conversations around outcomes rather than positions. “Did this tool help you ship faster, and if not, why not?” is a conversation that goes somewhere. “Do you believe in AI?” is not.

It also means modelling the behaviour yourself: share your own experiments openly, including the ones that didn’t work. If your team sees you being honest about what works and what doesn’t, they’ll follow your lead.

It’s also worth acknowledging openly that these tools don’t help equally in every context. A team working on a greenfield web application and a team maintaining a decades-old enterprise system with sparse documentation are going to have very different experiences, and both of those experiences are valid. The goal is a team that can figure out where AI helps them and say so honestly when it doesn’t, not one that agrees with you about it in the abstract.

Your turn

If you’re still sceptical about AI, ask yourself honestly: have you actually used the tools for real work, or have you formed your view from a distance? If it’s the latter, that’s the gap to close. Set aside an afternoon, pick a meaningful task, and see what happens. If it’s been a year since you genuinely tried, late-2025’s models were a huge step change for me.

If you’re enthusiastic, ask yourself the inverse: what would make you change your mind? If you can’t answer that, your enthusiasm might be doing the same work as the scepticism you’re dismissing.

Either way, find someone smart who disagrees with you and have a real conversation. Not a debate, not a performance: a genuine attempt to understand how they got to where they are. You might be surprised by what you learn.

Wrapping up

The people who thrive through this era won’t be the ones who called it right early. They’ll be the ones who thought clearly throughout: who engaged with the evidence as it changed, updated their views when the facts warranted it, and maintained intellectual honesty even when it was easier to pick a side and dig in.

Zealots are loud, but they’re brittle: the moment the evidence shifts, they break or double down. The people who hold their views lightly enough to update them are the ones you actually want to work with, and work for.

Buddhism has a name for this: the middle way, a path between extremes that favours clear seeing over fixed positions. You don’t have to be a Buddhist to recognise the wisdom in it.

That’s true for AI. It’s true for everything.

Until next time.