The jar of life
There have been umpteen articles written on the “jar of life”.
The story is attributed to a mythical philosophy professor, giving a life lesson to his students. Now, this isn’t going to be another one of those articles, so don’t worry. But for those unfamiliar with the concept, here’s a quick summary.
Your life is a jar (of course!). You fill it with golf balls, which represent the big important things, and it looks full.
Except it isn’t, because you can pour a whole load of pebbles in too, which take up the space around the golf balls. Now it looks full.
Except it isn’t yet, since you can then pour in a whole load of sand which fills the remaining space. Then – and you can see where this is going – it looks full again, but then you can pour liquid in there to make it legitimately full.
Aside from producing a jar full of wet objects that will be a pain to separate and clean, it follows that if you put them in the jar starting with the smallest first, then they wouldn’t have been able to fit.
Instead you need to put them in biggest first, so that the smaller things can fill the gaps. The metaphor is that you should focus on the big, critical facets of your life first, then the less critical things will naturally fill the gaps around them.
Right, pop-philosophy lesson over. Onward…
Reusing the idea
I had a conversation with a colleague at work that reused this metaphor in a way which I quite liked.
Let’s go back to the full jar. There are objects in there of all sizes.
Except this time it’s not your life, it’s your Engineering department (of course!) and the objects are the projects going on (right…) and their size denotes their importance to the rest of the company.
For example, mapping those objects to types of project:
- Golf balls: These are big ticket features and new products. They encompass anything that is going to have a significant marketing splash, or a new revenue stream, or both. They are the projects that your commercial staff and clients are hounding you about. They are the publicly visible deadlines.
- Pebbles: These are bugs, speed improvements, workflow tweaks and UX refinements. They encompass anything that gets a reaction of “nice” rather than “wow”. Yet, cumulatively they are impactful.
- Sand: This is everything that doesn’t concern the rest of the company. Developer tooling, deployment processes, infrastructure automation, “keeping the lights on”, and so on. Essential work, but mostly invisible, unless it breaks and causes bigger issues.
Reapply the metaphor and it still makes sense. You need to get all of these things done, all of the time. The big ticket items need fitting in first, but you can’t be successful without accounting for everything else.
However, I don’t want to use this article to convince you that you need to account for all of these things; I should hope that as a diligent engineering manager or developer you are managing to juggle the differently sized objects in order to deliver a good level of service to your users and the rest of the business.
Instead, I’d like to challenge the preconception that the best engineers are only drawn towards the golf balls. There are plenty of engineers that thrive, grow and enjoy their work even more when the correct space is created for the pebbles and sand involved in running a technology platform.
The curse of the high ticket features
Sometimes, high ticket features and products, it seems, are the only thing that the company cares about. You might have the fastest storage, the best uptime, no major incidents in months, but it doesn’t really matter; the company only cares about the next big thing and guess what? They want it yesterday.
Naturally, your most talented, career-focused engineers will be drawn to these projects because of the importance that the company puts on them.
If they want to continually make a case for more responsibility in their roles, more interesting work, and faster track towards promotion, then they’ll be clambering to get on next important high ticket project.
But, these projects can often become the breeding ground of the worst cultural aspects of our industry:
- Tight and arbitrary deadlines, often dictated by a noisy client or market pressure, creating stress, crunch and compromise.
- The highest value projects often attract the highest-paid stakeholders, thus creating an occasionally uncomfortable high stakes environment to complete the work in.
- Sometimes the big ticket item itself isn’t the most important thing: it may be the marketing message that goes along with it, the shift in company positioning, or the proverbial middle finger that is raised to a competitor. The usefulness of the feature can suffer as a result of timeliness, leaving people unsatisfied with what they’ve achieved.
The key issue here is that the work that is perceived externally to be most valuable is often the work with the highest stress attached to it. Your best engineers could be exposed to higher risk of burning out or getting fed up and leaving.
Is there another way for them to be happy, grow and progress?
Pebbles and sand
Your challenge is to make sure that – at least within your Engineering department – that the pebbles and sand projects are also perceived to be extremely valuable and career defining, even if the communication of that perception only reaches the boundaries of your department.
After all, the invisible parts of your infrastructure are the ones that make the highly visible ones really shine. They’re the parts that when they’re working, nobody cares, but when they’re not working, they’re the only part that the company is begging you to fix.
The challenge is to nurture the joy of caretaking in your platform. Caretakers traditionally look after existing buildings. They are invisible, but they make sure that the building is tidy, running efficiently, that broken parts are replaced and upgraded, that the electricity and water are flowing, that any problems are swept up and taken care of. That’s exactly what you want your caretaker engineers doing in your platform.
In fact, I’ll take that further: you should fully embrace, praise and reward caretaker roles, caretaker projects and even caretaker teams, and you should do so just as generously as those shipping the latest and greatest whizzbangs that get announced on stage to rapturous applause and whooping.
Caretaker engineers keep the wheels turning. They are absolutely essential.
If you want your department to be successful, you need to identify the need, and elevate the collective worth, of caretakers.
This fundamentally comes down to some simple steps that you need to consider:
- Changing the perception of caretaking work. This is both the internal perception that motivates staff to work on these work streams, and the external perception that – just because it isn’t the latest high ticket feature – it isn’t as valuable to be well resourced.
- Prioritizing and measuring. Given that your Product teams will be less involved in defining exactly what this work is, how can you do so? You need to work out how to capture requirements, set goals and measure success.
- Ensuring that success is celebrated. Given that high ticket features often get a marketing splash, you need to ensure that your caretaking streams have something to celebrate at regular intervals.
Let’s look at these in turn.
Changing the perception of caretaking
There have been times in my career where I have been worried about having to staff a seemingly unexciting project, making an incorrect assumption that it will be met with resistance when I raise it with my teams.
It turns out that there are a number of benefits to caretaking projects. Framed correctly, the work can actually be very rewarding:
- Less pressure from commercial deadlines. This alone can be extremely attractive to your engineers. In the same way that the janitor sweeps the halls after hours in relative silence, the increased distance from commercial staff in caretaking work can mean it can unfold in its own time.
- Less interruptions and pivots. Typically caretaking work is more clearly defined than product work. Your infrastructure may need upgrading. You may have a giant backlog of bugs to work through. Less glitzy, yes, but also more of a chance for uninterrupted flow within deep work.
- Increased self-directedness in solving problems. Caretaking projects by their nature are inherently technical, so those closest to the problem – the engineers working on it – can work out the best way to do it for themselves. Autonomy is happiness.
- A chance for mastery. Autonomy and deep work breed mastery. A seemingly unexciting caretaking project could create specialists in particular areas of your stack. That works well out for both your engineers and yourself.
So, if you have a project to update the major version of HBase, don’t assume that your engineers will run away screaming. You may find that it is a welcome break from the rollercoaster of high ticket work, and a chance to build expertise and specialism.
Just pitch it right.
Prioritizing and measuring
Since caretaking projects are lead by engineers, you’ll need to think about how to measure them to see progress. There’s no definitive answer here, but there are some scenarios:
- Boolean: get it done. Ah, simple! If you did need to upgrade the major version of HBase, you’ll know it’s done when you’ve done it. Easily measured.
- Hit a number. Let’s say you’ve amassed 300 bugs by neglecting the platform during a period of crunch (been there, done that). 50 of them are high priority and a killer 10 are critical. You can set achievement checkpoints as you clear the critical ones, then the high ones, and so on. Define the state that you want that area of the system to get in, then work towards it. Reassess whether it should continue when you get there. Measurably hack away at that long tail in checkpoint intervals.
- Reduce variance. Maybe your uptime has been suffering recently. Creating a caretaking project to address the root causes could continue until stability has been reached in the measurement over a period of time, say +/- 0.05% over three months.
Although these suggestions are pretty basic, it’s important to set success criteria, both so you can know when to stop working on them and also so that you can celebrate when you achieve them.
Ensuring that success is celebrated
Since high ticket features often get a marketing launch to shout about their completion, you need to ensure that you create a similar culture within your department for your caretaking projects.
When those projects reach milestones, shout about it. Let everyone know what has been achieved and by whom, and how important that is to your users, platform and engineers. Maybe do a technical talk or broadcast the achievement via email. Elevate the success of the engineers.
300 bugs cleared? Fantastic. The system is more stable than ever? Brilliant. Technical debt has been greatly reduced in the primary storage system? Great. That’s a big achievement.
The joy of caretaking
There’s a pleasure to be had in looking after the less glamorous parts of your application. It is meaningful work, and it can be extremely rewarding. It’s a chance for self-directed engineers to plot their course and succeed on their own terms.
If you are a smaller company, then you may feel it is too difficult to make your case for moving staff away from the frenetic high ticket items towards longer plays on your architecture, tooling and engineer efficiency. However, every time we’ve done so, we’ve wondered how we got by without it.
Here’s some things that we did:
- Attacking an extremely large bugs backlog with a whole team. A rapid period of growth left a lot of things on the floor. Deciding to put a whole team on it was a bold move, but it dramatically improved the quality of the application, and with time, the team progressed to work on many smaller enhancements for our users. The engineers that were on this team now have an incredible breadth of knowledge about our platform, and it was a fantastic place for graduate developers to land.
- Committing permanent staff towards engineering efficiency. This small team has built tooling that automates our workflows for deploying containers on to our cloud environment, including a GUI to check builds and do one-click deploys. Our other teams are deploying code faster and more regularly as a result. We’ve also created some Kubernetes domain experts on the side.
- Committing to reusing components for consistent look and feel. Our pattern library, Axiom, grew out of one team that built our most recent product – Audiences – but subsequent time was devoted to making those new components reusable across our other products. Axiom has since graduated to a permanent project that is staffed to help other teams reuse and expand the pattern library.
So what’s the take-home message? Never assume that the pebbles and sand are boring to engineers.
They’re a learning opportunity that benefits the skills of your staff and your departmental output; they are work that unfolds at a more controlled pace; they are an opportunity for autonomy and mastery, and they can create domain experts.
Sweeping the halls isn’t as bad after all.