Who could be your Jeff Dean?

comments 2
Growth
Portrait of Jeff Dean. From Faces of Open Source, who hopefully won’t mind me using it.

Facts!

If you’re a programmer, you’ve probably relished in some of the many Jeff Dean facts. Here’s a selection for you:

  • Jeff Dean’s PIN is the last 4 digits of pi.
  • Jeff Dean proved that P = NP when he solved all NP problems in polynomial time on a whiteboard.
  • Jeff Dean once shifted a bit so hard it ended up on another computer.
  • Jeff Dean can beat you at Connect 4 in three moves.
  • There is no ‘Ctrl’ key on Jeff Dean’s keyboard. Jeff Dean is always in control.

Indeed, these “facts” aren’t facts at all, and there’s plenty more where they came from. If Chuck Norris is the embodiment of machismo then Jeff Dean has evolved into the industry’s manifestation of superhuman computer science wizardry.  

You may not know who Jeff Dean is, which, I must add, is entirely unreasonable. Let’s bring you up to speed. Aside from Jeff’s primary job, which is being the Internet, what else has he done in his career?

jeff.describe()

Before he joined Google in 1999, Jeff attained his Ph.D. from the University of Washington. His research focused on optimizing compilers. His research took him to Digital Equipment Corporation’s (DEC) Western Research Lab, before being convinced to come and join a young company with some interesting problems to solve. That company was Google. According to Slate, there were only around 20 other staff there at the time. It was very early doors at a pivotal company.

During the last 19 years, Jeff has co-designed and implemented a large proportion of the most impactful and world-renowned Google infrastructure, including numerous iterations of the crawling and search platform, the initial development of AdSense, Protocol Buffers, GFS, MapReduce, BigTable (which you may use as HBase), Google Translate, and more recently, TensorFlow. If you’re working in a software company solving problems with large amounts of data, you’re probably using a system or library that evolved from something that Jeff created.

Work

It’s no accident that Jeff has been involved in Google’s most innovative projects. In addition to being a programming sorcerer, he is equally good at helping others grow through his mentorship and technical talks, thus allowing others to multiply their careers by working with him. He also is highly respected and listened to by Google’s higher ups, making sure that he gets to place himself on the most impactful projects.

A candid Quora answer was written about what it’s like working with Jeff, which, surprisingly, isn’t all roses:

I worked with Jeff Dean for about a year when I was on the Google Plus team.

He’s mostly just a really friendly, smart, helpful coworker.  Like any other Google engineer, you could walk over to his desk, ask why a particular product worked the way it did and he’d give you a really good description of the decision factors and thought process that went into that particular architecture. He gave tech talks and demos to show his work and fielded questions just like any other senior engineering staff member.

But he was also a little intimidating. Jeff definitely has the ear of upper management and they give him free rein to pretty much work on whatever he wants to.  So if you’re working on a particular project that is at cross-purposes with something that he wants to do, then you kind of just get railroaded because his code is getting *launched* no matter what…  So you may have to just adjust your plans to accommodate!

So there’s a bit of “jujitsu” to working with high-powered folks like him — you find ways to align your projects so that their weight is being thrown in behind you, not against you.

Partnership

Portrait of Sanjay Ghemawat, also from Faces of Open Source. I also hope they don’t mind me using it.

A recent New Yorker article gave a candid insight into Jeff’s life and also his partnership with Sanjay Ghemawat. The pair work inseparably, often choosing to pair program together, bouncing ideas off each other whilst one forges ahead on the keyboard. They worked together in a similar fashion at DEC. They even live inseparably: “Uncle Sanjay” often comes to dinner at the Dean’s household on Fridays.

To the purist programmer, and aspiring technical wizard, their situation seems perfect: two best friends working on the most interesting problems, together, in a trusted self-directed manner, free from the pressure of typical feature delivery and sprints. They’re some of the world’s best engineers, so they just get on with what naturally comes next, since they are implicitly drawn to the problems that are on the cusp of innovation that will be impactful for Google.

The reality for the 99%

But, as you’re reading this, you’re probably thinking that’s all well and good for Jeff and Sanjay, superstar engineers who have actually changed the world: they can have their freedom, but we can’t. They were in the right place with the right intellects at the right time.

Instead, us less-than-geniuses get our silly deadlines set by people who don’t know anything about engineering, we get processes and ceremonies that detract from actually getting good work done, and we sit in crowded open plan offices where we constantly hear noisy sales calls on the phone all day. Good for them, we say. Good for them.

Here’s the question: are Jeff and Sanjay a direct consequence of right brain, right place, right time, or is any of their situation replicable within our own working lives? After all, their situation seems unreachable. The New Yorker describes the leveling system within Google’s engineering department as follows:

Today, Google’s engineers exist in a Great Chain of Being that begins at Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. Most progression stops at Level 5. Level 6 engineers—the top ten per cent—are so capable that they could be said to be the reason a project succeeds; Level 7s are Level 6s with a long track record. Principal Engineers, the Level 8s, are associated with a major product or piece of infrastructure. Distinguished Engineers, the Level 9s, are spoken of with reverence. To become a Google Fellow, a Level 10, is to win an honor that will follow you for life. Google Fellows are usually the world’s leading experts in their fields. Jeff and Sanjay are Google Senior Fellows—the company’s first and only Level 11s.

Here’s another thought: if 31 year old Jeff and 33 year old Sanjay turned up at Google today, would they be given the same freedom and autonomy that they currently have at 51 and 53? Or would they end up in a regular development team for countless years, working on a tiny, tiny cog in a gigantic machine?

A formula for success

We could make some hypotheses about why Jeff and Sanjay have been so successful.

  • Preparation. They are both extremely smart individuals.
  • Luck. They joined a world-class company at a critical time in its growth, allowing them to work on the most important problems.
  • Riding the company growth curve. Their success in said problems allowed them more and more access to two critical things: the ear of the upper management and the visibility of the cutting edge of their domain.
  • Earned autonomy. Trust from management allowed them their own choice of work and protection from less innovative projects taking their time.
  • Multiplied output: Access to innovative work allowed them to apply themselves to the problems that made themselves and the company grow.

Not all of us have the fortune or geographical placement to get a foot in the door at the very beginning of the next Google or Facebook. However, there are ways that we can create opportunities for our staff to emulate how Jeff and Sanjay work once we get our departments to a reasonable size: say to the point that there are a small handful of engineering teams.

What we can do is to add roles to the top of the individual contributor (IC) track that allow staff to become more self-directed, to propose their own projects, and to live outside of traditional teams.

The top of the IC track

A year ago at Brandwatch we promoted a small number of senior and long tenured staff to the position of Principal Engineer. This isn’t a role that we have invented; a quick search on Google will show you that it’s pretty well known and has been around for a fairly long time. For example, Jay Kreps was a Principal Engineer at LinkedIn when they were working internally on Kafka. There are similar role names at similarly large companies, such as Distinguished Engineer.

Engineers in these roles aren’t just hitting deadlines by clearing out tickets. Instead, as described by Drew Eckhardt, they’re doing work that materially improves whole company gross profit. This can be innovative architecture that leads to new products, scaling and optimization that saves significant percentages of company budget, and acting as a role model and coach to other engineers who become dramatically better at their job for spending time with them.

From my research, the role at companies of that size can often include management. An answer on Quora suggests that people at this level at Google (Level 8+) have more than 50 people in their division. However, we are not at Google size, nor did we want to blur our individual contributor and management tracks together.

We kept our Principal Engineer role solely about the engineering, because that’s probably what Jeff would have wanted if I could hire him.

How I implemented a Principal Engineer role

We have an Engineering department of around 160 people. We wanted to allow the chance for our best performing senior engineers to break away from the traditional scrum team format and to become more self-directed in their work. The exact reporting line differs in our numerous divisions, but I wanted to share with you how it works in my division.

I run the Applications division, which is mostly focussed around a suite of our core SaaS products. This does involve a number of interesting infrastructure and architecture challenges, but the bulk of the development effort is closer to the user: much more so than the division working on our web crawling, data infrastructure and primary storage.

Due to this, the roadmap of my division is primarily led by Product rather than Engineering. Carving out the space in order to do Engineering-led innovative projects – notably those that may not have a immediate pay off, such as infrastructure that can invent new product possibilities – can be hard to prioritize against Product’s feature roadmap.

To reuse a compiler optimization term, I hoisted the promoted Principal Engineer out of a team and had them report to me instead.

A diagram of where the Principal Engineer sits in my division. Dotted lines represent that other parts of the org chart have been hidden.

The Principal Engineer’s own work stream is set mutually between myself and them. This sometimes means that they are working alone, and sometimes they may be doing some work for one or more teams. This allows them to be a welcome addition to any given project, rather than a spoken-for resource that could do any old piece of work dished out to them. It protects their time and their interests.

A recent article from Silvia Botros, a Principal Engineer at SendGrid, and the accompanying comments on Hacker News highlight how the Principal role requires cross-organizational collaboration, whole department and company understanding, and strategic thinking about where technology is going, both inside and outside of the company. This aspect is the key that sets Principal Engineers apart from their Senior Engineer counterparts.

Some of the traits that we outlined for Principal Engineer are below. These traits form part of our internal career tracks document.

  • Experience: Typically has 10+ years of experience. May have previously been a Senior Engineer, Principal Engineer, or CTO at a start-up. Is a deep technical expert in their area and has a drive and vision for where we should be going in the future.
  • Technical knowledge: An expert, potentially at a global level, in their area. Designs, builds and suggests architecture that will grow the company in the future. Has a track record of leading the design and implementation of high risk or mission critical projects, particularly those that require the application or invention of new technology or techniques.
  • Mentorship: The go-to mentor and confidante for our engineers. Is an expert and patient teacher.
  • Consensus building: Champions new technologies and the modernization of legacy systems. Is able to win the hearts and minds of senior leadership in the company and also motivate and drive their colleagues in Engineering.
  • Conflict resolution: Can contribute to discussions and even resolve conflict at a senior leadership level. Rides the balance of business needs and engineering needs expertly.
  • Time spent reading or writing code: 75%+

Currently, our three core products have one Principal Engineer mapped to each of them. As far as I can tell they are happy in their roles, but I’m sure they can correct me in the comments if they’re not.

It’s working out pretty well so far. The Principal Engineer in my division has delivered a new distributed bitmap index system to speed up our products. In another other division, we led a major project to rearchitect our storage to deliver 50x speed ups for our customers with the largest data sets.

In summary

You and I will probably never be as successful as Jeff Dean or Sanjay Ghemawat, however, we can create the space within our organizations to allow others to form similar skills and work on highly interesting and challenging problems.

Implementing Principal Engineer roles, even at smaller companies, can have an extremely positive effect: it can champion engineering-led projects, give individuals great career growth, inspire others to stay on the IC track rather than jumping into management for progression, and, most importantly, can result in the invention of some excellent technology.

Who could be your Jeff Dean? Think about it.

2 Comments

  1. What are your thoughts on a Principal Engineer having a small team of engineers directly reporting to him / her?

    • Hey Tony!

      I’m not an expert on this, but I guess my answer would be “it depends”! For some senior IC folk, management activities can be challenging and a burden; not just on time, but also emotionally. Some senior IC folk, on the other hand, can manage people with ease and not have their core work suffer. The most effective would be able to multiply their own output by mentoring their staff (assuming they’re working on the same project, of course).

      From my research it seems that even Jeff Dean is now managing people. I certainly wouldn’t be against it, but our own Principal Engineers, when asked, said that management wasn’t really for them. Although, I think the head fake is that they are demonstrating a number of core leadership behaviours by being in that role. They’re just not accountable for the career progression of some staff as well.

      That’s a wooly answer – I’m sorry.

Leave a Reply

Your email address will not be published. Required fields are marked *