Not invented here, revisited

Leave a comment

When should we reinvent the wheel? Photo by Tiffany Nutt on Unsplash.


Recently, we redesigned the main storage architecture of one of our products.

We did so by writing a distributed bitmap index of our own that was exactly suited to the type of searches and aggregations that users perform frequently. My colleague, Phil Messenger, who served as the lead engineer on the project, gave a technical talk at Devoxx 2018 about how the system was conceived and designed.

While this project was running, and despite my absolute trust in the team’s technical ability to deliver it, I couldn’t help feel that familiar nag in the back on my mind: are we definitely sure that this hasn’t been built elsewhere? We were convinced that it hadn’t, but it is always hard to be one hundred percent sure.

Since it was a new distributed system, development and production rollout was challenging and proceeded at the cost of developing less customer-facing features. This reality was a harder sell to our product team, as our application, at the time, was still playing catch-up in the market.

In retrospect, the storage system has been a resounding success. It is exponentially faster than our previous Elasticsearch index, scales linearly with usage, and has future-proofed the next few years of growth for this particular product.

But building something from scratch, rather than composing it from existing open source technologies, is a risky strategy. What if we’d missed an existing technology when we did our research? What if it had turned out much harder than originally expected? What if we couldn’t get buy-in from management and manage expectations accordingly?

How did we know we hadn’t we fallen under the spell of Not Invented Here syndrome?

Not Invented Here

Not Invented Here, or NIH for short, is a mindset that embodies a reluctance to reuse or purchase existing products or technologies because they originate from the “outside”. The NIH mindset rejects perfectly capable solutions that lie beyond the boundary of a company, department or team.

But why?

In technology specifically, NIH highlights situations where engineers side with reinventing the wheel due to a belief that building technology in-house will be better, quicker and cheaper than using an existing solution. In many cases, and especially with the current wealth of excellent open source software available for use, this is an anti-pattern and a big, bad smell to be picked up by management.

Assuming a suitable technology exists, why not use it, even if it does require a bit of modification? After all, isn’t software reuse the biggest lever that great engineers can use for getting the job done quickly and reliably?

Well, not always.

NIH as a differentiator

In 2001, Joel Spolsky wrote an article outlining his experience at Microsoft where the Excel team would reinvent the wheel in order to reduce the external dependencies in their software. They even had their own C compiler. The result? Excellent cross-platform compatibility and a small executable. He argued that when working with an outstanding team, reinventing the wheel can be a differentiator.

He concluded the article with a recommendation:

If it’s a core business function – do it yourself, no matter what.

Sound advice.

After all, as Joel states, if you’re a Web accounting company, then you should probably write your own accounting software. If you buy someone else’s accounting engine and re-skin it, you’re not really adding much value, and if you’re a technology company without valuable technology, what are you?

Your decisions are often smaller

So far, so good. You’ll want to make sure that you build the technology that supports the core business.

But as a manager, or as an engineer, you’re going to be thinking about whether to reuse or build components at a much more granular level. Yes, you may be writing your own Web accountancy software in general, but of which components should it be composed?

Should you use this storage system, or that one, or should you write our own? Should you use this CSS pattern library for your UI, or should you start writing one yourselves?

When Joel’s article was written 17 years ago, the open source community had not seen the surge in popularity and progress enabled by centralization around communities like Github. There’s more code to reuse than ever before, and it is much easier to find.

I would assume that reuse discussions in 2001 had a slightly different focus: a decision to possibly choose between implementing an established project from Apache or GNU, or between purchasing a system from another company, or building it from scratch.

I can see how many decisions to build rather than reuse could have stemmed from high licensing prices, since SaaS was in its infancy – Salesforce was just 2 years old – or from lack of faith in open source projects, because they moved slower.

Paralysis of choice

Although now we have the luxury of being able to find lots of software to reuse, there is another problem: which should we choose? There is a huge search space: one can choose from many open source implementations at differing levels of maturity, or from numerous paid-for SaaS offerings with a matrix of pricing options, to on-premise installations, or, of course, one could build their own. Argh!

Additionally, companies that operate with a closed-source model such as Google and Facebook also publish academic articles and white papers detailing how their technologies work. The algorithmic challenge or architectural quandary that is a team is facing may have also been solved before. Some papers are reimplemented as open source projects in their own right, such as Apache HBase which is based on Bigtable, but for those that aren’t, what search keywords do we use to find the paper containing our answer?

Despite the difficult problem of choosing, assuming you can find it, it’s highly likely that there is software to reuse that will solve your problem, albeit with a learning curve and possible modification to that software. For the business, this is great: it means things get done faster.

The correlation with staff happiness

I am a backend engineer by training, and the landscape of storage technologies has never been more diverse: hundreds of databases, each with unique design properties, exist in relatively mature states, actively maintained and often associated with notable technology companies that happened to be the first to solve this problem in the open. Unless you are working in an industry that is wrapped in red tape, such as finance, it’s likely that there’s always going to be something out there that does the job for you.

But what sort of effect does this have on the job satisfaction of developers? Karl L. Hughes argued that the bulk of software engineering in 2018 is just plumbing; that most production engineers are responsible for composing multiple pieces of existing technology in order to solve a business problem.

This, of course, means that the number of unique problems that one gets to solve from scratch, such as the challenges set in Computer Science coursework, or the fundamentals that were worked on in the halcyon days of computing, is greatly reduced. For the most creative and talented engineers, this can lead to boredom. Yet another CRUD API attached to Postgres to drive some JavaScript widget? Yawn.

Some NIH will give your best staff something to really get their teeth into. A chance to create. A chance to contribute something original. An incentive to remain at your company. Under which conditions should we allow original and innovative projects to occur?

Why we build from scratch

There are some areas of the computing industry working up against the cutting edge on tough, innovative problems. Consider the current race to produce the best hardware to support the needs of machine learning at scale, and the software frameworks being written to best utilize that compute power.

This new paradigm of computing has a lot of low level, challenging and exciting problems to solve. Take Google’s TPUs (specialized hardware for machine learning) and TensorFlow (a software framework for machine learning):

  • They are solving a problem that has clear market value and customer need. The current generation of data scientists and machine learning practitioners need the software frameworks and performant compute power in order to do their job more effectively. Their ability to do so drives innovation in most, if not all areas, of our industry, creating value everywhere.
  • They are attempting to solve a problem at scale. Demand goes far beyond what can be serviced by existing, non-specialized hardware and software. A scalable solution requires rethinking the problem and building something from scratch.
  • Demonstrable innovation encourages further buy-in. Continuing to develop the best hardware and software for doing machine learning is a large factor in encouraging engineers to choose GCP as a cloud provider, increasing its stickiness in organizations, making Google even more money.

The case for building from the ground up here is clear, and that’s all well and good: but you may be reading this and coming to the conclusion that your company is doing nothing remotely as interesting as solving the future machine learning needs of the whole industry (and yes, that is probably true; not many companies are working on problems at this scale outside of the FAANG group).

However, these principles can guide your approach to knowing the right inflection point to build something from the ground up.

When you should build something yourself

As mentioned previously, it’s been 17 years since Joel’s original post on NIH. The industry has changed: more technology is out there to reuse than ever before, and it is more mature and better supported. Hardware is also cheaper, meaning we can get away with less optimized implementations, because that slightly more powerful instance on AWS only costs us a little bit extra.

One could argue that the problems that many software creators face require less innovative solutions: many venture-backed companies are centered around solving a unique disruptive business problem with technology, rather than solving a unique technology problem.

In order to unlock the door to being able to build innovative technology yourself, it is important first to prove that your product has clear product-market fit and that you’re reaching the scale point at which existing solutions no longer work. Before this point, it is extremely hard to justify the time and money spent reinventing that wheel into something more specialized.

In my experience, the path travelled before reaching a NIH inflection point looks something like this:

  1. While proving product-market fit, reuse as much as possible. If the product hasn’t proved itself, reuse, reuse, reuse. Time is of the essence. Even if you know that Postgres deployment won’t scale, you haven’t earned the customers yet to deserve the opportunity to build something more innovative. It just isn’t worth the investment.
  2. After proving product-market fit, creatively scale your existing components as much as possible. During an initial growth phase, see how you can stretch the existing technologies. Even if you still know that singular Postgres deployment won’t scale, running a heavily sharded set up with your own load balancing is much less development effort than building your own storage system.

From that point onwards the choice to build your own technology will stem from one of two situations:

  1. A proposed high-value feature cannot be achieved by off-the-shelf components. Here, the future of the product simply cannot be supported by any existing reused solution. You have no choice but to roll your own.
  2. Your scale begins to push up against what feels like the limit for a given technology. Here, for example, predictions for the next 18 months of usage will highlight the current system not being able to run fast enough, and you can’t find any examples of other installations coping at that level of scale. Here it’s easier to bet on yourself, as you have superior domain knowledge of what your customers are trying to do, compared to adapting an existing generalized system even further to undocumented boundaries in production.

Going it on your own

If you find yourself in one of the two situations described above, then this is an exciting moment in time. You’re pushing up against a problem at a scale that requires real innovation. You’ve also got a product that’s performing well enough to be worth the investment of effort. You’ve earned the ticket to go it alone.

This, however, is where it starts to get hard:

  • Progress will be much slower than reusing existing software.
  • Progress will be much less predictable as you repeatedly hit tough challenges and unknowns.
  • Progress will be much less transparent and understandable by non-technical people in the business.

But, at last, you’re working on something truly innovative.

Good luck.

Is the HP Way still relevant today?

comments 6

Bill Hewlett and Dave Packard at work.

Linting code

When many engineers are contributing code to the same application, there needs to be an agreement on how that code should be written. Ideally, when reading a codebase, the style and logical layout should feel like it was written by one person, thus making it easier to understand.

This stylistic and structural consistency can be achieved by the maintainers of that codebase specifying contribution guidelines which are enforced during code review, such as the guidelines for the Atom text editor, and by use of automated tools such as linters that automatically run on each build and flag to the programmer as to whether there are stylistic inconsistencies with the proposed addition. Linters can also perform static analysis and highlight potential bugs.

Defining and enforcing code consistency is an expected part of maintaining sanity during growth of an application. But what about ensuring management consistency during the growth of an organization? The leaders in an organization, through their actions, demonstrate how they expect others to act. But, with increasing size and geographic disparity, how can we guarantee that the values that the founders abide by are continued to be replicated three, five or ten levels down the org chart?

Is management open to interpretation?

The implementation of management, compared to the implementation of coding standards, is subject to much wider interpretation. We’re talking about human social behavior here, after all. How each manager decides to do their job is very much a function of their personality, their morals and ethics, and their interpretation of what it means to be a leader. Therefore this interpretation of the perceived management standards will be much more varied than the interpretation of coding standards.

But are there management standards? A manager can find themselves without anything in particular to interpret, other than what they see going on around them combined with their own ideal that they want to work towards, which is either learned from people they’ve worked with in the past, or guided by a collective wisdom from reading around the subject, or by modeling characteristics they’ve seen in famous, successful leaders. This can work well assuming the organization is full of highly capable, self-motivated people.

However, if toxic people lead an organization, this culture often cascades downwards, because other managers look to the toxic characters for an example of how they should be acting in order to progress their own careers. The current Trump administration has been described as a “Lord of the Flies” leadership style, especially so when considering the number of high profile sackings and the perceived mental instability of their leader. Trump’s leadership style has also been described as “emotional abuse”. Is it any surprise that there are so many other malignant characters drawn into the dark vortex?

Towards consistency

Is it possible to move towards a definition of how people should be managed in an organization, much in the same way that one would expect coding standards to be defined, that ensures some consistency in the actions of leaders and managers? I believe so.

Consider how Shakespeare is still continually reinterpreted in new ways, even though over 400 years have passed since his death. There are classical interpretations of Shakespeare, aimed at the purist, and there are modern interpretations, such as Romeo + Juliet, McKellen’s Richard III, and the translation of The Taming of the Shrew into 10 Things I Hate About You. The adaptations are aimed to be meaningful to a different audience, whilst still delivering the essence of the original story.

Consider what happens with and without definition of values:

  • Without definition of values, the company evolves organically, for better or worse. Without defining how people should be treated, the canonical ethos of the founders evolves over time, much in the same way that the date assigned to Christmas, and the festivities around it, has evolved. Existing staff leave, new staff join, and the essence of how the founders wanted the original culture to be is lost in translation.
  • With definition of values, new reinterpretations are still true to the source. Written values can be reinterpreted in different ways to suit the staff that are implementing them. Not all staff in the organization will be able to emulate the personality traits of the founders, however, they can interpret their values and bring them to life in their own unique way, in their unique location, with their unique culture.

The HP Way

A company that is famous for the longevity of its values is Hewlett-Packard. Despite their modern presence as a predominant producer of laptops and printers, were once an incredible Silicon Valley success story. In fact, they were the Silicon Valley success story.

Founded in a rented garage in Palo Alto in 1938 (yes, 1938), Bill Hewlett and Dave Packard began doing part-time electrical engineering projects. After taking an initial investment of $538, and tossing a coin to decide whether they should be called Hewlett-Packard or Packard-Hewlett, they produced their first successful product: the HP200A low-distortion audio oscillator, a new design that was over half the price and of much better quality than others on the market. Walt Disney bought eight and used them to test the sound engineering of his psychedelic masterpiece, Fantasia.

From humble beginnings, over the following decades, HP grew through the semiconductor device era and began to produce electronic instruments such as oscilloscopes and calculators and continued through to computers and printers. Along this journey, the HP leadership achieved some landmark cultural implementations: establishing a health insurance plan for all employees in 1942, designing one of the first open plan offices in the same year, giving all employees with six months of service stock in the company in 1957, opening HP R&D Labs in 1966, and introducing flextime in 1973 for the wellbeing of their employees. These benefits are still seen as flagship items on job advertisements in 2018: HP had them a long time ago.

The front cover of the 1980 internal booklet describing the culture of HP.

Another artifact of the culture, also a radical act at the time, was the creation and adoption of the HP Way, a working philosophy for the company. Note that a working philosophy is quite different from corporate objectives, or a vision for the future; it intends to capture the manner in which the company should operate: how employees should treat each other and how the organization should maintain these values as it grows.

A scan of the paper booklet published in 1980 is a worthwhile read. It predates more recent “culture books”, which tend to be a collection of thoughts from employees about why working at a particular company is cool. It is a solid piece of thoughtful leadership work that stands the test of time.

The HP Way can be summarized in 5 values:

  • We have trust and respect for individuals.
  • We focus on a high level of achievement and contribution.
  • We conduct our business with uncompromising integrity.
  • We achieve our common objectives through teamwork.
  • We encourage flexibility and innovation.

The HP Alumni website has an expansion of each of these values with additional explanation.

Culturally, early to mid-stage HP must have been doing something right. Many staff spent their whole careers at the company. The HP Memory Project has lots of inspiring stories of what it was like to work there in the earlier years of the company, and it is clear that those with stories featured were incredibly proud of their job and appreciative of their colleagues.

Brandwatch values

The Brandwatch values.

The company that I work for – Brandwatch – is much younger than HP. We’re about 11 years old. Recently the company went through the exercise of defining our own values, which were to be accountable, authentic, bold, brilliant and connected. In July, the entire company came together in person for an entire day dedicated to exploring our own personal values and how they relate to those of the company. Many of our nearly 400-strong staff travelled from far reaching corners of the globe to be present, which represents a serious investment of their time and sacrifice to their commitments outside of work.

Naturally, many, including myself, can approach these days with an underlying worry that people wouldn’t fully engage in the exercises, and with a skepticism stemming from whether defining company values is actually meaningful and not just another corporate exercise to enforce the status quo. However, the day itself was fantastic and put my initial hesitation to shame.

Every single person that I interacted with engaged at a deep emotional level about the values that they used to orient their moral compass in their own personal lives, whether that is growth, having fun, building relationships, earning and keeping trust, or serving others. I gained some inspiring insight into people that I had only just met for the first time and had some extremely meaningful conversations. The morning was spent defining our ten most important personal values, narrowing them down to just one, and then discovering other people who were motivated in the same way.

We spent the afternoon discussing in groups using the World Cafe format. Around one of the Brandwatch company values, we were asked to explore what we think it means, how we embody that behavior, how we can tell when that value is not happening, and how we can encourage more of it. Again, despite my initial, potentially British, skepticism, each group that I rotated around was truly passionate about discussing the purpose of their work, both for themselves and for the wider company.

I left the day feeling uplifted and connected to those that I work with.

Brandwatch staff from above. Credit to Tom Millward for the photograph.

The kinds of conversation that were enabled by company values so very rarely happen in day to day work, where you are knee deep in email, or fixing a production issue, or juggling five priorities at once. The simple act of stepping away, physically and mentally, for just one day, was enlightening: a reminder that you work with humans, rather than job roles. I feel closer to all of them.

So is the HP Way still relevant today? Are values still effective in uniting large groups of people from around the world? I’d say that yes, they definitely are.

The importance of writing

comments 2

The three Rs: reading, writing and arithmetic.

As the tenet of basic education, children spend years learning how to input, manipulate and output information. It forms the foundations that allow more complex subjects to be taught such as science and literature.

However, some can often regard writing as an old skill, or a lost discipline that isn’t as important with the prevalence of communication via technology. Yet, I would argue that being able to write with clarity, skill and artistry is one of the most impactful skills that you can wield as a leader. It can elevate your presence and increase your impact dramatically.

Yes, you should write

Popular tropes of leadership in mainstream media have been loud and brash. Consider how the “big boss” character is portrayed in fiction, and how Alan Sugar or Donald Trump bounce around and deliver offensive quips on The Apprentice. Yet, this leadership is shallow. One needs only to observe the current term of the 45th President of the United States to see that leadership by brazen personality can only get you so far.

A resurgence of the academic leader is here.

Jeff Bezos, Elon Musk, Warren Buffett. Leaders who can communicate succinctly, leaders who can write. Leaders who are embracing one of the oldest skills that humans have had at their disposal, but are doing so at a time where distribution of written information globally is as easy as the click of a button.

Let’s enunciate further on Bezos. His annual Amazon shareholder letters are extremely well written. The 2017 incarnation explores ways to implement high standards, and whether they are intrinsic or teachable, universal or domain specific. That is far beyond just a table of profit and loss numbers, which is equally impressive, regardless. In fact, Bezos is a true champion of the written form. Recent information has shown that his executive meetings take the form of reading written memos as a way of making sure that a context is created for constructive debate and discussion. I have tried this out myself in recent important meetings. It works very well.

Why writing is so impactful

I am a firm believer in the power of good writing at work. I am also acutely aware that the time available to write well feels like it is increasingly diminishing: there is generally an expectation for lightning fast responses over precise communication, especially with the increase in popularity of chat software and direct messages when compared to email. However, I still find the time to write properly in my communication, and I will continue to do so.

Writing is a normalizing medium. No matter what you look like, how old you are, how you speak, or how confident you are, you can sit on your own and formulate your thoughts, draft and re-edit, and when you’re done, they can be presented in the same standard form as Bezos, Hemingway or Dickens: words on the page; a pure transfer of ideas from one brain to another with no judgement or discrimination based on creed, color, age or gender. An impactful message is an impactful message.

In an increasingly distributed world, where a senior member of an organization will rarely have all of their staff in the same physical location, the written word provides everybody with equal access to their leader, no matter whether they are sitting ten meters away from them, or whether they are oceans apart. Regular, frequent and considered communication unites. It aligns minds globally. It is a leveling platform.

In addition to communicating widely and equally, writing effectively is a differentiator. A considered written piece will almost always be able to persuade better, and to deliver a message better than a spoken argument given without preparation. I have had many experiences where retracting my vocal argument and committing my thoughts to paper has worked in my favor. A written argument can be constructed over many hours, even if it is only a few paragraphs long, ensuring maximum clarity and impact. You can take the space to ensure you represent your best self.

One of the reasons that I opted to begin writing regularly – and believe it or not – I have published an article every week for over a year now, is that it is a fantastic way of crystallizing one’s own thoughts. These articles are only a portion of the writing that I do each day. I write to others at work. I write to myself, privately. Continued focus on writing has improved the way that I think about problems and construct arguments. It helps me clarify the algorithmic paths of concepts in my brain, because it forces me to walk those lines slowly and to articulate them as I go along.

Writing as a leader

There are a number of ways in which you can use writing to improve your leadership ability, whether or not you have explicit control over a team, department or organization. Quality written communication is engaging and ensures that others will listen to your viewpoint, regardless of your place in the org chart.

Here’s a few ways in which you can practice being more impactful at work through your writing:

  • Consciously step back and think before sending messages. This shift in mindset is one of the easiest yet most effective things you can do. When you write a message to someone, resist the urge for it to be throwaway. Stop. Read it again. What would you think if you received this message from someone else? Could you make it better in order to portray exactly what you want? Does it represent the best communication that you can produce?
  • Journal your day to yourself. Even if you have no designated output for your writing, you can write for an audience of one: yourself. Start by briefly summarizing what you’ve done each day in a journal or private document. Or perhaps you’d like to take time each morning to describe how you’d really like the present day to play out. Either way, you can use this opportunity to plan or reflect and to improve your writing ability whilst doing so.
  • Think through problems by writing them out. Using whiteboards is an effective way of collaboratively thinking through problems, but the best way that I have found to help myself think through problems is to write them down, explain to myself how I feel about them, and to propose solutions to myself. The benefit of doing this is that once you’re happy with your written exploration, you can share the document with others for their feedback.
  • Write your thoughts for the subject of meetings ahead of time. Not only does this crystallize your thinking and allows you to come prepared and better able to convince others of your viewpoint, you can also share it with the attendees beforehand to help center the discussion.
  • Publish regular newsletters on your progress. Despite the belief that we all receive way too much email, from experience I have found that regular newsletters that people only have to read rather than issue a response to are well-received. People like updates! Get into the flow of writing regular written updates to your manager, team, stakeholders, or department.

And how do you get better at writing?

Many people who have read articles that I have written have asked me for my opinion on how to get better at writing. Let me take this opportunity to say that I still have a lot to improve before I would call myself a good writer. As such, I have no recommended course or educational materials to share, other than two extremely basic pieces of advice.

Firstly, the best way to get better at writing is just to write as much as you possibly can. I have found it similar to learning a sport, or an instrument, or a programming language: you simply need to practice, practice, practice. With time your speed will increase and you will find your voice. You will notice how you spend more time writing and less time editing. There will come a day that your stream of consciousness becomes almost exactly what you write, and what a liberating feeling that is! Try and capitalize on any opportunity to make your writing better. Every direct message, every email, every comment on a pull request. Take that bit of extra time to read and review. Use a thesaurus and learn a new word. Try to find opportunities to use semi-colons. Portray the feeling pictured in an emoji in your words, instead.

The only other piece of advice I can muster on how to improve your writing is to read. A lot. But don’t limit yourself to reading work-related materials. Read short stories on Medium. Raid the charity shop for that stack of Penguin classics and throw yourself into Orwell or Shelley. Work out how Zadie Smith and Kazuo Ishiguro pace their writing. How does that compare to Iain Banks? Consider the journalistic storytelling style of Steve Levy. Read a self-help book and study how the author makes their message so compelling, even if it may be tautological. Just read.

Writing is so powerful. Wield that power for your benefit.