Mnemosyne

Recently, we redesigned the main storage architecture of one of our products.

We did so by writing a distributed bitmap index of our own that was exactly suited to the type of searches and aggregations that users perform frequently. My colleague, Phil Messenger, who served as the lead engineer on the project, gave a technical talk at Devoxx 2018 about how the system was conceived and designed.

While this project was running, and despite my absolute trust in the team’s technical ability to deliver it, I couldn’t help feel that familiar nag in the back on my mind: are we definitely sure that this hasn’t been built elsewhere? We were convinced that it hadn’t, but it is always hard to be one hundred percent sure.

Since it was a new distributed system, development and production rollout was challenging and proceeded at the cost of developing less customer-facing features. This reality was a harder sell to our product team, as our application, at the time, was still playing catch-up in the market.

In retrospect, the storage system has been a resounding success. It is exponentially faster than our previous Elasticsearch index, scales linearly with usage, and has future-proofed the next few years of growth for this particular product.

But building something from scratch, rather than composing it from existing open source technologies, is a risky strategy. What if we’d missed an existing technology when we did our research? What if it had turned out much harder than originally expected? What if we couldn’t get buy-in from management and manage expectations accordingly?

How did we know we hadn’t we fallen under the spell of Not Invented Here syndrome?

Not Invented Here

Not Invented Here, or NIH for short, is a mindset that embodies a reluctance to reuse or purchase existing products or technologies because they originate from the “outside”. The NIH mindset rejects perfectly capable solutions that lie beyond the boundary of a company, department or team.

But why?

In technology specifically, NIH highlights situations where engineers side with reinventing the wheel due to a belief that building technology in-house will be better, quicker and cheaper than using an existing solution. In many cases, and especially with the current wealth of excellent open source software available for use, this is an anti-pattern and a big, bad smell to be picked up by management.

Assuming a suitable technology exists, why not use it, even if it does require a bit of modification? After all, isn’t software reuse the biggest lever that great engineers can use for getting the job done quickly and reliably?

Well, not always.

NIH as a differentiator

In 2001, Joel Spolsky wrote an article outlining his experience at Microsoft where the Excel team would reinvent the wheel in order to reduce the external dependencies in their software. They even had their own C compiler. The result? Excellent cross-platform compatibility and a small executable. He argued that when working with an outstanding team, reinventing the wheel can be a differentiator.

He concluded the article with a recommendation:

If it’s a core business function – do it yourself, no matter what.

Sound advice.

After all, as Joel states, if you’re a Web accounting company, then you should probably write your own accounting software. If you buy someone else’s accounting engine and re-skin it, you’re not really adding much value, and if you’re a technology company without valuable technology, what are you?

Your decisions are often smaller

So far, so good. You’ll want to make sure that you build the technology that supports the core business.

But as a manager, or as an engineer, you’re going to be thinking about whether to reuse or build components at a much more granular level. Yes, you may be writing your own Web accountancy software in general, but of which components should it be composed?

Should you use this storage system, or that one, or should you write our own? Should you use this CSS pattern library for your UI, or should you start writing one yourselves?

When Joel’s article was written 17 years ago, the open source community had not seen the surge in popularity and progress enabled by centralization around communities like Github. There’s more code to reuse than ever before, and it is much easier to find.

I would assume that reuse discussions in 2001 had a slightly different focus: a decision to possibly choose between implementing an established project from Apache or GNU, or between purchasing a system from another company, or building it from scratch.

I can see how many decisions to build rather than reuse could have stemmed from high licensing prices, since SaaS was in its infancy – Salesforce was just 2 years old – or from lack of faith in open source projects, because they moved slower.

Paralysis of choice

Although now we have the luxury of being able to find lots of software to reuse, there is another problem: which should we choose? There is a huge search space: one can choose from many open source implementations at differing levels of maturity, or from numerous paid-for SaaS offerings with a matrix of pricing options, to on-premise installations, or, of course, one could build their own. Argh!

Additionally, companies that operate with a closed-source model such as Google and Facebook also publish academic articles and white papers detailing how their technologies work. The algorithmic challenge or architectural quandary that is a team is facing may have also been solved before. Some papers are reimplemented as open source projects in their own right, such as Apache HBase which is based on Bigtable, but for those that aren’t, what search keywords do we use to find the paper containing our answer?

Despite the difficult problem of choosing, assuming you can find it, it’s highly likely that there is software to reuse that will solve your problem, albeit with a learning curve and possible modification to that software. For the business, this is great: it means things get done faster.

The correlation with staff happiness

I am a backend engineer by training, and the landscape of storage technologies has never been more diverse: hundreds of databases, each with unique design properties, exist in relatively mature states, actively maintained and often associated with notable technology companies that happened to be the first to solve this problem in the open. Unless you are working in an industry that is wrapped in red tape, such as finance, it’s likely that there’s always going to be something out there that does the job for you.

But what sort of effect does this have on the job satisfaction of developers? Karl L. Hughes argued that the bulk of software engineering in 2018 is just plumbing; that most production engineers are responsible for composing multiple pieces of existing technology in order to solve a business problem.

This, of course, means that the number of unique problems that one gets to solve from scratch, such as the challenges set in Computer Science coursework, or the fundamentals that were worked on in the halcyon days of computing, is greatly reduced. For the most creative and talented engineers, this can lead to boredom. Yet another CRUD API attached to Postgres to drive some JavaScript widget? Yawn.

Some NIH will give your best staff something to really get their teeth into. A chance to create. A chance to contribute something original. An incentive to remain at your company. Under which conditions should we allow original and innovative projects to occur?

Why we build from scratch

There are some areas of the computing industry working up against the cutting edge on tough, innovative problems. Consider the current race to produce the best hardware to support the needs of machine learning at scale, and the software frameworks being written to best utilize that compute power.

This new paradigm of computing has a lot of low level, challenging and exciting problems to solve. Take Google’s TPUs (specialized hardware for machine learning) and TensorFlow (a software framework for machine learning):

• They are solving a problem that has clear market value and customer need. The current generation of data scientists and machine learning practitioners need the software frameworks and performant compute power in order to do their job more effectively. Their ability to do so drives innovation in most, if not all areas, of our industry, creating value everywhere.
• They are attempting to solve a problem at scale. Demand goes far beyond what can be serviced by existing, non-specialized hardware and software. A scalable solution requires rethinking the problem and building something from scratch.
• Demonstrable innovation encourages further buy-in. Continuing to develop the best hardware and software for doing machine learning is a large factor in encouraging engineers to choose GCP as a cloud provider, increasing its stickiness in organizations, making Google even more money.

The case for building from the ground up here is clear, and that’s all well and good: but you may be reading this and coming to the conclusion that your company is doing nothing remotely as interesting as solving the future machine learning needs of the whole industry (and yes, that is probably true; not many companies are working on problems at this scale outside of the FAANG group).

However, these principles can guide your approach to knowing the right inflection point to build something from the ground up.

When you should build something yourself

As mentioned previously, it’s been 17 years since Joel’s original post on NIH. The industry has changed: more technology is out there to reuse than ever before, and it is more mature and better supported. Hardware is also cheaper, meaning we can get away with less optimized implementations, because that slightly more powerful instance on AWS only costs us a little bit extra.

One could argue that the problems that many software creators face require less innovative solutions: many venture-backed companies are centered around solving a unique disruptive business problem with technology, rather than solving a unique technology problem.

In order to unlock the door to being able to build innovative technology yourself, it is important first to prove that your product has clear product-market fit and that you’re reaching the scale point at which existing solutions no longer work. Before this point, it is extremely hard to justify the time and money spent reinventing that wheel into something more specialized.

In my experience, the path travelled before reaching a NIH inflection point looks something like this:

1. While proving product-market fit, reuse as much as possible. If the product hasn’t proved itself, reuse, reuse, reuse. Time is of the essence. Even if you know that Postgres deployment won’t scale, you haven’t earned the customers yet to deserve the opportunity to build something more innovative. It just isn’t worth the investment.
2. After proving product-market fit, creatively scale your existing components as much as possible. During an initial growth phase, see how you can stretch the existing technologies. Even if you still know that singular Postgres deployment won’t scale, running a heavily sharded set up with your own load balancing is much less development effort than building your own storage system.

From that point onwards the choice to build your own technology will stem from one of two situations:

1. A proposed high-value feature cannot be achieved by off-the-shelf components. Here, the future of the product simply cannot be supported by any existing reused solution. You have no choice but to roll your own.
2. Your scale begins to push up against what feels like the limit for a given technology. Here, for example, predictions for the next 18 months of usage will highlight the current system not being able to run fast enough, and you can’t find any examples of other installations coping at that level of scale. Here it’s easier to bet on yourself, as you have superior domain knowledge of what your customers are trying to do, compared to adapting an existing generalized system even further to undocumented boundaries in production.

Going it on your own

If you find yourself in one of the two situations described above, then this is an exciting moment in time. You’re pushing up against a problem at a scale that requires real innovation. You’ve also got a product that’s performing well enough to be worth the investment of effort. You’ve earned the ticket to go it alone.

This, however, is where it starts to get hard:

• Progress will be much slower than reusing existing software.
• Progress will be much less predictable as you repeatedly hit tough challenges and unknowns.
• Progress will be much less transparent and understandable by non-technical people in the business.

But, at last, you’re working on something truly innovative.

Good luck.