What does beta really mean?

comments 2
Growth
Beta. Just look at it. Majestic!

What’s the longest period of time that you’ve let software that you’ve shipped stay in beta? A week? A month? A year? 

What does beta even mean in the SaaS world of continuous delivery? Is it a relic of the past?

Let’s explore further.

Showing my age

Sarge.

It’s 1999. 

In the very early hours of a Sunday morning in April I am hunched over a desk, curtains closed, in my bedroom on the first floor of my childhood home. My face is lit dimly by the glow from a CRT monitor. 

I’ve been downloading a file called Q3Test.zip for most of the last twenty four hours, praying that my 56K modem doesn’t disconnect before it finishes. 

The dialog box reads 99%. 

This file was the public beta test of Quake III Arena, which to a die-hard Quake and Quake II fan like me was like having my birthday and Christmas rolled into one. It was the first chance to play the game that I was going to be spending many, many hours mastering (or, at least, trying to).

Beta releases of games were a big deal back then. It was a chance to feel part of the inner circle.

The anticipation of games in development was primarily generated by reading physical copies of PC Zone or PC Gamer, which would be the only reason my parents could convince me to come to the supermarket. 
I would browse the screenshots of current builds, which were hyped to fever pitch by the writers. I would imagine the secretive locations in which they were being created.

After many months of following the development of a game in a magazine, news would break that there was a beta available to download, and I would scramble online to IRC chat rooms to find out where it was hosted, and whether the mirror was reliable.

Reliability was key: in the late 90’s the fastest consumer Internet connection available in my area was dual ISDN, and that was way too expensive for my family. I needed somewhere to download it from that would not kick my little modem off while it was hammered by kids with fibre connections in the Nordics. 

99.6%… come on…

Stages

Following the games industry enabled me to begin to learn about how software was made. The beta stage was usually where people like me began interacting with it, as I was an avid gamer as a teenager, and I saw it as a matter of pride to be the first at my school to play the latest releases.

Jeff Atwood outlines the stages:

  • Pre-alpha: The software is under active development and isn’t ready for consumption by users.
  • Alpha: The software is ready enough for internal testing.
  • Beta: The software is ready for external testing, but has known bugs or limitations.
  • Release Candidate: The software is almost ready for final release and only tightly scoped bug fixes are allowed.
  • Gold: It’s done to the best of our knowledge.

Note: if you’ve ever been naughty and downloaded leaks of games, the “RC” in the filename stands for Release Candidate…

Now, for betas specifically, there are closed and open betas. 

A closed beta typically involves an invite-only period featuring users that have been selected to use the software, either because the number of concurrent users needs to be controlled, or because they are friendly customers who aren’t bothered by some bugs, or both. 

Open betas are where normal people like you and me get involved. As much as I would dream of id Software inviting me to their office, unfortunately it never happened.

What about in SaaS?

Now that I’m older, I don’t work in games; I work in SaaS. 

It’s an industry that moves very fast. You could posit that the software development stages listed above seem fairly archaic compared to the mindset of shipping to production continuously.

However, the formalism of the aforementioned stages is a defense mechanism: very large software projects such as AAA games and operating systems can’t ship broken code as the consequences would be devastating. 

Either a company could be humiliated by shipping millions of copies of physical media containing a broken build, or, even worse, a poorly tested operating system could ruin a person’s computer.

But, fortunately, in SaaS, we deliver via the browser and we can ship updates whenever we want. With the expectations of the expedience of delivery being high, we continually ride a line between acceptable speed of delivery and acceptable quality

This is why beta programs are so popular: allow an increase in the acceptable speed of delivery and a reduction in the acceptable quality. Win-win, right? 

Atwood pointed out two trends in 2008 which are more true than ever ten years later:

  • The definition of beta grows more all-encompassing and elastic every year.
  • We are awfully eager to throw alpha quality code over the wall to external users and testers.

Faster, faster, faster!

A mantra from Facebook’s past. It’s now “Move Fast With Stable Infra”, which isn’t quite as catchy.

Despite some parts of the software industry still following the release stages mentioned above, SaaS and the drive towards continuous delivery has created a trend of perpetual beta. 

There are a lot of large websites that continually have functionality in beta. Sometimes the entire service is beta: Gmail famously had a beta label for many years and so did Flickr.

I could summarize the current climate that I experience in my area of SaaS as:

  1. Expedited delivery of something of value to customers is the number one priority of any development effort.
  2. Done is always better than perfect.
  3. Perceived speed of development teams is the most important trait for Engineering in competitive VC-driven marketplaces. 

The last point may be cynical, but sometimes I feel it.

In 2010, an article suggested that delivering software using agile encouraged perpetual beta. But does this incremental, speed-first completion-second delivery mean that nothing ever gets finished properly, or is it actually useful?

Usage of beta in SaaS

In my own experience, there is often a lot of confusion around what a beta release actually means, and this confusion often compounds when views of engineers are mixed with those of non-engineers. 
Here is a list of situations that I’ve seen a beta flag used:

  • Features that are still slightly buggy but generally work and deliver value.
  • Features that aren’t yet complete (i.e. have a subset of proposed functionality).
  • Features that may or may not remain in the software based on customer feedback. This includes innovative or experimental features.
  • Features that are conceptually complete but haven’t yet reached the desired level of engineering quality (e.g. scaling, monitoring, or infrastructure robustness).

If this is the case, then there can be two forces that can drive the labelling of a feature as beta:

  1. Product: The feature or product isn’t complete or useful enough for all users to find value in it.
  2. Engineering: The feature or product is not scaled or robust enough for all users to have an acceptable experience.

This is very much a trade-off between Product and Engineering, and requires both disciplines to come  together to make a call on the use of the label, and most importantly, where progress on that software needs to get to before the label can be removed. 

And yes, it should get removed.

A healthy dialogue between both sides ensures that new features get into the hands of users through beta programs early enough so that feedback can be collected, but late enough so that they won’t be disappointed with what they see. You only get one chance at a first impression.

The positive side of beta

There is an overwhelmingly positive side to being able to run beta programs, and in users being more open to using beta software, even at work.

In fact, I’ve seen the excitement that I used to experience as a teenager when gaining access to beta releases of new games manifest in some of our own users when we invite them into beta programs that we are running. (OK, I exaggerate, but some do get very excited.)

By running beta programs in SaaS, there are a number of tangible benefits:

  • They allow you to get feedback much earlier in the development process.
  • They encourage more creative and innovative ideas because the impact of failure is much lower.
  • They allow your infrastructure to go under an initial stage of real load, rather than the simulated load you have done during development.
  • They can strengthen your relationship with your most engaged customers by giving them something new and exciting to play with earlier than everyone else.

We learn a lot from our beta programs, and some of our customers give such great feedback we should probably be paying them rather than the other way round.

Abuse of beta

However, for all of the positive talk about beta software, I’ve also seen the beta label be abused. 

Here are a list of things that I feel beta software should not be, yet have witnessed:

  • Untested. I mean come on, that’s just incredibly lazy. Your users are not your QA, and you shouldn’t feel comfortable or proud giving untested software to them.
  • Finished or indefinitely paused. Sticking a beta label on something that will never be completed is an anti-pattern. The point of a beta is to learn something, and should not be used as an excuse to ship substandard or orphaned work.
  • Used to mask diffidence. There is nuanced case where the beta label is used as a proxy for being nervous or embarrassed about the quality of what is being shipped.

Whenever I’ve observed the above happening, it’s often not via the intention of the team.

The first two situations often occur when there is external pressure to ship regardless of quality, such as when a feature has been promised and needs to go out to save face. Alternatively, they can both happen when reprioritization causes a team to drop what they are doing and work on something else immediately, and the “beta” launch is the salvage from the wreck.

The last situation stems from a nervous team, and may not even be indicative of their anxiety of how their users will react, but rather how important internal people will react, such as their managers, peers or the executives. The beta label is used as a buffer for any initial disappointment with the shipped feature.

Abuse of the beta label is often not because you have a poor performing team: you may have a poor performing culture or organization. It’s worth being mindful of this happening.

To conclude

Try not to spend over 5 years in beta. This product seemed to take off, though.

Beta releases, when handled correctly, are excellent tools to ship faster, please users, generate excitement and get valuable feature and infrastructure feedback. 

However, you must always ensure that beta releases are being done for the right reason and are not being used to hide smells of poor organization or a culture that doesn’t care about the quality of the code that is being given to its customers.

Beta is for learning and experimentation. It is not an excuse for unfinished work.

2 Comments

  1. Ben Strawson says

    Great article, but you never told us if you finished downloading Quake III Arena :-). Did it complete? Was it what you expected for a beta?

Leave a Reply

Your email address will not be published. Required fields are marked *