What does beta really mean?

comments 2
Growth
Beta. Just look at it. Majestic!

What’s the longest period of time that you’ve let software that you’ve shipped stay in beta? A week? A month? A year? 

What does beta even mean in the SaaS world of continuous delivery? Is it a relic of the past?

Let’s explore further.

Showing my age

Sarge.

It’s 1999. 

In the very early hours of a Sunday morning in April I am hunched over a desk, curtains closed, in my bedroom on the first floor of my childhood home. My face is lit dimly by the glow from a CRT monitor. 

I’ve been downloading a file called Q3Test.zip for most of the last twenty four hours, praying that my 56K modem doesn’t disconnect before it finishes. 

The dialog box reads 99%. 

This file was the public beta test of Quake III Arena, which to a die-hard Quake and Quake II fan like me was like having my birthday and Christmas rolled into one. It was the first chance to play the game that I was going to be spending many, many hours mastering (or, at least, trying to).

Beta releases of games were a big deal back then. It was a chance to feel part of the inner circle.

The anticipation of games in development was primarily generated by reading physical copies of PC Zone or PC Gamer, which would be the only reason my parents could convince me to come to the supermarket. 
I would browse the screenshots of current builds, which were hyped to fever pitch by the writers. I would imagine the secretive locations in which they were being created.

After many months of following the development of a game in a magazine, news would break that there was a beta available to download, and I would scramble online to IRC chat rooms to find out where it was hosted, and whether the mirror was reliable.

Reliability was key: in the late 90’s the fastest consumer Internet connection available in my area was dual ISDN, and that was way too expensive for my family. I needed somewhere to download it from that would not kick my little modem off while it was hammered by kids with fibre connections in the Nordics. 

99.6%… come on…

Stages

Following the games industry enabled me to begin to learn about how software was made. The beta stage was usually where people like me began interacting with it, as I was an avid gamer as a teenager, and I saw it as a matter of pride to be the first at my school to play the latest releases.

Jeff Atwood outlines the stages:

  • Pre-alpha: The software is under active development and isn’t ready for consumption by users.
  • Alpha: The software is ready enough for internal testing.
  • Beta: The software is ready for external testing, but has known bugs or limitations.
  • Release Candidate: The software is almost ready for final release and only tightly scoped bug fixes are allowed.
  • Gold: It’s done to the best of our knowledge.

Note: if you’ve ever been naughty and downloaded leaks of games, the “RC” in the filename stands for Release Candidate…

Now, for betas specifically, there are closed and open betas. 

A closed beta typically involves an invite-only period featuring users that have been selected to use the software, either because the number of concurrent users needs to be controlled, or because they are friendly customers who aren’t bothered by some bugs, or both. 

Open betas are where normal people like you and me get involved. As much as I would dream of id Software inviting me to their office, unfortunately it never happened.

What about in SaaS?

Now that I’m older, I don’t work in games; I work in SaaS. 

It’s an industry that moves very fast. You could posit that the software development stages listed above seem fairly archaic compared to the mindset of shipping to production continuously.

However, the formalism of the aforementioned stages is a defense mechanism: very large software projects such as AAA games and operating systems can’t ship broken code as the consequences would be devastating. 

Either a company could be humiliated by shipping millions of copies of physical media containing a broken build, or, even worse, a poorly tested operating system could ruin a person’s computer.

But, fortunately, in SaaS, we deliver via the browser and we can ship updates whenever we want. With the expectations of the expedience of delivery being high, we continually ride a line between acceptable speed of delivery and acceptable quality

This is why beta programs are so popular: allow an increase in the acceptable speed of delivery and a reduction in the acceptable quality. Win-win, right? 

Atwood pointed out two trends in 2008 which are more true than ever ten years later:

  • The definition of beta grows more all-encompassing and elastic every year.
  • We are awfully eager to throw alpha quality code over the wall to external users and testers.

Faster, faster, faster!

A mantra from Facebook’s past. It’s now “Move Fast With Stable Infra”, which isn’t quite as catchy.

Despite some parts of the software industry still following the release stages mentioned above, SaaS and the drive towards continuous delivery has created a trend of perpetual beta. 

There are a lot of large websites that continually have functionality in beta. Sometimes the entire service is beta: Gmail famously had a beta label for many years and so did Flickr.

I could summarize the current climate that I experience in my area of SaaS as:

  1. Expedited delivery of something of value to customers is the number one priority of any development effort.
  2. Done is always better than perfect.
  3. Perceived speed of development teams is the most important trait for Engineering in competitive VC-driven marketplaces. 

The last point may be cynical, but sometimes I feel it.

In 2010, an article suggested that delivering software using agile encouraged perpetual beta. But does this incremental, speed-first completion-second delivery mean that nothing ever gets finished properly, or is it actually useful?

Usage of beta in SaaS

In my own experience, there is often a lot of confusion around what a beta release actually means, and this confusion often compounds when views of engineers are mixed with those of non-engineers. 
Here is a list of situations that I’ve seen a beta flag used:

  • Features that are still slightly buggy but generally work and deliver value.
  • Features that aren’t yet complete (i.e. have a subset of proposed functionality).
  • Features that may or may not remain in the software based on customer feedback. This includes innovative or experimental features.
  • Features that are conceptually complete but haven’t yet reached the desired level of engineering quality (e.g. scaling, monitoring, or infrastructure robustness).

If this is the case, then there can be two forces that can drive the labelling of a feature as beta:

  1. Product: The feature or product isn’t complete or useful enough for all users to find value in it.
  2. Engineering: The feature or product is not scaled or robust enough for all users to have an acceptable experience.

This is very much a trade-off between Product and Engineering, and requires both disciplines to come  together to make a call on the use of the label, and most importantly, where progress on that software needs to get to before the label can be removed. 

And yes, it should get removed.

A healthy dialogue between both sides ensures that new features get into the hands of users through beta programs early enough so that feedback can be collected, but late enough so that they won’t be disappointed with what they see. You only get one chance at a first impression.

The positive side of beta

There is an overwhelmingly positive side to being able to run beta programs, and in users being more open to using beta software, even at work.

In fact, I’ve seen the excitement that I used to experience as a teenager when gaining access to beta releases of new games manifest in some of our own users when we invite them into beta programs that we are running. (OK, I exaggerate, but some do get very excited.)

By running beta programs in SaaS, there are a number of tangible benefits:

  • They allow you to get feedback much earlier in the development process.
  • They encourage more creative and innovative ideas because the impact of failure is much lower.
  • They allow your infrastructure to go under an initial stage of real load, rather than the simulated load you have done during development.
  • They can strengthen your relationship with your most engaged customers by giving them something new and exciting to play with earlier than everyone else.

We learn a lot from our beta programs, and some of our customers give such great feedback we should probably be paying them rather than the other way round.

Abuse of beta

However, for all of the positive talk about beta software, I’ve also seen the beta label be abused. 

Here are a list of things that I feel beta software should not be, yet have witnessed:

  • Untested. I mean come on, that’s just incredibly lazy. Your users are not your QA, and you shouldn’t feel comfortable or proud giving untested software to them.
  • Finished or indefinitely paused. Sticking a beta label on something that will never be completed is an anti-pattern. The point of a beta is to learn something, and should not be used as an excuse to ship substandard or orphaned work.
  • Used to mask diffidence. There is nuanced case where the beta label is used as a proxy for being nervous or embarrassed about the quality of what is being shipped.

Whenever I’ve observed the above happening, it’s often not via the intention of the team.

The first two situations often occur when there is external pressure to ship regardless of quality, such as when a feature has been promised and needs to go out to save face. Alternatively, they can both happen when reprioritization causes a team to drop what they are doing and work on something else immediately, and the “beta” launch is the salvage from the wreck.

The last situation stems from a nervous team, and may not even be indicative of their anxiety of how their users will react, but rather how important internal people will react, such as their managers, peers or the executives. The beta label is used as a buffer for any initial disappointment with the shipped feature.

Abuse of the beta label is often not because you have a poor performing team: you may have a poor performing culture or organization. It’s worth being mindful of this happening.

To conclude

Try not to spend over 5 years in beta. This product seemed to take off, though.

Beta releases, when handled correctly, are excellent tools to ship faster, please users, generate excitement and get valuable feature and infrastructure feedback. 

However, you must always ensure that beta releases are being done for the right reason and are not being used to hide smells of poor organization or a culture that doesn’t care about the quality of the code that is being given to its customers.

Beta is for learning and experimentation. It is not an excuse for unfinished work.

Project X

Leave a comment
Growth
“Ouch”, cries today. Photo by Johnson Wang on Unsplash.

In an open plan office in a galaxy not so far away…


Lisa rotates in her chair and looks upwards at the ceiling. She poses a question:  

“I’m just about to create the repository in Github. What should we call it?”

“Well, we’re not even sure what it’s going to do yet. I don’t even know if what they’re asking for is possible.”

Ben looks back towards his monitor, opens up Google and types in “cool project names” and hits the return key with vigor. Lisa scoots her chair over to Ben’s desk to look at the results. 

“That’s the one.”

“Oh yeah,” she replies.

Switching back to his initial browser tab, Ben types in “Project X” and creates the repository so that the Data Science team can start hacking around with ideas.

Lisa uploads the first file. It’s a Python script that prints “What on earth is Project X? 💁” when executed. Ben laughs. He commits a README.md that states:

The first rule of Project X is that you don’t talk about Project X.

Lisa stretches her arms into the air and cracks her knuckles. She sighs.


A gentle clinking sound repeats as Lisa whirls the milk into her teacup in the kitchen. She looks through the glass partition that separates her from the breakout area. 

The light in the open space is dimmed, and row upon row of chairs are packed tightly with what looks like the entire commercial team, listening intently to the presentation of the company roadmap.

Leaning against the kitchen wall, she watches the projection as screenshots of the latest iteration of the mobile app are shown. Laptops glow as the audience take notes.

Sipping her tea so that it doesn’t spill over, she turns to leave the kitchen, but is distracted by the reflection of a giant “X” on door. It’s coming from the presentation.

The slide says what she had immediately feared: “Project X”. The CEO is animated. The crowd are clapping. Lisa’s heart sinks. She fumbles with her phone to send Ben a Slack message.

Lisa: They’ve just presented Project X during the roadmap deck.

Ben: Are you serious?

Lisa: Yes.

Ben: But we don’t even know what it does yet.

Lisa: What do we do?

Ben: What did the slides say?

Lisa: It’s coming next quarter, and the whole of downstairs was hyped.

Ben bursts through the kitchen door. 

“What the hell is going on?”

“We are absolutely screwed,” replies Lisa, who is now pacing in circles.

“It’s literally a CSV file with two hundred lines of test data for a feature that hasn’t been designed. How are they announcing it already?”

Lisa stares into her mug. “Will they notice if I leave the country?”

“I’m looking up flights to Siberia,” replies Ben.


Lisa! Our rockstar data scientist! So good to see you!”

It’s the head of sales.

“Oh, hey Mia.”

“Let me tell you this, Lisa. I love the website. I love what you’re doing for this company. It’s going to absolutely blow the competition away. My team are going to push this so hard when it’s done.”

“Website? What do you mean?”

Double-you double-you double-you dot project x dot com. It looks incredible. The announcement last night on the webinar? Fantastic. Thousands of our customers watched it, Lisa. Thousands.”

“What?”

“I love it, Lisa. Love it!”

Mia is already out of the kitchen and involved in another discussion. Lisa walks back over to her team and taps Ben on the shoulder. After finishing the current line of code with a semi-colon, he removes his headphones.

“What’s up?”

“Can you go to project x dot com?”

“What?”

“Just type it.”

Hitting the return key, the widescreen display is filled with a humongous “X”. Wagner’s Ride of the Valkyries is audible from the headphones on the desk.

“Oh my God.”

The X fades away to reveal a satellite image of the Earth, spinning gently on its axis. Small X markers start appearing all over the globe. London. Tokyo. Hyderabad. Dubai. Lines then arc from city to city, forming an elaborate mesh across the planet.

There is stunned silence.

As the horn section reaches crescendo and the cymbals crash, the globe fades and a tagline appears:

Everything changes. November.

Ben is livid. “What the hell was that? Where did that November date come from? It’s already October!”

Lisa is staring through her hands. “We are so screwed.”


Errrrr… Photo by rawpixel on Unsplash.


Ben is having difficulty moving his mouse cursor because his trackpad is covered in sweat. Usually the Data Science demo sessions have a small handful of attendees, but today is the first live demo of Project X, and there isn’t anywhere left to stand.

Mia has brought her entire sales team. Most of the engineering team are present. Even the CEO is there. Lisa is leaning against the wall for support. Her legs don’t feel like they are able to stabilize the rest of her body.

Ben coughs. His cheeks are burning.

“OK, hello – wow, there’s lots of you here. A couple of demos today, but let’s kick off by showing you the latest build of Project X. Please bear in mind that we’re not finished yet and this is just a preview.”

Leaning over his laptop, he types his username and password into the login prompt and clicks the button to log in. An error pops up: login failed.

“Damn it,” he whispers to himself.

Lisa leans over so she can whisper to him. “It’s not hooked up to the live environment; try your development password.”

Ben enters his credentials again, but manages to type them in the address bar, revealing his password to the entire room. His cheeks flush for a second time.

Somebody from Engineering is humming the Ride of the Valkyries. Another sniggers.

Lisa rapidly shifts over and takes control of the laptop, successfully logging in to reveal a white screen with a text box on it.

“Er, OK, so this is our latest prototype. It’s probably just easiest if we give it a go. Can someone suggest a word to me?”

Somebody pipes up at the back of the room.

“Eggs!”

“OK, let’s type that in.”

Lisa successfully manages to type “Eggf”, which is quickly corrected to “Eggs”. She presses the return key. A table of data is returned with some fairly arbitrary numbers about eggs: their size, the quantity that are produced each year, and the ratio of egg size to the creature that laid it.

“There you go! Cool, huh?”

There is silence in the room. Faces look blank, confused, and disappointed. Ben and Lisa exchange glances. 

“Any other words to try?”

Mia pipes up. “Is this Project X? Is this our biggest new feature of the quarter? How are we meant to sell this?”


Ben is leaning over and prodding at the Nespresso machine with his left hand, which has managed to swallow his coffee pod and jam the input tray, much to the dismay of his colleagues who are impatiently waiting for the day’s first shot of caffeine. With his other hand, he is scrolling through a forum looking for instructions on how to hard reset the machine to release the drawer.

“Technology, eh?”

It’s Lisa, back from vacation after the rather intense end to Project X. 

“Why does the drawer have to be controlled by software, rather than it just being a manual latch?”

“Because software is eating the world, Ben. And now it’s eaten your coffee pod.”

Back at their desks, Lisa gets the rundown from Ben on the next project.

“So they’ve asked us to work out whether we can link together the browsing patterns of our users with the best times of the week to send them promotional emails.”

Lisa leans back in her chair.

“OK, sounds reasonable. Have you looked at any prior art?”

“Not yet. I’ve still been dealing with the fallout from this ridiculous Project X thing. I’m answering what feels like hundreds of emails a day about what it is, how it works, and why it didn’t change history, resurrect Martin Luther King or put another man on the Moon.”

Ben pauses for a second. “What did they expect?”

Lisa rests her head on her chin.

“Let’s just forget about that. I’m ready to start on this new thing. I think I saw something about behavior tracking at the last CHI conference that could be a good place to start. But before we do, this project needs a name.”

“Oh, how about we name it after a goddess? Oh, oh – or maybe one of the Grecian Fates? I always liked the name Atropos. Not a huge fan of Clotho though…” 

“No, no, no,” interjects Lisa. “We’re calling it ‘Best Time To Email’. We’re not having Project X happen all over again.”

Lisa commits README.md into the new best-time-to-email project on Github. It reads:

A project to find out the best time to email our users.

“Let’s see them make an animated spinning globe out of that.”

The trichotomy of control

Leave a comment
Growth
Nanga Parbat, nicknamed the “Killer Mountain”. Steve House and Vince Anderson won the Piolet d’Or in 2006 for their direct ascent of the Rupal Face. It is the ninth highest mountain in the world at 8,126 meters (26,660 feet).

The sum of zero

Have you ever worked exceptionally hard to achieve something and felt utterly crestfallen when you were unable to get it? Worse still, have you ever strived to achieve something, yet when you do, it leaves an ultimately hollow feeling?

These feelings are common. Humans are hardwired to be insatiable: our eternal dissatisfaction has moved humanity ever forward, but it can also play havoc on our minds.

Your parents may have once told you to enjoy the journey rather than seeking a goal, and there is truth in that statement. 

In fact, I was reminded of this when reading the story of Steve House and Vince Anderson completing the first alpine style ascent – that is, without fixed ropes, carrying all of their equipment, and without leaving anything on the mountain – of the Rupal Face of Nanga Parbat.

Success is the sum of zero. The choices we made, the weather we had, the mountain we climbed and descended, everything we risked: all these factors reached their culmination… and were erased. We lived the answer to every question presented. There is nothing left to ask. There is nothing left of our selves, only the ghost of what transformed us.”

Steve House – Beyond The Mountain

Whether we reach our goals in life, or whether we don’t, the process that we move through whilst working towards them is the valuable reward: it is the part that actually shapes us, makes us better, and defines us.

How can we look disappointment in the face calmly?

Hit the target

Although not comparable to completing a serious piece of alpinism, my work life, as with many of us who work in technology, consists of a horizontal axis of time, studded with milestones from the past to the distant future. 

Product launches, project beginnings and endings, performance reviews, acquisitions, pivots, sales targets; you name it. At any moment in time there are multiple things that we should be aiming for.

As those of you in the software industry will know, the resonance of moments of achievement is often short-lived. Success, once celebrated, yields to the next goal. In day to day operations, when everything in your system is fine, there is often little feeling of success: it is sometimes the case that only when everything is on fire that your name is called the loudest.

Given how complicated software is, and given how complicated entire software companies are, it is extremely rare for everything to go well all of the time. 

You may have an incredible quarter for engineering achievement in your infrastructure, but there haven’t been enough new features shipped. Conversely, there may have been some fantastic new features shipped, but another part of your system is on fire. You may have even achieved both, but something entirely out of your control makes it not matter, such as the commercial team not hitting their sales target, or a round of redundancies swinging the hatchet at morale.

If we work in such a complex system, how can we ever win and feel proud?

Dichotomy of control

The suicide of Seneca (1871), by Manuel Domínguez Sánchez. Seneca was a noted Stoic philosopher. His writings are extremely readable today.

Another excellent book that I read recently was a modern re-introduction to the Stoic philosophy called A Guide to the Good Life: The Ancient Art of Stoic Joy by William B. Irvine. One of the themes often written about by both the Greek and Roman Stoics was the dichotomy of control

This sounds academic, but it’s actually pretty simple. 

Typically the Stoics would advocate the use of the analytical brain to maintain tranquility over our emotions. One of the emotions was that of worry: how can we worry about the right things?

The Stoics stated that to choose whether to worry or not, simply categorize your issue into these two buckets:

  • Matters you have control over such as whether you can do a good job, whether you can be kind and whether you can get a good night’s sleep.
  • Matters you have no control over such as whether it is going to rain, whether you may get seriously ill, or whether or not there is going to be a natural disaster.

If your issue falls into the first category – that is, matters that you have control over – then you should absolutely worry about it. You affect the outcome, so do your utmost to make that outcome happen. 

However, if your issue falls into the second category – that is, matters that you have no control over – then no matter the anxiety or worry about what might happen, it doesn’t matter, since you can’t change it. Therefore you should not worry about it. 

Although easier said than done, the Stoics argued that frequent application of this analytical reasoning to problems would strengthen one’s resolve and make him or her more able to maintain their tranquility (the Stoics, as you may have guessed by now, loved their tranquility). 

OKRs

I was reminded of OKRs when I read the above dichotomy. Yes, like the comparison of work to alpinism, it is hardly comparable to philosophy either, but stay with me here.

You may already be familiar with the concept of OKRs, but if you are not, then they stand for Objectives and Key Results: quarterly goals that an organization should be aiming for (e.g. achieve 60% growth) that trickle down into tangible objectives for each department (e.g. sell $1.5M of new deals), every team (e.g. create $150K of upsell) and individual employees (e.g. create 150 new leads). 

We tried OKRs a few years ago, but they didn’t get a lot of buy in. Your own mileage at your own company may vary. 

One of the common difficulties was that the OKR process expects goals to be defined that are of sufficient difficulty such that you generally only achieve 60-70% of each target; if goals are completed then they are deemed not ambitious enough.

The latter point, which is meant to encourage a mentality of challenging stretch goals, was hard for people to embrace; after all, if you’re doing OKRs right, then you should be comfortable with never meeting the targets that you set for yourself. That’s quite hard to take!

Is there a way that we can set lofty goals but not feel disappointed when we don’t achieve them?

Trichotomy of control

In his analysis of the Stoics, Irvine makes an argument that the dichotomy of control we explored earlier in the article is incomplete. He states that life doesn’t present a binary choice between things we can control, and therefore worry about, and things we can’t control and therefore not worry about.

There’s a third dimension.

Take, for example, a competitive tennis match. Each player should care about the outcome of the match: each player trains for it intensely and on the day will play very much like they are concerned about losing. It would be silly not to worry about it.

Action at Wimbledon. Photo by Howard Lawrence B on Unsplash.

But to say that each tennis player is in full control of the match would be a lie: sometimes they’ll win, sometimes they’ll lose, and sometimes each of those outcomes won’t necessarily be up to them: their opponent may be on form or off form, be superior or inferior, and there are infinite ways in which the match can unfold.

Therefore the tennis scenario could be described as a situation they can neither control or not control. It is one that they exhibit some control. 

They’ll certainly have a better chance of winning the match if they play their hardest and have a lot of talent and preparation, but ultimately, they cannot guarantee success. 

Does this sound familiar to work?

Since the Stoics aim for maintaining tranquility, how can they cultivate it in situations that they only have some control over? 

Let’s expand our dichotomy into a trichotomy:

  • Matters you have control over should we worried about to ensure the desired outcome.
  • Matters you have no control over should not be worried about.
  • Matters you have some control over should not encourage obsession over external goals, but instead, should be approached by measuring our success against internal goals.

But what does this mean?

Internal goals

In the mindset of the tennis player, an external goal is “I must win this match”. Worrying about this too much can bring pain when the match is lost, especially when it’s due to a fluke or a bad day. After all, the player only has some control over this. 

Instead, they should worry about an internal goal, such as “I will play my best in this match”. Their own success can be measured against this because it is something that they do have control over.

This is the only way that we can maintain some Stoic tranquility in the world of work, where, as we saw at the beginning of the article, goals may or may not be met despite your best effort. 

If we admit that our aim for the company achieving 60% growth is something that we only have some control over, then we can internalize our measurement of progress against that goal and just make sure that we try our best to make it happen

The neat trick here is that by internalizing a goal, we simply avoid tying our emotional response to it happening or not. Regardless, we’re still trying our best, which is exactly what we wanted to do by having a goal anyway.

Take some inspiration from Seneca, Epictetus and Marcus Aureliusinternalize your goals.