The Eye of Sauron

comments 8

Oh dear…

“One moment only it stared out, but as from some great window immeasurably high there stabbed northward a flame of red, the flicker of a piercing Eye; and then the shadows were furled again and the terrible vision was removed.”

– The Lord of the Rings, The Return of the King

If you’ve been in the industry for some time, you’ll recall the moments that you were under intense pressure to deliver.

Sometimes this pressure can come from anticipation: your team just happens to be responsible for delivering the most important new feature for the company this year. Pressure to deliver can also come from catastrophe: parts of your infrastructure may not have scaled as expected and are continually on fire, and unless a new solution is developed, customers will start dropping away.

In these moments, you’ll have felt what I like to describe as the gaze of the Eye of Sauron: whichever way you turn, the entire business is looking toward you. It’s uncomfortable. You can feel the heat. There are emails, Slack messages, JIRA nudges, interruptions in person, you name it – it’s constant and stressful.

Depending on your mindset, you can turn these tough situations into a challenging but rewarding experience for your team, or conversely, you can totally fumble. Handled correctly, you’ll be looking at career growth. Handled poorly, and you may find the next high stakes project goes to another team instead.

The Eye glances at you…

But first, how do you know that the Eye has turned its gaze on to your team? There are a number of cues that begin to become more frequent and intense:

  • Increased interest in your project from stakeholders. You’ll have to bat them away, rather than ask repeatedly for them to turn up.
  • Senior members of the business beginning to probe into the status of your project, such as when you’re grabbing a coffee in the kitchen, or walking down the hallway. You may have never spoken to these people before. Why are they talking to you now?
  • Your boss, or bosses’ boss, being more direct and intense with the progress of your work. Why do they care more than usual?
  • Noticing how your upcoming feature is being hyped internally and externally. It may now be perceived as the headline launch of the year, even though that was not apparent when you started the project. Why is the business sending out teaser tweets when the UI isn’t even designed yet?
  • Or, quite simply, everything is on fire and the platform won’t work unless your team digs their way out of this hole. The support ticket queue is getting bigger, and bigger, and bigger…

Regardless of how the situation has unfolded, it’s important to increase your vigilance and take extra effort to manage you and your team’s way through this period of heightened pressure. Handled deftly, you’ll fully own the tough situation and have something to celebrate once you deliver. You’ll also earn the team some much needed breathing room afterwards.

Under the Eye

Are you feeling the heat?

Although the increased pressure will make your daily activities harder, there are some principles that can help you through these periods. Try to ensure that you are applying them daily, through your discussions, meetings and decisions.

  • Team alignment: When you’re under the Eye, your team will probably know. But if they don’t, or if you’re purposely shielding them from knowing, then that’s a bad idea. Utilize the pressure in a positive way: align the team around what they need to achieve, make sure everyone understands how to succeed, and then push them forwards towards the goal.
  • Communication: At times of immense pressure, you’ll want to increase the visibility of what the team are working on. Consider writing a weekly (or more frequent) update to stakeholders. Depending on the size of your company, a weekly newsletter to the wider organization might be suitable too. Either way, you’ll want to make it absolutely clear what you’re working on, how you’re progressing, and any key decisions that you’ve had to make. Always invite responses and feedback: it prevents frustration if you open up a channel of communication for others to reply.
  • Release frequently: Since your cadence is of utmost importance, ensure that you’re releasing frequently so that your stakeholders can follow along with your latest builds. The more time that you have for feedback in stressful situations, the better. Don’t keep code held back until the deadline; it just makes the event more stressful and the resulting mega-merge might cause all sorts of bugs. Use feature toggles and keep shipping to production.
  • Pragmatism: As dates loom nearer, or as the system continues to ignite, you’ll need to make pragmatic calls on speed of development, quality of the code, and creation of technical debt. As much as it can be painful for idealists in your team, you’ll likely end up shipping some shonky code in order to get the work over the line. However, make note of every hack you put in so that you can tidy up and refactor later.
  • Work hard: As a leader, you need to set the example for the rest of the team. Put in the work. The hardest projects can become career-defining moments. Own them and be there.

A successful high-stakes project is fantastic for your visibility.

Though the graft will be tough, you will succeed. When you’re done, what’s next?

Gaze averted

The pain has passed and you are now in the aftermath of the marketing launch for your new feature. Retweets are pinging off everywhere, the company blog posts are churning out, and you can hear the ringing of the virtual cash register.

However, what you do next is very important for the morale of your team.

  • Celebrate: This is one of the most critical things. The team has worked extra hard and they’ve met their commitment. Take them out for lunch, or drinks, get some cakes shipped out to the office, put on a gaming night – whatever makes them happy. Make sure that you say thank you for what they’ve done.
  • Tidy up and clear down technical debt: As the deadline approached, a whole bunch of little shortcuts may have been taken: a hack here, a missed unit test there. Put aside the next sprint to refactor and tidy up at a more leisurely pace, whilst fixing any production bugs if they arise.
  • Self-guided project time: Time can also be put aside in the coming weeks for some self-directed learning. Allow the team time to experiment and to learn something new. This change of pace and direction puts some mental space between the last project and the next one.
  • Reflect: Arrange a project retrospective meeting. It’s a focussed way of reflecting on the whole process and discussing what could be handled better next time. Even if the project was run perfectly, the project retrospective is an opportunity to mentally close the current book before opening up the new one.
  • Plan and regroup: Take some time to think about what’s next. Are there any initial explorations that can be picked up? Is it time for a design sprint? It’s time to start the discussion, to think about some initial planning, and to get excited about the future.

Protection from the Eye

It’s safe to say that – like any period of “crunch” – intense periods of work are not sustainable. If a team finds themselves under the Eye too often, it will cause burnout and attrition.

This may be unavoidable at a start-up, but that’s an exceptional circumstance: those that are there are fully committed to the ride. However, at a larger company, it is important to consider how projects and expectations can be managed to allow different teams to feel the heat of the Eye in an alternating sequence as time progresses, allowing the other teams to temporarily shift down a gear and regroup.

For example, it is advantageous for your marketing department to space out product launches to maintain a steady flow throughout the year. Why not take advantage of this? Your product organization and your engineers can balance these intense periods of contraction with periods of release and recuperation.

As a leader, you should also actively fight for after-project space for your teams if the business does not give them the opportunity by default. When a big project is over, push back on demands to create the room for the rejuvenating activities in the previous section.

It’s a marathon, not a sprint, after all.

Competitive sitting and leaving loudly

Leave a comment

Oh, well, this person isn’t working very hard, are they?

No, you first

Does your office suffer from a competitive sitting problem?

If you’ve not heard of the phrase before, unfortunately it doesn’t have anything to do with a game of musical chairs, or rolling across the office and jousting each other with broom handles.

It’s likely that you may have experienced it yourself.

Have you ever been sat next to your manager, or your team, and have felt anxiety about going home first at the end of the day, based on what they might think of you if you were the first to leave?

Likewise, have you felt that bubbling stress when traveling into work, making you walk that extra bit faster, so that you can be the first person in your team to arrive? Have you felt that being there first signals that you work harder than everyone else?

That is competitive sitting.

It is a cultural phenomenon where employees believe they are being monitored for how long they spend at their desk, regardless of their output. It can create a workplace where an employee may not go home until their manager goes home, who in turn may not leave until their own manager leaves. Everyone feels like they need to put in more hours because they think that others are putting in more hours.

Clearly this isn’t about the actual work that an individual is doing, but more about the appearance of doing said work.

In culture

The subject of competitive sitting was brought to my attention by an Instagram post by Anna Whitehouse (via my colleague Toby), who campaigns for better treatment of those that happen to be parents.

Check out her post.

Please leave loudly. Tell people, “I’m just off to pick up baby Ian”. (Noone called their baby Ian last year so trying to revive it). Tell people, “I’m heading off for a beer-and-a-burger down Wetherspoons”. Or simply tell people, “I’m heading home now”. Today I wrote a feature for @telegraph about being human at work in a bid to stop ‘competitive sitting’ – seeing who can remain strapped to their slab of MDF the longest. We’re not talking about a monologue on baby Ian’s weaning journey or an insight into nit comb research. But why hide our lives from work? When did it become OK for a woman to sit at her desk miscarrying for fear of telling anyone because they might know she is ‘trying’? When did life become something to sweep under the carpet with hole punch offcuts and rogue staples? When is that good for business? Every day, Robbert Rietbroek asks his executive team to “leave loudly”. For the chief executive of @pepsi Australia and New Zealand, it’s about sending a message to the entire company. "'Leaders Leaving Loudly’ is something we created to ensure that when team leaders leave, they feel comfortable doing so but also to declare it to the broader team,” he said. "So for instance, if I go at 4pm to pick up my daughters, I will make sure I tell the people around me, ‘I’m going to pick up my children.’ Because if it’s okay for the boss, then it’s okay for middle management and new hires.” Rietbroek said the goal is to reduce “presenteeism” and boost team morale because if you are “younger or more junior, you need to be able to see your leaders go home, to be comfortable to leave”. Since joining @pepsi, the father of two has been championing family-friendly, flexible work policies, as well as attempting to boost the number of women in senior management roles — currently at about 40 per cent. But he wants to challenge the perception that flexible working arrangements are “off limits” for men and non-parents. The head of procurement is an avid surfer, and is given flexibility to take time off when the surf conditions are good. "His entire team knows when he’s not in the office he’s catching waves 🌊.” #flexappeal

A post shared by Anna Whitehouse (@mother_pukka) on

The post references a workplace initiative from Pepsi Australia and New Zealand called “Leaders Leaving Loudly”. When team leaders go home early, they are encouraged to publicly announce why they are doing so. Perhaps they need to go and pick up their children. Perhaps their partner is sick and needs some help at home. Perhaps they’ve worked really hard this week and need a couple of extra hours to decompress. But, regardless of the reason, they announce it openly to their team before leaving the office.

The idea, as explained by CEO Robbert Rietbroek, is to reduce “presenteeism”, because employees that are younger or more junior will see their leaders going home for work-life balance reasons and will therefore also feel comfortable to leave when they need to. Leaving loudly drives a culture of working smartly and efficiently whilst being mindful of other commitments in our lives. It says that we aren’t primarily in the office to simply clock in and clock out, we’re here to get meaningful and impactful work done and then go get on with other equally important things.

Over the years, there have been a number of articles on the subject of presenteeism in Japanese culture. In fact, Japanese firms have been under scrutiny for the expectation of long hours as opposed efficient work. An unwritten code of how many hours one should be visibly sitting at their desk can create a culture of sleeping in the office, sinecure activities, and an unhealthy, stressed and unproductive workforce. At worst, it can lead to serious harm and death. There is the Japanese term karoshi, which means death from overwork. There are a shockingly high number of reports of this occurring.

Work is a large, meaningful part of life, but the best work is done when suitably rested and refreshed. And, like art and food, the greatest satisfaction is gained through contrast with other opposites: the rich diversity of the other facets of life. A day filled with productive work, fun family time, a walk, some exercise and a good night’s sleep is far more satisfying than 18 hours of unproductive graft due to tiredness, a silent takeaway dinner where both parties are on their phones while eating, and falling asleep on the sofa.

How can we move towards a healthy balance?

Virtue signalling

The oft-used term virtue signaling means the conspicuous expression of moral values. In recent years, its use has become loaded with negative connotation and has been used to highlight certain behavior on social media, where people partake in particular activities in order to appear virtuous. Such activity can be seen in changing a profile picture to support a cause, retweeting content from charities or politicians, and offering “thoughts and prayers” for any world crisis or notable figure passing away. Changing your profile picture is not the same as actually raising money for a charity.

Competitive sitting is a form of virtue signaling. It says “Look at me! I’m still here and I’m here until the end! I am the best employee!” But – and I don’t know about you – I’d much rather work with people who come in, get stuff done, then go and spend time with their friends and family so that they come back the next day refreshed and ready to do it all over again.

You don’t want employees sitting around in the office until 7:30PM – despite doing nothing particularly of use – to be congratulated as examples of people that are working hard. Instead, we need to ensure that people are held in high regard for their actual output, rather than their perceived output.

Setting an example

We need to consider the behavior and examples that we set to others in the company. This is why it is important for managers and leaders to make sure that they are sending the right message through their actions.

Think about your team, or your company. What is the culture like with regards to working hours and presence in the office? Do you think that your actions match your expectations for your staff, or is that something for them and not you?

Are you expecting your team to do 60 hour weeks, yet you regularly go home at 4PM? What message does that send? Conversely, do you want for your team to feel like they have the flexibility to come and go as they choose as long as they are getting their work done, but regardless, you’re there at your desk from 7:30AM to 7:30PM every day? How do you think that might make them feel, knowing that their manager is always there before them in the morning and after they go home at night?

Check whether your own actions align with the workplace that you wish to see around you. Ensure you are acting congruently with what you want to see. It’s unfair to expect the same from others otherwise.

How do data science projects work?

Leave a comment

A primer for managers, stakeholders and those that are interested

No, not that kind of science. But sort of.

According to the LinkedIn 2017 U.S. Emerging Jobs Report, data science roles on the social network have grown over 650% since 2012. The same report notes that there are 9.8 times more machine learning engineers working today than there were 5 years ago.

But, given that data science is a nascent field, do we really know how to run projects in a way that allows data scientists themselves to have the space required to experiment and for management and stakeholders to be satisfied with their progress? After all, despite 43 years passing since the original publication of The Mythical Man Month, and 17 years since the Agile Manifesto was coined in a ski lodge in Utah, you could sometimes argue that we’ve never learned how to deliver software!

Given the inevitability that data science projects will become ever more part of the software industry as a whole, and that more managers will be held accountable for them, and that more stakeholders will be expected to follow along and give feedback, we should all understand how these projects progress.

Here’s what I’ve learned.

But first, to set the scene, and because it’s fascinating, let’s have a very gentle introduction to deep learning, which is one of many techniques used in the field of data science. It uncovers some of the intricacies of doing data science projects.

Scaling deep learning

A number of weeks ago, I had my attention drawn to an excellent research paper via Adrian Colyer’s Morning Paper newsletter. The paper is an exploration into understanding how we can improve the state of the art in deep learning, and whether we can make progress more predictable.

Without going into a lot of technical detail, deep learning – the new, fashionable term for building neural networks with hidden layers to solve problems – can tend to be more of an art than a science. We don’t know a huge amount about exactly why they work so well for particular problems, and years of designing them hasn’t yielded bulletproof design principles or patterns. Clearly, this isn’t promising for the predictability of projects. After all, the industry always wants launch dates!

If you aren’t familiar with neural networks, then they can be described as a mathematical model inspired by how neurons work in the human brain. Each neuron takes an input, and depending on some condition, gives an output. Neural networks are software representations of chains of neurons. Deep learning – specifically the “deep” part – means that there multiple layers of neurons. More specifically, deep learning means that there are “hidden” layers of neurons: ones that exist in the chain between the input and output layer. These deep learning networks have been used successfully for making computers do “intelligent” tasks such as voice recognition and image recognition.

Building neural networks in software isn’t like regular programming, where a human writes out the specific instructions of what happens when a user clicks a button, or assembles exact queries to a database to retrieve data. Instead, you specify how the network looks: the number of neurons and the number of layers and how they all connect to one another, which data inputs make them activate, their individual activation functions and the algorithm used for training them. That’s called defining the architecture of the network. Some networks have had millions of neurons and billions of connections between them.

Once you have an architecture designed, creating a deep learning classifier requires training. For example, if you wanted to use a neural network to determine if pictures had a cat in them, you would expose the network to lots – often many thousands – of pictures of cats. Special algorithms tune the neurons to understand the commonalities in the cat images, meaning that when exposed to a picture of a new cat, it can classify it as such.

Creating one of these classifiers roughly unfolds in these stages:

  1. Defining the architecture of the network.
  2. Evaluating it with a small about of data to see whether it works.
  3. Iteratively training and adjusting the network based on the amount of error seen.
  4. Stopping training when the network performs well enough, or when it isn’t improving any more.

The paper contained the following diagram to show how increasing the amount of training data (x-axis) reduces the amount of error in a given deep learning model (y-axis).

Deep learning with little data is rarely better than random. But after finding a suitable architecture, throwing more data at it makes it improve, to a point.

The small data region involves prototyping approaches on small amounts of data, where often results can be no better than random. Once a design yields some results, more data is gathered and used to train the model to see whether it truly fits the problem; this is represented by the orange circle annotation. If it is improving, then as much data is fed into the network as possible while it improves (the power-law region). We continue until it cannot get any better (the irreducible error region).

If this all sounds a bit strange, then consider the similarities with doing a start-up: typically the founders keep iterating on MVPs and pivoting until the product-market fit is found and customers start signing up, and then decide to stick, invest and scale it into a larger business.

I hope that gives some idea of how deep learning projects work. As it turns out, there are many similarities in the steps taken in most data science projects. As a manager or a stakeholder, you’ll need to understand when projects are moving through these phases to have a greater appreciation for the work going on.

Managing data science projects

If you’re used to running or observing software teams who aren’t doing data science, then you’ll notice that the above sequence of steps sounds fairly peculiar and unpredictable when compared writing your typical CRUD applications, and you’d be right! If your business expects the same kind of predictability and time commitments for data science that they do for their regular software projects, then there may be some uncomfortable conversations required: they work very differently.

The approach to training deep learning networks above can be generalized to represent data science projects more broadly, regardless of the exact technique, whether it be NLP, regression techniques or statistics.
Data science projects iterates through multiple phases, many of which can result in failure, and many which require financial decisions for the business such as whether to invest in more training data or more compute power.

If you want data science success, then it’s incredibly important that managers and stakeholders understand how projects typically work. This ensures there is a mutual understanding and appreciation of risks and progress, and gives data scientists the trust and space they need to experiment.

Common iterative areas are as follows:

  1. Defining the problem: Typically there is a business problem to solve that needs formulating into a data science problem. Is the problem actually solvable? What data are available and what features do we need to model? How do we know that we have succeeded? Which trade-offs, notably in precision and recall are acceptable to the user?
  2. Finding a model: Assuming we think we can solve the problem, what sort of techniques may be likely to work? How can we prototype them and get some initial results that let us be confident enough to proceed?
  3. Training the model: How and where do we get more data from?
  4. Application to real data: Now that our model is trained and is giving acceptable results, how do we apply it to real data to prove that it works?
  5. Production: Now that we have a successful model, how does it get moved into production and by whom?
  6. Maintenance: Will this model degrade over time and, if so, how much? Does it need retraining at regular intervals in the future?

Steps 1-4 may result in failure and a decision to go back a number of steps, or even to not continue with the project. These steps are scientific processes. In steps 5-6, we should have a model that we would be confident delivering, whether that is to a customer, or building it into software. This makes them engineering processes requiring a different set of skills, and often staff.

1. Defining the problem

Contrary to belief, this can be the riskiest, most difficult and time consuming part of the entire process. Business problems aren’t often easily translatable into a scientific problem that can be immediately worked on. Stakeholders will be wanting to deliver a feature to users, or they may be wanting to gain an insight into some data. But the team will have to start with first principles and build the problem from the ground up.

Firstly, is the problem actually solvable? It may be the case that it isn’t – i.e. it’s intractable by definition, or requires data that don’t exist. It may be the case that a model could be built, but the insight is so subjective that many people will think that it is incorrect. If the data exist, is it clear which parts of those data are required for modelling, and what the features are?

Most importantly, how will the team know when they have succeeded? Is there a clear right answer that will allow them to prove that the model works? Or will it require lots of user testing, and if so, with who? Additionally, what trade-offs are acceptable? Does it need to work most of the time, or all of the time?

2. Finding a model

Given that there is a clear definition of a problem, such as whether it’s possible to predict whether users are about to leave your website, or whether you can classify images of credit cards, the project begins with finding a model. These stages are typically run in a time box, depending on the size and difficulty of the problem. Therefore the first question that the business has to answer is how long are we willing to spend seeing if this might be possible?

Secondly, in order to find a model, you’ll need data. Specifically, two types of data: training data and test data. You may already have the data you need. In the first example project above, it may be a case of taking a sample of user logs from your website. However, you might not have any data at all: in the case of the second example project, where are you going to get images of credit cards from?

So the next questions to ask are can we get any data to test with? This is then followed by can we annotate the data easily? Going back to pictures of cats, if a dataset doesn’t already exist, humans are going to have to look at those pictures and say whether they are a cat or not. If so, you will probably want to crowdsource the annotation using platforms like Mechanical Turk or Figure Eight. This costs time and money and requires a clear definition of the questions to be asked. Sometimes it is simple such as “is this a cat?” Sometimes is highly subjective, such as “does this text convey disappointment?”

Assuming the business is happy with the time and money investment, then this phase will run until a suitable model is found, or until the time runs out and we admit failure. This phase of the project will typically have staff running small experiments, and spending a few hundred dollars on data. The end of the time box is a great opportunity for a demo in front of stakeholders, regardless of whether there has been a success or not: there are always learnings to share.

3. Training the model

If the first phase has been a success, it’s time to train the model. Like the first phase, this centers around time and money. In broad terms, the more training data that are available, the better the model will get. You will need to discuss, again, where to get the data, how to get them annotated, and how much you are willing to spend on that task. It will typically be much more than in the first phase – maybe even thousands of dollars if you need human annotation to be done.

Additionally, depending on the scale of the problem, training the model may require more compute power than is available on local machines. For non-trivial training tasks you will want to utilize fast machines, often from a cloud provider. Even using spot instances can be pricey depending on the task, so upfront estimates are essential to avoid an expensive surprise.

Assuming the budget is acceptable, you’ll also want to have a conversation about when we’ll know the model is performing acceptably. Is there a precision and recall goal we are aiming for? Or are we going to commit a certain amount of time and money and then reassess? Given the variables above, this phase will, hopefully, produce a model that works well against test data. Demonstrating it against real data in the problem space is a great way to conclude this phase.

4. Application of the model to real data

Precision is the accuracy of your model on the data that it has classified. Recall is the amount of data that it has correctly classified out of all possible correct classifications. (Interested parties might find it fun to read a thorough definition.) There will always be errors in any model, but tuning it to balance precision and recall can greatly improve the model’s effectiveness: high recall typically lowers precision, which means your users may see more errors. Is that acceptable? It could be for classifiers diagnosing illness where you’d rather be safe than sorry, but it could be extremely irritating for users of text analysis software.

At this point, you’ll have a model that seems to work well at the task at hand. Before giving it the green light towards production, you’ll want to test it on real data, as precision and recall figures alone cannot be trusted to determine whether the model is of acceptable quality.

A good approach is to have the team produce multiple versions of the same model with different parameter tunings representing different precision/recall balances. These can then be applied to real data and shown to your stakeholders for their feedback. This can help them understand the implications and make an informed choice as to which models and tunings acceptably solve their problem.

5. Production

If your team has made it to this phase, then it’s looking very likely that you’ll be getting that feature delivered. But they’re not done yet. Unless you’re extremely lucky, your data scientists will not be experts at production engineering, so it’s at this point that they’ll partner up with other engineers to move the project forward.

Key considerations here include how to technically integrate the classifier into the production system, how to store the models, and the speed of classification which determines how the code around the models will need to be architected: perhaps it needs wrapping in an API because many parts of the system will need to use it. Maybe due to data volumes and speed of classification it will require tens or hundreds of instances.

Work at this point becomes easier to estimate. It fits more naturally into how your feature teams deliver their work. You can start giving more concrete deadlines for the feature becoming available, and assuming all goes well, you’ll be able to ship it.

Awesome. But is the project done? Not exactly…

6. Maintenance

Like regular software, models that you build will also need maintenance with time. Depending on the type of data you are processing, especially if it is topical data such as social media feeds, then inputs will change: consider how widespread the use of emojis are today compared with five years ago. Consider the names of popular video games now compared to last year. Models won’t know about these if they are abandoned once shipped.

If the input data does evolve and change rapidly over time, then the team will need to revisit it in the future to analyse it against new data. Documentation on how the team built, chose, and trained the model is essential as inevitably everyone will have forgotten in the future when it’s time to check how it is performing.

Whereas maintenance of the production system is the duty the production engineers, the maintenance of the models is the duty of the data scientists. You’ll need to find an ongoing balance of creation of new models and maintenance of the ones that you already have in production.

Cycle complete

And that’s it: the rough life cycle of a data science project. In many ways, they are harder to manage than traditional software projects.

Getting something into production involves much more chance of failure, many more unknowns, financial implications for data and compute, and cross-collaboration between disciplines. And given that people understand less about how this sort of work is done, and the hype in the industry about the promise of AI to solve all of our worldly problems, managing expectations is even more challenging. Not to mention the ethical implications of data science as a whole, but that’s for another article…

. . .

Thank you to my colleagues Alastair Lockie, Dan Chalmers, Hamish Morgan, Óskar Holm and Paul Siegel who gave invaluable input as I put together this article.