IT Tips & Insights: Learn how evidence-based management can improve decision making to produce better outcomes.
By Gerson Correia da Silva Junior, Scrum Master
Global software development has grown complex as business needs expand and deadlines become tighter. The main challenge faced by companies these days is determining how to maintain a fast delivery pace in synergy with real value being delivered to customers with frequency and effectiveness.
The Agile Mindset
When the industry started to really examine the effectiveness of software development, the agile mindset and methodologies came to shine as a promising savior with the release of the Agile Manifesto in 2001. Here was a solution for all of the wasted effort and budget experienced by traditional management processes.
Teams that chose to practice agile benefited from a lot of improvements. However, today the overwhelming growth of work fronts building solutions on top of the same backlog. This has created complex projects that are hard to synchronize in advancing their shared goals.
First of all, what we have to take into account to deliver real value is measuring the value that is hypothetically being delivered. It’s easy to be trapped by a false idea of effectiveness when dealing with a lot of parallel work that is put into customer usage rapidly, with a new sprint already started by the time the last one was released.
Evidence-Based Management
The Evidence-Based Management (EBM) approach, which later introduced a framework towards business agility, brings the empiricism once evangelized by the Agile Manifesto to the spotlights again.
First approached within a medical context, the discovery that inspired the discussion was that “only about 15% of their decisions are evidence based. For the most part, here’s what doctors rely on instead: obsolete knowledge gained in school, long-standing but never proven traditions, patterns gleaned from experience.”
The problematic nature of decisions being made based on fast thinking and no clear data-driven choices is also discussed in the book “Thinking, Fast and Slow” by Daniel Kahneman, which examines how our minds can trick us with false feelings of certainty during fast decisions making.
As Jeffrey Pfeffer said on Harvard Business Review, “Information acquired first-hand often feels richer and closer to real knowledge than do words and data in a journal article … Numerous decisions are driven by dogma and belief. When people are overly influenced by ideology, they often fail to question whether a practice will work.”
There are common assumptions that perpetuate the myth of being real and exact, even though there is no broad confirmation by rich evidence. The first-mover-advantage myth is one of them.
Research by Wharton’s Lisa Bolton showed that most people believe that the first to move to a new industry or market will have a big advantage. Even though empirical evidence shows that this is not a rule. Amazon, for example, wasn’t the first to sell books online. Actually, the second to arrive can learn a lot from the mistakes of the first.
This is like watching the news about plane crashes on an infrequent basis and starting to believe that planes are unsafe. The news represents a very low percentage of all successful flights, but the recency of the occurrence on your mind brings this assumption first and makes you believe that it’s true.
So how can we deal with the tricks of our mind to make more evidence-based decisions?
Jeffrey Pfeffer suggests 6 standards for it:
- Stop treating old ideas as if they were brand-new.
“To a hammer, everything looks like a nail.”
- Be suspicious of “breakthrough” ideas and studies.
“Empirical knowledge is not born ready; it comes from past experiences as an increment or advancing of previous ideas.”
- Celebrate and develop collective brilliance.
“Knowledge is rarely generated by lone geniuses who cook up brilliant new ideas in their gigantic brains.”
- Emphasize drawbacks as well as virtues.
“Enable who is going to be impacted by decisions to know the risks and join the decision process with clarity of possible problems.”
- Use success (and failure) stories to illustrate sound practices, but not in place of a valid research method.
“People tend to remember much different things when they are anointed winners (versus losers), and what they recall has little to do with what happened.”
- Adopt a neutral stance toward ideologies and theories.
“Establish clarity and consensus on the problem to be solved and on what constitutes evidence of efficacy.”
Applying the Practices Using Evidence-Based Management Framework
Published by scrum.org in 2020, the EBM Framework can help businesses achieve better value delivery with a guide for empirical management.
Breaking down business objectives can help businesses with a range of possibilities to improve their products and increase profits, make better decisions, and achieve business agility by mastering the process of making small steps.
Strategic Goal
A strategic goal is something important that the organization would like to achieve. This goal is generally so big and far away, with many uncertainties along the journey, that the organization must use empiricism.
Intermediate Goals
Achieving these intermediate goals will indicate that the organization is on the path to its Strategic Goal.
Immediate Tactical Goals
These are critical near-term objectives toward which a team or group of teams will work help toward Intermediate Goals.
Progressing towards the Strategic Goal in the EBM Framework is based on defining a hypothesis, running experiments to validate it, and inspecting / adapting the plan based on the evidence collected.
To really do it right we must first understand what value is and how we can explore it. Measuring value generally falls into the three categories: Activities (tasks performed), Output (what is produced) and Outcomes (new capabilities that the customer can achieve that weren’t available before).
The challenge here is that it’s easier for companies to measure activities and outputs than it is to measure outcomes that require gathering data from a lot of customers. And when dealing with complex projects, companies can fall into the false idea that doing more hours and delivering more features is improving customer value.
To deal with the challenges of measuring the right things and working on hypotheses and experiments toward goals, EBM provides a set of visions on value and the organization’s ability to deliver it.
Key Value Areas (KVA’s):
Current Value (CV)
The purpose of this measure is to have a real picture of the value that the organization delivers to the customer at the present moment. It is a starting point.
Questions that are useful here include:
- How happy are users and customers today? Is their happiness improving or declining?
- How happy are your employees today? Is their happiness improving or declining?
- How happy are your investors and other stakeholders today? Is their happiness improving or declining?
Unrealized Value (UV)
Unrealized Value includes the opportunities that can be explored to meet customer needs.
Questions that are useful here include:
- Can any additional value be created by our organization in this market or other markets?
- Is it worth the effort and risk to pursue these untapped opportunities?
- Should further investments be made to capture additional Unrealized Value?
Time to Market (T2M)
Time to Market indicates the ability of organizations to deliver new features and products quickly. It helps improve the frequency at which an organization can potentially change CV.
Questions that can be useful here include:
- How fast can the organization learn from new experiments and information?
- How fast can you adapt based on the information?
- How fast can you test new ideas with customers?
Ability to Innovate (A2I)
Ability to Innovate is the capability to deliver new features and products that might better meet customer needs.
Questions that are useful here include:
- What prevents the organization from delivering new value?
- What prevents customers or users from benefiting from that innovation?
Evidence-based decision making can improve our way of working, especially when you realize that our decisions are often based on biases and hypothesis that need to be validated and inspected. This will help us understand if our efforts are simply creating a lot of work to constantly release features, or if we are truly improving our customer experience and decreasing unrealized value.
Appendix from EBM scrum.org: Example key Value Measures
References:
https://www.infoq.com/articles/empiricism-business-agility/
https://scrumorg-website-prod.s3.amazonaws.com/drupal/2020-12/EBM%20Guide%202020_1.pdf
About
A native of Brazil, Gerson works on process consultancies mostly as an agile coach and scrum master. He is passionate about empirical thinking and loves to research better and sustainable ways to develop products. Gerson’s experience includes software development, product management, and agile coaching.
Join Our Team