Transforming Decision Making with Collective Reasoning

The Barriers to Organizational Transformation

Confidence, accuracy and speed in making decisions is the holy grail for organizational transformation. With all the work taking place in decision science, AI, and data science, we still find ourselves weary from long meetings seeking the collective confidence to know we are making the right decision. The situation has been made worse by the pandemic.

I recently spoke with the Chief Investment Officer of a top-tier endowment fund. In his position, he oversees investment committees, each of which allocates billions of dollars of funds through collaborative decisions. He knew that an improvement in their process of making investment decisions could have a significant positive impact on returns, and he was sure that the path to improved accuracy in decisions must be through technological enhancements. Specifically, he wanted to capture as many different perspectives as possible, make the discussions more transparent, and document the decision process to back-test it against results so they could learn how to improve.

Traditionally, collaborative decisions can be tricky in all of these areas, and are often fraught with unintentional bias towards senior leaders, or the ‘loudest voice in the room’. In addition, collaborative decisions are difficult to manage at scale, and even more challenging when the participants weigh in from multiple locations or different cultures. It is not by accident that most decisions are made with executive override, as anyone who has tried to reach a consensus across a large group can no doubt attest. The ‘echo chamber’ effect results in billions of dollars of lost opportunity across the global investment community every year. Fortunately, recent developments in collective intelligence, artificial intelligence, and natural language processing make it possible to solve many of these challenges to efficient, accurate decision making. This article articulates a method to increase confidence, speed and accuracy in organizational decisions by harnessing the collective power of these technological advances. By bringing together the insight and reasoning skills of people and the optimizing and refining clarity of artificial intelligence, we can begin to improve one of the most human and error prone aspects of business, the ‘art’ of well-informed and accurate decisions.

Before we begin, it is helpful to understand a few of the key challenges we need to address. These are the “big three” sources of dysfunction representing lost opportunities to effective decisions:

1. Collective alignment on the reasons for a decision: Poor alignment often results in an impasse, a frustrating end to a meeting, or a desire for leadership to override an otherwise open process. Too often the process of finding collective alignment on critical decisions invokes organizational politics, which often lead to lingering participant dissatisfaction, detachment, poor follow-up and implementation, and long term erosion of confidence in leadership.
2. Systemic bias: Unconscious systemic bias keeps organizations in a rut. Innovation is too often stymied by normative patterns of decision making and organizational structure rooted in years of “experience”. Valuable diverse thoughts fall to the floor while entrenched patterns and loud voices rule the day. Hidden bias relating to educational background, culture, age, gender, and seniority all conspire to silently kill potentially compelling and innovative ideas.
3. Evidence-based accuracy tracking: Insufficient recording of the reasons and influences underpinning a decision create opportunities for a revisionist history where comments surface such as “I think I was on leave when that decision was made,” or the always-helpful “I had my doubts all along”. Healthy organizations learn to shift the focus from personal accountability to collective organizational learnings taken directly from an audit-trail of evidence.

Advances in collective intelligence and artificial intelligence offer a breakthrough opportunity to overcome these three barriers and provide a transformative new approach to making well-informed and predictively accurate organizational decisions¹.

Collective Reasoning: A process that results in learning and evaluating a group’s reasons or beliefs about a decision or prediction

Creating alignment with collective reasoning

The solution to all three barriers and a path to confident, fast and accurate decision-making is to rapidly learn shared beliefs and priorities about the evidence supporting a decision while trying to eliminate bias. Without technology this is often done in meetings with an objective facilitator focused on creating shared priorities and alignment. This manual process does not scale well across locations and time zones, nor is it relevant in a time of crisis such as a pandemic. In addition, the meeting process often fails due to ‘the loudest voice in the room’, status in the organization, passion or the powers of persuasion ruling over evidence, logic and diverse perspectives. Clearly, this is a massive opportunity for a collaborative technology solution.

Work in collective intelligence has demonstrated that collectively we are smarter. Approaching a decision from multiple perspectives is a known best practice. As we enter a new age of collaboration and a generation that thinks of participation as a fundamental aspect of employment, we can find inspiration in a series of excellent books on the power of collective intelligence. Scott Page’s “Difference” lays out the power of diversity in thinking to increasing predictive accuracy. Philip Tetlock and Dan Gardner document and analysis demonstrated accuracy of collective intelligence in “Superforecasting”. Finally, Scott Page and Tom Malone contributed their substantial expertise in their very well received books, “The Model Thinker” and “Superminds”, respectively. In summary, these books show various ways the science of collective intelligence promised new ways to think about the accuracy.

Collective Intelligence Summary: Multiple perspectives and diverse backgrounds increase prediction accuracy and reduce systematic bias.

Collective reasoning goes a step beyond collective intelligence. In collective intelligence predictions are done completely independently. A diverse group is given a task to make a predicted outcome, each individual works independently and the results are analyzed. Reasons for the “why” behind different perspectives of a decision are analyzed independently.

In Collective reasoning we separate the prediction or rating of a decision from the reasons for the rating. Collective reasoning captures the dynamic potential often observed in great brainstorming sessions where we learn from each other’s thinking; it collectively creates new approaches to a decision. Collective reasoning captures the incredible power of collaborative thinking, moving beyond the mathematical calculus of simply aggregating a series of independent thoughts and capturing the power of the collective mind in creativity, problem solving and alignment.

An illustrative example

Let’s illustrate this with a simple example of making a decision to invest or not invest in an early-stage company — a startup. We can all relate to the need for confidence, speed and accuracy in betting on a startup that may have the next big idea.

Imagine you have been invited to be part of a select group of people to evaluate whether to invest in a new stealth startup or not. Collectively, you have the task of predicting whether the company will be successful in attracting major investors. You have access to all of the company’s materials, you participate in a Q&A with the founding team. You have all the materials you need to make a considered decision and prediction. The company you are evaluating has highly technical components to their business. While you don’t know their identity, you know that there are others with strong technical backgrounds appropriate for the task.

You are asked to rate the company on a scale of 1 to 10 on the following attributes:

1. Compelling business opportunity, considering market, competition, etc.
2. Team — is this the right team to lead the company in the target market?
3. Network effects of early advisors and investors — can they help propel the company?
4. Investment conviction — is this an investment you would make or recommend?

After reviewing the materials, you provide your input score and the reasons that caused you to give the score you gave. Once you submit your thinking, you are shown a sample of others’ reasons and their cause for their score. You do not see the other contributors’ score. Note you do not see who is behind the reasoning. You are asked to prioritize the reasons that align with your thinking. This process creates increased focus on the evidence driving a perspective on a decision and reduces the bias of organizational position or team politics. Once you have submitted the prioritized list of reasons, reading them may have caused you to go back and rethink your rating or other thoughts you might have about the decision. This reconsideration is an important part of the group process of moving toward points of alignment.

Learning relevance with scientific accuracy

We freeze frame the example to note the opportunity for a technological and scientific solution to collective reasoning. In fact, the roots of learning group preferences go back nearly 100 years to the law of comparative judgment first stated by L.L. Thurstone in 1927. The only way to learn ‘predictive’ preferences is an A/B test, an economic tradeoff. Many of us have been through the process at the optometrist’s office of answering “A or B?” … “B or A?” over and over and over again. Marketers are well versed in A/B tests as standard means of learning product preferences. But does this work for groups at scale in a dynamic discovery environment where the number of items to compare could require massive numbers of A/B tests? AI technology offers a solution.

Each time an individual in the evaluation team described above ranks a list of reasons, we can create a “web” of who prioritized whose reasons. The web from a session like we outlined above is shown below. The nodes are individual contributors, the links are generated by prioritizing each other’s ideas:

 

Relevance is learned from using a hidden Markov process. The sample list shown in the right panel of Figure 1 is a dynamically generated list based on prior rating events. The “hidden” part of the Markov process is a function that gradually shifts from “discovery mode” getting inputs on new ideas to “prioritization mode” attempting to estimate the list of reasons that best represent the group’s rank-ordered reasons. Note that the context of the sample list is determined by the “feature” under consideration. Team assessment is an example “feature” of the overall model. The figure below shows the learning curve of the algorithm. Note its rate of convergence is driven by the number of rating events but it converges rather quickly for the top rated items (near the origin of the plot).

 

From that web of connections we learn influence ranking of various members and we use natural language processing technologies to learn the reasons and topics that represent the collective reasoning associated with assessment of the business opportunity, the team, etc. Learning relevance vs. sentiment provides a meaningful signal of what items are both highly relevant. An example for an investment are shown in the figure below.

In this particular case, the evaluating group sees revenue as highly important but sentiment indicates that discussion about revenue is mixed. Detail on why this is true is learned by looking into the relevant reasons assigned to this topic.

This is the essence of a collective reasoning system. It automates the acquisition of knowledge applied to the decision at hand.

The background knowledge outlined above is only the baseline. Continue on to our next blog, Integrating Artificial and Human Intelligence, and dive further into the next generation of transformative intelligence.