For good reason, the book "Wisdom of the Crowds" has become a cornerstone of the WEB2.0 world. Authored by James Surowiecki in 2004, it introduces a revolutionary idea: under certain conditions, the wisdom of the masses surpasses that of experts. Surowiecki, an economic journalist from New York, meticulously details this concept, outlining the conditions and presenting numerous examples across diverse realms such as economics, contracts, politics, transportation, competitions, gambling, urban life, and our daily lives.
Living with this idea became more manageable in 2008; Wikipedia is a prime illustration of how a mass-written encyclopedia can be more up-to-date, prosperous, and better on average.
One intriguing aspect I found in the book is Surowiecki's inclusion of counterexamples, acknowledging cases and conditions where the idea doesn't hold. People not only dismiss it but also articulate opposition.
Another noteworthy aspect is how Surowiecki extrapolates lessons from the wisdom of the masses about the functioning of the scientific and organizational worlds.
The book covers the following topics:
The principle of the wisdom of crowds
Conditions that allow for it:
Variety
Independence between people
Aggregation
Types of decisions where the wisdom of the masses is relevant
Coordination issues
Collaboration issues
Thinking/cognitive issues
Insights in the scientific world
Insights in the corporate world
I highly recommend reading the entire book. I read the second edition (published in 2005), where Surowiecki added an afterword, incorporating insights gained during a year of lecturing and discussing the idea extensively. It is truly worthwhile.
Wishing you an enjoyable read.
The principle of the wisdom of crowds
The principle of the wisdom of crowds asserts that collective intelligence emerges within an audience. As the audience expands, its shared intelligence grows and often surpasses individual experts. Initially, this idea may seem implausible—how can a diverse and non-specialized audience outperform an expert? Nevertheless, various experiments have substantiated this principle across different populations and domains.
Let's delve into a classic example: estimating the number of peas in a jar. When a group of people is given the task of estimating multiple times, it becomes challenging to find a single expert whose accuracy exceeds the collective accuracy of the group. This is not a mere statistical exercise. Computer-generated results lack the nuanced intelligence that a group of diverse individuals possesses.
This phenomenon extends beyond estimating peas; it holds for predicting a person's weight, the current room temperature, election outcomes, and more. Even James Surowiecki, the author himself, initially found it difficult to embrace this concept fully. Despite writing the book, he continued testing the idea, and each time, the audience consistently outperformed, even in tasks like estimating the number of books in their library.
The wisdom of crowds is not confined to evaluation but applies to forecasting and quantitative and qualitative decision-making. The implications are extensive, advocating for increased board involvement in decision-making, comprehensive surveys within companies before product launches, and enhanced collaboration among intelligence agencies.
This principle underscores the notion that we may only sometimes discern which information is valuable. Encouraging diverse opinions and input from people, instead of solely relying on experts, enhances decision-making. While acknowledging the role of experts, two challenges arise from unquestionably relying on individual opinions, even if they are well-informed:
Identifying true experts is challenging, and the collective has an excellent ability to discern genuine expertise.
Even true experts possess biases and blind spots, making them susceptible to errors. Experiments on the wisdom of crowds reveal a low correlation between an expert's confidence and accuracy, highlighting that experts, too, can make mistakes.
The text also explores the role of incentives. Surprisingly, monetary incentives do not necessarily enhance the wisdom of crowds. While incentives can channel collective wisdom effectively, the rewards must be subtle and aligned with relevant motivations. In the long run, relying on the wisdom of the masses may transform decision-making, shifting away from dependence on individual leaders and their leadership style and fostering increased trust in collective capabilities.
Conditions that allow for it:
For mass wisdom to manifest, several conditions must be met:
Variety
Diversity is the primary prerequisite for the effectiveness of the wisdom of the masses principle. It necessitates various individuals with differing opinions, a range of products with varying levels of success, and a diversity of offerings, some superior and others less so. The system's success lies in identifying and swiftly eliminating less viable options. A system that facilitates generating commendable and subpar solutions discerns the weakest ones and promptly discards them, which is preferred over systems that promote reasonable solutions. One might argue that there is wisdom in the initial chaos.
Diversity contributes by offering multiple perspectives. In decision-making, a shortage of perspectives diminishes the overall efficacy. This holds for markets, but its impact is more profound in small groups or formal organizational units. While large groups from the general public may naturally possess diverse backgrounds compensating for the lack of alternative viewpoints, small groups or organizations, often characterized by a common viewpoint, benefit from an increased array of alternatives. Even suboptimal alternatives provide valuable perspectives that aid decision-making. It is crucial to recognize that the wisdom of the mass’s principle cannot thrive without diversity. Unthinkingly relying on this principle without careful analysis leads to erroneous results. Smaller groups or those within an organization may succumb to what psychologist Janis terms "groupthink." Groupthink can emerge when individuals share similar worldviews, making them more reliant on fellow group members and isolated from external perspectives, compromising objective judgment. Surowiecki illustrates this phenomenon with examples such as the failures at Pearl Harbor and the Bay of Pigs.
Furthermore, diversity offers an additional advantage—it helps maintain independence between individuals. Without diversity, observing crowds' wisdom in action would be challenging.
Independence between people
The second indispensable condition for the wisdom of the masses principle is independence among participants involved in evaluation, prediction, or any decision-making process. The significance of autonomy between individuals is twofold:
It prevents coordinated errors, ensuring that individual mistakes remain isolated and do not skew the collective judgment, as long as these errors don't systematically align in the same direction.
Independent individuals are likely to introduce new information, supplementing the existing knowledge already known to everyone. An intelligent audience is a diverse audience that maintains independence when reaching decisions.
The consequential insight is profound: individuals within the audience can harbor biases and engage in irrational thinking. As long as these biases vary in kinds, directions, and opinions, the collective judgment of the audience remains unimpaired. This underscores a critical point—while personal thought patterns may enhance individual intelligence, adherence to the same principles within a group can be detrimental, leading to a loss of independence and undue influence on one another.
Interestingly, the larger a group, the more susceptible an individual becomes to its influence. Consider a person on a street corner observing an empty sky: a few passersby might glance skyward, but it has minimal impact. Place ten people on the same corner, and a new observer might feel compelled to believe "there is something in the sky" merely because others seem to see it. In such cases, when a large group operates on the same strategy, it becomes challenging for an individual to advocate a new, different, and innovative idea, even if it is accurate.
The text also addresses the phenomenon of imitation, suggesting that it is acceptable as long as:
There is a broad spectrum of opinions initially.
Some individuals aspire to lead in decision-making ahead of others, acting as role models.
Surowiecki acknowledges the validity of the herd principle, emphasizing the wisdom of listening to the majority in many cases. However, he cautions against information cascades—instances where individuals follow a group without considering their information. Recognizing this distinction is crucial for maintaining a healthy state of a large, independent audience.
Fortunately, the text notes that more critical decisions are less prone to herd behavior and information cascades, making the wisdom of the masses a viable approach for significant decision-making. It highlights the importance of decentralization in decision-making, allowing independent groups to contribute and generate new, context-specific knowledge that may not emerge within centralized settings.
Aggregation
The third and final prerequisite for the effectiveness of the wisdom of the masses principle is the principle of aggregation. It is crucial to comprehend that having a large group of independent individuals with diverse opinions is insufficient if there is no practical way to gather, compile, and consolidate their views. This essence lies at the heart of aggregation.
An exemplary illustration is the Google search engine: the internet teems with information generated by independent individuals holding diverse opinions. The engine's excellence lies in its capability to assess the extensive, independent, and varied information and then reaggregate it in a practically usable manner. Conversely, a negative example, as per Surowiecki, is the conduct of the intelligence community pre-September 11. Despite decentralization and diverse organizations with independent opinions, there was a lack of integrated reaccumulation of information—an absence of organization.
Effective aggregation necessitates shared information accessibility for everyone, each bringing their unique perspective to allow a comprehensive view. This inclusive approach fosters acceptance of diverse opinions. However, a decentralized mechanism is required to bring these opinions together for informed decision-making. The challenge lies in achieving proper aggregation, a non-trivial issue that should prioritize enabling the majority's dominance in information sharing, decision-making, and local decisions based on this information. The recommended approach involves allowing the majority to influence these aspects before progressing to an overall decision-making step.
Types of decisions where the wisdom of the masses is relevant
Coordination issues
The wisdom of crowds proves to be highly beneficial in facilitating coordination among diverse individuals. Shared prior knowledge and a similar culture prompt people to make decisions that align, streamlining coordination even without direct communication. Successful resolution of coordination challenges stems from collective actions, often illogical at the individual level, contributing to an integrated logic and wisdom at the group level.
A notable experiment in New York exemplified this phenomenon. Students were tasked with arranging a meeting without specifying the location. Despite no explicit communication, almost all students converged at the same place—Central Park. A similar experiment regarding the meeting time demonstrated substantial coordination among the participants. This adherence to Schelling points highlights the tendency of people to converge on common choices under certain conditions, fostering coordination without direct communication.
Making decisions in such scenarios streamlines coordination among diverse individuals, contingent upon shared prior knowledge and conventions that facilitate reaching these reference points. Conversely, an example of complementary coordination is seen in crowded bars. When there's excessive congestion, people don't enjoy themselves and may avoid the place, and vice versa. However, over time, most bars and restaurants naturally achieve a balance, attracting a mix of regular patrons and new visitors who coordinate seamlessly without explicit communication.
It's essential to recognize that diversity is vital for the crowd's wisdom, although it can pose challenges in solving coordination problems. Flocks of birds provide insights into this concept, demonstrating that with a few simple rules or conventions and no direct communication, they can move with remarkable coordination, avoiding collisions. This behavior parallels the dynamics of the free market, showcasing excellent coordination constituting the wisdom of the masses, grounded in shared prior knowledge and derived conventions.
Collaboration issues
Cooperation issues resemble coordination challenges, with the vital distinction being that coordination problems find resolution even when individuals consider only their viewpoints. In contrast, cooperation problems necessitate a broader societal perspective. Examples of cooperation include tax regulation, snow removal from roads, and addressing environmental pollution. Collaboration is indispensable for the functionality of organizations and the broader societal framework. Relying solely on laws without citizen cooperation is insufficient.
The question of why people collaborate has been examined by political scientist Axelrod, who posits that people cooperate more readily in long-term relationships, where trust naturally evolves. Others argue that economic success results from embracing trust, even with unfamiliar counterparts. Being unreliable is economically unviable, leading most companies to consistently act reliably, fostering a healthy economy—an idea facilitated by capitalism, albeit with potential loopholes for those seeking to exploit it.
A comprehensive analysis of cooperation reveals that individuals are inclined to collaborate and contribute to the collective wisdom of the masses, even if it comes at the expense of their interests (such as paying taxes), as long as others engage in cooperation. Surowiecki's extensive experiments establish that, in most cases, individuals tend to cooperate when the majority does. However, there remains a steadfast core of non-cooperators (or even cheaters) and a core of altruists who consistently cooperate. The key lies in influencing the majority (conditionally agreed) in the right direction.
Three conditions foster an individual's cooperation for the greater good of the organization/society:
To some extent, trust in other individuals and their inclination to cooperate.
Trust in the system (organization, country, or company) seeking collaboration.
Trust in a system that imposes consequences for non-cooperation, acting as a deterrent.
Thinking/cognitive issues
On intellectual matters, Surowiecki's statements are clear and straightforward:
The audience may not provide better answers than experts individually, but the collective response of the audience consistently surpasses everyone's answer.
For the mass response to be genuinely qualitative, some bias is permissible if it is diverse. Caution is advised in markets that deviate from this principle, such as stock market bubbles or the reverse process of a stock market fall.
More information only sometimes enhances decision-making! Despite having information, people may not fully utilize it. Encompassing parameters, like the mere existence of knowledge, can disproportionately influence decisions, leading to errors even more significant than if no information were present.
The wisdom of the masses thrives in tackling complex thinking issues (stemming from the diversity conditions mentioned earlier): maintaining a balance between information and knowledge shared by the entire group and additional private knowledge held individually. This dynamic produces diverse perspectives crucial for wisdom.
However, thinking issues are more straightforward compared to coordination and cooperation challenges. While the latter evolve with the development of trust and familiarity, intellectual issues are consistently addressed by the wisdom of the masses, both presently and in the future. Of course, the necessary conditions must be met for such knowledge to prevail.
Insights in the scientific world
Through a series of examples, Surowiecki elucidates how the scientific world advances through the principles of the wisdom of the masses. An illustrative example is the response to SARS a few years ago, showcasing exemplary scientific work transcending boundaries. Within a few weeks, the cause of SARS was diagnosed and identified, with the process involving:
Contacting major countries and deploying teams of researchers and accompanying laboratories in each country.
Daily transatlantic conversations where each team shared progress, successes, and unsuccessful findings.
After knowledge sharing, each laboratory independently determined its approach to studying the disease over the next 24 hours and beyond.
This process involved a diverse group of people operating independently between teams, yet it served as an excellent tool for aggregating and sharing knowledge. No single manager was instructing actions, and some teams engaged in seemingly redundant activities. Nevertheless, the analysis and response development rate was among the highest recorded in the field.
Scientific work, in general, is characterized by decentralized efforts involving various factors. The sharing and publication of findings form a crucial cornerstone of scientific research. Motivating collaboration and sharing findings pose challenges, often resulting in delays in publication. According to Surowiecki, the primary incentive for researchers is recognition and acknowledgment, which are achieved through advertising and prompt continued sharing.
However, cooperation in all areas is not universally widespread. Experiments reveal that collaboration among those with knowledge is not guaranteed despite sharing findings. Researchers often prefer working with close associates rather than with other groups. Extraordinary endeavors like the response to SARS significantly advance scientific activity.
There has been a shift in recent years as commercial organizations, particularly pharmaceutical companies, conduct more studies. In the commercial world, information sharing is limited, and information concealment is prevalent. While there is partial protection with the publication of patent details as part of rights obligations, it does not offer a comprehensive solution. The resulting loss to science as a whole needs consideration.
Insights in the corporate world
Not all phenomena observed in the wisdom of the masses are replicated in the organizational world. Organizations typically operate in small groups, making it challenging to assert that these groups are diverse or that individuals within them make independent decisions. While there is an advantage to collective knowledge in group settings, managerial wisdom is essential to ensure beneficial teamwork without succumbing to erroneous decision-making, as evidenced in Surowiecki's detailed analysis of the Columbia spacecraft failure.
Surowiecki explores how group work can lead to opinion radicalization and polarization, particularly under certain conditions. In organizations where employees are familiar with each other, patterns of conduct and information exchange can emerge, hindering the free flow of knowledge. Selective listening is influenced by factors such as the speaker's hierarchical status, politics, and perceived talkativeness.
Surowiecki suggests defining rules for seating arrangements, information flow, personal decision-making, and sharing to navigate these challenges in small teams. Under such conditions, polarity is less likely, and teamwork can contribute positively to organizational progress.
Despite existing within a free market, the corporate world often operates differently, relying on programs, orders, and tasks to achieve its goals. Surowiecki recommends embracing more democracy, characterized by decentralizing decision-making, promoting the free flow of knowledge regardless of hierarchy, and altering the organizational incentive structure. Distributing options in the company is presented as an alternative incentive model, fostering a sense of partnership among employees and aligning their decisions with the organization's long-term goals.
Moreover, encouraging employees to propose ways to improve the organization and decentralization enhances the likelihood of implementation. Surowiecki emphasizes the importance of feedback from the market, especially from frontline workers, in instigating meaningful change.
Regarding management, the phenomenon of managers being perceived as superstars in the '90s is debunked. Surowiecki argues that such a perception is not only unrealistic but also detrimental to decentralization and long-term societal advancement. True managerial success lies in empowering subordinates and facilitating their contributions to organizational progress rather than aspiring to be a superstar. Effective managers acknowledge their limitations in knowledge and decision-making, showcasing true greatness.
Comments