Thursday, 31 January 2008

The Wisdom of Crowds

I was in two minds whether to read James Surowiecki's book, thinking that the main conclusions could be drawn from the title. And it was pretty hard to wade through, given that I could only read it in 22 minute bursts (the length of my train journey each day). But I'm glad I persevered!

The subtext is "Why the many are smarter than the few", but it could just as easily have been "How to get the best out of a team". It presents a remarkable stream of evidence to suggest that certain types of decisions made by groups of people are consistently better than decisions made by leaders or experts. The inevitable errors made by individuals in a group cancel themselves out time and time again. Numerous examples are given, ranging from poor decisions made by NASA prior to the Colombia disaster, to great decisions made by groups of people betting on the points spread at an American Football game, to the mechanics of the stock market. It's compelling stuff.

The repercussions for a team or a company are immense; top down leadership doesn't work unless the leaders can find a mechanism for harnessing the collective 'wisdom' of the people below them in the org chart. The deeper the hierarchy (or the bigger the ego of those in charge), the tougher the challenge.

Surowiecki identifies three characteristics that make a crowd 'wise':
  • Independence - the people in the crowd need to be unbiased
  • Diversity - that's cognitive diversity rather than e.g. race or gender, so they should come from as wide a range of backgrounds as possible, collectively bringing a wide range of information and perspectives with them
  • Private judgment - be allowed to make their decision without duress
Hmm. Independence, diversity and private judgment. Sounds rather like an open source project to me.

The book casts damning judgment on CEOs and their ilk, claiming that their success is more down to luck than skill. After analysing several CEOs who crashed and burned in a new job after success in their previous role, Surowiecki says:
It's natural for us to look at successful people and assume that their success is due to some innate quality they have rather than to think that it might be the result of circumstance or chance...."CEOs should come with the same disclaimer as mutual funds: Past success is no guarantee of future success".
Quite a humbling notion for those on the board who pay huge salaries to their CEO, or the trader who buys stock just because a new CEO has a good track record.

Of course there are some decisions which aren't suitable for collective decision making. Anyone who's tried to design a website by committee can tell you that. The challenge for anyone leading a team or a company is to try and identify which decisions are best made alone, which are best made by the crowd, and - if the latter - how this wisdom is harvested.


FND said...

I agree - Surowiecki's book is a must-read (also see my previous assessment).

If I ever find the time, I'll go through it again and summarize all the important aspects (like the conditions for success you've listed - there are a few similar things that are worth extracting).

shaidorsai said...

OpenID sign in fails again. I retype:
I was similarly impressed by Wisdom of Crowds. Do you feel that some of its contents supported "Fooled by Randomness" (Nassiem Taleb)?

I thought that the Tipping Point(Gladwell) was a worthwhile read alongside it - which emphasises the role of the maven.

Do we like the wisdom of crowds influenced by a key few?

Phil Whitehouse said...

I haven't read "Fooled by Randomness", but his other book "The Black Swan" is on my list of books to read!

My feeling is that The Tipping Point and The Wisdom of Crowds have a small overlap, to the extent that a large and diverse crowd is more likely to spot an emerging trend than a smaller team. And I rather like the notion that a diverse crowd can evaluate an emerging trend to determine whether it has value.

However, I'm thinking that this benefit only applies in the early stages of a trend emerging; as knowledge of the trend becomes more widespread, and people learn about the trend through e.g. the wb or newspapers, the people in the crowd become less capable of making an independent assessment, and their collective wisdom suffers as a result.

Interesting discussion point shaidorsai, thanks!