The Pareto Principle (also known as the 80-20 Rule) highlights the disproportionate importance of the most important elements of a system, set or list. Pareto first noticed this 100 years ago with respect to how much land (80%) was owned by the top part (20%) of Italian society. It has been applied widely as a general idea, as with software developers who were wise to really optimize the speed of the small part of their programs that did the vast majority of the work. Virtually everyone can understand the idea, "I spend 80% of time doing 20% of my job," even if the actual numbers are a bit off. Airport bookstores demonstrate this, as they carry very few titles, but because they are focused on best sellers they have decent total sales. In the figure below, that green part is quite a lot of the total area, even though it covers just a small fraction of the x-axis.
 |
| Pareto and Tail |
In The Long Tail (2008), Chris Andrews explores how with online commerce and massive centralized warehouse retail, the massive back catalogs of book publishers are much more valuable than they used to be. In fact, the total sales of all the books that sell relatively few copies in a year are greater than the total sales of the top ten best sellers. There just are so many poor selling books that in the aggregate they can swamp the few at the top of the lists. He explains how this phenomenon is not just limited to the books, but rather applies to a much wider world of commerce. He dubs this large set of low sellers "the long tail." A major theme of the book is that the Internet enables a virtually unlimited potential customer base, a virtually unlimited catalog or selection, and the means to connect them. Netflix offers a far larger selection of films through its website than any local video store could, and streaming video makes delivery even easier than its original distribution system. Above, you can see that the long yellow tail can add up to quite a bit as the tail extends.
In many ways, the Pareto Principle or 80-20 Rule seems at odds with The Long Tail. The former focuses on the most common/most popular/most important ones, and the latter emphasizes aggregate importance of all the rest. The 80/20 Rule, however, cheats. As variety increases (e.g. the size of the back catalog, length a program, parts of your job, etc.), one cannot simply say, "Oh, I'm still going to consider only the top 20%." Your brick and mortar bookstore is of a set size and your employee evaluation form still only has room for your top three responsibilities. The top 20% remains incredibly important, of course. But the top 10 become relatively less important as one further considers the sheer variety.
My Principle of Variety points out why the Long Tail phenomena is more important than the Pareto Principle. That is, the more complex the phenomena or system is, the more we tend to overestimate the importance of the top handful of elements.
Schwartz and Ward's Paradox of Choice (2004) explains why the Pareto principle has such outsized influence. As complexity, variety and choice increase, people can get paralyzed and overwhelmed. Thus, the idea that we can simply focus on a fraction of the total selection makes an unmanageable system or decision appear much more manageable. Reliance on the Pareto Principle, even unconsciously, inoculates us against the paralysis and overwhelmedness that Schwartz and Ward describe.
While the Pareto Principle underlies a important coping mechanism for day-to-day decision making, my Principle of Variety points out that it is incredibly problematic at other times. When thinking about whole systems or rules for whole systems, we are virtually guaranteed to overestimate the relative importance of the most easily examined elements. We need to be informed by Hamlet's advice to Horatio, "There are more things in heaven and earth than are dreamt of in your philosophy." We need to recognize that there often exist unaccounted for (or unknown) elements of the system whose aggregate importance is significant. The tail is often longer than we thought.
That matters quite a bit in education policy. There is enormous complexity and variety to the reality of accountability, but we have generally reduced it to test-based accountability. We have tried to increase the importance of test-based accountability by attaching sanctions to poor performance, by increasing publicity for performance, by standardizing tests more and more to ease comparisons, and sometimes (though rarely) trying to improve the quality of those tests. We also pay some attention to graduation rates, and perhaps to college preparedness. But all these remain a small fraction of the total number of accountabilities that schools and school leaders face.
The more one understands the breadth of variety of accountabilities for schools and school leaders, the better you see why tests and test scores have had so little influence on our schools. In fact, no matter how important we make some small set of accountability measures, they will be dwarfed by the long tail of the variety of real accountabilities.
The Pareto Principle suggests that we can deal with an outsized fraction of the total accountability, but we actually are addressing a far smaller fraction of accountability than we realize. The Principle of Variety points out that the long tail is far more important than we realize.
