Quantification has been on my mind. The question comes up in sales conversations, compensation discussions, product development processes and marketing messages. Heck, it even comes up when you’re putting together a resume or when you’re explaining to your parents why you’re fine making a little less money for a better job.
I find this search for quantification boring and missing the mark. Here’s my problem: numbers have severe limitations. There’s another issue too: numbers give this false sense of security in judgment. If you are analytical, you can rely on the numbers for too many of your answers. It limits you or allows you to be lazy. If you have a decision to make and the numbers go one way, it is easy to point to that and say yes, do that. When it goes wrong, it is easier to justify the mistake to colleagues when you go with the numbers. It is easier to justify that approach with investors. But here’s what I know, not everything important can be measured.
Player Worth Is More Than A Set Of Numbers
Here’s my handy sports analogy because it works: if you only followed the NBA by watching box scores and knowing the salaries of the players, you would be confused. For example, you’d see that the top salaries were paid to players that scored the most points. As you got further down the list though, the impact of a player’s scoring became less and less of a factor in determining their salary. Meanwhile, you’d see guys pop up on the list who were way down in scoring and had maybe only a couple more rebounds, blocks or steals per game than many of their contemporaries.
It happened everywhere: in every front office, in every city and on every team. Somehow these people were valuable but the numbers didn’t support it. Was everyone in the NBA that bad at evaluating talent?
The Rest Of The Story…
Offense in the NBA is important but so is defense. Unfortunately defense is incredibly hard to quantify in relation to impact. Sure, you had rebounding and block leaders but even those raw statistics didn’t capture what a great defensive player can do to change a game.
Don't mess with this guy.
Take one of my favorite guys from the Blazers named Buck Williams. On paper at least, his extra talent may have impacted 4-6 possessions of a 200 possession game. Why was he a bigger factor than that? Four things that can’t be quantified but were critical:
- Pulling down more rebounds on defense meant that the wing players could run out the court and get fast break opportunities they couldn’t have received if they had to stay in and help rebound.
- It made playing man to man defense much easier. If your guy got around you, you knew help was behind to assist. It allowed your other players to play more aggressively on the ball.
- Simply contesting a rebound impacts the flow of the game. When you fight for the ball on a rebound, you can disrupt the other team’s flow because they have to adjust for the extra time it will take to field the ball.
- He kept other players away from the basket and when they came close, he was able to contest shots. While an uncontested shot within ten feet may be a gimme, a contested shot had a substantial impact on scoring.
One of the other guys that was good at this was Mark Eaton of the hated Utah Jazz. He seemed to be omnipresent within eight to ten feet of the basket and made inside scoring difficult for any team. His blocking prowess was good but was only two blocks above what other centers were doing in the league. His defensive presence impacted game plans and allowed the Jazz to be a better team than they deserved to be despite fairly average career numbers outside of blocks.
Neither one of these guys are in the basketball hall of fame. Neither are guys like Alvin Robertson, Dennis Johnson, Michael Cooper, Horace Grant and Maurice Cheeks. They probably won’t ever be since that is often a numbers game too. But they made an indelible impact on their teams.
Not Everything That’s Important Can Be Quantified
Not to get too philosophical on you but if you’re married, what’s your ROI? What’s the break even mark for helping your grandma clean out her gutters? Unless you’re crass enough to marry for money or help family members for a slice of inheritance, I am guessing numbers didn’t cross your mind.There is great value there that adds to your life in ways that don’t show up on your bank statement or resume.
Even though we internally realize that many important things can’t be quantified, we still mindlessly pursue quantifying until we find our answer. Why is that? We certainly can’t trash the idea of ROI (we still do rely on money to run our businesses) but we can also focus on important parts of our business that don’t show up on spreadsheets, don’t have a profit margin and won’t show up in EBITDA.
You have to find the Buck Williams’ and Mark Eaton’s of your company and recognize them. Their individual stats may not show it but I guarantee that everyone around them is inexplicably better because they are around. The focus on quantification and supposed fairness has pushed these people out of organizations in some cases.
Are you willing to make that same mistake? Are you going to allow your spreadsheet to dictate a talent evaluation process that really requires deeper investigation than just a cursory glance at numbers?
{ 9 comments… read them below or add one }
Excellent points, Lance! I really like your example of basketball team performance, especially how you analyze the complex interactions that take place which show that box scores alone are insufficient to making hiring/pay decisions. For the moment staying solely in the “we’re in business to make money” train of thought vs. “I love my grandma” one, I believe that people get tripped up not so much on trying to measure things that can’t be measured, but in being overly focused on the idea of “one number”. Examples might be, “How much extra revenue will we get if we hire salesperson X?”, “How much will we save if we implement system Z?”, “How much oil will we extract from field A vs. field B?”, “What is the ROI of Project Y?”, etc. These are all very reasonable questions to ask because we need to make business decisions based on the answers. The trouble is, as you allude, people want a clear, yes/no or choice a/b/c kind of answer. “Why can’t you just give me the number?” is the common cry. The real answer, that folks would rather not hear, is that it’s “uncertain” or at least it’s “complicated.” Neither a basketball team’s performance nor the impact of one player on its performance is completely predictable. However, we can reduce the uncertainty in our estimates of what will happen by watching, measuring, analyzing, testing, experimenting (repeat). At some point we will be confident enough to make a decision, depending on our individual or organizational aversion to risk. It’s this unease that people have with uncertainty and risk that makes them wish for realistic levels of certainty, or worse yet, imagine that the numbers they see in a spreadsheet convey certainty. Thanks for bringing this point up, Lance.
Love the post, Lance. I remember Buck mostly from his days on the Nets and he was as you say just a tremendous and underrated player. Sometimes in organizations we miss the Bucks and Mark Eatons of the team since they may not always be in the ‘right’ place on the org chart or have what is assumed to be the correct pedigree. I know Mark Bennett and I both have written about the use of social network analysis to help uncover these unsung and really important members of the company, and perhaps in 2010 we will see more of this method getting adopted.
Basketball is still behind the sabermetricians covering baseball, but it is improving. Take Michael Lewis (of “Moneyball” fame) and his somewhat recent profile of Shane Battier and Daryl Morey (http://www.nytimes.com/2009/02/15/magazine/15Battier-t.html?_r=1&pagewanted=all)
2004 also saw the publication of one of the best books of basketball statistical analysis, “Basketball on paper” by Dean Oliver (http://www.amazon.com/Basketball-Paper-Rules-Performance-Analysis/dp/1574886886/ref=sr_1_2?ie=UTF8&s=books&qid=1262900680&sr=8-2) who set out the four factors key to basketball success: Shooting, rebounding, turnovers, and getting to the foul line.
David Berri is a sports economist who carried out some of Oliver’s work in the 2006 book, “The Wages of Wins” (http://www.amazon.com/Wages-Wins-Measure-Stanford-Business/dp/0804758441/ref=sr_1_1?ie=UTF8&s=books&qid=1262900680&sr=8-1) and he has a new book coming out in a couple of months.
I’m more of a college hoops guy, and I find ken Pomeroy’s site indispensible (www.kenpom.com).
Mark and Steve, Thanks for the thoughtful comments. You guys are both on the right track. Obviously we can’t discard quantification but we can look beyond it and look for meaning.
akaBruno, my feeling is that it is always going to be harder to quantify basketball players than baseball because baseball is still so much about individual talent. When you can figure out within a statistical probability what an ace pitcher will add to your win column (and it actually works consistently), I just don’t think you can do that with basketball.
Lance, I am a quantifier, so I’ve got to push back on this. I think things can be quantified, I don’t think it’s always easy so we often make the excuse that it can’t be quantified, or it’s not worth quantifying. I’ll start with Grandmas gutters example (I think AKABruno handled the BBall one well). Although you may not have reference point for quantifying the good will, there is still quantification that goes into the decision to clean her gutters. I love my grandma, but I also know what my time is worth. Therefore I have to quickly “run the numbers” and determine if it makes better sense economically to do it myself or pay someone else to do it. The good will factor is probably the same, Grams gets the gutters cleaned, so it’s all about numbers.
When it comes to performers in a company it is up to them and their managers to quantify their contribution. That’s one thing managers are paid to do, they shouldn’t have anyone working for them that they can’t justify having on the payroll. Again, I know it can be difficult in some cases, but it’s worth doing.
Excellent post and great discussion!
Lance, first, thanks for the shout out to us Bulls fans. Very classy.
Second, great post, and great point. I’m sure you’ve seen the performance review I did with my daughter when she was in kindergarten… I often use that to make the same point. I did have one group that was particularly anti-the-idea-that-numbers-don’t-tell-the-whole-story… so I asked them to quantify love. Where they ended up was a bit disturbing, b/c the name you give to a romantic relationship that is purely transactional is the opposite of the one that is formed on love—even though many of the quantifiable activities are the same.
I’ll leave it at that.
Again, nice post.
You’re a very clear-headed and intelligent person.
I like the way you think and I’ve only just read a couple specific posts.
Thanks for the more lateral thinking.
Good post, other than the fact that you mistakenly call the awesome Utah Jazz “hated”!
On a serious note however, you are correct that an overreliance on numbers can obscure the true value of talent, but the challenge is that HR practitioners are making decisions based purely on intuition and qualitative factors. If they do introduce quantitative measures, the basis of these measures is completely without a defensible methodology.
The corporate equivalent of your basketball and Hall of Fame analysis is the sales organization. Sales is the offensive powerhouse and those that succeed by pulling in the big deals get the deserved recognition. This is because without sales, there is no revenue, and without revenue, there is no company. It is also very easy to measure sales and sales effectiveness. That does not mean that other functions that support the sales team or build the products or run the operations of the company are not important, it just means that there is a recognition that the contribution of sales keeps the company in the game. Without Malone and Stockton, Mark Eaton’s efforts would not have mattered.
I think that we can agree though that it is important that these key contributors are identified and appropriately recognized. These folks typically occupy what I deem “Positions of Pain” because of the very fact that you mention; they are unrecognized yet provide an outsized contribution compared with the status of the role they occupy and position in the organization.
What I think would be a disservice is to not develop an appropriate means to highlight their contribution in a way that measurably shows the extent of their impact. I believe there is a way to intelligently use metrics to provide insight into performance based on the needs of the job as a way of supporting (or refuting) perceptions of one’s performance. That seems to me to be a much fairer and rational way of recognizing talent across an organization.
Good post, other than the fact that you mistakenly call the awesome Utah Jazz “hated”.
On a serious note however, you are correct that an overreliance on numbers can obscure the true value of talent, but the challenge is that HR practitioners are making decisions based purely on intuition and qualitative factors. If they do introduce quantitative measures, the basis of these measures is completely without a defensible methodology.
The corporate equivalent of your basketball and Hall of Fame analysis is the sales organization. Sales is the offensive powerhouse and those that succeed by pulling in the big deals get the deserved recognition. This is because without sales, there is no revenue, and without revenue, there is no company. It is also very easy to measure sales and sales effectiveness. That does not mean that other functions that support the sales team or build the products or run the operations of the company are not important, it just means that there is a recognition that the contribution of sales keeps the company in the game. Without Malone and Stockton, Mark Eaton’s efforts would not have mattered.
I think that we can agree though that it is important that these key contributors are identified and appropriately recognized. These folks typically occupy what I deem “Positions of Pain” because of the very fact that you mention; they are unrecognized yet provide an outsized contribution compared with the status of the role they occupy and position in the organization.
What I think would be a disservice is to not develop an appropriate means to highlight their contribution in a way that measurably shows the extent of their impact. I believe there is a way to intelligently use metrics to provide insight into performance based on the needs of the job as a way of supporting (or refuting) perceptions of one’s performance. That seems to me to be a much fairer and rational way of recognizing talent across an organization.
{ 2 trackbacks }