Malcolm Gladwell's NBA "Mismatch Problem" Problem

Also see: Analysis - Kobe & Pierce: A Tale of 2 Pick-and-Rolls

I've always generally enjoyed the work of Malcolm Gladwell, best-selling author of The Tipping Point and Blink as well as a staff writer for the New Yorker. In a piece that I wrote for Ballhype in March, I included a theoretical "Malcolm Gladwell's Sports Book" as one of the five sports books that I'd like to see written. As I wrote then:

We're thinking that something NFL-oriented might be up his alley - maybe a thorough look at theories on how a team or an organization should be built, how game plans are constructed and executed, or how draft prospects are thoroughly scouted, vetted and ultimately selected.

Unfortunately, after watching a speech in which Gladwell addressed NBA, NFL and NHL scouting combines, I may have to take it back.

Some background first: Gladwell's next book (due in Nov.) is called Outliers: Why Some People Succeed and Some Don't- a glimpse of which was seen in a recent New Yorker piece.

As described by publisher Little Brown (via kottke.org):

Outliers is a book about success. It starts with a very simple question: what is the difference between those who do something special with their lives and everyone else?... We're going to examine the bizarre histories of professional hockey and soccer players.... And in examining the lives of the remarkable among us -- the brilliant, the exceptional and the unusual -- I want to convince you that the way we think about success is all wrong.

One subject in Outliers is what Gladwell calls the "mismatch problem," which refers to his idea that the qualifications which people look for/companies require in candidates when hiring often do not necessarily have any correlation to successful job performance - and often leads to poor hiring decisions.

In a recent speech at "Stories from the Near Future," the 2008 New Yorker Conference (which can be seen here), Gladwell tried to explain his idea of the "mismatch problem" by relating it to the role that scouting combines play in terms of how NHL, NBA and NFL teams decide which players to draft.

The problem is that the logic Gladwell uses to relate combines to the "mismatch problem" is sloppy, simplistic and contradictory to the point of being nonsensical.

Gladwell says this about scouting combines in general:

As it turns out, if you look closely at the scouting combine as an institution, and as a kind of metaphor for hiring, it turns out it doesn't make sense at all. Scouting combines are, for the lack of a better word, a disaster.

Since this is a basketball blog (and since we're just a day away from the NBA Draft), we will focus on Gladwell's thoughts about the NBA scouting combine.

Just to clarify, when we talk about the "scouting combine," we mean the testing that is done at the pre-draft camp in five areas of athleticism: no-step vertical jump, max vertical jump, bench press (number of times 185 lbs. is lifted), lane agility, and 3/4-court sprint. Measurements of height, weight, wingspan and standing reach are also done at this time.

Prior to 2008, the players were given an overall ranking based on their combined results on the five tests.

The core of Gladwell's argument was that the top five rookies in the league this season had wildly variable rankings at the 2007 combine. Here are the players he cited as the five best rookies, with their combine rankings (out of 81 total players):

Greg Oden 62
Kevin Durant 78
Al Horford 21
Thaddeus Young 7
Joakim Noah 43

Meanwhile, he pointed out that the top three ranked players at the combine included one scrub and two guys who aren't in the league.

D.J. Strawberry 1
Russell Carter 2
Dominic James 3

That was that, pretty much the entirety of Gladwell's argument, which he concluded by saying that the combine is "an absolutely useless predictor of how someone performs at the professional level."

Here are my issues:
1. He's sloppy with the data.
Chad Ford published the results of the 2007 combine on ESPN.com (Insider) last year.

In a scan of the numbers, it sure seems like Oden's score of 0 times bench-pressing 185 pounds is curious, and indeed, Ford's first piece of analysis is this:

The big winner here seems to be Greg Oden, who tested off the charts in agility and speed for a big man. He didn't take the bench press test because of concerns about his wrist. But if he had he may have been in the Top 10 for best athletes in the draft. That's huge for a center.

So for Gladwell to assert Oden as ranking 62nd as his first piece of evidence is wildly misleading, and a lazy lack of simple analysis (the 2007 combine data, on Draft Express, is the second result returned by a Google search of "NBA combine" and Oden is the second name on the list, with "NA" listed under his Bench Press results).

2. The arguments - based on the data as it's presented - are contradictory.
Gladwell's statement that the NBA combine is "an absolutely useless predictor of how someone performs at the professional level" may actually be accurate.

One of the foremost NBA draft experts, Jonathan Givony of Draft Express, made a persuasive argument last year that the combine results are essentially pointless.

But think back to what Gladwell is trying to prove with the "mismatch problem": that people make poor hiring decisions because the criteria and data they use to evaluate candidates does not correlate with job performance.

That's the thing: the data that Gladwell presented actually indicates that scouting combines *do not* create a mismatch problem. If the top three picks in the draft were rated 62, 78 and 21, and the 1-2-3 athletes at the combine produced just one late second-round pick combined, then all Gladwell proved is that teams are *not* overvaluing the results of the scouting combine when making their hiring decision.

Note: Givony accurately notes several occasions when the scouting-combine results have been overvalued. I'm not saying it never happens - I'm mainly trying to note that Gladwell's argument is contradictory as presented.

Also, as a smaller point, let's go back to his list of the top five rookies and add another piece of data: where they were drafted.

Greg Oden 62 - Drafted 1
Kevin Durant 78 - Drafted 2
Al Horford 21 - Drafted 3
Thaddeus Young 7 - Drafted 12
Joakim Noah 43 - Drafted 9

Doesn't this data, as it's presented, indicate that Young was the one who was drafted furthest below his potential, considering he ended up being judged as the 4th-best rookie but was picked 12th? And wouldn't that seem to suggest that his high ranking at the combine might have actually been undervalued?

I'm not saying I agree with that point, I'm just saying - based on how the data was presented, with no other context - isn't that a valid piece of analysis that could be drawn?

It's just a mess to me at this point - Gladwell's trying to argue that the scouting combine is useless, but the data he presents seems to indicate the combines aren't actually used, and all the while there's no mention that there are other factors which go into NBA "hiring" decisions.

3. I believe there is value in the combine.
As much as I respect Jonathan Givony and the entire Draft Express site, I have to respectfully disagree with his argument that the combine is a complete waste of time.

I do believe that there is value in the combine data if the numbers are analyzed closely and with some sense of relative value - is this player more or less athletic than I might have realized?

In Chad Ford's analysis of the 2007 combine results, these were the winners (again, relatively - not the guys who should be drafted first, but guys whose draft "stock" could be most affected):

Greg Oden
Jason Smith
Nick Young
Rodney Stuckey
Jared Jordan

Al Horford was judged to have an advantage over Brandan Wright, and Corey Brewer did not seem to fare well against other small forwards.

Ford's other underachievers included:

Kevin Durant
Spencer Hawes
Josh McRoberts

Durant's note carried this disclaimer:
"Will it affect his draft stock? According to one NBA executive, 'No one will care, he's a basketball player. But if you're comparing him to Oden, then yes, Oden is the big winner.'"

The jury is still out on the Oden comparisons, of course, but people certainly kept Durant's poor combine results in perspective.

And overall, I think that the 2007 data is decent in retrospect, as it identified one of the biggest overachievers (Stuckey) and biggest underachievers (Brewer) relative to draft position in this rookie class. I'd also note that probably the biggest overachiever, second-rounder Carl Landry, ranked 5th at the combine.

I'm not saying this is foolproof or spectacular data, but I also don't think it's "an absolutely useless predictor of how someone performs at the professional level."

In 2006, these were among Ford's winners:

Ronnie Brewer
Jordan Farmar
Randy Foye
Brandon Roy
Rudy Gay
Tyrus Thomas
Rodney Carney
J.J. Redick

All but Thomas and Redick appear to be very good-to-excellent value picks.

These were the main losers:

Marcus Williams
Mardy Collins
Patrick O'Bryant
LaMarcus Aldridge
Adam Morrison

Big miss on Aldridge, but Morrison and O'Bryant look like two of the worst picks of the draft.

The 2005 combine analysis had some big hits - Deron Williams (whose athleticism had been questioned at Illinois) and David Lee boosted their stock by performing well - and some big misses - Joey Graham ranked no. 1 overall (sorry Raps fans) and Monta Ellis somehow scored near the bottom at the combine.

I really don't want to overstate my position on this point. I actually agree with Givony on the idea that scouting combine data is often overvalued, I just don't agree with the Gladwell sense that it's "absolutely useless."

Again, I think there is some value in the combine, but I fully realize it's a minor piece of the puzzle, and that the conclusions drawn from the data are often wrong. But you know what, subjective evaluations are often wrong, too. And as utterly cool as I think John Hollinger's work is in translating college stats into professional performance, you'll notice that his statistical projections have plenty of misses, too. The draft is just an inexact science no matter how you try to slice it.

I thought that TrueHoop's outstanding interview with Nets Coordinator of Statistical Analysis Ken Catanella from Tuesday really put things into perspective well:

I know a lot of teams wrestle with how to integrate new-breed analytical work into their decision making process. Can you give me glimpse of how that works for the Nets?
Taking a step back -- my background is in valuing companies for investment banks or mutual fund companies. It's making a projection based on past performance, trying to answer the question "what here really indicates likely future success?"

Figuring that out really starts in conversations with the coaching staff, and trying to get a sense of what kinds of stats could be important. I did a lot of that when I was on Coach K's staff, and I'm lucky enough to work with Lawrence Frank.

Then, based on those insights, we do a bit of regression, which helps to project how different players' careers might play out in the NBA.

What goes into the mix is not any one magic number, though.

There are typical efficiency stats. There are anthropometric ratings like wing span, body fat, vertical leap, standing reach and the like. There are strength of schedule ratings. There is looking at who they were on the court against, and who they were on the court with. There are +/- numbers. The numbers really run the whole gamut. You try to find anything that you can measure that might be helpful, and put it all in the mix.

Do I sense that you are not even looking at the same sets of numbers for every prospect?
It depends on the position. And it can get very detailed.

We chart, essentially, every game that every draft prospect has played on video, and we track just about every category you can imagine.

Closing out the shooter, winning loose balls ... all that stuff?
All that and much, much more.
...

Also, scouts are very valuable tools, too. I never want to underestimate them. We can work together by getting data into their assessments early, seeing a players' characteristics. Trying to get a full picture, using playing stats, evaluation stats, and the statistical part of the psychological evaluation ....

As Catanella paints the entire picture above, the point is that the combine data is just one set of data among a substantial number of metrics. To make the combine data an end-all/be-all would be absurd, but I do believe that evaluating combine numbers as pieces of data as necessary in a larger, well-defined context does have value in player evaluations.

4. It's all just too much of a sweeping, simplistic generalization.
That last point above ties into one of my main issues: Gladwell is just way too breezy and simplistic with the information. His argument is basically this: scouting combines are useless predictors of professional performance, NBA teams use scouting combines as part of their evaluations, therefore NBA teams make poor hiring decisions.

What Gladwell's missing is that NBA personnel decisions are actually made based on a tapestry of information, not just a single set of metrics, and it's really an increasingly fascinating story to dig into, as the window that Catanella opens to the process seems to indicate. It's a shame that Gladwell - especially given his influence as a leading cultural commentator - offers such a shallow view.

One of my favorite descriptions of the process was by Mike Born, Director of NBA Scouting for the Blazers, who described the team's approach of "Eyes, Ears and Numbers" in an interview with Blazers Edge:

Blazersedge: What kind of data do you analyze? How much of scouting is numbers, how much is observation, how much is "feel"?

Kevin Pritchard brought in a saying, I assume it was from San Antonio: "Eyes, ears, and numbers".

"Eyes" means what you see. Does a player have a feel for the game? What's his basketball IQ? Does he play winning basketball? What are his skills: shooter, ball handler, athleticism, size, length? You gather visual impressions of what you like and don't.

"Ears" has to do with culture, which is obviously a huge factor for our team. How does a guy fit? We do research on players so we know coming in whether they'll mesh easily or not. This entails talking in person with coaches, calling assistants and strength coaches, building the book on your man. Are they hard workers? Are they dependable? What are they like in the locker room? Do their teammates like them? Do they show leadership skills? And that's just the on-court stuff. We also want to know if they will be good in the community...what they do in their off time and that kind of thing.

Numbers are simply stats. For college we look at things like scoring, field goals, rebounds per minute, assist-to-turnover ratio. We also do quality of opponent analysis. We want to know if a guy has been playing against the best competition and how he fared. We try to look at back-to-back game and one-day-rest patterns to gauge how a guy will hold up physically. For the NBA we have a simulation guy who uses his own stats analysis. You weigh stats and strengths, digging deeper than the normal boxscore. For instance, which is better: a 90% free throw shooter who goes to the line 3 times a game or a 70% guy who goes 8 times? The boxscore highlights the 90% guy, but is he really more valuable? Our simulation guy doesn't watch many games. He just goes by the numbers. It gives us a different perspective. It lets you watch players differently.

The reality is that the process of making NBA hiring decisions is much more layered and multi-faceted and complex than Gladwell gives it credit in his simplistic speech. I wish that he would have, for example, dug into the San Antonio Spurs organization and the totality of their "Eyes, Ears and Numbers" approach around the world, instead. I can only hope he explored the topic with something closer to the depth it deserves in his book.

Also see: Analysis - Kobe & Pierce: A Tale of 2 Pick-and-Rolls