The longer I have been involved in hockey analytics the more and more I am finding myself interested in looking at the draft, the market inefficiencies and just how poorly teams have been performing at scouting to try and make the best draft selections.
Draft Situation
The draft is a really hard area for teams to succeed in. In a given year only 210 teenagers are selected, averaging to 7 per team, while hundreds, and if not, thousands of others are passed to never be drafted or see another chance at making the next level. Within the team’s 7 picks scouts and management need to ensure they make as many selections that will lead to NHLers as they can. On average teams hope to select at least 1 future NHLer, but 2 or more is considered a success. This is much easier said than done because the odds of making a selection, than converts to become an NHL full-timer, is very difficult.
Being able to draft successfully in today’s NHL is very critical. The current CBA allows you to keep your drafted players as Restricted Free Agents which in turn keeps their contracts much cheaper than equal talent in free agency. With the Salary Cap in place teams need to stuff as many good players, and their contracts under the cap, as possible without overspending. With the peak age of NHLers coming in at the mid-20s it becomes clearer and clearer why teams should be looking for every market inefficiency they can to help them succeed.
Scouting Experimental Data
Last summer one of my analytics colleagues Rhys Jessop started looking at how NHL teams perform at drafting. Using the Vancouver Canucks as his model, and a very simple algorithm, he found that Vancouver’s professional scouting staff, staffed with human experts personnel traveling around the world and viewing players first hand, could not out-select players in either NHL games played or points scored.
Jessop and I partnered for a much larger experiment looking at over a decade of drafting from the position of all 30 teams. Using metrics to determine when scouting staffs have outperformed the very simple algorithm, based on money spent and NHL games played to earn a profit, we found that 19 of 30 teams did not outperform the computer.
More recent work of ours has correlated scouts rankings and simple metrics to NHL games played. Using Pearson’s r correlation we found that the NHL’s Central Scouting Staff rankings compared to those players’ NHL games played only had a correlation of -0.19 (or “No or negligible relationship”). When using very simple metrics such as a prospect’s draft-year points/game, the correlation with NHL games played is much higher, at 0.41 (or “Strong positive relationship”).
This is absolutely mind blowing. An average small-market NHL teams are spending over $2 million dollars a year on scouting. They have staff of all sizes, full of human experts, traveling around the world for the majority of the year and watching games daily. Yet these humans cannot do better than simple algorithms? The return on investment is terrible
Scouting Reality
Given that scouting, on the whole, seems to be an issue within the NHL it should be noted that it’s not always the case with every team. There were still 11 teams who outperformed the simple model of scouting and there are those who trying to exploit market inefficiencies. Chicago Blackhawks seem to be really invested in scouting the USHL, an area that is under-scouted. But this is not the case for the majority of the league.
There’s a number of reasons why teams are failing to do well with their draft picks. With the low possibility of draft picks succeeding it’s easy to see that even the best selection can fail to yield results. However, there is also evidence that scouts constantly make the same mistakes over and over.
Size of prospects is an example of this. It’s shown time and time ago that size of prospects is given a heavy weight in scouting opinion; larger players are over-drafted, but there’s little evidence bigger players succeed more often than smaller players.
This fetish for size happens year in and year out. A good example of this is the Vancouver Canucks who drafted Kyle Pettit with a late round pick in the 2014 NHL Entry Draft. Listed at 6’4″ Pettit scored 10 points in over 50 games with the Erie Otters, a team boasting high quality of teammates. Pettit was selected for his size and his “tools” rather than his ability to excel at playing hockey. The Canucks wasted a pick on someone, who could barely hold on to a 4th line Centre position in the OHL, while leaving the shorter Spenser Watson on the board. Spenser Watson lead all CHL draft eligible players last year in even-strength scoring and almost went undrafted until he was selected by the LA Kings with the 209th pick.
Traditional scouting opinions is keeping the size bias alive. Most recently ISS Hockey released their Top 30 draft-eligible prospect rankings and put Lawson Crouse 3rd, most simply for his size (currently listed at 6’4″), while playing for Team Canada at the World Juniors did not hurt. Despite being third in the ISS rankings Crouse is scoring at a paltry rate of 0.74 points/game, 13th in the OHL of draft eligible players. Lawson Crouse’s comparables in the CHL are: Andrew Taylor, Nick Czinder, Lucas Bloodoof, Martin Boisvenue, Joey St. Aubin, Trent Ouellete, Jean-Francois Cyr, and Cody Chikie. If you have never heard of those players it is because amongst them they have a grand total of zero NHL games.
On the contrary, farther down the ISS list is Mitchell Marner, listed as a mere 5’11” – Marner is currently giving Connor McDavid a run for his money in the OHL. The closest comparable to Marner is Marek Svatos; Svatos played over 300 games in the NHL scoring over 170 points.
Conclusion
Drafting is a very tough and tricky subject to deal with, but like every other area it is possibly to quantify it and analyze it to measure how teams are performing. By moving focus away from the biggest players, and starting to focus on players who excel at hockey, teams can find market inefficiencies in their scouting to try and select future NHLers.
Scouting now is often too focused on finding the next Milan Lucic, but the Boston Bruins were lucky when they selected him. They did not expect him to be the next Lucic. He was one of the first players like him and there won’t likely be another for a while. Selecting the players who excel now at hockey will give teams the higher likelihood of players who can succeed in a much tougher league.
The link that is supposed to link to the article evaluating all teams leads to the same Canucks article.
Leafs did this when they drafted Tyler Biggs back in 2011…..drafted purely for his size rather then ability to play hockey, leafs passed on guys like Tomas Jurco, Boone Jenner, John Gibson, Victor Rask, and Brandon Saad…..all guys who have played significant time in the NHL already, while Biggs can barely get ice for the Marlies in the AHL….what makes it more sad is that Leafs traded up to get Biggs….for no reason.
I was pleasantly surprised and happy when the Leafs selected the high end skill of Nylander this past year instead of the big body of Nick Ritchie….Nylander showed his quality at the World Juniors while Ritchie was just a 3rd line checker who wasn’t noticeable most nights for Canada.
Nylander’s criticism was his size and “attitude” apparently, well he is the same size as Patrick Kane and he ain’t a half bad player last time I checked.
To many old timers in scouting these days it seems, lots of teams need fresh blood in there scouting departments.
Good Article, enjoyed the read.
I will never understand why the Vancouver Canucks passed on Milan Lucic,who played ALL his hockey in the Canuck’s back yard. Lucic played his Junior for the Vancouver Giants and was tournament MVP in the Memorial Cup when the Giants won it.
But they passed him up for Michael Grabner,who was quickly traded away. Then the Canucks management had to watch as Lucic and his Bruins beat hell out of the Canucks and won the Stanley Cup. Vancouver fans rioted and burned up a car,they should have burned the cars of the Canucks scouting staff instead.
I don’t disagree with the article, but I’d be pretty pissed if I got the next “Marek Svatos” with a Top 10 pick.
Do undersized teams ever win? No. They were a fad for about a year and a half after the 05 lockout. The Sabres – to their detriment – thought the ‘New’ NHL would continue so kept drafting midgets with talent.
Why would NHL GMs follow the Sabres into the dumper?
I can hear all the Lightning fans screaming for attention. Yah, well, call me when they actually win something in the playoffs.
Size matters in the NHL. No amount of fapping over made-up metrics and Excel spreadsheets is going to change that.
This kind of thinking is long overdue— in fact, any kind of thinking regarding the draft is long overdue. Having 30 squads of grizzled ‘hockey men’ drinking bad coffee at 6AM in Belarus is incredibly inefficient. Has anyone ever just run a simulation on taking the next best available player from Central Scouting?
Congratulations on discovering the overdrafting of size.
hehe… “on the hole”
I’m not a stat guy, but I think it’s extraordinarily stupid to ignore advanced stats as they’re developed. Still, this article picks and chooses a little too much. For instance, CHL and even AHL stats don’t actually equate to NHL success and that’s a historical fact at least 30 years old. Still, I appreciate any analysis, so please do continue the series and build on this! Thanks for taking the time!
Is age a factor? Since we’re talking about kids around the age of 18, is part of this problem physiological? A hell of a lot of physical and mental development in male humans is backloaded into the late-teens and early-20s, so are scouts necessarily looking at incomplete pictures?
Is there any scouting on coaches in the minors? The Toronto Maple Leafs could sure use a great one. What are the metrics they use to scout coaches?
This article is kind of surprising. One wants to assume that every major sport took a look at the movie Money Ball or read the book. Hockey scouting the way you describe it sounds like they have not learned the lessons of Money Ball. If this is true there is a huge opportunity out there for any one of the 30 NHL clubs to break parity for pennies on the dollar.