In grading which scouting service and/or ranking system made the best predictions or was most accurate in terms of how the NHL 2009 entry draft went down, there are many limitations. It will likely be somewhere between three to five years before we can begin to understand which scouting/ranking system was most accurate in their prognostications, and compare this to how NHL team’s professional scouting teams performed.
However, until time elapses and players develop, the only true comparator we have is to compare in some fashion the various rankings to how NHL teams actually drafted. Again, there are limitations to such analyses and I will try and outline some of these when discussing some of the comparisons I performed below.
First off, I did a straight up count of which of the main rankings out there got the most correct in the 1st round out of the sixty-one prospects drafted.
*The limitation of this analysis is that the various rankings systems out there are not necessarily predicting when a player will be picked, but ranking players based on their own scouting of the respective prospect.
Here are the results:
1. Bob McKenzie (TSN.ca): 54/61
As TSN states: “This is not a prediction of which team will take which prospect. In fact, when TSN puts its rankings together it doesn’t even take into account the order of selection at the draft. It is a ranking based on consensus opinion from 10 NHL scouts surveyed by McKenzie, who has been using the same process and formula for draft rankings since he originated The Hockey News Draft Preview in 1984, over 25 years ago.
And neither is it the personal or subjective opinion of McKenzie, who instead relies on the combined expertise of the scouts surveyed to calculate where prospects should be slotted on consensus from Nos. 1 through 60. Historically, it has been a good barometer of which players will be chosen in the first round.”
*My rankings and published here at THW.
3. (tie) McKeens: 49/61
3. (tie) THN: 49/61
5. Red Line Report: 48/61
6. Future Considerations: 47/61
7. International Scouting Services: 45/61
*Central Scouting Bureau/Services were not listed due to their segregated rankings. They separate North American skaters from European and also have skaters and goalies ranked separately.
Secondly, I went through each player taken in the first 2 rounds that was applicable to most rankings, and ranked each of the major scouting/ranking services (from 1 to 7) based on how close the predicted ranking was to where the player actually went in the 2009 NHL entry draft.
*This was by no means a sophisticated statistical analysis, but serves as a decent comparison between the rankings.
- Bias: of course, I did the analysis, so there may be my bias, although the numbers do not lie.
- The future: possibly in 3 to 5 years, based on how prospects perform, the various rankings could prove more accurate. But: the future is the future. It will be cool to look back after “X” number of years and see how the rankings match up then. Of course, scouting services perform their rankings as a projection of how the prospects will develop and the players they will become over the next 3 to five years.
- A couple players were left off if all of the rankings totally missed the boat, as it was next to impossible to analyze.
- Central Scouting Bureau/Services again were not listed due to their segregated rankings. They separate North American skaters from European and also have skaters and goalies ranked separately.
- ISS also separates out goalies, so in most cases I had to extrapolate, but I did give them the benefit of the doubt in these occasions.
The final totals calculated are listed in brackets below. In this analysis, the lower number indicates higher accuracy in terms of rankings closeness to where a respective prospect actually went in the draft.
1. The Hockey Spy (144) – Just edging out TSN and their 10 NHL scouts.
2. Bob McKenzie – TSN.ca (146) – An extremely close second – admittedly, with numbers this close, it likely could go just as easily in TSN’s favour in a more detailed analysis.
3. Future Considerations (156) – A very respectable and great score by FC!
4. Red Line Report (191) – Kyle Woodlief and crew finish a distant fourth.
5. THN (204) – The Hockey News are holding out hope their rankings look better five years from now.
6. McKeens (210) – well back in the pack!
7. ISS (224) – Someone had to bring up the rear!
I plan to do a more detailed analysis which I will post in the near future and may provide an even clearer picture.
Again, the significance of these analyses is unknown, but at present time it at the very least gives us some form of comparison. Given the limitations noted above and the fact I performed the analyses, I welcome and would appreciate anyone who wishes to perform a separate analysis. Feel free to comment below or email me at firstname.lastname@example.org .