Flight Numbers: Not All Ratings Changes Are Created Equal

Aaron Howard introduces the Weighted Ratings Change metric

February 12, 2019 by Aaron Howard in Analysis with 0 comments

When the 2018 touring season began with the Las Vegas Challenge, Garret Gurthie was rated 995.  After a full season on the road and 30 tournaments, at the start of the Hall of Fame Classic in October he was rated 1024.  That, my friends, is a large rating jump. In fact, it is the largest ratings jump for any player that competed in at least five of the elite events in 2018.¹

In contrast to Gurthie, Eagle McMahon started out the touring season rated 1031 and went into the HOFC rated 1048, an increase of 17 points.  Taking absolutely nothing away from Gurthie’s fantastic season, I think we can all agree that McMahon’s 17 point increase is the more impressive feat, even though 17 is much less than 29,² if for no other reason than McMahon built his rating up from a much higher starting point.  In other words, gaining one rating point when you are rated 1031 is much harder to do than if you are rated 995.

With that comparison in mind, my goal for this article is to provide more context for ratings changes so that seasons like Gurthie’s and McMahon’s 2018s can be compared more appropriately.

To do this, I generated a weighted ratings change metric.³ In simple terms, the weighting is based on the frequency of a player’s rating within the population of professionals. So, McMahon gets more “credit” for his rating increase than Gurthie because a rating of 1031 occurs much less frequently among MPO players than a rating of 995.⁴⁵

Below are tables of the top 20 MPO and top 10 FPO players from 2018 in weighted ratings change. For me, they pass the eye test. Gurthie is still in the top 10 because a ratings change of 29 points is impressive, independent of the starting point. In first is, no surprise, Eagle McMahon. The fact that Eagle is ranked above Gurthie indicates that the metric is doing its job. His 17 point increase is the more impressive feat.

2018 Weighted Ratings Change - MPO

2018 Weighted Ratings Change - FPO

On the FPO side, no. 1 by a long shot is Paige Bjerkaas. She improved a lot this year, to say the least. Madison Walker in second place is a bit of a surprise to me. I guess she flew under the radar a bit this season, but obviously performed quite well despite some injury issues. Interestingly, the three biggest increases on the FPO side — Bjerkaas, Walker, and Rebecca Cox — were all penalized quite a bit for their relatively low ratings to start the season.

That pattern, in addition to the fact that Calvin Heimburg and Gurthie are in the MPO top 10, indicates that this metric could be used as a data point for determining Breakout Player of the Year candidates. Along this same line of reasoning, some pundits were making an argument for Gregg Barsby as a 2018 BPOY candidate even though he has been touring for many years, and lo and behold, he is fifth on the MPO top 10 list.

The diligent among you may notice that my analysis stops with ratings going into the 2018 HOFC but does not include ratings changes as a result of it. There are two reasons for this: 1) the logistics of scraping ratings for individual players from the PDGA website is much harder than scraping ratings for players going into tournaments, and 2) it isn’t always clear when tournament results are included in ratings updates or when any given individual’s rating will be updated, making the decision regarding the appropriate ratings update to use very difficult.

As a result, I believe it’s best to think about these numbers as a proof of concept, and not a definitive measure of 2018 rating changes. As a proof of concept, they point to a viable method for comparing players with disparate ratings, such as McMahon and Gurthie. And, secondarily, they indicate a potential method for finding breakout player candidates. Look for me to improve upon and expand this analysis in future articles.

1. Majors, NTs, DGPT, and EPT events. ↩

2. Yay math! ↩

3. Initially, I wanted to label this metric wRC.  But, if there are any sabermetric nerds out there reading this you will know I didn’t use that label because it is already in use as an acronym for weighted runs created. ↩

4. For those interested in the statistical details, I ranked all players with ratings above or equal to 990 (MPO) or 880 (FPO).  Then, I transformed those rankings into proportions, which I used as the weights. For example, Eagle McMahon’s rating of 1031 going into LVC was higher than 94.8% of players in the dataset, so his weight was 0.948 and his weighted rating change was 0.948*17 = 16.12.  In layman’s terms, he gets almost full credit, or a weight of 1, for his rating change because his 1031 rating was so high. ↩

5. A methodological note: I did not use the first and last PDGA ratings of the year for my metric.  Instead, I used ratings going into the first and last big tournaments in which players competed. This is because I wanted my results to focus on the tournaments that had the best players: Majors, NTs, DGPT, and EPT events.  Often, players start and end the season with smaller tournaments that do not really influence their ratings within the context of competitors of the highest quality. Of course, there are many smaller tournaments played throughout the touring season, as well, but there is not much methodologically we can do about that except encourage the PDGA to update ratings more often. ↩