Shoddy Testing Compromises ISU Scoring Calculations
Today (28 Dec. 2011) the ISU released a statement that the calculation program used this season up through the Grad Prix Final contained a calculation error that caused Virtue & Moir to place second in the Free Dance instead of first. Davis & Whites margin of victory should have been 0.5 points lower, but this adjustment (luckily for the ISU) would not change the final overall result.
This error is also present in the software used at qualifying competitions in the U.S. this season and affects the scoring in those competitions. This error can be up to +/- 0.5 points for any couple with a combination lift.
The ISU statement said:
ISU Judging System - Calculation Program
28 December 2011
In all fairness to the sport we have to inform you that after the ISU Grand Prix of Figure Skating Final of Quebec City a calculation error in the ISU Judging System calculation program has been detected. The calculation program used up to and including the ISU Grand Prix of Figure Skating Final had erroneously calculated the Dance result with the previous Grade of Execution (GOE) for the Combination Lift, which was upgraded with ISU Communication 1677 in July 2011.
The difference of 0.5 points emanating from this calculation error was enough, to result in a switch in the Free Dance ranking of the ISU Grand Prix of Figure Skating Final 2011. As a matter of fact, Tessa Virtue / Scott Moir (CAN) should have been ranked in first place and Meryl Davis / Charlie White (USA) should have been ranked second in this segment when taking into account the corresponding correction. It is important to point out however that the final ranking (Short Dance and Free Dance combined) has not changed.
The calculation program has been updated immediately after detection of the error and random checks throughout other international results between July 1st, 2011 and today have shown no discrepancies in the final rankings.
We deeply regret and apologize for the inconveniences caused by this error in the calculation program.
That something like this is still possible does not surprise us. The ISU calculation program is an extraordinarily complex, convoluted piece of software. One of our early criticisms of the development of IJS was that the system as a whole, and its individual components, were not developed, tested and validated using good system engineering principles. Apparently that remains to be the case.For this particular error, one fundamental ISU practice that make such an error likely, is that the ISU publishes changes to the scale of values piecemeal, embedded in communications covering a variety of topics. Nowhere can you find a single ISU communication each season that contains the currently approved scale of values. Further, it would appear communication between the keepers of the scale of value and the software developers is not as good as it needs to be.
Fortunately this error did not change the final order of finish, but that is just dumb luck this time.
For those curious about the exact nature of the error, the file storing the scale of values includes entries for long lifts which are out of date. The software is pulling the correct base values from the file but pulling out of date GoE values which were lower in 2010/11 than they are for 2011/12.
It does not speak well of the ISU maintenance of this import part of the scoring system that one can still find errors in the calculation software five months into the start of the season, and eight months after the annual process of updating the software begins each year.
This error also highlights the inherent instability in the IJS method of scoring, where a small error in the SoV can significantly change the results, where intentional small adjustments to the SoV can significantly hange the results, as can any one error in the calling of levels, or assigning of GoEs. The problem still remains in IJS that any "victory" of less than three points may not actually mean the best performance has won.
Copyright 2011 by George S. Rossano