From PC World: Benchmarks, while inherently contentious and not always representative of real-world performance, are an important tool in any kind of quantitative evaluation. That’s why nerds are obsessed with them. And not just nerds: companies use third-party benchmark results to make decisions on millions, sometimes billions of dollars in investment. So when someone finds evidence of a company putting its figurative thumb on the scale, it has the potential for big ramifications. Such is the case with some recent, and very specific, Intel Xeon CPU benchmarks.
The Standard Performance Evaluation Corporation, better known as SPEC, has invalidated over 2600 of its own results testing Xeon processors in the 2022 and 2023 version of its popular industrial SPEC CPU 2017 test. After investigating, SPEC found that Intel had used compilers that were, quote, “performing a compilation that specifically improves the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using a priori knowledge of the SPEC code and dataset to perform a transformation that has narrow applicability.”
In layman’s terms, SPEC is accusing Intel of optimizing the compiler specifically for its benchmark, which means the results weren’t indicative of how end users could expect to see performance in the real world. Intel’s custom compiler might have been inflating the relevant results of the SPEC test by up to 9%. For more technical details (many of which are, frankly, beyond my level of compsci understanding) check out the reports from ServeTheHome and Phoronix, via Tom’s Hardware.
SPEC uncovered these results while looking back over its own benchmark database, and while it’s not deleting them for the sake of historical records, it is invalidating them for its own reports. Slightly newer versions of the compilers used in the latest industrial Xeon processors, the 5th-gen Emerald Rapids series, do not use these allegedly performance-enhancing APIs.
View: Full Article
