Benchmarking was once the holy grail for measuring PC speed and performance. But is the practice still relevant?
Well, Silicon Valley tech guru Charlie Demerjian posits that for 99+% of PC users, more CPU speed is probably irrelevant at this point.
“Everything but the bottom basement crap and hamstrung Celerons are more than sufficient, and the price difference between that and the same chip with a few less fuses blown is almost irrelevant,” he explained in an analysis posted on SemiAccurate.
“[Yes], if you run benchmarks on both chips, it will give you numbers, but not what you need.”
According to Demerjian, a number of industry heavyweights are eyeing “experiential benchmarks,” or a standard that attempts to measure the (arguably amorphous) user experience. However, says Demerjian, this may not be the most accurate method of measurement, as companies will inevitably fall into the trap of defining hyper-dynamic terms like “average use” and “typical user.”
“Average and typical are not words that lend themselves to real world behaviors of complex biological systems like people very well,” he opined.
“I guarantee that a typical 15 year old girl does not typically do the same things on a PC as a 45 year old man.”
Configurable benchmarks is yet another alternative to traditional benchmarking that is currently being floated in the PC industry. Essentially, companies would run numbers based on set use cases in such a scenario.
“Once again, they might spit out numbers based on what you do, but they can’t even begin to quantify good enough. Even if it is easy to tell within a few minutes of booting a system, good enough is almost impossible to measure with a program.
“Benchmarks can measure what you tell them to, customize themselves to your workload, your neighbors, or any average, typical, or mediocre individual, but they can’t tell you what you need to know. [Similarly], reviews will blather on about how much more X Y has, but never mention if it is good enough. That is what you need to know, and it can’t be measured in the now useless traditional ways… [So] what do we do now?”
