The recent pull out of AMD from BapCo and SYSmark certainly has a lot of drama associated with it.
Was Intel trying to pull a fast one, is AMD just being a poor loser because they can’t compete, or is there something secret and nefarious going on?
From where I’m standing, BapCo and SYSmark became obsolete as soon as AMD decided to take a different path. Indeed, when AMD diverged from the CPU-centric path that has historically defined the foundation, SYSmark essentially crumbled. Without the foundation, this benchmark no long had any real meaning.
Think about it: is SYSmark a failure because it couldn’t even run on an iPad? Of course not.
Well, how about if you had applied a Mainframe benchmark to UNIX boxes? Clearly, they wouldn’t pass. Similarly, if you published the results of a Windows Benchmark on a Mac you’d definitely have to hide from Apple fans.
Remember, SYSmark only worked because Intel and AMD agreed on what a PC should be. As soon as they stopped agreeing it was done, finished and uncermoniously rendered obsolete. Let’s explore the nature of benchmarks, and specifically, collaborative benchmarks like SYSmark.
Benchmarks and PCs
Benchmarks are created to compare similar products in an objective fashion; they are never supposed to be the only measure.
For example, if we all bought cars based on just gas mileage or performance alone and used the same combination of both we’d all be driving identical cars. In fact, companies that buy fleet motorcycles and cars rarely pick the same ones, although BMW has been rather impressive with police motorcycles of late.
This is because some want more room, a different look, convertible top, different interior options or favor a particular brand because they trust it. Subjective measures are balanced against objective benchmarks like gas mileage and 0 to 60 times based on how the buyer expects to drive the car.
But now let’s take something really different like the Tesla, where horsepower and torque really aren’t comparable to a gas car and neither is price. The damn thing doesn’t even burn gas. People buy a Tesla because they love the 0-30 acceleration, the fact it doesn’t use gas and can afford its admittedly exorbitant price. Frankly, I personally think they are nuts, but such folks generally seem happy with their choice.
My point is that benchmarks, particularly collaborative benchmarks, only work if people agree on the platform. Benchmarking an electric car against a gas car with benchmarks designed for one or the other isn’t a valid measure of value.
Intel and AMD
SYSmark was created at a time when both AMD and Intel agreed the CPU was the heart of the PC. And the GPU? Well, it didn’t really matter that much except for gamers. Besides, Nvidia and ATI were primarily concerned with that market.
The end result was that Intel and AMD actually agreed on parameters that may not have reflected either’s view of the real world – but did provide a balanced view of mutual technology. But AMD, which has always been substantially smaller and less well funded than Intel, recognized they could never get out of Santa Clara’s shadow and bought ATI to change the game.
AMD’s new Fusion is vastly different than anything SYSmark was originally created to measure. Even though Intel has a similar platform, their weighing of performance in their GPU/CPU blend favors the CPU while AMD’s favors the GPU.
This effectively means the two industry heavyweights are no longer able to longer agree on what defines performance, any more that a gas car company and an electric – or more accurately a hybrid car company – could.
Now if both are building hybrids, they could theoretically agree, but hybrids are much more expensive and tend to underperform gas cars so they tend to increasingly be compared to each other.
In tech we see the same thing between tablets – which are arguably crippled PCs with touch capability and long battery life – and PCs. A common objective benchmark just doesn’t reflect the strengths of either well, you first need to understand what you want to do and then pick a benchmark that best measures the unique aspects of that product.
Wrapping Up: All the Matters is What You Want to Do.
Clearly, a joint benchmark only works if the collaborators agree on a common standard. If they disagree for any reason, the benchmark is rendered obsolete. It doesn’t really matter why.
Although in this particular case, for AMD it simply made no sense to fund or co-brand a benchmark that didn’t reflect their product – in their opinion – in a fair light. As I noted above, if cooperation breaks down, the product is dead. Here, the collaboration ended, AMD changed, no one is really at fault, the market moved and SYSmark couldn’t move with it.
Personally, in a laptop all I really want is something light with a 10-hour battery life. In retrospect, I’ve had more performance than I can use and less battery life than I’ve needed for years. Now, for a desktop, I want enough graphics to play my favorite games and transcode what I need to transcode.
Really, you should define your needs based on what you want to do and SYSmark hasn’t helped me in years. Choosing any product should start with what you want, not what anyone else thinks you should have. SYSmark no longer helps with that, and if you doubt me, try it on an iPad.
