Yesterday Apple unveiled their new iPad Pro running the A9X CPU and a lot of people instantly started comparing it to Microsoft’s Surface Pro 3 running an Intel i5, but can you really compare the two?
For years I worked as a Technical Editor for various publications and while I was working at Interactivity Magazine in the late 90’s I was tasked with writing about high-end workstations.
I wrote comparative reviews of literally dozens of workstations from dozens of different companies. I used industry standard benchmarks for many tests but I also wrote a series of real-world benchmark tests using multiple high-end 3D modeling and rendering programs such as 3D Studio Max, SoftImage 3D, Maya, and LightWave 3D that stressed different components of the machines – 3D graphics rendering, CPU performance, storage I/O, etc. – all geared to provide objective results from one machine to the next.
We were planning a massive comparative review of a dozen different workstations and my boss at the time, a hard-core Mac fanatic, kept asking me to include Apple products in my roundups. He also wanted to see how the SGI products would stack up. The SGI people were always willing to help and sent us an Indigo 2 right away but the Apple folks were a bit reluctant to participate in any head-to-head tests. We finally did manage to convince the PR people at Apple to send us one of their top-of-the-line Power Macintosh 9600s (or an 8600, I can’t remember which).
I dove back into the world of benchmarks in an attempt to find an objective benchmark that could be used to test the Mac (and SGI) versus all the other Intel-based machines. I also looked into modifying my own benchmark tests using LightWave (at the time the only 3D modeling package that ran on just about any platform).
But there were problems.
Even though there were industry standard benchmarks for just about all systems they had to be written differently for different CPUs. It also turned out that when I spoke to engineers at NewTek they told me like benchmarks different versions of LightWave used completely different code bases on different architectures. So even if the benchmarks and LightWave provided the same basic functionality on different platforms, they were, at their cores, different programs. That meant any variations in test results could be because of the hardware or they could be attributed to how the test software was written in the first place, what compilers were used, how optimized they were for one system or another and any number of other factors.
It also turned out that since the processors used in the different machines were based on completely different architectures it was impossible to say that an Intel processor running at a certain clock speed was any faster or slower than say a PowerPC processor running at a different speed.
Apple used to claim that a PowerPC running at 100 MHz was faster than an Intel Pentium processor running at 120 MHz but that’s something that simply can’t be tested conclusively. Plus, it wasn’t uncommon for companies, including Apple and Intel, to tweak their benchmark results, sometimes by turning processor features on or off, in order to shine when performing certain tests (basically putting the CPU into a state you would never encounter in the real world).
You simply can’t say one way or the other that this system is faster than that system unless they are running the same CPU.
So when Apple claims that their newest iPad Pro with the A9X is 1.8x faster than its predecessor, the A8X, on a CPU level and 2x better on the GPU side that may (or may not) be true but when they say the new A9X is faster than “80% of portable PCs” it doesn’t really mean anything. Apple engineers wrote those benchmark tests and ran them and reported them. Do you think they would have announced that their latest and greatest processor was only marginally faster than its predecessor or worse, that it might be slower than a competitor’s CPU? I don’t think so.
So comparing the speed and performance of an iPad Pro running an A9X to a Microsoft Surface Pro 3 running an Intel i5 is essentially a pointless exercise.
