Linux versus Mac OS X and G5 versus Opteron Benchmarks

More update benchmarks are now available.

Summary

My 2.7 pound Pentium-M Linux laptop is faster than my 44 pound G5 running OS X. For statistical computing Apple's OS X is incredibly inefficient relative to Linux. The floating point performance of the 970 chip (the G5) leaves much to be desired, but OS X makes the performance problem significantly worse. OS X does not "just work" (at least for me). The latest release of the OS, Tiger, is very buggy and does not have the usability I would expect given Apple's reputation. Also, the issues of freedom and transparency should not be forgotten.

Background

People often ask me about my opinions of Apple's OS X both as a general operating system (as a replacement for Linux or other unixes) and as an operating system useful for statistical computing. People training in statistical methods expecially often ask for advice in trying to avoid Linux. Because I have to support my statistical software on various platforms, I own a 1.8Ghz dual G5 with 2GB of RAM. I have also run benchmarks on a dual 2Ghz G5. In order to save time repeating the same information to many people I have decided to post it on the web. The short answer: use Linux. If you want to use Mac OS X or Windows XP, go ahead. All of these operating systems are now above the line (not long ago the operating systems out of Redmond and Cupertino were a joke). However, if you decide to use Mac OS X for whatever reason, don't think it is just like Linux or some other efficient unix with a pretty and friendly gui. Life is full of tradeoffs and reasonable people can decide to make different choices. Don't pretend that tradeoffs don't exist, and don't fall victim to Apple's marketing which is an extention of the Steve Jobs Reality Distortion Field.

Benchmarks

I present here a set of benchmarks which are relevant to my work and to people working in statistical computing, particular people using the R Project for Statistical Computing. These benchmarks are floating point bound where the main IO is to memory and not to disk. Cache and Translation Look-aside Buffer (TLB) misses really matter as well as memory speed. This setup may be of more general interest. For example, my benchmarks generally match those provided by the well known SETI project (benchmarks). But they may not be relevant for what you do. If you need a computer to do Y, and these benchmarks are in no way related to Y, don't write me to complain about it. These benchmarks are useful for the work I and some other computational statistics people do.

OS X is incredibly slow by design not least because it uses the Mach Microkernel (see Linus vs. Tanenbaum). For example, in Linux, the variables for a system call are passed directly using the register file. In OS X, they are packed up in a memory buffer, passed to a variety of places, and the results are then passed back using another memory buffer before the results are written back to the register file. You can just imagine what that does for TLB and cache hits. This just adds to the context switching difficulties on the Power4 chips. Memory management in OS X is awful. To quote Kazushige Goto talking about his BLAS: "Performance is suppressed on purpose due to [the] awful memory management of OS X". Goto's work is described and praised on Apple's own website because he added a custom BLAS for the Apple super computer at Virginia Tech. On the Apple site it states that Goto was "pulling out incredible efficiencies". Well, given the Goto's own benchmarks and comments, this is just another example of the Steve Jobs Reality Distortion Field.

The benchmarks presented here are based on two of my statistical software packages for R: Matching (Multivariate and Propensity Score Matching Software) and rgenoud (R Version of GENetic Optimization Using Derivatives). The former uses C++ code extensively and the later mainly relies on C code. The two benchmark scripts are available here (matching) and here (rgenoud). All benchmarks were done using R-2.1.0 and gcc. The best timing results of three consecutive runs of the scripts are presented (if the first run numbers are used the results remain substantively the same).

The machines are:

Label	OS and Chip
OS X G5	Tiger, dual 1.8 Ghz G5, 2GB RAM
Linux G5	Ubuntu Linux 5.04 ppc livecd, dual 1.8 Ghz G5, 2GB RAM
Linux Intel-M	SuSE Professional 9.1, IBM Thinkpad X41 1.2 Pentium-M laptop, 1GB RAM
Linux Intel	SuSE Professional 9.1, IBM Intelistation 3 Ghz Pentium 4, 2GB RAM
Linux Opteron	SuSE Enterprise 9, Polywell Opteron Workstation, dual 1.8 Ghz, 4GB RAM
XP Intel	Windows XP Professional (SP2), IBM Intelistation 3 Ghz Pentium 4, 2GB RAM

My 2.7 pound Pentium-M laptop is faster than my 44 pound G5 running OS X!! Here were are not testing IO nor are we testing running multiple processes on the same chip. If we do either of these, the Opteron starts to outperform the Intel machine by a noticeable margin. The SETI people have a webpage which compares processors and it matches my own benchmarks well. They report theoretical peak cycles/flop for the Pentium M at 3.1, Opteron at 3.5, G5 (970) at 5.2 and the Pentium 4 at 6.4. Obviously, the Pentium 4 is able to maintain good performance only because it is clocked at a high speed. For actual performance numbers see their calculator.

It is worth noting that the distribution of Linux am running on the Mac, Ubuntu, is not optimized for the G5. One imagines that performance would improve if one were to use either Gentoo PPC64 or Yellow Dog Linux high performance version. Unlike, Ubuntu both of these distribution are 64-bit.

IBM designed the 970 chip (i.e,. the G5) as a brain damaged version of their server Power4 chips. They were worried that the 970 would eat into their server chip market so they limited it (512 cache, FPU etc). Just compare the IBM discussion of the 970 ("64-bit PowerPC microprocessor designed for desktops and entry-level servers") with Apple's marketing hyperbole "Apple and IBM team up to produce the world's most advanced processor".

I have conducted many more benchmarks on these and other machines (including a dual 2Ghz G5 and dual 252 Opteron workstations). I have also tested the HFS+ filesystem. It is vastly slower than reiser for small and medium sized files and vastly slower than XFS for large files (I use reiser partitions for system files and XFS partitions for storing large media files). If you want these additional benchmarks, let me know.

There are claims on the web that when Apple developers compile OS X on the 970, they use -Os. That is, they optimize for size and not for performance. "So even though Apple talked a lot of smack about having a first-class 64-bit RISC workstation chip under the hood of their towers, in the end they were more concerned about OS X's bulging memory requirements than they were about The Snappy(TM)." AnandTech has an article which offers another explanation for why OS X is so inefficient. See No more mysteries: Apple's G5 versus x86, Mac OS X versus Linux.

General Review

What about OS X for more general computing applications? After a few days of the Mac being my main desktop, I could no longer take it. I switched back to my Linux box. My Linux box has two monitors. There is a third monitor which I switch between XP and OS X. I run everything with just a single keyboard and mouse either through my KVM or even better through rdesktop (RDP) to Windows and Tight VNC to OS X (VNC controls the mouse and keyboard but I look at the actual monitor for the Mac screen output).

To make a long story short, the Mac has not been useful to me. I only use it to run iTunes. I cannot even use it to do video editing!?! I convert video from my PVR to mpeg-ts (mpegpes) on Windows XP. I do this using DirectShow Filters via graphedit which is an awesome interface to various video tools---one of Microsoft's biggest weapons has long been wonderful development tools. In this way, I can turn PVR files into portable mpeg-ts files. Which I can watch or burn to DVD for personal use. The problem? I cannot edit mepg-ts files using any Apple tool I can find. I could use third party software or open source stuff like mencoder and mplayer, but I might as well then use Linux (where the software runs much faster because of the OS and because I can run the software on an Opteron). On Windows, I can *easily* burn my shows to dvd using Nero which is fast and user friendly. I cannot use iDVD because it relies on Quicktime and QT does not support mpeg-ts. So, the Mac does not "just work" even for video editing. mpeg-ts is hardly an obscure format. And no, I do not want to add an encoding step to switch mpeg-ts to some Quicktime format just to change it again for burning to a dvd using iDVD. But if an encoding step is okay for you, you can use MPEG Steamclip for OS X, but it is slower than mplayer under Linux (particularly mplayer on x86 Linux).

Tiger has a LOT of bugs. It's ridiculous. This is what happens when the beta testing setup is mostly secret---much like a Bush rally, you need the secret decoder ring to get in. Apple does this because it wants the secrecy to generate buzz in the consumer market. It doesn't really care about enterprise users and it shows. The biggest bug I've found? Samba client support is broken. If Microsoft did this, Slashdot would go nuts. But Jobs's reality distortion field protects Apple. An example of the bug, on Linux (which is serving the smb file share), create the following two files:


musil:tmp/tmp> echo "a" > README
musil:tmp/tmp> echo "bb" > readme
musil:tmp/tmp> ls -l
total 8
-rw-r--r--  1 jsekhon research 3 2005-05-09 04:05 readme
-rw-r--r--  1 jsekhon research 2 2005-05-09 04:05 README

Mac OS X then hangs when it tries to view the directory. If this is done through finder, you are in big trouble. This is just stupid. smb has very basic rules for how clients are supposed to handle this. Windows has no problems and neither do Linux smb clients.

A related bug means that Macromedia MX does not work correctly on Tiger with a case sensitive filesystem. Damn. Macromedia MX on XP works fine on a case sensitive filesystem (enabled by the excellent UNIX Services for Windows package). See this Macromedia webpage.

Other bugs? Malicious web pages can install dashboard widgets. Egad! Didn't we hate pre-SP2 XP for allowing this?

I have to reboot to install the Quicktime mpeg plugin. What's up with that? I can't believe that I had to reboot the machine just to install the mpeg2 codec player for QuickTime. That is just so lame it's funny. Especially given that they are using the Mach Microkernel which is incredibly inefficient in part because it is a micro-kernel. They take the efficiency hit of a micro-kernel but they still don't have a modular system to the point that even a codec needs a reboot?!? At *worst* only the gui should need a reboot. And I guess they don't want to separate the gui because they don't want to scare most of their user base.

To be clear, Apple doesn't really care about techie users. People I know who work there are very honest about this. Apple thinks it is cool that a few hard core and famous hackers (such as Bill Joy) are using their machines. But that is not who they care about.

The color quality of the Mac is fantastic. Whites really are white and blues blue. But the bad engineering of the OS as well as the highly controlled nature of the apps offends me. Who would have thought that a decade after the famous 1984 Big Brother ad, it would be Apple with the highly controlled largely cathedral OS and IBM would be spending hundreds of millions of dollars on code it allows anyone to share and to contribute to?

Links

See similar benchmarks available on AnandTech's website: No more mysteries: Apple's G5 versus x86, Mac OS X versus Linux.

For another review see When a Linux user buys Apple's Mac mini.

Return to Jasjeet Sekhon's Homepage