Sun Fire ‘CoolThreads’ T1000 review
Ever since they first announced the Niagara processors at Sun, I've been excited. Could Niagara change my business? Who wouldn't want tons of physical cores coupled with tons of virtual cores? At every tech conference I've tried to get hard data from the people manning Sun's booths. At MySQL User's Conference they were hyping MySQL performance, for example - yet there's a huge MySQL bug where performance degrades with more CPUs, so that's clearly not a great target for us (yet).
Nonetheless, the geek in me remained intrigued - I've believed for years that scaling # of CPUs, rather than purely speed of CPUs, was the future. One of the great parts of my job is that I get to play around with new toys and new technology, like Amazon's S3 and Niagara, that can enhance our business or change it in some way. And every geek wants to dream that there's some hot new CPU around the corner that'll solve all their problems, right?
Sun has a great 60-day Try & Buy program. They make it basically as painless as clicking on the server you want, and a few days later it arrives. Very cool. Unfortunately, I haven't used Sun gear since 1994, when I was using SunOS 4 (remember when SunOS was BSD-based?), so it would likely be time-consuming to try out both new hardware and new software. No thanks, I'm a busy guy.
Enter Jonathan Schwartz and his famous blog. Jonathan probably doesn't remember me, but when I was 12 years old, I'd haunt the halls at NeXT every second I got and crashed NeXTWorld every year. I remember him. He was NeXT's most important developer, and my father got the thankless task of buffering Steve Jobs and Jonathan. Both of them needed the other, but they couldn't stand each other. Fun fun. 
I've been meaning to touch base with Jonathan and see how he's doing at his new job - and to see if a small web company like ours can shed any light on Sun's direction. I think he's got a very tough endeavor ahead of him - he's gotta turn a massive company with lots of inertia around to compete in a whole new ballgame. For more than a decade now, datacenter computing has been shifting more and more rapidly towards free operating systems coupled with commodity hardware, and Sun nearly missed the boat. Now they're scrambling to catch up. I believe Jonthan "gets it", but we'll have to see if he has the time and energy to really make the shift.
On June 16th, Jonathan posted a blog entry where he announced that Ubuntu Linux ran on Niagara, and that anyone who writes a thorough review would get to keep the box in question. Fantastic idea - I get to run Linux, which I know like the back of my hand, play with some hot new technology, and I get to keep the hardware for my time. Sold! So here we are, 60 days later, with a thorough review.
UPDATE: Jonathan has a new blog entry this morning about Niagara's power savings. Pretty cool that you can get a rebate for using lower-power servers - but it doesn't materially impact the conclusion of this review.
UPDATE #2: The comments here and on digg are pretty clear - you'd like to see Solaris results. Me too. Here's an open call for help from Sun.
THE GOAL
SmugMug has lots of different compute tasks, ranging from image manipulation (heavy CPU & math) to DB operations (threading and RAM heavy) to serving dynamic page content (lots of fast, light CPU requests). The obvious low-hanging fruit for something like Niagara, which has lots of slower cores, would be serving lightweight dynamic content. Specifically the thing we do most of - serving photos. We can load Apache processes up so each core gets a nice load.
As you can imagine, our "get photo" interface is heavily optimized for speed - it basically just checks to make sure you have permission to see the photo, grabs it from our distributed storage infrastructure (or Amazon's S3, depending), and shoves it down the network pipe. We serve millions and millions of them a day. I've often wondered if lots of smaller boxes with slower CPUs might be more efficient at serving them than the relatively small number of really fast CPUs we have doing the task now. Niagara provided a perfect opportunity to find out.
Our goal is a fairly simple one with any new piece of hardware we test. And now is a great time to test this stuff - we need to buy a bunch more servers. We always want to get the lowest cost per processing unit per watt. $/CPU/watt. That's it, it's that simple. Can Niagara deliver?
THE SERVERS
In the far corner, wearing red shorts, is the defending champion - SmugMug's baseline cluster member, Olde Faithful: a Rackable C1000. Weighing in at 1U half-depth, with two dual-core Opteron 270HE 2.0GHz processors, 4GB of RAM, no disks, and Gigabit Ethernet. Her contract is a svelte $3,595. Olde Faithful, and her precursors, have been getting the job done at SmugMug for years.
In the near corner, wearing the blue shorts, is the upstart contender from Sun, CoolThreads: a Sun Fire T1000, with a single 8-core 1.0GHz Niagara processor containing 4 virtual cores per physical core, 4GB of RAM, an 80GB SATA HDD, and quad Gigabit Ethernet. Her contract is a whopping $8,395*. CoolThreads looks sleek in her stylish 1U, 2/3rds depth metal case, but can she deliver the goods?
* Note, we actually are testing a T1000 with 16GB of RAM since that's what Sun sent us, but for this particular use case, we're only going to use a max of 1.5GB, so we'd buy the 4GB model. The price shown, $8,395, is for the 4GB version with an 80GB disk, which we don't need. Could probably save $100 if we could talk Sun into dropping the disk.
THE HYPOTHESIS
I'm not a big believer in the raw clock speed of CPUs, especially after watching our 2GHz Opterons clobber our 3GHz non-Woodcrest Xeons, but in this case I'm gonna give Sun the benefit of the doubt and guess that their clock speed is probably "high quality," just like AMD's.
CoolThreads' 8 cores x 1.0GHz = 8GHz. Olde Faithful's 4 cores x 2.0GHz = 8GHz. My hypothesis is that both are going to deliver roughly equivalent performance when it comes to the $/CPU portion of the equation. I predict a tie under load (which is what we care about most), but that when not loaded, Olde Faithful will serve photos faster.
Based on Sun's literature, and the fact that Sun also sells Opteron boxes (for less money) but continues to sell and promote the Niagara boxes, too, I think there's likely something there with the CPU/watt portion of the equation. I think CoolThreads will win the CPU/watt comparison, but it won't save nearly enough dollars to make up the 2.3X base cost.
Remember, the equation we care about is $/CPU/watt. Power efficiency alone isn't enough. If CoolThreads does win at the $/CPU/watt game, Sun can expect lots of future orders from us.
