Clearly, you haven’t read through the thread or the follow-up posts. Here’s a brief summary:

I set up a 2nd T1000 in our datacenter with Solaris on it, so we had one with Solaris and one with Linux. Then I went over to Sun and sat in a room for an entire day with the high-performance computing team and we fiddled with both servers.

The Linux one outperformed the Solaris one even after all the Sun guys had worked on it. Further, our AMD boxes outperformed both.

I believe we ran into limitations on network interrupts, rather than anything CPU related, but regardless - this fairly common workload fell down on T1000.

I believe Niagara2 would perform much better since it has on-board 10GigE NICs, but we haven’t done any testing yet.

On the one hand, your comparison is more than a joke. How can you be so ignorant and don’t even invest an evening to get the basics of Solaris?
I’m pretty sure that Solaris on that machine would be _much_ faster at your sweet spot, because the IP stack was lifted up. Plus you get administrative and debugging tools (e.g. DTrace) that you can’t have with Linux.
On the other hand, you’re right in that “your app is your benchmark”.
At least you should be aware that you benchmarked the L2 cache throughput. Is this workload really your daily business?

I’m long-time DGrin’ner (referred by Fish, early on) and SmugMug subscriber, who raves about SmugMug’s incredible responsiveness and customer service to anyone who will listen ;-). I also happen to receive paychecks from Sun.

The results you saw on the T1000 were not at all what I would have expected. The T1000/T2000 servers are the first systems Sun has shipped with *highly*-threaded (32-way) CPUs. I understand that one of the key issues with them is that customers don’t tend to see right out-of-the-box what those servers are capable of because the OS (Solaris, and possibly the Linux distros) don’t yet come pre-configured to take advantage of what the Niagara processor can do. The initial Solaris configuration settings are more appropriate to, say, a dual-threaded (dual-core, single-thread) UltraSPARC IV processor than the 32-thread (8-core, 4-thread) Niagara processor.

Sun is defintely working to remedy the out-of-the-box OS configuration issue. In the meantime, Sun provides software called “CoolTune” as part of the Cool Tools package that will ask you a few questions and set up all the *right* Solaris configuration parameters for you, for much better performance after that.

I agree, many of the tools (esp Linux ones) aren’t immediately available on the command line when you see that first Shell prompt. It’s not that they are missing; their directory just isn’t on the default $PATH. Arguably, they should be … but that aside, I think someone else already told you that many of them can be found in /usr/sfw/bin.

Secondly, did you get any help from Sun’s PAE group to uncover what the issues were? You might have found a significant improvement if they were involved. (I’ve heard that you did have some contact with that group re some other Sun gear you were evaluating)

I was so jazzed when I heard that SmugMug was considering (and as you announced in another blog, actually chose) Sun gear. Way cool. I suspect you’ll find some other Sun solutions that will be great fits at SmugMug, as time goes on ;-). I’m sure I’m not the first to offer, but if you’re not getting what you need, drop me a line and I’d be happy to help connect you up with the right folks at Sun.