The Enterprise Linux problem
I was talking with a friend last night about commercial OSes, specifically Linux and Solaris. It dawned on me that those of you in the midst of your own startups would find our experience useful. Sorry I didn’t think about blogging this earlier.
First of all, I love open source. I can’t imagine what it would be like to have a major OS problem and not have access to the source code and a community with access to it as well. I love the idea of building companies on open source and then using the success of that company to give back to the people and the community that made it possible.
There are lots of ways of contributing back, but the low hanging fruit for a tiny company like ours is simply buying service and support from one of the open source players, like MySQL or Red Hat. It’s a win-win, theoretically – they get funding to continue to build and test great software, and you get the service and support you need to grow your business.
We’ve been through a few commercial open-source OS distributions at SmugMug. All the gory details after the jump.
We started in 2002 with SuSE. At the time, I’d profiled the various filesystems, and XFS was the clear winner for our use case, and SuSE had it. At the time, we had a grand-total of 3 small, used servers that we got for pennies from a failed dot com. We’re bootstrapped, so every dollar we spent came out of our own pockets. The $800/seat price for SuSE with Enterprise support at the time seemed huge and steep (the hardware they were running on was much cheaper), but I choked down the cost and got a few copies.
Bad mistake. SuSE wouldn’t return my phone calls or emails when we ran into our first problem with the product. Literally wouldn’t even listen to see if our problem was legitimate or not. Since then, things over there have only gotten worse – Novell is now in bed with Microsoft, to the detriment of the entire community.
Next up at the plate was Red Hat. This time, I did some research before forking over my dough, and the community seemed in agreement that Red Hat actually responded to customers. Plus, their pricing was more stomachable: $350/seat instead of $800. The really good news? They actually did acknowledge me as a customer, reply to my emails, and best of all, have a fantastic web management interface / software update mechanism in the form of Red Hat Network.
We ended up using Red Hat Enterprise Linux (v3 at first, then later v4) for at least 4 years. During that time, we were mostly happy. The software updates were great, the software was stable and reliable, and we really had very few problems. Those that we did have, though, we were able to work out for ourselves or with a little help from the Linux community more quickly than Red Hat support was able to do. Even with the very most expensive Advanced Premium contracts, the support just wasn’t that useful to us. It does help that we have a lot of Linux experience in-house, so we’re able to take care of lots of things ourselves. Your mileage may vary if you don’t have our same level of expertise on tap 24/7.
When our service contract came up for renewal last year, we decided to evaluate whether it was worth paying or not, as we often do with any contract renewal. After looking at the price (tens of thousands of dollars, not chump change for a small business) and the relative speed and level of support, we balked. We had a dilemma: we loved Red Hat Linux, we loved how good they were at building & testing their software, we loved their mechanism for delivering software updates. We just didn’t need support.
We got on our knees, begging and pleading with Red Hat to let us pay for a “software updates only” license. They wouldn’t have it. “Support comes bundled with updates”, I was told, “no ifs, ands, or buts”. I *want* to pay Red Hat for the valuable service they do for us and the community. I just don’t want to pay for the part we don’t need – human support.
Around this time, Oracle announced Unbreakable Linux. Of course, the dirty little secret is that Unbreakable Linux is simply a repackage of Red Hat Enterprise. They’re identical. Better yet, they do offer a “software updates only” contract, and it’s reasonably priced. Now, this puts us in a strange position, because clearly Red Hat is doing most of the work, and Oracle is trying to profit. Brilliant of Oracle, you could argue, but it doesn’t make us feel that great – we want to give our money to the company actually doing the heavy lifting. Only they won’t take our money.
Nonetheless, we bought a couple of licenses from Oracle and gave it a whirl. It was indeed identical, and was a breeze to migrate our Red Hat boxes to it – we just changed one file. Unfortunately, Oracle released the product half-baked. The update service was broken in multiple ways (can’t have two people administer the same boxes, for example, and software updates inexplicably broke a few times). It quickly became clear that we couldn’t bank on something so unfinished, so we moved on.
I began asking friends of mine, at much larger companies than ours, what they were doing about this problem. Those companies have large, experienced Linux groups on hand to deal with isssues, and thousands of machines, so I couldn’t believe they were paying hundreds or thousands of dollars per Linux box for support they weren’t going to use. Turns out they weren’t – they were all using CentOS.
CentOS, it turns out, is just like Oracle Unbreakable Linux, only it’s free. The community has repackaged a few Red Hat Enterprise packages so Red Hat’s name and logo don’t appear in them, and that’s it. Other than that, it’s basically identical to Red Hat, and closely tracks their patches and updates. Migration from Red Hat Enterprise is just as easy as Oracle – you change a file or two and you’re golden. It’s a breeze. We have a handful of CentOS boxes in our datacenters now, and they’re working out beautifully – no learning curve, feels just like Red Hat, etc.
Now, again, this makes me feel a little slimy. I would really like to pay Red Hat for all their hard work building and testing the software. I’d happily pay $100/seat/year or something to Red Hat for exactly what I’m getting from CentOS for free. It’d be the right thing to do. But Red Hat won’t let me.
I’m certainly not going to waste my money, my shareholders’ money, and most importantly, my customer’s money on something frivolous like support we won’t use. I realize that normally I can’t just dictate prices – I can’t walk into a Honda dealer and tell them I’m buying their minivan for $20K instead of $35K and expect to win. It’s Red Hat’s product, they can sell it for whatever they like. I get that. But with open source, it’s also the community’s product. Which means I have options – including free.
Which brings us to Solaris. Solaris is now open-source, so it’s on my radar again. I love ZFS, I love the fault-tolerant stuff it has for when memory or CPUs go bad, etc. Sure sounds great. It’s a commercial OS, with free updates, and (I assume) good support options. But the last time I gave it a shot (last year), I was lost in userland. Solaris userland and Linux userland differ so greatly, there’s a steep learning curve for someone like me with 14 years of Linux under my belt. But I’m considering taking a closer look.
So at the moment, we’re sorta stuck in limbo. Our CentOS experiments are going well, but I’d rather find a middle ground where I can be part of the solution, rather than the problem. If and when we finally decide on our next course of action, I’ll be sure to let you know.
Love to hear any other experiences, if any, for people using open source software. (There’s no way we’d consider running something mission critical on a non-open source OS. I don’t know how people do it on Microsoft’s stuff).
UPDATE: I’m so dumb. Dag Wieers had a great point in the comments. I can just buy a handful of licenses (whatever I think Red Hat’s hard work on the software is “worth” to us) and not even use them. (And contribute to CentOS in the form of donations, too). Not as elegant as having a ’software updates only’ license, but still better than nothing. Thanks Dag!
