photo by: Local Sports Online

My poor little sister just got totally destroyed by awful Delta employees. She was trying to fly cross-country for the first time with her 4 month old baby. Disaster ensued, and Delta totally blew the customer service. Money quote?

“I asked [the supervisor] if she cared that I had been treated poorly by Delta employees, and she said if I wanted someone to care, I should write to “corporate” and they were required by law to respond to me in 30 days or less.”

My sister doesn’t fly as often as I do, clearly, or she would have known how terrible flying has gotten these days. She would have done online check-in, and would have arrived earlier. She made some mistakes, and seemed willing to accept the consequences. But that’s no excuse for downright rude customer service. Would an “I’m sorry” or a smile have been so hard? Shame on Delta.

Anne, for the record, flying with Virgin America has been an absolute dream, particularly when paired with Clear. (Want a free month with Clear? Use code SCA04501). I wish both were everywhere - maybe someday. In the meantime, avoid Delta like the plague.

UPDATE: I should have known Consumerist would have lots of stories like this. Enjoy!

Ever since Peter over at Percona wrote about MySQL and swap, I’ve been meaning to write this post. But after I saw Dathan Pattishall’s post on the subject, I knew I’d better actually do it.

There’s a nasty problem with Linux 2.6 even when you have a ton of RAM. No matter what you do, including setting /proc/sys/vm/swappiness = 0, your OS is going to prefer swapping stuff out rather than freeing up system cache. On a single-use machine, where the application is better at utilizing RAM than the system is, this is incredibly stupid. Our MySQL boxes are a perfect example - they run only MySQL and we want InnoDB to have a lot of RAM (32-64GB … and we’re testing 128GB).

You can’t just not have any swap partitions, though, or kswapd will literally dominate one of your CPU cores doing who-knows-what. But you can’t have it swapping to disk, or your performance goes into the toilet. So what to do?

Our solution is to make swap partitions out of RAM disks. Yes, I realize how insane that sounds, but the Linux kernel’s insanity drove us to it. Best part? It works. Here’s how:

mkdir /mnt/ram0
mkfs.ext3 -m 0 /dev/ram0
mount /dev/ram0 /mnt/ram0
dd bs=1024 count=14634 if=/dev/zero of=/mnt/ram0/swapfile
mkswap /mnt/ram0/swapfile
swapon /mnt/ram0/swapfile

That’ll give you a 14MB swap partition that’s actually in RAM, so it’s super-fast. This assumes your kernel is creating 16MB ramdisk partitions, but you can adjust your kernel paramenters and/or the ‘dd’ line above to suit whatever size you want.

We’ve found that anywhere from 20MB-40MB tends to be enough (so use /dev/ram1, /dev/ram2, etc), depending on load of the box. kswapd no longer uses any noticeable CPU, there’s always a few MB of free “swap”, and life is back in the fast lane. Just add those lines to your relevant startup file, like /etc/rc.d/rc.local, and it’ll persist after reboots.

Some Linux purists will probably hate this approach, others may have more efficient ways of achieving the same thing, but this works for us. Give it a shot.

Oh, and I hope it goes without saying, but make *darn* sure you know what you’re running on your box and what the maximum RAM footprint will be before you try running with only 20-40MB of swap. We’ve never OOMed (Out-Of-Memory) a production MySQL box - but that’s because we’re careful.

UPDATE: See what happens when I wait to blog? I forget that I read another related post over on Kevin Burton’s blog. Like Kevin, we’re using O_DIRECT, but unlike Kevin, this doesn’t solve the problem for us. Linux still swaps. We use the latest 2.6.18-53.1.14.el5 kernel from CentOS 5, btw. (Sorry, had posted 2.6.9 because I was dumb. We’re fully patched)

photo by: Jean-Yves

Have you reached stylesheet Nirvana? Are you the designer people go to for a fresh, clean, gorgeous look?

Have I got the dream job for you!

Come build beautiful stuff that millions of people all over the world can enjoy. We’re private, profitable, growing fast - and you could be a vital part of it.

BTW, as always, we’re permanently on the lookout for good software engineers, particularly if you have front-end web browser skills - JavaScript, DOM, AJAX, etc. Drop me a line.

The state of video codecs online has been a mess and there’s been no clear choice, making it very difficult to do awesome video sharing. Luckily, all of that changed when Adobe finally added H.264 support to Flash.

Thanks to Adobe, we finally have a video codec that we can get behind and that’ll be great for our customers. And so back in December, we released a major new update to our video offering that’s 100% based on H.264. And it supports resolutions all the way up to 1280×720p. That’s right - SmugMug has truly awesome hi-def video sharing.

Today, I’m thrilled to announce that our Flash player is out (we used QuickTime for a few months while we polished up our player), so it’s easier than ever to embed on your blogs and share with your friends:

Here’s all the gory details:

• Upload almost any video format you like. We’ll do our best to convert to H.264 in an extremely high quality way. (Thanks EC2!)
• We’ll generate multiple sizes for you, so you’ll have a version that’s perfect for sharing on the web (YouTube size), perfect for using on your iPod/iPhone (DVD size), and even your Hi-Def TV in your living room.
• We’ll automagically display just the right sized video for whichever browser and monitor you happen to be using. Ditto for your friends. Example from my friends in Dallas hard at work on Duke Nukem.
• You can embed the videos in your blog, website, or wherever else you like online. And you can do so at DVD quality resolution - 640×480 - more than 4X the pixels and quality of YouTube.
• You (and your friends and family, if you let them) can easily download all the different sized versions of your videos so you can do whatever else you’d like with them, like add them to YouTube or burn to a DVD.
• H.264 means it’ll play on a huge, wide variety of computers and devices, not just SmugMug. iPods, AppleTV, Playstation 3, and the list goes on…
• Speaking of Apple devices, we provide a complete podcast RSS feed for your account that you and your friends can subscribe to with a single click in iTunes. All your iPods, iPhones, and AppleTVs will then magically stay up-to-date. All your online videos in your pocket, and your living room, all the time. Neat, eh?
• I’m thrilled we’re making good use of the OpenShareIcon project, too. Rather than use some trademark-encumbered, company-owned, non-open ShareIcon, we’ve chosen to use the real deal. Viva open web standards!
• One gotcha: Flash takes 200% more CPU to play video on the Mac than QuickTime does, so in-gallery, Mac users will still see QuickTime. We can’t wait until that’s not true - but that’s up to Adobe, not us.

So there you have it. I’ll probably post again soon with lots more detail about how great the integration is with Apple devices: iPod, iPhone, iTunes, and AppleTV. We love us some Apple over here at SmugMug.

Oh, and you can count on our video player to continue to rapidly evolve. This is definitely just a 1.0 product - it may have some warts and it’ll get even better over time.

So go wild - share your crystal clear video with the world!

Oh, and demand your video to be awesome:

I’m on a panel called ‘Small Business Hacks’ at the Web2Open part of the Web 2.0 Expo today at 1:30pm.  Swing by if you’re interested in building a successful business on the cheap.  More details here.

One great thing about Web2Open is that it’s free - anyone can attend!

See ya there!

I realize I’m already way behind blogging about other new Amazon Web Services features like the recent EC2 release with static IPs, availability zones, and user kernels not to mention the new block storage service.  I’ll still try to get to them - but I didn’t want to wait for this one.

I’ve been pushing Amazon hard to do something like this, and I’m thrilled it’s finally out.  They have a great new service status dashboard complete with historical data and a mechanism for communicating to us, their customers, about any issues they may be having.  Especially cool is that the data is provided via RSS, so you can programmatically poll the status and take steps as necessary.  Awesome!  Get all the details here.

One possible gotcha is that it looks like the dashboard is hosted at Amazon.  We’ve run into outages (very rare) where all of amazon.com is down.  In those cases, it’d be nice to have an externally-hosted site where they could post updates.  Our customers asked us for this recently, so on January 29th, we were happy to comply.  Perhaps Amazon could post to their TypePad blog in events like these, rare as they may be?

Next, they now offer paid premium support.  Need some sort of help that’s not provided on the AWS forums or via searching Google?  No worries - whip out your credit card and pay for it.  Looks like they have two plans which should cover lots of use cases I’ve seen in my own comments and on the forums.

I’d still like to see a pay-per-incident model, personally, even with an extremely high price-tag for each incident.  We rarely use support for AWS, but at the same time, we’re very big customers of theirs, so the monthly price is quite high.  But if we really come up against a big problem, it’d be nice to know I could pay for support just that one time.  I imagine most of their customers will like their Silver and Gold monthly  packages, but for us, they’re just not quite the right fit.  Do they work for you? 

I’m pretty thrilled about this release, but maybe our use case is different from yours.  Do you like these new features?  Are they missing things you’d like to see?

I know I’m a little late to the discussion, but Brian Aker posted a thought-provoking piece on the imminent death of MySQL replication to scale reads.  His premise is that memcached is so cool and scales so much better, that read replication scaling is going to become a think of the past.  Other MySQL community people, like Arjen and Farhan, chimed in too.

Now, I love memcached.  We use it as a vital layer in our datacenters, and we couldn’t live without it.  But it’s not a total solution to all reads, so at least for our use case, it’s not going to kill our replica slaves that we use to scale reads.  

Why?  Because we still need to do index lookups to get the keys that we can extract from memcached.  And we have to do lots of those indexed queries.  Most of the row data lives inside of memcached, so this turns out to be a great solution, but we still need read slaves to provide the lists of keys.  Bottom line is that we still use read replication heavily - but we use it for different things that we did in years past.

And then, of course, there’s the issue of memcached failure.  For us, it’s very rare, and thanks to the way memcached works, it rarely hampers system performance, but when a node fails and needs to be re-filled, we have to go back to disk to get it.  And doing that efficiently means read slaves again.

For us, memcached plus MySQL replication is true magic.  Brian’s a very smart guy, and I realize he wrote the post to get people thinking and talking about the issue, but at least for us, read slaves are here to stay.

There’s quite a bit of buzz on the blogosphere from people I respect a great deal, like Jeremy Cole at Proven Scaling and Vadim at Percona, about MySQL’s new Enterprise backup plans.  

The big deal?  They’re releasing a Community version that doesn’t have all the same features as the Enterprise version of Online Backup, including compression and encryption.  The Community version is open-sourced under GPL, the Enterprise version is not.

Personally, I think this is awesome. Don’t get me wrong - I love open source.  We couldn’t have built our business without it, and we love it when we get a chance to contribute back to the community.

But let’s not forget that MySQL is a business.  And that business helps the community and improves the software.  They have customers (I’m one - we’re a paying MySQL Enterprise Platinum customer), and they have to solve those customers’ problems.  This is a virtuous cycle where the community benefits directly as MySQL thrives financially.  

Every time a business like us pays MySQL for a service or feature, MySQL can then invest in better software that benefits all.  The end result in MySQL’s case is more GPL’d code.   In a very real way, without companies like mine, there wouldn’t be a new backup tool at all - let alone the differences this debate is focused on.

Every day, I hear someone saying “Man, I love SmugMug so much!  It has [insert features here] which I love!  Why isn’t it free?”

The answer?  ”It wouldn’t be SmugMug if it was free.”  MySQL’s situation is very similar.

I wish more open source projects would make it easier for this cycle to ignite.  Some of them, like Red Hat, refuse to even take our money.  Talk about stupid.  There are *lots* of businesses out there willing to pay for extra services and features, and the community can harness that revenue in amazing ways, including getting more (or better) GPL’d code.

Couple more thoughts:

• I wouldn’t be surprised if future releases add new Enterprise-only features and some existing Enterprise-only features migrate down to Community.
• The Community version is open-sourced, so I’m sure the community will develop their own compression and encryption features.
• This is really no different from Enterprise Monitor, which has been only for Enterprise customers for awhile.
• Lots of other projects do this (and I would argue this benefits those projects and their communities, too)
• I’m 99% sure that this was the plan before Sun acquired MySQL.

First:  Very cool.

Next:  I think it’s interesting that Google has basically taken a sniper scope out and aimed it at a specific cloud computing target.  App Engine is only for web applications.  No batch computing, no cron jobs, no CPU/disk/network access, etc.  

I think this is very smart of Google.  Rather than attacking Amazon head-on, Google has realized there’s a huge playing field for cloud computing, and are attempting to dominate another portion of it, one where they have a lot of expertise.  Very good business move, imho.

Will we use it?  I wouldn’t be surprised.  I’ve long thought that we’ll continue to mix in web services from a variety of providers, and it looks like App Engine can solve a slice of our datacenter need that other providers don’t yet provide.  

I’m more than a little concerned, though, by how much vendor lock-in there is with App Engine.  At first glance, it doesn’t look like the apps will be portable at all.  If I want to switch providers, or add in other providers so I’m not relying solely on Google, I’m outta luck.  

I’m hopeful other languages get supported, too.  I think Python is great - don’t get me wrong - but we have a lot more experience with other languages, so there’ll be a learning curve.

Finally, I’m dying to find out what pricing for an application of our scale will look like.  I can see some immediate, obvious things I’d like to try to do on App Engine, but the beta limits aren’t gonna cut it for us.  :(

Will it replace Amazon?  It sure doesn’t look like it from where I sit.  In fact, I don’t see this as much of a competitor to Amazon Web Services.  There’s some overlap in some small area (hosted web apps on EC2), but I doubt that’s the bulk of Amazon’s business.  As I said, we’ll likely end up using both (and other providers as they come along, too).

My favorite bit?  In theory, Google has solved the data scaling problem.  I don’t mean raw binary (blob) storage, which S3, SmugFS, MogileFS, and plenty of other things have solved, but the “database” scaling problem.  Every popular web app runs into this problem, and it’s typically solved with a combination of memcached, federation, and replication.  But it’s messy.  In theory, Google has automated that piece for us.  I can’t wait to play with it and see if that’s true.

I also can’t wait to see who else is going to wade into this fray.  Sun?  Microsoft?  Yahoo?  IBM?  

Bring it on!

Sitting in the Virgin America part of the SFO International terminal.  Talk about an awesome terminal.  Can’t wait to fly Virgin, too - my first time (hah!).  I doubt anyone on the plane throws down in DOOM the way I do, so I expect a river of tears in the aisle.

Wish I could be at the Google Campfire tonight, cuz the news is awesome, but Microsoft got to me first, so up to the frigid North I go….  Hope this MS stuff is as good as it sounds.

It turns out the Seattle Photography Group is meeting on Wednesday night.  Cool!  So I’m going to that. You should come.  :)

Then, afterwards, I’m hoping we’ll head out for some food.  If you want to get food afterwards, please leave a comment so we have at least a rough head count.  Attendance at the SPG isn’t mandatory, but I’m sure you’re welcome to come.

On Tuesday, I already have dinner plans, but wouldn’t mind hanging out with some geeks afterwards.  Doubt I’m up for going into Seattle, though, so if you’re in Redmond or Bellevue, holler.

(I’ll try to email everyone who sent emails, commented, and twittered - but if you haven’t heard from me, re-comment because I’m lame)