Pimp My Drive by Richard and Barb

There’s remarkably little information online about using MySQL on ZFS, successfully or not, so I did what any enterprising geek would do: Built a box, threw some data on it, and tossed it into production to see if it would sink or swim.

I’m a Linux geek, have been since 1993 (Slackware!). All of SmugMug’s datacenters (and our EC2 images) are built on Linux. But the current state of filesystems on Linux is awful, and it’s been awful for at least 8 years. As a result, we’ve put our first OpenSolaris box into production at SmugMug and I’ve been pleasantly surprised with the performance (the userland portions of the OS, though, leave a lot to be desired). Why OpenSolaris?

ZFS.

ZFS is the most amazing filesystem I’ve ever come across. Integrated volume management. Copy-on-write. Transactional. End-to-end data integrity. On-the-fly corruption detection and repair. Robust checksums. No RAID-5 write hole. Snapshots. Clones (writable snapshots). Dynamic striping. Open source software. It’s not available on Linux. Ugh. Ok, that sucks. (GPL is a double-edged sword, and this is a perfect example). Since it’s open-source, it’s available on other OSes, like FreeBSD and Mac OS X, but Linux is a no go. *sigh* I have a feeling Sun is working towards GPL’ing ZFS, but these things take time and I’m sick of waiting.

The OpenSolaris project is working towards making Solaris resemble the Linux (GNU) userland plus the Solaris kernel. They’re not there yet, but the goal is commendable and the package management system has taken a few good steps in the right direction. It’s still frustrating, but massively less so. Despite all the rough edges, though, ZFS is just so compelling I basically have no choice. I need end-to-end data integrity. The rest of the stuff is just icing on an already delicious cake.

The obvious first place to use ZFS was for our database boxes, so that’s what I did. I didn’t have the time, knowledge of OpenSolaris, or inclination to do any synthetic benchmarking or attempt to create an apples-to-apples comparison with our current software setup, so I took the quickest route I could to have a MySQL box up and running. I had two immediate performance metrics I cared about:

• Can a MySQL slave on OpenSolaris with ZFS keep up with the write load with no readers?
• If yes, can the slave shoulder its fair share of the reads, too?

Simple and to the point. Here’s the system:

• SunFire X2200 M2 w/64GB of RAM and 2 x dual-core 2.6GHz Opterons
• Dell MD3000 w/15 x 15K SCSI disks and mirrored 512MB battery-backed write caches (these are really starting to piss us off, but that’s another post…)

The quickest path to getting the system up and running resulted in lots of variables in the equation changing:

• Linux -> OpenSolaris (snv_95 currently)
• MySQL 5.0 -> MySQL 5.1
• LVM2 + ext3 -> ZFS
• Hardware RAID -> Software RAID
• No compression -> gzip9 volume compression

Whew! Lots of changes. Let me break them down one by one, skipping the obvious first one:

MySQL - MySQL 5.1 is nearing GA, and has a couple of very important bug fixes for us that we’ve been working around for an awfully long time now. When I downloaded the MySQL 5.0 Enterprise Solaris packages and they wouldn’t install properly, that made the decision to dabble with 5.1 even easier - the CoolStack 5.1 binaries from Sun installed just fine.

Going to MySQL 5.1 on a ~1TB DB is painful, though, I should warn you up front. It forced ‘REPAIR TABLE’ on lots of my tables, so this step took much longer than I expected. Also, we found that the query optimizer in some cases did a poor job of choosing which indexes to use for queries. A few “simple” SELECTs (no JOINs or anything) that would take a few milliseconds on our 5.0 boxes took seconds on our 5.1 boxes. A little bit of code solved the problem and resulted in better efficiency even for the 5.0 boxes, so it was a net win, but painful for a few hours while I tracked it down.

Finally, after running CoolStack for a few days, we switched (on advice from Sun) to the 5.1.28 Community Edition to fix some scalability issues. This made a huge difference so I highly recommend it. (On a side note, I wish MySQL provided Enterprise binaries for 5.1 for their paying customers to test with). The Google & Percona patches should make a monster difference, too.

Volume management and the filesystem - There’s some debate online as to whether ZFS is a “layering violation” or not. I could care less - it’s pure heaven to work with. This is how filesystems should have always been. The commands to create, manage, and extend pools are so simple and logical you basically don’t even need man pages (discovering disk names, on the other hand, isn’t easy. I finally used ‘format’ but even typing it gives me the shivers…).

zpool create MYPOOL c0t0d0

You just created a ZFS pool. Want a mirror?

zpool create MYPOOL mirror c0t0d0 c0t0d1

Want a striped mirror (RAID-1+0) w/spare?

zpool create MYPOOL mirror c0t0d0 c0t0d1 mirror c0t0d2 c0t0d3 spare c0t0d4

Want to add another mirror to an already striped mirror (RAID-1+0) pool?

zpool add MYPOOL mirror c0t0d5 c0t0d6

Get the idea? Super-easy. Massively easier than LVM2+ext3 where adding a mirror is at least 4 commands: pvcreate, vgextend, lvextend, resize2fs - usually with an fsck in there too.

Software RAID - This is something we’ve been itching for for quite some time. With modern system architectures and modern CPUs, there’s no real reason “storage” should be separate from “servers”. A storage device should be just a server with some open-source software and lots of disks. (The “open source” part is important. I’m sick of relying on closed-source RAID firmware). The amount of flexibility, performance, reliability and operational cost savings you can achieve with software RAID rather than hardware is enormous. With real datacenter-grade flash storage devices just around the corner, this becomes even more vital. ZFS makes all of this stuff Just Work, including properly adjusting the write caches on the disk, eliminating the RAID-5 write hole, etc. Our first box still has a battery-backed write-cache between the disks and the CPU for write performance, but all the disks are just exposed as JBOD and striped + mirrored using ZFS. It rocks.

Compression - Ok, so this is where the geek in me decided to get a little crazy. ZFS allows you to turn on (and off) a variety of compression mechanisms on-the-fly on your pool. This comes with some unknown (depends on lots of factors, including your workload, CPUs, etc) performance penalty (CPU is required to compress/decompress), but can have performance upsides too (smaller reads and writes = less busy disk).

InnoDB is notoriously bad at disk usage (we see 2X+ space usage using InnoDB) and while it’s not an enormous concern, it’d be something nice to curtail. On most of our DB boxes, we have idle CPU around (we’re not really I/O bound either - MySQL is a strange duck in that you can be concurrency bound without being either CPU or I/O bound fairly easily thanks to poor locking), so I figured I’d go wild and give it a shot.

Lo and behold, it worked! We’re getting a 2.12X compression ratio on our DB, and performance is keeping up just fine. I ran some quick performance tests on large linear reads/writes and we were measuring 45.6MB/s sustained uncompression and 39MB/s sustained compression on a single-threaded app on an Opteron CPU. We’ll probably continue to test compression stuff, and of course if we run into performance bottlenecks, we’ll turn it off immediately, but so far the mad science experiment is working.

Configuration

Configuring everything was relatively painless. I bounced a few questions off of Sun (imho, this is where Sun really shines - they listen to their customers and put technical people with real answers within arms reach) and read the Evil Tuning Guide to ZFS. In the end I really only ended up tweaking two things (plus setting compression to gzip-9):

• I set the recordsize to match InnoDB’s - 16KB.

  zfs set recordsize=16K MYPOOL

• I turned off file-level prefetching. See the Evil Tuning Guide. (I’m testing with this on, now, and so far it seems fine).

I believe since ZFS is fully checksummed and transactional (so partial writes never occur) I can disable InnoDB’s doublewrite buffer. I haven’t been brave enough to do this yet, but I plan to. I like performance.

Performance

This box has been in production in our most important DB cluster for two weeks now. On the metrics I care about (replication lag, query performance, CPU utliization, etc) it’s pulling its fair share of the read load and keeping completely up on replication. Just eyeballing the stats (we haven’t had time to number crunch comparison stats, though we gave some to Sun that I’m hoping they crunch), I can’t tell a difference between this slave and any of the others in the cluster running Linux. I sure feel a lot better about the data integrity, though.

Why not [insert other OS here]?

We could have gone with Nexenta, FreeBSD, Mac OS X, or even *gulp* tried ZFS on FUSE/Linux. To be honest, Nexenta is the most interesting because it actually *is* the Solaris kernel plus Linux userland, exactly what I wanted. I’ve played with it a tiny bit, and plan to play with it more, but this is a mission-critical chunk of data we’re dealing with, so I need a company like Sun in my corner. I find myself wishing Sun had taken the Nexenta route (or offered support for it that I could buy or something). Instead, we’ll be buying software service & support from Sun for this and any other mission-critical OpenSolaris boxes.

FreeBSD also doesn’t have the support I need, Mac OS X wasn’t performant enough the last time I fiddled with it as a server, and most FUSE filesystems are slow so I didn’t even bother.

Gotchas

• On my 64GB Linux boxes, I give InnoDB 54GB of buffer pool size. With otherwise exactly the same my.cnf settings, MySQL on OpenSolaris crashes with anything more than 40GB. 14GB, or 21.9% of my RAM, that I can’t seem to use effectively. Sun is looking into this, I’ll let you know if I find anything out.
• For a Linux geek, OpenSolaris userland is still painful. Bear in mind that this is a single-purpose box, so all I really want to do is install and configure MySQL, then monitor the software and hardware. If this were a developer box, I would have already given up. OpenSolaris is still very early, so I’m still hopeful, but be prepared to invest some time. Some of my biggest peeves:
  • Common commands, like ‘ps’, have very different flags.
  • Some GNU bins are provided in /usr/gnu/bin - but a better ‘ps’ is missing, as is ‘top’ (no, ‘prstat’ is *not* the same!), ’screen’, etc (Can anyone even use remote command-line Unix boxes without ’screen’? If so, how?)
  • Packages are crazily named, making finding your stuff to install tough. Like instead of Apache being called ‘apache’ or ‘httpd’, it’s called ‘SUNWapch’. What?
  • After finally figuring out how to search for packages to get the names (’pkg search -r Apache’ - which doesn’t provide pleasant results), I discovered that ‘top’ and ’screen’ just simply aren’t provided (or they’re named even worse than I thought). Instead, I had to go to a 3rd party repository, BlastWave, to get them. And then, of course, the ‘top’ OpenSolaris package wouldn’t actually install and I had to manually break into the package and extract the binary. Ugh.

Whew! Big post, but there was a lot of ground to cover. I’m sure there are questions, so please post in the comments and I’ll try to do a follow-up. As I fiddle, tweak, and change things I’ll try to post updates, too - but no promises.

UPDATE: One other gotcha I forgot to mention. When MySQL (or, presumably, anything else running on the box) gets really busy, user interactivity evaporates on OpenSolaris. Just hitting enter or any other key at a bash prompt over SSH can take many seconds to register. I remember when Linux had these sort of issues in the past, but had blissfully forgotten about them.

©Vincent Laforet - Blog.vincentlaforet.com

So you may have seen all the hooplah yesterday over Canon and Vincent Laforet’s amazing Canon 5D MkII footage. I thought maybe a little explanation was in order. First, a little background on me and Canon:

• I, personally, am a monster Canon fanboy. I have a lot of cameras, and all of them - my collection of happy-snappys, our dSLRs, and even our video cameras - are Canon.
• Our company is filled with Canon fanboys. We have more dSLR Canon bodies and lenses lying around than I can count.
• The 5D MkII is the coolest camera I’ve ever heard of. Dozens of SmugMuggers have already pre-ordered them.
• I’ve been dying to work with Canon since we started SmugMug. We’re a Top 500 website, we reach 6.5M people a month, our demographic is definitely high-end, and Nikon’s already in bed with Flickr. Sounds like a match made in heaven to me.

Ok, so now that I’ve set the stage, let’s talk about Vincent’s movie a little bit:

• SmugMug had nothing to do with the production of the film. We didn’t even know it existed until we read this post on Vincent’s blog on Saturday afternoon.
• The entire company caught fire. We lost our minds, we were so excited. Within minutes, we’d offered to provide *unlimited* HD bandwidth to Vincent. Bear in mind this was an unknown, but likely very large, cost with no real tangible upside. But we built this company because we love photography, video, and gadgets - and we’ve gotta stick with what we love.
• Vincent enthusiastically took us up on our offer, and we all started brainstorming about how we could best release the film. Then we started brainstorming on how great this camera would be for indie photographers and filmmakers, and we lost our minds again. By Sunday morning, we had committed $25-50K to create a community-driven film using the Canon 5D MkII. (Note how fast things are moving - they were moving so fast, none of us had time to catch our breath).
• We found out that Vincent had some awesome Behind-the-Scenes footage of the making of his film, Reverie, and so of course we offer to host that for free again.
• The time for release arrived. Now, this entire time, we’ve never talked to anyone at Canon. As far as I knew, this wasn’t a Canon deal - Vincent clearly says Canon told him “You can then produce a video and stills completely independently from Canon U.S.A.”
• We posted full HD versions of both Reverie and the behind-the-scenes footage for the world to see, crossing our fingers that our bandwidth bill wouldn’t be more than we could bear.
• Our customers went bananas. Awesome! They’re thrilled we’re interested in this stuff, because they’re interested in this stuff. Ok, great, so maybe this bandwidth bill will pay of in goodwill.
• The press went bananas - both mainstream and online. Awesome! They’re gaga over the user response and the remarkable camera.
• We got busy (and I personally got busy) telling everyone, press and non alike, who called, emailed, tweeted, blogged, etc that the Canon 5D MkII is a game-changing camera the likes of which we haven’t seen before.
• Canon asked Vincent to ask us to take Reverie down.

SAY WHAT?!

Canon asked Vincent to ask us to take Reverie down.

Being a Canon fanboy, I quickly complied - with a very heavy heart. I felt like I’d been kicked in the gut by one of my heroes. I felt betrayed. I also wrote a few things in the heat of the moment that came out harsher than they should have (and thankfully I didn’t publish what I’d original written - whew!). I’ve now edited my blog post and would like to apologize to anyone at Canon who I offended - I certainly wasn’t attacking Canon’s great employees, I was just lashing out.

But look at it from my point of view. I was risking an awful lot of money on bandwidth (I doubt it would have topped 6 figures, but easily could have been in the 5s) because I’m a camera geek and I love this stuff. Customer goodwill is fabulous, and we love generating it, but we were really doing this because we love the camera, love the passion that went into the film, and love to help our industry. We were hopeful that that goodwill would come back to us someday - but even if it didn’t, the chance to be a part of something as momentous as this film from this camera was worth it. And a good chunk of the company busted their butts over the weekend to make this happen. We could have been playing with our kids or out shooting photographs, but instead we spent the weekend setting things up for Vincent’s release.

And instead of appreciating how generous I thought we were being, and appreciating the monster amount of PR they were getting (better PR than any amount of money can buy), it felt like Canon was arbitrarily cutting us off for no good reason. I found myself asking “Well, if they want to host it on their pages, why don’t they just embed the video from SmugMug? Then they get it for free and we still get to be involved. It doesn’t even have to show our logo or anything - just use Quicktime but use a file from SmugMug’s servers. We’d save them money!”. We just wanted to be involved. And no-one at Canon called or emailed us at all - as I’m writing this, I’ve still never talked to anyone at Canon on this “independent from Canon” project.

In the cold light of the next day, though, I can see that I overreacted. It’s a sign of my passion for Canon and their products. No-one overreacts when some bad company does something stupid. But just look at Apple - the instant they make a mis-step (or even perceived mis-step), everyone is up in arms, ready to lynch Steve. Why? Because their products are so dang good, everyone’s super-passionate about them. So I let my passion get the better of me. I still wish Canon had wanted to work together, or at least let us be part of the project, but does it really matter?

I’m still buying a Canon 5D MkII and, I’m sure, lots of Canon goodies to go along with it. So what are you waiting for? Go get your own.

©Vincent Laforet - Blog.vincentlaforet.com

Pulitzer Prize-winning photographer Vincent Laforet got his hands on a Canon 5D MkII for a weekend. Rather than shoot some quick stills, he rounded up an entire film crew and put them to work using the amazing 1080p video capture it offers - in helicopters, no less! When SmugMug heard about this, we went bananas and offered to host both the short film itself, Reverie (want it in HD?):

UPDATE: There should be an embedded video of the short film right here, and a link to the HD version. But there isn’t (anymore). Go check it out on Canon’s own website instead.

Meanwhile, you can see the Behind the Scenes footage (want it in HD?):

Then we went a little more bananas, and ponied up $25K to sponsor a community-created film led by Vincent, with another $25K to follow if other sponsors get on the train. We think this camera is truly a game-changer and we’re thrilled to help visionaries like Vincent prove it to the world.

Now, the astute geeks in the audience will note that Reverie isn’t hosted in 1080p, but instead is at 720p. I wish it weren’t so, and we’re actively trying to get our hands on the 1080p footage right out of Final Cut so we can let everyone take a peek - but it’s not our footage, so I don’t actually have it. I believe Canon may be putting it online themselves, but if they don’t, I’ll do everything I can to put it up - so stay tuned to Vincent’s blog as well as my own.

Man I love this industry! Thanks Canon!

photo by: ikegami

I’ve been too busy to blog lately, and for that I apologize.  But here’s a quicky detailing the technologies (internet related and not) I’m excited about right now:

• Drizzle.  For years now, I’ve felt that MySQL has been doing in a direction in opposition to my use case.  Stored procedures, views, etc etc have added bloat and complexity without offering me anything useful.  Turns out I’m not alone - and thus Drizzle was born.  To say I’m *super* excited about this is a serious understatement.
• Google & Percona’s MySQL patches.  While I wait for Drizzle, I’m stuck dealing with terrible concurrency issues in MySQL/InnoDB that force us to partition data way before we really should have to, making our system more complex.  It’s crazy having a server keel over when it shouldn’t be either CPU-bound *or* IO-bound but that’s life with MySQL and InnoDB these days - or at least, it was until Google and Percona fixed what I couldn’t get MySQL to fix with our Platinum Enterprise subscriptions.  Open source rules!
• Flash storage.  I really wish I could talk about this some more (pesky NDAs), but there are datacenter changes coming that are more dramatic than anything I’ve seen in 14 years of working on them. I hope I’ve talked to everyone in the space (and from the companies I’ve talked to, one of them seems to be the *very* clear winner for this upcoming round), but if you’re a storage vendor working on flash appliances and I haven’t talked to you, ping me.  We’re a bleeding edge customer and we’ll put your stuff in production faster than you can deliver it to us.  :)
• ZFS.  Regardless of flash storage, ZFS is the filesystem of choice - head and shoulders over everything we’ve used or heard of.  The advent of flash just makes this even more compelling.  The downside?  It’s not on Linux.  :(
• OpenSolaris.  ZFS is so incredible, my hand has been forced, and we’re about to put our first OpenSolaris system into production.  OpenSolaris is, in theory, the Solaris kernel (think ZFS, DTrace, SMF, high concurrency, etc) with the GNU-like userland (think Linux-like).  In practice, it’s still extremely painful for a Linux expert and Solaris n00b like me to use - even on a single-purpose machine like a MySQL server.  Only ZFS makes the pain worth it.  For development, it’s basically unusable for Linuxers (it’s probaby fabulous for Solaris guys - lucky ducks).
• Nexenta.  Unlike OpenSolaris, Nexenta *is* the Solaris kernel plus GNU userland.  Unfortunately, it’s not backed by Sun or anyone else I have any relationship with.  Sun has been absolutely the very best technology vendor we’ve ever dealt with in terms of support, technical knowledge, and just plain listening to us, so that’s a big issue.  I wish Sun had taken Nexenta’s approach (or would just buy them or offer support or something).  If OpenSolaris continues to be painful, we may fall back on Nexenta instead - remember, ZFS is the driving factor here.
• Amazon Web Services competitors.  They’ve been promising they’d be coming out for years now and I’m shocked they’ve given Amazon this much runway.  But I believe a few more are getting very close (can’t say more, again, pesky NDAs).  Now, we’re extremely happy with Amazon, so we have no plans to switch, but competition is good for everyone - and Amazon is a fierce competitor.  Plus there are still gaps in Amazon’s strategy, and if I can mix & match to plug some of those gaps, awesome - sign me up.
• Memcached.  This one’s been on my list for years, and it’s still way up there.  Binary protocol on the verge of shipping, nice patch to resolve some networking issues we’ve seen, and talk about scabability.  If you’re building web apps and this isn’t a core part of your infrastructure, you’re doing it wrong.
• Big RAM.  4GB DIMMs are dirt cheap, so if you’re not loading your DB and Memcached boxes to the gills, you’re missing the boat.  Cheap 2-socket 64GB (and relatively cheap 128GB at 4-sockets) are here.
• Sun Fire X4140 and X4440.  The best 1U (2-socket) and 2U (4-socket) servers on earth.  Despite being late to the game with quad-core, Opteron RAM performance kills Xeon, so these are the servers we’re buying.  You can load them to the gills with 4GB DIMMs, enjoy the dual-power supplies (yes, in the 1U box too), and crank out some great stuff.
• OpenSocial, Y!OS, etc.  The big boys are finally getting real about getting open and cross-pollinating data and I think we’re finally nearing an inflection point.  We’re hiring a Sorcerer to do nothing but think and build in this space.  I’m sure magic will ensue.
• Nikon D90 and Canon 5D MkII.  Nikon’s taken the photography world by storm with amazing high-ISO performance, and Canon just announced a DSLR that shoots full 1080p video.  Both look amazing and both are game-changers.
• Onkyo TX-SR806.  I’m an A/V junkie and this thing is amazing.  5 HDMI inputs (need more?), THX Ultra2 Plus (the low-volume enhancements are *awesome* with young kids sleeping at home), automatic room EQ, decodes every modern audio encoding, etc.  I don’t even use the amplifier section (I have separates), but it’s turning out to be the best Pre/Pro I’ve ever owned.  Sounds fabulous on my gear.
• iPhone App Store.  That thing is a game changer, and we’re barely seeing the tip of the iceberg.  All the other players have to respond - which is great for you and I.  And talk about a platform that’s a dream to develop on!

          

photo by: Bill Evans Photography

How would you like to be the 8th Sorcerer here at SmugMug?  (We don’t hire engineers, programmers, or even coders - we only hire Sorcerers.  If you can’t work magic, I’m sure our competitors would love to see your resume…)

At SmugMug, everything we build is a direct result of customer feedback. We do very little, if any, competitive research - our customers keep us plenty busy. As a result, we’ve largely ignored social networking, especially outside of SmugMug. It just hasn’t been something our customers have asked for.

That’s changing. I’ve started getting tweets, blog comments, and forum posts about our “broken Facebook app”. Problem is, we don’t have a Facebook app.

The good news is we listen. So we’re ready to take the plunge. The geek in me has *always* wanted to dive into this stuff (and I’m the one who built and/or pushed us to build the building blocks we already have: an open API, Atom/RSS feeds, OpenID support, OAuth support, etc), so I’m thrilled we finally have the “ok” from my boss - our customers.

So if you’re high on social networking, particularly sharing photos anywhere and everywhere, we’d love to have you come work your magic. The job is extremely open-ended: You’d create our strategy, build our apps on other platforms, interact with our API developers who’ve already built some, and generally make it even easier for our customers to share their photos outside SmugMug. You’ll have to get your hands dirty - you’ll be writing the software (with the help of the other Sorcerers as needed), so managers and architects who no longer dirty their hands need not apply.

If that sounds like fun, we’re the best company to work for you’ve ever heard of (ok, this list sounds unbelievable, but I swear it’s all true):

• We’re all super heroes.
• We’re a privately held, profitable-for-years, fast-growing company (100%+ year-on-year for multiple years)
• Fun projects. You choose what to work on rather than being assigned some fluff job. (I know, I know, unheard of - but I swear it’s true).
• Small team. Your projects are your projects, not some multi-layer management effort doomed to fail.
• Fast paced. Any week where we don’t do at least one software release is rare.
• Large scale. Top 500 site. 350M+ photos, 800TB+ storage, 300M page views/month. Fun problems to solve
• Big impact. Hundreds of thousands of paying customers and 6.5M+ visitors a month will use your work.
• Family friendly. Full healthcare coverage, kids welcome for company meals and events. (Ex: We’re taking the whole company to Tahoe to ski & relax, including spouses and kids).
• Distributed. Nearly 75% of our employees aren’t in Silicon Valley - they’re scattered all over the world, from Australia to Europe and a dozen US states.
• Crazy benefits. We pay better salaries than the giants in Silicon Valley plus “early” stock options, profit sharing bonuses, matching 401k, 100% healthcare coverage for you and your family, gym memberships, iPhone 3G + minutes & data, 3G data cards, cable/DSL at home. Free drinks, free meals while working (new private chef too!). And more.
• Great office. Walking distance to downtown Mountain View, across the street from train & lightrail, near Highway 85. 7.1 channel home theater, dual 30″ displays + Mac Pro + MacBook Pro/Air, jaw-dropping photography on the walls (and an in-house studio to shoot your own). Healthy cube/office decoration budget. (Ok, this is getting really fun to write )

Whew!  (Yes, I think our employees are our most valuable asset.  Can you tell?)

So, do you have what it takes? At the very least, you’ll need:

• A passion for open data.
• An understanding of how important privacy controls are.
• Experience with web services, especially REST. SOAP and XML-RPC fans, this isn’t the place for you (but knowledge of black magic ain’t bad - just don’t practice it here!).
• Modern scripting language experience (PHP, Python, Ruby). We use PHP (and so will you!).
• History building apps (big or small) for platforms like Facebook, OpenSocial, etc.
• Understanding of current and upcoming social networking technologies: OpenID, OAuth, microformats, etc
• Experience with the SmugMug API a big plus.

If this sounds like your brand of magic, please contact us and let us know you’re our next Sorcerer.   If not, please tell your magic-working friends that the opportunity of a lifetime is right here…

Apparently it’s the best superhero movie ever made, and must be seen on IMAX. And you’re invited! We’re taking our company, family, customers and friends to Batman: The Dark Knight in IMAX! If you read my blog, that means you’re a friend - and eligible for a free ticket out of our block!

WHERE: Regal Hacienda Crossings Stadium 21 & IMAX, 5000 Dublin Blvd., Dublin, CA 94568
WHEN: 12:50pm on Friday, July 18th. Look for people in red SmugMug hats.

If you’d like a ticket, post in the comments, email me, Tweet me, something - I’ll reply to confirm we have enough tickets for you. If you want to bring your SO, friend(s), or family, let us know - we’ll try to accomodate as best we can.

Please arrive by 12:20pm so we can get you your ticket in time. We’ll give away any that aren’t there by the time all the SmugMuggers take their seats.

Man, to say I’m excited about this would be a major understatement. We’re huge Apple fanboys over here, so when we got accepted to the first wave of SDK developers at Apple, we were stoked. Shizam went to town almost immediately and after a few months of hard work, SmugShot was born. (And as I’m writing this, we’re #1 in “What’s Hot” on both iTunes and the iPhone interface!)

So what is it? Well, we knew early on we wanted something very simple and elegant that did only one thing - but did it extremely well. We didn’t want a kitchen-sink photo-sharing / -browsing / -taking application. We already have a fantastic iPhone application on Safari, so the obvious thing to tackle first was actually taking the photos on your iPhone and getting them up to SmugMug.

SmugShot makes it incredibly simple to simply whip your phone out at a moment’s notice and take as many snapshots as you’d like. The photos will be automagically geotagged with your location, should you wish it, and you can quickly and easily enter a caption and some keywords - or not. Your call. We’ll queue them up and send them along to the SmugMug gallery of your choice - over EDGE, WiFi, or 3G.

And that’s basically it. Simple, elegant, clean - just the way we like it. If you’re new to SmugMug, you can create a free trial account right from SmugShot. You can set up a default Caption and some default Keywords to make entering them a breeze. And you can even upload photos that are already in your Photo Library, rather than from your camera (and you iPod Touch users can do this, too). One big Apple bug with that, though - the SDK only give us access to 640×480 versions of photos in your Library. I’m hoping they’ll fix that soon.

The really wild thing is how much I actually use the app. I’m a bit of a snob when it comes to things like cameras and lenses, and lets face it - the iPhone’s lens can’t compare to some fabulous Canon glass. But as the app has spread throughout the office, everyone’s learned the same lesson I have: There’s an awful lot of value in convenience.

SmugShot is so shockingly convenient and easy to use, it trumps the limited image quality for almost all of my normal everyday shots.

So go grab it from iTunes, read more about it, or even get some answers. Definitely let us know if you like it and what we can improve on - we already have our own list but would love to hear yours!

SmugMug has always allowed everyone to upload an unlimited number of web-displayable files - JPEG, GIF, PNG, and MP4 - but to date we haven’t been able to accept the RAW files generated by modern digital cameras. For years our customers have been asking, begging, and pleading for us to let them upload their priceless archives.  I’m happy to announce that day has come!

SmugVault is a new SmugMug product that lets you upload all the RAW, PSD, BMP, and TIFF files you’d like.  And not just those - we’ll accept XMP sidecars, PDF files, Word documents, Excel spreadsheets, video archives, and anything else you might want to store with your photos.  What’s more, we’ll bundle your files together for easy, intuitive browsing and safe retrieval.

Thanks to an innovative new product from Amazon Web Services, DevPay, you only pay pennies per GB for the storage you actually use each month.  There’s no huge fee with a maximum storage amount - it’s truly unlimited and pay-by-the-drink.  Store one megabyte or one billion megabytes - we don’t care.  Whatever works best for your workflow and archival needs, SmugVault can handle it.

photo by: Andy Williams

Compose a beautiful panorama out of 20 RAW files?  No problem - upload your final JPEG and bundle all 20 RAW files with it, along with your Photoshop PSD containing all your layers and edits and the XMP sidecar detailing the Adobe Lightroom changes you made during the editing process.  You’ll see just the single perfect photo on your SmugMug site, but with a single click, you have access to every component you’ve associated with it.

Don’t want to upload final corrected JPEGs for all the RAWs you shot at that huge event, but still want them stored somewhere safe and sound?  No problem.  Just upload the RAWs straight off your camera and we’ll store them for safe retrieval.  Want us to generate JPEG previews of those uncorrected RAW files so you can browse your SmugVault visually to find that perfect shot?  We’ll do that too.

Loving SmugMug’s new HD video features, but wishing you had somewhere safe to archive the original footage rather than the web-friendly lower bitrate copies?  Not a problem.  Just add them to your SmugVault.

Unfortunately, we hear about people losing their priceless memories to hurricanes, fire, and computer failure almost every day.  We’ve always been glad we can simply help them get the JPEGs back - remember, your photos are yours, not ours - and I’m even more excited that we can now help everyone recover their priceless archives too!

Read more: Release Notes | Pricing | Help | Wiki FAQ

photo by: Andrew Tobin

I’m thrilled that O’Reilly is putting on a great performance and operations conference, so I’m especially happy to be speaking there. I’m on a great-sounding panel, Success: A Survival Guide. I’m sure you’ll hear about our first few years where, like clockwork on the same days each year, we got massively hammered with traffic and what we did to handle it.

If you’re going to the conference, come say “Hi!”. I’ll be wearing a red SmugMug hat, as always.

Oh, and if you haven’t signed up yet, use ‘vel08js’ to get a nice discount.

I’m really honored that SmugMug made LifeHacker’s Five Best Photo Sharing Web Sites.

They have voting open to pick the best - go vote for your favorite!