Comments on: The Perfect DB Storage Array http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/ Thought stream from SmugMug's CEO & Chief Geek Sun, 20 Jul 2008 11:31:06 +0000 http://wordpress.org/?v=2.6-bleeding2 By: Glen Shok http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-95699 Glen Shok Wed, 26 Dec 2007 17:30:05 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-95699 Hey, although I am a marketing puke (with a highly technical background) I came by this blog post and found it interesting. We have an array that has been used by very large MySQL database shops and we have implemented features specifically for DB (1MB stripes) among others. I know this is an obvious plug, but you may want to take a look at it. http://www.pillardata.com If you want to yell at me for posting this: gshok@pillardata.com. :-) And to add to Tao's post - I worked with a defense contractor to implement something similar, using cheap 1U servers and a GFS like Lustre...this solution also works well and makes the storage and servers disposable when implemented correctly. Although I work for an array vendor, this is a very elegant solution that is resilient, but needs a lot of care and feeding when using open systems stuff like Lustre. There are some "private" GFS vendors out there that have better uptime. Good luck. And Happy New Year! Hey, although I am a marketing puke (with a highly technical background) I came by this blog post and found it interesting. We have an array that has been used by very large MySQL database shops and we have implemented features specifically for DB (1MB stripes) among others. I know this is an obvious plug, but you may want to take a look at it. http://www.pillardata.com

If you want to yell at me for posting this: gshok@pillardata.com. :-)

And to add to Tao’s post - I worked with a defense contractor to implement something similar, using cheap 1U servers and a GFS like Lustre…this solution also works well and makes the storage and servers disposable when implemented correctly. Although I work for an array vendor, this is a very elegant solution that is resilient, but needs a lot of care and feeding when using open systems stuff like Lustre. There are some “private” GFS vendors out there that have better uptime.

Good luck. And Happy New Year!

]]> By: Looking for the perfect DB storage array for your hot Web 2.0 company? « Hardware http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-72382 Looking for the perfect DB storage array for your hot Web 2.0 company? « Hardware Fri, 19 Oct 2007 17:16:58 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-72382 [...] read more | digg story [...] [...] read more | digg story [...]

]]> By: SmugBlog: Don MacAskill » Blog Archive » Dell MD3000 - Great DAS DB Storage http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-70270 SmugBlog: Don MacAskill » Blog Archive » Dell MD3000 - Great DAS DB Storage Tue, 02 Oct 2007 00:02:34 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-70270 [...] So I’ve written about storage before, specifically our quest for The Perfect DB Storage Array and how Sun’s storage didn’t stack up with their excellent servers. As you can probably tell, I spend a lot of my time thinking about and investigating storage - both small-and-fast for our DBs and huge-and-slower (like S3) for our photos. [...] [...] So I’ve written about storage before, specifically our quest for The Perfect DB Storage Array and how Sun’s storage didn’t stack up with their excellent servers. As you can probably tell, I spend a lot of my time thinking about and investigating storage - both small-and-fast for our DBs and huge-and-slower (like S3) for our photos. [...]

]]> By: Tao Shen http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-52472 Tao Shen Tue, 19 Jun 2007 02:17:50 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-52472 Don: one last thing: you should look into Iwill H4103 barebone...plug 4 dual core Opterons in there gives you 8 cores, 16 slots mem, and use the PCI-X expansion for your fibre channel connection to DAS. I think it's cheaper than buying Sun's boxes, and smaller 1U profile. Don:

one last thing: you should look into Iwill H4103 barebone…plug 4 dual core Opterons in there gives you 8 cores, 16 slots mem, and use the PCI-X expansion for your fibre channel connection to DAS. I think it’s cheaper than buying Sun’s boxes, and smaller 1U profile.

]]> By: Tao Shen http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-52385 Tao Shen Mon, 18 Jun 2007 09:30:35 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-52385 oh just to add to my post above, no RAID DAS boxes needed in that setup, just cheap drives. You are basically running an architecture similar to google's fs and bigtable over 1000s of cheap 1U nodes. In terms of cost and scalability, which way is better. Since google bought youtube, I was wondering if google will convert youtube's db architecture to use GoogleFS and Bigtable instead of master-N slave mysql clusters. oh just to add to my post above, no RAID DAS boxes needed in that setup, just cheap drives. You are basically running an architecture similar to google’s fs and bigtable over 1000s of cheap 1U nodes. In terms of cost and scalability, which way is better.

Since google bought youtube, I was wondering if google will convert youtube’s db architecture to use GoogleFS and Bigtable instead of master-N slave mysql clusters.

]]> By: Tao Shen http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-52384 Tao Shen Mon, 18 Jun 2007 09:25:51 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-52384 Hi,Don: Nice blog, and nice to see someone with experience posting their quest for nice database IO storage subsystems. I have been thinking about this same problem for a while now, what do you think about the following architecture? Using a OpenSSI cluster for your compute nodes and run mysqld on the SSI over say 24 nodes of cheap commodity 1Udual core CPUs(don't know if MySQL or PostgreSQL would scale horizontally that way) and then have another cluster of cheap 1U dual core system with 4 commodity hotswappable hard drives(SATA even) and run either lustre fs, or hadoop file system. You can choose to use Infiniband switch if you want, but dual GigE over say 24 nodes will give you 250MB/s*24=6GB/s theoretical IO bandwidth. In the case of TSUBAME(japan's supercomputer) they use lustrefs, and hadoop file system is simply a open source google file system. I don't know how much IO smugmug currently require, but linearly scaling IO bandwidth over GigE sounds a good solution oh, only if I had 1000 nodes to test this idea LOL Hi,Don:

Nice blog, and nice to see someone with experience posting their quest for nice database IO storage subsystems.

I have been thinking about this same problem for a while now, what do you think about the following architecture?

Using a OpenSSI cluster for your compute nodes and run mysqld on the SSI over say 24 nodes of cheap commodity 1Udual core CPUs(don’t know if MySQL or PostgreSQL would scale horizontally that way)

and then have another cluster of cheap 1U dual core system with 4 commodity hotswappable hard drives(SATA even) and run either lustre fs, or hadoop file system. You can choose to use Infiniband switch if you want, but dual GigE over say 24 nodes will give you 250MB/s*24=6GB/s theoretical IO bandwidth. In the case of TSUBAME(japan’s supercomputer) they use lustrefs, and hadoop file system is simply a open source google file system. I don’t know how much IO smugmug currently require, but linearly scaling IO bandwidth over GigE sounds a good solution

oh, only if I had 1000 nodes to test this idea LOL

]]> By: mfc http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-51798 mfc Wed, 13 Jun 2007 23:25:50 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-51798 Our write to read ratio is 2:1. We are also considering flash drives for redo logs. Sandisk (M-Systems) FFD35 U3S 8 P80 which is around 2k a pop! Our write to read ratio is 2:1.
We are also considering flash drives for redo logs.
Sandisk (M-Systems) FFD35 U3S 8 P80 which is
around 2k a pop!

]]> By: Timmy J http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-51772 Timmy J Wed, 13 Jun 2007 17:40:44 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-51772 I've read through this 3 times; great stuff. I'm curious: What is your approximate ratio of writes to reads? (I have a situation where we have tons of writes compared to much fewer reads. But it's critical the reads have the lowest possible latency. We're considering flash drives. Any thoughts?) I’ve read through this 3 times; great stuff. I’m curious: What is your approximate ratio of writes to reads?

(I have a situation where we have tons of writes compared to much fewer reads. But it’s critical the reads have the lowest possible latency. We’re considering flash drives. Any thoughts?)

]]> By: Jimmy B http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-50927 Jimmy B Wed, 06 Jun 2007 01:03:56 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-50927 Hi Don, good stuff. It seems the question of disk size comes up quite a bit. With disk sizes getting upwards of 300 GB now, what are your thoughts on the performance on these larger drives....Vendors will tell you that they are just as fast but I don't entirely trust the larger spindles... What size and speed of drives do you guys lean towards? Hi Don, good stuff. It seems the question of disk size comes up quite a bit. With disk sizes getting upwards of 300 GB now, what are your thoughts on the performance on these larger drives….Vendors will tell you that they are just as fast but I don’t entirely trust the larger spindles…

What size and speed of drives do you guys lean towards?

]]> By: Hostgator » Blog Archive » Looking for the perfect DB storage array for your hot Web 2.0 company? http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-50184 Hostgator » Blog Archive » Looking for the perfect DB storage array for your hot Web 2.0 company? Mon, 28 May 2007 09:43:05 +0000 http://blogs.smugmug.com/don/2007/04/27/the-perfect-db-storage-array/#comment-50184 [...] We do billions of queries per day in our DBs, so we have very specific requirements for what our storage does. It turns out YouTube has very similar requirements, so I wrote up a nice, detailed list of things to watch for if you’re a startup looking to scale your web application.read more | digg story [...] [...] We do billions of queries per day in our DBs, so we have very specific requirements for what our storage does. It turns out YouTube has very similar requirements, so I wrote up a nice, detailed list of things to watch for if you’re a startup looking to scale your web application.read more | digg story [...]

]]>