Comments on: Death of MySQL read replication highly exaggerated http://blogs.smugmug.com/don/2008/04/16/death-of-mysql-read-replication-highly-exaggerated/ Thought stream from SmugMug's CEO & Chief Geek Sat, 11 Oct 2008 00:55:15 +0000 http://wordpress.org/?v=2.7-hemorrhage hourly 1 By: Thomas Hurst http://blogs.smugmug.com/don/2008/04/16/death-of-mysql-read-replication-highly-exaggerated/#comment-102708 Thomas Hurst Sat, 26 Apr 2008 06:45:32 +0000 http://blogs.smugmug.com/don/?p=213#comment-102708 At Newzbin, we operate the other way around; we have our own custom C daemon doing sorting, limiting, filtering etc, and giving us a list of ID's to go ask MySQL for. We could squeeze memcached in there too, but I don't think it's going to be much of a win; primary key lookups aren't exactly tough :) We use the same framework for stream processing, so we can do alerts for users when a search matches something new (or starts matching something old that's changed that they've not seen before). My main concern with replication, aside from it being awkward making sure you're talking to a current enough slave (we use a serialized counter to checkpoint critical db changes), is any long write queries on the master get serialized on slaves; have a 50s long "DELETE FROM BigTable WHERE created_at < really_old"? Well, now all your slaves are gonna sit doing that and nothing else for a minute or more, so you either give users stale data or hammer your master. I guess we need some sort of service which replicates queries like this async. Hurray, more infrastructure ;) At Newzbin, we operate the other way around; we have our own custom C daemon doing sorting, limiting, filtering etc, and giving us a list of ID’s to go ask MySQL for. We could squeeze memcached in there too, but I don’t think it’s going to be much of a win; primary key lookups aren’t exactly tough :)

We use the same framework for stream processing, so we can do alerts for users when a search matches something new (or starts matching something old that’s changed that they’ve not seen before).

My main concern with replication, aside from it being awkward making sure you’re talking to a current enough slave (we use a serialized counter to checkpoint critical db changes), is any long write queries on the master get serialized on slaves; have a 50s long “DELETE FROM BigTable WHERE created_at < really_old”? Well, now all your slaves are gonna sit doing that and nothing else for a minute or more, so you either give users stale data or hammer your master.

I guess we need some sort of service which replicates queries like this async. Hurray, more infrastructure ;)

]]> By: mike503 http://blogs.smugmug.com/don/2008/04/16/death-of-mysql-read-replication-highly-exaggerated/#comment-102600 mike503 Thu, 17 Apr 2008 00:22:24 +0000 http://blogs.smugmug.com/don/?p=213#comment-102600 Sounds like you're using a two-fetch method - use MySQL for sorting, limiting, whatever else to get a list of IDs, then do read-through caching for the bulk of the row data using memcached and fill in the missing from the DB. Is that correct? That's been my strategy so far too, nice to know if other people have adopted it successfully :) Sounds like you’re using a two-fetch method - use MySQL for sorting, limiting, whatever else to get a list of IDs, then do read-through caching for the bulk of the row data using memcached and fill in the missing from the DB. Is that correct? That’s been my strategy so far too, nice to know if other people have adopted it successfully :)

]]>