Comments on: MySQL and the Linux swap problem

By: Linux Memory Usage at btm.geek

Linux Memory Usage at btm.geek — Thu, 08 May 2008 21:54:24 +0000

[...] smugmug discussion about swappiness is interesting, as that was originally my problem because running out [...]

By: Kevin Burton

Kevin Burton — Mon, 05 May 2008 22:20:52 +0000

Thanks Bernd……

I started a thread on LKML about this so hopefully this patch will fix this problem.

Now I need to find some time to apply the patch to our kernel.

Kevin

By: Don MacAskill

Don MacAskill — Mon, 05 May 2008 21:39:51 +0000

@Bernd Eckenfels

Thanks for the info! (Reads just joining, here’s a link to Rik’s post in that thread: http://www.ussg.iu.edu/hypermail/linux/kernel/0805.0/2261.html )

By: Bernd Eckenfels

Bernd Eckenfels — Mon, 05 May 2008 21:28:12 +0000

Rik van Riel recently commented on that on lkml: 20080505164446.2ec9a543@cuia.bos.redhat.com

One of the problems is that the process pages (anonymous memory) and
page cache pages live on the same LRU, so the kernel cannot always
easily find the page cache pages when it is trying to evict something.

Once that is fixed, and replacement is biased towards evicting page
cache pages, the system may do the right thing by itself.

I realize that this is no quick fix for your issue, but I am working
on a split LRU patch series to make sure Linux does the right thing in
the future.

You can find the latest patch at http://people.redhat.com/riel/splitvm/
and info on the design at http://linux-mm.org/PageReplacementDesign

By: Kevin Burton

Kevin Burton — Mon, 05 May 2008 03:40:13 +0000

Just FYI… I’m getting the same behavior on 2.6.24 on Debian Etch.

I loaded MySQL and a robot process which took up 3.8G out of 4G… a good bit of it left over.

I then decided to go for a hike in the Marin Headlands for a few hours.

I come back and Linux has decided to swap out 2G of MySQL.

Brilliant. Kill me now please.

At least my hike was good.

Hm…. I wonder if I could LD_PRELOAD the kernel to disable using the page cache……..

Kevin

By: A Linux Fan now gone

A Linux Fan now gone — Mon, 05 May 2008 00:15:09 +0000

Why not forget all of Linux’s kernel insanity and just run on OpenSolaris.

By: just a bar

just a bar — Sun, 04 May 2008 11:08:20 +0000

That’s a known problem with swappiness=0. Set it to 10 and kswapd won’t hog the CPU anymore.

By: Finite

Finite — Sun, 04 May 2008 06:33:06 +0000

This is some kind of joke, right?

As if swapping to a ramdisk wasn’t retarded enough, you’ve decided to throw a filesystem in the middle? Hilarious stuff!

So, what happens when the kernel decides to swap out some of that memory used by the ramdisk?

By: Olaf van der Spek

Olaf van der Spek — Sat, 03 May 2008 18:36:07 +0000

> You can’t just not have any swap partitions, though, or kswapd will literally dominate one of your CPU cores doing who-knows-what.

I assume this bug has been reported, but I don’t see a link. Got one?

> Zeek, some people do run without swap but there shouldn’t be a need to disable it or run without swap.

There shouldn’t be a need to enable swap in the first place.

> The problem is the OOM killer.

> If you’re on a 16GB box… and you have 16GB of memory in InnoDB… an errant ls or top shouldn’t be able to trigger the OOM killer and thus kill -9 your mysqld.

I think what is needed is something like memory quotas. You could allocate/reserve 15 gb for MySQL. The OoM killer can then make a better decision about what to kill when more memory is needed.

By: Kevin Burton

Kevin Burton — Sat, 03 May 2008 06:54:10 +0000

Zeek, some people do run without swap but there shouldn’t be a need to disable it or run without swap.

The problem is the OOM killer.

If you’re on a 16GB box… and you have 16GB of memory in InnoDB… an errant ls or top shouldn’t be able to trigger the OOM killer and thus kill -9 your mysqld.

Things like overcommit make it worse…