Comments on: Silent data corruption on AMD servers

By: Chris

Chris — Fri, 07 Sep 2007 20:07:01 +0000

Hey Don,
Just a heads up looks like that first Kernel.org link is broken. I realize this is a month+ late though .

BTW, just started reading random parts of this blog a few weeks ago, and I have say thanks. Your incite and experience sharing is priceless and I wish more companies were as open as you are regarding hardware/scaling/experiences. Keep it up.

By: Somewhere out there! » Blog Archive » Silent data corruption on AMD servers with 4G+ RAM

Fri, 27 Jul 2007 23:05:24 +0000

[...] From smugmug. [...]

By: Don MacAskill

Don MacAskill — Fri, 27 Jul 2007 08:02:09 +0000

@Jeff:

Thanks for the insight – that’s what I was thinking, too.

By: Jeff Bonwick

Jeff Bonwick — Fri, 27 Jul 2007 07:29:07 +0000

Hey Don — I don’t know enough about the bug and the BIOS/Linux interaction to say for sure whether ZFS would save you here. If, as the bug report implies, it’s related to the iommu, that would suggest that the data is corrupted during DMA. In which case, ZFS would detect it on the next read; and if you were running with mirrors or RAID-Z, ZFS would correct it as well.

Wout is right that if you get silent in-memory corruption *before* the data is written to disk, we currently have no way to detect that. We’ve considered adding an option to keep in-memory buffers checksummed and verify them before any modification and before any disk write. It would be insanely expensive, of course, but could come in handy when trying to track down broken hardware.

By: Don MacAskill

Don MacAskill — Thu, 26 Jul 2007 16:28:32 +0000

@Matt:

No, we’re running Linux on our Sun boxes. CentOS5 to be exact.

But ZFS has us perpetually curious about Solaris.

By: Matt Culbreth

Matt Culbreth — Thu, 26 Jul 2007 15:31:44 +0000

Don,

I’m assuming you’re running Solaris on your Sun boxes, not Linux?

P.S.–Great blog. I’m trying out a Sun box here soon mostly on your blog’s recommendation.

By: Wout

Wout — Thu, 26 Jul 2007 08:42:56 +0000

Well, from the bugs it seems that data read from the disk is not what was stored. So the system memory is not affected?

ZFS adds the checksum before the write. As soon as you hand off a block of data to the ZFS subsystem, it’s protected. Anything that happens at a lower level, like disk errors, wire problems, controller issues etc will be detected.

If the data you give to ZFS is wrong though, there’s nothing that can help you ZFS will simply hand you back the same wrong data when you read it.

So if this bug silently overwrites main memory, ZFS might not be in a position to help.