Comments on: ZFS & MySQL/InnoDB Compression Update

By: Ethan

Ethan — Mon, 12 Oct 2009 18:57:36 +0000

@Guest: Not a limit in either, simply a requirement by Solaris (unlike Linux) to have virtual memory backing memory allocation.

By: Guest

Guest — Mon, 12 Oct 2009 02:10:24 +0000

Ethan,
Is it an allocation limit in Solaris, or in InnoDB?

By: Ethan

Ethan — Mon, 30 Mar 2009 17:14:11 +0000

I know this is an old posting, but did you settle on using lzjb or gzip in production, for InnoDB data files? Also, we ran into memory allocation limits in Solaris as well, and it turned out we just didn't have enough swap. We found a need for ~50% swap -> memory allocated. So for a 48G system, with a 40G innodb_buffer_pool, we needed about 18-20G of swap.

By: roland

roland — Sat, 20 Dec 2008 04:29:53 +0000

>Someone really needs to find a (legal) way to use lzo compression with zfs, this is imho the real contender

yes, indeed.

i don`t know why this did not yet attract sun developers.
regarding licensing, i think there is a way to proceed, as the author may (perhaps) give explicit permission to use lzo with zfs. there is even a commercial lzo implementation with some optimization. zfs+lzo would be "win-win" for sun and for lzo author.

there exists a patch for zfs-fuse to enable lzo compression (just demonstration purpose) and the results are real interesting as lzo can give better performance and better compression than lzjb – see http://www.mail-archive.com/zfs-discuss@opensolar...

By: SYSTEMHELDEN.COM /* HELDENFunk

SYSTEMHELDEN.COM /* HELDENFunk — Mon, 08 Dec 2008 17:53:12 +0000

[...] Smugmug’s CEO und Chief Geek Don McAskill probierte neulich OpenSolaris mit ZFS und MySQL aus – und liebt es. Aufgrund der großen Nachfrage, hat er gleich einen draufgelegt und Kompressions-Benchmarks gemacht. [...]

By: Scott C.

Scott C. — Tue, 28 Oct 2008 19:04:38 +0000

Since ZFS compresses only one block at a time, the gzip blocks and tree structure discard mentioned by Jacques above do not apply.
For 16K blocks, what happens is a 16K data block (uncompressed size) is compressed with lzjb or gzip into a new block of smaller size, and stored on disk in the smaller size (which has to be a binary size, IIRC — 8k, 4k, 2k, etc) and like all blocks in ZFS, there is a checksum elsewhere to make sure the data is ok.

LZJB, as mentioned by Jacques, is a LZ-77 style algorithm. It is supposedly faster and a bit better at compression than the original, and versus the lzo suggested by Marc above. Either way, its basically the same class of small-window streaming compression algorithm.

Gzip is also a streaming compression algorithm, and as mentioned above is basically just a LZ 77 followed immediately by huffman encoding. Most of the time, a default huffman encoder is used and the dictionary isn’t built up by analyzing the data first. Gzip has a default dictionary that is used for almost all streams to avoid this step (or more correctly, deflate does) If it is, it is done by analyzing a smaller chunk. Either way for ZFS, this chunk size is at most the size of the file block.

Deflate/Gzip are RFC 1951, and 1952. RFC 1950 is the related zlib wrapper format commonly used in coordination with deflate. (just search for RFC 1952 to find more about gzip than you ever want to know).

For info on LZJB, look at the source code in zfs It is well commented from what I recall.

By: Jacques Chester

Jacques Chester — Wed, 22 Oct 2008 06:02:49 +0000

Just to follow up my previous remark.

LZJB seems to follow the basic scheme in Lempel-Ziv 77 (LZRW1 actually) with some modifications. It has a moving window through which data streams. When an item comes in, the algo checks to see if it’s been seen. If it has, the element is replaced with a pointer to the earliest instance in the window. Otherwise it’s added to a dictionary of such addresses and the algorithm moves to the next item.

This is ideal for filesystem compression for two reasons:

1. You don’t need to store header data for the decompressor. All the data needed to decompress the data is already in situ in the data.
2. You don’t need to see all the data to begin compressing. As soon as you have data streaming in, you can begin to compress it.

By contrast gzip uses an algorithm called Deflate, which is actually two algorithms run in series. The first step is a Lempel-Ziv 77 pass like the one in LZJB, followed by a second pass of Huffman coding, where repeated blocks are replaced with short symbolic codes. This involves a tree structures being built and then discarded every 64k or so, on top of the pass already made by the Lempel-Ziv algorithm.

I think that accounts for the observed differences in performance.

By: Steve

Steve — Mon, 20 Oct 2008 15:02:50 +0000

Did you set the ZFS record size to 16kb before doing this? The MySQL docs say the ZFS record size should match the InnoDB record size of 16kb.

http://dev.mysql.com/tech-resources/articles/mysql-zfs.html#Set_the_ZFS_Recordsize_to_match_the_block_size

If not, I would love to see the results with the record size set to 16kb.

By: Ricardo Correia

Ricardo Correia — Sat, 18 Oct 2008 12:03:44 +0000

Did you run “sync” at the end of each “cp” command and waited for the sync to complete before collecting the elapsed time?

Note that ZFS only compresses data when it’s actually writing to the disks, which typically only happens every 5 seconds given enough RAM. So if you didn’t sync at the end, the first 2 tables are not really useful for compression comparison purposes and the other ones are very suspect

Also, if you used sync at the end, the average of the 10 runs would be more representative of the actual performance than only taking the fastest run.

As for the different checksum options, I see no difference in performance when enabling/disabling the default checksum algorithm, but I noticed that in zfs-fuse the SHA1 checksum is way too heavy on the CPUs (especially considering all the context switching going on), but it’s possible that this isn’t so bad in Solaris/OpenSolaris.

By: Jacques Chester

Jacques Chester — Wed, 15 Oct 2008 05:14:29 +0000

I think the difference is that LZJB is a compression algorithm designed for streaming data — it doesn’t rely on a full scan of the data to perform its compression. The gzip algorithm builds a dictionary first, so that explains the big slowdowns on larger datasets.

Obviously building a dictionary of items for compression purposes lets you do things you can’t do otherwise, which is why gzip is better for total compression ratio.

A good followup test would be to see performance on a random-access basis. What happens with lots of writes and updates? I’m not sure if gzip requires a from-the-top effort or not. I imagine Jon Bonwick could tell you.

Of course, it could be that I’m an undergraduate who knows not whereof he speaks. YMMV.