The joys of ZFS: transparent compression

One of the great things of ZFS is its built-in compression capability. Enabling compression on a ZFS filesystem allows you to save a significant amount of space, especially in a field like ours where one sometimes needs to use large plain text files. It is completely transparent: the user doesn’t notice anything. Well… unless they check the output of the ls command carefully, of course. I just noticed the most extreme example I have seen to date:

▶ ls -lh
total 640G
-rw-r----- 1 user1 group1  12T May 15 00:27 chr10impv1.gen
-rw-r----- 1 user1 group1 695G May 13 15:01 chr22impv1.gen
-rw-r----- 1 user1 group1 1.1K May 15 12:53 fcgene_out_fcgene.log
-rw-r----- 1 user1 group1 2.7M May 14 17:40 file1.sample
-rw-r----- 1 user1 group1 3.2M May 14 17:46 file2.sample

Notice how ls says that the first .gen file is 12TB in size, but the actually used total disk space (see the first line) is 640GB!

An alternative way to check this is using the du command to show disk space usage:

▶ du -bh
13T     .

▶ du -h
640G    .

This ZFS filesystem uses the default LZ4 compression, and as can be seen below, the compression ratio on that filesystem is quite good:

▶ sudo zfs get compression,compressratio storage/projects/Project1
NAME                        PROPERTY       VALUE     SOURCE
storage/projects/Project1   compression    lz4       inherited from storage
storage/projects/Project1   compressratio  7.70x     -

More information on ZFS compression can be found at, for example, the following to blogs:

• Jim Salter’s blog: ZFS compression: yes, you want this
• Aaron Toponce’s blog: ZFS Administration, Part XI- Compression and Deduplication