1229159Sadrian 2229159SadrianXZ data compression in Linux 3229159Sadrian============================ 4229159Sadrian 5229159SadrianIntroduction 6229159Sadrian 7229159Sadrian XZ is a general purpose data compression format with high compression 8229159Sadrian ratio and relatively fast decompression. The primary compression 9229159Sadrian algorithm (filter) is LZMA2. Additional filters can be used to improve 10229159Sadrian compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters 11229159Sadrian improve compression ratio of executable data. 12229159Sadrian 13229159Sadrian The XZ decompressor in Linux is called XZ Embedded. It supports 14229159Sadrian the LZMA2 filter and optionally also BCJ filters. CRC32 is supported 15229159Sadrian for integrity checking. The home page of XZ Embedded is at 16229159Sadrian <http://tukaani.org/xz/embedded.html>, where you can find the 17229159Sadrian latest version and also information about using the code outside 18229159Sadrian the Linux kernel. 19229159Sadrian 20229159Sadrian For userspace, XZ Utils provide a zlib-like compression library 21229159Sadrian and a gzip-like command line tool. XZ Utils can be downloaded from 22229159Sadrian <http://tukaani.org/xz/>. 23229159Sadrian 24229159SadrianXZ related components in the kernel 25229159Sadrian 26229159Sadrian The xz_dec module provides XZ decompressor with single-call (buffer 27229159Sadrian to buffer) and multi-call (stateful) APIs. The usage of the xz_dec 28229159Sadrian module is documented in include/linux/xz.h. 29229159Sadrian 30229159Sadrian The xz_dec_test module is for testing xz_dec. xz_dec_test is not 31229159Sadrian useful unless you are hacking the XZ decompressor. xz_dec_test 32229159Sadrian allocates a char device major dynamically to which one can write 33229159Sadrian .xz files from userspace. The decompressed output is thrown away. 34229159Sadrian Keep an eye on dmesg to see diagnostics printed by xz_dec_test. 35229159Sadrian See the xz_dec_test source code for the details. 36229159Sadrian 37229159Sadrian For decompressing the kernel image, initramfs, and initrd, there 38229159Sadrian is a wrapper function in lib/decompress_unxz.c. Its API is the 39229159Sadrian same as in other decompress_*.c files, which is defined in 40229159Sadrian include/linux/decompress/generic.h. 41229159Sadrian 42229159Sadrian scripts/xz_wrap.sh is a wrapper for the xz command line tool found 43229159Sadrian from XZ Utils. The wrapper sets compression options to values suitable 44229159Sadrian for compressing the kernel image. 45229159Sadrian 46229159Sadrian For kernel makefiles, two commands are provided for use with 47229159Sadrian $(call if_needed). The kernel image should be compressed with 48229159Sadrian $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 49229159Sadrian dictionary. It will also append a four-byte trailer containing the 50229159Sadrian uncompressed size of the file, which is needed by the boot code. 51229159Sadrian Other things should be compressed with $(call if_needed,xzmisc) 52229159Sadrian which will use no BCJ filter and 1 MiB LZMA2 dictionary. 53229159Sadrian 54229159SadrianNotes on compression options 55229159Sadrian 56229159Sadrian Since the XZ Embedded supports only streams with no integrity check or 57229159Sadrian CRC32, make sure that you don't use some other integrity check type 58229159Sadrian when encoding files that are supposed to be decoded by the kernel. With 59229159Sadrian liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 60229159Sadrian when encoding. With the xz command line tool, use --check=none or 61229159Sadrian --check=crc32. 62229159Sadrian 63229159Sadrian Using CRC32 is strongly recommended unless there is some other layer 64229159Sadrian which will verify the integrity of the uncompressed data anyway. 65229159Sadrian Double checking the integrity would probably be waste of CPU cycles. 66229159Sadrian Note that the headers will always have a CRC32 which will be validated 67229159Sadrian by the decoder; you can only change the integrity check type (or 68229159Sadrian disable it) for the actual uncompressed data. 69229159Sadrian 70229159Sadrian In userspace, LZMA2 is typically used with dictionary sizes of several 71229159Sadrian megabytes. The decoder needs to have the dictionary in RAM, thus big 72229159Sadrian dictionaries cannot be used for files that are intended to be decoded 73229159Sadrian by the kernel. 1 MiB is probably the maximum reasonable dictionary 74229159Sadrian size for in-kernel use (maybe more is OK for initramfs). The presets 75229159Sadrian in XZ Utils may not be optimal when creating files for the kernel, 76229159Sadrian so don't hesitate to use custom settings. Example: 77229159Sadrian 78229159Sadrian xz --check=crc32 --lzma2=dict=512KiB inputfile 79229159Sadrian 80229159Sadrian An exception to above dictionary size limitation is when the decoder 81229159Sadrian is used in single-call mode. Decompressing the kernel itself is an 82229159Sadrian example of this situation. In single-call mode, the memory usage 83229159Sadrian doesn't depend on the dictionary size, and it is perfectly fine to 84229159Sadrian use a big dictionary: for maximum compression, the dictionary should 85229159Sadrian be at least as big as the uncompressed data itself. 86229159Sadrian 87229159SadrianFuture plans 88229159Sadrian 89229159Sadrian Creating a limited XZ encoder may be considered if people think it is 90229159Sadrian useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at 91229159Sadrian the fastest settings, so it isn't clear if LZMA2 encoder is wanted 92229159Sadrian into the kernel. 93229159Sadrian 94229159Sadrian Support for limited random-access reading is planned for the 95229159Sadrian decompression code. I don't know if it could have any use in the 96229159Sadrian kernel, but I know that it would be useful in some embedded projects 97229159Sadrian outside the Linux kernel. 98229159Sadrian 99229159SadrianConformance to the .xz file format specification 100229159Sadrian 101229159Sadrian There are a couple of corner cases where things have been simplified 102229159Sadrian at expense of detecting errors as early as possible. These should not 103229159Sadrian matter in practice all, since they don't cause security issues. But 104229159Sadrian it is good to know this if testing the code e.g. with the test files 105229159Sadrian from XZ Utils. 106229159Sadrian 107229159SadrianReporting bugs 108229159Sadrian 109229159Sadrian Before reporting a bug, please check that it's not fixed already 110229159Sadrian at upstream. See <http://tukaani.org/xz/embedded.html> to get the 111229159Sadrian latest code. 112229159Sadrian 113229159Sadrian Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on 114229159Sadrian Freenode and talk to Larhzu. I don't actively read LKML or other 115229159Sadrian kernel-related mailing lists, so if there's something I should know, 116229159Sadrian you should email to me personally or use IRC. 117229159Sadrian 118229159Sadrian Don't bother Igor Pavlov with questions about the XZ implementation 119229159Sadrian in the kernel or about XZ Utils. While these two implementations 120229159Sadrian include essential code that is directly based on Igor Pavlov's code, 121229159Sadrian these implementations aren't maintained nor supported by him. 122229159Sadrian 123