1
2XZ Utils
3========
4
5 0. Overview
6 1. Documentation
7 1.1. Overall documentation
8 1.2. Documentation for command line tools
9 1.3. Documentation for liblzma
10 2. Version numbering
11 3. Reporting bugs
12 4. Translating the xz tool
13 5. Other implementations of the .xz format
14 6. Contact information
15
16
170. Overview
18-----------
19
20 XZ Utils provide a general-purpose data compression library and
21 command line tools. The native file format is the .xz format, but
22 also the legacy .lzma format is supported. The .xz format supports
23 multiple compression algorithms, which are called "filters" in
24 context of XZ Utils. The primary filter is currently LZMA2. With
25 typical files, XZ Utils create about 30 % smaller files than gzip.
26
27 To ease adapting support for the .xz format into existing applications
28 and scripts, the API of liblzma is somewhat similar to the API of the
29 popular zlib library. For the same reason, the command line tool xz
30 has similar command line syntax than that of gzip.
31
32 When aiming for the highest compression ratio, LZMA2 encoder uses
33 a lot of CPU time and may use, depending on the settings, even
34 hundreds of megabytes of RAM. However, in fast modes, LZMA2 encoder
35 competes with bzip2 in compression speed, RAM usage, and compression
36 ratio.
37
38 LZMA2 is reasonably fast to decompress. It is a little slower than
39 gzip, but a lot faster than bzip2. Being fast to decompress means
40 that the .xz format is especially nice when the same file will be
41 decompressed very many times (usually on different computers), which
42 is the case e.g. when distributing software packages. In such
43 situations, it's not too bad if the compression takes some time,
44 since that needs to be done only once to benefit many people.
45
46 With some file types, combining (or "chaining") LZMA2 with an
47 additional filter can improve compression ratio. A filter chain may
48 contain up to four filters, although usually only one two is used.
49 For example, putting a BCJ (Branch/Call/Jump) filter before LZMA2
50 in the filter chain can improve compression ratio of executable files.
51
52 Since the .xz format allows adding new filter IDs, it is possible that
53 some day there will be a filter that is, for example, much faster to
54 compress than LZMA2 (but probably with worse compression ratio).
55 Similarly, it is possible that some day there is a filter that will
56 compress better than LZMA2.
57
58 XZ Utils doesn't support multithreaded compression or decompression
59 yet. It has been planned though and taken into account when designing
60 the .xz file format.
61
62
631. Documentation
64----------------
65
661.1. Overall documentation
67
68 README This file
69
70 INSTALL.generic Generic install instructions for those not familiar
71 with packages using GNU Autotools
72 INSTALL Installation instructions specific to XZ Utils
73 PACKAGERS Information to packagers of XZ Utils
74
75 COPYING XZ Utils copyright and license information
76 COPYING.GPLv2 GNU General Public License version 2
77 COPYING.GPLv3 GNU General Public License version 3
78 COPYING.LGPLv2.1 GNU Lesser General Public License version 2.1
79
80 AUTHORS The main authors of XZ Utils
81 THANKS Incomplete list of people who have helped making
82 this software
83 NEWS User-visible changes between XZ Utils releases
84 ChangeLog Detailed list of changes (commit log)
85 TODO Known bugs and some sort of to-do list
86
87 Note that only some of the above files are included in binary
88 packages.
89
90
911.2. Documentation for command line tools
92
93 The command line tools are documented as man pages. In source code
94 releases (and possibly also in some binary packages), the man pages
95 are also provided in plain text (ASCII only) and PDF formats in the
96 directory "doc/man" to make the man pages more accessible to those
97 whose operating system doesn't provide an easy way to view man pages.
98
99
1001.3. Documentation for liblzma
101
102 The liblzma API headers include short docs about each function
103 and data type as Doxygen tags. These docs should be quite OK as
104 a quick reference.
105
106 I have planned to write a bunch of very well documented example
107 programs, which (due to comments) should work as a tutorial to
108 various features of liblzma. No such example programs have been
109 written yet.
110
111 For now, if you have never used liblzma, libbzip2, or zlib, I
112 recommend learning *basics* of zlib API. Once you know that, it
113 should be easier to learn liblzma.
114
115 http://zlib.net/manual.html
116 http://zlib.net/zlib_how.html
117
118
1192. Version numbering
120--------------------
121
122 The version number format of XZ Utils is X.Y.ZS:
123
124 - X is the major version. When this is incremented, the library
125 API and ABI break.
126
127 - Y is the minor version. It is incremented when new features are
128 added without breaking existing API or ABI. Even Y indicates
129 stable release and odd Y indicates unstable (alpha or beta
130 version).
131
132 - Z is the revision. This has different meaning for stable and
133 unstable releases:
134 * Stable: Z is incremented when bugs get fixed without adding
135 any new features.
136 * Unstable: Z is just a counter. API or ABI of features added
137 in earlier unstable releases having the same X.Y may break.
138
139 - S indicates stability of the release. It is missing from the
140 stable releases where Y is an even number. When Y is odd, S
141 is either "alpha" or "beta" to make it very clear that such
142 versions are not stable releases. The same X.Y.Z combination is
143 not used for more than one stability level i.e. after X.Y.Zalpha,
144 the next version can be X.Y.(Z+1)beta but not X.Y.Zbeta.
145
146
1473. Reporting bugs
148-----------------
149
150 Naturally it is easiest for me if you already know what causes the
151 unexpected behavior. Even better if you have a patch to propose.
152 However, quite often the reason for unexpected behavior is unknown,
153 so here are a few things to do before sending a bug report:
154
155 1. Try to create a small example how to reproduce the issue.
156
157 2. Compile XZ Utils with debugging code using configure switches
158 --enable-debug and, if possible, --disable-shared. If you are
159 using GCC, use CFLAGS='-O0 -ggdb3'. Don't strip the resulting
160 binaries.
161
162 3. Turn on core dumps. The exact command depends on your shell;
163 for example in GNU bash it is done with "ulimit -c unlimited",
164 and in tcsh with "limit coredumpsize unlimited".
165
166 4. Try to reproduce the suspected bug. If you get "assertion failed"
167 message, be sure to include the complete message in your bug
168 report. If the application leaves a coredump, get a backtrace
169 using gdb:
170 $ gdb /path/to/app-binary # Load the app to the debugger.
171 (gdb) core core # Open the coredump.
172 (gdb) bt # Print the backtrace. Copy & paste to bug report.
173 (gdb) quit # Quit gdb.
174
175 Report your bug via email or IRC (see Contact information below).
176 Don't send core dump files or any executables. If you have a small
177 example file(s) (total size less than 256 KiB), please include
178 it/them as an attachment. If you have bigger test files, put them
179 online somewhere and include an URL to the file(s) in the bug report.
180
181 Always include the exact version number of XZ Utils in the bug report.
182 If you are using a snapshot from the git repository, use "git describe"
183 to get the exact snapshot version. If you are using XZ Utils shipped
184 in an operating system distribution, mention the distribution name,
185 distribution version, and exact xz package version; if you cannot
186 repeat the bug with the code compiled from unpatched source code,
187 you probably need to report a bug to your distribution's bug tracking
188 system.
189
190
1914. Translating the xz tool
192--------------------------
193
194 The messages from the xz tool have been translated into a few
195 languages. Before starting to translate into a new language, ask
196 the author that someone else hasn't already started working on it.
197
198 Test your translation. Testing includes comparing the translated
199 output to the original English version by running the same commands
200 in both your target locale and with LC_ALL=C. Ask someone to
201 proof-read and test the translation.
202
203 Testing can be done e.g. by installing xz into a temporary directory:
204
205 ./configure --disable-shared --prefix=/tmp/xz-test
206 # <Edit the .po file in the po directory.>
207 make -C po update-po
208 make install
209 bash debug/translations.bash | less
210 bash debug/translations.bash | less -S # For --list outputs
211
212 Repeat the above as needed (no need to re-run configure though).
213
214 Note especially the following:
215
216 - The output of --help and --long-help must look nice on
217 a 80-column terminal. It's OK to add extra lines if needed.
218
219 - In contrast, don't add extra lines to error messages and such.
220 They are often preceded with e.g. a filename on the same line,
221 so you have no way to predict where to put a \n. Let the terminal
222 do the wrapping even if it looks ugly. Adding new lines will be
223 even uglier in the generic case even if it looks nice in a few
224 limited examples.
225
226 - Be careful with column alignment in tables and table-like output
227 (--list, --list --verbose --verbose, --info-memory, --help, and
228 --long-help):
229
230 * All descriptions of options in --help should start in the
231 same column (but it doesn't need to be the same column as
232 in the English messages; just be consistent if you change it).
233 Check that both --help and --long-help look OK, since they
234 share several strings.
235
236 * --list --verbose and --info-memory print lines that have
237 the format "Description: %s". If you need a longer
238 description, you can put extra space between the colon
239 and %s. Then you may need to add extra space to other
240 strings too so that the result as a whole looks good (all
241 values start at the same column).
242
243 * The columns of the actual tables in --list --verbose --verbose
244 should be aligned properly. Abbreviate if necessary. It might
245 be good to keep at least 2 or 3 spaces between column headings
246 and avoid spaces in the headings so that the columns stand out
247 better, but this is a matter of opinion. Do what you think
248 looks best.
249
250 - Be careful to put a period at the end of a sentence when the
251 original version has it, and don't put it when the original
252 doesn't have it. Similarly, be careful with \n characters
253 at the beginning and end of the strings.
254
255 - Read the TRANSLATORS comments that have been extracted from the
256 source code and included in xz.pot. If they suggest testing the
257 translation with some type of command, do it. If testing needs
258 input files, use e.g. tests/files/good-*.xz.
259
260 - When updating the translation, read the fuzzy (modified) strings
261 carefully, and don't mark them as updated before you actually
262 have updated them. Reading through the unchanged messages can be
263 good too; sometimes you may find a better wording for them.
264
265 - If you find language problems in the original English strings,
266 feel free to suggest improvements. Ask if something is unclear.
267
268 - The translated messages should be understandable (sometimes this
269 may be a problem with the original English messages too). Don't
270 make a direct word-by-word translation from English especially if
271 the result doesn't sound good in your language.
272
273 In short, take your time and pay attention to the details. Making
274 a good translation is not a quick and trivial thing to do. The
275 translated xz should look as polished as the English version.
276
277
2785. Other implementations of the .xz format
279------------------------------------------
280
281 7-Zip and the p7zip port of 7-Zip support the .xz format starting
282 from the version 9.00alpha.
283
284 http://7-zip.org/
285 http://p7zip.sourceforge.net/
286
287 XZ Embedded is a limited implementation written for use in the Linux
288 kernel, but it is also suitable for other embedded use.
289
290 http://tukaani.org/xz/embedded.html
291
292
2936. Contact information
294----------------------
295
296 If you have questions, bug reports, patches etc. related to XZ Utils,
297 contact Lasse Collin <lasse.collin@tukaani.org> (in Finnish or English).
298 I'm sometimes slow at replying. If you haven't got a reply within two
299 weeks, assume that your email has got lost and resend it or use IRC.
300
301 You can find me also from #tukaani on Freenode; my nick is Larhzu.
302 The channel tends to be pretty quiet, so just ask your question and
303 someone may wake up.
304
305