1Name
2 Text::WordDiff - Track changes between documents
3
4Synopsis
5 use Text::WordDiff;
6
7 my $diff = word_diff 'file1.txt', 'file2.txt', { STYLE => 'HTML' };
8 my $diff = word_diff \$string1, \$string2, { STYLE => 'ANSIColor' };
9 my $diff = word_diff \*FH1, \*FH2; \%options;
10 my $diff = word_diff \&reader1, \&reader2;
11 my $diff = word_diff \@records1, \@records2;
12
13 # May also mix input types:
14 my $diff = word_diff \@records1, 'file_B.txt';
15
16Description
17 This module is a variation on the lovely Text::Diff module. Rather than
18 generating traditional line-oriented diffs, however, it generates
19 word-oriented diffs. This can be useful for tracking changes in
20 narrative documents or documents with very long lines. To diff source
21 code, one is still best off using Text::Diff. But if you want to see how
22 a short story changed from one version to the next, this module will do
23 the job very nicely.
24
25 What is a Word?
26 I'm glad you asked! Well, sort of. It's a really hard question to
27 answer. I consulted a number of sources, but really just did my best to
28 punt on the question by reformulating it as, "How do I split text up
29 into individual words?" The short answer is to split on word boundaries.
30 However, every word has two boundaries, one at the beginning and one at
31 the end. So splitting on "/\b/" didn't work so well. What I really
32 wanted to do was to split on the *beginning* of every word. Fortunately,
33 _Mastering Regular Expressions_ has a recipe for that:
34 "/(?<!\w)(?=\w)/". I've borrowed this regular expression for use in
35 Perls before 5.6.x, but go for the Unicode variant in 5.6.0 and newer:
36 "/(?<!\p{IsWord})(?=\p{IsWord})/". With either of these regular
37 expressions, this sentence, for example, would be split up into the
38 following tokens:
39
40 my @words = (
41 'With ',
42 'either ',
43 'of ',
44 'these ',
45 'regular ',
46 "expressions,\n",
47 'this ',
48 'sentence, ',
49 'for ',
50 'example, ',
51 'would ',
52 'be ',
53 'split ',
54 'up ',
55 'into ',
56 'the ',
57 'following ',
58 'tokens:'
59 );
60
61 Note that this allows the tokens to include any spacing or punctuation
62 after each word. So it's not just comparing words, but word-like tokens.
63 This makes sense to me, at least, as the diff is between these tokens,
64 and thus leads to a nice word-and-space-and-punctation type diff. It's
65 not unlike what a word processor might do (although a lot of them are
66 character-based, but that seemed a bit extreme--feel free to dupe this
67 module into Text::CharDiff!).
68
69 Now, I acknowledge that there are localization issues with this
70 approach. In particular, it will fail with Chinese, Japanese, and Korean
71 text, as these languages don't put non-word characters between words.
72 Ideally, Test::WordDiff would then split on every charaters (since a
73 single character often equals a word), but such is not the case when the
74 "utf8" flag is set on a string. For example, This simple script:
75
76 use strict;
77 use utf8;
78 use Data::Dumper;
79 my $string = '뼈뼉뼘뼙뼛뼜뼝뽀뽁뽄뽈뽐뽑뽕뾔뾰뿅뿌뿍뿐뿔뿜뿟뿡쀼쁑쁘쁜쁠쁨쁩삐';
80 my @tokens = split /(?<!\p{IsWord})(?=\p{IsWord})/msx, $string;
81 print Dumper \@tokens;
82
83 Outputs:
84
85 $VAR1 = [
86 "\x{bf08}\x{bf09}\x{bf18}\x{bf19}\x{bf1b}\x{bf1c}\x{bf1d}\x{bf40}\x{bf41}\x{bf44}\x{bf48}\x{bf50}\x{bf51}\x{bf55}\x{bf94}\x{bfb0}\x{bfc5}\x{bfcc}\x{bfcd}\x{bfd0}\x{bfd4}\x{bfdc}\x{bfdf}\x{bfe1}\x{c03c}\x{c051}\x{c058}\x{c05c}\x{c060}\x{c068}\x{c069}\x{c090}"
87 ];
88
89 Not so useful. It seems to be less of a problem if the "use utf8;" line
90 is commented out, in which caase we get:
91
92 $VAR1 = [
93 '뼈',
94 '뼉',
95 '뼘',
96 '뼙',
97 '뼛',
98 '뼜',
99 '뼝',
100 '뽀',
101 '뽁',
102 '뽄',
103 '뽈',
104 '뽐',
105 '뽑',
106 '뽕',
107 '뾔',
108 '뾰',
109 '뿅',
110 '뿌',
111 '뿍',
112 '뿐',
113 '뿔',
114 '뿜',
115 '뿟',
116 '뿡',
117 '?',
118 '?쁑',
119 '쁘',
120 '쁜',
121 '쁠',
122 '쁨',
123 '쁩',
124 '삐'
125 ];
126
127 Someone whose more familiar with non-space-using languages will have to
128 explain to me how I might be able to duplicate this pattern when "utf8;"
129 is on, seing as it may very well be important to have it on in order to
130 ensure proper character semantics.
131
132 However, if my word tokenization approach is just too naive, and you
133 decide that you need to take a different approach (maybe use
134 Lingua::ZH::Toke or similar module), you can still use this module;
135 you'll just have to tokenize your strings into words yourself, and pass
136 them to word_diff() as array references:
137
138 word_diff \@my_words1, \@my_words2;
139
140Options
141 word_diff() takes two arguments from which to draw input and an optional
142 hash reference of options to control its output. The first two arguments
143 contain the data to be diffed, and each may be in the form of any of the
144 following (that is, they can be in two different formats):
145
146 * String
147 A bare scalar will be assumed to be a file name. The file will be
148 opened and split up into words. word_diff() will also "stat" the
149 file to get the last modified time for use in the header, unless the
150 relevant option ("MTIME_A" or "MTIME_B") has been specified
151 explicitly.
152
153 * Scalar Reference
154 A scalar reference will be assumed to refer to a string. That string
155 will be split up into words.
156
157 * Array Reference
158 An array reference will be assumed to be a list of words.
159
160 * File Handle
161 A glob or IO::Handle-derived object will be read from and split up
162 into its constituent words.
163
164 The optional hash reference may contain the following options.
165 Additional options may be specified by the formattting class; see the
166 specific class for details.
167
168 * STYLE
169 "ANSIColor", "HTML" or an object or class name for a class providing
170 "file_header()", "hunk_header()", "same_items()", "delete_items()",
171 "insert_items()", "hunk_footer()" and "file_footer()" methods.
172 Defaults to "ANSIColor" for nice display of diffs in an ANSI
173 Color-supporting terminal.
174
175 If the package indicated by the "STYLE" has no "new()" method,
176 "word_diff()" will load it automatically (lazy loading). It will
177 then instantiate an object of that class, passing in the options
178 hash reference with which the formatting class can initialize the
179 object.
180
181 Styles may be specified as class names ("STYLE => "My::Foo""), in
182 which case they will be instantiated by calling the "new()"
183 construcctor and passing in the options hash reference, or as
184 objects ("STYLE => My::Foo->new").
185
186 The simplest way to implement your own formatting style is to create
187 a new class that inherits from Text::WordDiff::Base, wherein the
188 "new()" method is already provided, and the "file_header()" returns
189 a Unified diff-style header. All of the other formatting methods
190 simply return empty strings, and are therefore ripe for overriding.
191
192 * FILENAME_A, MTIME_A, FILENAME_B, MTIME_B
193 The name of the file and the modification time "files" in epoch
194 seconds. Unless a defined value is specified for these options, they
195 will be filled in for each file when word_diff() is passed a
196 filename. If a filename is not passed in and "FILENAME_A" and
197 "FILENAME_B" are not defined, the header will not be printed by the
198 base formatting base class.
199
200 * OUTPUT
201 The method by which diff output should be, well, *output*. Examples
202 and their equivalent subroutines:
203
204 OUTPUT => \*FOOHANDLE, # like: sub { print FOOHANDLE shift() }
205 OUTPUT => \$output, # like: sub { $output .= shift }
206 OUTPUT => \@output, # like: sub { push @output, shift }
207 OUTPUT => sub { $output .= shift },
208
209 If "OUTPUT" is not defined, word_diff() will simply return the diff
210 as a string. If "OUTPUT" is a code reference, it will be called once
211 with the file header, once for each hunk body, and once for each
212 piece of content. If "OUTPUT" is an IO::Handle-derived object,
213 output will be sent to that handle.
214
215 * FILENAME_PREFIX_A, FILENAME_PREFIX_B
216 The string to print before the filename in the header. Defaults are
217 "---", "+++".
218
219 * DIFF_OPTS
220 A hash reference to be passed as the options to
221 "Algorithm::Diff->new". See Algorithm::Diff for details on available
222 options.
223
224Formatting Classes
225 Text::WordDiff comes with two formatting classes:
226
227 Text::WordDiff::ANSIColor
228 This is the default formatting class. It emits a header and then the
229 diff content, with deleted text in bodfaced red and inserted text in
230 boldfaced green.
231
232 Text::WordDiff::HTML
233 Specify "STYLE => 'HTML'" to take advantage of this formatting
234 class. It outputs the diff content as XHTML, with deleted text in
235 "<del>" elements and inserted text in "<ins>" elements.
236
237 To implement your own formatting class, simply inherit from
238 Text::WordDiff::Base and override its methods as necssary. By default,
239 only the "file_header()" formatting method returns a value. All others
240 simply return empty strings, and are therefore ripe for overriding:
241
242 package My::WordDiff::Format;
243 use base 'Text::WordDiff::Base';
244
245 sub file_footer { return "End of diff\n"; }
246
247 The methods supplied by the base class are:
248
249 "new()"
250 Constructs and returns a new formatting object. It takes a single
251 hash reference as its argument, and uses it to construct the object.
252 The nice thing about this is that if you want to support other
253 options in your formatting class, you can just use them in the
254 formatting object constructed by the Text::WordDiff::Base class and
255 document that they can be passed as part of the options hash
256 refernce to word_diff().
257
258 "file_header()"
259 Called once for a single call to "word_diff()", this method outputs
260 the header for the whole diff. This is the only formatting method in
261 the base class that returns anything other than an empty string. It
262 collects the filenames from "filname_a()" and "filename_b()" and, if
263 they're defined, uses the relevant prefixes and modification times
264 to return a unified diff-style header.
265
266 "hunk_header()"
267 This method is called for each diff hunk. It should output any
268 necessary header for the hunk.
269
270 "same_items()"
271 This method is called for items that have not changed between the
272 two sequnces being compared. The unchanged items will be passed as a
273 list to the method.
274
275 "delete_items"
276 This method is called for items in the first sequence that are not
277 present in the second sequcne. The deleted items will be passed as a
278 list to the method.
279
280 "insert_items"
281 This method is called for items in the second sequence that are not
282 present in the first sequcne. The inserted items will be passed as a
283 list to the method.
284
285 "hunk_footer"
286 This method is called at the end of a hunk. It should output any
287 necessary content to close out the hunk.
288
289 "file_footer()"
290 This method is called once when the whole diff has been procssed. It
291 should output any necessary content to close out the diff file.
292
293 "filename_a"
294 This accessor returns the value specified for the "FILENAME_A"
295 option to word_diff().
296
297 "filename_b"
298 This accessor returns the value specified for the "FILENAME_B"
299 option to word_diff().
300
301 "mtime_a"
302 This accessor returns the value specified for the "MTIME_A" option
303 to word_diff().
304
305 "mtime_b"
306 This accessor returns the value specified for the "MTIME_B" option
307 to word_diff().
308
309 "filename_prefix_a"
310 This accessor returns the value specified for the
311 "FILENAME_PREFIX_A" option to word_diff().
312
313 "filename_prefix_b"
314 This accessor returns the value specified for the
315 "FILENAME_PREFIX_B" option to word_diff().
316
317See Also
318 Text::Diff
319 Inspired the interface and implementation of this module. Thanks
320 Barry!
321
322 Text::ParagraphDiff
323 A module that attempts to diff paragraphs and the words in them.
324
325 Algorithm::Diff
326 The module that makes this all possible.
327
328Bugs
329 Please send bug reports to <bug-text-worddiff@rt.cpan.org>.
330
331Author
332 David Wheeler <david@kineticode.com>
333
334Copyright and License
335 Copyright (c) 2005 Kineticode, Inc. All Rights Reserved.
336
337 This module is free software; you can redistribute it and/or modify it
338 under the same terms as Perl itself.
339
340