MFC bsdgrep(1) fixes: r320414, r328559, r332805-r332806, r332809, r332832,
r332850-r332852, r332856, r332858, r332876, r333351, r334803,
r334806-r334809, r334821, r334837, r334889, r335188, r351769, r352691

r320414:
Expect :mmap_eof_not_eol to fail

It relies on a jemalloc feature (opt.redzone) no longer available after
r319971.

r328559:
Remove t_grep:mmap_eof_not_eol test

The test was marked as an expected failure in r320414 after r319971's import
of a newer jemalloc removed an essential feature (opt.redzone) for
reproducing the behavior it was testing. Since then, no way has been found
or demonstrated to reliably test the behavior, so remove the test.

r332805:
bsdgrep: Split match processing out of procfile

procfile is getting kind of hairy, and it's not going to get better as we
correct some more bits that assume we process one line at a time.

r332806:
bsdgrep: Clean up procmatches a little bit

r332809:
bsdgrep: Add some TODOs for future work on operating on chunks

r332832:
bsdgrep: Break procmatches down a little bit more

Split the matching and non-matching cases out into their own functions to
reduce future complexity. As the name implies, procmatches will eventually
process more than one match itself in the future.

r332850:
bsdgrep: Some light cleanup

There's no point checking for a bunch of file modes if we're not a
practicing believer of DIR_SKIP or DEV_SKIP.

This also reduces some style violations that were particularly ugly looking
when browsing through.

r332851:
bsdgrep: More trivial cleanup/style cleanup

We can avoid branching for these easily reduced patterns

r332852:
bsdgrep: if chain => switch

This makes some of this a little easier to follow (in my opinion).

r332856:
bsdgrep: Fix --include/--exclude ordering issues

Prior to r332851:
* --exclude always win out over --include
* --exclude-dir always wins out over --include-dir

r332851 broke that behavior, resulting in:
* First of --exclude, --include wins
* First of --exclude-dir, --include-dir wins

As it turns out, both behaviors are wrong by modern grep standards- the
latest rule wins. e.g.:

`grep --exclude foo --include foo 'thing' foo`
foo is included

`grep --include foo --exclude foo 'thing' foo`
foo is excluded

As tested with GNU grep 3.1.

This commit makes bsdgrep follow this behavior.

r332858:
bsdgrep: Use grep_strdup instead of grep_malloc+strcpy

r332876:
bsdgrep: Fix build failure WITHOUT_LZMA (incorrect bracket placement)

r333351:
bsdgrep: Allow "-" to be passed to -f to mean "standard input"

A version of this patch was originally sent to me by se@, matching behavior
from newer versions of GNU grep.

While there have been some differences of opinion on whether stdin should be
closed or not after depleting it in process of -f, I've opted to leave stdin
open and just let the later matching stuff fail and result in a no-match.
I'm not married to the current behavior- it was generally chosen since we
are adopting this in particular from GNU grep, and I would like to stay
consistent without a strong argument to the contrary. The current behavior
isn't technically wrong, it's just fairly unfriendly to the developer-user
of grep that may not realize their usage is trivially invalid.

r334803:
netbsd-tests: grep(1): Add test for -c flag

Someone might be inclined to accidentally break this. someone might have
written said test because they broke it locally.

r334806:
bsdgrep(1): Do some less dirty things with return types

Neither procfile nor grep_tree return anything meaningful to their callers.
None of the callers actually care about how many lines were matched in all
of the files they processed; it's all about "did anything match?"

This is generally just a light refactoring to remind me of what actually
matters as I'm rewriting these bits to care less about 'stuff'.

r334807:
bsdgrep(1): whoops, garbage collect the now write-only variable

r334808:
bsdgrep(1): Don't initialize fts_flags twice

Admittedly, this is a clang-scan complaint... but it wasn't wrong. fts_flags
is initialized by all cases in the switch(), which should be fairly obvious.
Annotate this anyways.

r334809:
netbsd-tests: bsdgrep(1): Add a test for -m, too

r334821:
bsdgrep(1): Slooowly peel away the chunky onion

(or peel off the band-aid, whatever floats your boat)

This addresses two separate issues:

1.) Nothing within bsdgrep actually knew whether it cared about line numbers
or not.

2.) The file layer knew nothing about the context in which it was being
called.

#1 is only important when we're *not* processing line-by-line. #2 is
debatably a good idea; the parsing context is only handy because that's
where we store current offset information and, as of this commit, whether or
not it needs to be line-aware.

r334837:
bsdgrep(1): Evict character sequence that moved in

r334889:
bsdgrep(1): Some more int -> bool conversions and name changes

Again motivated by upcoming work to rewrite a bunch of this- single-letter
variable names and slightly misleading variable names ("lastmatches" to
indicate that the last matched) are not helpful.

r335188:
bsdgrep(1): Remove redundant initialization; unconditionally assigned later

r351769:
bsdgrep(1): add some basic tests for some GNU Extension support

These will be expanded later as I come up with good test cases; for now,
these seem to be enough to trigger bugs in base gnugrep and expose missing
features in bsdgrep.

r352691:
bsdgrep(1): various fixes of empty pattern/exit code/-c behavior

When an empty pattern is encountered in the pattern list, I had previously
broken bsdgrep to count that as a "match all" and ignore any other patterns
in the list. This commit rectifies that mistake, among others:

- The -v flag semantics were not quite right; lines matched should have been
counted differently based on whether the -v flag was set or not. procline
now definitively returns whether it's matched or not, and interpreting
that result has been kicked up a level.
- Empty patterns with the -x flag was broken similarly to empty patterns
with the -w flag. The former is a whole-line match and should be more
strict, only matching blank lines. No -x and no -w will will match the
empty string at the beginning of each line.
- The exit code with -L was broken, w.r.t. modern grep. Modern grap will
exit(0) if any file that didn't match was output, so our interpretation
was simply backwards. The new interpretation makes sense to me.

Tests updated and added to try and catch some of this.

This misbehavior was found by autoconf while fixing ports found in PR 229925
expecting either a more sane or a more GNU-like sed.

MFC r326276:

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

MFC r326483: grep test: Fix copyright notice

The copyright notice was erroneously introduced as one from the NetBSD
foundation due to it being copied from a file in the NetBSD test suite, but
this file itself is not derived from or supplied with the NetBSD test suite.

MFC r323611: fastmatch.h: remove duplicate #defines

bsdgrep: add a primitive literal matcher to unbreak fgrep in some scenarios

MFC r322825: bsdgrep: add some additional tests for fgrep

Previously added tests only check that fgrep is somewhat sane and works. Add
some more tests that check that the implementation is basically functional
and not producing incorrect results with various flags.

MFC r322826: bsdgrep: add a primitive literal matcher

fgrep/grep -F will error out at runtime if compiled with a regex(3)
that does not define REG_NOSPEC or REG_LITERAL. glibc is one such regex(3)
implementation, and as it turns out they don't support literal matching at
all.

Provide a primitive literal matcher for use with glibc and other
implementations that don't support literal matching so that we don't
completely lose fgrep/grep -F if compiled against libgnuregex on stable/10,
stable/11, or other systems that we don't necessarily support.

This is a wholly unoptimized implementation with no plans to optimize it as
of now. This is due to both its use-case being primarily on unsupported
systems in the near-distant future and that it's reinventing the wheel that
we already have available as a feature of regex(3).

PR: 222201
Approved by: emaste (mentor, blanket MFC)

MFC r321450: bsdgrep(1): Don't exit before processing every file

Given an empty pattern (i.e. grep "" A B), bsdgrep(1) would previously
exit() with the appropriate exit code upon encountering an empty file.
Likely intended as an optimization, but this behavior is technically
incorrect since an empty pattern should match every line.

PR: 220924
Approved by: emaste (mentor, blanket MFC)

bsdgrep: bump version number to 2.6.0 and update copyright information

MFC r319132: bsdgrep: bump version number and add Kyle Evans copyright

The following changes have been made over the last couple of months:

Features:

- With bsdgrep -r, the working directory is implied if no directory is
specified
- bsdgrep will now behave as bsdgrep -r does when it's named rgrep
- bsdgrep now understands -z/--null-data to use \0 as EOL
- GNU regex compatibility is now indicated with a "GNU compatible" in
the version string

Fixes:

- --mmap no longer hangs when coming across an EOF without an
accompanying EOL
- -o/--color matching generally improved, now produces earliest /
longest matches
- Context output now more closely aligns with GNU grep
- Zero-length matches no longer exhibit broken behavior
- Every output line now honors -b/-H/-n flags

Tests have been added for previous regressions as well as other
previously untested behaviors.

Various other fixes have been commited, and refactoring for further /
later improvements has taken place.

(The original submission changed the version string to 2.5.2, but I
decided to use 2.6.0 to reflect the addition of new features.)

MFC r320754: Update copyright e-mail address to @FreeBSD.org address

Approved by: emaste (mentor, blanket MFC)

MFC r318916: bsdgrep: use safer sizeof() construct

Approved by: emaste (mentor, blanket MFC)

MFC r318914: bsdgrep: correct assumptions to prepare for chunking

Correct a couple of minor BSD grep assumptions that are valid for line
processing but not future chunk-based processing.

Approved by: emaste (mentor, blanket MFC)

bsdgrep: fix build when linking against libgnuregex

MFC r322618: bsdgrep: cast pmatch.rm_so to fix build when linking against
libgnuregex

Approved by: emaste (mentor)

MFC r318574: bsdgrep: Correct per-line line metadata printing

Metadata printing with -b, -H, or -n flags suffered from a few flaws:

1) -b/offset printing was broken when used in conjunction with -o

2) With -o, bsdgrep did not print metadata for every match/line, just
the first match of a line

3) There were no tests for this

Address these issues by outputting this data per-match if the -o flag is
specified, and prior to outputting any matches if -o but not --color,
since --color alone will not generate a new line of output for every
iteration over the matches.

To correct -b output, fudge the line offset as we're printing matches.

While here, make sure we're using grep_printline in -A context. Context
printing should *never* look at the parsing context, just the line.

The tests included do not pass with gnugrep in base due to it exhibiting
similar quirky behavior that bsdgrep previously exhibited.

Approved by: emaste (mentor, blanket MFC)

MFC r318571: bsdgrep: emit more than MAX_LINE_MATCHES per line

We should not set an arbitrary cap on the number of matches on a line,
and in any case MAX_LINE_MATCHES of 32 is much too low. Instead, if we
match more than MAX_LINE_MATCHES, keep processing and matching from the
last match until all are found.

For the regression test, we produce 4096 matches (larger than we expect
we'll ever set MAX_LINE_MATCHES) and make sure we actually get 4096
lines of output with the -o flag.

We'll also make sure that every distinct line is getting its own line
number to detect line metadata not being printed as appropriate along
the way.

PR: 218811
Approved by: emaste (mentor, blanket MFC)

bsdgrep: fix segfault with --mmap and add relevant test

MFC r318565: bsdgrep: fix segfault with --mmap

r313948 partially fixed --mmap behavior but was incomplete. This commit
generally reverts it and does it the more correct way- by just consuming
the rest of the buffer and moving on.

MFC r318908: bsdgrep: add --mmap tests

Basic sanity tests as well as coverage for the bug fixed in r318565.

PR: 219402
Approved by: emaste (mentor, blanket MFC)

bsdgrep: Don't allow negative context flags, add more tests

MFC r318302: bsdgrep: don't allow negative -A / -B / -C

Previously, when given a negative -A/-B/-C argument bsdgrep would
overflow the respective context flag(s) and exhibited surprising
behavior.

Fix this by removing unsignedness of Aflag/Bflag and erroring out if
we're given a value < 0. Also adjust the type used to track 'tail'
context in procfile() so that it accurately reflects the Aflag value
rather than overflowing and losing trailing context.

This also fixes an inconsistency previously existing between -n and
-C "n" behavior. They are now both limited to LLONG_MAX, to be
consistent.

Add some test cases to make sure grep errors out properly for both
negative context values as well as non-numeric context values rather
than giving bogus matches.

MFC r318317: bsdgrep: add more tests for different binary flags

The existing 'binary' test in netbsd-tests/ does a basic check of the
default treatment for binary behavior, but not much more than that.
Given some opportunity for breakage recently that did not trigger any
failures, add some tests to cover the three different binary file
behaviors (a, -I, -U) and their --binary-files= equivalent values.

Approved by: emaste (mentor, blanket MFC)

MFC r317705: bsdgrep: avoid use of magic number for REG_NOSPEC

Approved by: emaste (mentor, blanket MFC)

bsdgrep: fix -w flag matching with an empty pattern

MFC r317703: bsdgrep: fix -w flag matching with an empty pattern

-w flag matching with an empty pattern was generally 'broken', allowing
matches to occur on any line whether or not it actually matches -w
criteria.

This fix required a good amount of refactoring to address. procline()
is altered to *only* process the line and return whether it was a match
or not, necessary to be able to short-circuit the whole function in case
of this matchall flag. -m flag handling is moved out as well because it
suffers from the same fate as context handling if we bypass any actual
pattern matching.

The matching context (matches, mostly) didn't previously exist outside
of procline(), so we go ahead and create context object for file
processing bits to pass around. grep_printline() was created due to
this, for the scenarios where the matches don't actually matter and we
just want to print a line or two, a la flushing the context queue and
no -o or --color specified.

Damage from this broken behavior would have been mitigated by the fact
that it is unlikely users would invoke grep -w with an empty pattern.

This was identified while checking PR 105221 for problems it this may
cause in BSD grep, but PR 105221 is *not* a report of this behavior.

MFC r317741: bsdgrep: correct uninitialized variable introduced in r317703

MFC r317842: bsdgrep: don't ouptut matches with -c, -l, -L

Refactoring done in r317703 broke -c, -l, and -L flags implying
suppression of match printing. Fortunately this is just a matter of not
doing any printing of the resulting matches and context printing was not
broken in this refactoring.

Add some regression tests since this area may still see further
refactoring, include different context flags as well even though they
were not broken in this case.

PR: 219077
Approved by: emaste (mentor, blanket MFC)

bsdgrep: fix escape map building when using TRE (BSD_GREP_FASTMATCH)

MFC r317700: bsdgrep: use calloc where appropriate in grep's tre-fastmatch

Also apply style(9) to a related NULL check.

MFC r317701: bsdgrep: correct test sense from r317700

Kyle's change in review D10098 was correct. I introduced the error when
extracting a portion of that change.

MFC r317704: bsdgrep: fix escape map building for multibyte strings

In BSD grep, fix escape map building in the regex parser. It was
previously using memory not explicitly initialized, and the MBS escape
map was being built based on a version of the pattern with escapes
already parsed out.

This is Kyle's change, but I restored the broken style that already
exists in this file.

Approved by: emaste (mentor, blanket MFC)

MFC r317678: bsdgrep: Add GNU compatible version string indicator

As reported in r218614 it's useful to have an indication of whether or not
BSD grep was built with GNU_GREP_COMPAT.

Approved by: emaste (mentor, blanket MFC)

MFC r317665: bsdgrep: fix -w -v matching improperly with certain patterns

-w and -v flag matching was mostly functional but had some minor
problems:

1. -w flag processing only allowed one iteration through pattern
matching on a line. This was problematic if one pattern could match
more than once, or if there were multiple patterns and the earliest/
longest match was not the most ideal, and

2. Previous work "fixed" things to not further process a line if the
first iteration through patterns produced no matches. This is clearly
wrong if we're dealing with the more restrictive -w matching.

#2 breakage could have also occurred before recent broad rewrites, but
it would be more arbitrary based on input patterns as to whether or not
it actually affected things.

Fix both of these by forcing a retry of the patterns after advancing
just past the start of the first match if we're doing more restrictive
-w matching and we didn't get any hits to start with. Also move -v flag
processing outside of the loop so that we have a greater change to match
in the more restrictive cases. This wasn't strictly wrong, but it could
be a little more error prone.

While here, introduce some regressions tests for this behavior and fix
some excessive wrapping nearby that hindered readability. GNU grep
passes these new tests.

PR: 218467, 218811
Approved by: emaste (mentor, blanket MFC)

MFC r317254: bsdgrep: add BSD_GREP_FASTMATCH knob for built-in fastmatch

Bugs have been found in the fastmatch implementation as used in bsdgrep.
Some have been fixed (r316495) while fixes for others are in review
(D10098).

In comparison with the fastmatch implementation, Kyle Evans found that:

- regex(3)'s performance with literal expressions offers a speed
improvement over fastmatch

- regex(3)'s performance, both with simple BREs and EREs, seems to be
comparable

The regex implementation was imported in r226035, and the commit message
reports:

This is a temporary solution until the whole regex library is
not replaced so that BSD grep development can continue and the
backported code gets some review and testing. This change only
improves scalability slightly, there is no big performance boost
yet but several minor bugs have been found and fixed.

Introduce a WITH_/WITHOUT_BSD_GREP_FASTMATCH knob to support testing
of both approaches.

Regenerate src.conf(5) as per the original commit

PR: 175314, 194823
Approved by: emaste (mentor, blanket MFC)

MFC r303444 (ed): Call basename() in a portable way.

Pull a copy of the filename string before calling basename(). Change the
loop to not return on its own, so we can put a free() statement at the
bottom.

Approved by: emaste (mentor, blanket MFC)

bsdgrep: Use implied working directory for -r if no directories are passed

MFC r317050: bsdgrep: for -r, use the working directory if none specified

This is more sensible than the previous behaviour of grepping stdin,
and matches newer GNU grep behaviour.

MFC r317300 (ngie): Only expect :grep_r_implied to pass with bsdgrep(1)

The test fails with gnu grep from base and ports.

MFC r319002 (ngie): :rgrep : use atf-check to check the exit code/save the
output of grep -r instead of calling grep -r without it, and saving the
output to a file

This ensures that any errors thrown via grep -r are caught, not lost, and
uses existing atf-sh idioms for saving files.

PR: 216307
Approved by: emaste (mentor, blanket MFC)
Relnotes: yes

MFC r317051: bsdgrep: remove output separators between overlapping segments

Make bsdgrep more sensitive to context overlaps. If it's printing
context that either overlaps or is immediately adjacent to another bit
of context, don't print a separator.

- Non-overlapping segments no longer have two separators between them

- Overlapping segments no longer have separators between them with
overlapping sections repeated

Approved by: emaste (mentor, blanket MFC)

bsdgrep: add -z/--null-data support and update NLS catalogs accordingly

MFC r317049: bsdgrep: add -z/--null-data support

-z treats input and output data as sequences of lines terminated by a
zero byte instead of a newline. This brings it more in line with GNU grep
and brings us closer to passing the current tests with BSD grep.

MFC r317679: bsdgrep: correct nls usage data after r317049

r317049 added -z/--null-data to BSD grep but missed the update to nls
catalogs.

Approved by: emaste (mentor, blanket MFC)
Relnotes: yes

MFC r316542: bsdgrep: Handle special case of single-byte NUL pattern

PR: 202022
Approved by: emaste (mentor, blanket MFC)

MFC r316495: bsdgrep(1): Fix errors with invalid expressions

Invalid expressions with an ultimate compiled pattern length of 0 (e.g.,
"grep -E {") were not taken into account and caused a segfault while trying
to fill in the good suffix table.

Approved by: emaste (mentor, blanket MFC)

MFC r316492: bsdgrep(1): Rip out "xmalloc" bits

xmalloc was a debug malloc implementation, but the x{malloc,calloc,free}
functions default to calling the malloc(3) equivalents.

Instead of relying on this malloc shim, we can devise better ways to debug
malloc issues that aren't misleading upon initial inspection. (I.e., using
jemalloc's various built-in debugging capabilities.)

Approved by: emaste (mentor, blanket MFC)

bsdgrep: Fix matching behavior and add regression tests

MFC r316477: bsdgrep: fix matching behaviour

- Set REG_NOTBOL if we've already matched beginning of line and we're
examining later parts

- For each pattern we examine, apply it to the remaining bits of the
line rather than (potentially) smaller subsets

- Check for REG_NOSUB after we've looked at all patterns initially
matching the line

- Keep track of the last match we made to later determine if we're
simply not matching any longer or if we need to proceed another byte
because we hit a zero-length match

- Match the earliest and longest bit of each line before moving the
beginning of what we match to further in the line, past the end of the
longest match; this generally matches how gnugrep(1) seems to behave,
and seems like pretty good behavior to me

- Finally, bail out of printing any matches if we were set to print all
(empty pattern) but -o (output matches) was set

MFC r316489: bsdgrep: Initialize vars to avoid a false positive GCC warning

MFC r316491: bsdgrep: revert color changes from r316477

r316477 changed the color output to match exactly the in-tree GNU grep,
but introduces unnecessary escape sequences.

MFC r316536: bsdgrep: create additional tests for coverage on recent fixes

Create additional tests to cover regressions that were discovered by
PRs linked to reviews D10098, D10102, and D10104.

It is worth noting that neither bsdgrep(1) nor gnugrep(1) in the base
system currently pass all of these tests, and gnugrep(1) not quite being
up to snuff was also noted in at least one of the PRs.

MFC r317052: bsdgrep: fix zero-length matches without the -o flag

r316477 broke zero-length matches when not using the -o flag, by
skipping over them entirely.

Add a regression test so that it doesn't break again in the future.

PR: 175314, 180990, 181263, 195763, 197531, 197555, 202022, 209116
Approved by: emaste (mentor, blanket MFC)
Relnotes: yes

bsdgrep: treat rgrep as grep -r and install rgrep symlink

MFC r316473: bsdgrep: treat rgrep as grep -r

MFC r316484: bsdgrep(1): create rgrep link

Create a convenience rgrep link for bsdgrep(1) that observes 'grep -r'
behavior.

A follow-up to r316473.

Approved by: emaste (mentor)

MFC r313948: bsdgrep: fix EOF handling with --mmap

Rework part of the loop in grep_fgetln to return the rest of the line
and ensure that we still advance the buffer by the length of the rest
of the line.

PR: 165471
Approved by: emaste (mentor)

MFC r313955 (emaste):
bsdgrep: document ignored option -u

MSDOS and Windows GNU grep uses -u to mean "print byte offsets as if
running on an UNIX system." The option has no effect on systems that
do not use CRLF line endings.

PR: 171200
Sponsored by: The FreeBSD Foundation

MFC r303676:

Fix a segfault in bsdgrep when parsing the invalid extended regexps "?"
or "+" (these are invalid, because there is no preceding operand).

When bsdgrep attempts to emulate GNU grep in discarding and ignoring the
invalid ? or + operators, some later logic in tre_compile_fast() goes
beyond the end of the buffer, leading to a crash.

Fix this by bailing out, and reporting a bad pattern instead.

Approved by: re (gjb, kib)
Reported by: Steve Kargl

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

Fix a bug in bsdgrep that caused the program to hang in a tight loop for
some combinations of command line options and search patterns. The code was
examining regexec flags looking for a regcomp flag value. The fix is to
look in the struct field where the decoded regcomp flag was stored when the
regex was compiled.

With this fix, it's possible to build WITHOUT_GNU_GREP_COMPAT and
WITH_BSDGREP and have a usable GPL-free grep (which of course lacks gnugrep
extensions). It now passes the kyua tests except for one test that requires
the -z/--null-data gnu extension, and one test involving outputting context
lines across multiple files which appears to sometimes output an extra
delimiter line ("--") between matches (a rather obscure failure of a rather
obscure feature, so bsdgrep should be generally usable now).

DIRDEPS_BUILD: Connect MK_TESTS.

Sponsored by: EMC / Isilon Storage Division

Refactor the test/ Makefiles after recent changes to bsd.test.mk (r289158) and
netbsd-tests.test.mk (r289151)

- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison

Bump GCC max-inline-insns-single in libiconv_modules and grep

This is required by our FORTIFY_SOURCE implementation as it
does more inlining. As a rule of thumb, FORTIFY_SOURCE doubles
the number of inlines except that in grep inlining
blows up for some reason.

new depends

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

bsdgrep: fix regression in the -f option since r268799

Caused by an incomplete merge from NetBSD.

PR: 198725
MFC after: 3 days

Create MANLINKS for lzgrep, etc when MK_LZMA_SUPPORT != no

MFC after: 3 days
Sponsored by: EMC / Isilon Storage Division

Fix xz handling for files larger than 32K.

Submitted by: Stefan Ehmann <shoesoft gmx net>
PR: bin/186861
MFC after: 2 weeks

Add a ${CP} alias for copying files in the build.

Some users build FreeBSD as non-root in Perforce workspaces. By default,
Perforce sets files read-only unless they're explicitly being edited.
As a result, the -f argument must be used to cp in order to override the
read-only flag when copying source files to object directories. Bare use of
'cp' should be avoided in the future.

Update all current users of 'cp' in the src tree.

Reviewed by: emaste
MFC after: 1 week
Sponsored by: Spectra Logic

Convert to usr.bin/ to LIBADD
Reduce overlinking

Integrate usr.bin/grep/tests from NetBSD into atf/kyua

Sponsored by: EMC / Isilon Storage Division

bsdgrep: Work-around for segmentation fault.

Fix by David Carlier.

Obtained from: HardenedBSD
PR: 167921
MFC after: 1 month

Fix incremental builds involving non-root users with read-only source files.

Makefiles should not assume that source files can be overwritten. This is the
common case for Perforce source trees.

This is a followup commit to r211243 in the same vein.

MFC after: 1 month
Sponsored by: Spectra Logic
MFSpectraBSD: r1036319 on 2014/01/29, r1046711 on 2014/03/06

- Do not look for more matching lines if -L is specified

Submitted by: eadler (based on)
MFC after: 2 weeks

grep: fix some memory leaks.
Add memory leak fix missing from r268799.

Obtained from: NetBSD

grep: fix some memory leaks.

Bring a couple of changes from NetBSD:

queue.c (CVS Rev. 1.4. 1.5)
Fix memory leaks.
NULL does not need a cast.

grep.c (CVS Rev. 1.6)
Use the more portable getline.

Obtained from: NetBSD
MFC after: 3 days

grep: Fix type.

Obtained from: NetBSD (CVS rev. 1.17)
MFC after: 3 days

Fix a bug in bsdgrep(1) where patterns are not correctly
detected.

Certain criteria must be met for this bug to show up:

* the -w flag is specified, and
* neither -o or --color are specified, and
* the pattern is part of another word in the line, and
* the other word that contains the pattern occurs first

PR: 181973
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Various style(9) fixes and typos in grep, sort and patch.

MFC after: 3 days

Don't test WITHOUT_FOO in program makefiles, test MK_FOO instead.

mdoc: minor paragraph fixes.

grep -i does not work for simple patterns and single byte locales, like
LANG=ru_RU.KOI8-R grep -i <some single KOI8-R letter>
Fix it.

MFC after: 3 days

Adjust CFLAGS to pick up correct regex.h and posix/regex.h. Note this
actually reverts r250860 and r250861.

Reported by: gjb, tinderbox

Revert r246917, as it is causing incorrect behaviour as reported on
freebsd-current.

PR: bin/175213
Approved by: emaste (co-mentor)

Strengthen the check in IS_OUT_OF_BOUNDS to ensure that (j - 1) is a
valid index into the input buffer.

PR: bin/175213
Reviewed by: gabor
Approved by: emaste (co-mentor)
MFC after: 1 week

- Avoid unnecessary strdup()

Submitted by: ache
MFC after: 5 days

Following r226271, allow disabling lzma support with "WITHOUT_LZMA_SUPPORT".
Correct r226271 which should have used WITHOUT_BZIP2_SUPPORT per r166255.

Obtained from: Juniper Networks

Fix a segfault when bsdgrep -i is given an empty pattern string.

PR: bin/172865
Reviewed by: gabor
Approved by: emaste (co-mentor)
MFC after: 1 week

- Fix handling of the case when multiple patterns are specified in a single
command line argument, separated by newlines

PR: bin/173673
Submitted by: ache
MFC after: 1 week

Make bsdgrep behave as gnugrep and as documented: -m should only stop
reading the specific file, not any file.

Tested by: frogs (irc)
Reviewed by: gabor
Approved by: cperciva (implicit)
MFC after: 1 week

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variables into ed.h.
- sbin/camcontrol: Move `verbose' into camcontrol.h and fix shadow warnings.
- usr.bin/calendar: Remove unneeded variables.
- usr.bin/chat: Make `line' local instead of global.
- usr.bin/elfdump: Comment out unneeded function.
- usr.bin/rlogin: Use _Noreturn instead of __dead2.
- usr.bin/tset: Pull `Ospeed' into extern.h.
- usr.sbin/mfiutil: Put global variables in mfiutil.h.
- usr.sbin/pkg: Remove unused `os_corres'.
- usr.sbin/quotaon, usr.sbin/repquota: Remove unused `qfname'.

Remove duplicate line from usage

PR: bin/164139
Submitted by: Yuri Pankov <yuri.pankov@gmail.com>
Approved by: nwhitehorn
MFC after: 3 days

Add missing "static const" to long options table.

This table is only used in this C file and passed to getopt_long(), so
we can safely add static and const to it.

- Match GNU behavior of exit code
- Rename variable that has a different meaning now

PR: bin/162930
Submitted by: Jan Beich <jbeich@tormail.net>
MFC after: 1 week

- Create links to the xz and lzma versions even if BSD grep is not the
default. Nor GNU nor liblzma in base provides such functionality so
it may be useful.

MFC after: 3 days

- Call warnx() instead of errx() if a directory is not readable when using
a recursive search. This is the expected behavior instead of aborting.

PR: bin/162907
Submitted by: Jan Beich <jbeich@tormail.net>
MFC after: 3 days

- Fix behavior of --null to match GNU grep

PR: bin/162906
Submitted by: Jan Beich <jbeich@tormail.net>
MFC after: 3 days

- Fix installation when WITH_BSD_GREP is set to yes

Submitted by: Aleksandr Rybalko <ray@ddteam.net>

- Fix counting of match limit (-m)

Reported by: Nali Toja <nalitoja@gmail.com>
Approved by: delphij (mentor)

- Use getprogname() instead of __progname
- Allow disabling bzip2 support with WITHOUT_BZIP2
- Fix handling patterns that start with a dot
- Remove superfluous semicolon

Approved by: delphij (mentor)

Fix build on i386 and arm.

Tested with: make universe
Pointy hat to: delphij

Update BSD grep to the latest development version. It has some code
backported that was written for the TRE integration project in Google
Summer of Code 2011. This is a temporary solution until the whole
regex library is not replaced so that BSD grep development can continue
and the backported code gets some review and testing. This change only
improves scalability slightly, there is no big performance boost yet
but several minor bugs have been found and fixed.

Approved by: delphij (mentor)
Sposored by: Google Summer of Code 2011
MFC after: 1 week

- Fix exclusion of directories from a recursive search
- Use FTS_SKIP for exclusion instead of custom code

Submitted by: ttsestt@gmail.com
Approved by: re (kib), delphij (mentor)

- Fix handling of environmental variables when they are set to empty string

Submitted by: ttsestt@gmail.com
Approved by: re (kib), delphij (mentor)

- Use REG_NOSUB to bypass submatch counting when not necessary. This may
yield in somewhat better performance in a few cases.

Approved by: delphij (mentor)

- Fix -w behavior
- Make -F and -w work together
- Fix --color to colorize all of the matches

PR: bin/156826
Submitted by: Yuri Pankov <yuri.pankov@gmail.com>
Approved by: delphij (mentor)

Remove redundant assignments to WARNS.

For these directories, WARNS is already implied to be 6.

Build and install a BSD licensed grep.
If WITH_BSD_GREP is not set, it will be 'bsdgrep' and GNUgrep will be
'[ef]grep'. Otherwise, BSD-grep will be the grep family, and GNUgrep
will be 'gnugrep'.

Discussed with: brooks

- Adjust a comment to actual behaviour
- Makefile nit
- Add more CVS/SVN keywords to make it easier to track changes from NetBSD
in case they add further improvements

Approved by: delphij (mentor)
Obtained from: The NetBSD Project

- Simplify the fixed string pattern preprocessing code
- Improve readability

Approved by: delphij (mentor)
Obtained from: The NetBSD Project

- Replace some strcpy()-family functions with memcpy() ones. It has been
discussed earlier that the extra safeness is not required in these
cases and we can avoid the overhead by using the more general
memory copy functions.

Approved by: delphij (mentor)
Obtained from: The NetBSD Project

Add Simplified Chinese messages for BSD grep.

Revert a minor part of revision 211364:

- Imply -h if single file is grepped, this is the GNU behaviour

This is already done by code above the change and have caused a regression
since this instance of code does not check Hflag.

Reported by: davidxu
Pointy hat to: delphij

UTFize my name.

- Refactor file reading code to use pure syscalls and an internal buffer
instead of stdio. This gives BSD grep a very big performance boost,
its speed is now almost comparable to GNU grep.

Submitted by: Dimitry Andric <dimitry@andric.com>
Approved by: delphij (mentor)

- Revert strlcpy() changes to memcpy() because it's more efficient and
former may be safer but in this case it doesn't add extra
safety [1]
- Fix -w option [2]
- Fix handling of GREP_OPTIONS [3]
- Fix --line-buffered
- Make stdin input imply --line-buffered so that tail -f can be piped
to grep [4]
- Imply -h if single file is grepped, this is the GNU behaviour
- Reduce locking overhead to gain some more performance [5]
- Inline some functions to help the compiler better optimize the code
- Use shortcut for empty files [6]

PR: bin/149425 [6]
Prodded by: jilles [1]
Reported by: Alex Kozlov <spam@rm-rf.kiev.ua> [2] [3],
swell.k@gmail.com [2],
poyopoyo@puripuri.plala.or.jp [4]
Submitted by: scf [5],
Shuichi KITAGUCHI <ki@hh.iij4u.or.jp> [6]
Approved by: delphij (mentor)

- Some fixes to Ukranian catalog

Submitted by: avg, Alex Kozlov <spam@rm-rf.kiev.ua>

Fix spelling.

- Add Ukranian catalog

Submitted by: Alex Kozlov <spam@rm-rf.kiev.ua>

- Some minor changes to the messages to increase usefulness of error msgs

Reviewed by: hrs (Japanese catalogs),
pluknet <pluknet at gmail dot com> (Russian catalog)
Approved by: delphij (mentor)

- Add Japanese NLS catalogs

Submitted by: hrs
Approved by: delphij (mentor)

- Use the traditional behaviour for filename and directory name inclusion
and exclusion patterns [1]
- Some improvements on the exiting code, like replacing memcpy with
strlcpy/strcpy

Approved by: delphij (mentor)
Pointed out by: bf [1], des [1]

- Fix -l and -L by really surpressing output and just showing filenames

Submitted by: swell.k@gmail.com
Approved by: delphij (mentor)

- Fix --color behaviour to only output color sequences if stdout is a tty
or if forced mode is specified [1]
- While here, add some alternative names for the options and make then
case-insensitive
- Fix -q and -l behaviour [2]
- Some small changes to make the code easier to review

Submitted by: swell.k@gmail.com [1],
dougb [2]
Approved by: delphij (mentor)

Fix crashes when using grep -R:

- Explicitly pre-zero memory for fts_open parameters.
- Don't test against directory patterns when we are testing direct
leaf of current directory.

While I'm there plug a few of memory leaks.

- Add Russian catalog [1]
- Fix two minor nits in manpage [2]
- style.Makefile(5)

Submitted by: pluknet <pluknet at gmail.com> [1],
Alex Kozlov <spam@rm-rf.kiev.ua> [2]
Reviewed by: delphij

Add BSD grep to the base system and make it our default grep.

Deliverables: Small and clean code (1,4 KSLOC vs GNU's 8,5 KSLOC),
lower memory usage than GNU grep, GNU compatibility,
BSD license.

TODO: Performance is somewhat behind GNU grep but it is only
significant for bigger searches. The reason is complex, the
most important factor is that GNU grep uses lots of
optimizations to improve the speed of the regex library.
First, we need a modern regex library (practically by adopting
TRE), add support for GNU-style non-standard regexes and then
reevalute the performance issues and look for bottlenecks. In
the meantime, for those, who need better performance, it is
possible to build GNU grep by setting WITH_GNU_GREP.

Approved by: delphij (mentor)
Obtained from: OpenBSD (http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/grep/),
freegrep (http://github.com/howardjp/freegrep)
Sponsored by: Google SoC 2008
Portbuild tests run by: kris, pav, erwin
Acknowledgements to: fjoe (as SoC 2008 mentor),
everyone who helped in reviewing and testing