MFC bsdgrep(1) fixes: r320414, r328559, r332805-r332806, r332809, r332832,
r332850-r332852, r332856, r332858, r332876, r333351, r334803,
r334806-r334809, r334821, r334837, r334889, r335188, r351769, r352691

r320414:
Expect :mmap_eof_not_eol to fail

It relies on a jemalloc feature (opt.redzone) no longer available after
r319971.

r328559:
Remove t_grep:mmap_eof_not_eol test

The test was marked as an expected failure in r320414 after r319971's import
of a newer jemalloc removed an essential feature (opt.redzone) for
reproducing the behavior it was testing. Since then, no way has been found
or demonstrated to reliably test the behavior, so remove the test.

r332805:
bsdgrep: Split match processing out of procfile

procfile is getting kind of hairy, and it's not going to get better as we
correct some more bits that assume we process one line at a time.

r332806:
bsdgrep: Clean up procmatches a little bit

r332809:
bsdgrep: Add some TODOs for future work on operating on chunks

r332832:
bsdgrep: Break procmatches down a little bit more

Split the matching and non-matching cases out into their own functions to
reduce future complexity. As the name implies, procmatches will eventually
process more than one match itself in the future.

r332850:
bsdgrep: Some light cleanup

There's no point checking for a bunch of file modes if we're not a
practicing believer of DIR_SKIP or DEV_SKIP.

This also reduces some style violations that were particularly ugly looking
when browsing through.

r332851:
bsdgrep: More trivial cleanup/style cleanup

We can avoid branching for these easily reduced patterns

r332852:
bsdgrep: if chain => switch

This makes some of this a little easier to follow (in my opinion).

r332856:
bsdgrep: Fix --include/--exclude ordering issues

Prior to r332851:
* --exclude always win out over --include
* --exclude-dir always wins out over --include-dir

r332851 broke that behavior, resulting in:
* First of --exclude, --include wins
* First of --exclude-dir, --include-dir wins

As it turns out, both behaviors are wrong by modern grep standards- the
latest rule wins. e.g.:

`grep --exclude foo --include foo 'thing' foo`
foo is included

`grep --include foo --exclude foo 'thing' foo`
foo is excluded

As tested with GNU grep 3.1.

This commit makes bsdgrep follow this behavior.

r332858:
bsdgrep: Use grep_strdup instead of grep_malloc+strcpy

r332876:
bsdgrep: Fix build failure WITHOUT_LZMA (incorrect bracket placement)

r333351:
bsdgrep: Allow "-" to be passed to -f to mean "standard input"

A version of this patch was originally sent to me by se@, matching behavior
from newer versions of GNU grep.

While there have been some differences of opinion on whether stdin should be
closed or not after depleting it in process of -f, I've opted to leave stdin
open and just let the later matching stuff fail and result in a no-match.
I'm not married to the current behavior- it was generally chosen since we
are adopting this in particular from GNU grep, and I would like to stay
consistent without a strong argument to the contrary. The current behavior
isn't technically wrong, it's just fairly unfriendly to the developer-user
of grep that may not realize their usage is trivially invalid.

r334803:
netbsd-tests: grep(1): Add test for -c flag

Someone might be inclined to accidentally break this. someone might have
written said test because they broke it locally.

r334806:
bsdgrep(1): Do some less dirty things with return types

Neither procfile nor grep_tree return anything meaningful to their callers.
None of the callers actually care about how many lines were matched in all
of the files they processed; it's all about "did anything match?"

This is generally just a light refactoring to remind me of what actually
matters as I'm rewriting these bits to care less about 'stuff'.

r334807:
bsdgrep(1): whoops, garbage collect the now write-only variable

r334808:
bsdgrep(1): Don't initialize fts_flags twice

Admittedly, this is a clang-scan complaint... but it wasn't wrong. fts_flags
is initialized by all cases in the switch(), which should be fairly obvious.
Annotate this anyways.

r334809:
netbsd-tests: bsdgrep(1): Add a test for -m, too

r334821:
bsdgrep(1): Slooowly peel away the chunky onion

(or peel off the band-aid, whatever floats your boat)

This addresses two separate issues:

1.) Nothing within bsdgrep actually knew whether it cared about line numbers
or not.

2.) The file layer knew nothing about the context in which it was being
called.

#1 is only important when we're *not* processing line-by-line. #2 is
debatably a good idea; the parsing context is only handy because that's
where we store current offset information and, as of this commit, whether or
not it needs to be line-aware.

r334837:
bsdgrep(1): Evict character sequence that moved in

r334889:
bsdgrep(1): Some more int -> bool conversions and name changes

Again motivated by upcoming work to rewrite a bunch of this- single-letter
variable names and slightly misleading variable names ("lastmatches" to
indicate that the last matched) are not helpful.

r335188:
bsdgrep(1): Remove redundant initialization; unconditionally assigned later

r351769:
bsdgrep(1): add some basic tests for some GNU Extension support

These will be expanded later as I come up with good test cases; for now,
these seem to be enough to trigger bugs in base gnugrep and expose missing
features in bsdgrep.

r352691:
bsdgrep(1): various fixes of empty pattern/exit code/-c behavior

When an empty pattern is encountered in the pattern list, I had previously
broken bsdgrep to count that as a "match all" and ignore any other patterns
in the list. This commit rectifies that mistake, among others:

- The -v flag semantics were not quite right; lines matched should have been
counted differently based on whether the -v flag was set or not. procline
now definitively returns whether it's matched or not, and interpreting
that result has been kicked up a level.
- Empty patterns with the -x flag was broken similarly to empty patterns
with the -w flag. The former is a whole-line match and should be more
strict, only matching blank lines. No -x and no -w will will match the
empty string at the beginning of each line.
- The exit code with -L was broken, w.r.t. modern grep. Modern grap will
exit(0) if any file that didn't match was output, so our interpretation
was simply backwards. The new interpretation makes sense to me.

Tests updated and added to try and catch some of this.

This misbehavior was found by autoconf while fixing ports found in PR 229925
expecting either a more sane or a more GNU-like sed.

bsdgrep: add a primitive literal matcher to unbreak fgrep in some scenarios

MFC r322825: bsdgrep: add some additional tests for fgrep

Previously added tests only check that fgrep is somewhat sane and works. Add
some more tests that check that the implementation is basically functional
and not producing incorrect results with various flags.

MFC r322826: bsdgrep: add a primitive literal matcher

fgrep/grep -F will error out at runtime if compiled with a regex(3)
that does not define REG_NOSPEC or REG_LITERAL. glibc is one such regex(3)
implementation, and as it turns out they don't support literal matching at
all.

Provide a primitive literal matcher for use with glibc and other
implementations that don't support literal matching so that we don't
completely lose fgrep/grep -F if compiled against libgnuregex on stable/10,
stable/11, or other systems that we don't necessarily support.

This is a wholly unoptimized implementation with no plans to optimize it as
of now. This is due to both its use-case being primarily on unsupported
systems in the near-distant future and that it's reinventing the wheel that
we already have available as a feature of regex(3).

PR: 222201
Approved by: emaste (mentor, blanket MFC)

MFC r321450: bsdgrep(1): Don't exit before processing every file

Given an empty pattern (i.e. grep "" A B), bsdgrep(1) would previously
exit() with the appropriate exit code upon encountering an empty file.
Likely intended as an optimization, but this behavior is technically
incorrect since an empty pattern should match every line.

PR: 220924
Approved by: emaste (mentor, blanket MFC)

MFC r318574: bsdgrep: Correct per-line line metadata printing

Metadata printing with -b, -H, or -n flags suffered from a few flaws:

1) -b/offset printing was broken when used in conjunction with -o

2) With -o, bsdgrep did not print metadata for every match/line, just
the first match of a line

3) There were no tests for this

Address these issues by outputting this data per-match if the -o flag is
specified, and prior to outputting any matches if -o but not --color,
since --color alone will not generate a new line of output for every
iteration over the matches.

To correct -b output, fudge the line offset as we're printing matches.

While here, make sure we're using grep_printline in -A context. Context
printing should *never* look at the parsing context, just the line.

The tests included do not pass with gnugrep in base due to it exhibiting
similar quirky behavior that bsdgrep previously exhibited.

Approved by: emaste (mentor, blanket MFC)

MFC r318571: bsdgrep: emit more than MAX_LINE_MATCHES per line

We should not set an arbitrary cap on the number of matches on a line,
and in any case MAX_LINE_MATCHES of 32 is much too low. Instead, if we
match more than MAX_LINE_MATCHES, keep processing and matching from the
last match until all are found.

For the regression test, we produce 4096 matches (larger than we expect
we'll ever set MAX_LINE_MATCHES) and make sure we actually get 4096
lines of output with the -o flag.

We'll also make sure that every distinct line is getting its own line
number to detect line metadata not being printed as appropriate along
the way.

PR: 218811
Approved by: emaste (mentor, blanket MFC)

bsdgrep: fix segfault with --mmap and add relevant test

MFC r318565: bsdgrep: fix segfault with --mmap

r313948 partially fixed --mmap behavior but was incomplete. This commit
generally reverts it and does it the more correct way- by just consuming
the rest of the buffer and moving on.

MFC r318908: bsdgrep: add --mmap tests

Basic sanity tests as well as coverage for the bug fixed in r318565.

PR: 219402
Approved by: emaste (mentor, blanket MFC)

bsdgrep: Don't allow negative context flags, add more tests

MFC r318302: bsdgrep: don't allow negative -A / -B / -C

Previously, when given a negative -A/-B/-C argument bsdgrep would
overflow the respective context flag(s) and exhibited surprising
behavior.

Fix this by removing unsignedness of Aflag/Bflag and erroring out if
we're given a value < 0. Also adjust the type used to track 'tail'
context in procfile() so that it accurately reflects the Aflag value
rather than overflowing and losing trailing context.

This also fixes an inconsistency previously existing between -n and
-C "n" behavior. They are now both limited to LLONG_MAX, to be
consistent.

Add some test cases to make sure grep errors out properly for both
negative context values as well as non-numeric context values rather
than giving bogus matches.

MFC r318317: bsdgrep: add more tests for different binary flags

The existing 'binary' test in netbsd-tests/ does a basic check of the
default treatment for binary behavior, but not much more than that.
Given some opportunity for breakage recently that did not trigger any
failures, add some tests to cover the three different binary file
behaviors (a, -I, -U) and their --binary-files= equivalent values.

Approved by: emaste (mentor, blanket MFC)

MFC r318004 (ngie): Remove expected failure that no longer fails with gnu
grep in base

Approved by: emaste (mentor, blanket MFC)

bsdgrep: fix -w flag matching with an empty pattern

MFC r317703: bsdgrep: fix -w flag matching with an empty pattern

-w flag matching with an empty pattern was generally 'broken', allowing
matches to occur on any line whether or not it actually matches -w
criteria.

This fix required a good amount of refactoring to address. procline()
is altered to *only* process the line and return whether it was a match
or not, necessary to be able to short-circuit the whole function in case
of this matchall flag. -m flag handling is moved out as well because it
suffers from the same fate as context handling if we bypass any actual
pattern matching.

The matching context (matches, mostly) didn't previously exist outside
of procline(), so we go ahead and create context object for file
processing bits to pass around. grep_printline() was created due to
this, for the scenarios where the matches don't actually matter and we
just want to print a line or two, a la flushing the context queue and
no -o or --color specified.

Damage from this broken behavior would have been mitigated by the fact
that it is unlikely users would invoke grep -w with an empty pattern.

This was identified while checking PR 105221 for problems it this may
cause in BSD grep, but PR 105221 is *not* a report of this behavior.

MFC r317741: bsdgrep: correct uninitialized variable introduced in r317703

MFC r317842: bsdgrep: don't ouptut matches with -c, -l, -L

Refactoring done in r317703 broke -c, -l, and -L flags implying
suppression of match printing. Fortunately this is just a matter of not
doing any printing of the resulting matches and context printing was not
broken in this refactoring.

Add some regression tests since this area may still see further
refactoring, include different context flags as well even though they
were not broken in this case.

PR: 219077
Approved by: emaste (mentor, blanket MFC)

MFC r317665: bsdgrep: fix -w -v matching improperly with certain patterns

-w and -v flag matching was mostly functional but had some minor
problems:

1. -w flag processing only allowed one iteration through pattern
matching on a line. This was problematic if one pattern could match
more than once, or if there were multiple patterns and the earliest/
longest match was not the most ideal, and

2. Previous work "fixed" things to not further process a line if the
first iteration through patterns produced no matches. This is clearly
wrong if we're dealing with the more restrictive -w matching.

#2 breakage could have also occurred before recent broad rewrites, but
it would be more arbitrary based on input patterns as to whether or not
it actually affected things.

Fix both of these by forcing a retry of the patterns after advancing
just past the start of the first match if we're doing more restrictive
-w matching and we didn't get any hits to start with. Also move -v flag
processing outside of the loop so that we have a greater change to match
in the more restrictive cases. This wasn't strictly wrong, but it could
be a little more error prone.

While here, introduce some regressions tests for this behavior and fix
some excessive wrapping nearby that hindered readability. GNU grep
passes these new tests.

PR: 218467, 218811
Approved by: emaste (mentor, blanket MFC)

bsdgrep: Revise tests based on recent fixes and future changes

MFC r317297: Remove the expected failures for :context and :context2 with
bsdgrep(1)

They're no longer needed after recent fixes made to bsdgrep(1).

MFC r317299: Add more sanity tests for grep, egrep, and fgrep

The test suite currently lacks basic sanity checks to ensure that egrep,
fgrep, and grep are actually matching the right expression types, i.e.
passing the right flags to regcomp(3). Amend the test suite to make sure
that not only are the individual versions doing the right thing, but also
that we don't have some kind of frankenregex situation happening where egrep
is accepting a BRE or grep an ERE.

I've chosen to not expand the 'basic' test but to add the 'grep_sanity'
checks to their own test case since this is testing for more than just
'grep matches things', but actual expression types.

MFC r317694: bsdgrep: revise test case which will soon become a failure

Work in progress (D10315) is going to make egrep_empty_invalid an
actually invalid regex, to be consistent with the equivalent BRE "{"
behavior, when using regex(3).

Any non-0 exit value is acceptable, depending on how the installed grep
interprets the expression. GNU grep interprets it as non-matching, and
in the future BSD grep will interpret it is an error.

Approved by: emaste (mentor, blanket MFC)

MFC r316750 (ngie): Fix expectations for testcases per bsdgrep vs gnu grep

The following failures occur with various versions of grep:

BSD grep:
- :context
- :context2

GNU grep (base):
- :color
- :oflag_zerolen

GNU grep (ports):
- :recurse_symlink

Approved by: emaste (mentor, blanket MFC)

bsdgrep: Fix matching behavior and add regression tests

MFC r316477: bsdgrep: fix matching behaviour

- Set REG_NOTBOL if we've already matched beginning of line and we're
examining later parts

- For each pattern we examine, apply it to the remaining bits of the
line rather than (potentially) smaller subsets

- Check for REG_NOSUB after we've looked at all patterns initially
matching the line

- Keep track of the last match we made to later determine if we're
simply not matching any longer or if we need to proceed another byte
because we hit a zero-length match

- Match the earliest and longest bit of each line before moving the
beginning of what we match to further in the line, past the end of the
longest match; this generally matches how gnugrep(1) seems to behave,
and seems like pretty good behavior to me

- Finally, bail out of printing any matches if we were set to print all
(empty pattern) but -o (output matches) was set

MFC r316489: bsdgrep: Initialize vars to avoid a false positive GCC warning

MFC r316491: bsdgrep: revert color changes from r316477

r316477 changed the color output to match exactly the in-tree GNU grep,
but introduces unnecessary escape sequences.

MFC r316536: bsdgrep: create additional tests for coverage on recent fixes

Create additional tests to cover regressions that were discovered by
PRs linked to reviews D10098, D10102, and D10104.

It is worth noting that neither bsdgrep(1) nor gnugrep(1) in the base
system currently pass all of these tests, and gnugrep(1) not quite being
up to snuff was also noted in at least one of the PRs.

MFC r317052: bsdgrep: fix zero-length matches without the -o flag

r316477 broke zero-length matches when not using the -o flag, by
skipping over them entirely.

Add a regression test so that it doesn't break again in the future.

PR: 175314, 180990, 181263, 195763, 197531, 197555, 202022, 209116
Approved by: emaste (mentor, blanket MFC)
Relnotes: yes

MFC r313439,r314450:

r313439:

Merge content from ^/projects/netbsd-tests-upstream-01-2017 into ^/head

The primary end-goal of this drop is ease future merges with NetBSD and
collaborate further with the NetBSD project.

The goal was (largely, not completely as some items are still oustanding
in the NetBSD GNATS system) achieved by doing the following:
- Pushing as many changes required to port contrib/netbsd-tests
back to NetBSD as possible, then pull the upstream applied changes
back in to FreeBSD.
- Diff reduce with upstream where possible by:
-- Improving libnetbsd header, etc compat glue.
-- Using _SED variables to modify test scripts on the fly for items
that could not be upstreamed to NetBSD.

As a bonus for this work, this change also introduces testcases for
uniq(1).

Many thanks to Christos for working with me to get many of the changes
back into the NetBSD project.

In collaboration with: Christos Zoulas <christos@netbsd.org>

r314450:

Add additional __FreeBSD_version guards around the hsearch_r testcases

The reasoning for this is the same as r276046: to ease MFCing the tests
to ^/stable/10 .

This was accidentally missed in r313439

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix grep_test:recurse with ZFS and TMPFS tmpdirs

contrib/netbsd-tests/usr.bin/grep/t_grep.sh
Fix grep_test:recurse when /tmp is either zfs or tmpfs. The test was
relying on an implicit ordering of directory recursion which happens
to be true when using UFS. grep's specification requires no such
ordering. The solution is to ignore the order of grep's results.

Reviewed by: ngie
MFC after: 32 days
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D4925

Use stable output to a test file instead of depending on the OS name being
grep'able in /bin/sh

This fixes the situation where the OS has been rebranded to something other
than `FreeBSD`

MFC after: 1 week
Obtained from: Isilon OneFS (^/onefs/head@r511419)
Reviewed by: cem, Daniel O'Connor <darius@dons.net.au>
Sponsored by: EMC / Isilon Storage Division

Import the NetBSD test suite from ^/vendor/NetBSD/tests/09.30.2014_20.45 ,
minus the vendor Makefiles

Provide directions for how to bootstrap the vendor sources in
FREEBSD-upgrade

MFC after 2 weeks
Discussed with: rpaulo
Sponsored by: EMC / Isilon Storage Division

Tag ^/base/vendor/NetBSD/tests/09.30.2014_20.45 from
^/base/vendor/NetBSD/tests/dist

Sponsored by: EMC / Isilon Storage Division

Check in first src/tests snapshot from NetBSD anoncvs

Sources were obtained like so:

% export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
% cvs -z9 co -D "09/30/2014 20:45" -P src/tests
% mv src/tests/* tests/dist/.

'*CVS*' has been added to svn:ignore to ease updating periodically from
upstream

Some line ending issues had to be resolved with test outputs and scripts
via dos2unix and by deleting the eol-style property set in usr.bin/sort

Discussed with: rpaulo
Sponsored by: EMC / Isilon Storage Division

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix grep_test:recurse with ZFS and TMPFS tmpdirs

contrib/netbsd-tests/usr.bin/grep/t_grep.sh
Fix grep_test:recurse when /tmp is either zfs or tmpfs. The test was
relying on an implicit ordering of directory recursion which happens
to be true when using UFS. grep's specification requires no such
ordering. The solution is to ignore the order of grep's results.

Reviewed by: ngie
MFC after: 32 days
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D4925

Use stable output to a test file instead of depending on the OS name being
grep'able in /bin/sh

This fixes the situation where the OS has been rebranded to something other
than `FreeBSD`

MFC after: 1 week
Obtained from: Isilon OneFS (^/onefs/head@r511419)
Reviewed by: cem, Daniel O'Connor <darius@dons.net.au>
Sponsored by: EMC / Isilon Storage Division

Import the NetBSD test suite from ^/vendor/NetBSD/tests/09.30.2014_20.45 ,
minus the vendor Makefiles

Provide directions for how to bootstrap the vendor sources in
FREEBSD-upgrade

MFC after 2 weeks
Discussed with: rpaulo
Sponsored by: EMC / Isilon Storage Division

Tag ^/base/vendor/NetBSD/tests/09.30.2014_20.45 from
^/base/vendor/NetBSD/tests/dist

Sponsored by: EMC / Isilon Storage Division

Check in first src/tests snapshot from NetBSD anoncvs

Sources were obtained like so:

% export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
% cvs -z9 co -D "09/30/2014 20:45" -P src/tests
% mv src/tests/* tests/dist/.

'*CVS*' has been added to svn:ignore to ease updating periodically from
upstream

Some line ending issues had to be resolved with test outputs and scripts
via dos2unix and by deleting the eol-style property set in usr.bin/sort

Discussed with: rpaulo
Sponsored by: EMC / Isilon Storage Division