newsyslog(8): Implement a new 'E' flag to not rotate empty log files

Based on an idea from dvl's coworker, László DANIELISZ, implement
a new flag, 'E', that prevents newsyslog(8) from rotating the empty
log files. This 'E' flag ist mostly usable in conjunction with 'B'
flag that instructs newsyslog(8) to not insert an informational
message into the log file after rotation, keeping it still empty.

Reviewed by: markj, ian, manpages (rpokala)
Approved by: markj, ian, manpages (rpokala)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D28940

(cherry picked from commit c7d27b225df8d7fb36a31a21737d4309593c4604)

Git Hash: 7e863cb9893b96f4ca6e0597572e3df218dd3830
Git Author: otis@FreeBSD.org

MFC r362233: newsyslog(8): make configuration parser more robust.

MFC r344470:

Fix several Coverity-detected issues in newsyslog.

- CID 1394815, CID 1305673: Dereference before null check - memory was
allocated and the allocation checked for NULL with a call to errx()
if it failed. Code below that was guaranteed that the pointer was
non-NULL, but there was another check for NULL at the exit of the
function (after the memory had already been referenced). Eliminate
the useless NULL check.

- CID 1007452: Resource leak - Storage intended to be allocated and
returned to the caller was never freed. This was the result of a
regression in the function signature introduced in r208648 (2010)
(thanks for that find, @cem!). Fixed by altering the function
signature and passing the allocated memory to the caller as
intended. This also fixes PR158794.

- CID 1008620: Logically dead code in newsyslog.c - This was a direct
result of CID 1007452. Since the memory allocated as described there
was not returned to the caller, a subsequent check for the memory
having been allocated was dead code. Returning the memory
re-animates the code that is the subject of this CID.

- CID 1006131: Unused value - in parsing a configuration file, a
pointer to the end of the last field was saved, but not used after
that. Rewrite to use the pointer value. This could have been fixed
by avoiding the assignment altogether, but this solutions more
closely follows the pattern used in the preceding code.

PR: 158794
Reported by: Coverity, Ken-ichi EZURA <k.ezura@gmail.com> (PR158794)

Sponsored by: Dell EMC Isilon

MFC r336086:

Use the FQDN in the newsyslog log message when RFC 5424 is enabled.

The RFC 5424 spec mentions that logging FQDNs over short hostnames is
preferred. Alter this code, so that the hostname doesn't get truncated
on startup. Keep track of the length of the short hostname, so that
fprintf() can do the truncation where necessary.

MFC r337323:
Fix a flag collision introduced in r327451.

PR: 230350

MFC r326276:

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

MFC r327451:

newsyslog: implement 'p' flag

Implement the 'p' flag for newsyslog from NetBSD. This flag results in
the first log file for a given file to not be compressed.

While here, don't change file attributes during a no-op run

PR: 162798

MFC 328035: Sort the list of flags in newsyslog.conf entries.

Move the 'X' entry into its sorted location in the list
of flags just above 'Z'.

MFC r318960,r319545,r319546,r319548,r321261:

r318960 (by dab):

Add newsyslog capability to write RFC5424 compliant rotation message.

This modification adds the capability to newsyslog to write the
rotation message in a format that is compliant with RFC5424. This
capability is enabled on a per-log file basis through a new value
("T") in the flags field in newsyslog.conf. This is useful on systems
that use the RFC5424 format for log files so that the rotation message
format matches that of the other log messages. There has been recent
mention of adding an RFC5424 compliant mode to syslogd and at least
one alternative system log daemon (rsyslogd) that already has the
capability to use that format.

Relnotes: yes

r319545:

Don't execute the TODO cases in a subshell

This messes up the testcase counter, as seen in bug 219756.

PR: 212160, 219756

r319546:

Fix the testplan after ^/head@r318960

The number of executed testcases is 128, not 126.

MFC with: r318960

r319548:

Remove TODO for sub testcases added for bug 212160

On closer inspection, the past failures no longer occur on ^/head.

PR: 212160

r321261:

Clean up leading whitespace (convert single column spaces to hard tabs)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

newsyslog: Eliminate unnecessary sleep(10) when -R and -s are specified

After going through the signal work list, during which do_sigwork()
is called and essentially does nothing because -s and -R were
specified on the command line, newsyslog will sleep for 10 seconds
as the (verbose) code says: "Pause 10 seconds to allow daemon(s)
to close log file(s)".

However, the man page verbiage for -R (and -s) seems quite clear
that this sleep() is unnecessary because the daemon was expected
to have already closed the log file before calling newsyslog.

PR: 210020
Submitted by: David A. Bright <david_a_bright@dell.com>
MFC after: 1 week
Sponsored by: Dell Inc.
Differential Revision: https://reviews.freebsd.org/D6727

Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

DIRDEPS_BUILD: Connect MK_TESTS.

Sponsored by: EMC / Isilon Storage Division

newsyslog: treat 'c' flag in the config as 'C'

When -C was introduced in r114137 the plan was to have -C and -c being used for
"create" due to a typo in FreeBSD <= 4.8 a temporary compatibility hack has been
added to make -c being like -G aka GLOB and a warning was issued for the user to
be aware of the futur change for -c.

12 years later it is more than time to remove that hack and finish the what was
intent in r114137

Submitted by: Alexandre Perrin <alex@kaworu.ch>
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D4000

newsyslog.conf: allow to configure the signal using the signal name.

Submitted by: Alexandre Perrin <alex@kaworu.ch>
MFC after: 1 week
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D3961

Fix a ton of speelling errors

arc lint is helpful

Reviewed By: allanjude, wblock, #manpages, chris@bsdjunk.com
Differential Revision: https://reviews.freebsd.org/D3337

Refactor the test/ Makefiles after recent changes to bsd.test.mk (r289158) and
netbsd-tests.test.mk (r289151)

- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

If we fail to send a signal after rotation, print the pidfile from which
the corresponding PID was obtained.

PR: 194143
Submitted by: Lyndon Nerenberg <lyndon@orthanc.ca>
MFC after: 1 week

Make it more explicitly clear that -t will not change filename.

MFC after: 3 days

- Include /etc/newsyslog.conf.d/* and /usr/local/etc/newsyslog.conf.d/* by
default for newsyslog(8).

The /usr/local/etc/newsyslog.conf.d will give packages an opportunity to
install a default configuration to handle their own log files.

MFC after: 2 weeks
Relnotes: yes

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Migrate tools/regression/{usr.bin/lastcomm,usr.sbin}/ to the new tests layout.

This change was originally going to only migrate the usr.sbin tests but, as
it turns out, the usr.sbin/sa/ tests require files from usr.bin/lastcomm/
so it's better to just also migrate the latter at the same time. The other
usr.bin tests will be moved separately.

To make these tests work within the test suite, some of them have required
changes to prevent modifying the source directory and instead just rely on
the current directory for file manipulation.

Fix newsyslog(8) to use the size of the file instead of the blocks it
takes on disk, as advertised in newsyslog.conf(5).

This fixes newsyslog(8) on ZFS with compression enabled to not have
large files compared to the expected rotation size.

# grep remotes/messages /etc/newsyslog.conf
/var/log/remote/messages root:info 640 5 500 * JC
# ls -alh /var/log/remote/messages
-rw-r----- 1 root info 3.2M Jan 31 20:02 /var/log/remote/messages
# newsyslog -vN|grep remote/messages
/var/log/remote/messages <5J>: size (Kb): 464 [500] --> skipping
# stat -f "st_size: %z st_blocks: %b" /var/log/remote/messages
st_size: 3372627 st_blocks: 928
# zfs get -H compressratio zroot/syslogs
zroot/syslogs compressratio 3.77x -

With fix:

# newsyslog -v | grep remote/messages
/var/log/remote/messages <5J>: size (Kb): 3338 [500] --> trimming log....

Approved by: bapt (mentor)
PR: docs/150877
Reported by: Joshua Isom <jrisom@gmail.com>
MFC after: 2 weeks

Initialize the struct tm before handing it to strptime(3).

PR: 183550
MFC after: 2 weeks

Fix -Wunsequenced

Submitted by: dt71@gmx.com

We want to stat the archived log file rather than the logfile itself.

PR: bin/179122
Submitted by: Oliver Fromme <olli@secnetix.de>
MFC after: 3 days

Some filesystems (NFS in particular) do not fill out the d_type field when
returning directory entries through readdir(3). In this case we need to
obtain the file type ourselves; otherwise newsyslog -t will not be able to
find archived log files and will fail to both delete old log files and to
do interval-based rotations properly.

Reported by: jilles
Reviewed by: jilles
MFC after: 2 weeks

Fix interval-based rotations when the -t flag is used. In this case, find
the most-recently archived logfile and use its mtime to determine whether
or not to rotate, as in the non-timestamped case.

Previously we would just try to use the mtime of <logfile>.0, which always
results in a rotation since it generally doesn't exist in the -t case.

PR: bin/166448
Approved by: emaste (co-mentor)
Tested by: Marco Steinbach <coco executive-computing.de>
MFC after: 2 weeks

Rename the run_cmd field to sw_runcmd to make it consistent with the
other fields in struct sigwork_entry.

Approved by: rstone (co-mentor)
MFC after: 1 week

Ensure that newsyslog -n prints the correct message for a rotation rule
that uses the 'R' flag.

Reviewed by: gad
Approved by: rstone (co-mentor)
MFC after: 1 week

When the 'R' flag is used with a newsyslog.conf entry, some fields of
the corresponding struct sigwork_entry were left uninitialized,
potentially causing an early return from do_sigwork(). Ensure that these
fields are initialized, and handle the 'R' flag properly in
do_sigwork().

PR: bin/175330
Reviewed by: gad
Approved by: rstone (co-mentor)
MFC after: 1 week

Make sure to update the mtime of a logfile after archiving it. This
ensures that the next rotation happens at the correct time when using
interval-based rotations.

PR: bin/174438
Reviewed by: gad
Approved by: rstone (co-mentor)
MFC after: 1 week

Have -n imply -r, since dry-run mode obviously doesn't require root
privileges.

Approved by: rstone (co-mentor)
MFC after: 1 week

Fix a typo in an error message.

Approved by: rstone (co-mentor)
MFC after: 1 week

More -Wmissing-variable-declarations fixes.

In addition to adding missing `static' keywords:
- bin/dd: Pull in `extern.h' to guarantee consistency with source file.
- libexec/rpc.rusersd: Move shared globals into an extern.h.
- libexec/talkd: Move `debug' and `hostname' into extern.h.
- usr.bin/cksum: Put counters in extern.h, as they are used by ckdist/mtree.
- usr.bin/m4: Move `end_result' into extern.h.
- usr.sbin/services_mkdb: Move shared globals into an extern.h.

We don't need to check the result of sending signal when -R option is
specified.

Submitted by: Ilya A. Arkhipov
MFC after: 1 week

mdoc: fix mandoc "Oc breaks Op" warning.

Minor spelling fixes.

Hide DIR definition by making it an opaque struct typedef.

Introduce dirfd() libc exported symbol replacing macro with same name,
preserve _dirfd() macro for internal use.

Replace dirp->dd_fd with dirfd() call. Avoid using dirfd as variable
name to prevent shadowing global symbol.

Sponsored by: Google Summer Of Code 2011

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

Fix a sentence in a paragraph that describes time and interval based
trimming. This sentence vaguely can be interpreted as if it was speaking
about time and size interaction, while it wasn't about it.

Don't run through time checks when entry is definitely oversized. This
leads to newsyslog rotating on (size OR time) if both are specified.

PR: 100018, 160432

Further fix a typo and spelling classic correctly in function names,
too.

Submitted by: Ben Kaduk <minimarmot@gmail.com>

Replace index() and rindex() calls with strchr() and strrchr().

The index() and rindex() functions were marked LEGACY in the 2001
revision of POSIX and were subsequently removed from the 2008 revision.
The strchr() and strrchr() functions are part of the C standard.

This makes the source code a lot more consistent, as most of these C
files also call into other str*() routines. In fact, about a dozen
already perform strchr() calls.

Spelling fixes for usr.sbin/

- Remove extraneous null ptr deref checks
- Fix memory leak

Submitted by: Slono Slono <slonoman2011@yandex.ru>
Approved by: jhb
MFC after: 1 week

Add new modifier - "R", when it is specified the path to pid file
will be considered as a path to a binary or a shell script to be executed
after rotation has been completed instead of sending signal to the process
id in that file.

Sponsored by: Sippy Software, Inc.
From the: FreeBSD hacking lounge at BSDCan

Fix an old bug in newsyslog where we kept one log file more than was
requested in newsyslog.conf. This was only the case using the non-time
based filenames (.0, .1, .2 etc.).

The change also makes newsyslog clean clean up the old extra logfile so
users don't end up with a single stale logfile which won't be rotated
out.

This change also cleans up some code a bit to avoid more copy / paste
code and removes some old copy / paste code in the process.

PR: bin/76697
MFC after: 2 weeks

Add an example for the use of the <include> entry to help others
who are as slow as I am.

Discussed with: gordon

Sync manpage's SYNOPSIS with program's usage.

Make code more friendly to the non-C99 compilers - don't allocate
local variables in the `for' loop declaration. This allows trunk
newsyslog.c to be compiled on 7.x. This change should be no-op from
the functional POV.

Use correct alphabetical order for xz(1) in SEE ALSO section
of newsyslog manpages.

Reported by: maxim
MFC after: 3 weeks

Add xz(1) support to newsyslog.
Rewrite and simplify logfile compression code.

Approved by: gad
MFC after: 3 weeks

Make -S functional.

MFC after: 1 week

Fix a typo in a comment.

MFC after: 3 days

Add a -S switch to override the default syslog pid file. This can be useful
if syslogd's -P switch or a syslogd alternative is being used.

MFC after: 3 weeks

Add support for creating the archived log filenames using a time-stamp
instead of the traditional simple counter.

Using the time-stamp based file-names, once a log file is archived, it
will not change name until it is deleted. This means that many backup
systems will only perform one backup of the archived log file, instead
for performing a new backup of the logfile upon each logfile rotation.

This implementation is separate from the patches in the mentioned PR,
as I wasn't aware of the existence of the PR until after I had
implemented the same functionality as the patches in the PR provide.
Unlike the PR, this new code does honor the 'log count' in
newsyslog.conf so old logfiles are deleted. This new code does not
currently support never deleting the archived logfiles.

PR: bin/29363
MFC after: 3 weeks

Add file include processing for newsyslog.

Format for the include line in /etc/newsyslog.conf is:
<include> /etc/defaults/newsyslog.conf

Other notes of interest:
Globbing is supported in <include> statements.
Properly detect circular include loop dependencies.

Reviewed by: gad@
Approved by: wes@ (mentor)
MFC after: 2 months

Convert newsyslog to using queue(3) macros instead of a home rolled version.

Reviewed by: gad@
Approved by: wes@ (mentor)
MFC after: 2 months

mdoc: move remaining sections into consistent order

This pertains mostly to FILES, HISTORY, EXIT STATUS and AUTHORS sections.

Found by: mdocml lint run
Reviewed by: ru

Add a new option, -P, which reverts newsyslog(8) to the old behavior,
which stops to proceed further, as it is possible that processes which
fails to create PID file get screwed by rotation.

Requested by: stas
MFC after: 2 weeks
X-MFC with: r200806

The last big commit: let usr.sbin/ use WARNS=6 by default.

Don't consider non-existence of a PID file an error, we should be able
to proceed anyway as this most likely mean that the process has been
terminated.

PR: bin/140397
Submitted by: Dan Lukes <dan obluda cz>
MFC after: 1 month

Fix 6-year old cut&paste error. The # could be escaped with '\', not
with '\\'.

MFC after: 2 weeks

Fix two typos.

s/effect/affect/

PR: docs/118266
Submitted by: "Kim Scarborough" <sluggo at unknown dot nu>
MFC After: 3 days

Markup fixes.

Use same signedness for i and matched_c. This should make
newsyslog(8) WARNS=6 clean.

Remove mention of the `W' flag, which has been turned to no-op by the
neworder change. Keep the option in a config file parser, to not
violate POLA.

MFC after: 2 weeks

Remove the last three calls to strncpy(), two of which would have
been bugs if the source had ever been too big for the destination.

MFC after: 3 weeks

Remove the "oldorder" processing. The "neworder" processing has
been the default for two years now, without any problems reported.

MFC after: 3 weeks

Improve error-handling related to the fork() done to compress files after
they have been rotated. Among other things, use warnx() instead of warn()
for some messages where the value if errno is irrelevant to the problem
being reported.

MFC after: 5 days

Fix logic error which causes <null> to be printed instead of the
actual file name in error message.

MFC After: 2 weeks

Markup fixes.

Approved by: re (blanket)

Add the -N option to not rotate any files. This is to be used in
cojunction with -C and is used by /etc/rc.d/newsyslog.

I forgot that this was in my perforce tree and not my running system and
thus committed a non-working newsyslog script.

Reported by: des
Pointy hat: brooks

Add a HISTORY section.

PR: 75282

Expand *n't contractions.

Fixed xrefs.

Sort sections.

Because the `permission' field in conf_entry is intended to be used as
parameter 2 in chmod(2), which is a mode_t (and in turn a __uint_16_t),
it's more likely that it should be defined as an unsigned variable.

This commit should make newsyslog WARNS=6 clean, but don't bump the knob
until I have a universe build.

MFC After: 1 month

Mention "-d directory" in usage().

Pointy hat to: brooks

Don't prepend the directory specified by -d when the file is a relative
path. Doing so makes no sense. I'm not sure allowing relative paths
makes sense either, but I'm not going to break that now.

Add a new -d argument which is used to specify an alternate root for log
files similar to DESTDIR in the BSD make process. This only affects log
file paths not config file (-f) or archive directory (-a) paths.

Nothing says that /var/log can't be not a directory but a symbolic link
to a directory. Therefore, use stat(2) instead of lstat(2) to check if
/var/log exists.

MFC after: 7 days

Mechanically kill hard sentence breaks.

Fix the format-string in a call to err(). It was causing a warning if
compiled on 4.x-stable.

Fix the check for a "duplicate filename to compress", so that we're checking
the *filename* and not the pid_file(!). Stupid brain-fault on my part.
This could cause a segfault under -neworder if newsyslog had to rotate
multiple files, and later ones had specifed the 'N' flag.

Bug first reported by: le
MFC after: 3 days

Assorted markup, grammar, and spelling fixes.

Switch to using the "neworder" for rotating log files, by default. The
main advantage of this is that daemon's are only signalled once per run,
instead of once for each file that is rotated.

MFC after: 2 weeks

Drop the include for <stdint.h>, it was only needed when this was
using __DECONST() for something, and that reference has been removed.

Noticed by: Helge Oldach
MFC after: 13 days

Add an "oldorder" option, so that when the default changes to "neworder",
people have a way to drop back to the previous logic.

MFC after: 13 days

In "neworder" processing, reduce the delay between signals to separate
processes, and balance that by adding a 10-second delay after all the
processes have been signaled. Also improvement a few messages printed
with `-n' or `-v' processing (mostly signal-related messages).

MFC after: 13 days

Major re-ordering of the steps that newsyslog will use when processing
files to rotate. The new order will first rotate all files that need
to be rotated, and then send a single signal to each process which
needs to be signaled, and finally it will compress all the files which
were rotated.

This means daemons will be signaled once per run of newsyslog, instead
of once per file rotated. Also, files will be compressed in order of
file-size (smallest to largest). Also, it waits for each file to be
completely compressed before starting the next one (effectively as if
the 'w' flag is specified for all entries in newsyslog.conf). This
avoids the situation of having 10 gzip's going at the same time (each
with a log.0 and a log.0.gz file active), and it also means that file
attributes can be reliably set on files after they are compressed.

NOTE: This commit does define NEWORDER (which you could get rid of if
you really don't trust this), but it does not flip the "-D neworder"
switch. So, at the moment none of these changes happen unless you
request them (perhaps by adding '<debug> neworder' in newsyslog.conf).

PR: bin/25070 inspired some parts of this
Submitted by: parts from bin/25070 done by Helge Oldach
MFC after: 14 days

A variety of minor changes. Allow users to set a debugging option via
the newsyslog.conf file. Rename one size-related variable, and move
another one from the stack into conf_entry. Add a routine to change
file-attributes (chown, chmod, chflags), instead of having several
places doing the same sequence of system-calls. A few cosmetic/style
changes.

These should not effect any users. Most of these probably look
pointless, but they are the "insignificant parts" of a much larger
update that I'll be committing soon. Doing these as a separate update
should make that update easier to read.

MFC after: 14 days

Style-istic fix to a number of #define's that were not followed by a tab...

MFC after: 16 days

Add a 'D' flag that can be specified on entries in newsyslog.conf.
If specified, the matching log files will have the NODUMP flag set
on them after they are created.

Submitted by: Sean Eric Fagan
MFC after: 16 days

Change standard processing to use the newer createlog() routine
that had been written some months ago for other processing. This
should get rid of a few subtle situations where an existing log
file would not exist (for a short time) while it is being rotated.

MFC after: 16 days

Improved versions of the is*ch() and tolowerch() macros that I like to use.

When rotating some "blah.log" file, make sure that a chmod and
(if requested) a chown is done on the "blah.log.0" file.

PR: bin/67137
Submitted by: jeh
MFC after: 10 days

Do not :-terminate err(3) string, one will be added anyway.

Move newsyslog.conf.5 to usr.sbin/newsyslog. There is no real history
other than 'initial revision' thus I did not request a repocopy.

Requested by: ru, gad

Remove information about the configuration file.
Add an Xref to newsyslog.conf.5 and bzip2.1.

Handle a 'const' parameter without using __DECONST().

MFC after: 14 days

Restructure the time processing routines, mainly to fix up the
"will trim at" message printed when the user requests '-v'. The
previous code would often print the wrong time, such as:
On Sept 22, run: newsyslog -nv /var/log/wtmp
And see: will trim at Mon Sep 1 05:00:00 2003
correct msg: will trim at Wed Oct 1 05:00:00 2003

MFC after: 20 days

Correct the calculation of "a leap year" in parseDWM. The calculation
would only match a leap year every 400 years. The parseDWM code first
showed up in April 2000, so the first time this bug would cause any
confusion is in Feb 2004.

MFC after: 18 days

Move the parse8601 and parseDWM routines into a new ptime.c file. The
only code-change is to add a "next_time" parameter to both routines (and
that is not used yet). A later update will make "next_time" more useful.

MFC after: 20 days

Switch dotrim() to take advantage of the 'struct conf_entry' that
is already passed in, instead of having the caller copy values from
that struct into additional parameters.

MFC after: 22 days

Reduce the annoying compiler warnings that pop up when compiling with
gcc 3.3.x and -Wshadow. Just renames 'log' variables to be 'logname'.

MFC after: 22 days

Correct the comment about which timezone-change loses an hour...

MFC after: 23 days

Add a '-D <something>' command line arg, which can be used to set
debugging options. Initial option is '-D TN=<time>', which can be
used to see how newsyslog would work if run at the specified time.
(time format is ISO 8601, since that is already supported).

MFC after: 23 days

Use strtol() instead of strtoul() in parse8601, so we can detect
negative values. Mainly done to sync this routine with OpenBSD.

Obtained from: OpenBSD
MFC after: 23 days

Fix typo in the previous commit. Was checking wrong variable...

MFC after: 23 days

Change parse8601 and parseDWM so they return an alternate error value
for invalid times, and have the caller print the error message.

MFC after: 23 days

Cosmetic change to move parse8601 right next to parseDWM. No code
is changed. (that will come in later updates).

MFC after: 23 days

When checking the 'user:group' field in newsyslog.conf, freebsd's source
was mistakenly calling the standard isnumber() function to find out if
the given 'user' or 'group' were all numeric. This meant that only the
first character of the fields were actually checked, so a username of
(say) '3com' would look like a number, and thus get mapped to uid=3 (bin)
instead of username=3com.

This bug was introduced back in freebsd's v1.1. That initial import
almost matches netbsd's v1.9, except that an internal isnumber()
routine was removed in favor of the standard library version. The thing
is, that internal routine was checking the entire string, and not just
the first digit. In OpenBSD, isnumber() was eventually renamed to
isnumberstr() to make the distinction more obvious, and I'm going to
follow that lead.

I believe this also happens to remove the last references to isnumber()
in the entire freebsd base system.

Obtained from: OpenBSD, by a long circuitous route
MFC after: 5 days

Fix an 'bad file descriptor' error which would come up when using
the 'C' flag on an entry that also specifies 'user:group' info.

Submitted by: Riccardo Torrini <riccardo@torrini.org> in -current
MFC after: 4 days

Fix age_old_log checking so that it will notice log files which were
rotated and then compressed with bzip2 instead of gzip. Otherwise,
any file which had a time-interval specified for 'when' and also
specified the 'J' flag would be rotated every time newsyslog was run.
(this is a quick-fix, trying to beat the code-freeze for 5.1-release)

PR: bin/51519
MFC after: 1 week

Slightly improve the checking of the 'size' field, so people will get
a warning message if they switch the values for 'size' and 'when'.

MFC after: 1 week

Use __FBSDID over rcsid[]. Protect copyright[] where needed.

Add a -C (create) option for newsyslog, and a 'C' flag for entries in the
config file. If the -C option is specified once, then newsyslog will create
any entries which specify the 'C' option. If -C is given twice, then
newsyslog will create all missing log files. Some of this code comes
from NetBSD, although this implementation does not exactly match theirs.

Reviewed by: freebsd-arch
MFC after: 10 days

Document support for the new 'U' flag, which was implemented back in March
as revision 1.64 of newsyslog.c. I plan to MFC these changes next week.

MFC after: 1 week

Document support for the new -s and -R command-line options, and
the 'N' flag. These were coded in March as revisions 1.55 and 1.56
of newsyslog.c. I intend to MFC all the matching changes next week.
This also reorganizes the description of the 'flags' field to give
list of the valid flags, instead of a long paragraph explaining
each of the possible values.

Obtained from: NetBSD (in spirit at least, for -s and N)
MFC after: 1 week

Document support for the <default> entry, which was implemented in
revision 1.50 of newsyslog.c , back in February. I intend to MFC
those changes soon.

MFC after: 4 days

Fix interactions between entries for a specific file vs entries for
a filename pattern, and also wrt filenames given on the command line.
Now if a file is listed as a specific entry, it will not *also* be
processed by an entry specifying a pattern. And filename-patterns
will now only match existing files (ignoring directories, etc).

MFC after: 3 weeks

The processing of files given on the command-line will work a little better
if we return the entire worklist, instead of just the last entry. A minor
fix to revision 1.57.

MFC after: 15 days

Add a config-file flag of 'U' or 'u' to indicate that the pid-file
will contain the pid for a process group. This means the file must
contain a negative value (as would be needed in the 'kill' commmand).
I still need to write man-page update before MFC-ing.

This started by rewriting the get_pid() routine. Later I looked at
what OpenBSD has, and included a few ideas from their send_signal()
routine. So, parts of this change are from OpenBSD, even though
OpenBSD does not actually have a 'U' flag.

PR: bin/28435
Reviewed by: no objections on freebsd-arch
MFC after: 3 weeks

Prefix all -n (noaction) "unix cmds" with tab, thus making it much easier
to sort out noaction messages from -v (verbose) messages. This also
improves a few noaction messages.

Obtained from: OpenBSD
MFC after: 3 weeks

Move the -s (nosignal) processing, as newsyslog needs to print out a
warning message if -s is specified and it rotates a file that expects
to be compressed. This warning message is not printed if -R is also
specified, because we assume a -sR request is coming from the process
which would have been signaled, and that it has already released the
logfile.

Indirectly noticed by: sheldonh

Add missing 'static' on two routines, so they match their declarations.

PR: 29363 (a small part of)
Submitted by: dinoex
MFC after: 3 weeks

A variety of cosmetic changes, mostly to (slightly) reduce diffs with
OpenBSD. Except for one added '\n', the object code is not changed.

MFC after: 3 weeks

Collapse two more parameters to dotrim() into the 'ent' parameter that is
already there. This should not effect anything.

MFC after: 3 weeks

Change to using uid_t and gid_t instead of int. Also change this won't
bother doing a chmod() if neither of these two fields are set.

Obtained from: OpenBSD
MFC after: 3 weeks

Fix the interactions between specific log files given on the command line,
and config-file entries which specify a filename-pattern (glob). It is
still not perfectly-right, but at least it isn't completely-wrong.

Reviewed by: no objections on freebsd-arch
MFC after: 3 weeks
MFC addendum: (or after the code-freeze of 4.x is lifted)

Add a command-line option of '-R somename', which indicates that newsyslog
should rotate all files given on the command, even if they don't seem to
need to be rotated. This would be used by some other command that decides
the given log file(s) should be rotated, but wants the "how" of that rotation
to be determined by entries to newsyslog. Wes expects to change syslogd to
take advantage of this. Man page will be updated after we're sure this is
all working the way we want it to.

Reviewed by: no objections on freebsd-arch
MFC after: 3 weeks
MFC addendum: (or after the code-freeze of 4.x is lifted)

Add a command-line option of '-s', which indicates that newsyslog should
not send a signal to any processes. Also add a config-file flag of 'N' or
'n', which indicates that the given logfile has no process which needs a
signal when it is rotated. Both of these are based on changes NetBSD
has made, although the implementation is somewhat different.

PR: bin/36553 (2nd half)
Reviewed by: no objections on freebsd-arch
Obtained from: NetBSD (in spirit, at least)
MFC after: 3 weeks

Fix an error which caused 'free(): error: chunk is already free'
if an entry in the config file included flag 'G'.

MFC after: 3 weeks

Check the value given for number-of-logs to keep in config-file entries,
and error-out if the value is negative (avoiding an infinite loop).

PR: bin/30654 (part of it)
MFC after: 3 weeks

1) Fix so -a honors the -n ("do nothing") option. Before, if the directory
given for -a did not exist, then newsyslog would always try to create
it, even if -n was specified.
2) When -a processing *does* create the directory, have it check the result
from mkdir(), and immediately error-out if that failed.

PR: bin/46974
MFC after: 3 weeks

Minor style fix. Add parenthesis around 'return' values.

Reviewed by: md5
MFC after: 3 weeks

Add the idea of a "default rotate action", which will be used if a file is
specified at runtime, but that filename is not listed in the newsyslog.conf
file. This default-action can be changed by having a line in newsyslog.conf
with the filename of "<default>". Before this change, the program would
quietly ignore the given file. An update to the man page will be written
after I finish some other updates to newsyslog.c.

Reviewed by: no objections from freebsd-arch
MFC after: 3 weeks

english(4) police.

Make newsyslog(8) print the correct date when the logs are turned over.

PR: 46395
Submitted by: maxim
MFC: eventually

Add a new `W' flag, that when used in conjunction with the `Z' flag or the
`J' flag means that newsyslog should wait for previously started compression
jobs to complete before starting a new one for this entry. When it is used
along with the `G' flag, in the case when several logfiles match the pattern
and should be compressed, the newsyslog will compress logs one by one,
ensuring that only one compression job is running at a time.

This prevents newsyslog(8) from overloading system by starting several
compression jobs on big files simultaneously.

Sponsored by: Porta Software Ltd
MFC after: 2 weeks

Backout unapproved WARNS level change occasionally slipped into previous
commit.

Big pointy hat to: sobomax

- Allow comments to be placed at the end of configuration line.

Sponsored by: Porta Software Ltd
Approved by: re
MFC after: 2 weeks

mdoc(7) police: markup fixes.

Approved by: re

Add a new newsyslog.conf flag - 'G', which if set tells newsyslog(8) that
the specified filename of the log to be rotated is in fact shell glob
pattern. In this case, all files matching this pattern will be rotated
using the same options. Useful in the case when there is no pre-defined
name for the logfiles (e.g. xtradius, samba etc).

Sponsored by: PortaOne Software Ltd
MFC after: 2 weeks

Fix getopt(3) optstring: remove a flag we does not handle.

PR: bin/44066
Submitted by: Stefan Farfeleder <e0026813@stud3.tuwien.ac.at>
MFC after: 1 week

The .Nm utility

Remove the last EOF spaces I missed in rev. 1.43.

MFC after: 1 week

style(9): sort includes, remove EOL spaces.

MFC after: 1 week

wait(2) for compress_log() processes to complete before exit.

Spotted by: Grigoriy Strokin <grg@isabase.philol.msu.ru>
Reviewed by: -current
MFC after: 1 month

Close a very small window during which new (empty) instances of rotated log
files are owned by the caller of newsyslog (usually root:wheel) even if
alternative ownerships were specified in newsyslog.conf.

Note that this is part of a wider problem which is fully addressed in
OpenBSD. Anyone with the time and inclination to incorporate the full
fix for the wider problem will receive no complaints from me and should
feel free to walk all over this delta.

PR: bin/36738
MFC after: 1 week

Produce a different error message for mktime(3) failure than for bad
newsyslog.conf syntax.

PR: bin/36641

Fix -F with logfiles rotated on time only.

Approved by: silence on -audit
MFC after: 1 month

Fix the type of the NULL arg to execl().

WARNS= -> WARNS?=

jhb says a 10 line change containing casts is more invasive than a 28 line
change.

Use the POSIX syntax for chown.

Submmited by: garrett

Simplify (and hopefully clarify) some date-parsing arithmetic.

style(9)

WARNS-2'ify.
(this included removing the funny "if (unsigned_var < 0)" test)

Add the ability to Bzip2 your logs.

PR: 27901
Submitted by: Anders Nordby <anders@fix.no> (stylistic changes by me)

Perform a major cleanup of the usr.sbin Makefiles.
These are not perfectly in agreement with each other style-wise, but they
are orders of orders of magnitude more consistent style-wise than before.

Note the equivalence of certain time formats.

Instead of giving an example of rotating logs at the end of the month,
give an example of how to rotate logs at the beginning of the month.
Although they sound the same, since both of them rotate logs at the
beginning of the day, the former ended up taking place on, e.g., July
31 00:00 instead of the expected July 31 23:59. This is contraty to POLA.

Submitted by: Dan Langille <dan@langille.org>

Remove whitespace at EOL.

Fix the type of the NULL arg to execl()

Idea from: Theo de Raadt <deraadt@openbsd.org>

Fix a grammar bogon.

- Backout botched attempt to introduce MANSECT feature.
- MAN[1-9] -> MAN.

Set the default manual section for usr.sbin/ to 8.

mdoc(7) police: split punctuation characters + misc fixes.

- Correct MAXPATHLEN/MAXHOSTNAMELEN usage
- Check return values of malloc() and strdup()
- Replace strcpy()/strcat()/sprintf() usage with strlcpy()/snprintf

Reviewed by: -audit

mdoc(7) police: use the new features of the Nm macro.

When warning that a log file can't be compressed, tell the reader which
log file it was.

Allow user to specify which logs to rotate

- implement logfile archivation to a separate, user
configurable directory
- implement alternate and more flexible way to specify
logfile rotation time in addition to the ISO 8601
restricted format
- cleanup the source which was a mix of several styles
of persons who maintained it so far, ran through
knfom script got from bde.

Reviewed by: (in part) sheldonh and garyj

Run newsyslog source through knfom script obtained from Bruce Evans.
Joerg Wunsch suggested to do this to make the functional changes in
the next commit to newsyslog (which were run through the same script)
better visible.

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

.Nm corrections

$Id$ -> $FreeBSD$

Correctly refer to the companion program rather than as "chown".
Clean up wording a little (like I have any idea what real English looks like...)

Go back to allowing ``.'' as a username.group separator for backward
compatibility. : is still the documented non-ambiguous approach. The
algorithm used will correctly parse david.obrien.staff as strrchar() is
used, and in my mind more people would use a ``.'' in the username than
the group name.

Convinced by argument and patch by: sheldonh (with slight changes by me)

Syntax for user/group is changed from "user.group" to "user:group" to be
consistant with chown(8).

Fix description of size, it's in kilobytes, not bytes.

jkoshy forgot up update the heading date on the man page.

Note that the 'owner.group' field is optional in the config file.

PR: docs/9602
Submitted by: Kazuo Horikawa <horikawa@jp.freebsd.org>

A slight bit of code and doco cleanup, but mostly:

Add the much-needed and long-desired ability to force the log rotation
to take place on a specific day or time, and document same. This works
by extending the syntax of the `when' field to allow a restricted
ISO 8601 time specification, and performing the log rotation if newsyslog
is run during the same hour as is specified.

The largest PID is now 99999
Reported by: Ben Stuyts <ben@stuyts.nl>

Remove useless `BINOWN=root' now that it is the default.

Add optional config file field: signal number to send (defaulted to HUP)
F.e. apache restarts less agressively on USR1 then on HUP

1) kill -HUP syslog, even if compressing supposedly "binary" files
PR: docs/6385

2) -n (noaction) does not imply -r (run as non-root), since as of
Rev. 1.12 (ache), -r changes actual behaviour.

3) missing \n from if(noaction) messages.

Arg! Someone fixed the synopsis in -stable's newsyslog.8, but not
in -current. However, they fixed both the ordering and the missing -F
flag. Merge from 22 and fix ordering here, too.

Fix the usage here, too (-F option).

Typo fix in comment ("Force the tim now matter what" s/tim/trim/)

1) Remove a line that somehow strayed from the one true path of
its sentence.
2) Fix usage (-F option).

.Sh AUTHOR -> .Sh AUTHORS. Use .An/.Aq.

Add "-F" (force) flag to newsyslog.
Submitted by: Terry Lambert <terry@freebsd.org>

Sleep 10secs after kill -HUP instead of 3secs, some daemons like Apache
may have log time to re-configure

Cosmetic in err() strings.

Show the real revision date and not the date that this
manpage is being viewed.

Allow a '-' flag as a placeholder so that path_to_pid_file
can be specified without the flags field.
Fix bogus "trail" left in parse routine where the code jumps
past the end of the line and wanders into oblivion.

Do not kill -HUP syslog if running in user mode (-r), it allows
private user logs rotation without error diagnostics

1) Add yet one optional field: pid file to send SIGHUP to if log rotated
(good thing for apache f.e.) This change is backward compatible with old
newsyslog.conf files.

2) Do not compress log if SIGHUP sending failed for some reason
(f.e. pid file deleted). Newcoming messages will be lost
otherwise.

3) Misc cleanup while I am here.

compare return value from getopt against -1 rather than EOF, per the final
posix standard on the topic.

Add a missing period so that the -n option is display properly
in the man page output. Closes PR# 2831.

Submitted by: Marc Slemko <marcs@znep.com>

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Convert to mdoc format.

Doh! Actually read the patch I applied and see what it was *trying* to do,
then make it do that.
Error-of-my-ways-pointed-out-by: Bob Willcox <bob@luke.pmr.com>

PR#2724 in NetBSD notes that newsyslog save log files even if the number
of copies to save is zero. Incorporate suggested fix with some stylistic
cleanup to make the resulting code more readable.
Submitted-By: Kenneth Stailey <kstailey@dol-esa.gov>

Added function prototypes and removed in-line prototypes of standard
functions such as malloc and time.

Fix uninitialized variable that caused a core dump if /etc/newsyslog.conf
was empty. Closes PR bin/1300.

Found by: Brian Tao <taob@io.org>

Submitted by: charnier@lirmm.fr (Philippe Charnier)
1) correct MAX_PID according to system definition.
2) replace fprintf(stderr,...)/perror/exit by 4.4 equivalence.
3) make strings enough larger so they can't overflow: MAXHOSTNAMELEN for
host and MAXPATHLEN for filenames.
4) call execl with correct parameters (compress -> gzip).

Obtained from: NetBSD (PR#1906)
removed date formatting glitch in newsyslog

This commit was generated by cvs2svn to compensate for changes in r13244,
which included commits to RCS files with non-trunk default branches.