periodic/daily/801.trim-zfs: Add a daily zfs trim script

As mentioned in zpoolprops(7), on some SSDs, it may not be desirable to
use ZFS autotrim because a large number of trim requests can degrade
disk performance; instead, the pool should be manually trimmed at
regular intervals.

Add a new daily periodic script for this purpose, 801.trim-zfs. If
enabled (daily_trim_zfs_enable=YES; the default is NO), it will run a
'zpool trim' operation on all online pools, or on the pools listed in
'daily_trim_zfs_pools'.

The trim is not started if the pool is degraded (which matches the
behaviour of the existing 800.scrub-zfs script) or if a trim is already
running on that pool. Having autotrim enabled does not inhibit the
periodic trim; it's sometimes desirable to run periodic trims even with
autotrim enabled, because autotrim can elide trims for very small
regions.

PR: 275965
MFC after: 1 week
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/956

periodic: Make daily diff(1) output as small is possible

Make, by default, daily diff(1) ignore whitespace changes and the unified output
a context of zero (0) lines. This reduces output of unrelated lines in e-mails
delivered to root.

PR: 270266
Approved by: jrm (mentor), karels
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D42762

periodic: Make daily diff(1) flags configurable with daily_diff_flags

PR: 270266
Approved by: jrm (mentor), karels
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D42900

periodic: Make security diff(1) output as small is possible

Make, by default, security diff(1) produce a unified output with a context of
zero (0) lines. This reduces output of unrelated lines in e-mails delivered
to root.

PR: 270266
Approved by: jrm (mentor), karels
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D43071

Remove $FreeBSD$: two-line nroff pattern

Remove /^\.\\"\n\.\\"\s*\$FreeBSD\$$\n/

pfdenied: support reporting on additional anchors

The security/520-pfdenied script only reports blocked packets from the
main ruleset or any blocklistd(8) anchor.

Add an option to periodic.conf(5) to make it possible to specify
additional anchors to report.

PR: 262446
Reviewed by: kp

periodic: remove obsolete news.expire

With the initial import of 386BSD 0.1 in 1993, the daily execution of
/etc/news.expire was introduced (see commit 1bf9d5d9518e).

In 1997, this was brought into periodic resulting in daily/330.news
(see commit 28dce04d1975). But as far as I see, /etc/news.expire has
never existed.

PR: 256238
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30631

periodic: add support for .xz and .zcat compressed logs

Also improve temporary file usage in 200.accounting, add an xref to
zstd(1) to newsyslog.conf.5, and clarify in periodic.conf that
"daily accounting" means process accounting and "monthly accounting"
is login accounting.

PR: 253868
Reviewed by: allanjude
Approved by: blackend (mentor)
Differential Revision: https://reviews.freebsd.org/D29267

Add a periodic script to backup output generated from `gmirror list`.

Disabled by default.

PR: 86388
Submitted by: Miroslav Lachman <000.fbsd@quip.cz>
Reviewed by: allanjude, gbe
Approved by: allanjude (mentor)
MFC after: 4 weeks
Event: July 2020 Bugathon
Differential Revision: https://reviews.freebsd.org/D25631

Add a periodic script to backup output generated from `zfs list`, `zfs get`,
`zpool list`, and `zpool get` commands.

Disabled by default.

PR: 86388
Submitted by: Miroslav Lachman <000.fbsd@quip.cz>
Reviewed by: allanjude, 0mp
Approved by: allanjude (mentor)
MFC after: 4 weeks
Event: July 2020 Bugathon
Differential Revision: https://reviews.freebsd.org/D25638

man5: Fix a few typos spotted by igor

- fstab(5): conjuction -> conjunction
- mount.conf(5): repeated 'the'
- periodic.conf(5): Partion -> Partition

MFC after: 1 week

periodic.conf(5): don't truncate interface names reported by 420.status-network.

The daily periodic script, 420.status-network uses netstat(1) to report
interface status. By default, netstat(1) truncates interface names.

Add the '-W' flag (avoid truncating interface names) as the default for
'daily_status_network_netstat_flags' in periodic.conf(5).

The default 420.status-network command is now 'netstat -i -d -W -n'.

PR: 163572
Reported by: kes-kes@yandex.ru
Reviewed by: allanjude (mentor) bcr (manpages)
Approved by: allanjude (mentor)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26023

periodic.conf: correct capitailization in the middle of a sentence

Reported by: yuripv
Event: July 2020 Bugathon

Add a periodic script to backup the partition table and boot code

Optionally, alert you if the contents change from the previous backup

PR: 86388
Submitted by: Rob Fairbanks <rob.fx907@gmail.com>, Miroslav Lachman <000.fbsd@quip.cz> (Original Version)
MFC after: 4 weeks
Relnotes: yes
Sponsored by: Klara Inc.
Event: July 2020 Bugathon
Differential Revision: https://reviews.freebsd.org/D25628

Fix typo in the documentation about the daily ntpd status

PR: 245679
Submitted by: Taylor Stearns <t@tstearns.com>
MFC after: 3 days

Add extremely useful calendar(1) application to FreeBSD

It does extremely useful things like execute sendmail and spew dubiously
accurate factoids.

From the feedback, it seems like it is an essential utility in a modern unix
and not at all a useless bikeshed. How do those Linux people live without it?
Reverts r358561.

Fix typo in r278616

FreeBSD isn't an encyclopedia.

Chase rename of rwho script in r290252

The script and associated variable was changed in r290252. Now just
chase it.

MFC With: r290252
Reported by: Aaron LI <aly@aaronly.me>

Deorbit catman. The tradeoff of disk for performance has long since tipped
in favor of just rendering the manpage instead of relying on pre-formatted
catpages. Note, this does not impede the ability to use existing catpages,
it just removes the utility to generate them.

Reviewed by: imp, allanjude
Approved by: emaste (mentor)
Differential Revision: https://reviews.freebsd.org/D12317

Add 410.status-mfi, a periodic script for mfi(4) arrays

PR: 176049
Submitted by: doconnor@gsoft.com.au
Reviewed by: scottl, Larry Rosenman <ler@lerctr.org>
MFC after: 3 weeks
Relnotes: yes

Fix man page typo from r316342

Reported by: rgrimes
MFC after: 20 days
X-MFC-With: 316342

Consolidate random sleeps in periodic scripts

Multiple periodic scripts sleep for a random amount of time in order to
mitigate the thundering herd problem. This is bad, because the sum of
multiple uniformly distributed random variables approaches a normal
distribution, so the problem isn't mitigated as effectively as it would be
with a single sleep.

This change creates a single configurable anticongestion sleep. periodic
will only sleep if at least one script requires it, and it will never sleep
more than once per invocation. It also won't sleep if periodic was run
interactively, fixing an unrelated longstanding bug.

PR: 217055
PR: 210188
Reviewed by: cy
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D10211

Fix bunch of .Xrs.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Allow additional flags to be passed to netstat -i in the daily status check.
In particular, this allows an administrator to specify "-h" for human
readable output if that is preferred.

The default setting passes "-d", so that can be excluded by using a custom
setting.

Differential Revision: https://reviews.freebsd.org/D2034
Submitted by: Lystopad Aleksandr <laa@laa.zp.ua>
(patch to add option for -h)
Reviewed by: bz
MFC after: 1 week

- Align comment for df flags variable in periodic.conf.
- Note default value of df flags variable in periodoc.conf(5).

MFC after: 1 week

o Typo: securiy -> security.

PR: 197927
Submitted by: mike.543@comcast.net
MFC after: 1 week

Evaluate running userland/kernel version in daily
periodic(8) run, taken from uname(1) '-U' and '-K'
flags.

Reviewed by: allanjude, dvl
Differential Revision: https://reviews.freebsd.org/D1541
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Various mdoc fixes.

Found with: mandoc -Tlint

use .Mt to mark up email addresses consistently (part6)

PR: 191174
Submitted by: Franco Fichtner <franco at lastsummer.de>

Clean up manual pages after BIND removal.

MFC after: 1 week

Since r254974, periodic scripts' period can be configured
independently. There is no reason to leave their options
with the daily ones, so move them to their own section.

mdoc fix

Make the period of each periodic security script configurable.

There are now six additional variables
weekly_status_security_enable
weekly_status_security_inline
weekly_status_security_output
monthly_status_security_enable
monthly_status_security_inline
monthly_status_security_output
alongside their existing daily counterparts. They all have the same
default values.

All other "daily_status_security_${scriptname}_${whatever}"
variables have been renamed to "security_status_${name}_${whatever}".
A compatibility shim has been introduced for the old variable names,
which we will be able to remove in 11.0-RELEASE.

"security_status_${name}_enable" is still a boolean but a new
"security_status_${name}_period" allows to define the period of
each script. The value is one of "daily" (the default for backward
compatibility), "weekly", "monthly" and "NO".

Note that when the security periodic scripts are run directly from
crontab(5) (as opposed to being called by daily or weekly periodic
scripts), they will run unless the test is explicitely disabled with a
"NO", either for in the "_enable" or the "_period" variable.

When the security output is not inlined, the mail subject has been
changed from "$host $arg run output" to "$host $arg $period run output".
For instance:
myfbsd security run output -> myfbsd security daily run output
I don't think this is considered as a stable API, but feel free to
correct me if I'm wrong.

Finally, I will rearrange periodic.conf(5) and default/periodic.conf
to put the security options in their own section. I left them in
place for this commit to make reviewing easier.

Reviewed by: hackers@

Remove references to ataraid(4) and atacontrol(8).

Bump date for content missed in r236290

Submitted by: gjb
Approved by: cperciva (implicit)
MFC after: 3 days
X-MFC-With: r236290

Document daily_status_security_chkportsum_enable

PR: docs/167980
Submitted by: "Bryan Drewery" <bryan@shatow.net>
Reported by: rank1seeker@gmail.com
Approved by: cperciva
MFC after: 2 weeks

Add an option to 404.status-zfs (enabled by default) to list all
zfs pools on the system.

While here, document daily_status_zfs_enable in periodic.conf(5).

Discussed on: -fs [1]
Reviewed by: netchild [1]
Approved by: jhb
MFC after: 1 week

[1] - http://lists.freebsd.org/pipermail/freebsd-fs/2011-June/011869.html

As of r226865, daily_scrub_zfs_default_threshold is 35 days; document
accordingly.

PR: 162890
Submitted by: Oliver Hartmann (ohartman ! mail.zedat.fu-berlin.de)
Patch by: Niclas Zeising (niclas.zeising ! gmail.com)
MFC after: 1 week
X-Need-MFC: r226865

Add an (off by default) check for negative permissions (where the
group on a object has less permissions that everyone). These
permissions will not work reliably over NFS if you have more than
14 supplemental groups and are usually not what you mean.

MFC after: 1 week

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Fix typos, spelling, formatting and mdoc mistakes found by Nobuyuki while
translating these manual pages. Minor corrections by me.

Submitted by: Nobuyuki Koganemaru <n-kogane@syd.odn.ne.jp>

Add a periodic zfs scrub script.

Features:
- configurable amount of days between scrubs (default value or per pool)
- do not scrub directly after pool creation (respects the configured
number of days between scrubs)
- do not scrub if a scrub is in progress
- tells how to see the status of the scrub
- tells how many days since the last scrub if it skips the scrubbing
- warns if a non-existent pool is specified explicitely
(default: no pools specified -> all currently imported pools are
handled)
- runs late in the periodic run to not slow down the other periodic daily
scripts

Discussed on: fs@

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

- Import the HEAD csup code which is the basis for the cvsmode work.

o Add a script to check ntpd(8) state. Default is off.

PR: conf/112604
Submitted by: Oliver Fromme
MFC after: 1 month

Reword previous commit, now supporting better English.

Discussed with: ceri

Belated update for login.conf checks.
Bump doc date.

The kvm_mkdb(8) is long dead.

Markup polishing.

Move etc/rc.firewall6 to ipfw2+v6, update related rc.d and periodic scripts.
Since ipfw2 now does dual-stack, statistics for IPv6 come from the ipfw
scripts as well.

Actually bump the date to the commit-date for the last commit.

Document the graid3(8), gstripe(8) and gconcat(8) periodic scripts.

Document the daily_status_ata_raid_enable and daily_status_gmirror_enable
variables.

Remove references to rdist(1) and friends.

Submitted by: Björn König

Use 'manual page' instead of 'man page' for consistency.

Approved by: re (hrs)

Expand contractions.

Scheduled mdoc(7) sweep.

Teach periodic(8) security output to display information about blocked
packet counts by pf(4).

This adds a ``daily_status_security_pfdenied_enable'' variable to
periodic.conf, which defaults to ``YES'' as the matching IPF(W) versions.

The output will look like this (line wrapped):

pf denied packets:
> block drop log on rl0 proto tcp all [ Evaluations: 504986 Packets: 0
Bytes: 0 States: 0 ]
> block drop log on rl0 all [ Evaluations: 18559 Packets: 427 Bytes: 140578
States: 0 ]

Submitted by: clive (thanks a lot!)
MFC after: 2 weeks

Add a knob 'daily_status_security_diff_flags' controlling the
format of the 'diff' output generated during periodic(8) scripts.

Submitted by: keramida (script changes)
Reviewed by: keramida (man page changes)

Assorted markup, spelling, and grammar fixes.

Spelling and grammar fixups.

Describe pkg_version_index

Reviewed by: joe

mdoc(7) police: Scheduled sweep.

Give more information to users replacing sendmail regarding periodic's
submit mail queue check.

PR: docs/38924

Do not do manually what sendmail(8) can do better automatically.
Tell sendmail to clean up its own host status cache.
The error condition handling could probably be done better.

Uniformly refer to a file system as "file system".

Approved by: re

Add a new /etc/periodic/security script to check for packets
rejected by ipfilter (510.ipfdenied), and a corresponding periodic.conf
knob (daily_status_security_ipfdenied_enable).

Reviewed by: roberto
Approved by: re@

Add a pkg_version variable so that it's possible to run portsversion instead
of pkg_version in periodic/weekly/400.status-pkg.

Update mail queue related periodic scripts to account for sendmail 8.12's
clientmqueue (submit mail queue).

The new mailq display is only active if both the old
daily_status_mailq_enable is set to "YES" and the new
daily_status_include_submit_mailq is set to "YES" so people who disabled
440.status-mailq won't have any surprises.

Likewise, the new queue run is only active if both the old
daily_queuerun_enable is set to "YES" and the new daily_submit_queuerun
is set to "YES" so people who disabled 500.queuerun won't have any
surprises.

While I am here, remove the [ ! -d /var/spool/mqueue ] checks from
both scripts as the queue directory isn't always /var/spool/mqueue for
the main daemon -- it can be set to anything in the sendmail.cf file.

MFC after: 1 week

Fix spelling errors.

Mark daily_news_expire_enable as a variable, not a path.

Long ago, there was just /etc/daily. Then /etc/security was split out
of /etc/daily. Some time later, /etc/daily became a set of periodic(8)
scripts. Now, this evolution continues, and /etc/security has been
broken into periodic(8) scripts to make local customization easier and
more maintainable.

Reviewed by: ru
Approved by: ru

Fixed two typos.

PR: docs/32327
Submitted by: Frans Haarman <frans@haarman.com>

Oops, another obsolete UUCP knob

Note removal of UUCP periodic maintenance knobs

mdoc(7) police: Fixed broken xrefs.

mdoc(7) police:

Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text.
Not only this slows down the mdoc(7) processing significantly, but it also
has an undesired (in this case) effect of disabling hyphenation within the
entire enclosed block.

Remove $daily_status_named_logs and figure out which /var/log/messages*
files to look an (in the same way that /etc/security does).

Don't single-quote $start, reducing it to an empty string.

MFC after: 3 days

Removed whitespace at end-of-line; no content changes. I simply did
cd src/share; find man[1-9] -type f|xargs perl -pi -e 's/[ \t]+$//'

BTW, what editors are the culprits? I'm using vim and it shows
me whitespace at EOL in troff files with a thick blue block...

Reviewed by: Silence from cvs diff -b
MFC after: 7 days

mdoc(7) police: removed HISTORY info from the .Os call.

mdoc(7) police: fixed markup in rev. 1.26.

Remove vestiges of MFS.

Default daily_accounting_flags to -q. I thought this was a typo in the
originally submitted patch (oops!).

Also check for an empty $daily_accounting_save.

Submitted by: Udo Schweigert <Udo.Schweigert@cert.siemens.de>

Add $daily_accounting_save and $daily_accounting_flags

Submitted by: Udo Schweigert <Udo.Schweigert@cert.siemens.de>
MFC after: 2 weeks

mdoc(7) police: use .Va and .Vt where appropriate

Check for denied zone transfers (AXFR and IXFR).

mdoc(7) police: normalize .Nd.

Move the sendmail -q from cron to periodic, as suggested by a few people.
This has the benefit of adding a random start time element as daily
processing takes a different amount of time on different machines.

Fix typo: seperate -> separate.

Seperate does not exist in the english language.

Submitted to look at by: kris

mdoc(7) police: split punctuation characters + misc fixes.

Allow the output of /etc/security to be logged or mailed to different
users in line with ${daily,weekly,monthly}_output using a new
$daily_status_security_output variable.

PR: 24643

Prepare for mdoc(7)NG.

mdoc(7) police: use the new features of the Nm macro.

remove fullstops from the end of .Xr lines in SEE ALSO sections.

* Fix inconsistency between "int" and "num"; as far as I can see they mean
the same thing in this context.
* Only include one fullstop in the SEE ALSO line.

PR: 22658
Submitted by: Yoshihiko Sarumaru <mistral@imasy.or.jp>

Change .Os FreeBSD 5.0 to .Os FreeBSD.

PR: 22684
Submitted by: Akinori YAMADA <yamada-a@nextcom.co.jp>

Correct the description of a few options to reflect that commands and/or
the results that they actualy use/generate.

PR: docs/22267
Submitted by: SUZUKI Koichi <koich@cac.co.jp>

Put temporary output in ${TMPDIR:-/tmp}
If $<basedir>_output is not set, don't redirect output

PR: 21395

Another overhaul of the periodic stuff.

All periodic sub-scripts <larf> now have their return codes interpreted
by periodic(8). Output may be masked based on variable values in
periodic.conf.

It's also now possible to email periodic output to arbitrary addresses,
or to send it to a log file, examples of which can be found in
newsyslog.conf.

The upshot of it all should be no discernable changes to the default
behaviour of periodic(8).

PR: 21250

Complete migration of aliases file to /etc/mail/aliases.
The maintainers of share/examples/diskless/README.TEMPLATING and mergemaster
have been contacted so those may be updated as well.

Correct a typo

Spotted by: Jimmy Olgeni <olgeni@uli.it>

Introduce a new option, daily_status_disks_df_flags, which specifies
the command-line arguments to be used for the call to df(1) when
daily_status_disks_enable is set to YES.

The name of the new variable was chosen by the maintainer of our
periodic hierarchy, Brian Somers.

PR: 19631

Use "yesterday's" instead of "yesterdays" as required for correct
grammar.

Since this pushes the line to an unacceptable length, apply the
accepted line breaking style.

De-spam (oops!)

Spotted by: sheldonh

Add $daily_status_mail_rejects_logs, defaulting to 3 to control
how many /var/log/maillog* files to check

PR: 19587

Reword the description of weekly_status_pkg_enable (although not
quite how Sheldon suggested)

Suggested by: sheldonh

Add weekly_status_pkg_enable (defaults to NO)

Mention $daily_accounting_compress

Forgotten by: Ben & I

Introduce /etc/defaults/periodic.conf, similar in concept to rc.conf.
The only change in the default functionality should be that
the output reports are slightly more verbose WRT files deleted.

Not objected to by: freebsd-arch