MFC r316945-r316946

r316945:
Add 410.status-mfi, a periodic script for mfi(4) arrays

PR: 176049
Submitted by: doconnor@gsoft.com.au
Reviewed by: scottl, Larry Rosenman <ler@lerctr.org>
Relnotes: yes

r316946:
Reorder Makefile entries from r316945

PR: 176049
Reported by: Oliver Pinter
X-MFC-With: 316945

MFC r316342, r316358

r316342:
Consolidate random sleeps in periodic scripts

Multiple periodic scripts sleep for a random amount of time in order to
mitigate the thundering herd problem. This is bad, because the sum of
multiple uniformly distributed random variables approaches a normal
distribution, so the problem isn't mitigated as effectively as it would be
with a single sleep.

This change creates a single configurable anticongestion sleep. periodic
will only sleep if at least one script requires it, and it will never sleep
more than once per invocation. It also won't sleep if periodic was run
interactively, fixing an unrelated longstanding bug.

PR: 217055
PR: 210188
Reviewed by: cy
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D10211

r316358:
Fix man page typo from r316342

Reported by: rgrimes
MFC after: 20 days
X-MFC-With: 316342

MFC r302798 [cy]: Restore lost comment from r301295.

PR: 211027
Reported by: Trond.Endrestol@ximalas.info

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Enable daily_ntpd_leapfile_enable by default. Otherwise an expired
leapfile will be ignored and ntpd will behave as if it has no
leapfile.

While here, remove an extraneous blank line.

Suggested by: ache
MFC after: 1 week

Better document security_show_{success,info,badconfig} in /etc/periodic.conf

periodic(8) already handles the security_show_{success,info,badconfig}
variables correctly. However, those variables aren't explicitly set in
/etc/defaults/periodic.conf or anywhere else, which suggests to the user
that they shouldn't be used.

etc/defaults/periodic.conf
Explicitly set defaults for security_show_{success,info,badconfig}

usr.sbin/periodic/periodic.sh
Update usage string

usr.sbin/periodic/periodic.8
Minor man page updates

One thing I'm _not_ doing is recommending setting security_output to
/var/log/security.log or adding that file to /etc/newsyslog.conf, because
periodic(8) would create it with default permissions, usually 644, and
that's probably a bad idea.

Reviewed by: brd
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6477

Add support for automatic leap-second file updates.

The working copy of leapfile resides in /var/dbntpd.leap-seconds.list.
/etc/ntp/leap-seconds (periodically updated from ftp://time.nist.gov/pub/
or ftp://tycho.usno.navy.mil/pub/ntp/) contains the master copy should
automatic leapfile updates be disabled (default).

Automatic leapfile updates are fetched from $ntp_leapfile_sources,
defaulting to https://www.ietf.org/timezones/data/leap-seconds.list,
within $ntp_leapfile_expiry_days (default 30 days) from leap-seconds
file expiry. Automatic updates can be enabled by setting
$daily_ntpd_leapfile_enable="YES" in periodic.conf. To avoid congesting
the ntp leapfile source the automatic update randomized by default but
can be disabled through daily_ntpd_avoid_congestion="NO" in
periodic.conf.

Suggested by: des
Reviewed by: des, roberto, dwmalone, ian, cperciva, glebius, gjb
MFC after: 1 week
X-MFC with: r289421, r293037

periodic: Fix backwards compatibility for daily_status_security_* vars.

Most daily_status_security_* variables in periodic.conf were changed to
security_status_* in SVN r254974. The compatibility code for the old names
did not work.

PR: 204331
Submitted by: martin at lispworks.com
MFC after: 1 week

Rename etc/periodic/daily/430.status-rwho to periodic/daily/430.status-uptime

The command was checking local/remote system uptime, so rename the script to
match its function and to avoid confusion

The controlling variable in /etc/periodic.conf has been renamed from
daily_status_rwho_enable to daily_status_uptime_enable.

MFC after: 3 days
Reported by: Peter Jeremy <peter@rulingia.com>
Relnotes: yes
Sponsored by: EMC / Isilon Storage Division

Allow again periodic scripts to be run from command-line.

PR: 188109
Submitted by: Jason Unovitch
MFC after: 1 week

Allow additional flags to be passed to netstat -i in the daily status check.
In particular, this allows an administrator to specify "-h" for human
readable output if that is preferred.

The default setting passes "-d", so that can be excluded by using a custom
setting.

Differential Revision: https://reviews.freebsd.org/D2034
Submitted by: Lystopad Aleksandr <laa@laa.zp.ua>
(patch to add option for -h)
Reviewed by: bz
MFC after: 1 week

- Align comment for df flags variable in periodic.conf.
- Note default value of df flags variable in periodoc.conf(5).

MFC after: 1 week

Evaluate running userland/kernel version in daily
periodic(8) run, taken from uname(1) '-U' and '-K'
flags.

Reviewed by: allanjude, dvl
Differential Revision: https://reviews.freebsd.org/D1541
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Remove settings for pkg_* scripts which are no longer present.

MFC after: 1 week

Remove pkg_* related info from periodic.conf

Reported by: Robin Brocks <robin.brocks@brocks.de>
MFC after: 3 days

Remove remnants of BIND from /etc, since there is no BIND in base now.

Sorry, that would break users running head and BIND from ports, since
ports rely on these scripts. The ports will be fixed soon.

Reviewed by: erwin

Fix indentation.

MFC after: 4 days

Fix compatibility function for old daily_status_security_${name}_enable
variable

PR: conf/183137
Reported by: Adam McDougall <mcdouga9 at egr msu edu>
MFC after: 3 days

Turn it all the way up to 11:

- Update FreeBSD version in:
- UPDATING
- sys/conf/newvers.sh

- Add 11.0 FreeBSD version for manual pages

- Bump __FreeBSD_version to 1100000

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Since r254974, periodic scripts' period can be configured
independently. There is no reason to leave their options
with the daily ones, so move them to their own section.
Move periodic scripts' options into their own section. Since r254974,

Make the period of each periodic security script configurable.

There are now six additional variables
weekly_status_security_enable
weekly_status_security_inline
weekly_status_security_output
monthly_status_security_enable
monthly_status_security_inline
monthly_status_security_output
alongside their existing daily counterparts. They all have the same
default values.

All other "daily_status_security_${scriptname}_${whatever}"
variables have been renamed to "security_status_${name}_${whatever}".
A compatibility shim has been introduced for the old variable names,
which we will be able to remove in 11.0-RELEASE.

"security_status_${name}_enable" is still a boolean but a new
"security_status_${name}_period" allows to define the period of
each script. The value is one of "daily" (the default for backward
compatibility), "weekly", "monthly" and "NO".

Note that when the security periodic scripts are run directly from
crontab(5) (as opposed to being called by daily or weekly periodic
scripts), they will run unless the test is explicitely disabled with a
"NO", either for in the "_enable" or the "_period" variable.

When the security output is not inlined, the mail subject has been
changed from "$host $arg run output" to "$host $arg $period run output".
For instance:
myfbsd security run output -> myfbsd security daily run output
I don't think this is considered as a stable API, but feel free to
correct me if I'm wrong.

Finally, I will rearrange periodic.conf(5) and default/periodic.conf
to put the security options in their own section. I left them in
place for this commit to make reviewing easier.

Reviewed by: hackers@

Move daily_status_security_noamd next to 200.chkmounts's variables.

Remove periodic script for ataraid(4) and add instead script for graid(8).

Set default for ${pkg_info} like ${pkg_version}.

MFC after: 1 week

Use correct INDEX on 10-CURRENT.

Whitespace nit

Don't attempt to delete .sujournal in /tmp

PR: conf/163828
Submitted by: Tatsuki Makino <tatsuki_makino@hotmail.com>
Approved by: cperciva
MFC after: 1 week

Add an option to 404.status-zfs (enabled by default) to list all
zfs pools on the system.

While here, document daily_status_zfs_enable in periodic.conf(5).

Discussed on: -fs [1]
Reviewed by: netchild [1]
Approved by: jhb
MFC after: 1 week

[1] - http://lists.freebsd.org/pipermail/freebsd-fs/2011-June/011869.html

Increase default scrub threshold from 30 days to 5 weeks. Using
whole weeks makes it easier to predicate when the scrub would
happen.

MFC after: 1 week

Add missing default values for daily/800.scrub-zfs for documentation
purposes. No functional change, since all parameters are set to their
default values.
MFC after: 1 week

Add a daily period script to back up /var/db/pkg

The final product contains work from the originator, and
Florent Thoumie <florent.thoumie@gmail.com>. The final
product contains considerable re-working by me, so all
responsibility for bugs rests under my pointy hat.

PR: ports/145957
Submitted by: Eitan Adler <EitanAdlerList@gmail.com>

Enable the check for negative permissions (the group on a file can't do
something "everyone" can) by default.

X-MFC after: never

Add an (off by default) check for negative permissions (where the
group on a object has less permissions that everyone). These
permissions will not work reliably over NFS if you have more than
14 supplemental groups and are usually not what you mean.

MFC after: 1 week

Add a daily script to the periodic framework that reports
changes to the package database, i.e. any packages that
have been added, updated or deleted in the past 24 hours.
The format is intentionally simple and concise.

That information is particularly useful on servers that
are maintained by multiple administrators. When someone
adds, updates or deletes a package, the others will see
it in the daily periodic output.

This script is disabled by default.

PR: conf/113913
Submitted by: olli
Approved by: des (mentor)
MFC after: 3 weeks

- Add a periodic script, which can be used to find installed ports' files with
mismatched checksum

PR: conf/124641
Submitted by: Alex Kozlov <spam@rm-rf.kiev.ua>
Approved by: delphij (mentor)

Add .snap to daily_clean_tmps_ignore; /tmp/.snap ist not supposed to
be auto-removed (and /tmp is a filesystem of its own now by default).

MFC after: 3 days

Update name of INDEX file as part of 8.0 -> 9.0 transition.

A new configuration variable, daily_status_mail_rejects_shorten, allows
the rejected mail reports to tally the rejects per blacklist without
providing details about individual sender hosts. The default configuration
keeps the reports in their original form.

MFC after: 1 week

Update pkg_version_index to INDEX-8

Don't delete files in the X11 socket directories under /tmp (.X11-unix,
.ICE-unix, .font-unix, .XIM-unix) when purging files from /tmp via the
daily 100.clean-tmps job. If you are logged into an X session longer
than the timeout period (default of 3 days), then this job can delete
the X11 sockets out from under the session without this fix.

MFC after: 3 days

Now that a separate /usr/X11R6 directory is no longer in fashion,
stop looking there for things like rc.d and periodic. This avoids
duplicating effort when /usr/X11R6 is a symlink to /usr/local,
which it is by default now.

It is not anticipated at this time that we will MFC this change, since
we'd like to avoid breaking legacy systems. However, there is a fix for
/etc/rc.subr in the works to avoid running any rc.d scripts twice which
we should be able to MFC.

o Add a script to check ntpd(8) state. Default is off.

PR: conf/112604
Submitted by: Oliver Fromme
MFC after: 1 month

Add ZFS periodic scripts that monitors status of ZFS pools.

Submitted by: des

The kvm_mkdb(8) is long dead.

Use ports INDEX-7 instead of INDEX-6

Submitted by: Niclas Zeising <lothrandil@n00b.apagnu.se>

Add login.conf checking to periodic security scripts. If the login.conf file
is not UID/GID 0, limits will be ignored and a strange error sent to auth.log.

Head nod: ru, rwatson

Move etc/rc.firewall6 to ipfw2+v6, update related rc.d and periodic scripts.
Since ipfw2 now does dual-stack, statistics for IPv6 come from the ipfw
scripts as well.

Add the graid3(8), gstripe(8) and gconcat(8) status scripts, default is "off".

Approved by: rwatson (mentor)

Make df output more consistent:
Remove -k now that -h is present
use -l instead of -t nonfs to match smbfs too [1]
PR: conf/50956 [1]
Approved by: philip (mentor)
MFC after: 3 days

Make df output in periodic mail human readable

PR: conf/87196
Submitted by: Mike <mspam@ideaway.net>
Approved by: philip (mentor)
MFC after: 3 days

Add a daily script to show the status of gmirror(8) devices.

Add a reference to the periodic.conf(5) manual page.

Suggested by: simon

Ports index file is now INDEX-6

Teach periodic(8) security output to display information about blocked
packet counts by pf(4).

This adds a ``daily_status_security_pfdenied_enable'' variable to
periodic.conf, which defaults to ``YES'' as the matching IPF(W) versions.

The output will look like this (line wrapped):

pf denied packets:
> block drop log on rl0 proto tcp all [ Evaluations: 504986 Packets: 0
Bytes: 0 States: 0 ]
> block drop log on rl0 all [ Evaluations: 18559 Packets: 427 Bytes: 140578
States: 0 ]

Submitted by: clive (thanks a lot!)
MFC after: 2 weeks

Add a knob 'daily_status_security_diff_flags' controlling the
format of the 'diff' output generated during periodic(8) scripts.

Submitted by: keramida (script changes)
Reviewed by: keramida (man page changes)

Allow the location of the INDEX file to specified to pkg_version.
This is particularly convenient on a cluster of machines to prevent
having to rebuild the INDEX file on each.

Reviewed by: portmgr

Add script for checking ipv6 blocked packets from PR.

PR: misc/50154
Submitted by: Kimura Fuyuki <fuyuki@hadaly.org>

Fix typo, I forgot daily_ in front of the status_ata_raid_enable

Add status checking of ATA raid to the daily periodic scripts.

Complete removal of 320.rdist by removing its entry from periodic.conf and
removing the related 220.backup-distfile script and associatd periodic.conf
entry.

Discussed with: obrien

Tighten wording of comment.

Suggested by: gshapiro

Do not do manually what sendmail(8) can do better automatically.
Tell sendmail to clean up its own host status cache.
The error condition handling could probably be done better.

Add a new /etc/periodic/security script to check for packets
rejected by ipfilter (510.ipfdenied), and a corresponding periodic.conf
knob (daily_status_security_ipfdenied_enable).

Reviewed by: roberto
Approved by: re@

Add a pkg_version variable so that it's possible to run portsversion instead
of pkg_version in periodic/weekly/400.status-pkg.

Remove trailing whitespace.

Update mail queue related periodic scripts to account for sendmail 8.12's
clientmqueue (submit mail queue).

The new mailq display is only active if both the old
daily_status_mailq_enable is set to "YES" and the new
daily_status_include_submit_mailq is set to "YES" so people who disabled
440.status-mailq won't have any surprises.

Likewise, the new queue run is only active if both the old
daily_queuerun_enable is set to "YES" and the new daily_submit_queuerun
is set to "YES" so people who disabled 500.queuerun won't have any
surprises.

While I am here, remove the [ ! -d /var/spool/mqueue ] checks from
both scripts as the queue directory isn't always /var/spool/mqueue for
the main daemon -- it can be set to anything in the sendmail.cf file.

MFC after: 1 week

Long ago, there was just /etc/daily. Then /etc/security was split out
of /etc/daily. Some time later, /etc/daily became a set of periodic(8)
scripts. Now, this evolution continues, and /etc/security has been
broken into periodic(8) scripts to make local customization easier and
more maintainable.

Reviewed by: ru
Approved by: ru

Finish the removal of uucp scripts.

Forgotten by: kris

Remove $daily_status_named_logs and figure out which /var/log/messages*
files to look an (in the same way that /etc/security does).

Don't single-quote $start, reducing it to an empty string.

MFC after: 3 days

Remove vestiges of MFS.

Default daily_accounting_flags to -q. I thought this was a typo in the
originally submitted patch (oops!).

Also check for an empty $daily_accounting_save.

Submitted by: Udo Schweigert <Udo.Schweigert@cert.siemens.de>

Add $daily_accounting_save and $daily_accounting_flags

Submitted by: Udo Schweigert <Udo.Schweigert@cert.siemens.de>
MFC after: 2 weeks

Check for denied zone transfers (AXFR and IXFR).

Fix a comment

PR: 25831
Submitted by: quinot@inf.enst.fr

Move the sendmail -q from cron to periodic, as suggested by a few people.
This has the benefit of adding a random start time element as daily
processing takes a different amount of time on different machines.

Allow the output of /etc/security to be logged or mailed to different
users in line with ${daily,weekly,monthly}_output using a new
$daily_status_security_output variable.

PR: 24643

Another overhaul of the periodic stuff.

All periodic sub-scripts <larf> now have their return codes interpreted
by periodic(8). Output may be masked based on variable values in
periodic.conf.

It's also now possible to email periodic output to arbitrary addresses,
or to send it to a log file, examples of which can be found in
newsyslog.conf.

The upshot of it all should be no discernable changes to the default
behaviour of periodic(8).

PR: 21250

The previous commit changed the df(1) units flag from -k to -h, which
produced human-readable output. I like this, but it's certainly not
something to change willy-nilly without discussion. Revert to -k.

Anyway, the new variable allows folks to pick any units flag that
fits their fancy.

Introduce a new option, daily_status_disks_df_flags, which specifies
the command-line arguments to be used for the call to df(1) when
daily_status_disks_enable is set to YES.

The name of the new variable was chosen by the maintainer of our
periodic hierarchy, Brian Somers.

PR: 19631

Add $daily_status_mail_rejects_logs, defaulting to 3 to control
how many /var/log/maillog* files to check

PR: 19587

Fix a comment

Submitted by: joe

Add weekly_status_pkg_enable (defaults to NO)

Allow compressed acct files

PR: 19483
Submitted by: Ben Smithurst <ben@scientia.demon.co.uk>

Introduce /etc/defaults/periodic.conf, similar in concept to rc.conf.
The only change in the default functionality should be that
the output reports are slightly more verbose WRT files deleted.

Not objected to by: freebsd-arch