sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vfs: drop the mostly unused flags argument from VOP_UNLOCK

Filesystems which want to use it in limited capacity can employ the
VOP_UNLOCK_FLAGS macro.

Reviewed by: kib (previous version)
Differential Revision: https://reviews.freebsd.org/D21427

Rework the logic around quick checks for auditing that take place at
system-call entry and whenever audit arguments or return values are
captured:

1. Expose a single global, audit_syscalls_enabled, which controls
whether the audit framework is entered, rather than exposing
components of the policy -- e.g., if the trail is enabled,
suspended, etc.

2. Introduce a new function audit_syscalls_enabled_update(), which is
called to update audit_syscalls_enabled whenever an aspect of the
policy changes, so that the value can be updated.

3. Remove a check of trail enablement/suspension from audit_new() --
at the point where this function has been entered, we believe that
system-call auditing is already in force, or we wouldn't get here,
so simply proceed to more expensive policy checks.

4. Use an audit-provided global, audit_dtrace_enabled, rather than a
dtaudit-provided global, to provide policy indicating whether
dtaudit would like system calls to be audited.

5. Do some minor cosmetic renaming to clarify what various variables
are for.

These changes collectively arrange it so that traditional audit
(trail, pipes) or the DTrace audit provider can enable system-call
probes without the other configured. Otherwise, dtaudit cannot
capture system-call data without auditd(8) started.

Reviewed by: gnn
Sponsored by: DARPA, AFRL
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D17348

sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Add an experimental DTrace audit provider, which allows users of DTrace to
instrument security event auditing rather than relying on conventional BSM
trail files or audit pipes:

- Add a set of per-event 'commit' probes, which provide access to
particular auditable events at the time of commit in system-call return.
These probes gain access to audit data via the in-kernel audit_record
data structure, providing convenient access to system-call arguments and
return values in a single probe.

- Add a set of per-event 'bsm' probes, which provide access to particular
auditable events at the time of BSM record generation in the audit
worker thread. These probes have access to the in-kernel audit_record
data structure and BSM representation as would be written to a trail
file or audit pipe -- i.e., asynchronously in the audit worker thread.

DTrace probe arguments consist of the name of the audit event (to support
future mechanisms of instrumenting multiple events via a single probe --
e.g., using classes), a pointer to the in-kernel audit record, and an
optional pointer to the BSM data and its length. For human convenience,
upper-case audit event names (AUE_...) are converted to lower case in
DTrace.

DTrace scripts can now cause additional audit-based data to be collected
on system calls, and inspect internal and BSM representations of the data.
They do not affect data captured in the audit trail or audit pipes
configured in the system. auditd(8) must be configured and running in
order to provide a database of event information, as well as other audit
configuration parameters (e.g., to capture command-line arguments or
environmental variables) for the provider to operate.

Reviewed by: gnn, jonathan, markj
Sponsored by: DARPA, AFRL
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D10149

Busy the mount point which is the owner of the audit vnode, around
audit_record_write(). This is important so that VFS_STATFS() is not
done on the NULL or freed mp and the check for free space is
consistent with the vnode used for write.

Add vn_start_write() braces around VOP_FSYNC() calls on the audit vnode.

Move repeated code to fsync vnode and panic to the helper
audit_worker_sync_vp().

Reviewed by: rwatson
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

IFp4 @208383:

Currently when we discover that trail file is greater than configured
limit we send AUDIT_TRIGGER_ROTATE_KERNEL trigger to the auditd daemon
once. If for some reason auditd didn't rotate trail file it will never
be rotated.

Change it by sending the trigger when trail file size grows by the
configured limit. For example if the limit is 1MB, we will send trigger
on 1MB, 2MB, 3MB, etc.

This is also needed for the auditd change that will be committed soon
where auditd may ignore the trigger - it might be ignored if kernel
requests the trail file to be rotated too quickly (often than once a second)
which would result in overwriting previous trail file.

Sponsored by: FreeBSD Foundation (auditdistd)
MFC after: 2 weeks

IFp4 @208382:

Currently on each record write we call VFS_STATFS() to get available space
on the file system as well as VOP_GETATTR() to get trail file size.

We can assume that trail file is only updated by the audit worker, so instead
of asking for file size on every write, get file size on trail switch only
(it should be zero, but it's not expensive) and use global variable audit_size
protected by the audit worker lock to keep track of trail file's size.

This eliminates VOP_GETATTR() call for every write. VFS_STATFS() is satisfied
from in-memory data (mount->mnt_stat), so shouldn't be expensive.

Sponsored by: FreeBSD Foundation (auditdistd)
MFC after: 2 weeks

IFp4 @208381:

For VOP_GETATTR() we just need vnode to be shared-locked.

Sponsored by: FreeBSD Foundation (auditdistd)
MFC after: 2 weeks

Remove the support for using non-mpsafe filesystem modules.

In particular, do not lock Giant conditionally when calling into the
filesystem module, remove the VFS_LOCK_GIANT() and related
macros. Stop handling buffers belonging to non-mpsafe filesystems.

The VFS_VERSION is bumped to indicate the interface change which does
not result in the interface signatures changes.

Conducted and reviewed by: attilio
Tested by: pho

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Remove the thread argument from the FSD (File-System Dependent) parts of
the VFS. Now all the VFS_* functions and relating parts don't want the
context as long as it always refers to curthread.

In some points, in particular when dealing with VOPs and functions living
in the same namespace (eg. vflush) which still need to be converted,
pass curthread explicitly in order to retain the old behaviour.
Such loose ends will be fixed ASAP.

While here fix a bug: now, UFS_EXTATTR can be compiled alone without the
UFS_EXTATTR_AUTOSTART option.

VFS KPI is heavilly changed by this commit so thirdy parts modules needs
to be recompiled. Bump __FreeBSD_version in order to signal such
situation.

Wrap sx locking of the audit worker sleep lock in macros, update comments.

MFC after: 2 months
Sponsored by: Apple, Inc.

- Import the HEAD csup code which is the basis for the cvsmode work.

Decontextualize the couplet VOP_GETATTR / VOP_SETATTR as the passed thread
was always curthread and totally unuseful.

Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com>

Further synchronization of copyrights, licenses, white space, etc from
Apple and from the OpenBSM vendor tree.

Obtained from: Apple Inc., TrustedBSD Project
MFC after: 3 days

Generally avoid <space><tab> as a white space anomoly.

Obtained from: Apple Inc.
MFC after: 3 days

In preparation to sync Apple and FreeBSD versions of security audit,
pick up the Apple Computer -> Apple change in their copyright and
license templates.

Obtained from: Apple Inc.
MFC after: 3 days

When the file-system containing the audit log file is running low on
disk space a warning is printed. Make this warning a bit more
informative.

Approved by: rwatson

Don't use LK_DRAIN before calling VOP_FSYNC() in the two further
panic cases for audit trail failure -- this doesn't contribute
anything, and might arguably be wrong.

MFC after: 1 week
Requested by: attilio

Don't use LK_DRAIN before calling VOP_FSYNC() in the panic case for
audit trail failure -- this doesn't contribute anything, and might
arguably be wrong.

MFC after: 1 week
Requested by: attilio

Use __FBSDID() for $FreeBSD$ IDs in the audit code.

MFC after: 3 days

Remove XXX to remind me to check the free space calculation, which to my
eyes appears right following a check.

MFC after: 3 days

Rename globally exposed symbol send_trigger() to audit_send_trigger().

MFC after: 1 month

Replace somewhat awkward audit trail rotation scheme, which involved the
global audit mutex and condition variables, with an sx lock which protects
the trail vnode and credential while in use, and is acquired by the system
call code when rotating the trail. Previously, a "message" would be sent
to the kernel audit worker, which did the rotation, but the new code is
simpler and (hopefully) less error-prone.

Obtained from: TrustedBSD Project
MFC after: 1 month

VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used in
conjuction with 'thread' argument passing which is always curthread.
Remove the unuseful extra-argument and pass explicitly curthread to lower
layer functions, when necessary.

KPI results broken by this change, which should affect several ports, so
version bumping and manpage update will be further committed.

Tested by: kris, pho, Diego Sardina <siarodx at gmail dot com>

vn_lock() is currently only used with the 'curthread' passed as argument.
Remove this argument and pass curthread directly to underlying
VOP_LOCK1() VFS method. This modify makes the code cleaner and in
particular remove an annoying dependence helping next lockmgr() cleanup.
KPI results, obviously, changed.

Manpage and FreeBSD_version will be updated through further commits.

As a side note, would be valuable to say that next commits will address
a similar cleanup about VFS methods, in particular vop_lock1 and
vop_unlock.

Tested by: Diego Sardina <siarodx at gmail dot com>,
Andrea Di Pasquale <whyx dot it at gmail dot com>

Rename the kthread_xxx (e.g. kthread_create()) calls
to kproc_xxx as they actually make whole processes.
Thos makes way for us to add REAL kthread_create() and friends
that actually make theads. it turns out that most of these
calls actually end up being moved back to the thread version
when it's added. but we need to make this cosmetic change first.

I'd LOVE to do this rename in 7.0 so that we can eventually MFC the
new kthread_xxx() calls.

Clean up audit comments--formatting, spelling, etc.

Remove AUDIT_PRINTF() debugging statements and definition; clean up or
remove associated comments.

Slip audit_file_rotate_wait assignment in audit_rotate_vnode() before
the drop of the global audit mutex.

Obtained from: TrustedBSD Project

Short name of kernel audit worker thread from "audit_worker" to "audit".

MFC after: 3 days

Update a number of comments:

- Replace XXX with Note: in several cases where observations are made about
future functionality rather than problems or bugs.

- Remove an XXX comment about byte order and au_to_ip() -- IP headers must
be submitted in network byte order. Add a comment to this effect.

- Mention that we don't implement select/poll for /dev/audit.

Obtained from: TrustedBSD Project

Rework the way errors are handled with respect to how audit records are
written to the audit trail file:

- audit_record_write() now returns void, and all file system specific
error handling occurs inside this function. This pushes error handling
complexity out of the record demux routine that hands off to both the
trail and audit pipes, and makes trail behavior more consistent with
pipes as a record destination.

- Rate limit kernel printfs associated with running low on space. Rate
limit audit triggers for low space. Rate limit printfs for fail stop
events. Rate limit audit worker write error printfs.

- Document in detail the types of limits and space checks we perform, and
combine common cases.

This improves the audit subsystems tolerance to low space conditions by
avoiding toasting the console with printfs are waking up the audit daemon
continuously.

MFC after: 3 days
Obtained from: TrustedBSD Project

Merge OpenBSM 1.0 alpha 11 changes into src/sys/bsm and src/sys/security;
primarily, add new event identifiers and update trigger names.

Obtained from: TrustedBSD Project

Correct a slight regression which was introduced with the implementation of
audit pipes. If the kernel record was not selected for the trail or the pipe,
any user supplied record attached to it would be tossed away, resulting in
otherwise selected events being lost.

- Introduce two new masks: AR_PRESELECT_USER_TRAIL AR_PRESELECT_USER_PIPE,
currently we have AR_PRESELECT_TRAIL and AR_PRESELECT_PIPE, which tells
the audit worker that we are interested in the kernel record, with
the additional masks we can determine if either the pipe or trail is
interested in seeing the kernel or user record.

- In audit(2), we unconditionally set the AR_PRESELECT_USER_TRAIL and
AR_PRESELECT_USER_PIPE masks under the assumption that userspace has
done the preselection [1].

Currently, there is work being done that allows the kernel to parse and
preselect user supplied records, so in the future preselection could occur
in either layer. But there is still a few details to work out here.

[1] At some point we need to teach au_preselect(3) about the interests of
all the individual audit pipes.

This is a RELENG_6 candidate.

Reviewed by: rwatson
Obtained from: TrustedBSD Project
MFC after: 1 week

Extract pointer value for mnt_stat from vp after the NULL check, not
before.

Coverity ID: 134394
Found with: Coverity Prevent (tm)

Introduce support for per-audit pipe preselection independent from the
global audit trail configuration. This allows applications consuming
audit trails to specify parameters for which audit records are of
interest, including selecting records not required by the global trail.
Allowing application interest specification without changing the global
configuration allows intrusion detection systems to run without
interfering with global auditing or each other (if multiple are
present). To implement this:

- Kernel audit records now carry a flag to indicate whether they have
been selected by the global trail or by the audit pipe subsystem,
set during record commit, so that this information is available
after BSM conversion when delivering the BSM to the trail and audit
pipes in the audit worker thread asynchronously. Preselection by
either record target will cause the record to be kept.

- Similar changes to preselection when the audit record is created
when the system call is entering: consult both the global trail and
pipes.

- au_preselect() now accepts the class in order to avoid repeatedly
looking up the mask for each preselection test.

- Define a series of ioctls that allow applications to specify whether
they want to track the global trail, or program their own
preselection parameters: they may specify their own flags and naflags
masks, similar to the global masks of the same name, as well as a set
of per-auid masks. They also set a per-pipe mode specifying whether
they track the global trail, or user their own -- the door is left
open for future additional modes. A new ioctl is defined to allow a
user process to flush the current audit pipe queue, which can be used
after reprogramming pre-selection to make sure that only records of
interest are received in future reads.

- Audit pipe data structures are extended to hold the additional fields
necessary to support preselection. By default, audit pipes track the
global trail, so "praudit /dev/auditpipe" will track the global audit
trail even though praudit doesn't program the audit pipe selection
model.

- Comment about the complexities of potentially adding partial read
support to audit pipes.

By using a set of ioctls, applications can select which records are of
interest, and toggle the preselection mode.

Obtained from: TrustedBSD Project

No longer unconditionally drain the audit record queue if there is
not an active audit trail: instead, continue to iterate through
each record in case an audit pipe is interested.

Obtained from: TrustedBSD Project

Pull BSM conversion logic out of audit_record_write(), as well as
knowledge of user vs. kernel audit records into
audit_worker_process_record(). This largely confines vnode
knowledge to audit_record_write(), but avoids that logic knowing
about BSM as opposed to byte streams. This will allow us to
improve our ability to support real-time audit stream processing
by audit pipe consumers while auditing is disabled, but this
support is not yet complete.

Obtained from: TrustedBSD Project

Assert audit mtx in audit_worker_drain().

Break out logic to call audit_record_write() and handle error
conditions into audit_worker_process_record(). This will be the
future home of some logic now present in audit_record_write()
also.

Obtained from: TrustedBSD Project

Use struct kaudit_queue instead of a hand-crafted queue type for
audit records in the audit_worker thread.

Obtained from: TrustedBSD Project

Rename audit_cv to audit_worker_cv, as it wakes up the audit
worker.

Rename audit_commit_cv to audit_watermark_cv, since it is there to
wake up threads waiting on hitting the low watermark. Describe
properly in comment.

Obtained from: TrustedBSD Project

Merge Perforce change 93581 from TrustedBSD audit3 branch:

Mega-style patch.

Obtained from: TrustedBSD Project

Merge Perforce changes 93512, 93514, 93515 from TrustedBSD audit3
branch:

Integrate audit.c to audit_worker.c, so as to migrate the worker
thread implementation to its own .c file.

Populate audit_worker.c using parts now removed from audit.c:

- Move audit rotation global variables.
- Move audit_record_write(), audit_worker_rotate(),
audit_worker_drain(), audit_worker(), audit_rotate_vnode().
- Create audit_worker_init() from relevant parts of audit_init(),
which now calls this routine.
- Recreate audit_free(), which wraps uma_zfree() so that
audit_record_zone can be static to audit.c.
- Unstaticize various types and variables relating to the audit
record queue so that audit_worker can get to them. We may want
to wrap these in accessor methods at some point.
- Move AUDIT_PRINTF() to audit_private.h.

Addition of audit_worker.c to kernel configuration, missed in
earlier submit.

Obtained from: TrustedBSD Project