sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

vfs: retire KERN_VNODE

It got disabled in 2003:

commit acb18acfec97aa7fe26ff48f80a5c3f89c9b542d
Author: Poul-Henning Kamp <phk@FreeBSD.org>
Date: Sun Feb 23 18:09:05 2003 +0000

Bracket the kern.vnode sysctl in #ifdef notyet because it results
in massive locking issues on diskless systems.

It is also not clear that this sysctl is non-dangerous in its
requirements for locked down memory on large RAM systems.

There does not seem to be practical use for it and the disabled routine
does not work anyway.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D39127

cache: drop the always curthread argument from reverse lookup routines

Note VOP_VPTOCNP keeps getting it as temporary compatibility for zfs.

Tested by: pho

audit_canon_path_vp: don't panic if cdir == NULL

cdir may have simply failed to resolve (e.g. fget_cap failure in namei
leading to NULL dp passed to AUDIT_ARG_UPATH*_VP); restore the pre-rS358191
behavior of setting cpath[0] = '\0' and bailing out instead of panicking.

This was found by inadvertently running the libc/c063 tests with auditing
enabled, resulting in a panic.

Reviewed by: mjg (committed version actually his)
Differential Revision: https://reviews.freebsd.org/D24445

fd: move vnodes out of filedesc into a dedicated structure

The new structure is copy-on-write. With the assumption that path lookups are
significantly more frequent than chdirs and chrooting this is a win.

This provides stable root and jail root vnodes without the need to reference
them on lookup, which in turn means less work on globally shared structures.
Note this also happens to fix a bug where jail vnode was never referenced,
meaning subsequent access on lookup could run into use-after-free.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23884

audit: provide audit_canon_path variant which accepts vnodes

audit: simplify path resolving logic

audit: rely on use count instead of hold count in audit_canon_path

audit(4): fix a typo in a comment

no functional change

sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Break audit_bsm_klib.c into two files: one (audit_bsm_klib.c)
retaining various utility functions used during BSM generation,
and a second (audit_bsm_db.c) that contains the various in-kernel
databases supporting various audit activities (the class and
event-name tables).

(No functional change is intended.)

Obtained from: TrustedBSD Project
MFC after: 3 weeks
Sponsored by: DARPA, AFRL

When handling msgsys(2), semsys(2), and shmsys(2) multiplex system calls,
map the 'which' argument into a suitable audit event identifier for the
specific operation requested.

Obtained from: TrustedBSD Project
MFC after: 3 weeks
Sponsored by: DARPA, AFRL

Add an experimental DTrace audit provider, which allows users of DTrace to
instrument security event auditing rather than relying on conventional BSM
trail files or audit pipes:

- Add a set of per-event 'commit' probes, which provide access to
particular auditable events at the time of commit in system-call return.
These probes gain access to audit data via the in-kernel audit_record
data structure, providing convenient access to system-call arguments and
return values in a single probe.

- Add a set of per-event 'bsm' probes, which provide access to particular
auditable events at the time of BSM record generation in the audit
worker thread. These probes have access to the in-kernel audit_record
data structure and BSM representation as would be written to a trail
file or audit pipe -- i.e., asynchronously in the audit worker thread.

DTrace probe arguments consist of the name of the audit event (to support
future mechanisms of instrumenting multiple events via a single probe --
e.g., using classes), a pointer to the in-kernel audit record, and an
optional pointer to the BSM data and its length. For human convenience,
upper-case audit event names (AUE_...) are converted to lower case in
DTrace.

DTrace scripts can now cause additional audit-based data to be collected
on system calls, and inspect internal and BSM representations of the data.
They do not affect data captured in the audit trail or audit pipes
configured in the system. auditd(8) must be configured and running in
order to provide a database of event information, as well as other audit
configuration parameters (e.g., to capture command-line arguments or
environmental variables) for the provider to operate.

Reviewed by: gnn, jonathan, markj
Sponsored by: DARPA, AFRL
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D10149

Introduce an audit event identifier -> audit event name mapping
database in the kernel audit implementation, similar the exist
class mapping database. This will be used by the DTrace audit
provider to map audit event identifiers originating in the
system-call table back into strings for the purposes of setting
probe names. The database is initialised and maintained by
auditd(8), which reads values in from the audit_events
configuration file, and then manages them using the A_GETEVENT
and A_SETEVENT auditon(2) operations.

Obtained from: TrustedBSD Project
Sponsored by: DARPA, AFRL
MFC after: 3 weeks

Remove slightly used const values that can be replaced with nitems().

Suggested by: jhb

fd: make 'rights' a manadatory argument to fget* functions

audit: plug FILEDESC_LOCK leak in audit_canon_path.

MFC after: 3 days

There is no sysctl with the MIB { CTL_KERN, KERN_MAXID }.

MFC after: 2 weeks

Handle cases where capability rights are not provided.

Reported by: kib

IFp4 @219811:

VFS is now fully MPSAFE, fix compilation.

IFp4 @208451:

Fix path handling for *at() syscalls.

Before the change directory descriptor was totally ignored,
so the relative path argument was appended to current working
directory path and not to the path provided by descriptor, thus
wrong paths were stored in audit logs.

Now that we use directory descriptor in vfs_lookup, move
AUDIT_ARG_UPATH1() and AUDIT_ARG_UPATH2() calls to the place where
we hold file descriptors table lock, so we are sure paths will
be resolved according to the same directory in audit record and
in actual operation.

Sponsored by: FreeBSD Foundation (auditdistd)
Reviewed by: rwatson
MFC after: 2 weeks

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Replace sbuf_overflowed() with sbuf_error(), which returns any error
code associated with overflow or with the drain function. While this
function is not expected to be used often, it produces more information
in the form of an errno that sbuf_overflowed() did.

Audit file descriptors passed to fooat(2) system calls, which are used
instead of the root/current working directory as the starting point for
lookups. Up to two such descriptors can be audited. Add audit record
BSM encoding for fooat(2).

Note: due to an error in the OpenBSM 1.1p1 configuration file, a
further change is required to that file in order to fix openat(2)
auditing.

Approved by: re (kib)
Reviewed by: rdivacky (fooat(2) portions)
Obtained from: TrustedBSD Project
MFC after: 1 month

Change various routines that are responsible for transforming audit
event IDs based on arguments to return au_event_t rather than int.

Obtained from: TrustedBSD Project
Sponsored by: Apple, Inc.
MFC after: 1 week

Don't lock the vnode around calls to vn_fullpath().

Reviewed by: csjp, rwatson

Protect the event->class lookup database using an rwlock instead of a
mutex, as it's rarely changed but frequently accessed read-only from
multiple threads, so a potentially significant source of contention.

MFC after: 1 month
Sponsored by: Apple, Inc.

- Import the HEAD csup code which is the basis for the cvsmode work.

Use sbuf_putc instead of sbuf_cat. This makes more sense, since we are
appending a single character to the buffer.

MFC after: 2 weeks

Currently, BSM audit pathname token generation for chrooted or jailed
processes are not producing absolute pathname tokens. It is required
that audited pathnames are generated relative to the global root mount
point. This modification changes our implementation of audit_canon_path(9)
and introduces a new function: vn_fullpath_global(9) which performs a
vnode -> pathname translation relative to the global mount point based
on the contents of the name cache. Much like vn_fullpath,
vn_fullpath_global is a wrapper function which called vn_fullpath1.

Further, the string parsing routines have been converted to use the
sbuf(9) framework. This change also removes the conditional acquisition
of Giant, since the vn_fullpath1 method will not dip into file system
dependent code.

The vnode locking was modified to use vhold()/vdrop() instead the vref()
and vrele(). This will modify the hold count instead of modifying the
user count. This makes more sense since it's the kernel that requires
the reference to the vnode. This also makes sure that the vnode does not
get recycled we hold the reference to it. [1]

Discussed with: rwatson
Reviewed by: kib [1]
MFC after: 2 weeks

Further synchronization of copyrights, licenses, white space, etc from
Apple and from the OpenBSM vendor tree.

Obtained from: Apple Inc., TrustedBSD Project
MFC after: 3 days

Minor white space tweak.

Obtained from: Apple Inc.
MFC after: 3 days

Use #define<tab> rather than #define<space>.

Obtained from: Apple Inc.
MFC after: 3 days

In preparation to sync Apple and FreeBSD versions of security audit,
pick up the Apple Computer -> Apple change in their copyright and
license templates.

Obtained from: Apple Inc.
MFC after: 3 days

Use __FBSDID() for $FreeBSD$ IDs in the audit code.

MFC after: 3 days

Add audit_prefixes to two more globally visible functions in the Audit
implementation.

MFC after: 1 month

Rename several audit functions in the global kernel symbol namespace to
have audit_ on the front:

- canon_path -> audit_canon_path
- msgctl_to_event -> audit_msgctl_to_event
- semctl_to_event -> audit_semctl_to_event

MFC after: 1 month

vn_lock() is currently only used with the 'curthread' passed as argument.
Remove this argument and pass curthread directly to underlying
VOP_LOCK1() VFS method. This modify makes the code cleaner and in
particular remove an annoying dependence helping next lockmgr() cleanup.
KPI results, obviously, changed.

Manpage and FreeBSD_version will be updated through further commits.

As a side note, would be valuable to say that next commits will address
a similar cleanup about VFS methods, in particular vop_lock1 and
vop_unlock.

Tested by: Diego Sardina <siarodx at gmail dot com>,
Andrea Di Pasquale <whyx dot it at gmail dot com>

Replace use of AU_NULL with 0 when no audit classes are in use; this
supports the removal of hard-coded audit class constants in OpenBSM
1.0. All audit classes are now dynamically configured via the
audit_class database.

Obtained from: TrustedBSD Project

Clean up audit comments--formatting, spelling, etc.

Replace custom file descriptor array sleep lock constructed using a mutex
and flags with an sxlock. This leads to a significant and measurable
performance improvement as a result of access to shared locking for
frequent lookup operations, reduced general overhead, and reduced overhead
in the event of contention. All of these are imported for threaded
applications where simultaneous access to a shared file descriptor array
occurs frequently. Kris has reported 2x-4x transaction rate improvements
on 8-core MySQL benchmarks; smaller improvements can be expected for many
workloads as a result of reduced overhead.

- Generally eliminate the distinction between "fast" and regular
acquisisition of the filedesc lock; the plan is that they will now all
be fast. Change all locking instances to either shared or exclusive
locks.

- Correct a bug (pointed out by kib) in fdfree() where previously msleep()
was called without the mutex held; sx_sleep() is now always called with
the sxlock held exclusively.

- Universally hold the struct file lock over changes to struct file,
rather than the filedesc lock or no lock. Always update the f_ops
field last. A further memory barrier is required here in the future
(discussed with jhb).

- Improve locking and reference management in linux_at(), which fails to
properly acquire vnode references before using vnode pointers. Annotate
improper use of vn_fullpath(), which will be replaced at a future date.

In fcntl(), we conservatively acquire an exclusive lock, even though in
some cases a shared lock may be sufficient, which should be revisited.
The dropping of the filedesc lock in fdgrowtable() is no longer required
as the sxlock can be held over the sleep operation; we should consider
removing that (pointed out by attilio).

Tested by: kris
Discussed with: jhb, kris, attilio, jeff

Add a witness sleep warning to canon_path(), which invokes vput() and hence
may perform an unbounded sleep. Remove an XXX comment suggesting that one
be added.

Obtained from: TrustedBSD Project

Introduce support for per-audit pipe preselection independent from the
global audit trail configuration. This allows applications consuming
audit trails to specify parameters for which audit records are of
interest, including selecting records not required by the global trail.
Allowing application interest specification without changing the global
configuration allows intrusion detection systems to run without
interfering with global auditing or each other (if multiple are
present). To implement this:

- Kernel audit records now carry a flag to indicate whether they have
been selected by the global trail or by the audit pipe subsystem,
set during record commit, so that this information is available
after BSM conversion when delivering the BSM to the trail and audit
pipes in the audit worker thread asynchronously. Preselection by
either record target will cause the record to be kept.

- Similar changes to preselection when the audit record is created
when the system call is entering: consult both the global trail and
pipes.

- au_preselect() now accepts the class in order to avoid repeatedly
looking up the mask for each preselection test.

- Define a series of ioctls that allow applications to specify whether
they want to track the global trail, or program their own
preselection parameters: they may specify their own flags and naflags
masks, similar to the global masks of the same name, as well as a set
of per-auid masks. They also set a per-pipe mode specifying whether
they track the global trail, or user their own -- the door is left
open for future additional modes. A new ioctl is defined to allow a
user process to flush the current audit pipe queue, which can be used
after reprogramming pre-selection to make sure that only records of
interest are received in future reads.

- Audit pipe data structures are extended to hold the additional fields
necessary to support preselection. By default, audit pipes track the
global trail, so "praudit /dev/auditpipe" will track the global audit
trail even though praudit doesn't program the audit pipe selection
model.

- Comment about the complexities of potentially adding partial read
support to audit pipes.

By using a set of ioctls, applications can select which records are of
interest, and toggle the preselection mode.

Obtained from: TrustedBSD Project

Check to see if the rootdir is the same as the current working directory.
If it is, and the pathname was relative, do not separate the componenets
with a '/' character.

Obtained from: TrustedBSD Project

Merge Perforce change 93581 from TrustedBSD audit3 branch:

Mega-style patch.

Obtained from: TrustedBSD Project

Import kernel audit framework:

- Management of audit state on processes.
- Audit system calls to configure process and system audit state.
- Reliable audit record queue implementation, audit_worker kernel
thread to asynchronously store records on disk.
- Audit event argument.
- Internal audit data structure -> BSM audit trail conversion library.
- Audit event pre-selection.
- Audit pseudo-device permitting kernel->user upcalls to notify auditd
of kernel audit events.

Much work by: wsalamon
Obtained from: TrustedBSD Project, Apple Computer, Inc.