MFC r281436 (by mjg@):

fd: remove filedesc argument from fdclose

Just accept a thread instead. This makes it consistent with fdalloc.

No functional changes.

MFH: r306306

Give the user a clue as to which process hit maxfiles.

Sponsored by: Panzura

MFC r298819:

sys/kern: spelling fixes in comments.

MFC 287442,287537,288944:
Fix corruption of coredumps due to procstat notes changing size during
coredump generation. The changes in r287442 required some reworking
since the 'fo_fill_kinfo' file op does not exist in stable/10.

287442:
Detect badly behaved coredump note helpers

Coredump notes depend on being able to invoke dump routines twice; once
in a dry-run mode to get the size of the note, and another to actually
emit the note to the corefile.

When a note helper emits a different length section the second time
around than the length it requested the first time, the kernel produces
a corrupt coredump.

NT_PROCSTAT_FILES output length, when packing kinfo structs, is tied to
the length of filenames corresponding to vnodes in the process' fd table
via vn_fullpath. As vnodes may move around during dump, this is racy.

So:

- Detect badly behaved notes in putnote() and pad underfilled notes.

- Add a fail point, debug.fail_point.fill_kinfo_vnode__random_path to
exercise the NT_PROCSTAT_FILES corruption. It simply picks random
lengths to expand or truncate paths to in fo_fill_kinfo_vnode().

- Add a sysctl, kern.coredump_pack_fileinfo, to allow users to
disable kinfo packing for PROCSTAT_FILES notes. This should avoid
both FILES note corruption and truncation, even if filenames change,
at the cost of about 1 kiB in padding bloat per open fd. Document
the new sysctl in core.5.

- Fix note_procstat_files to self-limit in the 2nd pass. Since
sometimes this will result in a short write, pad up to our advertised
size. This addresses note corruption, at the risk of sometimes
truncating the last several fd info entries.

- Fix NT_PROCSTAT_FILES consumers libutil and libprocstat to grok the
zero padding.

287537:
Follow-up to r287442: Move sysctl to compiled-once file

Avoid duplicate sysctl nodes.

288944:
Fix core corruption caused by race in note_procstat_vmmap

This fix is spiritually similar to r287442 and was discovered thanks to
the KASSERT added in that revision.

NT_PROCSTAT_VMMAP output length, when packing kinfo structs, is tied to
the length of filenames corresponding to vnodes in the process' vm map
via vn_fullpath. As vnodes may move during coredump, this is racy.

We do not remove the race, only prevent it from causing coredump
corruption.

- Add a sysctl, kern.coredump_pack_vmmapinfo, to allow users to disable
kinfo packing for PROCSTAT_VMMAP notes. This avoids VMMAP corruption
and truncation, even if names change, at the cost of up to PATH_MAX
bytes per mapped object. The new sysctl is documented in core.5.

- Fix note_procstat_vmmap to self-limit in the second pass. This
addresses corruption, at the cost of sometimes producing a truncated
result.

- Fix PROCSTAT_VMMAP consumers libutil (and libprocstat, via copy-paste)
to grok the new zero padding.

Approved by: re (gjb)

MFC r285134 (by mjg):
fd: de-k&r-ify functions + some whitespace fixes

MFC r285269:
Handle copyout for the fcntl(F_OGETLK) using oflock structure.

MFC r282213:

Add kern.racct.enable tunable and RACCT_DISABLED config option.
The point of this is to be able to add RACCT (with RACCT_DISABLED)
to GENERIC, to avoid having to rebuild the kernel to use rctl(8).

MFC r282901:

Build GENERIC with RACCT/RCTL support by default. Note that it still
needs to be enabled by adding "kern.racct.enable=1" to /boot/loader.conf.

Note those two are MFC-ed together, because the latter one changes the
name of RACCT_DISABLED option to RACCT_DEFAULT_TO_DISABLED. Should have
committed the renaming separately...

Relnotes: yes
Sponsored by: The FreeBSD Foundation

Merge r263233 from HEAD to stable/10:

Update kernel inclusions of capability.h to use capsicum.h instead; some
further refinement is required as some device drivers intended to be
portable over FreeBSD versions rely on __FreeBSD_version to decide whether
to include capability.h.

Sponsored by: Google, Inc.

MFC r272566:
Convert -1 from sbuf_bcat() to ENOMEM.

MFC r263710, r273377, r273378, r273423 and r273455:

- De-vnet hash sizes and hash masks.
- Fix multiple issues related to arguments passed to SYSCTL macros.

Sponsored by: Mellanox Technologies

MFC r269023,r272503,r272505,r272523,r272567,r272569,r272574

Prepare fget_unlocked for reading fd table only once.

Some capsicum functions accept fdp + fd and lookup fde based on that.
Add variants which accept fde.

===============================

Add sequence counters with memory barriers.

Current implementation is somewhat simplistic and hackish,
will be improved later after possible memory barrier overhaul.

===============================

Plug capability races.

fp and appropriate capability lookups were not atomic, which could result in
improper capabilities being checked.

This could result either in protection bypass or in a spurious ENOTCAPABLE.

Make fp + capability check atomic with the help of sequence counters.

===============================

Put and #ifdef _KERNEL around the #include for opt_capsicum.h to
hopefully allow the build to finish after r272505.

===============================

filedesc: fix up breakage introduced in 272505

Include sequence counter supports incoditionally [1]. This fixes reprted build
problems with e.g. nvidia driver due to missing opt_capsicum.h.

Replace fishy looking sizeof with offsetof. Make fde_seq the last member in
order to simplify calculations.

===============================

Keep struct filedescent comments within 80-char limit.

===============================

seq_t needs to be visible to userspace

MFC r272132:
Fix fcntl(2) compat32 after r270691.

Approved by: re (glebius)

MFC r270993:

Fix up proc_realparent to always return correct process.

Prior to the change it would always return initproc for non-traced processes.

This fixes a regression in inferior().

Approved by: re (marius)

MFC r268505, r268507:

Avoid relocking filedesc lock when closing fds during fdp destruction.

Don't call bzero nor fdunused from fdfree for such cases. It would do
unnecessary work and complain that the lock is not taken.

=======

Don't zero fd_nfiles during fdp destruction.

Code trying to take a look has to check fd_refcnt and it is 0 by that time.

This is a follow up to r268505, without this the code would leak memory for
tables bigger than the default.

MFC r268001:
Make fdunshare accept only td parameter.

Proc had to match the thread anyway and 2 parameters were inconsistent
with the rest.

MFC r268000:

Make sure to always clear p_fd for process getting rid of its filetable.

Filetable can be shared with other processes. Previous code failed to
clear the pointer for all but the last process getting rid of the table.
This is mostly cosmetics.

Get rid of 'This should happen earlier' comment. Clearing the pointer in
this place is fine as consumers can reliably check for files availability
by inspecting fd_refcnt and vnodes availabity by NULL-checking them.

MFC r267760:
Tidy up fd-related functions called by do_execve

o assert in each one that fdp is not shared
o remove unnecessary NULL checks - all userspace processes have fdtables
and kernel processes cannot execve
o remove comments about the danger of fd_ofiles getting reallocated - fdtable
is not shared and fd_ofiles could be only reallocated if new fd was about to be
added, but if that was possible the code would already be buggy as setugidsafety
work could be undone

MFC r267755:

Don't take filedesc lock in fdunshare().

We can read refcnt safely and only care if it is equal to 1.

If it could suddenly change from 1 to something bigger the code would be
buggy even in the previous form and transitions from > 1 to 1 are equally racy
and harmless (we copy even though there is no need).

MFC r267710:

fd: replace fd_nfiles with fd_lastfile where appropriate

fd_lastfile is guaranteed to be the biggest open fd, so when the intent
is to iterate over active fds or lookup one, there is no point in looking
beyond that limit.

Few places are left unpatched for now.

MFC r267708:

do_dup: plug redundant adjustment of fd_lastfile

By that time it was already set by fdalloc, or was there in the first place
if fd is replaced.

MFC r265247:
Request a non-exiting process in sysctl_kern_proc_{o,}filedesc

This fixes a race with exit1 freeing p_textvp.

MFC r263530:
Mark the following sysctls as MPSAFE:
kern.file
kern.proc.filedesc
kern.proc.ofiledesc

MFC r263460:
Take filedesc lock only for reading when allocating new fdtable.

Code populating the table does this already.

MFC r262006,r262328:

r262006:
Fix M_FILEDESC leak in fdgrowtable() introduced in r244510.
fdgrowtable() now only reallocates fd_map when necessary.
r262328:
Style.

Approved by: bapt (mentor, implicit)

MFC r262005:

Remove redundant memcpy of fd_ofiles in fdgrowtable()

Approved by: bapt (mentor, implicit)

MFC r262309:

Fix a race between kern_proc_{o,}filedesc_out and fdescfree leading
to use-after-free.

fdescfree proceeds to free file pointers once fd_refcnt reaches 0, but
kern_proc_{o,}filedesc_out only checked for hold count.

MFC r260233:

Plug a memory leak in dup2 when both old and new fd have ioctl caps.

MFC r260232:
Don't check for fd limits in fdgrowtable_exp.

Callers do that already and additional check races with process
decreasing limits and can result in not growing the table at all, which
is currently not handled.

MFC r285134 (by mjg):
fd: de-k&r-ify functions + some whitespace fixes

MFC r285269:
Handle copyout for the fcntl(F_OGETLK) using oflock structure.

MFC r282213:

Add kern.racct.enable tunable and RACCT_DISABLED config option.
The point of this is to be able to add RACCT (with RACCT_DISABLED)
to GENERIC, to avoid having to rebuild the kernel to use rctl(8).

MFC r282901:

Build GENERIC with RACCT/RCTL support by default. Note that it still
needs to be enabled by adding "kern.racct.enable=1" to /boot/loader.conf.

Note those two are MFC-ed together, because the latter one changes the
name of RACCT_DISABLED option to RACCT_DEFAULT_TO_DISABLED. Should have
committed the renaming separately...

Relnotes: yes
Sponsored by: The FreeBSD Foundation

Merge r263233 from HEAD to stable/10:

Update kernel inclusions of capability.h to use capsicum.h instead; some
further refinement is required as some device drivers intended to be
portable over FreeBSD versions rely on __FreeBSD_version to decide whether
to include capability.h.

Sponsored by: Google, Inc.

MFC r272566:
Convert -1 from sbuf_bcat() to ENOMEM.

MFC r263710, r273377, r273378, r273423 and r273455:

- De-vnet hash sizes and hash masks.
- Fix multiple issues related to arguments passed to SYSCTL macros.

Sponsored by: Mellanox Technologies

MFC r269023,r272503,r272505,r272523,r272567,r272569,r272574

Prepare fget_unlocked for reading fd table only once.

Some capsicum functions accept fdp + fd and lookup fde based on that.
Add variants which accept fde.

===============================

Add sequence counters with memory barriers.

Current implementation is somewhat simplistic and hackish,
will be improved later after possible memory barrier overhaul.

===============================

Plug capability races.

fp and appropriate capability lookups were not atomic, which could result in
improper capabilities being checked.

This could result either in protection bypass or in a spurious ENOTCAPABLE.

Make fp + capability check atomic with the help of sequence counters.

===============================

Put and #ifdef _KERNEL around the #include for opt_capsicum.h to
hopefully allow the build to finish after r272505.

===============================

filedesc: fix up breakage introduced in 272505

Include sequence counter supports incoditionally [1]. This fixes reprted build
problems with e.g. nvidia driver due to missing opt_capsicum.h.

Replace fishy looking sizeof with offsetof. Make fde_seq the last member in
order to simplify calculations.

===============================

Keep struct filedescent comments within 80-char limit.

===============================

seq_t needs to be visible to userspace

MFC r272132:
Fix fcntl(2) compat32 after r270691.

Approved by: re (glebius)

MFC r270993:

Fix up proc_realparent to always return correct process.

Prior to the change it would always return initproc for non-traced processes.

This fixes a regression in inferior().

Approved by: re (marius)

MFC r268505, r268507:

Avoid relocking filedesc lock when closing fds during fdp destruction.

Don't call bzero nor fdunused from fdfree for such cases. It would do
unnecessary work and complain that the lock is not taken.

=======

Don't zero fd_nfiles during fdp destruction.

Code trying to take a look has to check fd_refcnt and it is 0 by that time.

This is a follow up to r268505, without this the code would leak memory for
tables bigger than the default.

MFC r268001:
Make fdunshare accept only td parameter.

Proc had to match the thread anyway and 2 parameters were inconsistent
with the rest.

MFC r268000:

Make sure to always clear p_fd for process getting rid of its filetable.

Filetable can be shared with other processes. Previous code failed to
clear the pointer for all but the last process getting rid of the table.
This is mostly cosmetics.

Get rid of 'This should happen earlier' comment. Clearing the pointer in
this place is fine as consumers can reliably check for files availability
by inspecting fd_refcnt and vnodes availabity by NULL-checking them.

MFC r267760:
Tidy up fd-related functions called by do_execve

o assert in each one that fdp is not shared
o remove unnecessary NULL checks - all userspace processes have fdtables
and kernel processes cannot execve
o remove comments about the danger of fd_ofiles getting reallocated - fdtable
is not shared and fd_ofiles could be only reallocated if new fd was about to be
added, but if that was possible the code would already be buggy as setugidsafety
work could be undone

MFC r267755:

Don't take filedesc lock in fdunshare().

We can read refcnt safely and only care if it is equal to 1.

If it could suddenly change from 1 to something bigger the code would be
buggy even in the previous form and transitions from > 1 to 1 are equally racy
and harmless (we copy even though there is no need).

MFC r267710:

fd: replace fd_nfiles with fd_lastfile where appropriate

fd_lastfile is guaranteed to be the biggest open fd, so when the intent
is to iterate over active fds or lookup one, there is no point in looking
beyond that limit.

Few places are left unpatched for now.

MFC r267708:

do_dup: plug redundant adjustment of fd_lastfile

By that time it was already set by fdalloc, or was there in the first place
if fd is replaced.

MFC r265247:
Request a non-exiting process in sysctl_kern_proc_{o,}filedesc

This fixes a race with exit1 freeing p_textvp.

MFC r263530:
Mark the following sysctls as MPSAFE:
kern.file
kern.proc.filedesc
kern.proc.ofiledesc

MFC r263460:
Take filedesc lock only for reading when allocating new fdtable.

Code populating the table does this already.

MFC r262006,r262328:

r262006:
Fix M_FILEDESC leak in fdgrowtable() introduced in r244510.
fdgrowtable() now only reallocates fd_map when necessary.
r262328:
Style.

Approved by: bapt (mentor, implicit)

MFC r262005:

Remove redundant memcpy of fd_ofiles in fdgrowtable()

Approved by: bapt (mentor, implicit)

MFC r262309:

Fix a race between kern_proc_{o,}filedesc_out and fdescfree leading
to use-after-free.

fdescfree proceeds to free file pointers once fd_refcnt reaches 0, but
kern_proc_{o,}filedesc_out only checked for hold count.

MFC r260233:

Plug a memory leak in dup2 when both old and new fd have ioctl caps.

MFC r260232:
Don't check for fd limits in fdgrowtable_exp.

Callers do that already and additional check races with process
decreasing limits and can result in not growing the table at all, which
is currently not handled.