sys/malloc.h: move sys/proc.h MALLOC_DECLAREs here

This avoids the need to explicitly include sys/malloc.h before
sys/proc.h.

Suggested by: kib
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D44766

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .h pattern

Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/

Remove bzero declaration

The kernel was migrated to memset in ba96f37758412151 ("Use __builtin
for various mem* and b* (e.g. bzero) routines.") and there are no
remaining architectures using the file.

malloc is augmented to prevent KMSAN from breaking.

Sponsored by: Rubicon Communications, LLC ("Netgate")

malloc(9): Document/complete aligned variants

Comments on a pending kvmclock driver suggested adding a
malloc_aligned() to complement malloc_domainset_aligned(); add it now,
and document both.

Reviewed by: imp, kib, allanjude (manpages)
Differential Revision: https://reviews.freebsd.org/D31004

stand/kmem_zalloc: panic when a M_WAITOK allocation fails

Malloc() might return NULL, in which case we will panic with a NULL
pointer deref. Make it panic when the allocation fails to preserve the
postcondtion that we never return a non-NULL value.

Reviewed by: tsoome
PR: 249859
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D31106

Implement mallocarray_domainset(9) variant of mallocarray(9).

Reviewed by: kib @
MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

Add a VM flag to prevent reclaim on a failed contig allocation

If a M_WAITOK contig alloc fails, the VM subsystem will try to
reclaim contiguous memory twice before actually failing the
request. On a system with 64GB of RAM I've observed this take
400-500ms before it finally gives up, and I believe that this
will only be worse on systems with even more memory.

In certain contexts this delay is extremely harmful, so add a flag
that will skip reclaim for allocation requests to allow those
paths to opt-out of doing an expensive reclaim.

Sponsored by: Dell Inc
Differential Revision: https://reviews.freebsd.org/D28422
Reviewed by: markj, kib

Implement malloc_domainset_aligned(9).

Change the power-of-two malloc zones to require alignment equal to the
size [*]. Current uma allocator already provides such alignment, so in
fact this change does not change anything except providing future-proof
setup.

Suggested by: markj [*]
Reviewed by: andrew, jah, markj
Tested by: pho
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28147

[malloc] quieten -Werror=missing-braces with malloc.h wth gcc-6.4

This sets off gcc-6.4 to spit out a 'error: missing braces around initializer'
error when compiling this.

Remove it as it isn't needed.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D27183

malloc: retire malloc_last_fail

The routine does not serve any practical purpose.

Memory can be allocated in many other ways and most consumers pass the
M_WAITOK flag, making malloc not fail in the first place.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27143

malloc: retire mt_stats_zone in favor of pcpu_zone_64

Reviewed by: markj, imp
Differential Revision: https://reviews.freebsd.org/D27142

malloc: move malloc_type_internal into malloc_type

According to code comments the original motivation was to allow for
malloc_type_internal changes without ABI breakage. This can be trivially
accomplished by providing spare fields and versioning the struct, as
implemented in the patch below.

The upshots are one less memory indirection on each alloc and disappearance
of mt_zone.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27104

malloc: add a helper returning size allocated for given request

Sample usage: kernel modules can decide whether to stick to malloc or
create their own zone.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27097

malloc: delegate M_EXEC handling to dedicacted routines

It is almost never needed and adds an avoidable branch.

While here do minior clean ups in preparation for larger changes.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27019

Create a standalone version of sys/malloc.h

The ZSTD support for the boot loader will need to include files that
use the kernel's malloc interface. Create a standalone stub version
that's functional enough to allow this to work. There's some
limitations in this interface, and it's not quite a perfect
match. Specifically, M_WAITOK allocations can fail because there's
nothing that can be done we no memory is available.

LinuxKPI: Implement ksize() function.

In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It also maps LinuxKPI ksize() to newly created function.

ksize() function is used by drm-kmod.

Reviewed by: hselasky, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26215

Remove free_domain() and uma_zfree_domain().

These functions were introduced before UMA started ensuring that freed
memory gets placed in domain-local caches. They no longer serve any
purpose since UMA now provides their functionality by default. Remove
them to simplyify the kernel memory allocator interfaces a bit.

Reviewed by: cem, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D25937

Add zfree to zero allocation before free

Key and cookie management typically wants to
avoid information leaks by explicitly zeroing
before free. This routine simplifies that by
permitting consumers to do so without carrying
the size around.

Reviewed by: jeff@, jhb@
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC (Netgate)
Differential Revision: https://reviews.freebsd.org/D22790

The alignment is passed into contigmalloc_domainset in the 7th argument.

KUBSAN was complaining the pointer contigmalloc_domainset returned was
misaligned. Fix this by using the correct argument to find the alignment
in the function signature.

Reported by: KUBSAN
MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Implement the M_NEXTFIT allocation strategy for vmem(9).

This is described in the vmem paper: "directs vmem to use the next free
segment after the one previously allocated." The implementation adds a
new boundary tag type, M_CURSOR, which is linked into the segment list
and precedes the segment following the previous M_NEXTFIT allocation.
The cursor is used to locate the next free segment satisfying the
allocation constraints.

This implementation isn't O(1) since busy tags aren't coalesced, and we
may potentially scan the entire segment list during an M_NEXTFIT
allocation.

Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D17226

Add malloc_domainset(9) and _domainset variants to other allocator KPIs.

Remove malloc_domain(9) and most other _domain KPIs added in r327900.
The new functions allow the caller to specify a general NUMA domain
selection policy, rather than specifically requesting an allocation from
a specific domain. The latter policy tends to interact poorly with
M_WAITOK, resulting in situations where a caller is blocked indefinitely
because the specified domain is depleted. Most existing consumers of
the _domain KPIs are converted to instead use a DOMAINSET_PREF() policy,
in which we fall back to other domains to satisfy the allocation
request.

This change also defines a set of DOMAINSET_FIXED() policies, which
only permit allocations from the specified domain.

Discussed with: gallatin, jeff
Reported and tested by: pho (previous version)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17418

Eliminate false sharing in malloc due to statistic collection

Currently stats are collected in a MAXCPU-sized array which is not
aligned and suffers enormous false-sharing. Fix the problem by
utilizing per-cpu allocation.

The counter(9) API is not used here as it is too incomplete and does
not provide a win over per-cpu zone sized for malloc stats struct. In
particular stats are being reported for each cpu separately by just
copying what is supposed to be an array element for given cpu.

This eliminates significant false-sharing during malloc-heavy tests
e.g. on Skylake. See the review for details.

Reviewed by: markj
Approved by: re (kib)
Differential Revision: https://reviews.freebsd.org/D17289

Remove a redundant #ifdef _KERNEL.

Submitted by: Sebastian Huber <sebastian.huber@embedded-brains.de>
MFC after: 3 days

inpcb: don't gratuitously defer frees

Don't defer frees in sysctl handlers. It isn't necessary
and it just confuses things.
revert: r333911, r334104, and r334125

Requested by: jtl

Make UMA and malloc(9) return non-executable memory in most cases.

Most kernel memory that is allocated after boot does not need to be
executable. There are a few exceptions. For example, kernel modules
do need executable memory, but they don't use UMA or malloc(9). The
BPF JIT compiler also needs executable memory and did use malloc(9)
until r317072.

(Note that a side effect of r316767 was that the "small allocation"
path in UMA on amd64 already returned non-executable memory. This
meant that some calls to malloc(9) or the UMA zone(9) allocator could
return executable memory, while others could return non-executable
memory. This change makes the behavior consistent.)

This change makes malloc(9) return non-executable memory unless the new
M_EXEC flag is specified. After this change, the UMA zone(9) allocator
will always return non-executable memory, and a KASSERT will catch
attempts to use the M_EXEC flag to allocate executable memory using
uma_zalloc() or its variants.

Allocations that do need executable memory have various choices. They
may use the M_EXEC flag to malloc(9), or they may use a different VM
interfact to obtain executable pages.

Now that malloc(9) again allows executable allocations, this change also
reverts most of r317072.

PR: 228927
Reviewed by: alc, kib, markj, jhb (previous version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D15691

malloc: elaborate on r334545 due to frequent questions

While here annotate the NULL check as probably true.

Make Coverity more happy with r334545

Coverity complains about:

if (((flags) & M_WAITOK) || _malloc_item != NULL)

saying:

The expression
1 /* (2 | 0x100) & 2 */ || _malloc_item != NULL
is suspicious because it performs a Boolean operation
on a constant other than 0 or 1.

Although the code is correct, add "!= 0" to make it slightly
more legible and to silence hundreds(?) of Coverity warnings.

Reported by: Coverity
Discussed with: mjg
Sponsored by: Dell EMC

malloc: try to use builtins for zeroing at the callsite

Plenty of allocation sites pass M_ZERO and sizes which are small and known
at compilation time. Handling them internally in malloc loses this information
and results in avoidable calls to memset.

Instead, let the compiler take the advantage of it whenever possible.

Discussed with: jeff

malloc: Add flag for conditionally zeroing for INVARIANTS builds

style: Remove remaining deprecated MALLOC/FREE macros

Mechanically replace uses of MALLOC/FREE with appropriate invocations of
malloc(9) / free(9) (a series of sed expressions). Something like:

* MALLOC(a, b, ... -> a = malloc(...
* FREE( -> free(
* free((caddr_t) -> free(

No functional change.

For now, punt on modifying contrib ipfilter code, leaving a definition of
the macro in its KMALLOC().

Reported by: jhb
Reviewed by: cy, imp, markj, rmacklem
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D14035

malloc(9): Change nominal size to size_t to match standard C

No functional change -- size_t matches unsigned long on all platforms.

Reported by: bde
Discussed with: jhb
Sponsored by: Dell EMC Isilon

Use the __alloc_size2 attribute where relevant.

This follows the documented use in GCC. It is basically only relevant for
calloc(3), reallocarray(3) and mallocarray(9).

Suggested by: Mark Millard

Reference:
https://docs.freebsd.org/cgi/mid.cgi?9DE674C6-EAA3-4E8A-906F-446E74D82FC4

Implement NUMA support in uma(9) and malloc(9). Allocations from specific
domains can be done by the _domain() API variants. UMA also supports a
first-touch policy via the NUMA zone flag.

The slab layer is now segregated by VM domains and is precise. It handles
iteration for round-robin directly. The per-cpu cache layer remains
a mix of domains according to where memory is allocated and freed. Well
behaved clients can achieve perfect locality with no performance penalty.

The direct domain allocation functions have to visit the slab layer and
so require per-zone locks which come at some expense.

Reviewed by: Attilio (a slightly older version)
Tested by: pho
Sponsored by: Netflix, Dell/EMC Isilon

mallocarray(9): panic if the requested allocation would overflow

Additionally, move the overflow check logic out to WOULD_OVERFLOW() for
consumers to have a common means of testing for overflowing allocations.
WOULD_OVERFLOW() should be a secondary check -- on 64-bit platforms, just
because an allocation won't overflow size_t does not mean it is a sane size
to request. Callers should be imposing reasonable allocation limits far,
far, below overflow.

Discussed with: emaste, jhb, kp
Sponsored by: Dell EMC Isilon

Revert r327697:
malloc(9): drop the __result_use_check attribute for the kernel allocator.

My bad: __result_use_check just checks the for the general and we always
want to make sure allocated memory is used, not only checked for nullness.

Add it to reallocf since that was missing.

malloc(9): drop the __result_use_check attribute for the kernel allocator.

The __result_use_check attribute was brought to the kernel malloc in
r281203 for consistency with the userland malloc.

For the case of the M_WAITOK flag, the kernel malloc(), realloc(), and
reallocf() cannot return NULL so in that case the __result_use_check
attribute makes no sense.

We don't have any way of conditionalizing such attributes so just drop it.

MFC after: 3 days

Introduce mallocarray() in the kernel

Similar to calloc() the mallocarray() function checks for integer
overflows before allocating memory.
It does not zero memory, unless the M_ZERO flag is set.

Reviewed by: pfg, vangyzen (previous version), imp (previous version)
Obtained from: OpenBSD
Differential Revision: https://reviews.freebsd.org/D13766

sys: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

Move IPv6 malloc tag definitions into the IPv6 code.

Make use of GCC alloc_align attribute

This lets the compiler know about the alignment of pointers returned
by aligned_alloc(3), posix_memalign(3). and contigmalloc(9)

Currently this is only supported in recent gcc but we are ready to
use it if clang implements it.

Relnotes: yes

Make use of allocation attributes in system headers.

Start using 'alloc_size' attribute in the allocator functions.
This is useful as it helps the compiler generate warnings on suspicious
code and can also enable some small optimizations.

This is based on r281130, which brought similar enhnacements
to the standard libc headers.

Pull in r267961 and r267973 again. Fix for issues reported will follow.

Revert r267961, r267973:

These changes prevent sysctl(8) from returning proper output,
such as:

1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory

Extend the meaning of the CTLFLAG_TUN flag to automatically check if
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.

Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

- Add a general purpose resource allocator, vmem, from NetBSD. It was
originally inspired by the Solaris vmem detailed in the proceedings
of usenix 2001. The NetBSD version was heavily refactored for bugs
and simplicity.
- Use this resource allocator to allocate the buffer and transient maps.
Buffer cache defrags are reduced by 25% when used by filesystems with
mixed block sizes. Ultimately this may permit dynamic buffer cache
sizing on low KVA machines.

Discussed with: alc, kib, attilio
Tested by: pho
Sponsored by: EMC / Isilon Storage Division

- Change contigmalloc() to use the vm_paddr_t type instead of an unsigned
long for specifying a boundary constraint.
- Change bus_dma tags to use bus_addr_t instead of bus_size_t for boundary
constraints.

These allow boundary constraints to be fully expressed for cases where
sizeof(bus_addr_t) != sizeof(bus_size_t). Specifically, it allows a
driver to properly specify a 4GB boundary in a PAE kernel.

Note that this cannot be safely MFC'd without a lot of compat shims due
to KBI changes, so I do not intend to merge it.

Reviewed by: scottl

exclude kmem_alloc'ed ARC data buffers from kernel minidumps on amd64
excluding other allocations including UMA now entails the addition of
a single flag to kmem_alloc or uma zone create

Reviewed by: alc, avg
MFC after: 2 weeks

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Add MALLOC_DEBUG_MAXZONES debug malloc(9) option to use multiple uma
zones for each malloc bucket size. The purpose is to isolate
different malloc types into hash classes, so that any buffer overruns
or use-after-free will usually only affect memory from malloc types in
that hash class. This is purely a debugging tool; by varying the hash
function and tracking which hash class was corrupted, the intersection
of the hash classes from each instance will point to a single malloc
type that is being misused. At this point inspection or memguard(9)
can be used to catch the offending code.

Add MALLOC_DEBUG_MAXZONES=8 to -current GENERIC configuration files.
The suggestion to have this on by default came from Kostik Belousov on
-arch.

This code is based on work by Ron Steinke at Isilon Systems.

Reviewed by: -arch (mostly silence)
Reviewed by: zml
Approved by: zml (mentor)

Garbage collect now-unused struct malloc_type fields, bump __FreeBSD_version
as kernel modules will need to be rebuilt. These fields existed to support
binary compatibility with kernel modules from before the introduction of
libmemstat(3) in FreeBSD 6.x, so they are no longer required.

Add a function attribute called `__malloc_like', which informs gcc
that the annotated function returns a pointer that doesn't alias any
extant pointer. This results in a 50%+ speedup in microbenchmarks such
as the following:

char *cp = malloc(1), *buf = malloc(BUF);
for (i = 0; i < BUF; i++) buf[i] = *cp;

In real programs, your mileage will vary. Note that gcc already
performs this optimization automatically for any function called
`malloc', `calloc', `strdup', or `strndup' unless -fno-builtins is
used.

Revert the removal of the MALLOC and FREE macros from the net80211 code.

Requested by: sam

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

- Import the HEAD csup code which is the basis for the cvsmode work.

Add support for the DTrace malloc provider which can enable probes
on a per-malloc type basis.

Eliminate a stale comment concerning contigmalloc(9).

Improve memguard a bit:
- Provide tunable vm.memguard.desc, so one can specify memory type without
changing the code and recompiling the kernel.
- Allow to use memguard for kernel modules by providing sysctl
vm.memguard.desc, which can be changed to short description of memory
type before module is loaded.
- Move as much memguard code as possible to memguard.c.
- Add sysctl node vm.memguard. and move memguard-specific sysctl there.
- Add malloc_desc2type() function for finding memory type based on its
short description (ks_shortdesc field).
- Memory type can be changed (via vm.memguard.desc sysctl) only if it
doesn't exist (will be loaded later) or when no memory is allocated yet.
If there is allocated memory for the given memory type, return EBUSY.
- Implement two ways of memory types comparsion and make safer/slower the
default.

End the MALLOC_DEFINE macro without the semi-colon, the caller supplies
that.

Spotted by: Flexelint

Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:

- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.

- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.

- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).

This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.

While here:

- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.

- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.

- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.

A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.

Due to a last minute change in the #ifdefing in malloc.h before committing,
a nested include of param.h is required so that MAXCPU is visible to all
consumers of sys/malloc.h. In an earlier version of the patch, the
malloc_type_internal structure was only conditionally visible.

Pointed out by: delphij

Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.

This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:

- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.

- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.

- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.

- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.

- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.

This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.

Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.

Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others

/* -> /*- for license, minor formatting changes

Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.

The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.

The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.

There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.

A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.

When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core

correct typo in comment

- Add vm_paddr_t, a physical address type. This is required for systems
where physical addresses larger than virtual addresses, such as i386s
with PAE.
- Use this to represent physical addresses in the MI vm system and in the
i386 pmap code. This also changes the paddr parameter to d_mmap_t.
- Fix printf formats to handle physical addresses >4G in the i386 memory
detection code, and due to kvtop returning vm_paddr_t instead of u_long.

Note that this is a name change only; vm_paddr_t is still the same as
vm_offset_t on all currently supported platforms.

Sponsored by: DARPA, Network Associates Laboratories
Discussed with: re, phk (cdevsw change)

Make malloc and mbuf allocation mode flags nonoverlapping.

Under INVARIANTS whine if we get incompatible flags.

Submitted by: imp

GC M_STRING, no longer required as strdup() accepts a malloc type.

Requested by: phk

Add an implementation of strdup() to libkern. Allocated memory is of
type M_STRING, now defined in malloc.h. Useful when string parsing
must occur using the kernel strsep() and we want to avoid toasting
the source string.

Obtained from: TrustedBSD Project
Sponsored by: DARPA, Network Associates Laboratories

Back out M_* changes, per decision of the TRB.

Approved by: trb

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

Introduce malloc_last_fail() which returns the number of seconds since
malloc(9) failed last time. This is intended to help code adjust
memory usage to the current circumstances.

A typical use could be:
if (malloc_last_fail() < 60)
reduce_cache_by_one();

Garbage-collected splmem.

Moved the declaration of malloc_mtx to be with the other extern declarations
and not exposed to userland.

Fixed some minor style bugs.

Removed most namespace pollution in this header: don't include <vm/uma.h>;
include <sys/_mutex.h> and its prerequisites instead of <sys/mutex.h> and
its prerequisite.

fix whitespace botch in previous commit.

- Introduce the new M_NOVM option which tells uma to only check the currently
allocated slabs and bucket caches for free items. It will not go ask the vm
for pages. This differs from M_NOWAIT in that it not only doesn't block, it
doesn't even ask.

- Add a new zcreate option ZONE_VM, that sets the BUCKETCACHE zflag. This
tells uma that it should only allocate buckets out of the bucket cache, and
not from the VM. It does this by using the M_NOVM option to zalloc when
getting a new bucket. This is so that the VM doesn't recursively enter
itself while trying to allocate buckets for vm_map_entry zones. If there
are already allocated buckets when we get here we'll still use them but
otherwise we'll skip it.

- Use the ZONE_VM flag on vm map entries and pv entries on x86.

malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the
mallochash. Mallochash is going to go away as soon as I introduce the
kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen
until all users of the malloc api that expect memory to be aligned on the size
of the allocation are fixed.

Convert longs to u_longs in stats. This will hold off wrap arounds for a
while longer.

Parenthesise macro arguments to reduce lint warnings.

Remove malloc_type's ks_limit.

Updated the kmemzones logic such that the ks_size bitmap can be used as an
index into it to report the size of the zone used.

Create the kern.malloc sysctl which replaces the kvm mechanism to report
similar data. This will provide an easy place for statistics aggregation if
malloc_type statistics become per cpu data.

Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing
the malloc buckets.

Change ks_calls from int64_t to uint64_t, since it cannot be negative.

PR: 32342
Submitted by: ryan beasley <ryanb@goddamnbastard.org>
Reviewed by: jeff, Tim J Robbins

Fixed some style bugs in the removal of __P(()). The main ones were
not removing tabs before "__P((", and not outdenting continuation lines
to preserve non-KNF lining up of code with parentheses. Switch to KNF
formatting and/or rewrap the whole prototype in some cases.

Remove __P

This is the first part of the new kernel memory allocator. This replaces
malloc(9) and vm_zone with a slab like allocator.

Reviewed by: arch@

Add realloc() and reallocf(), and make free(NULL, ...) acceptable.

Reviewed by: alfred

- Remove asleep(), await(), and M_ASLEEP.
- Callers of asleep() and await() have been converted to calling tsleep().
The only caller outside of M_ASLEEP was the ata driver, which called both
asleep() and await() with spl-raised, so there was no need for the
asleep() and await() pair. M_ASLEEP was unused.

Reviewed by: jasone, peter

Let M_PANIC go back to the private tree as its intention isn't understood well
for now.

Add M_PANIC flag to the list of available flags passed to malloc().
With this flag set malloc() will panic if memory allocation failed.
This usable only in critical places where failed allocation is fatal.

Reviewed by: peter

Introduce the M_ZERO flag to malloc(9)

Instead of:

foo = malloc(sizeof(foo), M_WAIT);
bzero(foo, sizeof(foo));

You can now (and please do) use:

foo = malloc(sizeof(foo), M_WAIT | M_ZERO);

In the future this will enable us to do idle-time pre-zeroing of
malloc-space.

sys/malloc.h:
Order the SYSINIT() for MALLOC_DEFINE() correctly so that malloc()
doesn't have to waste time initializing itself. The
(SI_SUB_KMEM, SI_ORDER_ANY) order was shared with syscons' SYSINIT()
for scmeminit(), and scmeminit() calls malloc(), so malloc()
initialization was not always complete on the first call to malloc().

kern/kern_malloc.c:
- Removed self-initialization in malloc().
- Removed half-baked sanity check in free(). Trust MALLOC_DEFINE().

Removed support for generating inline code for MALLOC() and FREE()
in the dysfunctional !KMEMSTATS case. This hasn't compiled since
rev.1.31 of kern_malloc.c quietly removed the core of the support
for the !KMEMSTATS case. I fixed it to see if it was worth saving
and found that (as usual) inlining just wasted space and increased
complexity without significantly affecting time, at least for the
lmbench2 micro-benchmark on a Celeron. The space bloat was
surprisingly large - the text size increased from 1700K to 1840K
for a version with the entire malloc() family inlined.

Removed even older garbage (kmemxtob() and btokmemx() macros).

Attempt to deprecate MALLOC() and FREE(). Given current compilers
(gcc-2.x or C99), they don't do anything that (safe) function-like
macros or inline functions named malloc() and free() couldn't do.

Fixed missing casts of macro args in MALLOC() and FREE().

Change #ifdef KERNEL to #ifdef _KERNEL in the public headers. "KERNEL"
is an application space macro and the applications are supposed to be free
to use it as they please (but cannot). This is consistant with the other
BSD's who made this change quite some time ago. More commits to come.

KAME related header files additions and merges.
(only those which don't affect c source files so much)

Reviewed by: cvs-committers
Obtained from: KAME project

Force the "calls" count for malloc types to be 64 bit where it keeps track
of the number of times a particular type has been used. It's rather easy
to overflow. One site I'm looking at seems to do it in a matter of days.
On the Alpha this is a no-op since 'long' is 64 bit already. The sole
user of this interface seems to be vmstat -m and friends which will need
a recompile. The overheads of using a 64 bit int should be pretty light
as the kernel just does "calls++" type operations and that's it.

Fixed some style bugs (mainly disorderd prototypes).

Get rid of MALLOC_INSTANTIATE and MALLOC_MAKE_TYPE(). Just handle the 3
malloc types declared in <sys/malloc.h> like other global malloc types.

$Id$ -> $FreeBSD$

Add a contigfree() as a corollary to contigmalloc() as it's not clear
which free routine to use and people are tempted to use free() (which
doesn't work)

Fix warnings in preparation for adding -Wall -Wcast-qual to the
kernel compile

Renamed M_KERNEL to a more appropriate M_USE_RESERVE.

This is a rather large commit that encompasses the new swapper,
changes to the VM system to support the new swapper, VM bug
fixes, several VM optimizations, and some additional revamping of the
VM code. The specific bug fixes will be documented with additional
forced commits. This commit is somewhat rough in regards to code
cleanup issues.

Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>

Have MALLOC_DECLARE() initialize malloc types explicitly, and have them
removed at module unload (if in a module of course).
However; this introduces a new dependency on <sys/kernel.h> for things
that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h
to the handful of files that need it rather than add an extra include to
sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.

Reviewed by: dyson@freebsd.org (john Dyson), dg@root.com (david greenman)
Submitted by: Kirk McKusick (mcKusick@mckusick.com)
Obtained from: WHistle development tree

Unspammed nested include of <vm/vm_zone.h>.

Removed one `const' from the declaration of `ks_shortdesc'. The pointer
isn't actually const in vmstat.

Fixed pedantic syntax errors caused by trailing semicolons in macro
definitions.

Fixed style bugs and typos in revisions 1.26-1.33.

Moved declaration of M_IOV to a less bogus place. It belongs in
<sys/uio.h>, but it doesn't work there because of header pollution
(<sys/uio.h> is prematurely included by <sys/param.h>).

Some fixes from John Hood:
1) Fix the initialization of malloc structure that changed
due to perf opt.
2) Remove unneeded include.
3) An initialization assert added to malloc.
Submitted by: John Hood <cgull@smoke.marlboro.vt.us>

shuffle structs for better cacheline behavior.

Remove the long description from the in-kernel datastructure.
Put a magic field in there instead, to help catch uninitialized
malloc types.

Last major round (Unless Bruce thinks of somthing :-) of malloc changes.

Distribute all but the most fundamental malloc types. This time I also
remembered the trick to making things static: Put "static" in front of
them.

A couple of finer points by: bde

Distribute and statizice a lot of the malloc M_* types.

Substantial input from: bde

Remove all traces of M_VFSCONF, which were for all practical
purposes unused.

Remove a bunch of unused malloc types.
A couple of potential bogons flagged.
Various prototypes changed.

Make malloc more extensible. The malloc type is now a pointer to
the struct kmemstats that describes the type.

This allows subsystems to declare their malloc types locally
and <sys/malloc.h> doesn't need tweaked everytime somebody
gets an idea. You can even have a type local to a lkm...

I don't know if we really need the longdesc, comments welcome.

TODO: There is a single nit in ext2fs, that will be fixed later,
and I intend to remove all unused malloc types and distribute
the rest closer to their use.

Change the M_NAMEI allocations to use the zone allocator. This change
plus the previous changes to use the zone allocator decrease the useage
of malloc by half. The Zone allocator will be upgradeable to be able
to use per CPU-pools, and has more intelligent usage of SPLs. Additionally,
it has reasonable stats gathering capabilities, while making most calls
inline.

Fixed staticization. buckets[] was staticized but was still declared
extern in <sys/malloc.h> and it should not have been staticized for
the !(KMEMSTATS || DIAGNOSTIC) case.

Fixed the !(KMEMSTATS || DIAGNOSTIC) case. The MALLOC() and FREE()
macros are evil, but code generally doesn't allow for this and some code
involving else clauses did not compile.

Finished staticization.

Fix all areas of the system (or at least all those in LINT) to avoid storing
socket addresses in mbufs. (Socket buffers are the one exception.) A number
of kernel APIs needed to get fixed in order to make this happen. Also,
fix three protocol families which kept PCBs in mbufs to not malloc them
instead. Delete some old compatibility cruft while we're at it, and add
some new routines in the in_cksum family.

Get rid of the ad-hoc memory allocator for vm_map_entries, in lieu of
a simple, clean zone type allocator. This new allocator will also be
used for machine dependent pmap PV entries.

This is an upgrade so that the kernel supports the AIO calls from
POSIX.4. Additionally, there is some initial code that supports LIO.
This code supports AIO/LIO for all types of file descriptors, with
few if any restrictions. There will be a followup very soon that
will support significantly more efficient operation for VCHR type
files (raw.) This code is also dependent on some kernel features
that don't work under SMP yet. After I commit the changes to the
kernel to support proper address space sharing on SMP, this code
will also work under SMP.

Back out part 1 of the MCFH that changed $Id$ to $FreeBSD$. We are not
ready for it yet.

This is the kernel Lite/2 commit. There are some requisite userland
changes, so don't expect to be able to run the kernel as-is (very well)
without the appropriate Lite/2 userland changes.

The system boots and can mount UFS filesystems.

Untested: ext2fs, msdosfs, NFS
Known problems: Incorrect Berkeley ID strings in some files.
Mount_std mounts will not work until the getfsent
library routine is changed.

Reviewed by: various people
Submitted by: Jeffery Hsu <hsu@freebsd.org>

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Reserve 4 malloc types for generic geometry handling.

Remove the kernel FD_SETSIZE limit for select().
Make select()'s first argument 'int' not 'u_int'.

Reviewed by: bde

Add separate kmalloc classes for BIO buffers and Ktrace info.

This is the `netkey' kernel key-management service (the PF_KEY analogue
to PF_ROUTE) from NRL's IPv6 distribution, heavily modified by me for
better source layout, formatting, and textual conventions. I am told
that this code is no longer under active development, but it's a useful
hack for those interested in doing work on network security, key management,
etc. This code has only been tested twice, so it should be considered
highly experimental.

Obtained from: ftp.ripe.net

removed:
CLBYTES PD_SHIFT PGSHIFT NBPG PGOFSET CLSIZELOG2 CLSIZE pdei()
ptei() kvtopte() ptetov() ispt() ptetoav() &c &c
new:
NPDEPG

Major macro cleanup.

Import 4.4BSD-Lite2 onto the vendor branch, note that in the kernel, all
files are off the vendor branch, so this should not change anything.

A "U" marker generally means that the file was not changed in between
the 4.4Lite and Lite-2 releases, and does not need a merge. "C" generally
means that there was a change.
[new sys/syscallargs.h file, to be "cvs rm"ed]

Merge in Lite2: resolve manifest constant numbering in favor of least
required changes and also add parentheses around BUCKETINDX macro body.
Reviewed by: davidg & bde

Added a `boundary' arg to vm_alloc_page_contig(). Previously the only
way to avoid crossing a 64K DMA boundary was to specify an alignment
greater than the size even when the alignment didn't matter, and for
sizes larger than a page, this reduced the chance of finding enough
contiguous pages. E.g., allocations of 8K not crossing a 64K boundary
previously had to be allocated on 8K boundaries; now they can be
allocated on any 4K boundary except (64 * n + 60)K.

Fixed bugs in vm_alloc_page_contig():
- the last page wasn't allocated for sizes smaller than a page.
- failures of kmem_alloc_pageable() weren't handled.

Mutated vm_page_alloc_contig() to create a more convenient interface
named contigmalloc(). This is the same as the one in 1.1.5 except
it has `low' and `high' args, and the `alignment' and `boundary'
args are multipliers instead of masks.

malloc.h: add sysctl class.
sysctl.h add "AUTO" OID.

both of which will be need RSN.

Remove bogus (caddr_t) casts from FREE macros; they can obscure serious
bugs that the type-checking would otherwise have found. (This has no
effect on anything that I noticed in the current system other than finding
the bug in some code that I am writing.)

Reviewed by: no-one yet, but non-intrusive
Submitted by: julian@tfs.com
Obtained from: written from scratch

slight changes to make space for devfs..
(also conditional test code in i386/isa/fd.c)

===================================================================
RCS file: /home/ncvs/src/sys/sys/malloc.h,v
retrieving revision 1.7
diff -r1.7 malloc.h
113a114,117
> #define M_DEVFSMNT 62 /* DEVFS mount structure */
> #define M_DEVFSBACK 63 /* DEVFS Back node */
> #define M_DEVFSFRONT 64 /* DEVFS Front node */
> #define M_DEVFSNODE 65 /* DEVFS node */
184c188,192
< NULL, NULL, NULL, NULL, NULL, \
---
> "DEVFS mount", /* 62 M_DEVFSMNT */ \
> "DEVFS back", /* 63 M_DEVFSBACK */ \
> "DEVFS front", /* 64 M_DEVFSFRONT */ \
> "DEVFS node", /* 65 M_DEVFSNODE */ \
> NULL, \
Index: sys/mount.h
===================================================================
RCS file: /home/ncvs/src/sys/sys/mount.h,v
retrieving revision 1.16
diff -r1.16 mount.h
100c100,101
< #define MOUNT_MAXTYPE 15
---
> #define MOUNT_DEVFS 16 /* existing device Filesystem */
> #define MOUNT_MAXTYPE 16
118a120
> "devfs", /* 15 MOUNT_DEVFS */ \
Index: sys/vnode.h
===================================================================
RCS file: /home/ncvs/src/sys/sys/vnode.h,v
retrieving revision 1.19
diff -r1.19 vnode.h
61c61
< VT_UNION, VT_MSDOSFS
---
> VT_UNION, VT_MSDOSFS, VT_DEVFS

Add M_DEVS for list of saved isa_device's
and M_IPFW for firewall chans memory on the same opportunity,
theese types of memory are unique enough to have own identifiers and
besides vmstat looks much more useful and right for those now

Calling semantics for kmem_malloc() have been changed...and the third
argument is now more than just a single flag.

Submitted by: John Dyson

Added M_GZIP for the imgact_gzip code. The gzip-code is likely to be used
for other weird things in the future (hint, hint!)

Added msdosfs.

Obtained from: NetBSD

Added $Id$

The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch.

Reviewed by: Rodney W. Grimes
Submitted by: John Dyson and David Greenman

BSD 4.4 Lite Kernel Sources