MFC r356315: Avoid duplicate I/O statistics accounting.

Alike to geom_disk free the provider statistics structure and point GEOM
toward local statistics. It allows to save some CPU time.

MFC of 345758

Properly flush outstanding I/Os when forcibly deleteing a memory disk device.

Sponsored by: Netflix

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r327032:

Add a new kernel config option, MD_ROOT_READONLY, which forces on the
MD_READONLY flag for the md device automatically instantiated during
kernel init for an mdroot filesystem.

Note that there is specifically and by design no tunable or sysctl
control over this feature. Without this option, you already have control
over whether the mdroot fs is writeable using vfs.root.mountfrom.options
from loader(8), the root_rw_mount rcvar, and by using "mount -u[rw] /"
or equivelent on the fly. This option is being added to provide a way
to make the mdroot fs truly immutable before userland code begins running.

Differential Revision: https://reviews.freebsd.org/D13411

MFC r319933:
Free the request page if an I/O error occurs while reading from swap.

MFC r319934:
Don't call vm_pager_page_unswapped() when writing or deleting a dirty page.

MFC r319932:
Fix handling of subpage BIO_WRITE and BIO_DELETE requests on swap MDs.

Approved by: re (gjb)

MFC r313982, r314068:
sys: Replace zero with NULL for pointers.

Found with: devel/coccinelle

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

For the MD_ROOT option don't inject /dev/md0 as root dev when ROOTDEVNAME
is defined explicitly. It's kinda pointless and results in extra step in
boot sequence which is not really needed, i.e.:

md0: Embedded image 1331200 bytes at 0x8038b7b4
Trying to mount root from ufs:/dev/md0 []...
Mounting from ufs:/dev/md0 failed with error 22.
Trying to mount root from ufs:md0.uzip []...
warning: no time-of-day clock registered, system time will not be set accurately
start_init: trying /sbin/init

Fix MFS builds when both MD_ROOT_SIZE and MFS_IMAGE are specified

MD_ROOT_SIZE and embed_mfs.sh were basically retired as part of
https://reviews.freebsd.org/D2903 .
However, when building a kernel with 'options MD_ROOT_SIZE' specified, this
results in a non-working MFS, as within sys/dev/md/md.c we fall within the
wrong # ifdef.

This patch implements the following:

* Allow kernels to be built without the MD_ROOT_SIZE option, which results
in a kernel built as per D2903.
* Allow kernels to be built with the MD_ROOT_SIZE option, which results
in a kernel built similarly to the pre-D2903 way, with the following
differences:
* The MFS is now put in a separate section within the kernel (oldmfs,
so it differs from the mfs section introduced by D2903).
* embed_mfs.sh is changed, so it looks up the oldmfs section within the
kernel, gets its size and offset, sees if the MFS will fit within the
allocated oldmfs section and only if all is well does a dd of the MFS
image into the kernel.

Submitted by: Stanislav Galabov <sgalabov@gmail.com>
Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D5093

A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().

o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.

Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.

Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().

o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.

This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.

Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix

In md(4) over vnode, correct handling of the unaligned unmapped io
requests which page alignment + size is greater than MAXPHYS. Right
now md(4) over vnode would use the physical buffer of the size MAXPHYS
to map a data of size MAXPHYS + page offset of the user buffer. This
typically corrupts next pbuf, or, if the pbuf used was the last pbuf
in the map, the next page after the pbuf's map.

Split request up to the size of io which fits into pbuf KVA with
alignment, and retry if a part of the bio is left unprocessed.

Reported by: Fabian Keil <fk@fabiankeil.de>
Tested by: Fabian Keil, pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Add asynchronous command support to the pass(4) driver, and the new
camdd(8) utility.

CCBs may be queued to the driver via the new CAMIOQUEUE ioctl, and
completed CCBs may be retrieved via the CAMIOGET ioctl. User
processes can use poll(2) or kevent(2) to get notification when
I/O has completed.

While the existing CAMIOCOMMAND blocking ioctl interface only
supports user virtual data pointers in a CCB (generally only
one per CCB), the new CAMIOQUEUE ioctl supports user virtual and
physical address pointers, as well as user virtual and physical
scatter/gather lists. This allows user applications to have more
flexibility in their data handling operations.

Kernel memory for data transferred via the queued interface is
allocated from the zone allocator in MAXPHYS sized chunks, and user
data is copied in and out. This is likely faster than the
vmapbuf()/vunmapbuf() method used by the CAMIOCOMMAND ioctl in
configurations with many processors (there are more TLB shootdowns
caused by the mapping/unmapping operation) but may not be as fast
as running with unmapped I/O.

The new memory handling model for user requests also allows
applications to send CCBs with request sizes that are larger than
MAXPHYS. The pass(4) driver now limits queued requests to the I/O
size listed by the SIM driver in the maxio field in the Path
Inquiry (XPT_PATH_INQ) CCB.

There are some things things would be good to add:

1. Come up with a way to do unmapped I/O on multiple buffers.
Currently the unmapped I/O interface operates on a struct bio,
which includes only one address and length. It would be nice
to be able to send an unmapped scatter/gather list down to
busdma. This would allow eliminating the copy we currently do
for data.

2. Add an ioctl to list currently outstanding CCBs in the various
queues.

3. Add an ioctl to cancel a request, or use the XPT_ABORT CCB to do
that.

4. Test physical address support. Virtual pointers and scatter
gather lists have been tested, but I have not yet tested
physical addresses or scatter/gather lists.

5. Investigate multiple queue support. At the moment there is one
queue of commands per pass(4) device. If multiple processes
open the device, they will submit I/O into the same queue and
get events for the same completions. This is probably the right
model for most applications, but it is something that could be
changed later on.

Also, add a new utility, camdd(8) that uses the asynchronous pass(4)
driver interface.

This utility is intended to be a basic data transfer/copy utility,
a simple benchmark utility, and an example of how to use the
asynchronous pass(4) interface.

It can copy data to and from pass(4) devices using any target queue
depth, starting offset and blocksize for the input and ouptut devices.
It currently only supports SCSI devices, but could be easily extended
to support ATA devices.

It can also copy data to and from regular files, block devices, tape
devices, pipes, stdin, and stdout. It does not support queueing
multiple commands to any of those targets, since it uses the standard
read(2)/write(2)/writev(2)/readv(2) system calls.

The I/O is done by two threads, one for the reader and one for the
writer. The reader thread sends completed read requests to the
writer thread in strictly sequential order, even if they complete
out of order. That could be modified later on for random I/O patterns
or slightly out of order I/O.

camdd(8) uses kqueue(2)/kevent(2) to get I/O completion events from
the pass(4) driver and also to send request notifications internally.

For pass(4) devcies, camdd(8) uses a single buffer (CAM_DATA_VADDR)
per CAM CCB on the reading side, and a scatter/gather list
(CAM_DATA_SG) on the writing side. In addition to testing both
interfaces, this makes any potential reblocking of I/O easier. No
data is copied between the reader and the writer, but rather the
reader's buffers are split into multiple I/O requests or combined
into a single I/O request depending on the input and output blocksize.

For the file I/O path, camdd(8) also uses a single buffer (read(2),
write(2), pread(2) or pwrite(2)) on reads, and a scatter/gather list
(readv(2), writev(2), preadv(2), pwritev(2)) on writes.

Things that would be nice to do for camdd(8) eventually:

1. Add support for I/O pattern generation. Patterns like all
zeros, all ones, LBA-based patterns, random patterns, etc. Right
Now you can always use /dev/zero, /dev/random, etc.

2. Add support for a "sink" mode, so we do only reads with no
writes. Right now, you can use /dev/null.

3. Add support for automatic queue depth probing, so that we can
figure out the right queue depth on the input and output side
for maximum throughput. At the moment it defaults to 6.

4. Add support for SATA device passthrough I/O.

5. Add support for random LBAs and/or lengths on the input and
output sides.

6. Track average per-I/O latency and busy time. The busy time
and latency could also feed in to the automatic queue depth
determination.

sys/cam/scsi/scsi_pass.h:
Define two new ioctls, CAMIOQUEUE and CAMIOGET, that queue
and fetch asynchronous CAM CCBs respectively.

Although these ioctls do not have a declared argument, they
both take a union ccb pointer. If we declare a size here,
the ioctl code in sys/kern/sys_generic.c will malloc and free
a buffer for either the CCB or the CCB pointer (depending on
how it is declared). Since we have to keep a copy of the
CCB (which is fairly large) anyway, having the ioctl malloc
and free a CCB for each call is wasteful.

sys/cam/scsi/scsi_pass.c:
Add asynchronous CCB support.

Add two new ioctls, CAMIOQUEUE and CAMIOGET.

CAMIOQUEUE adds a CCB to the incoming queue. The CCB is
executed immediately (and moved to the active queue) if it
is an immediate CCB, but otherwise it will be executed
in passstart() when a CCB is available from the transport layer.

When CCBs are completed (because they are immediate or
passdone() if they are queued), they are put on the done
queue.

If we get the final close on the device before all pending
I/O is complete, all active I/O is moved to the abandoned
queue and we increment the peripheral reference count so
that the peripheral driver instance doesn't go away before
all pending I/O is done.

The new passcreatezone() function is called on the first
call to the CAMIOQUEUE ioctl on a given device to allocate
the UMA zones for I/O requests and S/G list buffers. This
may be good to move off to a taskqueue at some point.
The new passmemsetup() function allocates memory and
scatter/gather lists to hold the user's data, and copies
in any data that needs to be written. For virtual pointers
(CAM_DATA_VADDR), the kernel buffer is malloced from the
new pass(4) driver malloc bucket. For virtual
scatter/gather lists (CAM_DATA_SG), buffers are allocated
from a new per-pass(9) UMA zone in MAXPHYS-sized chunks.
Physical pointers are passed in unchanged. We have support
for up to 16 scatter/gather segments (for the user and
kernel S/G lists) in the default struct pass_io_req, so
requests with longer S/G lists require an extra kernel malloc.

The new passcopysglist() function copies a user scatter/gather
list to a kernel scatter/gather list. The number of elements
in each list may be different, but (obviously) the amount of data
stored has to be identical.

The new passmemdone() function copies data out for the
CAM_DATA_VADDR and CAM_DATA_SG cases.

The new passiocleanup() function restores data pointers in
user CCBs and frees memory.

Add new functions to support kqueue(2)/kevent(2):

passreadfilt() tells kevent whether or not the done
queue is empty.

passkqfilter() adds a knote to our list.

passreadfiltdetach() removes a knote from our list.

Add a new function, passpoll(), for poll(2)/select(2)
to use.

Add devstat(9) support for the queued CCB path.

sys/cam/ata/ata_da.c:
Add support for the BIO_VLIST bio type.

sys/cam/cam_ccb.h:
Add a new enumeration for the xflags field in the CCB header.
(This doesn't change the CCB header, just adds an enumeration to
use.)

sys/cam/cam_xpt.c:
Add a new function, xpt_setup_ccb_flags(), that allows specifying
CCB flags.

sys/cam/cam_xpt.h:
Add a prototype for xpt_setup_ccb_flags().

sys/cam/scsi/scsi_da.c:
Add support for BIO_VLIST.

sys/dev/md/md.c:
Add BIO_VLIST support to md(4).

sys/geom/geom_disk.c:
Add BIO_VLIST support to the GEOM disk class. Re-factor the I/O size
limiting code in g_disk_start() a bit.

sys/kern/subr_bus_dma.c:
Change _bus_dmamap_load_vlist() to take a starting offset and
length.

Add a new function, _bus_dmamap_load_pages(), that will load a list
of physical pages starting at an offset.

Update _bus_dmamap_load_bio() to allow loading BIO_VLIST bios.
Allow unmapped I/O to start at an offset.

sys/kern/subr_uio.c:
Add two new functions, physcopyin_vlist() and physcopyout_vlist().

sys/pc98/include/bus.h:
Guard kernel-only parts of the pc98 machine/bus.h header with
#ifdef _KERNEL.

This allows userland programs to include <machine/bus.h> to get the
definition of bus_addr_t and bus_size_t.

sys/sys/bio.h:
Add a new bio flag, BIO_VLIST.

sys/sys/uio.h:
Add prototypes for physcopyin_vlist() and physcopyout_vlist().

share/man/man4/pass.4:
Document the CAMIOQUEUE and CAMIOGET ioctls.

usr.sbin/Makefile:
Add camdd.

usr.sbin/camdd/Makefile:
Add a makefile for camdd(8).

usr.sbin/camdd/camdd.8:
Man page for camdd(8).

usr.sbin/camdd/camdd.c:
The new camdd(8) utility.

Sponsored by: Spectra Logic
MFC after: 1 week

s/as/at/ in previous commit.

Pointed out by: jmallett@

Change md(4) to use weak symbols as start, end and size for the embedded
root disk. The embedded image is linked into the kernel in the .mfs
section.

Add rules and variables to kern.pre.mk and kern.post.mk that handle the
linking of the image. First objcopy is used to generate an object file.
Then, the object file is linked into the kernel.

Submitted by: Steve Kiernan <stevek@juniper.net>
Reviewed by: brooks@
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D2903

Use g_conf_printf_escaped() to escape illegal symbols in file name.

PR: 202289
MFC after: 1 week

For md(4), posix shm(3) and tmpfs(5), free swap space used by paged in
dirty page, which is written by the process.

Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

vm_page_grab() and vm_pager_get_pages() can drop the vm_object lock,
then threads can sleep on the pip condition.
Avoid to deadlock such threads by correctly awakening the sleeping ones
after the pip is finished.
swapoff side of the bug can likely result in shutdown deadlocks.

Sponsored by: EMC / Isilon Storage Division
Reported by: pho, pluknet
Tested by: pho

Only assert the length of the passed bio in the mdstart_vnode() when
the bio is unmapped, so we must map the bio pages into pbuf. This
works around the geom classes which do not follow the MAXPHYS limit on
the i/o size, since such classes do not know about unmapped bios
either.

Reported by: Paolo Pinto <paolo.pinto@netasq.com>
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Change comment to match code.

Discussed with: thompsa
Sponsored by: The FreeBSD Foundation

Add "null" backend to mdconfig(8). This does exactly what the name
suggests, and is somewhat useful for benchmarking.

MFC after: 1 month
No objections from: kib
Sponsored by: The FreeBSD Foundation

Merge GEOM direct dispatch changes from the projects/camlock branch.

When safety requirements are met, it allows to avoid passing I/O requests
to GEOM g_up/g_down thread, executing them directly in the caller context.
That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid
several context switches per I/O.

The defined now safety requirements are:
- caller should not hold any locks and should be reenterable;
- callee should not depend on GEOM dual-threaded concurency semantics;
- on the way down, if request is unmapped while callee doesn't support it,
the context should be sleepable;
- kernel thread stack usage should be below 50%.

To keep compatibility with GEOM classes not meeting above requirements
new provider and consumer flags added:
- G_CF_DIRECT_SEND -- consumer code meets caller requirements (request);
- G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done);
- G_PF_DIRECT_SEND -- provider code meets caller requirements (done);
- G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request).
Capable GEOM class can set them, allowing direct dispatch in cases where
it is safe. If any of requirements are not met, request is queued to
g_up or g_down thread same as before.

Such GEOM classes were reviewed and updated to support direct dispatch:
CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE,
VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL,
MAP, FLASHMAP, etc).

To declare direct completion capability disk(9) KPI got new flag equivalent
to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk
drivers got it set now thanks to earlier CAM locking work.

This change more then twice increases peak block storage performance on
systems with manu CPUs, together with earlier CAM locking changes reaching
more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to
256 user-level threads).

Sponsored by: iXsystems, Inc.
MFC after: 2 months

Give the page allocations initiated by the swap-backed md(4) a higher
priority. If the write is requested by a system daemon, sleeping
there would starve resources and cause deadlock.

Reported and tested by: pho
Sponsored by: The FreeBSD Foundation

Remove the deprecated VM_ALLOC_RETRY flag for the vm_page_grab(9).
The flag was mandatory since r209792, where vm_page_grab(9) was
changed to only support the alloc retry semantic.

Suggested and reviewed by: alc
Sponsored by: The FreeBSD Foundation

The soft and hard busy mechanism rely on the vm object lock to work.
Unify the 2 concept into a real, minimal, sxlock where the shared
acquisition represent the soft busy and the exclusive acquisition
represent the hard busy.
The old VPO_WANTED mechanism becames the hard-path for this new lock
and it becomes per-page rather than per-object.
The vm_object lock becames an interlock for this functionality:
it can be held in both read or write mode.
However, if the vm_object lock is held in read mode while acquiring
or releasing the busy state, the thread owner cannot make any
assumption on the busy state unless it is also busying it.

Also:
- Add a new flag to directly shared busy pages while vm_page_alloc
and vm_page_grab are being executed. This will be very helpful
once these functions happen under a read object lock.
- Move the swapping sleep into its own per-object flag

The KPI is heavilly changed this is why the version is bumped.
It is very likely that some VM ports users will need to change
their own code.

Sponsored by: EMC / Isilon storage division
Discussed with: alc
Reviewed by: jeff, kib
Tested by: gavin, bapt (older version)
Tested by: pho, scottl

Fix the data corruption on the swap-backed md.

Assign the rv variable a success code if the pager was not asked for
the page. Using an error code from the previous processed page caused
zeroing of the valid page, when e.g. the previous page was not
available in the pager.

Reported by: lstewart
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Do not declare that preloaded md(4) supports unmapped bio requests, it
does not.

Reported by: <mh@kernel32.de>
Sponsored by: The FreeBSD Foundation

Support unmapped i/o for the md(4).

The vnode-backed md(4) has to map the unmapped bio because VOP_READ()
and VOP_WRITE() interfaces do not allow to pass unmapped requests to
the filesystem. Vnode-backed md(4) uses pbufs instead of relying on
the bio_transient_map, to avoid usual md deadlock.

Sponsored by: The FreeBSD Foundation
Tested by: pho, scottl

Switch the vm_object mutex to be a rwlock. This will enable in the
future further optimizations where the vm_object lock will be held
in read mode most of the time the page cache resident pool of pages
are accessed for reading purposes.

The change is mostly mechanical but few notes are reported:
* The KPI changes as follow:
- VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK()
- VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK()
- VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK()
- VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED()
(in order to avoid visibility of implementation details)
- The read-mode operations are added:
VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(),
VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED()
* The vm/vm_pager.h namespace pollution avoidance (forcing requiring
sys/mutex.h in consumers directly to cater its inlining functions
using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h
consumers now must include also sys/rwlock.h.
* zfs requires a quite convoluted fix to include FreeBSD rwlocks into
the compat layer because the name clash between FreeBSD and solaris
versions must be avoided.
At this purpose zfs redefines the vm_object locking functions
directly, isolating the FreeBSD components in specific compat stubs.

The KPI results heavilly broken by this commit. Thirdy part ports must
be updated accordingly (I can think off-hand of VirtualBox, for example).

Sponsored by: EMC / Isilon storage division
Reviewed by: jeff
Reviewed by: pjd (ZFS specific review)
Discussed with: alc
Tested by: pho

Print correct unit number when attaching preloaded memory disks.
Retire now unused mdunits variable.

Disallow attaching preloaded memory disks via ioctl.

- The feature is dangerous because the kernel code didn't check
validity of the memory address provided from user space.
- It seems that mdconfig(8) never really supported attaching preloaded
memory disks.
- Preloaded memory disks are automatically attached during md(4)
initialization. Thus there shouldn't be much use for the feature.

PR: kern/169683
Discussed on: freebsd-hackers

Zero the newly allocated md(4) swap-backed page to prevent random
kernel memory leakage to userspace. For the typical use, when a
filesystem put on the md disk, the change only results in CPU and
memory bandwidth spent to zero the page, since filsystems make sure
that user never see unwritten content. But if md disk is used as raw
device by userspace, the garbage is exposed.

Reported by: Paul Schenkeveld <freebsd@psconsult.nl>
MFC after: 2 weeks

Add a MD_ROOT_FSTYPE kernel option. The option specifies the
file system part for the MD_ROOT mount string. Hardcoding the
the file system type as "ufs" is too restrictive.

Remove the support for using non-mpsafe filesystem modules.

In particular, do not lock Giant conditionally when calling into the
filesystem module, remove the VFS_LOCK_GIANT() and related
macros. Stop handling buffers belonging to non-mpsafe filesystems.

The VFS_VERSION is bumped to indicate the interface change which does
not result in the interface signatures changes.

Conducted and reviewed by: attilio
Tested by: pho

After the PHYS_TO_VM_PAGE() function was de-inlined, the main reason
to pull vm_param.h was removed. Other big dependency of vm_page.h on
vm_param.h are PA_LOCK* definitions, which are only needed for
in-kernel code, because modules use KBI-safe functions to lock the
pages.

Stop including vm_param.h into vm_page.h. Include vm_param.h
explicitely for the kernel code which needs it.

Suggested and reviewed by: alc
MFC after: 2 weeks

Remove verbose unused commented out debugging printf.

MFC after: 1 week
Reviewed by: alc

Disallow sectorsize larger than MAXPHYS and mediasize smaller than
sectorsize.

PR: 169947
Submitted by: Filip Palian (original version)
Reviewed by: kib

Make it possible to resize md(4) devices.

Reviewed by: kib
Sponsored by: FreeBSD Foundation

Document a large number of currently undocumented sysctls. While here
fix some style(9) issues and reduce redundancy.

PR: kern/155491
PR: kern/155490
PR: kern/155489
Submitted by: Galimov Albert <wtfcrap@mail.ru>
Approved by: bde
Reviewed by: jhb
MFC after: 1 week

Add information about MD_READONLY and MD_COMPRESS flags to the
configuration dump.

MFC after: 1 week

Include sys/sbuf.h directly.

Move the ZERO_REGION_SIZE to a machine-dependent file, as on many
architectures (i386, for example) the virtual memory space may be
constrained enough that 2MB is a large chunk. Use 64K for arches
other than amd64 and ia64, with special handling for sparc64 due to
differing hardware.

Also commit the comment changes to kmem_init_zero_region() that I
missed due to not saving the file. (Darn the unfamiliar development
environment).

Arch maintainers, please feel free to adjust ZERO_REGION_SIZE as you
see fit.

Requested by: alc
MFC after: 1 week
MFC with: r221853

Usa a globally visible region of zeros for both /dev/zero and the md
device. There are likely other kernel uses of "blob of zeros" than can
be converted.

Reviewed by: alc
MFC after: 1 week

Implement BIO_DELETE for vnode devices by simply overwriting the deleted
sectors with all-zeroes.

The zeroes come from a static buffer; null(4) uses a dynamic buffer for
the same purpose (for /dev/zero). It might be a good idea to have a
static, shared, read-only all-zeroes page somewhere in the kernel that
md(4), null(4) and any other code that needs zeroes could use.

Reviewed by: kib
MFC after: 3 weeks

Use the preload_fetch_addr() and preload_fetch_size() convenience
functions and only create the MD device when we have a non-zero
pointer and size.

Sponsored by: Juniper Networks

Add support for BIO_DELETE on swap-backed md(4). In the case of BIO_DELETE
covering the whole page, free the page. Otherwise, clear the region and
mark it clean. Not marking the page dirty could reinstantiate cleared
data, but it is allowed by BIO_DELETE specification and saves unneeded
write to swap.

Reviewed by: alc
Tested by: pho
MFC after: 2 weeks

Bio shall not be accessed after g_io_deliver(9).

Reported and tested by: pho
Reviewed by: ae, phk
MFC after: 1 week

Add missed ().

Noted by: alc
MFC after: 3 days

There is no point in calling vm_object_set_writeable_dirty() on an object
that is definitively known to be swap backed since its only effects are on
vnode-backed objects.

Reviewed by: kib

Add reporting of GEOM::candelete BIO_GETATTR for md(4) and geom_disk(4).
Non-zero value of attribute means that device supports BIO_DELETE.

Suggested and reviewed by: pjd
Tested by: pho
MFC after: 1 week

Add sysctl vm.md_malloc_wait, non-zero value of which switches malloc-backed
md(4) to using M_WAITOK malloc calls.

M_NOWAITOK allocations may fail when enough memory could be freed, but not
immediately. E.g. SU UFS becomes quite unhappy when metadata write return
error, that would happen for failed malloc() call.

Reported and tested by: pho
MFC after: 1 week

Allow the MDIOCATTACH ioctl operation to originate from within the kernel.
To protect against malicious software, we demand that the file name is at
a particular location (i.e. appended to the mdio structure) for it to be
treated as in-kernel.

- Remove some extra white space.
- Wrap g_md_dumpconf() prototype to 80 columns.

Convert md(4) to use alloc_unr(9) and alloc_unr_specific(9) for unit
number allocation. The old approach had some problems such as it allowed
an overflow to occur in the unit number calculation.

PR: kern/122288

Calculate nshift only once.

Also noted by: avg
MFC after: 1 week

Eliminate unnecessary page queues locking.

Lock the page around vm_page_activate() and vm_page_deactivate() calls
where it was missed. The wrapped fragments now protect wire_count with
page lock.

Reviewed by: alc

Fix panic on invalid 'mdconfig -at preload' usage.

PR: kern/80136

(S)LIST_HEAD_INITIALIZER takes a (S)LIST_HEAD as an argument.
Fix some wrong usages.
Note: this does not affect generated binaries as this argument is not used.

PR: 137213
Submitted by: Eygene Ryabinkin (initial version)
MFC after: 1 month

Implement global and per-uid accounting of the anonymous memory. Add
rlimit RLIMIT_SWAP that limits the amount of swap that may be reserved
for the uid.

The accounting information (charge) is associated with either map entry,
or vm object backing the entry, assuming the object is the first one
in the shadow chain and entry does not require COW. Charge is moved
from entry to object on allocation of the object, e.g. during the mmap,
assuming the object is allocated, or on the first page fault on the
entry. It moves back to the entry on forks due to COW setup.

The per-entry granularity of accounting makes the charge process fair
for processes that change uid during lifetime, and decrements charge
for proper uid when region is unmapped.

The interface of vm_pager_allocate(9) is extended by adding struct ucred *,
that is used to charge appropriate uid when allocation if performed by
kernel, e.g. md(4).

Several syscalls, among them is fork(2), may now return ENOMEM when
global or per-uid limits are enforced.

In collaboration with: pho
Reviewed by: alc
Approved by: re (kensmith)

Add cpu_flush_dcache() for use after non-DMA based I/O so that a
possible future I-cache coherency operation can succeed. On ARM
for example the L1 cache can be (is) virtually mapped, which
means that any I/O that uses temporary mappings will not see the
I-cache made coherent. On ia64 a similar behaviour has been
observed. By flushing the D-cache, execution of binaries backed
by md(4) and/or NFS work reliably.
For Book-E (powerpc), execution over NFS exhibits SIGILL once in
a while as well, though cpu_flush_dcache() hasn't been implemented
yet.

Doing an explicit D-cache flush as part of the non-DMA based I/O
read operation eliminates the need to do it as part of the
I-cache coherency operation itself and as such avoids pessimizing
the DMA-based I/O read operations for which D-cache are already
flushed/invalidated. It also allows future optimizations whereby
the bcopy() followed by the D-cache flush can be integrated in a
single operation, which could be implemented using on-chips DMA
engines, by-passing the D-cache altogether.

Add a new internal mount flag (MNTK_EXTENDED_SHARED) to indicate that a
filesystem supports additional operations using shared vnode locks.
Currently this is used to enable shared locks for open() and close() of
read-only file descriptors.
- When an ISOPEN namei() request is performed with LOCKSHARED, use a
shared vnode lock for the leaf vnode only if the mount point has the
extended shared flag set.
- Set LOCKSHARED in vn_open_cred() for requests that specify O_RDONLY but
not O_CREAT.
- Use a shared vnode lock around VOP_CLOSE() if the file was opened with
O_RDONLY and the mountpoint has the extended shared flag set.
- Adjust md(4) to upgrade the vnode lock on the vnode it gets back from
vn_open() since it now may only have a shared vnode lock.
- Don't enable shared vnode locks on FIFO vnodes in ZFS and UFS since
FIFO's require exclusive vnode locks for their open() and close()
routines. (My recent MPSAFE patches for UDF and cd9660 already included
this change.)
- Enable extended shared operations on UFS, cd9660, and UDF.

Submitted by: ups
Reviewed by: pjd (ZFS bits)
MFC after: 1 month

Remove unnecessary page queues locking around vm_page_wakeup(). (This
change is applicable to RELENG_7 but not RELENG_6.)

MFC after: 1 week

Add the possibility to specify "-o force" with "mdconfig -du".

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Fix forced mdconfig -du. E.g. the following would previously
result in panic:

mdconfig -af blah.img -o force
mount /dev/md0 /mnt
mdconfig -du 0

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Decontextualize the couplet VOP_GETATTR / VOP_SETATTR as the passed thread
was always curthread and totally unuseful.

Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com>

Remove the distinction between device minor and unit numbers.

Even though we got rid of device major numbers some time ago, device
drivers still need to provide unique device minor numbers to make_dev().
These numbers are only used inside the kernel. They are not related to
device major and minor numbers which are visible in devfs. These are
actually based on the inode number of the device.

It would eventually be nice to remove minor numbers entirely, but we
don't want to be too agressive here.

Because the 8-15 bits of the device number field (si_drv0) are still
reserved for the major number, there is no 1:1 mapping of the device
minor and unit numbers. Because this is now unused, remove the
restrictions on these numbers.

The MAXMAJOR definition was actually used for two purposes. It was used
to convert both the userspace and kernelspace device numbers to their
major/minor pair, which is why it is now named UMINORMASK.

minor2unit() and unit2minor() have now become useless. Both minor() and
dev2unit() now serve the same purpose. We should eventually remove some
of them, at least turning them into macro's. If devfs would become
completely minor number unaware, we could consider using si_drv0 directly,
just like si_drv1 and si_drv2.

Approved by: philip (mentor)

Zero sc->vnode if mdsetcred() fails.

This fixes the panic which happens when mdcreate_vnode() calls vn_close()
and mddestroy() calls it again further down the error handling path.

Reviewed by: kris, kib
MFC after: 3 days

VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used in
conjuction with 'thread' argument passing which is always curthread.
Remove the unuseful extra-argument and pass explicitly curthread to lower
layer functions, when necessary.

KPI results broken by this change, which should affect several ports, so
version bumping and manpage update will be further committed.

Tested by: kris, pho, Diego Sardina <siarodx at gmail dot com>

vn_lock() is currently only used with the 'curthread' passed as argument.
Remove this argument and pass curthread directly to underlying
VOP_LOCK1() VFS method. This modify makes the code cleaner and in
particular remove an annoying dependence helping next lockmgr() cleanup.
KPI results, obviously, changed.

Manpage and FreeBSD_version will be updated through further commits.

As a side note, would be valuable to say that next commits will address
a similar cleanup about VFS methods, in particular vop_lock1 and
vop_unlock.

Tested by: Diego Sardina <siarodx at gmail dot com>,
Andrea Di Pasquale <whyx dot it at gmail dot com>

Put back devstat support that was lost during GEOM transition. Initially,
I've tried to move md(4) to use geom_disk class, like real disks do, but
this requires major rework of some of the existing features such as
configuration dumping for example. Therefore just putting devstat support
directly into md(4) seems to be optimal solution.

Now you can see md(4) stats in `systat -vm' again.

MFC after: 2 weeks

Rename the kthread_xxx (e.g. kthread_create()) calls
to kproc_xxx as they actually make whole processes.
Thos makes way for us to add REAL kthread_create() and friends
that actually make theads. it turns out that most of these
calls actually end up being moved back to the thread version
when it's added. but we need to make this cosmetic change first.

I'd LOVE to do this rename in 7.0 so that we can eventually MFC the
new kthread_xxx() calls.

Commit 14/14 of sched_lock decomposition.
- Use thread_lock() rather than sched_lock for per-thread scheduling
sychronization.
- Use the per-process spinlock rather than the sched_lock for per-process
scheduling synchronization.

Tested by: kris, current@
Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc.
Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)

Revert UF_OPENING workaround for CURRENT.
Change the VOP_OPEN(), vn_open() vnode operation and d_fdopen() cdev operation
argument from being file descriptor index into the pointer to struct file.

Proposed and reviewed by: jhb
Reviewed by: daichi (unionfs)
Approved by: re (kensmith)

Resolve two deadlocks that could be caused by busy md device backed
by vnode. Allow for md thread and the thread that owns lock on vnode
backing the md device to do the write even when runningbufspace is
exhausted.

Tested by: Peter Holm
Reviewed by: tegge
MFC after: 2 weeks

Style nits.

Fix md(4) panic which occurs when I/O request different than
BIO_READ/BIO_WRITE is sent to vnode-backed provider (BIO_DELETE or
BIO_FLUSH).

Reported by: ceri

Add support for BIO_FLUSH to vnode-backed md(4) devices based on
VOP_FSYNC().

- Conditionally acquire Giant in mdstart_vnode(), mdcreate_vnode(), and
mddestroy() only if the file is from a non-MPSAFE VFS.
- No longer unconditionally hold Giant in the md kthread for vnode-backed
kthreads.
- Improve the handling of the thread exit race when destroying an md
device.

Teach md(4) and mdconfig(8) how to understand XML. Right now there won't be
a problem with listing large number of md(4) devices. Either 'list' or
'query' mode uses XML.

Additionally, new functionality was introduced. It's possible to pass
multiple devices to -u:

# ./mdconfig -l -u md0,md1

Approved by: cognet (mentor)

make sure that the start and end preloaded MFS markers are
in contiguous strings, and that the compiler does not optimize them
away because it thinks they are unused.

Call NDFREE() only when vn_open() succeeded.

MFC after: 3 days

o Fix typos in the comments.

Submitted by: Wojciech A. Koszek

Normalize a significant number of kernel malloc type names:

- Prefer '_' to ' ', as it results in more easily parsed results in
memory monitoring tools such as vmstat.

- Remove punctuation that is incompatible with using memory type names
as file names, such as '/' characters.

- Disambiguate some collisions by adding subsystem prefixes to some
memory types.

- Generally prefer lower case to upper case.

- If the same type is defined in multiple architecture directories,
attempt to use the same name in additional cases.

Not all instances were caught in this change, so more work is required to
finish this conversion. Similar changes are required for UMA zone names.

Make sure that the worker thread knows the type early enough to
grab Giant for vnode backing.

Found by: pho & tegge

Fix configuration locking in MD.

Remove md_mtx.

Remove GIANT from the mdctl device driver and avoid DROP_GIANT,
PICKUP_GIANT and geom events since we can call into GEOM directly
now.

Pick up Giant around vn_close().

Apply an exclusive sx around mdctls ioctl and preloading to protect
lists etc..

Don't initialize our lock (md_mtx or md_sx) from a
SYSINIT when there is a perfectly good pair of _fini/_init
functions to do it from.

Prune any final fractional sector from the mediasize to
keep GEOM happy.

Cleanups:

Unify MDIOVERSION check in (x)mdctlioctl()

Add pointer to start() routine to softc to eliminate a switch{}

Inline guts of mddetach().

Always pass error pointer to mdnew(), simplify implementation.

Do not destroy the queue mutex until the thread is done with it.

- Add md_mtx lock to protect ID number and list of devices.
- Always check mdnew() return value, as even in !autounit case
kthread_create() can fail.

Those two changes fix serval panics provked by simple stress test.

Tested by: Kris The BugMagnet
MFC after: 3 days

Ensure that file flags such as schg, sappnd (and others) are honored
by md(4). Before this change, it was possible to by-pass these flags
by creating memory disks which used a file as a backing store and
writing to the device.

This was discussed by the security team, and although this is problematic,
it was decided that it was not critical as we never guarantee that root will
be restricted.

This change implements the following behavior changes:

-If the user specifies the readonly flag, unset write operations before
opening the file. If the FWRITE mask is unset, the device will be
created with the MD_READONLY mask set. (readonly)
-Add a check in g_md_access which checks to see if the MD_READONLY mask
is set, if so return EROFS
-Do not gracefully downgrade access modes without telling the user. Instead
make the user specify their intentions for the device (assuming the file is
read only). This seems like the more correct way to handle things.

This is a RELENG_6 candidate.

PR: kern/84635
Reviewed by: phk

Request a CPU private mapping from sf_buf_alloc(). If the swap-backed
memory disk is larger than the number of available sf_bufs, this improves
performance on SMPs by eliminating interprocessor TLB shootdowns. For
example, with 6656 sf_bufs, the default on my test machine, and a 256MB
swap-backed memory disk, I see the command
"dd if=/dev/md0 of=/dev/null bs=64k" achieve ~489MB/sec with the default,
shared mappings, and ~587MB/sec with CPU private mappings.

Use MAXMINOR

- Don't destroy UMA zone on error in mdcreate_malloc(), because we need it
in mddestroy() to properly free already allocated memory.
This fixes a panic when we want to create too big memory backed device
with preallocate memory (-o reserve).
- Remove redundant { }.

MFC after: 1 week

Add a couple of mtx_asserts() to try to narrow down the window on
a bug repeatedly reported.

Start each of the license/copyright comments with /*-, minor shuffle of lines

Add needed synchronization to the error handling code that was introduced
in revision 1.141.

Lock assertion failures reported by: Kris Kennaway

Stop explicitly touching td_base_pri outside of the scheduler and simply
set a thread's priority via sched_prio() when that is the desired action.
The schedulers will start managing td_base_pri internally shortly.

Rewrite piece of code which I committed some time ago that allows to
show file name for 'mdconfig -l -u <x>' command.
This allows to preserve API/ABI compatibility with version 0 (that's why
I changed version number back to 0) and will allow to merge this change
to RELENG_5.

MFC after: 5 days

Fix the MDIOCDETACH ioctl() for md(4). Now that the md_file field in
the mdio structure is an array and not a pointer, we cannot test for
it to be NULL. It never is. Instead, test for md_file[0] to be '\0'.

Be consistent and use 'if (error != 0)' instead of 'if (error)' everywhere.

For file backed md(4) devices output their source file via
'mdconfig -l -u <unit>'.
Bump version number, as this change breaks ABI/API.

Don't explicitly call g_waitidle(), it happens automagically now.

Account for failure in vm_pager_allocate() or vm_pager_get_pages() in
md(8). The former is generally not going to fail, but the latter can
fail when the underlying swap device returns an error.

There are still plenty of other places where vm_pager_get_pages() failing
will lead directly to crashes, so it's a good idea to put your swap on
RAID if you care enough to put any of your disks on RAID....

Actually this order (unlock, wakeup) in this case is race-safe and can
save us 2 context switches.

Explained by: njl

- Make md(4) 64-bit clean.
After this change it should be possible to use very big md(4) devices.
- Clean up and simplify the code a bit.
- Use humanize_number(3) to print size of md(4) devices.
- Add 't' suffix which stands for terabyte.
- Make '-S' to really work with all types of devices.
- Other minor changes.

There is no need to keep 'npage' value inside our softc structure,
it is only used in one function. While doing so, change its type to
vm_ooffset_t.
We are still limited for swap-backed devices to 16TB on 32-bit architectures
where PAGE_SIZE is 4096 bytes.

- Do not use bio_pblkno as it is going away anyway.
- Prefer bio_length than bio_bcount.

First wakeup, then unlock.

Type 'int' is too small for 'i' and 'lastp' variables. Use proper type,
which is vm_pindex_t (unsigned 64bit on i386).

Deallocate VM object on failure.

One more missing NDFREE(9).

- Don't forget about NDFREE() in case of vn_open() failure.
- Don't forget about vn_close() in case of failure.

Fix UMA zone leak.

Use bioq_takefirst()

Don't g_waitidle() when initializing a preloaded md. This fixes a
deadlock which otherwise occurs during the boot process.

Reported by: kensmith
MFC after: 3 days
(assuming that re@ approves)

When creating a new md, wait for geom's event queue to become empty
before returning. Device nodes are created via the "taste" mechanism,
so this is necessary in order to make sure that devfs entries are
created before mdconfig(8) returns.

This may be a MFC candidate for 5.3.

Suggested by: phk

Tag all geom classes in the tree with a version number.

Use a ->fini() from the geom class to destroy the control device.
Use default initialization of geom methods.

Do a pass over all modules in the kernel and make them return EOPNOTSUPP
for unknown events.

A number of modules return EINVAL in this instance, and I have left
those alone for now and instead taught MOD_QUIESCE to accept this
as "didn't do anything".

Do the dreaded s/dev_t/struct cdev */
Bump __FreeBSD_version accordingly.

Fix panic which occurs when given sector size for memory-backed device
is less than DEV_BSIZE (512) bytes.

Reported by: Mike Bristow <mike@urgle.com>
Approved by: phk

Ooops, removed this acknowledgement bogusly.

Eagle Eyes: bde

Remove advertising clause from University of California Regent's
license, per letter dated July 22, 1999 and email from Peter Wemm,
Alan Cox and Robert Watson.

Approved by: core, peter, alc, rwatson

In some cases, sf_buf_alloc() should sleep with pri PCATCH; in others, it
should not. Add a new parameter so that the caller can specify which is
the case.

Reported by: dillon

Fix a bug with preloaded image -- for some reason [that i don't
completely understand], md_takeroot() runs before md_preloaded(),
rendering both useless.
As a fix, move the body (effectively one line!) of md_takeroot()
into md_preloaded(), and get rid of the stuff that has become useless.

Bug and fix reported 10 days ago on -current, no reply.

- Remove some unused #includes.
- Apply some style fixes to mdstart_swap().

Utilize sf_buf_alloc() and sf_buf_free() to implement the ephemeral
mappings required by mdstart_swap(). On i386, if the ephemeral mapping
is already in the sf_buf mapping cache, a swap-backed md performs
similarly to a malloc-backed md. Even if the ephemeral mapping is not
cached, this implementation is still faster. On 64-bit platforms, this
change has the effect of using the direct virtual-to-physical mapping,
avoiding ephemeral mapping overheads, such as TLB shootdowns on SMPs.

On a 2.4GHz, 400MHz FSB P4 Xeon configured with 64K sf_bufs and
"mdmfs -S -o async -s 128m md /mnt"

before:
dd if=/dev/md0 of=/dev/null bs=64k
134217728 bytes transferred in 0.430923 secs (311465697 bytes/sec)

after with cold sf_buf cache:
dd if=/dev/md0 of=/dev/null bs=64k
134217728 bytes transferred in 0.367948 secs (364773576 bytes/sec)

after with warm sf_buf cache:
dd if=/dev/md0 of=/dev/null bs=64k
134217728 bytes transferred in 0.252826 secs (530870010 bytes/sec)

malloc-backed md:
dd if=/dev/md0 of=/dev/null bs=64k
134217728 bytes transferred in 0.253126 secs (530240978 bytes/sec)

Allow swap-backed devices to run without Giant.

Fix a long-standing deadlock issue with vnode backed md(4) devices:

On vnode backed md(4) devices over a certain, currently undetermined
size relative to the buffer cache our "lemming-syncer" can provoke
a buffer starvation which puts the md thread to sleep on wdrain.

This generally tends to grind the entire system to a stop because the
event that is supposed to wake up the thread will not happen until a fair
bit of the piled up I/O requests in the system finish, and since a lot
of those are on a md(4) vnode backed device which is currently waiting
on wdrain until a fair amount of the piled up ... you get the picture.

The cure is to issue all VOP_WRITES on the vnode backing the device
with IO_SYNC.

In addition to more closely emulating a real disk device with a
non-lying write-cache, this makes the writes exempt from rate-limited
(there to avoid starving the buffer cache) and consequently prevents
the deadlock.

Unfortunately performance takes a hit.

Add "async" option to give people who know what they are doing the
old behaviour.

kthread_exit() no longer requires Giant, so don't force callers to acquire
Giant just to call kthread_exit().

Requested by: many

Make swapbacked md(4) devices respect the -x and -y emulation arguments.

Use DEV_BSIZE byte sectors instead of PAGE_SIZE byte sectors for
swap-backed memory disks. This reduces filesystem allocation overhead
and makes swap-backed memory disks compatible with broken code (dd,
for example) which expects to see 512 byte sectors. The size of a
swap-backed memory disk must still be a multiple of the page size.

When performing page-aligned operations, this change has zero
performance impact.

Reviewed by: phk
Approved by: rwatson (mentor)

Device megapatch 4/6:

Introduce d_version field in struct cdevsw, this must always be
initialized to D_VERSION.

Flip sense of D_NOGIANT flag to D_NEEDGIANT, this involves removing
four D_NOGIANT flags and adding 145 D_NEEDGIANT flags.

Allow specification of a geometry for vnode backed devices as well as
for malloc backed devices.

Fix a locking problem with MD_ROOT_SIZE.

Retire md(4)'s static major number.

Use the class->init() to hitch up preload devices, rather than rely on
the "old" SYSINIT. This makes sure things happen in the right order.

XXX: md(4) needs to be fully geom-ified and in particluar /dev/md.ctl
should be abandonded for the GEOM OaM api.

Approved by: re@

Don't initialize unused bio_blkno field.

The present defaults for the open and close for device drivers which
provide no methods does not make any sense, and is not used by any
driver.

It is a pretty hard to come up with even a theoretical concept of
a device driver which would always fail open and close with ENODEV.

Change the defaults to be nullopen() and nullclose() which simply
does nothing.

Remove explicit initializations to these from the drivers which
already used them.

Consistently use the BSD u_int and u_short instead of the SYSV uint and
ushort. In most of these files, there was a mixture of both styles and
this change just makes them self-consistent.

Requested by: bde (kern_ktrace.c)

Change the implementation of swap backing to use the VM system in normal
ways, and drop the need for vm_pager_strategy().

Add fdidx argument to vn_open() and vn_open_cred() and pass -1 throughout.

Remove 256 unit limit, there is no evil minor number encoding to
deal with any more.

Spotted by: "Darren Freestone" <df@cops.org>

Remove the G_CLASS_INITIALIZER, we do not need it anymore.

The IO_NOWDRAIN and B_NOWDRAIN hacks are no longer needed to prevent
deadlocks with vnode backed md(4) devices because md now uses a
kthread to run the bio requests instead of doing it directly from
the bio down path.

Use vm_object_deallocate(), not vm_pager_deallocate(), to destroy a
vm object. (vm_pager_deallocate() does not, in fact, destroy a vm object.)

Approved by: re (scottl)
Reviewed by: phk

Call g_wither_geom(), instead of just setting the flag.

Add a couple of undocumented test options to MD(4) to aid in regression
testting of GEOM.

Remove all references to BIO_SETATTR. We will not be using it.

Use bioq_flush() to drain a bio queue with a specific error code.
Retain the mistake of not updating the devstat API for now.

Spell bioq_disksort() consistently with the remaining bioq_*().

#include <geom/geom_disk.h> where this is more appropriate.

Don't include <sys/disk.h>.

remove a blank line.

Allocate the toplevel indir with M_WAITOK to avoid complicating things
needlessly.

Detected by: rwatsons EvilMalloc(9)

Change g_class initialization to sparse format.

Including <sys/stdint.h> is (almost?) universally only to be able to use
%j in printfs, so put a newsted include in <sys/systm.h> where the printf
prototype lives and save everybody else the trouble.

Centralize the devstat handling for all GEOM disk device drivers
in geom_disk.c.

As a side effect this makes a lot of #include <sys/devicestat.h>
lines not needed and some biofinish() calls can be reduced to
biodone() again.

Add a "-S sectorsize" option to enable Kirk to find a bug :-)

Gigacommit to improve device-driver source compatibility between
branches:

Initialize struct cdevsw using C99 sparse initializtion and remove
all initializations to default values.

This patch is automatically generated and has been tested by compiling
LINT with all the fields in struct cdevsw in reverse order on alpha,
sparc64 and i386.

Approved by: re(scottl)

Back out M_* changes, per decision of the TRB.

Approved by: trb

Mark our provider with G_PF_CANDELETE in the cases where this is actually
the case.

NO_GEOM cleanup: unifdef

Implement MDIOCLIST which returns the unit numbers of configured md(4)
devices.

We use the md_pad[] array and if there are more units than its size the
last returned unit number will be -1, but the number of units returned
is correct.

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

OK Ok, so I didn't check the NO_GEOM case for the final version...

Stumbled on by: bde

Enable the new h0h0magic code which on GEOM kernels make the md(4)
driver a _real_ GEOM driver.

Add a mutex around the per unit bioqueue.

Only grab giant in the per unit kthread for SWAP and VNODE backed devices.

Initialize the bioq before the kthread gets a chance to study it.

Don't lock Giant in mddone_swap, we shouldn't need it.

Remove the printf which announces the creation of malloc disks: it is
inconsistent when we do not do it for swap or vnode.

We still printf for preloaded disks because of the weak debugging
options people have in embedded/tiny environments where this is
usually used.

Add code to make md(4) a GEOM device driver instead of relying in
the disk mini-layer.

This is currently not enabled.

Shift things around a bit in preparation for future evilness.

Move the check for the MD_SHUTDOWN flag to before the tsleep() call
in the per-device kthread. This ensures that synchronisation with
mddestroy() succeeds even if the kthread was not waiting in tsleep()
at the time of the wakeup(). Among other things, this fixes the
problem of mdconfig getting stuck when an attempt is made to use a
zero-length file as a vnode-type backing store.

Approved by: re

We want /dev/md0 for ramdisk roots, not /dev/md0c.

Sponsored by: DARPA & NAI Labs

Use ENOSPC error return, not ENOMEM.

Use %jd rather than %lld.

MODINFO_SIZE metadata has type size_t, not unsigned. This makes preloaded
md root work on sparc64.

Some kernel threads try to do significant work, and the default KSTACK_PAGES
doesn't give them enough stack to do much before blowing away the pcb.
This adds MI and MD code to allow the allocation of an alternate kstack
who's size can be speficied when calling kthread_create. Passing the
value 0 prevents the alternate kstack from being created. Note that the
ia64 MD code is missing for now, and PowerPC was only partially written
due to the pmap.c being incomplete there.
Though this patch does not modify anything to make use of the alternate
kstack, acpi and usb are good candidates.

Reviewed by: jake, peter, jhb

Put the casts on the right hand side of =.

Initialize fwsectors/fwheads to allow the DIOCGFWSECTORS and
DIOCGFWHEADS ioctls to return meaningful values to disklabel/newfs

Approved by: phk

(This commit touches about 15 disk device drivers in a very consistent
and predictable way, and I apologize if I have gotten it wrong anywhere,
getting prior review on a patch like this is not feasible, considering
the number of people involved and hardware availability etc.)

If struct disklabel is the messenger: kill the messenger.

Inside struct disk we had a struct disklabel which disk drivers used to
communicate certain metrics to the disklayer above (GEOM or the disk
mini-layer). This commit changes this communication to use four
explicit fields instead.

Amongst the benefits is that the fields do not get overwritten by
wrong or bogus on-disk disklabels.

Once that is clear, <sys/disk.h> which is included in the drivers
no longer need to pull <sys/disklabel.h> and <sys/diskslice.h> in,
the few places that needs them, have gotten explicit #includes for
them.

The disklabel inside struct disk is now only for internal use in
the disk mini-layer, so instead of embedding it, we malloc it as
we need it.

This concludes (modulus any mistakes) the series of disklabel related
commits.

I belive it all amounts to a NOP for all the rest of you :-)

Sponsored by: DARPA & NAI Labs.

Replace (ab)uses of "NULL" where "0" is really meant.

Yet another warning fix for 64 bits platforms.

Reviewed by: phk

mdcreate_vnode() isn't correctly clearing things out of the linked
list if the file is of 0 size or mdsetcred() fails.

Submitted by: Martin Faxer <gmh003532@brfmasthugget.se>

- Whitespace only: use return statement consistentlt (return (foo), not
return(foo)), kill extra blank names between function names;
- fix format string in printf(): devtoname() returns string, not pointer.

Use a per-device worker thread to avoid blocking in mdstrategy()
until the I/O completes. This fixes some easily reproducable deadlocks
that occur when using md(4) with GEOM.

Reviewed by: phk

Mis-edit in last commit.

Be a bit smarter about rewriting data so we don't loose too much performance.

Sponsored by: DARPA & NAI Labs.

Use an umazone per unit for allocating the sectors for malloc backing.

Clean up things properly when we unconfigure malloc backed units.

Sponsored by: DARPA & NAI Labs.

Give the "malloc" backing of md(4) an adaptive multilevel index tree to
remove the need for a contiguous array with pointers to all the sectors.

Try to make failure to malloc(9) memory a non-hang situation.

Eventually this will allow us to test the 64bit cleanness of the disk
I/O patch, but more work is outstanding here and elsewhere.

Sponsored by: DARPA & NAI Labs.

Fix a memory-leak when configuring a vnode backed md(4) device fails.

Submitted by: Martin Faxér <gmh003532@brfmasthugget.se>
MFC after: 4 weeks

Remove unused include.

The previous commit missed fixing 2 old printf format errors and
introduced a format printf error.

Fix printf warning caused by recent changes in bio_pblkno's type.

Introduce the new 64-bit size disk block, daddr64_t. Change
the bio and buffer structures to have daddr64_t bio_pblkno,
b_blkno, and b_lblkno fields which allows access to disks
larger than a Terabyte in size. This change also requires
that the VOP_BMAP vnode operation accept and return daddr64_t
blocks. This delta should not affect system operation in
any way. It merely sets up the necessary interfaces to allow
the development of disk drivers that work with these larger
disk block addresses. It also allows for the development of
UFS2 which will use 64-bit block addresses.

Simple p_ucred -> td_ucred changes to start using the per-thread ucred
reference.

Staticize the malloc definitions.

Obtained from: ~bde/sys.dif.gz

Gah! last commit botched indentation, fix indentation and some other
white-space nits while at it.

Restructure slightly, eliminating some repetitive source lines and
making GEOM patches simpler and more readable at the same time.

Actually make use of the md_version field of 'struct mdio'. In order
not to needlessly break compatibility, decrement MDIOVERSION to 0.

Approved by: phk

Implement IO_NOWDRAIN and B_NOWDRAIN - prevents the buffer cache from blocking
in wdrain during a write. This flag needs to be used in devices whos
strategy routines turn-around and issue another high level I/O, such as
when MD turns around and issues a VOP_WRITE to vnode backing store, in order
to avoid deadlocking the dirty buffer draining code.

Remove a vprintf() warning from MD when the backing vnode is found to be
in-use. The syncer of buf_daemon could be flushing the backing vnode at
the time of an MD operation so the warning is not correct.

MFC after: 1 week

Change the kernel's ucred API as follows:
- crhold() returns a reference to the ucred whose refcount it bumps.
- crcopy() now simply copies the credentials from one credential to
another and has no return value.
- a new crshared() primitive is added which returns true if a ucred's
refcount is > 1 and false (0) otherwise.

Use crhold() instead of crdup(). The md(4) driver doesn't modify the ucred
that it uses, so it merely needs to bump its refcount to make it immutable
rather than obtain its own copy.

KSE Milestone 2
Note ALL MODULES MUST BE RECOMPILED
make the kernel aware that there are smaller units of scheduling than the
process. (but only allow one thread per process at this time).
This is functionally equivalent to teh previousl -current except
that there is a thread associated with each process.

Sorry john! (your next MFC will be a doosie!)

Reviewed by: peter@freebsd.org, dillon@freebsd.org

X-MFC after: ha ha ha ha

OOPS, remove local change that somehow slipped into a commit (I swear that
I already deleted it some time ago). This should fix problem people have with
unsefined reference to `MD_PRELOAD_COMPRESSED'.

Submitted by: Manfred Antar <null@pozo.com>

- On module unload try to detach all configured disks and let unload proceed
if all disks were detached sucessfully;
- use consistent style for return statements and fix several others style
inconsistencies.

Reviewed by: ru
Approved by: phk

There is no MD_OBJET disk type, it's actually MD_SWAP. I guess the
former was either a previous or proposed name that kind of snuck in.

Introduce a force option, MD_FORCE, that instructs the driver to
bypass some extra anti-foot-shooting measures. Currently, its only
effect is to allow detaching a device while it's still open (e.g.,
mounted). This is useful for testing how the system reacts to a disk
suddenly going away, which can happen with some removeable media.

At this point, the force option is only checked on detach, so it
would've been possible to allow the option to be passed with the
MDIOCDETACH operation. This was not done to allow the possibility of
having the force flag influence other tests in the future, which may
not necessarily deal with detaching the device.

Reviewed by: sobomax
Approved by: phk

- Deny detaching requests until device is still open, otherwise it is possible
to hang or panic kernel by detaching disk from which fs is mounted;
- replace "md" with MD_NAME in yet another place.

Reviewed by: phk
Approved by: phk

Make sure the total number of sectors is not 0 for a vnode-type md to
avoid a division by zero which would occur on open() in this case.

Reviewed by: phk

Use MD_NAME and MDCTL_NAME constants where appropriate.

With Alfred's permission, remove vm_mtx in favor of a fine-grained approach
(this commit is just the first stage). Also add various GIANT_ macros to
formalize the removal of Giant, making it easy to test in a more piecemeal
fashion. These macros will allow us to test fine-grained locks to a degree
before removing Giant, and also after, and to remove Giant in a piecemeal
fashion via sysctl's on those subsystems which the authors believe can
operate without Giant.

We don't need the vm lock to perform a few simple calculations on the
md device's softc.

Remove MFS compat bits.

Acquire vm_mtx before calling vm_pager_deallocate.

Reviewed by: phk

Sort includes.

Introduce a global lock for the vm subsystem (vm_mtx).

vm_mtx does not recurse and is required for most low level
vm operations.

faults can not be taken without holding Giant.

Memory subsystems can now call the base page allocators safely.

Almost all atomic ops were removed as they are covered under the
vm mutex.

Alpha and ia64 now need to catch up to i386's trap handlers.

FFS and NFS have been tested, other filesystems will need minor
changes (grabbing the vm lock when twiddling page properties).

Reviewed (partially) by: jake, jhb

Polish error handling code using biofinish()

Actually biofinish(struct bio *, struct devstat *, int error) is more general
than the bioerror().

Most of this patch is generated by scripts.

Fix a panic if MD devices were left half-created.

XXX: the real bug is that devstat isn't part of the disk minilayer.

PR: 27158
Submitted by: Anders Nordby <anders@fix.no>

Undo part of the tangle of having sys/lock.h and sys/mutex.h included in
other "system" header files.

Also help the deprecation of lockmgr.h by making it a sub-include of
sys/lock.h and removing sys/lockmgr.h form kernel .c files.

Sort sys/*.h includes where possible in affected files.

OK'ed by: bde (with reservations)

Fix a reference to the "vn" driver in a warning message.

Use a more BIOS friendly geometry.

Submitted by: joe

Make "md" and "mdctl" macroized parameters.

Implement "-l" option to mdconfig which can list one or all md devices.

Submitted by: Dima Dorfman <dima@unixfreak.org>

Make md/mdconfig do kld.

Submitted by: dcs

Remove devstat entries in mddelete()

Spotted: tegge

General cleanup.

Fix a maybe-not-so-harmless warning.

Either cvs(1) or I forgot this file in my last commit.

Please see commit log for rev 1.4 of src/sbin/mdconfig/mdconfig.c

This is the first snapshot of the new all-singing-and-dancing md(4).

Using the mdconfig(8) program you can now configure memory disks
on malloc(9), swap or a file/vnode. preloaded md disks also work
as usual.

Enforce disk unit numbers upper limit in cloning.

Convert more malloc+bzero to malloc+M_ZERO.

Submitted by: josh@zipperup.org
Submitted by: Robert Drehmel <robd@gmx.net>

Avoid the modules madness I inadvertently introduced by making the
cloning infrastructure standard in kern_conf. Modules are now
the same with or without devfs support.

If you need to detect if devfs is present, in modules or elsewhere,
check the integer variable "devfs_present".

This happily removes an ugly hack from kern/vfs_conf.c.

This forces a rename of the eventhandler and the standard clone
helper function.

Include <sys/eventhandler.h> in <sys/conf.h>: it's a helper #include
like <sys/queue.h>

Remove all #includes of opt_devfs.h they no longer matter.

* Completely rewrite the alpha busspace to hide the implementation from
the drivers.
* Remove legacy inx/outx support from chipset and replace with macros
which call busspace.
* Rework pci config accesses to route through the pcib device instead of
calling a MD function directly.

With these changes it is possible to cleanly support machines which have
more than one independantly numbered PCI busses. As a bonus, the new
busspace implementation should be measurably faster than the old one.

Remove all traces of Julians DEVFS (incl from kern/subr_diskslice.c)

Remove old DEVFS support fields from dev_t.

Make uid, gid & mode members of dev_t and set them in make_dev().

Use correct uid, gid & mode in make_dev in disk minilayer.

Add support for registering alias names for a dev_t using the
new function make_dev_alias(). These will show up as symlinks
in DEVFS.

Use makedev() rather than make_dev() for MFSs magic devices to prevent
DEVFS from noticing this abuse.

Add a field for DEVFS inode number in dev_t.

Add new DEVFS in fs/devfs.

Add devfs cloning to:
disk minilayer (ie: ad(4), sd(4), cd(4) etc etc)
md(4), tun(4), bpf(4), fd(4)

If DEVFS add -d flag to /sbin/inits args to make it mount devfs.

Add commented out DEVFS to GENERIC

Rename MDNSECT to MD_NSECT and declare it as something that isn't
default in NOTES.

Requested by: bde
Approved by: phk

Fix the "almost clone" semantics.

Separate the struct bio related stuff out of <sys/buf.h> into
<sys/bio.h>.

<sys/bio.h> is now a prerequisite for <sys/buf.h> but it shall
not be made a nested include according to bdes teachings on the
subject of nested includes.

Diskdrivers and similar stuff below specfs::strategy() should no
longer need to include <sys/buf.> unless they need caching of data.

Still a few bogus uses of struct buf to track down.

Repocopy by: peter

Complete the bio/buf divorce for all code below devfs::strategy

Exceptions:
Vinum untouched. This means that it cannot be compiled.
Greg Lehey is on the case.

CCD not converted yet, casts to struct buf (still safe)

atapi-cd casts to struct buf to examine B_PHYS

Remove B_READ, B_WRITE and B_FREEBUF and replace them with a new
field in struct buf: b_iocmd. The b_iocmd is enforced to have
exactly one bit set.

B_WRITE was bogusly defined as zero giving rise to obvious coding
mistakes.

Also eliminate the redundant struct buf flag B_CALL, it can just
as efficiently be done by comparing b_iodone to NULL.

Should you get a panic or drop into the debugger, complaining about
"b_iocmd", don't continue. It is likely to write on your disk
where it should have been reading.

This change is a step in the direction towards a stackable BIO capability.

A lot of this patch were machine generated (Thanks to style(9) compliance!)

Vinum users: Greg has not had time to test this yet, be careful.

Initialize type correctly.

Retire MFS_ROOT and MFS_ROOT_SIZE options from the MFS implementation.

Add MD_ROOT and MD_ROOT_SIZE options to the md driver.

Make the md driver handle MFS_ROOT and MFS_ROOT_SIZE options for compatibility.

Add md driver to GENERIC, PCCARD and LINT.

This is a cleanup which removes the need for some of the worse hacks in
MFS: We really want to have a rootvnode but MFS on a preloaded image
doesn't really have one. md is a true device, so it is less trouble.

This has been tested with make release, and if people remember to add
the "md" pseudo-device to their kernels, PicoBSD should be just fine
as well. If people have no other use for MFS, it can be removed from
the kernel.

Teach the md driver to use preloaded files of type "md_image".

remove unused #include

Remove five now unused fields from struct cdevsw. They should never
have been there in the first place. A GENERIC kernel shrinks almost 1k.

Add a slightly different safetybelt under nostop for tty drivers.

Add some missing FreeBSD tags

Fix includes (remove unnecessary ones, reorder necessary ones.)
Also, correct an %x to be %lx.

Reviewed by: phk

fix a buglet which jordan made me provoke :-)

Add an experimental Memory-Disk driver. This driver will allocate
memory with malloc(9) using a few tricks to save space on the way.