Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Implement lockless resource limits.

Use the same scheme implemented to manage credentials.

Code needing to look at process's credentials (as opposed to thred's) is
provided with *_proc variants of relevant functions.

Places which possibly had to take the proc lock anyway still use the proc
pointer to access limits.

Add a mmap flag (MAP_32BIT) on 64-bit platforms to request that a mapping use
an address in the first 2GB of the process's address space. This flag should
have the same semantics as the same flag on Linux.

To facilitate this, add a new parameter to vm_map_find() that specifies an
optional maximum virtual address. While here, fix several callers of
vm_map_find() to use a VMFS_* constant for the findspace argument instead of
TRUE and FALSE.

Reviewed by: alc
Approved by: re (kib)

Replace kernel virtual address space allocation with vmem. This provides
transparent layering and better fragmentation.

- Normalize functions that allocate memory to use kmem_*
- Those that allocate address space are named kva_*
- Those that operate on maps are named kmap_*
- Implement recursive allocation handling for kmem_arena in vmem.

Reviewed by: alc
Tested by: pho
Sponsored by: EMC / Isilon Storage Division

- Correct mispellings of the word necessary

Submitted by: Christoph Mallon <christoph.mallon@gmx.de> (via private mail)

Remove an old hack I noticed years ago, but never committed.

Fix misuse of the kernel map in miscellaneous image activators.
Vnode-backed mappings cannot be put into the kernel map, since it is a
system map.

Use exec_map for transient mappings, and remove the mappings with
kmem_free_wakeup() to notify the waiters on available map space.

Do not map the whole executable into KVA at all to copy it out into
usermode. Directly use vn_rdwr() for the case of not page aligned
binary.

There is one place left where the potentially unbounded amount of data
is mapped into exec_map, namely, in the COFF image activator
enumeration of the needed shared libraries.

Reviewed by: alc
MFC after: 2 weeks

Add accounting for most of the memory-related resources.

Sponsored by: The FreeBSD Foundation
Reviewed by: kib (earlier version)

VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used in
conjuction with 'thread' argument passing which is always curthread.
Remove the unuseful extra-argument and pass explicitly curthread to lower
layer functions, when necessary.

KPI results broken by this change, which should affect several ports, so
version bumping and manpage update will be further committed.

Tested by: kris, pho, Diego Sardina <siarodx at gmail dot com>

vn_lock() is currently only used with the 'curthread' passed as argument.
Remove this argument and pass curthread directly to underlying
VOP_LOCK1() VFS method. This modify makes the code cleaner and in
particular remove an annoying dependence helping next lockmgr() cleanup.
KPI results, obviously, changed.

Manpage and FreeBSD_version will be updated through further commits.

As a side note, would be valuable to say that next commits will address
a similar cleanup about VFS methods, in particular vop_lock1 and
vop_unlock.

Tested by: Diego Sardina <siarodx at gmail dot com>,
Andrea Di Pasquale <whyx dot it at gmail dot com>

Fix for the panic("vm_thread_new: kstack allocation failed") and
silent NULL pointer dereference in the i386 and sparc64 pmap_pinit()
when the kmem_alloc_nofault() failed to allocate address space. Both
functions now return error instead of panicing or dereferencing NULL.

As consequence, vmspace_exec() and vmspace_unshare() returns the errno
int. struct vmspace arg was added to vm_forkproc() to avoid dealing
with failed allocation when most of the fork1() job is already done.

The kernel stack for the thread is now set up in the thread_alloc(),
that itself may return NULL. Also, allocation of the first process
thread is performed in the fork1() to properly deal with stack
allocation failure. proc_linkup() is separated into proc_linkup()
called from fork1(), and proc_linkup0(), that is used to set up the
kernel process (was known as swapper).

In collaboration with: Peter Holm
Reviewed by: jhb

Maintain the lock on the vnode for most of exec_elfN_imgact().
Specifically, it is required for the I/O that may be performed by
elfN_load_section().

Avoid an obscure deadlock in the a.out, elf, and gzip image
activators. Add a comment describing why the deadlock does not occur
in the common case and how it might occur in less usual circumstances.

Eliminate an unused variable from exec_aout_imgact().

In collaboration with: tegge

- Change the vm_mmap() function to accept an objtype_t parameter specifying
the type of object represented by the handle argument.
- Allow vm_mmap() to map device memory via cdev objects in addition to
vnodes and anonymous memory. Note that mmaping a cdev directly does not
currently perform any MAC checks like mapping a vnode does.
- Unbreak the DRM getbufs ioctl by having it call vm_mmap() directly on the
cdev the ioctl is acting on rather than trying to find a suitable vnode
to map from.

Reviewed by: alc, arch@

o Split out kernel part of execve(2) syscall into two parts: one that
copies arguments into the kernel space and one that operates
completely in the kernel space;

o use kernel-only version of execve(2) to kill another stackgap in
linuxlator/i386.

Obtained from: DragonFlyBSD (partially)
MFC after: 2 weeks

/* -> /*- for copyright notices, minor format tweaks as necessary

Locking for the per-process resource limits structure.
- struct plimit includes a mutex to protect a reference count. The plimit
structure is treated similarly to struct ucred in that is is always copy
on write, so having a reference to a structure is sufficient to read from
it without needing a further lock.
- The proc lock protects the p_limit pointer and must be held while reading
limits from a process to keep the limit structure from changing out from
under you while reading from it.
- Various global limits that are ints are not protected by a lock since
int writes are atomic on all the archs we support and thus a lock
wouldn't buy us anything.
- All accesses to individual resource limits from a process are abstracted
behind a simple lim_rlimit(), lim_max(), and lim_cur() API that return
either an rlimit, or the current or max individual limit of the specified
resource from a process.
- dosetrlimit() was renamed to kern_setrlimit() to match existing style of
other similar syscall helper functions.
- The alpha OSF/1 compat layer no longer calls getrlimit() and setrlimit()
(it didn't used the stackgap when it should have) but uses lim_rlimit()
and kern_setrlimit() instead.
- The svr4 compat no longer uses the stackgap for resource limits calls,
but uses lim_rlimit() and kern_setrlimit() instead.
- The ibcs2 compat no longer uses the stackgap for resource limits. It
also no longer uses the stackgap for accessing sysctl's for the
ibcs2_sysconf() syscall but uses kernel_sysctl() instead. As a result,
ibcs2_sysconf() no longer needs Giant.
- The p_rlimit macro no longer exists.

Submitted by: mtm (mostly, I only did a few cleanups and catchups)
Tested on: i386
Compiled on: alpha, amd64

Use __FBSDID().

Correct typos, mostly s/ a / an / where appropriate. Some whitespace cleanup,
especially in troff files.

Use the fields in the sysentvec and in the vm map header in place of the
constants VM_MIN_ADDRESS, VM_MAXUSER_ADDRESS, USRSTACK and PS_STRINGS.
This is mainly so that they can be variable even for the native abi, based
on different machine types. Get stack protections from the sysentvec too.
This makes it trivial to map the stack non-executable for certain abis, on
machines that support it.

Infrastructure tweaks to allow having both an Elf32 and an Elf64 executable
handler in the kernel at the same time. Also, allow for the
exec_new_vmspace() code to build a different sized vmspace depending on
the executable environment. This is a big help for execing i386 binaries
on ia64. The ELF exec code grows the ability to map partial pages when
there is a page size difference, eg: emulating 4K pages on 8K or 16K
hardware pages.

Flesh out the i386 emulation support for ia64. At this point, the only
binary that I know of that fails is cvsup, because the cvsup runtime
tries to execute code in pages not marked executable.

Obtained from: dfr (mostly, many tweaks from me).

Modernize my email address.

Remove __P.

Make MAXTSIZ, DFLDSIZ, MAXDSIZ, DFLSSIZ, MAXSSIZ, SGROWSIZ loader
tunable.

Reviewed by: peter
MFC after: 2 weeks

Undo part of the tangle of having sys/lock.h and sys/mutex.h included in
other "system" header files.

Also help the deprecation of lockmgr.h by making it a sub-include of
sys/lock.h and removing sys/lockmgr.h form kernel .c files.

Sort sys/*.h includes where possible in affected files.

OK'ed by: bde (with reservations)

Fix a typo.

Reported by: albert

Proc locking.

useracc() the prequel:

Merge the contents (less some trivial bordering the silly comments)
of <vm/vm_prot.h> and <vm/vm_inherit.h> into <vm/vm.h>. This puts
the #defines for the vm_inherit_t and vm_prot_t types next to their
typedefs.

This paves the road for the commit to follow shortly: change
useracc() to use VM_PROT_{READ|WRITE} rather than B_{READ|WRITE}
as argument.

$Id$ -> $FreeBSD$

Correctly return ENOEXEC for really short zipped files. The way this is
done is less-than cute, but this whole file is suffering from some amount
of bitrot. Execution of zipped files should probably be implemented in a
manner similar to that of #!/interpreted files.

PR: kern/10780

Fix a couple of warnings and some bitrot in comments.

*_execsw static structures cannot be const due to the way they interact
with EXEC_SET, DECLARE_MODULE, and module_register. Specifically,
module_register. We may eventually be able to make these const, but
not now.

*gulp*. Jordan specifically OK'ed this..

This is the bulk of the support for doing kld modules. Two linker_sets
were replaced by SYSINIT()'s. VFS's and exec handlers are self registered.
kld is now a superset of lkm. I have converted most of them, they will
follow as a seperate commit as samples.
This all still works as a static a.out kernel using LKM's.

Cast u_longs to uintptr_t before casting them to pointers. Don't
attempt to even partially support systems with function pointers
larger than object pointers.

Use copyout() instead of bcopy() to copy the image to user space.
bcopy() caused panics under heavy paging (not quite as suspected -
the kernel stack seemed to get corrupted).

Fixed long lines.

Reviewed by: phk

Make gzipped dynamically linked executables work again. There was
an old bug here that failed to copy the a.out header into memory
properly. It didn't matter until changes were made recently to
the dynamic linker.

Removed unused #includes.

Removed redundant test against MAXDSIZ (the rlimit test is stronger).

Fully implement vfork. Vfork is now much much faster than even our
fork. (On my machine, fork is about 240usecs, vfork is 78usecs.)

Implement rfork(!RFPROC !RFMEM), which allows a thread to divorce its memory
from the other threads of a group.

Implement rfork(!RFPROC RFCFDG), which closes all file descriptors, eliminating
possible existing shares with other threads/processes.

Implement rfork(!RFPROC RFFDG), which divorces the file descriptors for a
thread from the rest of the group.

Fix the case where a thread does an exec. It is almost nonsense for a thread
to modify the other threads address space by an exec, so we
now automatically divorce the address space before modifying it.

Back out part 1 of the MCFH that changed $Id$ to $FreeBSD$. We are not
ready for it yet.

This is the kernel Lite/2 commit. There are some requisite userland
changes, so don't expect to be able to run the kernel as-is (very well)
without the appropriate Lite/2 userland changes.

The system boots and can mount UFS filesystems.

Untested: ext2fs, msdosfs, NFS
Known problems: Incorrect Berkeley ID strings in some files.
Mount_std mounts will not work until the getfsent
library routine is changed.

Reviewed by: various people
Submitted by: Jeffery Hsu <hsu@freebsd.org>

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Make DFLDSIZ and MAXDSIZ fully-supported options.

"Don't forget to do a ``make depend''" :-)

Fixed the easy cases of const poisoning in the kernel. Cosmetic.

Write protect the text segment of gzip'ed programs.
Various other cleanups by phk.

Reviewed by: phk
Submitted by: Igor Khasilev <igor@jabber.paco.odessa.ua>

First pass at cleaning up macros relating to pages, clusters and all that.

Removed unnecessary #includes from <sys/imgact.h> so that it is
self-sufficient and added explicit #includes where required.

Fixed unsigned longs that should have been vm_offset_t.

vm_offset_t is currently unsigned long but should probably be plain
unsigned for i386's to match the choice of minimal types to represent
for fixed-width types in Lite2. Anyway, it shouldn't be assumed
to be unsigned long.

I only fixed the type mismatches that were detected when I changed
vm_offset_t to unsigned. Only pointer type mismatches were detected.

rewrap some long lines.

Eliminated many redundant vm_map_lookup operations for vm_mmap.
Speed up for vfs_bio -- addition of a routine bqrelse to greatly diminish
overhead for merged cache.
Efficiency improvement for vfs_cluster. It used to do alot of redundant
calls to cluster_rbuild.
Correct the ordering for vrele of .text and release of credentials.
Use the selective tlb update for 486/586/P6.
Numerous fixes to the size of objects allocated for files. Additionally,
fixes in the various pagers.
Fixes for proper positioning of vnode_pager_setsize in msdosfs and ext2fs.
Fixes in the swap pager for exhausted resources. The pageout code
will not as readily thrash.
Change the page queue flags (PG_ACTIVE, PG_INACTIVE, PG_FREE, PG_CACHE) into
page queue indices (PQ_ACTIVE, PQ_INACTIVE, PQ_FREE, PQ_CACHE),
thereby improving efficiency of several routines.
Eliminate even more unnecessary vm_page_protect operations.
Significantly speed up process forks.
Make vm_object_page_clean more efficient, thereby eliminating the pause
that happens every 30seconds.
Make sequential clustered writes B_ASYNC instead of B_DELWRI even in the
case of filesystems mounted async.
Fix a panic with busy pages when write clustering is done for non-VMIO
buffers.

Untangled the vm.h include file spaghetti.

Staticized.

Added prototypes.

All:
Changed vnodep -> vp for consistency with the rest of the kernel, and
changed iparams -> imgp for brevity.

kern_exec.c:
Explicitly initialized some additional parts of the image_params struct
to avoid bzeroing it. Rewrote the set-id code to reduce the number of
logical tests. The rewrite exposed a mostly benign bug in the algorithm:
traced set-id images would get ktracing disabled even if the set-id didn't
happen for other reasons.

Remove trailing whitespace.

Add and move declarations to fix all of the warnings from `gcc -Wimplicit'
(except in netccitt, netiso and netns) and most of the warnings from
`gcc -Wnested-externs'. Fix all the bugs found. There were no serious
ones.

Use of vm_allocate() and vm_deallocate() has been deprecated.

Actually access the right first page if the file.
Bruce finally caught this bogon for me, Thank you Bruce !
Due to some part of the VM/buffer/pmap magic doing clustering, this bogon
managed to work better than 99.9% of the time. Amazing.
If You ever again see a weird message from the gzip code, please tell me.

Make the diagnostics a little more useful.

A word of wisdom, don't do this:

| cd /usr/bin
| for i in *
| do
| cp $i /tmp/a
| gzip -9 < /tmp/a > $i
| done

It will compress files with multiple links several times. do it this way:

| cd /usr/bin
| for i in *
| do
| gunzip -f < $i > /tmp/a
| gzip -9 < /tmp/a > $i
| done

I belive imgact_gzip is finally reentrant. It is also a whole lot more
readable.

inflate is now much more general, and is there if anybody feels like making
a uncompressing filesystem or something like that (hint hint !)

- remove unnecessary #includes (I think a couple of redundant ones remain)
- excise some unused code (#if 0'd out - don't want to nuke it yet)
- fix problems with "make depend" - some macros were screwing it up
- get rid of some static local variables

There still seems to be a small reentrancy problem somewhere.

First stage of getting imgact_gzip reentrant:
1) cut this up into /sys/sys/inflate.h, sys/kern/inflate.c
sys/kern/ingact_gzip.c
2) make a lot more things static
3) make a lot of globals const
4) make some args const
5) first stage of making globals into a struct (not used yet)

The vm_allocate() call which was introduced between revisions 1.4 and
1.5 of imagact_gzip.c broke things. I have backed that out for the time
being. (Davidg: help please)

WARNING: if you have gzip enabled in your kernel, you must now run
config again, as another source file has been added. Otherwise your
kernel compile will fall over.

This is all still WIP. More commits to come.

Suggestions from: phk.

Steven Wallace provided a program which broke this stuff. I guess there are
more weird kinds of a.out than anyone can argue for. This code failed to
load the first 28K of the text-segment, in the case where the first page
of the a.out contains only the a.out-header, and the text is still at 0x0.
Thanks Steven !

David Greenman told me to do this: (Thanks!) use vm_allocate to allocate
the uncompression buffer. Now malloc(M_GZIP) is used for all the Huffman-
tree stuff only. Numbers so far indicate < 15Kb Malloc use + 32 Kb for
the abovementioned buffer while uncompressing.

Added M_GZIP for the imgact_gzip code. The gzip-code is likely to be used
for other weird things in the future (hint, hint!)

Based on the applause (in this case: not downright rejection :-) I have
cleaned up much of the cruft in this thing.

No printf's in the case where things go well.

Gzip-headers can contain filenames and comments (as long as they're
shorter than the page-size.)

I don't think we leak memory, in the "exec/aout" code. I'm not quite sure
about the inflate code yet, but I don't think memory is lost.

Q: Can I add a class M_GZIP to <sys/malloc.h> and bump M_LAST one up
without any thing else needing tweaking ?

Poul-Henning

First bug-fix. This this depends on something odd. I am looking at it,
but every now and then it will fail without an explanation :-(

***
WARNING: THIS MATERIAL MIGHT GO AWAY!
This material needs the core-groups approval to stay here for the 2.0 release.
If the core-group does not concent to this commit, it will be backed out.
***

It is a non-gpl'ed "unzip" which will allow execution of a.out files which
have been sent through "gzip -9". The idea being saved disk-space.

Just now this code has quality rating: "working prototype".

To compress a file to be used with this, do it exactly this way:

gzip -9 -v < /bin/FOO > /tmp/FOO
remember to chmod /tmp/FOO as needed.

DON'T compress all of you binaries right away ! There are several things
which you should consider first:

1. Using compressed binaries, you use >MUCH< more VM, and thus swap-space.

2. It is slow.

3. It might crash your machine.

Apart from that, I welcome comments...

NB: There is also a change to sys/conf/files, but cvs core-dumped on me,
so it didn't get into the logs or emailed, but the commit seems to have
happended OK.