Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r227293 (by ed):
Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.

This means that their use is restricted to a single C file.

MFC r228864:
Discard local array based on return values.

Pointed out by: uqs
Found with: Coverity Prevent(tm)
CID: 10089

MFC 228023:
Add unicode support to ntfs

Obtained from: imura

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Fix ntfs such that it understand media with a non-512-bytes sector size:
1. Fixups are always done on 512 byte chunks (in stead of sectors). This
is kind of stupid.
2. Conevrt between NTFS blocknumbers (the blocksize equals the media
sector size) and the bread() and getblk() blocknr (which are 512-byte
sized)

NB: this change should not affect ntfs for 512-byte sector sizes.

Fix a number of style issues in the MALLOC / FREE commit. I've tried to
be careful not to fix anything that was already broken; the NFSv4 code is
particularly bad in this respect.

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

lockinit() can't accept LK_EXCLUSIVE as an initializaiton flag, so just
drop it.

Reported by: Josh Carroll <josh dot carroll at gmail dot com>
Submitted by: jhb

Fix a lock leak in the ntfs locking scheme:
When ntfs_ntput() reaches 0 in the refcount the inode lockmgr is not
released and directly destroyed. Fix this by unlocking the lockmgr() even
in the case of zero-refcount.

Reported by: dougb, yar, Scot Hetzel <swhetzel at gmail dot com>
Submitted by: yar

Cleanup lockmgr interface and exported KPI:
- Remove the "thread" argument from the lockmgr() function as it is
always curthread now
- Axe lockcount() function as it is no longer used
- Axe LOCKMGR_ASSERT() as it is bogus really and no currently used.
Hopefully this will be soonly replaced by something suitable for it.
- Remove the prototype for dumplockinfo() as the function is no longer
present

Addictionally:
- Introduce a KASSERT() in lockstatus() in order to let it accept only
curthread or NULL as they should only be passed
- Do a little bit of style(9) cleanup on lockmgr.h

KPI results heavilly broken by this change, so manpages and
FreeBSD_version will be modified accordingly by further commits.

Tested by: matteo

Remove explicit calling of lockmgr() with the NULL argument.
Now, lockmgr() function can only be called passing curthread and the
KASSERT() is upgraded according with this.

In order to support on-the-fly owner switching, the new function
lockmgr_disown() has been introduced and gets used in BUF_KERNPROC().
KPI, so, results changed and FreeBSD version will be bumped soon.
Differently from previous code, we assume idle thread cannot try to
acquire the lockmgr as it cannot sleep, so loose the relative check[1]
in BUF_KERNPROC().

Tested by: kris

[1] kib asked for a KASSERT in the lockmgr_disown() about this
condition, but after thinking at it, as this is a well known general
rule, I found it not really necessary.

Fix an integer overflow and allow access to files larger than 4GB on
NTFS.

o Typo in the debug message: s/skiped/skipped.

PR: kern/91346
Submitted by: Gavin Atkinson

Fix -Wundef.

Normalize a significant number of kernel malloc type names:

- Prefer '_' to ' ', as it results in more easily parsed results in
memory monitoring tools such as vmstat.

- Remove punctuation that is incompatible with using memory type names
as file names, such as '/' characters.

- Disambiguate some collisions by adding subsystem prefixes to some
memory types.

- Generally prefer lower case to upper case.

- If the same type is defined in multiple architecture directories,
attempt to use the same name in additional cases.

Not all instances were caught in this change, so more work is required to
finish this conversion. Similar changes are required for UMA zone names.

The printf(9) `%p' conversion specifier puts an "0x" in
front of the pointer value. Therefore, remove the "0x"
from the format string.

Fix our NTFS readdir function.

To check a directory's in-use bitmap bit by bit, we use
a pointer to an 8 bit wide unsigned value.

The index used to dereference this pointer is calculated
by shifting the bit index right 3 bits. Then we do a
logical AND with the bit# represented by the lower 3
bits of the bit index.

This is an idiomatic way of iterating through a bit map
with simple bitwise operations.

This commit fixes the bug that we only checked bits
3:0 of each 8 bit chunk, because we only used bits 1:0
of the bit index for the bit# in the current 8 bit value.
This resulted in files not being returned by getdirentries(2).

Change the type of the bit map pointer from `char *' to
`u_int8_t *'.

Make NTFS at least minimally usable after bufobj and GEOM fallout.

mmap() on NTFS files was hosed, returning pages offset from the
start of the disk rather than the start of the file. (ie, "cp" of
a 1-block file would get you a copy of the boot sector, not the
data in the file.) The solution isn't ideal, but gives a functioning
filesystem.

Cached vnode lookup was also broken, resulting in vnode haemorrhage.
A lookup on the same file twice would give you two vnodes, and the
resulting cached pages.

Just recently, mmap() was broken due to a lack of a call to
vnode_create_vobject() in ntfs_open().

Discussed with: phk@

Make a more whole-hearted attempt at GEOM'ifying NTFS.

I must have been sleepy when I did the first pass.

Spotted by: tegge

Avoid casts as lvalues.

Avoid sign extension when casting signed characters to unsigned wide
characters in ntfs_u28(). This fixes the conversion of filenames containing
single-byte characters with the high bit set.

- Support for multibyte charsets in LIBICONV.
- CD9660_ICONV, NTFS_ICONV and MSDOSFS_ICONV kernel options
(with corresponding modules).
- kiconv(3) for loadable charset conversion tables support.

Submitted by: Ryuichiro Imura <imura@ryu16.org>

Revise and improve ntfs_subr.c 1.30: read only a single cluster at a time
in ntfs_writentvattr_plain and ntfs_readntvattr_plain, and purge the boot
block from the buffer cache if isn't exactly one cluster long. These two
changes work around the same buffer cache bug that ntfs_subr.c 1.30 tried
to, but in a different way. This may decrease throughput by reading smaller
amounts of data from the disk at a time, but may increase it by avoiding
bogus writes of clean buffers.
Problem (re)reported by Karel J. Bosschaart on -current.

Merge from NetBSD src/sys/ntfs/ntfs_subr.c 1.5 & 1.30 (jdolecek):
- Avoid calling bread() with different sizes on the same blkno.
Although the buffer cache is designed to handle differing size
buffers, it erroneously tries to write the incorrectly-sized buffer
buffer back to disk before reading the correctly-sized one, even
when it's not dirty. This behaviour caused a panic for read-only
NTFS mounts when INVARIANTS was enabled ("bundirty: buffer x still
on queue y"), reported by NAKAJI Hiroyuki.
- Fix a bug in the code handling holes: a variable was incremented
instead of decremented, which could cause an infinite loop.

- Add a new 'flags' parameter to getblk().
- Define one flag GB_LOCK_NOWAIT that tells getblk() to pass the LK_NOWAIT
flag to the initial BUF_LOCK(). This will eventually be used in cases
were we want to use a buffer only if it is not currently in use.
- Convert all consumers of the getblk() api to use this extra parameter.

Reviwed by: arch
Not objected to by: mckusick

Back out M_* changes, per decision of the TRB.

Approved by: trb

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

Regularize the vop_stdlock'ing protocol across all the filesystems
that use it. Specifically, vop_stdlock uses the lock pointed to by
vp->v_vnlock. By default, getnewvnode sets up vp->v_vnlock to
reference vp->v_lock. Filesystems that wish to use the default
do not need to allocate a lock at the front of their node structure
(as some still did) or do a lockinit. They can simply start using
vn_lock/VOP_UNLOCK. Filesystems that wish to manage their own locks,
but still use the vop_stdlock functions (such as nullfs) can simply
replace vp->v_vnlock with a pointer to the lock that they wish to
have used for the vnode. Such filesystems are responsible for
setting the vp->v_vnlock back to the default in their vop_reclaim
routine (e.g., vp->v_vnlock = &vp->v_lock).

In theory, this set of changes cleans up the existing filesystem
lock interface and should have no function change to the existing
locking scheme.

Sponsored by: DARPA & NAI Labs.

Change callers of mtx_init() to pass in an appropriate lock type name. In
most cases NULL is passed, but in some cases such as network driver locks
(which use the MTX_NETWORK_LOCK macro) and UMA zone locks, a name is used.

Tested on: i386, alpha, sparc64

Remove __P.

Add a flags parameter to VFS_VGET to pass through the desired
locking flags when acquiring a vnode. The immediate purpose is
to allow polling lock requests (LK_NOWAIT) needed by soft updates
to avoid deadlock when enlisting other processes to help with
the background cleanup. For the future it will allow the use of
shared locks for read access to vnodes. This change touches a
lot of files as it affects most filesystems within the system.
It has been well tested on FFS, loopback, and CD-ROM filesystems.
only lightly on the others, so if you find a problem there, please
let me (mckusick@mckusick.com) know.

Expand LOCKMGR() compat macro.

GC non-FreeBSD code that didn't work anyways.

Missing KSE s/curproc/curthread/

KSE Milestone 2
Note ALL MODULES MUST BE RECOMPILED
make the kernel aware that there are smaller units of scheduling than the
process. (but only allow one thread per process at this time).
This is functionally equivalent to teh previousl -current except
that there is a thread associated with each process.

Sorry john! (your next MFC will be a doosie!)

Reviewed by: peter@freebsd.org, dillon@freebsd.org

X-MFC after: ha ha ha ha

Stole unicode translation table from mount_msdos. Add kernel code
to support this translation.

MFC after: 2 weeks

Fix opening particular file's attributes (as described in man page).
This is useful for debug purposes.

MFC after: 2 weeks

Reference devvp on ntnode creation and dereference on removal. Previous
code lead to page faults becouse i_devvp went zero after VOP_RECLAIM, but
ntnode was reused (not reclaimed).

MFC after: 2 weeks

Fix errors and warnings when compiling with NTFS_DEBUG > 1

MFC after: 2 weeks

- sys/n[tw]fs moved to sys/fs/n[tw]fs
- /usr/include/n[tw]fs moved to /usr/include/fs/n[tw]fs

Change and clean the mutex lock interface.

mtx_enter(lock, type) becomes:

mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks)
mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized)

similarily, for releasing a lock, we now have:

mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN.
We change the caller interface for the two different types of locks
because the semantics are entirely different for each case, and this
makes it explicitly clear and, at the same time, it rids us of the
extra `type' argument.

The enter->lock and exit->unlock change has been made with the idea
that we're "locking data" and not "entering locked code" in mind.

Further, remove all additional "flags" previously passed to the
lock acquire/release routines with the exception of two:

MTX_QUIET and MTX_NOSWITCH

The functionality of these flags is preserved and they can be passed
to the lock/unlock routines by calling the corresponding wrappers:

mtx_{lock, unlock}_flags(lock, flag(s)) and
mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN
locks, respectively.

Re-inline some lock acq/rel code; in the sleep lock case, we only
inline the _obtain_lock()s in order to ensure that the inlined code
fits into a cache line. In the spin lock case, we inline recursion and
actually only perform a function call if we need to spin. This change
has been made with the idea that we generally tend to avoid spin locks
and that also the spin locks that we do have and are heavily used
(i.e. sched_lock) do recurse, and therefore in an effort to reduce
function call overhead for some architectures (such as alpha), we
inline recursion for this case.

Create a new malloc type for the witness code and retire from using
the M_DEV type. The new type is called M_WITNESS and is only declared
if WITNESS is enabled.

Begin cleaning up some machdep/mutex.h code - specifically updated the
"optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN
and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently
need those.

Finally, caught up to the interface changes in all sys code.

Contributors: jake, jhb, jasone (in no particular order)

Mechanical change to use <sys/queue.h> macro API instead of
fondling implementation details.

Created with: sed(1)
Reviewed by: md5(1)

Convert more malloc+bzero to malloc+M_ZERO.

Submitted by: josh@zipperup.org
Submitted by: Robert Drehmel <robd@gmx.net>

Move suser() and suser_xxx() prototypes and a related #define from
<sys/proc.h> to <sys/systm.h>.

Correctly document the #includes needed in the manpage.

Add one now needed #include of <sys/systm.h>.
Remove the consequent 48 unused #includes of <sys/proc.h>.

Convert lockmgr locks from using simple locks to using mutexes.

Add lockdestroy() and appropriate invocations, which corresponds to
lockinit() and must be called to clean up after a lockmgr lock is no
longer needed.

Separate the struct bio related stuff out of <sys/buf.h> into
<sys/bio.h>.

<sys/bio.h> is now a prerequisite for <sys/buf.h> but it shall
not be made a nested include according to bdes teachings on the
subject of nested includes.

Diskdrivers and similar stuff below specfs::strategy() should no
longer need to include <sys/buf.> unless they need caching of data.

Still a few bogus uses of struct buf to track down.

Repocopy by: peter

Merged NetBSD version, as they have done improvements:
1. ntfs_read*attr*() functions now accept
uio structure to eliminate one data copying.
2. found and removed deadlock caused
by 6 concurent ls -lR.
3. started implementation of nromal
Unicode<->unix recodeing.

Obtained from: NetBSD

$Id$ -> $FreeBSD$

Decommision miscfs/specfs/specdev.h. Most of it goes into <sys/conf.h>,
a few lines into <sys/vnode.h>.

Add a few fields to struct specinfo, paving the way for the fun part.

Driver is now ported to NetBSD.

Submitted by: Christos Zoulas <christos@zoulas.com>

Removed annoying messaged during boot,added some check
before mounting (should help to do not mount extended partitions:-).
Fixed problem with hanging while unmounting busy fs.

And (the most important) added some locks to prevent
simulaneous access to kernel structures!

Added limited write ability. Now we can use some kind
of files for swap holders. See mount_ntfs..8 for details.

First version.
Reviewed by: David O'Brien <obrien@NUXI.com>