MFC r333239:
msdosfs: long names of files are created incorrectly.

This fixes a regression that happened in r120492 (2003) where libkiconv
was introduced and we went from checking unlen to checking for '\0'.

PR: 111843
Patch by: Damjan Jovanovic
Approved by: re (gjb)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

sys/fs: spelling fixes in comments.

No functional change.

msdosfs: Prevent buffer overflow when expanding win95 names

In win2unixfn() we expand Windows 95 style long names. In some cases that
requires moving the data in the nbp->nb_buf buffer backwards to make room. That
code failed to check for overflows, leading to a stack overflow in win2unixfn().

We now check for this event, and mark the entire conversion as failed in that
case. This means we present the 8 character, dos style, name instead.

PR: 204643
Differential Revision: https://reviews.freebsd.org/D6015

fs misc: for pointers replace 0 with NULL.

Mostly cosmetical, no functional change.

Found with devel/coccinelle.

Fix a bunch of -Wcast-qual warnings in msdosfs_conv.c, by using
__DECONST. No functional change.

MFC after: 3 days

ANSIfy msdosfs

Add a few cases and style(9) fixes missed in r276887

Sponsored by: The FreeBSD Foundation

ANSIfy sys/fs/msdosfs

There are a number of msdosfs improvements in NetBSD that may be worth
bringing over, and this reduces noise in the comparison.

Differential Revision: https://reviews.freebsd.org/D1466
Reviewed by: kib
Sponsored by: The FreeBSD Foundation

- Correct spelling in comments

Submitted by: Christoph Mallon <christoph.mallon@gmx.de> (via private mail)

- Correct mispellings of the word necessary

Submitted by: Christoph Mallon <christoph.mallon@gmx.de> (via private mail)

Discarding local array based on return values

Put all the messages from msdosfs under the MSDOSFS_DEBUG ifdef.
They are confusing to user, and not informative for general consumption.

MFC after: 1 week

Add unicode support to msdosfs and smbfs; original pathes from imura,
bug fixes by Kuan-Chung Chiu <buganini at gmail dot com>.

Tested by me in production for several days at work.

- Add idempotency guards so the structures can be used in other utilities.
- Update bpb structs with reserved fields.
- In direntry struct join deName with deExtension. Although a
fix was attempted in the past, these fields were being overflowed,
Now this is consistent with the spec, and we can now share the
WinChksum code with NetBSD.

Submitted by: Pedro F. Giffuni <giffunip tutopia com>
Mostly obtained from: NetBSD
Reviewed by: bde
MFC after: 2 weeks

Revert previous commit and add myself to the list of people who should
know better than to commit with a cat in the area.

Add necessary include.

Turn a "panic: non-decreasing id" into an error printf. This seems
to be caused by a metadata corruption that occurs quite often after
unplugging a pendrive during write activity.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Fix races in msdosfs_lookup() and msdosfs_readdir(). These functions
can easily block in bread(), and then there was nothing to prevent the
static buffer (nambuf_{ptr,len,last_id}) being clobbered by another
thread.

The effects of the bug seem to have been limited to failed lookups and
mangled names in readdir(), since Giant locking provides enough
serialization to prevent concurrent calls to the functions that access
the buffer. They were very obvious for multiple concurrent tree walks,
especially with a small cluster size.

The bug was introduced in msdosfs_conv.c 1.34 and associated changes,
and is in all releases starting with 5.2.

The fix is to allocate the buffer as a local variable and pass around
pointers to it like "_r" functions in libc do. Stack use from this
is large but not too large. This also fixes a memory leak on module
unload.

Reviewed by: kib
Approved by: re (kensmith)

Remove unused include(s).

Remove banal comments before includes.

Approved by: re (kensmith) (blanket)

Remove unused include(s).

Approved by: re (kensmith) (blanket)

o From the submitter: dos2unixchr will convert to lower case if
LCASE_BASE or LCASE_EXT or both are set. But dos2unixfn uses
dos2unixchr separately for the basename and the extension. So if
either LCASE_BASE or LCASE_EXT is set, dos2unixfn will convert both
the basename and extension to lowercase because it is blindly
passing in the state of both flags to dos2unixchr. The bit masks I
used ensure that only the state of LCASE_BASE gets passed to
dos2unixchr when the basename is converted, and only the state of
LCASE_EXT is passed in when the extension is converted.

PR: kern/86655
Submitted by: Micah Lieske
MFC after: 3 weeks

Replace slightly crummy fattime<->timespec conversion functions.

Use utc_offset() where applicable, and hide the internals of it
as static variables.

First part of a little cleanup in the calendar/timezone/RTC handling.

Move relevant variables to <sys/clock.h> and fix #includes as necessary.

Use libkern's much more time- & spamce-efficient BCD routines.

Do not assume that `char direntry::deExtension[3]' starts right after
`char direntry::deName[8]' and access deExtension[] explicitly.

Found by: Coverity Prevent(tm)
CID: 350, 351, 352

[1] unix2doschr()
If a character cannot be converted to DOS code page,
unix2doschr() returned `0'. As a result, unix2dosfn()
was forced to return `0', so we saw a file which was
composed of these characters as `Invalid argument'.
To correct this, if a character can be converted to
Unicode, unix2doschr() now returns `1' which is a magic
number to make unix2dosfn() know that the character
must be converted to `_'.

[2] unix2dosfn()
The above-mentioned solution only works if a file
has both of Unicode name and DOS code page name.
Unicode name would not be recorded if file name
can be settled within 11 bytes (DOS short name)
and if no conversion from Unix charset to DOS code
page has occurred. Thus, FreeBSD can create a file
which has only short name, but there is no guarantee
that the short name contains allways valid characters
because we leave it to people by using mount_msdosfs(8)
to select which conversion is used between DOS code
page and unix charset.
To avoid this, Unicode file name should be recorded
unless a character is an ascii character. This is
the way Windows XP do.

PR: 77074 [1]
MFC after: 1 week

Fix mbnambuf support for multi-byte characters. If a substring is larger
than WIN_CHARS bytes, we shift the suffix (previous substrings) upwards
by the amount this substring exceeds its WIN_CHARS slot. Profiling shows
this change is indistinguishable from the previous code at 95% confidence.
This bug would result in attempts to access or create files or directories
with multi-byte characters returning an error but no data loss.

Reported and tested by: avatar
MFC after: 3 days

Used unsigned version.

Submitted by: jmallett

Fix kernel build on 64-bit machines.

Correct a last-minute thinko. Instead of copying the nul with the string,
nul-terminate the dp->d_name directly and only copy the string.

The mbnambuf routines combine multiple substrings into a single
long filename. Each substring is indexed by the windows ID, a
sequential one-based value. The previous code was extremely slow,
doing a malloc/strcpy/free for each substring.

This code optimizes these routines with this in mind, using the ID
to index into a single array and concatenating each WIN_CHARS chunk
at once. (The last chunk is variable-length.)

This code has been tested as working on an FS with difficult filename
sizes (255, 13, 26, etc.) It gives a 77.1% decrease in profiled
time (total across all functions) and a 73.7% decrease in wall time.
Test was "ls -laR > /dev/null".

Per-function time savings:
mbnambuf_init: -90.7%
mbnambuf_write: -18.7%
mbnambuf_flush: -67.1%

MFC after: 1 month

Unroll the loop for calculating the 8.3 filename checksum. In testing
on my P3, microbenchmarks show the unrolled version is 78x faster. In
actual use (recursive ls), this gives an average of 9% improvement in
system time and 2% improvement in wall time.

/* -> /*- for copyright notices, minor format tweaks as necessary

Use size_t or ssize_t wherever appropriate instead of casting from int *
to size_t *, which is incorrect because they may have different widths.
This caused some subtle forms of corruption, the mostly frequently
reported one being that the last character of a filename was sometimes
duplicated on amd64.

Forced commit to note that my previous commit was a Darwin merge.

Make msdosfs long filenames matching case insensitive again.

PR: 59765
Submitted by: Ryuichiro Imura <imura@ryu16.org>

- Support for multibyte charsets in LIBICONV.
- CD9660_ICONV, NTFS_ICONV and MSDOSFS_ICONV kernel options
(with corresponding modules).
- kiconv(3) for loadable charset conversion tables support.

Submitted by: Ryuichiro Imura <imura@ryu16.org>

Split the global timezone structure into two integer fields to
prevent the compiler from optimizing assignments into byte-copy
operations which might make access to the individual fields non-atomic.

Use the individual fields throughout, and don't bother locking them with
Giant: it is no longer needed.

Inspired by: tjr

Fix winChkName() to match when the last slot contains nothing but the
terminating zero (it was treated as length missmatch). The mtools create
such slots if the name len is the product of 13 (max number of unicode
chars fitting in directory slot).

MFC after: 1 week

Remove __P.

- sys/msdosfs moved to sys/fs/msdosfs
- msdos.ko renamed to msdosfs.ko
- /usr/include/msdosfs moved to /usr/include/fs/msdosfs

$Id$ -> $FreeBSD$

Support NT VFAT lower case flags.

PR: 8383
(Mostly) Submitted by: Carl Mascott <cmascott@world.std.com>

Fix support for pre-Win95 filesystems: Make it possible to lookup just
created short file name. Don't insert "generation numbers".

Support compiling with `gcc -ansi'.

Fix a problem with the conversion of Unix filenames into the VFAT
namespace.
Submitted by: Dmitrij Tejblum <dima@tejblum.dnttm.rssi.ru>

Fixes for some bugs in the VFAT/FAT32 support:

- 'mv longnamedfile1 longnamedfile2' would cause longnamedfile2 to lose its
long name.
- Long names have trailing spaces/dots stripped for lookup as well as
assignment.
- A lockup when the mdsosfs was accessed from within the Linux emulator is fixed.
- A bug whereby long filenames were recognised by Microsoft operating systems but
not FreeBSD is fixed.

Submitted by: Dmitrij Tejblum <dima@tejblum.dnttm.rssi.ru>

Back out "always view in lowercase" part
Return to previous variant "comparing in lowercase" in winChkName

Implement loadable DOS<->local conversion tables for DOS names
Always create DOS name in uppercase
Always view DOS name in lowercase

Implement loadable upper->lower local conversion table

Reduce new arguments number added in my changes

Add Unicode support to winChkName, now lookup works!

Implement loadable local<->unicode file names conversion
Note: it produce correct names only for Win95, DOS names are still
incorrect and need similar work
mount_msdos support coming soon

Replace all unknown Unicode characters with '?' in win->unix mapping

Add initial support to map 0x4XX Unicode Cyrillic range names:
only win->unix part is implemented at this time with 256-byte
table defaulted to KOI8-R (will be loadable in future).
Since back mapping not supported yet, you'll get "No such file or directory"
on each Cyrillic name with 'ls -l', only 'echo *' work at this moment.
Teach current code to understand Unicode a bit.

Update MSDOSFS code using NetBSD's msdosfs as a guide to support
FAT32 partitions. Unfortunately, we looked around here at
Walnut Creek CDROM for any newer FAT32-supporting versions
of Win95 and we were unsuccessful; only the older stuff here.
So this is untested beyond simply making sure it compiles and
someone with access to an actual FAT32 fs will have
to let us know how well it actually works.
Submitted by: Dmitrij Tejblum <dima@tejblum.dnttm.rssi.ru>
Obtained from: NetBSD

Staticize.

Back out part 1 of the MCFH that changed $Id$ to $FreeBSD$. We are not
ready for it yet.

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Fixed an off by 1 error in unix2dostime(). The first day of each month
was converted to the last day of the previous month. This bug was
introduced in the optimizations in rev.1.4.

In sys/time.h, struct timespec is defined as:

/*
* Structure defined by POSIX.4 to be like a timeval.
*/
struct timespec {
time_t ts_sec; /* seconds */
long ts_nsec; /* and nanoseconds */
};

The correct names of the fields are tv_sec and tv_nsec.

Reminded by: James Drobina <jdrobina@infinet.com>

Fix adjkerntz expression priority.
Make filetimes the same as DOS times for UTC cmos clock.

Don't adjust file times for UTC clock to have the same timestamps
for DOS/FreeBSD.

staticize private parts.

Second batch of cleanup changes.
This time mostly making a lot of things static and some unused
variables here and there.

Remove trailing whitespace.

Fix timestamps when using Wall CMOS clock,
optimize dos2unixtime()
Submitted by: pritc003@maroon.tc.umn.edu

Fix numerous timestamp bugs.

DE_UPDATE was confused with DE_MODIFIED in some places (they do have
confusing names). Handle them exactly the same as IN_UPDATE and
IN_MODIFIED. This fixes chmod() and chown() clobbering the mtime
and other bugs.

DE_MODIFIED was set but not used.

Parenthesize macro args.

DE_TIMES() now takes a timeval arg instead of a timespec arg. It was
stupid to use a macro for speed and do unused conversions to prepare
for the macro.

Restore the left shifting of the DOS seconds count by 1. It got
lost among the shifts for the bitfields, so DOS seconds counts
appeared to range from 0 to 29 seconds (step 1) instead of 0 to 58
seconds (step 2).

Actually use the passed-in mtime in deupdat() as documented so that
utimes() works.

Change `extern __inline's to `static inline's so that msdosfs_fat.o
can be linked when it is compiled without -O.

Remove faking of directory mtimes to always be the current time. It's
more surprising for directory mtimes to change when you read the
directories than for them not to change when you write the directories.
This should be controlled by a mount-time option if at all.

Added declarations, fixed bugs due to missing decls. At least one of them
could panic a system. (I know, it paniced mine!).

Added msdosfs.

Obtained from: NetBSD