MFC r304910, r304912, r304915, r304952, r325019, r328164, r331094, r332664,
r335341-r335345, r335347, r335379-r335380, r335382

r304910:
Introduce cnv man page.

r304912:
Add missed header file for cnv.h .

r304915:
Bump date in the man page.

r304952:
Remove duplicated declaration.

r325019:
Introduce cnvlist_name() and cnvlist_type() functions.

Those function can be used when we are iterating over nvlist to reduce
amount of extra variables we need to declare.

r328164:
libnv: Use mallocarray(9) for the nv_calloc.

r331094:
Fix formatting errors that resulted in apropos(1) output looking weird.

r332664:
Add missing argument in the cnv man page.

r335341:
libnv: change name of cookie from cookiep to cookie.

The name was inconsistent with rest of the library.
No functional change intended.

r335342:
libnv: add const to cookies arguments

r335343:
libnv: Remove nvlist argument from cnvlist_{take,free}_* functions.

All information which are need for those operations is already stored in
the cookie.

We decided not to bump libnv version because this API is not used yet in the
base system.

r335344:
libnv: clean parent in nvlist_array when removing it.

When we are removing element form the nvlist we should also clean parent,
because the array is not a part of the nvlist anymore.

r335345:
libnv: add regression test for r335344.

r335347:
libnv: Add nvlist_append_*_array() family of functions.

The nvlist_append_{bool,number,string,nvlist,descriptor}_array() functions
allows to dynamically extend array stored in the nvlist.

r335379:
Set prev to NULL so its garaunteed to have a value of some kind and
gcc doesn't explode. Feel free to fix this correctly or whatever for
gcc builds.

This *should* quiesce tinderbox after r335347 for the gcc builds.

r335380:
style(9) fix, I was also going to silence gcc.

r335382:
Really fix the style.

MFC r314709,r314790,r314794:

r314709:
Fix bootstrapping mtree after r313404 for older systems.
r314790:
Added comments for why nmtree/libmd are bootstrapped.
r314794:
Fix bootstrapping libmd on older systems after r314709.

PR: 217673

MFC r312452-r312512:

r312452-r312512:

- Use SRCTOP-relative paths to other directories instead of .CURDIR-relative ones

This simplifies pathing in make/displayed output

- Use .CURDIR:H instead of .CURDIR to simplify pathing in output, etc

MFC r307521: libmd: introduce functions that operate on fd instead of filename

MFC r307584

Fix C++ includability of crypto headers with static array sizes

C99 allows array function parameters to use the static keyword for their
sizes. This tells the compiler that the parameter will have at least the
specified size, and calling code will fail to compile if that guarantee is
not met. However, this syntax is not legal in C++.

This commit reverts r300824, which worked around the problem for
sys/sys/md5.h only, and introduces a new macro: min_size(). min_size(x) can
be used in headers as a static array size, but will still compile in C++
mode.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Import the skein hashing algorithm, based on the threefish block cipher

Connect it to userland (libmd, libcrypt, sbin/md5) and kernel (crypto.ko)

Support for skein as a ZFS checksum algorithm was introduced in r289422
but is disconnected because FreeBSD lacked a Skein implementation.

A further commit will enable it in ZFS.

Reviewed by: cem
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D6166

Implement SHA-512 truncated (224 and 256 bits)

This implements SHA-512/256, which generates a 256 bit hash by
calculating the SHA-512 then truncating the result. A different initial
value is used, making the result different from the first 256 bits of
the SHA-512 of the same input. SHA-512 is ~50% faster than SHA-256 on
64bit platforms, so the result is a faster 256 bit hash.

The main goal of this implementation is to enable support for this
faster hashing algorithm in ZFS. The feature was introduced into ZFS
in r289422, but is disconnected because SHA-512/256 support was missing.
A further commit will enable it in ZFS.

This is the follow on to r292782

Reviewed by: cem
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D6061

libmd: Work around C++'s inability to understand C

Reported by: antoine@ (x265)

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

DIRDEPS_BUILD: Regenerate without local dependencies.

These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.

Sponsored by: EMC / Isilon Storage Division

Improvements to the MDXFileChunk() template function:
- Remove unneeded fstat()/lseek() calls.
- Return NULL and set errno to EINVAL on negative length.
- Fix small style problems and expand variable names.

After this change, it is possible to use this code for some irregular
files. For example, 'md5 /dev/md0' should now succeed.

Differential Revision: https://reviews.freebsd.org/D4748
Suggested by: bde
Reviewed by: bde, allanjude, delphij

DIRDEPS_BUILD: Update dependencies.

Sponsored by: EMC / Isilon Storage Division

Fix a file descriptor leak in mdXhl.c (which is used by numerous hashing
algorithms.

CID: 1305669,1305611,1305663,1305603,1305584,1305639,1346865,1305601
Differential Revision: https://reviews.freebsd.org/D4732
Reviewed by: allanjude, delphij
MFC after: 2 weeks
Sponsored by: Juniper Networks

Replace sys/crypto/sha2/sha2.c with lib/libmd/sha512c.c

cperciva's libmd implementation is 5-30% faster

The same was done for SHA256 previously in r263218

cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation

Extend sbin/md5 to create sha384(1)

Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}

Reviewed by: cperciva, des, delphij
Approved by: secteam, bapt (mentor)
MFC after: 2 weeks
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3929

META MODE: Remove unneeded libmd.host dependency for xinstall.host.

This is chasing r291026.

Sponsored by: EMC / Isilon Storage Division

META MODE: Fix 'make bootstrap-tools'.

The main problem was bitrot after elftoolchain being swapped in for the
GNU toolchain.

This also reworks how the list of 'host allowed' libraries is determined
to only allow INTERNALLIBs, which is needed for libelftc to come in.

For usr.bin/readelf use the same hack, as libelf and libdward, to bring in
the needed sys/ headers for host builds. This has not yet been a problem due
to readelf not being built as a host tool in buildworld. This is possible
in the meta build though when building the toolchain.

Sponsored by: EMC / Isilon Storage Division

META MODE: Prefer INSTALL=tools/install.sh to lessen the need for xinstall.host.

This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.

Sponsored by: EMC / Isilon Storage Division

Add new include path for sha256.h

This fixes the bootstrap build on FreeBSD 10.

Submitted by: andrew

Complete the move that was started w/ r263218.. For some reason I
didn't delete the files, so that means we need to bring the changes in
r282726 to the correct files..

make tinderbox completed with this patch...

increase buffer size to significantly increase performance...

see:
https://docs.freebsd.org/cgi/mid.cgi?20150513080342.GE37063@funkthat.com

for benchmarks...

new depends

Revert r284417 it is not necessary anymore

Enforce overwritting SHLIBDIR

Since METAMODE has been added, sys.mk loads bsd.mkopt.mk which ends load loading
bsd.own.mk which then defines SHLIBDIR before all the Makefile.inc everywhere.

This makes /lib being populated again.

Reported by: many

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

Unbreak MIPS build following rev. 282726

Introduce further adjustments to the renaming of libmd
symbols: make sure that we do not generate dangling weak
aliases, as this causes build failures on MIPS.

Tested by: sbruno

Unbreak build following rev. 282726

(Makefile.inc1): add dependency of xinstall on libmd to
avoid failure of parallel bootstrap.

(lib/libmd/*.h): do not redefine symbols if already
defined as macros (libcrypt uses the same sources internally,
redefining symbols with a prefix of its own).

Fixes build failures caused by previous change to libmd.

Reported by: ian
Pointy hat to: thomas

Ensure libmd symbols do not clash with libcrypto

Add a prefix to all symbols in libmd to avoid incompatibilites
with same-named, but not binary compatible, symbols from libcrypto.

Also introduce Weak aliases to avoid the need to rebuild dependent
binaries and a major version bump.

PR: 199119
Differential Revision: D2216
Reviewed by: roberto, delphij
MFC after: 2 weeks

Avoid an infinite loop by ensuring that the amount of bytes read is greater
than 0 in MDXFileChunk when calculating the checksum

This edgecase can be triggered if the file is truncated while the checksum
is being calculated (i.e. the EOF is reached)

Differential Revision: https://reviews.freebsd.org/D2351 (patch by darius)
PR: 196694
Reviewed by: delphij, ngie
Submitted by: Daniel O'Connor <darius@dons.net.au>
Sponsored by: EMC / Isilon Storage Division

Revert r274772: it is not valid on MIPS

Reported by: sbruno

Use canonical __PIC__ flag

It is automatically set when -fPIC is passed to the compiler.

Reviewed by: dim, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D1179

use .Mt to mark up email addresses consistently (part3)

PR: 191174
Submitted by: Franco Fichtner <franco at lastsummer.de>

Dereference nonexistent md2(3) manual.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

replace the kernel's version w/ cperciva's implementation... In all
my tests, it is faster ~20%, even on an old IXP425 533MHz it is ~45%
faster... This is partly due to loop unrolling, so the code size does
significantly increase... I do plan on committing a version that
rolls up the loops again for smaller code size for embedded systems
where size is more important than absolute performance (it'll save ~6k
code)...

The kernel implementation is now shared w/ userland's libcrypt and
libmd...

We drop support for sha256 from sha2.c, so now sha2.c only contains
sha384 and sha512...

Reviewed by: secteam@

use a real uint64_t instead of writing code to emulate one..

I verified w/ a:
dd if=/dev/zero bs=1m count=5000 | sha256
a33351fafd00e4c4bcdee2a1c5d019026500f8cdfeaf91a9b8dbbb2619429659

Reviewed by: cperciva
MFC after: 1 week

Remove the RFC 1319 MD2 Message-Digest Algorithm routines from libmd.

1. The licensing terms for the MD2 routines from RFC is not under a BSD-like
license. Instead it is only granted for non-commercial Internet
Privacy-Enhanced Mail.
2. MD2 is quite deprecated as it is no longer considered a cryptographically
strong algorithm.

Discussed with: so (cperciva), core

Rewrite HISTORY: The SHA256 code first appeared in 6.0, and the SHA512
code in 9.0; neither existed in FreeBSD 4.0.

Add SHA512 (Actually, this is Colin Percival's code for SHA256, with
relevant constants changed).

While I'm here clean up the tests and Makefile.

PR: misc/124164
Submitted by: KIMURA Yasuhiro < yasu utahime org >
MFC after: 1 month

Fix some leftover binaries and shared libraries in the system that still
have an executable stack, due to linking in hand-assembled .S or .s
files, that have no .GNU-stack sections:

RWX --- --- /lib/libcrypto.so.6
RWX --- --- /lib/libmd.so.5
RWX --- --- /lib/libz.so.6
RWX --- --- /lib/libzpool.so.2
RWX --- --- /usr/lib/liblzma.so.5

These were found using scanelf, from the sysutils/pax-utils port.

Reviewed by: kib

Build lib/ with WARNS=6 by default.

Similar to libexec/, do the same with lib/. Make WARNS=6 the norm and
lower it when needed.

I'm setting WARNS?=0 for secure/. It seems secure/ includes the
Makefile.inc provided by lib/. I'm not going to touch that directory.
Most of the code there is contributed anyway.

Remove reference to no longer existing document.

Spotted by: Bjoern Voigt <bjoern@cs.tu-berlin.de>

Make the "test" target test the compiled version, instead of the
installed version of the md library.

Use unsigned comparisons. Prior to this commit, SHA1_Update and
RIPEMD160_Update were broken when all of the following conditions
applied:
(1) The platform is i386.
(2) The program calling *_Update is statically linked to libmd.
(3) The buffer provided to *_Update is aligned modulo 4 bytes.
(4) The buffer extends beyond 2GB.

Due to the design of this code, SHA1_Update and RIPEMD160_Update will
still be broken if conditions (1)-(3) apply AND the buffer extends
beyond 4GB (i.e., there is an integer overflow in computing "data + len").
Since this remaining bug simply replaces SIGSEGV with a bogus hash (and
non-broken programs should never provide such operands) I don't consider
it to be a serious problem.

MFC After: 1 week
PR: kern/102795

Fix an 11 year old mistake: Let the hash functions take a void* instead
of unsigned char* argument.

-mdoc sweep.

Correct type.

OK'ed by: cperciva

Removed redundnt empty line.

Approved by: re (blanket)

Markup fixes.

Approved by: re

In light of the recent 2^69 operation collision-finding attack on SHA1,
add support for SHA256.

Tested on: i386, sparc64
Tested using: NIST test vectors, built-in tests
X-MFC-after: 5.4-RELEASE

The third parameter to SHA_Update and SHA1_Update is a "size_t", not a
"unsigned int".

Expand contractions.

Expand *n't contractions.

Sort sections.

Eliminate double whitespace.

Backed out rev.1.6 and subsequent copying of it (bogus addition of
-static to CFLAGS). It just turned rev.1.5 into an obfuscated no-op.
As explained in the log for rev.1.5, testing should be done in the
host environment but there is a problem in cross-compilation environments.
As not explained in the log for rev.1.6, there was apparently a practical
problem with cross-compiling (makeworld should have set -static in
LDFLAGS but apparently didn't). Cross-compilation was especially
complicated because the relevant programs are test programs that were
run at beforeinstall time -- dynamic libraries might or might not exist
depending on the build options. The complications became moot in
rev.1.8 when beforeinstall was renamed "test".

Assorted mdoc(7) fixes:

- fix hard sentence breaks
- sprinkle a few .Vt's where neccessary
- remove incorrect use of `\-'
- proper quoting using .Dq, instead of manual ``...''

Approved by: des@ (mentor)
Reviewed by: ru@

style.Makefile(5)

Stage 3 of dynamic root support. Make all the libraries needed to run
binaries in /bin and /sbin installed in /lib. Only the versioned files
reside in /lib, the .so symlink continues to live /usr/lib so the
toolchain doesn't need to be modified.

Rename local variables to not mask global names of same name. This
fixes lots of lint(1) warnings.

The .Fn function

Initiate deorbit burn for the i386-only a.out related support. Moves are
under way to move the remnants of the a.out toolchain to ports. As the
comment in src/Makefile said, this stuff is deprecated and one should not
expect this to remain beyond 4.0-REL. It has already lasted WAY beyond
that.

Notable exceptions:
gcc - I have not touched the a.out generation stuff there.
ldd/ldconfig - still have some code to interface with a.out rtld.
old as/ld/etc - I have not removed these yet, pending their move to ports.
some includes - necessary for ldd/ldconfig for now.

Tested on: i386 (extensively), alpha

Get this file closer to style(9).

Fix a bug where calling MD5File(3) with a zero-length file generated an
error, due to an uninitialized variable.

Reviewed by: phk, archie

Add missing const's.

Improve the handling of Encode and Decode operations in MD5.

Use memcpy for all little-endian architectures, sys/kern/md5c.c indicates
this should be safe for all currently supported LE archs.

Change the Encode and Decode functions for other archs to use le32toh()
and htole32() functions instead of explicit byte shuffling.

On sparc64 this gives md5(1) about 8% speed increase.

Don't build manuals at inappropriate time.
Collapse generation of md[245].3 manpages.

Modernize my email address.

Modernize my email address.

Remove __P() usage.

Remove 'register' keyword.

mdoc(7) police: Use the new .In macro for #include statements.

add __FBSDID()s to libmd

mdoc(7) police:

Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text.
Not only this slows down the mdoc(7) processing significantly, but it also
has an undesired (in this case) effect of disabling hyphenation within the
entire enclosed block.

Remove whitespace at EOL.

mdoc(7) police: removed HISTORY info from the .Os call.

mdoc(7) police: use proper markup for function arguments.

MAN[1-9] -> MAN.

Add a new entrypoint to the hashes in libmd:
char *
FooFileChunk(const char *filename, char *buf, off_t offset, off_t length)
Which only hashes part of a file.
Implement FooFile() in terms of this function.

Submitted by: roam

mdoc(7) police: removed history info from the .Os FreeBSD call.

Introduce .Lb macro to libmd manpages

Replace beforeinstall target with new variables used by .mk system.

Reviewed by: marcel, and make world

Change #ifdef KERNEL to #ifdef _KERNEL in the public headers. "KERNEL"
is an application space macro and the applications are supposed to be free
to use it as they please (but cannot). This is consistant with the other
BSD's who made this change quite some time ago. More commits to come.

$Id$ -> $FreeBSD$

add more MLINKS

Don't even attempt to build the assembly-language versions of RIPEMD160
and SHA-1 when OBJFORMAT is not ELF. Add a warning to the man page
about how SHA-1 uses bswapl, which will trap on 80386es (and the kernel
should, but doesn't currently, emulate).

fix for incorrect specification of alignment (it worked for the elf
build, but broke while doing the aout legacy build). Now using
.p2align instead of .align. Fixes broken buildworld.

Submitted by: John Polstra
Reviewed by: John Polstra

No, this isn't what I submitted. Seems like the patch I sent
was ignored and an incorrect hand edit was used instead.

Fixes for Alpha.

Submitted by: mjacob

Don't build the assembly for shared linkage, PIC isn't implemented.

Add Eric Young's RIPEMD160 implementation as well, in case SHA-1
should prove weak. Also fix a few problems with the SHA-1 build.

Fix bug in MDx test suite.
Add Eric Young's SHA-[01] implementations.

Add a note about the insecurity of MD4 and potential vulnerability of
MD5 to similar attacks.

Build libmd shared for a.out too. Required for some PAM modules.

$@ -> ${.TARGET}

BINFORMAT -> OBJFORMAT ready for E-day.

Split the padding out into a separate function.
Synchronize the kernel and libmd versions of md5c.c

PR: misc/6127
Reviewed by: phk
Submitted by: Ari Suutari <ari@suutari.iki.fi>

Don't use the beforedepend target. It was a no-op here except for
helping bsd.dep.mk break `make -jN depend'.

.Sh AUTHOR -> .Sh AUTHORS. Use .An/.Aq

If not building a.out, still build a pic version of libmd. This allows
things like libskey.so to be dynamically self contained.

Things like md5(1) where speed is critical should still link with libmd.a,
but for things like login, where it's a once-off call if skey is used, it's
not worth the hassle.

Make the MD* header files C++-aware. Also, string arguments are supposed
to be of type `const char *'.

PR: 3291
Submitted by: dima@tejblum.dnttm.rssi.ru (Dmitrij Tejblum)

FIxed arg types (mostly missing consts) in synopsis.

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Just pick up the real thing from <sys/md5.h>

Make this compile in the kernel too, major cosmetic cleanup.

Moved #include of <sys/types.h> earlier so that this compiles when
<stdio.h> doesn't (bogusly) include <sys/types.h>.

Cleaned up #includes.

Add #include <sys/types>

close bin/1648 libmd not 64bit safe.

if something fails to compile now, you need to add #include <sys/types.h>

Partially Submitted by: Jason Thorpe <thorpej@nas.nasa.gov>

Use -mdoc macros properly.
Translate from Danglish into English.
Refer to the alogrithms as ``RSA Data Security MDX message digest'' as
required by the license.

cmp -s || install -c ==> install -C, use a .for loop

Some minor man page cleanup.

Fix a memory leak in MD[245]End()

Submitted by: Ikuo Nakagawa <ikuo@isl.intec.co.jp>
PR: misc/1424

Correct two errors in the manual page.

Don't install mdX.3, it's the template used to build the other pages.

Bill was a little to fast here...

Add some missing MLINKS, correct some cross references, correct some
file locations and some minor formatting/style problems.

Added some missing MLINKS for section 3 man pages.
Also corrected a few minor formatting errors, file location and cross
references in some of the section 3 man pages.

This shuts up a lot of the output from "manck" for section 3.

Add a structure definition to the MD* Contexts, so that cvs can use
the standard libmd version of MD5 instead of it's own seperate copy..

Back out that last change; we don't build this shared.

Bumped shared lib rev to 2.1.

Change `install' to `${INSTALL}' so that default install flags can be
specified in the top level Makefiles.

Previously I missed dozens of Makefiles that skip the install after
using `cmp -s' to decide that the install isn't necessary.

Change this to do what it should have done from the start.
Add argument for buffer for output.
Fix manuals.

Remove trailing whitespace.

No sense in copying the hex[] array in each call to MDXEnd(), it's a
constant. (There probably ought to be a single libc version of it.)

Clean a bunch of -Wall warnings.

Speed md5 up around 30% by shorting out a couple of cumbersome
memcpy equivalent functions.

Eliminate a bogus tab.

Added "const" to the arguments here and there.

Clean up beforeinstall

Added beforeinstall rule to install .h files. We might need some .mk
support for this kind of thing. Look at src/Makefile target "includes".

libmd no longer built as shared-lib, only static.
Renamed the beforeinstall to test.

Build *.3 and *.ref at build time, not at install time.

Make mddriver static.
Submitted by: jkh

Use ${LDFLAGS} instead of static for compiling binaries. Neither is
sufficient for cross compiling but it's best to test with the flags
normally used.

Use ${ECHO} instead of `echo' so that `make -s' is fairly quiet.

Added ${.CURDIRb to vuilding of man pages so it works with obj

Submitted by: Paul Richards

Boy, was *this* ever bollixed!

1. Copyright files looked for in the wrong place

2. cmp was looking in wrong place for test data.

3. Driver for test not linked static, thus dynamic resolution of library
not working.

4. Man page installation not consistent with source.

Reviewed by:
Submitted by: jkh

Make obj directory work right. md?hl.c should probably be
deleted, but I'll leave that up to PHK.

This commit was generated by cvs2svn to compensate for changes in r1802,
which included commits to RCS files with non-trunk default branches.