MFC r316312, r332361, r333438-r333439, r339804: GELI dry-run

r316312:
sys/geom/eli: Switch bzero() to explicit_bzero() for sensitive data

In GELI, anywhere we are zeroing out possibly sensitive data, like
the metadata struct, the metadata sector (both contain the encrypted
master key), the user key, or the master key, use explicit_bzero.

Didn't touch the bzero() used to initialize structs.

r332361:
Introduce dry run option for attaching the device.
This will allow us to verify if passphrase and key is valid without
decrypting whole device.

r333438:
Change option dry-run from 'n' to 'C' in geli attach command.

'n' is used in other commands to define the key index.
We should be consistent with that.
'C' option is used by patch(1) to perform dryrun so lets use that.

r333439:
Introduce the 'n' flag for the geli attach command.

If the 'n' flag is provided the provided key number will be used to
decrypt device. This can be used combined with dryrun to verify if the key
is set correctly. This can be also used to determine which key slot we want to
change on already attached device.

r339804:
Restore backward compatibility for "attach" verb.

In r332361 and r333439, two new parameters were added to geli attach
verb using gctl_get_paraml, which requires the value to be present.
This would prevent old geli(8) binary from attaching geli(4) device
as they have no knowledge about the new parameters.

Restore backward compatibility by treating the absense of these two
values as seeing the default value supplied by userland.

MFC r308137, r316312, r332361

r308137:
Fix alignment issues on MIPS: align the pointers properly.

All the 5520 GEOM_ELI tests passed successfully on MIPS64EB.

r316312:
sys/geom/eli: Switch bzero() to explicit_bzero() for sensitive data

In GELI, anywhere we are zeroing out possibly sensitive data, like
the metadata struct, the metadata sector (both contain the encrypted
master key), the user key, or the master key, use explicit_bzero.

Didn't touch the bzero() used to initialize structs.

r332361:
Introduce dry run option for attaching the device.
This will allow us to verify if passphrase and key is valid without
decrypting whole device.

MFC Loader Fixes 2017q3: r320547,r320553,r321621,r321844,r321969,r321991,
r322037,r322038,r322039,r322040,r322056,r322074,r322542,r322592,r322593,
r322896,r322923,r323671,r322930,r322931,r322932,r322933,r322934,r322935,
r322936,r322937,r322938,r322939,r322941,r323062,r323063,r323064,r323065,
r323100,r323131,r323174,r323258,r323261,r323272,r323367,r323379,r323389,
r323407,r323428,r323436,r323494,r323496,r323497,r323541,r323554,r323589,
r323707,r323867,r323885,r323886,r323895,r323896,r323897,r323905,r323906,
r323907,r323908,r323909,r323952,r323991,r324099,r324558,r326445,r326609,
r326610

This batch includes a special kludge to fix powerpc loader build; <stdlib.h>
was included after <stand.h> there, causing problems with DEBUG_MALLOC bits.
Include <stdlib.h> a little bit earlier to fix the build with the intention
of removing this when eventually libsa silently replaces stdlib.h with
stand.h.

r320547: Link EFI/uboot loaders with -znotext

r320553: Integer underflow in efipart_realstrategy when I/O starts after end
of disk

r321621: Always set the receive mask in loader.efi.

r321844: Clean up style in print_state(..) and pager_printf(..)

r321969: Fix the return types for printf and putchar to match their libc

r321991: Revert r321969

r322037: Add stpcpy and stpncpy to libstand

r322038: Add definitions and utilities for EFI drivers

r322039: Move EFI ZFS functions to libefi

r322040: Add EFI utility functions to libefi

r322056: Move EFI fmtdev functionality to libefi

r322074: libefi/time.c cstyle cleanup

r322542: loader.efi: repace XXX with real comments in trap.c

r322592: Remove unused defines.

r322593: Define proposed GUID for FreeBSD boot loader variables.

r322896: Make spinconsole platform independent and hook it up into EFI
loader

r322923: Hide length of geli passphrase during boot.

r323671: Fix language used in the r322923.

r322930: Move efi_main into efi/loader

r322931: Cleanup efi_main return type

r322932: Use the loader.efi conventions for the various EFI tables.

r322933: No need for MK_ZFS around these: they are by their nature only
active when MK_ZFS is true.

r322934: _STAND is sometimes defined on the command line. Make the define
here match.

r322935: Fix warnings due to type mismatch.

r322936: Remove useless 'static' for an enum definition.

r322937: Forward declare struct dsk to avoid warnings when building libi386.

r322938: Link in libefi for boot1

r322939: Use efi_devpath_str for debug path info.

r322941: Eliminate redunant device path matching.

r323062: Make efichar.c routines available to libefi.

r323063: boot1.efi: print more info about where boot1.efi is loaded from

r323064: Exit rather than panic for most errors.

r323065: Save where we're booted from

r323100: libstand: nfs_readlink() should return proper return code

r323131: Revert r322941: Eliminate redundant device matching functions

r323174: Fix loader bug causing too many pages allocation when bootloader
is U-Boot

r323258: ucs2len

r323261: Fix armv6 build

r323272: Be consistent and do return (1);

r323367: Mark init_chroot and init_script variables as deprecated.

r323379: It's been pointed out that init_script at least is useful w/o

r323389: loader.efi: chain loader should provide proper device handle

r323407: boot1 generate-fat: generate all templates at once

r323428: r323389 breaks the kernel build when WITHOUT_ZFS is defined in
src.conf

r323436: boot1: remove BOOT1_MAXSIZE default value

r323494: loader should support large_dnode

r323496: libstand: tftp_open() can leak pkt on error

r323497: libefi: efipart_open should check the status from disk_open

r323541: libefi: efipart_realstrategy rsize pointer may be NULL

r323554: Increase EFI boot file size frok 128k to 384k

r323589: loader: biosmem.c cstyle cleanup

r323707: loader: biosmem allocate heap just below 4GB

r323867: libefi: devicename.c cleanups

r323885: libefi: efi_devpath_match() should return bool

r323886: libefi: efipart.c should use calloc()

r323895: libefi: efi_devpath_match local len should be unsigned

r323896: r323885 did miss efilib.h update

r323897: efilib.h: typo in structure member description

r323905: libefi: pdinfo_t pd_unit and pd_open should be unsigned

r323906: libefi: efipart_strategy() should return ENXIO when there is no
media

r323907: libefi: efipart.c cstyle fix for efipart_print_common()

r323908: libefi: efipart_hdinfo_add_filepath should check strtol result

r323909: libefi: define EISA PNP constants

r323952: After the r317886 support for TFTP and NFS can be enable
simultaneously.

r323991: libefi: efipart_floppy() will should not pass acpi pointer if the
HID test fails

r324099: Compile loader as Little-Endian on PPC64/POWER8

r324558: Define prototype for exit and ensure references

r326445: Fix random() and srandom() prototypes to match the standard.

r326609: Make putenv and getenv match the userland definition

r326610: Fix random() prototype to match the system.

PR: 219000 221001 222215
Relnotes: yes ("The length of the geli passphrase is hidden during boot")

MFC Loader Fixes 2017q1: r311458,r312237,r312314,r312374,r312947,r313042,
r313047,r313166,r313328,r313332,r313333,r313337,r313348,r313349,r313389,
r313442,r313451,r313575,r313645,r313710,r314114,r314213,r314275,r314945,
r314948,r315008,r315408,r315427,r315645,r315646,r315648,r315653,r315850,
r316064,r316078,r316079,r316100,r316104,r316111,r316112,r316171,r316279,
r316280,r316287,r316311,r316343,r316424,r316436

r311458: Use compiler driver to link BERI boot loaders

r312237: loader.efi: find_currdev() can leak memory

r312314: loader: move device path definitions to include/efidevp.h

r312374: loader: efi devpath api usage should be more aware of NULL pointers

r312947: Remove "-Xassembler -G0" from CFLAGS.

r313042: loader.efi environment related cleanups

r313047: loader: disk/part api needs to use uint64_t offsets

r313166: loader: libefi/env.c warnings in arm build

r313328: loader: Implement disk_ioctl() to support DIOCGSECTORSIZE and
DIOCGMEDIASIZE.

r313332: loader: bcache read ahead block count should take account the large
sectors

r313333: loader: Replace EFI part devices.

r313337: loader: 313329 missed ZFS guard in loader/main.c

r313348: loader: biosdisk fix for 2+TB disks

r313349: loader: disk io should not use alloca()

r313389: efipart is also using the '%S' printf format, add -Wno-format for
it.

r313442: loader: possible NULL pointer dereference in efipart.c

r313451: loader: possible NULL pointer dereference in bcache.c

r313575: makefs: make the buffer functions look exactly like the kernel ones

r313645: loader: implement MEDIA_FILEPATH_DP support in efipart

r313710: loader: cstyle fixes and DIOCGMEDIASIZE should use uint64_t

r314114: Use LDFLAGS with CC instead of _LDFLAGS.

r314213: Remove control+r handling from geliboot's pwgets()

r314275: Remove unused macro from common/drv.c.

r314945: Some style(9) fixes. No functional changes.

r314948: Try to extract the RFC1048 data from PXE.

r315008: r314948 seems to be missing a variable or two that will break

r315408: loader: remove open_disk cache

r315427: loader: biosdisk should report IO error from INT13

r315645: loader: disk_cleanup was left in userboot_disk.c

r315646: loader: pxe.h constants have wrong values

r315648: libstand: verify value provided by nfs.read_size

r315653: loader: verify the value from dhcp.interface-mtu and use snprintf
o set mtu

r315850: The original author abused Nd (one-line description, used by
makewhatis)

r316064: Fix build with path names with 'align' or 'nop' in them.

r316078: gpt*boot: Save a bit more memory when LOADER_NO_GELI_SUPPORT is
specified

r316079: Simply retire the sedification of the boot2.s file.

r316100: Remove -fno-guess-branch-probability and -fno-unit-at-a-time.

r316104: Use `NO_WCAST_ALIGN` instead of spelling it out as -Wno-cast-align
in CFLAGS

r316111: loader: move bios getsecs into time.c

r316112: loader: ls command should display file types properly

r316171: xfsread inlined uses more space, so remove the inline tag.

r316279: loader: efipart should check disk size from partition table

r316280: loader: simplify efi_zfs_probe and avoid double probing for zfs.

r316287: Remove OLD_NFSV2 from loader and libstand

r316311: Add explicit_bzero() to libstand, and switch GELIBoot to using it

r316343: Implement boot-time encryption key passing (keybuf)

r316424: Fix sparc64 build broken by r316343 and r316076

r316436: Restore EFI boot environment functionality broken in r313333

PR: 216940 217298 217935

MFC r313962, r313972-r313973, r315230

r313962 by allanjude:
improve PBKDF2 performance

The PBKDF2 in sys/geom/eli/pkcs5v2.c is around half the speed it could be

GELI's PBKDF2 uses a simple benchmark to determine a number of iterations
that will takes approximately 2 seconds. The security provided is actually
half what is expected, because an attacker could use the optimized
algorithm to brute force the key in half the expected time.

With this change, all newly generated GELI keys will be approximately 2x
as strong. Previously generated keys will talk half as long to calculate,
resulting in faster mounting of encrypted volumes. Users may choose to
rekey, to generate a new key with the larger default number of iterations
using the geli(8) setkey command.

Security of existing data is not compromised, as ~1 second per brute force
attempt is still a very high threshold.

PR: 202365
Original Research: https://jbp.io/2015/08/11/pbkdf2-performance-matters/
Submitted by: Joe Pixton <jpixton@gmail.com> (Original Version), jmg (Later Version)
Reviewed by: ed, pjd, delphij
Approved by: secteam, pjd (maintainer)
Differential Revision: https://reviews.freebsd.org/D8236

r313972 by ngie:
Unbreak the build when "make obj" is executed beforehand

Using relative paths imply working directory (in this case .OBJDIR), whereas the
sources live in the .CURDIR-relative path.

X-MFC with: r313962
Pointyhat to: allanjude
Sponsored by: Dell EMC Isilon

r313973 by ngie:
A forced commit to note other portion of the Makefile change accidentally
committed in r313972

The code committed in r313962 implicitly relies on python 2.x to generate
testvect.h . There are a handful of issues with this approach:

- python is not an explicit build dependency for FreeBSD
- python 2.x is deprecated and will be removed sometime in the future
(potentially before 11.x's EOL), and the script does not function with
python 3.5 (it uses deprecated idioms and incompatible function calls).
- python(1) (by default) lives in /usr/local/bin (${LOCALBASE}/bin) and
gentestvect.py is a dependency of testvect.h (prior to r313972) which
means that if the mtime of the generator script was newer than the
mtime of the test vector, it could cause a spurious build failure in
build time or at install time.

A better solution using C/C++ should be devised.

Discussed with: allanjude
X-MFC with: r313962, r313972
Sponsored by: Dell EMC Isilon

r315230 by ngie:
Move .../sys/geom/eli/pbkdf2... to .../sys/geom/class/eli/...

This change moves the tests added in r313962 to an existing directory
structure used by the geli TAP tests. It also, renames the test from
pbkdf2 to pbkdf2_test .

The changes to ObsoleteFiles.inc are being committed separately as they
aren't needed for the MFC to ^/stable/11, etc, if the MFC for the tests
is done all in one commit.

X-MFC with: r313962, r313972-r313973
Reviewed by: allanjude
Sponsored by: Dell EMC Isilon
Differential Revision: D9985

MFC r317247: Dump md_iterations as signed, which it really is.

PR: 208305
PR: 196834

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Create the GELIBOOT GEOM_ELI flag

This flag indicates that the user wishes to use the GELIBOOT feature to boot from a fully encrypted root file system.
Currently, GELIBOOT does not support key files, and in the future when it does, they will be loaded differently.
Due to the design of GELI, and the desire for secrecy, the GELI metadata does not know if key files are used or not, it just adds the key material (if any) to the HMAC before the optional passphrase, so there is no way to tell if a GELI partition requires key files or not.

Since the GELIBOOT code in boot2 and the loader does not support keys, they will now only attempt to attach if this flag is set. This will stop GELIBOOT from prompting for passwords to GELIs that it cannot decrypt, disrupting the boot process

PR: 208251
Reviewed by: ed, oshogbo, wblock
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D5867

Make additional parts of sys/geom/eli more usable in userspace

The upcoming GELI support in the loader reuses parts of this code
Some ifdefs are added, and some code is moved outside of existing ifdefs

The HMAC parts of GELI are broken out into their own file, to separate
them from the kernel crypto/openssl dependant parts that are replaced
in the boot code.

Passed the GELI regression suite (tools/regression/geom/eli)
Files=20 Tests=14996
Result: PASS

Reviewed by: pjd, delphij
MFC after: 1 week
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D4699

Replace sys/crypto/sha2/sha2.c with lib/libmd/sha512c.c

cperciva's libmd implementation is 5-30% faster

The same was done for SHA256 previously in r263218

cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation

Extend sbin/md5 to create sha384(1)

Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}

Reviewed by: cperciva, des, delphij
Approved by: secteam, bapt (mentor)
MFC after: 2 weeks
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3929

Enable BIO_DELETE passthru in GELI, so TRIM/UNMAP can work as expected when
GELI is used on a SSD or inside virtual machine, so that guest can tell
host that it is no longer using some of the storage.

Enabling BIO_DELETE passthru comes with a small security consequence - an
attacker can tell how much space is being really used on encrypted device and
has less data no analyse then. This is why the -T option can be given to the
init subcommand to turn off this behaviour and -t/T options for the configure
subcommand can be used to adjust this setting later.

PR: 198863
Submitted by: Matthew D. Fuller fullermd at over-yonder dot net

This commit also includes a fix from Fabian Keil freebsd-listen at
fabiankeil.de for 'configure' on onetime providers which is not strictly
related, but is entangled in the same code, so would cause conflicts if
separated out.

Use correct part of the Master-Key for generating encryption keys.
Before this change the IV-Key was used to generate encryption keys,
which was incorrect, but safe - for the XTS mode this key was unused
anyway and for CBC mode it was used differently to generate IV
vectors, so there is no risk that IV vector collides with encryption
key somehow.

Bump version number and keep compatibility for older versions.

MFC after: 2 weeks

Before this change when GELI detected hardware crypto acceleration it will
start only one worker thread. For software crypto it will start by default
N worker threads where N is the number of available CPUs.

This is not optimal if hardware crypto is AES-NI, which uses CPU for AES
calculations.

Change that to always start one worker thread for every available CPU.
Number of worker threads per GELI provider can be easly reduced with
kern.geom.eli.threads sysctl/tunable and even for software crypto it
should be reduced when using more providers.

While here, when number of threads exceeds number of CPUs avilable don't
reduce this number, assume the user knows what he is doing.

Reported by: Yuri Karaban <dev@dev97.com>
MFC after: 3 days

Add support for creating GELI devices with older metadata version for use
with older FreeBSD versions:
- Add -V option to 'geli init' to specify version number. If no -V is given
the most recent version is used.
- If -V is given don't allow to use features not supported by this version.
- Print version in 'geli list' output.
- Update manual page and add table describing which GELI version is
supported by which FreeBSD version, so one can use it when preparing GELI
device for older FreeBSD version.

Inspired by: Garrett Cooper <yanegomi@gmail.com>
MFC after: 3 days

When decoding metadata, check magic string, so we know this is not GELI device
before we check its version. We don't want to report that some garbage is
unsupported version if this is not even GELI provider.

MFC after: 3 days

When metadata is at newer version than the highest supported, return
EOPNOTSUPP when decoding.

MFC after: 3 days

Detect and handle metadata of version 6.

MFC after: 1 week

When support for multiple encryption keys was committed, GELI integrity mode
was not updated to pass CRD_F_KEY_EXPLICIT flag to opencrypto. This resulted in
always using first key.

We need to support providers created with this bug, so set special
G_ELI_FLAG_FIRST_KEY flag for GELI provider in integrity mode with version
smaller than 6 and pass the CRD_F_KEY_EXPLICIT flag to opencrypto only if
G_ELI_FLAG_FIRST_KEY doesn't exist.

Reported by: Anton Yuzhaninov <citrin@citrin.ru>
MFC after: 1 week

Remove prototype for a function that no longer exist.

MFC after: 1 week

Instead of allocating memory for all the keys at device attach,
create reasonably large cache for the keys that is filled when
needed. The previous version was problematic for very large providers
(hundreds of terabytes or serval petabytes). Every terabyte of data
needs around 256kB for keys. Make the default cache limit big enough
to fit all the keys needed for 4TB providers, which will eat at most
1MB of memory.

MFC after: 2 weeks

Free opencrypto sessions on suspend, as they also might keep encryption keys.

Bring in geli suspend/resume functionality (finally).

Before this change if you wanted to suspend your laptop and be sure that your
encryption keys are safe, you had to stop all processes that use file system
stored on encrypted device, unmount the file system and detach geli provider.

This isn't very handy. If you are a lucky user of a laptop where suspend/resume
actually works with FreeBSD (I'm not!) you most likely want to suspend your
laptop, because you don't want to start everything over again when you turn
your laptop back on.

And this is where geli suspend/resume steps in. When you execute:

# geli suspend -a

geli will wait for all in-flight I/O requests, suspend new I/O requests, remove
all geli sensitive data from the kernel memory (like encryption keys) and will
wait for either 'geli resume' or 'geli detach'.

Now with no keys in memory you can suspend your laptop without stopping any
processes or unmounting any file systems.

When you resume your laptop you have to resume geli devices using 'geli resume'
command. You need to provide your passphrase, etc. again so the keys can be
restored and suspended I/O requests released.

Of course you need to remember that 'geli suspend' won't clear file system
cache and other places where data from your geli-encrypted file system might be
present. But to get rid of those stopping processes and unmounting file system
won't help either - you have to turn your laptop off. Be warned.

Also note, that suspending geli device which contains file system with geli
utility (or anything used by 'geli resume') is not very good idea, as you won't
be able to resume it - when you execute geli(8), the kernel will try to read it
and this read I/O request will be suspended.

Change g_eli_debug to int, so one can turn off any GELI output by setting
kern.geom.eli.debug sysctl to -1.

MFC after: 2 weeks

Update copyright years.

MFC after: 1 week

Add support for AES-XTS. This will be the default now.

MFC after: 1 week

Implement switching of data encryption key every 2^20 blocks.
This ensures the same encryption key won't be used for more than
2^20 blocks (sectors). This will be the default now.

MFC after: 1 week

Define default overwrite count, so that userland can use it.

MFC after: 1 week

Support attaching version 4 metadata

Reviewed by: pjd

Correct offset conversion to little endian. It was implemented in version 2,
but because of a bug it was a no-op, so we were still using offsets in native
byte order for the host. Do it properly this time, bump version to 4 and set
the G_ELI_FLAG_NATIVE_BYTE_ORDER flag when version is under 4.

MFC after: 2 weeks

Add support for Camellia encryption algorithm.

PR: kern/113790
Submitted by: Yoshisato YANAGISAWA <yanagisawa@csg.is.titech.ac.jp>
Approved by: re (bmah)

Add 'configure' subcommand which for now only allows setting and removing
of the BOOT flag. It can be performed on both attached and detached
providers.

Requested by: Matthias Lederhofer <matled@gmx.net>
MFC after: 1 week

Before using byte offset for IV creation, covert it to little endian.
This way one will be able to use provider encrypted on eg. i386 on
eg. sparc64. This doesn't really buy us much today, because UFS isn't
endian agnostic.

We retain backward compatibility by setting G_ELI_FLAG_NATIVE_BYTE_ORDER
flag on devices with version number less than 2 and not converting the
offset.

Forgot to bump version number after G_ELI_FLAG_READONLY flag addition.

Allow geli to operate on read-only providers.

Initial patch from: vd
MFC after: 2 weeks

Implement data integrity verification (data authentication) for geli(8).

Supported by: Wheel Sp. z o.o. (http://www.wheel.pl)

Revert previous change, as I fixed MD5(9).

md_hash field in g_eli_metadata structure is not 4 byte aligned, which
case panic on sparc64.

The problem is in MD5(9) implementation. The Encode() function takes
'unsigned char *output' as its first argument, which is then assigned to
'u_int32_t *op'. If the 'output' argument is not 4 byte aligned (and in
geli(8) case it is not), sparc64 machine will panic.

I don't know how to fix MD5(9) in a clean way, so I'm implementing a
work-around in geli(8).

Reported by: brueffer
MFC after: 3 days

Remove trailing spaces.

Add a __packed keyword to g_eli_metadata struct definition, so
sizeof(struct g_eli_metadata) will return the exact number of bytes needed
for storing it on the disk.
Without this change GELI was unusable on amd64 (and probably other 64-bit
archs), because sizeof(struct g_eli_metadata) was greater than 512 bytes
and geli(8) was failing on assertion.

Reported by: Michael Reifenberger <mike@Reifenberger.com>
MFC after: 3 days

- Add a missing period.
- Fix number of spaces.

MFC after: 3 days

Be case-insensitive when dealing with algorithm names.

PR: kern/84659
Submitted by: Benjamin Lutz <benlutz@datacomm.ch>

Add GEOM_ELI class which provides GEOM providers encryption.
For features list and usage see manual page: geli(8).

Sponsored by: Wheel Sp. z o.o.
http://www.wheel.pl
MFC after: 1 week