Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge the Cavium Octeon SDK 2.3.0 Simple Executive code and update FreeBSD to
make use of it where possible.

This primarily brings in support for newer hardware, and FreeBSD is not yet
able to support the abundance of IRQs on new hardware and many features in the
Ethernet driver.

Because of the changes to IRQs in the Simple Executive, we have to maintain our
own list of Octeon IRQs now, which probably can be pared-down and be specific
to the CIU interrupt unit soon, and when other interrupt mechanisms are added
they can maintain their own definitions.

Remove unmasking of interrupts from within the UART device now that the
function used is no longer present in the Simple Executive. The unmasking
seems to have been gratuitous as this is more properly handled by the buses
above the UART device, and seems to work on that basis.

Remove octeon_bootinfo from all files except octeon_machdep.c.
Use instead cvmx_sysinfo_get() consistently.

Reviewed by: jmallett, marcel
Obtained from: Juniper Networks, Inc
Author: Andrew Duane

Partially revert r224661:
octeon_ap_boot is not a bitmask. It holds the CPU Id of the
AP currently being started. As such there's no need to make
it a 64-bit integral and we're not limited to 64 CPUs.

Reported by: jmallet
Obtained from: Andre Duane

Make octeon_ap_boot 64 bits to handle MAXCPU up to 64.
Better construction of CPU mask in platform_cpu_mask().
Release cores still in reset when platform_start_ap() is called.

Reviewed by: imp, marcel
Obtained from: Juniper Networks, Inc
Author: Andrew Duane
Approved by: re (kib)

etire the cpumask_t type and replace it with cpuset_t usage.

This is intended to fix the bug where cpu mask objects are
capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever
value. Anyway, as long as several structures in the kernel are
statically allocated and sized as MAXCPU, it is suggested to keep it
as low as possible for the time being.

Technical notes on this commit itself:
- More functions to handle with cpuset_t objects are introduced.
The most notable are cpusetobj_ffs() (which calculates a ffs(3)
for a cpuset_t object), cpusetobj_strprint() (which prepares a string
representing a cpuset_t object) and cpusetobj_strscan() (which
creates a valid cpuset_t starting from a string representation).
- pc_cpumask and pc_other_cpus are target to be removed soon.
With the moving from cpumask_t to cpuset_t they are now inefficient
and not really useful. Anyway, for the time being, please note that
access to pcpu datas is protected by sched_pin() in order to avoid
migrating the CPU while reading more than one (possible) word
- Please note that size of cpuset_t objects may differ between kernel
and userland. While this is not directly related to the patch itself,
it is good to understand that concept and possibly use the patch
as a reference on how to deal with cpuset_t objects in userland, when
accessing kernland members.
- KTR_CPUMASK is changed and now is represented through a string, to be
set as the example reported in NOTES.

Please additively note that no MAXCPU is bumped in this patch, but
private testing has been done until to MAXCPU=128 on a real 8x8x2(htt)
machine (amd64).

Please note that the FreeBSD version is not yet bumped because of
the upcoming pcpu changes. However, note that this patch is not
targeted for MFC.

People to thank for the time spent on this patch:
- sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested
several revision of the patches and really helped in improving
stability of this work.
- marius fixed several bugs in the sparc64 implementation and reviewed
patches related to ktr.
- jeff and jhb discussed the basic approach followed.
- kib and marcel made targeted review on some specific part of the
patch.
- marius, art, nwhitehorn and andreast reviewed MD specific part of
the patch.
- marius, andreast, gonzo, nwhitehorn and jceel tested MD specific
implementations of the patch.
- Other people have made contributions on other patches that have been
already committed and have been listed separately.

Companies that should be mentioned for having participated at several
degrees:
- Yahoo! for having offered the machines used for testing on big
count of CPUs.
- The FreeBSD Foundation for having sponsored my devsummit attendance,
which has been instrumental.
- Sandvine for having offered offices and infrastructure during
development.

(I really hope I didn't forget anyone, if it happened I apologize in
advance).

Allow the platform code to return a bitmask of running cores rather than just
a number of cores, this allows for a sparse set of CPUs. Implement support
for sparse core masks on Octeon.

XXX jeff@ suggests that all_cpus should include cores that are offline or
running other applications/OSes, so the platform API should be further
extended to allow us to set all_cpus to include all cores that are
physically-present as opposed to only those that are running FreeBSD.

Submitted by: Bhanu Prakash (with modifications)
Reviewed by: jchandra
Glanced at by: kib, jeff, jhb

o) Unmask Central Interrupt Unit interrupts on APs, too.
o) There's no good reason to make the low bits of the ebase the core
number. While per-CPU exception bases would be nice, for now we just
need to make ebase the same on all cores.

Make the meaning of the 'mask' argument to 'set_intr_mask(mask)' consistent
with the meaning of IM bits in the status register.

Reviewed by: jmallett, jchandra

Update the port of FreeBSD to Cavium Octeon to use the Cavium Simple Executive
library:
o) Increase inline unit / large function growth limits for MIPS to accommodate
the needs of the Simple Executive, which uses a shocking amount of inlining.
o) Remove TARGET_OCTEON and use CPU_CNMIPS to do things required by cnMIPS and
the Octeon SoC.
o) Add OCTEON_VENDOR_LANNER to use Lanner's allocation of vendor-specific
board numbers, specifically to support the MR320.
o) Add OCTEON_BOARD_CAPK_0100ND to hard-wire configuration for the CAPK-0100nd,
which improperly uses an evaluation board's board number and breaks board
detection at runtime. This board is sold by Portwell as the CAM-0100.
o) Add support for the RTC available on some Octeon boards.
o) Add support for the Octeon PCI bus. Note that rman_[sg]et_virtual for IO
ports can not work unless building for n64.
o) Clean up the CompactFlash driver to use Simple Executive macros and
structures where possible (it would be advisable to use the Simple Executive
API to set the PIO mode, too, but that is not done presently.) Also use
structures from FreeBSD's ATA layer rather than structures copied from
Linux.
o) Print available Octeon SoC features on boot.
o) Add support for the Octeon timecounter.
o) Use the Simple Executive's routines rather than local copies for doing reads
and writes to 64-bit addresses and use its macros for various device
addresses rather than using local copies.
o) Rename octeon_board_real to octeon_is_simulation to reduce differences with
Cavium-provided code originally written for Linux. Also make it use the
same simplified test that the Simple Executive and Linux both use rather
than our complex one.
o) Add support for the Octeon CIU, which is the main interrupt unit, as a bus
to use normal interrupt allocation and setup routines.
o) Use the Simple Executive's bootmem facility to allocate physical memory for
the kernel, rather than assuming we know which addresses we can steal.
NB: This may reduce the amount of RAM the kernel reports you as having if
you are leaving large temporary allocations made by U-Boot allocated
when starting FreeBSD.
o) Add a port of the Cavium-provided Ethernet driver for Linux. This changes
Ethernet interface naming from rgmxN to octeN. The new driver has vast
improvements over the old one, both in performance and functionality, but
does still have some features which have not been ported entirely and there
may be unimplemented code that can be hit in everyday use. I will make
every effort to correct those as they are reported.
o) Support loading the kernel on non-contiguous cores.
o) Add very conservative support for harvesting randomness from the Octeon
random number device.
o) Turn SMP on by default.
o) Clean up the style of the Octeon kernel configurations a little and make
them compile with -march=octeon.
o) Add support for the Lanner MR320 and the CAPK-0100nd to the Simple
Executive.
o) Modify the Simple Executive to build on FreeBSD and to build without
executive-config.h or cvmx-config.h. In the future we may want to
revert part of these changes and supply executive-config.h and
cvmx-config.h and access to the options contained in those files via
kernel configuration files.
o) Modify the Simple Executive USB routines to support getting and setting
of the USB PID.

o) Fix XKPHYS physical address extraction. Also define cache coherency
attributes for XKPHYS.
o) Make coprocessor 0 accessor function macros for register+selector registers
take the full name so that e.g. (as done in this commit), prid selector 1
can be written through mips_wr_ebase() rather than mips_wr_prid1().
o) Allow for sign extension of 32-bit segment addresses.
o) Remove an unused MIPS-I register number.

o) Add SMP support for Octeon using U-Boot to launch all the processors at the
same time.
o) Remove some unused trivial uart functions from octeon_machdep now that the
uart part is fully working and they are unused.
o) Use __func__ instead of __FUNCTION__.
o) Use intr_*() instead of other routines that do the same thing.
o) Remove some duplicate printfs from the Octeon port, as well as duplicate
setting of Maxmem.
o) Use the right frequency divider on Octeon.
o) Use PCPU_GET(cpuid) consistently to get the cpuid of the running core.
o) Remove some unused macros in the Octeon port.
o) Use mips_sync() around use of the global dpcpu, whose value may not be
visible to APs at first.
o) When loading the first thread's stack, use macros to make the code correct
for n64 as well.
o) Remove stub, do-nothing FAU init/enable/disable functions from the RGMX
driver.