xref: /barrelfish-master/doc/022-armv8/report.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2013, ETH Zurich.
% All rights reserved.
%
% This file is distributed under the terms in the attached LICENSE file.
% If you do not find this file, copies can be found by writing to:
% ETH Zurich D-INFK, Universitaetstrasse 6, CH-8092 Zurich. Attn: Systems Group.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[a4paper,twoside]{report}

\usepackage{bftn}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{hyphenat}
\usepackage{listings}
\usepackage{makeidx}
\usepackage{natbib}
\usepackage{xspace}

\def\chapterautorefname{Chapter}
\def\sectionautorefname{Section}
\def\subsectionautorefname{Section}
\def\subsubsectionautorefname{Section}
\def\tableautorefname{Table}
\def\qemu{QEMU\xspace}

\lstdefinelanguage{armasm}{
    numbers=left,
    numberstyle=\tiny,
    numbersep=5pt,
    basicstyle=\ttfamily\small
}
\lstset{language=armasm}

\title{Barrelfish on ARMv8}
\author{David Cock}

\tnnumber{022}  % give the number of the tech report
\tnkey{ARMv8} % Short title, will appear in footer

\begin{document}
\maketitle

\begin{versionhistory}
\vhEntry{1.0}{11.04.2016}{DC}{Initial version}
\end{versionhistory}

\chapter{Summary}

Barrelfish now supports ARMv7 and ARMv8 as primary platforms, and we have
discontinued support for all older architecture revisions (ARMv5, ARMv6). The
current Barrelfish release contains a port to a simulated ARMv8 environment,
derived from the existing ARMv7 codebase and running under GEM5, with generous
contributions from HP Research.

Simultaneously, we are undertaking a clean-slate redesign of the CPU driver
for ARMv8, as it presents a number of novel features, and greatly improved
platform standardisation (\autoref{s:sbsa}), that should allow for a much
cleaner and simpler implementation. This redesigned CPU driver will for the
basis for ongoing research into large-scale non-cache-coherent systems using
ARMv8 cores. This document presents the new CPU driver design
(\autoref{c:design}), briefly covering those features of ARMv8 of greatest
relevance (\autoref{c:background}), and discusses a number of technical
challenges presented by the new architecture (\autoref{c:tech}).

\chapter{Background}\label{c:background}

The Barrelfish research operating system is a vehicle for research into
software support for likely future architectures, where large numbers of
non-coherent (or weakly-coherent) heterogeneous processor cores are assembled
into a single large-scale system. As such, support for a common non-x86
architecture has always been part of the project, beginning with the ARMv5
(XScale) port, which permitted the embedded processor on a network interface
card to be integrated as a first-class part of the system, with its own CPU
driver. We have also actively maintained an ARMv7 port, to the OMAP4460
processor on the Pandaboard ES, which we use as a teaching platform in the
Advanced Operating Systems course at ETH Z\"urich. These ports are described
more fully in the accompanying technical report, \citet{btn017-arm}.

\section{The ARMv8 Architecture}

The ARMv8 architecture is quite a radical departure from previous versions,
and represents the culmination of a trend that has been developing for quite
some time. While the first wave of ARM-based microservers, based on the 32-bit
ARMv7 architecture, was largely a commercial failure, it's clear that ARM is
now actively targeting the server market, where Intel currently has near-total
dominance.

ARMv8 discards some long-standing features of the ARM instruction set:
universal conditional execution, multiple loads/stores, and the program
counter as a general-purpose register. These most likely caused difficulty in
scaling the processor pipeline to high clock rates, and we present some
consequences of their loss in \autoref{s:threads} and \autoref{s:traps}. The
instruction-set changes, however challenging to the systems programmer, are
ultimately of little consequence compared to the consolidation of the ARM
ecosystem into a serious server platform. The two features of most interest at
this stage in the design process are the standardisation of hardware features
and memory maps, and of the boot process.

\subsection{ARM Server Base System Architecture}\label{s:sbsa}

ARM has long been criticised by systems programmers for its highly fragmented,
and non-uniform programming interface. Linux, in particular, has struggled for
years with supporting the great multiplicity of ARM platforms.  The principal
reason for this is the lack of any concept of a \emph{platform}: a set of
assumptions (available hardware, memory map, etc.), that programmers can rely
on when initialising and managing a system. The Linux source tree famously
contained a vastly greater amount of code in the ARM platform support
subtrees, than that for x86.

The relative standardisation of the x86 platform is largely a historical
accident, due to the rapid proliferation of PC/AT clones in the early 1980s.
The x86 platform thus contains layers of ossified legacy interfaces, necessary
to ensure broad cross and backward compatibility.  ARM's business model, on
the contrary, has long emphasised the specialisation of implementations: an
ARM licensee would take their ARM-designed CPU core, and integrate it
themselves in to a complex SoC (system on a chip), with their own specialised,
proprietary interfaces. The upsides of this were the possibility to highly
optimise a particular design, and no requirement on ARM itself to maintain a
coherent platform.

While ARM's customisable platform worked well for embedded devices, and scaled
reasonably well to relatively powerful smartphones, it's a disaster for
producing high-quality systems software, able to execute on a broad range of
hardware from competing vendors: exactly what a competitive server platform
requires. ARM clearly know this, and since 2014 have published the Server Base
System Architecture \citep{arm:sbsa}. To the extent that manufacturers adhere
to these guidelines, our job as systems programmers is significantly simpler:
it should be possible to write a single set of initialisation and
configuration code for ARMv8, that will run on any SBSA-compliant system, much
as we already do for x86-64.

\begin{table}
\begin{center}
\begin{tabular}{lllp{6cm}}
\toprule
Supplier & Processor & Name & \\
\midrule
APM & APM883208 & Mustang  & 1P 8-core X-Gene 1 with serial trace. \\
\addlinespace[2pt]
    & APM883408-X2 & X-C2  & 1P 8-core X-Gene 2. \\
\addlinespace[2pt]
Cavium & CN8890 & StratusX & 1P 48-core ThunderX. \\
\addlinespace[2pt]
       &        & Cirrus   & 2P 48-core ThunderX. \\
\addlinespace[2pt]
ARM & AEM & Fixed Virtual Platform & The \emph{architectural envelope model}
covers the range of behaviour permitted by ARMv8. Bare-metal debug. \\
\addlinespace[2pt]
    &     & Foundation Platform & Freely available, compatible with FVP. \\
\bottomrule
\end{tabular}
\end{center}
\caption{ARMv8 platforms of interest}\label{t:platforms}
\end{table}

Our target platforms listed in \autoref{t:platforms} all support SBSA to some
extent, and absent any compelling reason, we will only support SBSA-compliant
platforms.

\subsection{UEFI}\label{s:uefi}

One aspect of the SBSA which eases portability is the specification, for the
first time, of a boot process for ARM systems. ARM has specified that SBSA
systems must support UEFI \citep{uefi} (the unified extensible firmware
interface). UEFI is a descendant of the EFI specification, developed by Intel
for the Itanium project. While Itanium is no longer a platform of any great
commercial interest, UEFI support is now widespread in the x86-64 market.
UEFI, in turn, specifies the use of ACPI \citep{acpi} (the advanced
configuration and power interface) for platform discovery and control.

Supporting ACPI and UEFI requires a one-off investment of effort to design a
new boot and configuration subsystem, but should pay off in the long term, as
ports to new ARM boards will no longer require extensive manual configuration.
The code should also be largely reusable for x86 UEFI systems. Our new UEFI
bootloader is described in \autoref{s:hagfish}.

\section{A Direct Port from ARMv7}

As already described, the Barrelfish release current at time of writing
includes an initial ARMv8 port to the GEM5 simulator. This port contains code
generously contributed by HP Research.

Being developed from the existing codebase, this ARMv8 port follows the
structure of the existing ARMv7 code closely. While it is highly useful to
have a running port, we are nevertheless continuing with a significant
redesign of the CPU driver, as significant improvements and simplifications
will be possible, once we no longer need to follow the existing structure,
originally developed for a significantly different platform.

The GEM5 simulator's model of an ARMv8 platform is relatively primitive, and
does not conform to modern platform conventions, for example placing RAM at
address \texttt{0}, rather than \texttt{0x80000000} as mandated by the SBSA.
For this reason, in addition to better integration with ARM debugging tools,
we have switched to the ARM Fixed Virtual Platform as our default simulation
environment, with the Foundation Platform supported as a freely available
simulator.

\section{Registers}

\subsection{General-purpose Registers}

In total there are 31+1 general purpose registers (\texttt{r0-r30}) of size 
64bits(\autoref{tab:registers}). They are usually referred to by the names 
\texttt{x0-x30}. The 32-bit content of the registers are referred to as 
\texttt{w0-w30}. The additional stack pointer \texttt{SP} register can be 
accessed with a restricted number of instructions.

\begin{table}[!h]
    \begin{center}

    \begin{tabular}{lll}
        \textbf{Register} & \textbf{Special} & \textbf{Description} \\
        \hline
        \texttt{X0-X7}   & Caller-save & function call arguments and return value 
        \\
        \texttt{X8}      & & indirect result e.g. location of large return value 
                             (struct) \\
        \texttt{X9-X15}  & Caller-save& temporary registers  \\
        \texttt{X16}     & IP0 & The first intra-procedure-call scratch 
        register\footnote{can be used by call veneers and PLT code; at other 
        times may be used as a temporary register. Same for X17}\\
        \texttt{X17}     & IP1 & The second intra-procedure-call temporary 
        register  \\
        \texttt{X18}     & & The Platform Register (TLS), if needed; otherwise a 
        temporary register. \\
        \texttt{X19-X28} & Callee-save & need to be preseved and restored when 
                           modified\\
        \texttt{X29}     & FP & frame pointer \\
        \texttt{X30}     & LR & link register \\
        \texttt{SP}      & & stack pointer (XZR) \\
        
    \end{tabular}
    \caption{ARMv8 General purpose Registers}
    \label{tab:registers}
        \end{center}
\end{table}

\paragraph{Procedure call}
\begin{itemize}
    \item The registers \texttt{x19-28} and \texttt{SP} are callee-saved and 
          hence must be preserved by the called subroutine. All 64 bits have to 
          be preserved even when executing in the 32-bit mode.
    \item The registers \texttt{x0-x7} and \texttt{x9-x15} are caller saved.
    \item During procedure calls the registers \texttt{x16}, \texttt{x17}, 
          \texttt{x29} and \texttt{x30} have special roles i.e.\ they store 
          relevant addresses such as the return address.
    \item Arguments for calls are passed in the registers \texttt{x0-x7}, 	    
          \texttt{v0-v7} for floats/SIMD and on the stack
\end{itemize}

\paragraph{Indirect result} This register is used when returning a large value 
such as declared by this function: \texttt{struct mystruct foo(int arg);}.


\paragraph{Platform specific} The use of register \texttt{x18} is platform 
specific and needs to be defined by the platform ABI. This register can hold
inter-procedural state such as the thread context. 

\paragraph{Linker} The registers \texttt{IP0} and \texttt{IP1} can be used by the 
linker as a scratch register or to hold intermediate values between subroutine 
calls.

\subsection{SIMD and Floating point}
There are 32 registers to be used by floating point and SIMD operations. The name 
of those registers will change, depending on the size of the operation.

\begin{table}[!h]
    \begin{center}
        
        \begin{tabular}{ll}
            \textbf{Register} & \textbf{Description} \\
            \hline
            \texttt{v0-v7}   & function call arguments, intermediate values and 
                               return value, caller save registers \\
            \texttt{v8-v15}  & Callee-save registers. They need to be 
            preserved\\
            \texttt{v16-v31} & Caller-save registers
        \end{tabular}
        \caption{ARMv8 General purpose Registers}
        \label{tab:registers}
    \end{center}
\end{table}

\chapter{Design and Implementation}\label{c:design}




\section{Redesigning the CPU Driver}

Given that ARMv8 is a significantly different platform to ARMv7, and that the
ARMv7 codebase carries a significant legacy, reaching right back to ARMv5, we
are pursuing substantial redesign of the CPU driver. Taking advantage of the
standardisation of the hardware platform mandated by the SBSA
(\autoref{s:sbsa}), and the facilities provided by UEFI (\autoref{s:uefi}), in
addition to a relatively unrestricted virtual address space, we are able to
significantly reduce the complexity of the CPU driver.  In this section we
describe the updated design, and our progress on its implementation, while the
UEFI interface (Hagfish) is described separately, in \autoref{s:hagfish}.

\paragraph{Terminology}
In the interest of clarity, in the discussion that follows, we use a few terms
with precise intent:
\begin{description}
\item[shall]
    indicates features or characteristics of the design to which the
    Barrelfish implementation must conform.
\item[should]
    indicates features which should be supported if at all possible.
\item[initially]
    indicates features which will be provided from the outset in the
    Barrelfish implementation.
\item[eventually]
    indicates features which will be provided later in the Barrelfish
    implementation, and which the initial design will aim to facilitate.
\end{description}

\subsection{Goals}

Our goal is to provide a reference design for the CPU driver and user-space
execution environment for Barrelfish on an ARMv8 core, in order to understand
both positive and negative implications of the architecture for a multikernel
system.  The design \textbf{should} be applicable to any ARMv8 with
virtualisation (\texttt{EL2}) support.

\textbf{Initially}, our hardware development platform is the APM X-Gene 1,
using the Mustang Development Board. We are using the Mustang principally as
it was relatively easily available, as well as being a comparatively complex
and powerful CPU. The ThunderX platform from Cavium is very interesting for
Barrelfish, as it ties a large number (48) of less-powerful (2-issue) cores.
We do not have the resources to develop for two platforms simultaneously, but
we hope to \textbf{eventually} add support for the ThunderX.

Our target simulation environment is the ARM Fixed Virtual Platform, and the
Foundation Platform. These models are supplied by ARM. The Foundation Platform
is freely available, and will be the default supported simulation platform for
the public Barrelfish tree, while we will use the FVP internally to allow
bare-metal debugging. Future support for \qemu is desirable, to the extent that
it models a compatible system --- GEM5, which the ARMv7 port targets,
currently does not.

\textbf{Initially}, the design will support running both the CPU driver and
user-space processes in AArch64 mode without support for virtualisation.
\textbf{Eventually} the design will support running the CPU driver in AArch64
mode, and user-space processes in both AArch64 and AArch32 modes without
virtualisation, and virtual machines in AArch64 mode. We will only support
virtualisation on ARMv8.1 or later platforms, that support the VHE extensions,
as described in \autoref{s:layout}.

\subsection{Processor Modes and Virtualisation}

Where possible, we will keep the virtualisation model similar to that on
Barrelfish/x86. In particular, it \textbf{should} be possible to implement
native applications, fully virtualised (e.g. Linux) VMs, and VM-level
applications e.g. Arrakis \citep{peter:osdi14}.

ARMv8 has a somewhat different virtualisation model to x86, and different
again from the ARMv7 virtualisation extensions. Rather than having exception
levels (rings) duplicated between guest and host, ARMv8 provides 4 exception
levels (ELs):

\begin{itemize}
\item \texttt{EL0} is unprivileged --- user applications.
\item \texttt{EL1} is privileged --- OS kernel.
\item \texttt{EL2} is hypervisor state.
\item \texttt{EL3} is for switching between secure and non-secure (TrustZone)
                   modes. The X-Gene 1 does not implement \texttt{EL3}, and it
                   is currently not of interest for Barrelfish.
\end{itemize}

Explicit traps (syscalls/hypercalls) target only the next level up:
\texttt{EL0} can call \texttt{EL1} using \texttt{svc} (syscall), and
\texttt{EL1} can call \texttt{EL2} using \texttt{hvc} (hypercall), but
\texttt{EL0} cannot directly call \texttt{EL2}, unless \texttt{EL1} is
completely disabled.  Exceptions return to the caller's exception level.

ELs \textbf{shall} be distributed as follows: The CPU driver \textbf{shall}
exist at both \texttt{EL1} and \texttt{EL2}, and take both syscalls
(\texttt{svc}, from \texttt{EL0} applications) and hypercalls (\texttt{hvc},
from \texttt{EL1} applications). The system \textbf{shall} support
applications both at \texttt{EL0}, and at \texttt{EL1} (e.g.  Arrakis, VMs).
Most code paths \textbf{should} be identical, as most CPU driver operations do
not depend on \texttt{EL2} privileges.  Hypercalls from \texttt{EL0}
\textbf{shall} be chained via \texttt{EL1} (with appropriate permission
checks).

\texttt{EL1} apps such as Arrakis, and paravirtualised VMs using hypercalls
know that they are being virtualised, and will use \texttt{hvc} explicitly.
Fully-virtualised \texttt{EL1} VMs do not make hypercalls.

ARMv8 implements two-level address translation: VA (virtual address) to IPA
(intermediate physical address), and IPA to PA (physical address).
\texttt{EL1} guests \textbf{shall} be isolated at the L1 translation layer,
and by trapping all accesses to system control registers.

\subsection{Virtual Address Space Layout}\label{s:layout}

ARMv8 has an effective 48-bit virtual address space. At the lowest execution
levels (0 --- BF user \& 1 --- BF CPU driver), the hardware supports two (up to)
48-bit (256TB) 'windows' in a 64-bit space: one at the bottom, and one at the
top.  Each region has its own translation table base register (\texttt{TTBR0}
\& \texttt{TTBR1}). \texttt{TTBR0} is used at \texttt{EL0}, and \texttt{TTBR1}
at \texttt{EL1}.

In the initial ARMv8 specification, this split address space was not
implemented at \texttt{EL2}, which would require a separate CPU driver
instance for virtualisation, and hypercalls (e.g. for Arrakis). ARMv8.1
introduced the virtualisation host extensions (VHE) which, among other things,
extends the split address space to \texttt{EL2}. As this provides a cleaner
implementation model, and to avoid having to support a now-deprecated
interface, virtualisation will \textbf{only} be supported on ARMv8.1 and
later. This means that we will not support virtualisation on the X-Gene 1.
Both the simulation environment (FVP/FP) and, seemingly, the ThunderX chips,
support VHE.

The CPU driver \textbf{shall} use \texttt{TTBR1} to provide a complete
physical window. The ARMv8 CPU driver \textbf{shall not} dynamically map
device memory into its own window (as the ARMv7 CPU driver does) --- the few
memory-mapped devices required will be statically mapped on boot, with
appropriate memory attributes. All physical addresses, RAM and device,
\textbf{shall} be accessible at a static, standard offset (the base of the
\texttt{TTBR1} region).

User-level page tables will \textbf{initially} be limited to a 4k translation
granularity.  \textbf{Eventually} user-level page tables \textbf{should} have
access to all page-table formats and page sizes, as is the case in the current
Barrelfish x86 implementation.

\subsection{Address Space, Context, and Thread Identifiers}

ARMv8 also provides address-space identifiers (ASIDs) in the TLB to avoid
flushing the translation cache on a context switch.

ARMv8 ASIDs (referred to in ARM documentation as context IDs) are
architecturally allowed to be either 8 or 16 bits, although the SBSA
specifies that they must be at least 16. Relying on the SBSA platform will
allow us to avoid multiplexing IDs among active processes, on any
reasonably-sized system.  Managing the reuse of context IDs can be left to
user-level code, and does not need to be on the critical path of a context
switch. The CPU driver need only ensure that every allocated dispatcher has a
unique ASID, which is loaded into the \texttt{ContextID} register on dispatch.

The value in the \texttt{ContextID} register is also checked against the
hardware breakpoint and watchpoint registers, in generating debug exceptions.
Therefore, it \texttt{shall} be possible for authorised user-level code to
load the Context ID for a given dispatcher into a breakpoint register --- this
\texttt{may} be an invocation on the dispatcher capability.

\begin{table}
\begin{center}
\begin{tabular}{ll}
\texttt{tpidrro\_el0} & EL0 Read-Only Software Thread ID Register \\
\texttt{tpidr\_el0} & EL0 Read/Write Software Thread ID Register \\
\texttt{tpidr\_el1} & EL1 Read/Write Software Thread ID Register \\
\texttt{tpidr\_el2} & EL2 Read/Write Software Thread ID Register \\
\texttt{tpidr\_el3} & EL3 Read/Write Software Thread ID Register \\
\end{tabular}
\end{center}
\caption{Thread ID registers in ARMv8}
\label{t:threadid}
\end{table}

In addition to the \texttt{ContextID} register, used to tag TLB entries, ARMv8
also provides a set of thread ID registers with no architecturally-defined
semantics, as listed in \autoref{t:threadid}. The client-writeable
\texttt{tpidr\_el0} and \texttt{tpidr\_el1} \textbf{shall} have no CPU
driver-defined purpose, but \textbf{shall} be saved and restored in a
dispatcher's trap frame, to allow their use as thread-local storage (TLS).
Recall that the Barrelfish CPU driver has no awareness of threads, which are
implemented purely at user level.

To implement the upcall/dispatch mechanism of Barrelfish, the CPU driver and
the user-level dispatcher need to share a certain amount of state --- the
user-visible portion of the dispatcher control block, which contains the trap
frames, and the disabled flag (used to achieve atomic dispatch). The address
of this structure needs to be known to both the CPU driver, and to user-level
code, and moreover be efficiently-accessible, as the CPU driver needs to find
the trap frame on the critical path of system calls and exceptions. This
pointer also needs to be trustworthy, from the CPU driver's perspective, and
thus cannot be directly modifiable by user-level code.

The x86-32, x86-64, and ARMv7 CPU drivers all store the address of the running
dispatcher's shared segment at a fixed known address, \texttt{dcb\_current},
which is loaded by the trap handler. At user level, on x86 this address is
held in a \emph{segment register} (\texttt{fs} on x86-64, and \texttt{gs} on
x86-32), while on ARMv7 we sacrifice a general-purpose register (\texttt{r9})
for this purpose. Using the \texttt{tpidrro\_el0} register to hold the address
of the current dispatcher structure will allow us to avoid both a memory load
on the fast path, and sacrificing a register in user-level code, thus
\texttt{tpidrro\_el0} \textbf{shall} hold the address of the currently-running
dispatcher.

\subsection{Instruction Sets}

ARMv8 supports both AArch64, and legacy ARM/Thumb (renamed AArch32). Switching
execution mode is only possible when switching execution level i.e. on a trap
or return, and can only be changed while at the higher execution level. Thus,
\texttt{EL2} can set execution mode for \texttt{EL1}, and \texttt{EL1} for
\texttt{EL0}. There is no way for a program to change its own execution mode.
If \texttt{ELn} is in AArch64, then \texttt{EL(n-1)} can be in either AArch64
or AArch32. If \texttt{ELn} is in AArch32, all lower ELs must also be AArch32.

The CPU driver \textbf{shall} execute in AArch64.

\textbf{Initially}, the CPU driver will enforce that all directly-scheduled
threads also use AArch64, by controlling all downward EL transitions. An
\texttt{EL1} client (such as Arrakis or a full virtual machine) may execute
its own \texttt{EL0} clients in AArch32 (and there is no way to prevent this).
However, all transitions into the CPU driver (\texttt{svc}, \texttt{hvc} or
exception) must come from a direct client of the CPU driver, and thus from
AArch64. The syscall ABI \textbf{shall} be AArch64.

\textbf{Eventually}, Barrelfish \textbf{should} also support the execution of
AArch32 dispatcher processes, by marking each dispatcher with a flag
indicating the instruction set to be used (much as is already done with
VM/non-VM mode in the Arrakis CPU driver).

\subsection{User-Space Access to Architectural Functions}

Generally, anything that can be safely exported, \textbf{should} be made
available outside of the CPU driver, preferable as a memory-mapped interface,
at 4kiB granularity. The SBSA mandates that devices be present at addresses
that can be individually mapped, thus this should not be a problem.

\subsection{Cache Management}

ARMv8 has moved most cache and TLB management from the system control
coprocessor (cp15), into the core ISA. Several cache operations
(invalidate/clean by VA) are executable at \texttt{EL0}, and thus no kernel
interface is required. The system must take into account that user-directed
flushes may have occurred, or may occur concurrently with any memory
operation.

\subsection{Performance Monitors}

Performance monitors \textbf{should} be exposed, if it can be done safely.

\subsection{Debugging}

Self-hosted debug \textbf{should} be exposed, if it can be done safely. This
is under active development.

\subsection{Booting}

Platform support i.e.~a standard set of peripherals, and a defined boot
process, has improved dramatically on ARM, as it has been repositioned as a
server platform. UEFI and ACPI support are widespread, including on the
Mustang development board. We will assume support for UEFI booting, make use
of ACPI data, where available.

The Barrelfish CPU driver and initial image \textbf{shall} be loaded and
executed by a UEFI shim, which will pass through all UEFI-supplied
information, such as ACPI tables, and be able to interpret a Barrelfish
Multiboot image.  This shim, or second-stage bootloader, is called Hagfish,
and is described in \autoref{s:hagfish}.

\subsection{Interrupts}

ARMv8 interrupt handling is not substantially different from the existing
architectures and platforms supported by Barrelfish. While a redesign of the
Barrelfish interrupt system is under way (to use capabilities to grant access
to receive interrupts), we do not anticipate ARMv8 to impose any particular
challenges.

The ARMv8 systems we \textbf{initially} target all use minor variations on the
ARM Generic Interrupt Controller (GIC) design, already supported in
Barrelfish. We currently have support for version 2 of the GIC, with which
later implementations are backward-compatible. We will \textbf{eventually}
support GICv3, the current specification at time of writing.

\subsection{Inter-Domain Communication}

User-level communication between cache-coherent cores in Barrelfish for ARMv8
is likely to the same as with ARMv7 and x86, and we expect the existing
User-level Message-Passing over Cache-Coherence (UMP-CC) interconnect driver
to work unmodified.

Between dispatchers on the same core, however, the different register set on
the ARMv8 is likely to result in a very different Local Message Passing (LMP)
interconnect driver---this is always an architecture-specific part of the CPU
driver. In practice, its design will be closely tied to the context switch and
upcall dispatch code.

\chapter{Booting}\label{c:booting}

Booting ARM systems has always been difficult to do in a standard way,
and ARMv8 systems are no exception.  Barrelfish uses one of two
methods of booting an initial ARMv8 core, depending on whether the
hardware platform supports UEFI~\cite{uefi} or U-Boot.  If a platform
supports neither, more work will be required to boot the board.

If a board has full support for UEFI (such as TianoCore), you can use
Hagfish~\ref{s:hagfish} to individually load the modules needed to
boot Barrelfish and set up the initial CPU/MMU environment before
entering the CPU driver proper. 

Note that U-Boot also claims to support UEFI.  However, in practice it
supports a small subset of UEFI functionality sufficient to boot
\texttt{grub} or the Linux kernel as an EFI binary.   If your board
boots via U-Boot, you should use the minimal EFI
bootloader~\ref{s:uboot} which loads a single multiboot image into
memory and sets up the environment similar to Hagfish. 

\section{Hagfish}\label{s:hagfish}

The Barrelfish/ARMv8 UEFI loader prototype is called Hagfish\footnote{A
hagfish is a basal chordate i.e. something like the ancestor of all fishes.}.
Hagfish is a second-stage bootloader for Barrelfish on UEFI platforms,
initially the ARMv8 server platform.  Hagfish is loaded as a UEFI application,
and uses the large set of supplied services to do as much of the one-time
(boot core) setup that the CPU driver needs as is reasonably possible. More
specifically, Hagfish:

\begin{itemize}
\item Is loaded over BOOTP/PXE.
\item Reuses the PXE environment to load a menu.lst-style configuration.
\item Loads the kernel image and the initial applications, as directed, and
builds a Multiboot image.
\item Allocates and builds the CPU driver's page tables.
\item Activates the initial page table, and allocates a stack.
\end{itemize}

\subsection{Why Another Bootloader?}

The ARMv8 machines that we're porting to are different to both existing ARM
boards, and to x86. They have a full pre-boot environment, unlike most
embedded boards, but it's not a PC-style BIOS. The ARM Server Base Boot
Requirements specify UEFI. Moreover, there is no mainline support from GNU
GRUB for the ARMv8 architecture, so no matter what, we need some amount of
fresh code.

Given that we had to write at least a shim loader, and keeping in mind that
UEFI is multi-platform (and becoming more and more common in the x86 world),
we're taking the opportunity to simplify the initial boot process within the
CPU driver by moving the once-only initialisation into the bootloader. In
particular, while running under UEFI boot services, we have memory allocation
available for free, e.g. for the initial page tables. By moving ELF loading
and relocation code into the bootloader, we can eliminate the need to relocate
running code, and can cut down (hopefully eliminate) special-case code for
booting the initial core. Subsequent cores can rely on user-level Coreboot
code to relocate them, and to construct their page tables.

\subsection{Assumptions and Requirements}

Hagfish is (initially at least) intended to support development work on
AArch64 server-style hardware and, as such, makes the following assumptions:

\begin{itemize}
\item 64-bit architecture, using ELF binaries. Porting to 32-bit architectures
wouldn't be hard, if it were ever necessary (probably not).
\item PXE/BOOTP/TFTP available for booting. Hagfish expects to load its
configuration, and any binaries needed, using the same PXE context with which
it was booted. Changing this to boot from a local device (e.g. HDD) wouldn't
be hard, as the UEFI \texttt{LoadFile} interface abstracts from the hardware.
\end{itemize}

\subsection{Boot Process}

In detail, Hagfish currently boots as follows:

\begin{enumerate}
\item \texttt{Hagfish.efi} is loaded over PXE by UEFI, and is executed at a
runtime-allocated address, with translation (MMU) and caching enabled.
\item Hagfish queries EFI for the PXE protocol instance used to load it, and
squirrels away the current network configuration.
\item Hagfish loads the file \texttt{hagfish.A.B.C.D.cfg} from the TFTP server
root (where \texttt{A.B.C.D} is the IP address on the interface that ran PXE).
\item Hagfish parses its configuration, which is essentially a GRUB menu.lst,
and loads the kernel image and any additional modules specified therein. All
ELF images are loaded into page-aligned regions of type
\texttt{EfiBarrelfishELFData}.
\item Hagfish queries UEFI for the system memory map, then allocates and
initialises the inital page tables for the CPU driver (mapping all occupied
physical addresses, within the \texttt{TTBR1} window, see \autoref{s:layout}).
The frames holding these tables are marked with the EFI memory type\\
\texttt{EfiBarrelfishBootPagetable}, allocated from the OS-specific range
(\texttt{0x80000000}--\texttt{0x8fffffff}). All memory allocated by Hagfish on
behalf of the CPU driver is page-aligned, and tagged with an OS-specific type,
to allow EFI and Hagfish regions to be safely reclaimed.
\item Hagfish builds a Multiboot 2 information structure, containing as much
information as it can get from EFI, including:
    \begin{itemize}
    \item ACPI 1.0 and 2.0 tables.
    \item The EFI memory map (including Hagfish's custom-tagged regions).
    \item Network configuration (the saved DHCP ack packet).
    \item The kernel command line.
    \item All loaded modules.
    \item The kernel's ELF section headers.
    \end{itemize}
\item Hagfish allocates a page-aligned kernel stack (type
\texttt{EfiBarrelfishCPUDriverStack}), of the size specified in the
configuration.
\item Hagfish terminates EFI boot services (calls \texttt{ExitBootServices}),
activates the CPU driver page table, switches to the kernel stack, and jumps
into the relocated CPU driver image.
\end{enumerate}

\subsection{Post-Boot state}

When the CPU driver on the boot core begins executing, it can assume the
following:

\begin{itemize}
\item The MMU is configured with all RAM and I/O regions mapped via
\texttt{TTBR1}.
\item The CPU driver's code and data are both fully relocated into one or more
distinct 4kiB-aligned regions.
\item The stack pointer is at the top of a distinct 4kiB-aligned region of at
least the requested size.
\item The first argument register holds the Multiboot 2 magic value.
\item The second holds a pointer to a Multiboot 2 information structure, in
its own distinct 4kiB-aligned region.
\item The console device is configured.
\item Only one core is enabled.
\item The Multiboot structure contains at least:
    \begin{itemize}
    \item The final EFI memory map, with all areas allocated by Hagfish to
    hold data passed to the CPU driver marked with OS-specific types, all of
    which refer to non-overlapping 4k-aligned regions:
        \begin{description}
        \item[\ttfamily EfiBarrelfishCPUDriver]
        The currently-executing CPU driver's text and data segments.
        \item[\ttfamily EfiBarrelfishCPUDriverStack]
        The CPU driver's stack.
        \item[\ttfamily EfiBarrelfishMultibootData]
        The Multiboot structure.
        \item[\ttfamily EfiBarrelfishELFData]
        The unrelocated ELF image for a boot-time module (including that for
        the CPU driver itself), as loaded over TFTP.
        \item[\ttfamily EfiBarrelfishBootPageTable]
        The currently-active page tables.
        \end{description}
    \item The CPU driver (kernel) command line.
    \item A copy of the last DHCP Ack packet.
    \item A copy of the section headers from the CPU driver's ELF image.
    \item Module descriptions for the CPU driver and all other boot modules.
    \item If UEFI provided an ACPI root table, the Multiboot structure
    contains a pointer to it.
    \end{itemize}
\end{itemize}

\subsection{Configuration}

Hagfish configures itself by loading a file whose path is generated from its
assigned IP address. Thus if your development machine receives the address
192.168.1.100, Hagfish will load the file\\
\texttt{hagfish.192.168.1.100.cfg}
from the same TFTP server used to load it. The format is intended to be as
close as practical to that of an old-style GRUB menu.lst file. The example
configuration in \autoref{f:hag_config} loads
\texttt{/armv8/sbin/cpu\_apm88xxxx} as the CPU driver, with arguments
\texttt{loglevel=3}, and an 8192B (2-page) stack.

\begin{figure}[htb]
\begin{center}
\begin{lstlisting}
kernel /armv8/sbin/cpu_apm88xxxx loglevel=3
stack 8192
module /armv8/sbin/cpu_apm88xxxx
module /armv8/sbin/init

# Domains spawned by init
module /armv8/sbin/mem_serv
module /armv8/sbin/monitor

# Special boot time domains spawned by monitor
module /armv8/sbin/chips boot
module /armv8/sbin/ramfsd boot
module /armv8/sbin/skb boot
module /armv8/sbin/kaluga boot
module /armv8/sbin/spawnd boot bootarm=0
module /armv8/sbin/startd boot

# General user domains
module /armv8/sbin/serial auto portbase=2
module /armv8/sbin/fish nospawn
module /armv8/sbin/angler serial0.terminal xterm

module /armv8/sbin/memtest

module /armv8/sbin/corectrl auto
module /armv8/sbin/usb_manager auto
module /armv8/sbin/usb_keyboard auto
module /armv8/sbin/sdma auto
\end{lstlisting}
\end{center}
\caption{Hagfish configuration file}
\label{f:hag_config}
\end{figure}



\subsection{Booting with Hagfish in \qemu}\label{c:qemu}

When booting a \qemu image for 64-bit ARM, a number of options are
available (see \texttt{make help-boot}).  Building a boot image for
\qemu with ARMv8 will typically result in a file in the build directory
called \texttt{armv8_<core_type>_qemu_image}.  This is a disk image which can be
read by Hagfish through EFI calls.

Booting this with a boot target from \texttt{make} will run the
following:
\begin{lstlisting}
  srcdir/tools/qemu-wrapper.sh \\
     --image armv8_<core_type>_qemu_image \\
     --arch armv8 \\
     --bios ../git/barrelfish/tools/hagfish/QEMU_EFI.fd
\end{lstlisting}

This wrapper script is complex, but reasonably well documented (use
'\texttt{--help}').  It will invoke \qemu as follows:
\begin{lstlisting}
  qemu-system-aarch64 \\
  -m 1024 \\
  -cpu cortex-a57 \\
  -M virt \\
  -d guest_errors \\
  -M gic_version=3 \\
  -smp 1 \\
  -bios ../git/barrelfish/tools/hagfish/QEMU_EFI.fd \\
  -device virtio-blk-device,drive=image \\
  -drive if=none,id=image,file=armv8_<core_type>_qemu_image,format=raw \\
  -nographic
\end{lstlisting}


Note that for this script to work, you need to have \texttt{mtools}
(the MS-DOS file system manipulation tools) installed, since they are
used to prepare the \texttt{armv8_<core_type>_qemu_image} file.

More specifically, the \texttt{armv8_<core_type>_qemu_image} file is generated
by \texttt{tools/harness/efiimage.py}.  This creates an EFI file
system image out of the plain Barrelfish binaries built in
\texttt{\textit{builddir}/armv8/sbin}, plus the Hagfish EFI image we
regularly use for real hardware.  The \texttt{QEMU\_EFI.fd} file is
the UEFI runtime built for \qemu. 

\section{Booting from U-Boot}\label{s:uboot}

Where a full UEFI environment is not available, it is possible to boot
Barrelfish from U-Boot~\cite{uboot}.  We boot Barrelfish from U-Boot
using U-Boot's limited EFI support: a build-time tool
(\texttt{armv8\_bootimage} builds a single binary which only requires
the minimal EFI environment provided by U-Boot.   This binary contains
a loader (\texttt{efi\_loader}) which sets up the rest of the image as
a multiboot image in memory before starting the CPU driver.

\subsection{Booting in \qemu with U-Boot}

A ``platform'' target like \texttt{QEMU\_UBoot} which build such an image
for \qemu, and the \texttt{qemu-wrapper.sh} script can be invoked to
use U-Boot instead of Hagfish:

\begin{lstlisting}
  srcdir/tools/qemu-wrapper.sh \\
  --image armv8_a57_qemu_image.efi \\
  --arch armv8 \\
  --uboot-img srcdir/tools/qemu-armv8-uboot.bin
\end{lstlisting}

This invoked \qemu as follows:

\begin{lstlisting}
  qemu-system-aarch64 \\
  -m 1024 \\
  -cpu cortex-a57 \\
  -M virt \\
  -d guest_errors \\
  -M gic_version=3 \\
  -smp 1 \\
  -bios srcdir/tools/qemu-armv8-uboot.bin \\
  -device loader,addr=0x50000000,file=armv8_a57_qemu_image.efi \\
  -nographic
\end{lstlisting}

As you can see, the UBoot binary is given as the BIOS, and the minimal
EFI image with the complete set of multiboot modules compiled in is
pre-loaded into memory when \qemu starts.

\chapter{Technical Observations}\label{c:tech}

\section{User-Space Threading}\label{s:threads}

\begin{figure}[htb]
\begin{center}
\begin{minipage}[t]{0.3\textwidth}
\begin{lstlisting}
clrex
/* Restore CPSR */
ldr r0, [r1], #4
msr cpsr, r0
/* Restore registers */
ldmia r1, {r0-r15}
\end{lstlisting}
\end{minipage}
\hspace{2cm}
\begin{minipage}[t]{0.5\textwidth}
\begin{lstlisting}
/* Restore PSTATE, load resume
 * address into x18 */
ldp x18, x2, [x1, #(PC_REG * 8)]
/* Set only NZCV. */
and x2, x2, #0xf0000000
msr nzcv, x2
/* Restore the stack pointer and x30. */
ldp x30, x2, [x1, #(30 * 8)]
mov sp, x2
/* Restore everything else. */
ldp x28, x29, [x1, #(28 * 8)]
ldp x26, x27, [x1, #(26 * 8)]
ldp x24, x25, [x1, #(24 * 8)]
ldp x22, x23, [x1, #(22 * 8)]
ldp x20, x21, [x1, #(20 * 8)]
/* n.b. don't reload x18 */
ldr      x19, [x1, #(19 * 8)]
ldp x16, x17, [x1, #(16 * 8)]
ldp x14, x15, [x1, #(14 * 8)]
ldp x12, x13, [x1, #(12 * 8)]
ldp x10, x11, [x1, #(10 * 8)]
ldp  x8,  x9, [x1, #( 8 * 8)]
ldp  x6,  x7, [x1, #( 6 * 8)]
ldp  x4,  x5, [x1, #( 4 * 8)]
ldp  x2,  x3, [x1, #( 2 * 8)]
/* n.b. this clobbers x0&x1 */
ldp  x0,  x1, [x1, #( 0 * 8)]
/* Return to the thread. */
br x18
\end{lstlisting}
\end{minipage}
\end{center}
\caption{\texttt{disp\_resume\_context} on ARMv7 (left) and ARMv8 (right)}
\label{f:disp_resume}
\end{figure}

The ARMv8 architecture is in some ways an improvement, and in other ways
problematic, for the sort of user-level threading implemented in Barrelfish,
via \emph{scheduler activations}. Under this scheme, the kernel (in Barrelfish
terms, the \emph{CPU driver}), does not schedule threads directly, but instead
exposes all scheduling-relevant events via \emph{upcalls} to predefined
user-level handlers (in Barrelfish, the \emph{dispatcher}), which then
implements thread scheduling (or something else entirely), as it sees fit.
This differs from the behaviour of a system such as UNIX, which only ever
restores a user-level execution context simultaneously with dropping from a
privileged to an unprivileged execution level.

Processor architectures are, understandably, designed with common software in
mind. Thus, the primitives available for restoring an execution context i.e.
register state are often tied closely to those for changing privilege level. A
common design (which ARMv8 also implements) is the \emph{exception return},
where privileged code can atomically drop its privilege, and jump to a
user-level execution address. In ARMv8, the \texttt{eret} instruction
atomically updates the program state (PSTATE, most importantly the privilege
level bits), and branches to the address held in the \emph{exception link
register}, \texttt{elr}.

In implementing user-level threading, we're not concerned with privilege
levels, but the lack of some equivalent of \texttt{elr} is frustrating. Not
only does \texttt{eret} provide an atomic update of the program counter and
the program state, it does so without modifying any general-purpose register.
Replicating this behaviour at \texttt{EL0}, where \texttt{eret} is unavailable
is problematic. ARMv8 differs from ARMv7, in that the program counter can no
longer be the target of a load instruction, but can only be loaded via a
general-purpose register.

Specifically, the only PC-modifying instructions (other than \texttt{eret})
are PC-relative branches (which are useless in this scenario) and
branch-to-register (of which \texttt{br}, \texttt{blr} and \texttt{ret} are
all special encodings). Since ARMv8 has also removed the \texttt{ldm} (load
multiple) instruction, there is no way to load the program counter with an
arbitrary value (the thread's restart address), without overwriting one of the
general-purpose registers. We cannot restore the thread's register value
\emph{before} we branch to it, as we'd overwrite the return address, and we
obviously can't do so afterwards, as the thread likely has no idea that it's
been interrupted. The only alternative is to trampoline through kernel mode in
order to use \texttt{eret} (which would eliminate the speed benefit of
user-level threading), or to reserve a general-purpose register for use by the
dispatcher. Neither option is appealing, but we went with the second option,
reserving \texttt{x18}, reasoning that with 31 general-purpose registers
available, the loss of one isn't a huge penalty. Register \texttt{x18} is
explicitly marked as the \emph{platform register} in the AArch64 ABI
\citep{arm:aa64pcs}, for such a purpose.

Future revisions of the ARM architecture could prevent this issue in a number
of ways: allowing the use of \texttt{eret} at \texttt{EL0} or providing an
equivalent functionality (specifically a non-general-purpose register such as
\texttt{elr}, that doesn't need to be restored); or alternatively, adding
indirect jumps (load to PC) back to the instruction set.

\autoref{f:disp_resume} compares the user-level thread resume code for the
Barrelfish dispatcher (function \texttt{disp\_resume}) for ARMv7 and ARMv8
side-by-side. The effect of removing the load-multiple instructions, and
direct-to-SP loads, on code density is clearly visible: everything on lines
8--29 for ARMv8 corresponds to the single \texttt{ldmia} instruction on lines
9 for ARMv7 --- one instruction is now 18, on the thread-switch critical path!
Note also, on line 17, that the ARMv8 code does not restore the thread's
\texttt{r18}, but instead uses it to hold the branch address for use on line
29. The only improvement on ARMv8 is that the \texttt{clrex} (clear exclusive
monitor) instruction is no longer required, as the monitor is cleared on
returning from the kernel. Note also that the usual method to efficiently load
multiple registers, using 16-word SIMD (NEON) loads, isn't available, as
there's no guarantee that the SIMD extensions are enabled on this dispatcher,
and we cannot handle a fault in this code.

\section{Trap Handling}\label{s:traps}

\begin{figure}
\begin{lstlisting}
el0_aarch64_sync:
    msr daifset, #3 /* IRQ and FIQ masked, Debug and Abort enabled. */

    stp x11, x12, [sp, #-(2 * 8)]!
    stp x9,  x10, [sp, #-(2 * 8)]!

    mrs x10, tpidr_el1
    mrs x9, elr_el1

    ldp x11, x12, [x10, #OFFSETOF_DISP_CRIT_PC_LOW]
    cmp x11, x9
    ccmp x12, x9, #0, ls
    ldr w11, [x10, #OFFSETOF_DISP_DISABLED]
    ccmp x11, xzr, #0, ls
    /* NE <-> (low <= PC && PC < high) || disabled != 0 */

    mrs x11, esr_el1  /* Exception Syndrome Register */
    lsr x11, x11, #26 /* Exception Class field is bits [31:26] */

    b.ne el0_sync_disabled

    add x10, x10, #OFFSETOF_DISP_ENABLED_AREA

save_syscall_context:
    str x7,       [x10, #(7 * 8)]

    stp x19, x20, [x10, #(19 * 8)]
    stp x21, x22, [x10, #(21 * 8)]
    stp x23, x24, [x10, #(23 * 8)]
    stp x25, x26, [x10, #(25 * 8)]
    stp x27, x28, [x10, #(27 * 8)]
    stp x29, x30, [x10, #(29 * 8)] /* FP & LR */

    mrs x20, sp_el0
    stp x20, x9, [x10, #(31 * 8)]

    mrs x19, spsr_el1
    str x19, [x10, #(33 * 8)]

    cmp x11, #0x15 /* SVC or HVC from AArch64 EL0 */
    b.ne el0_abort_enabled

    add sp, sp, #(4 * 8)

    mov x7, x10

    b sys_syscall
\end{lstlisting}
\caption{BF/ARMv8 synchronous exception handler}
\label{f:sync_el0}
\end{figure}

\autoref{f:sync_el0} shows the CPU driver exception stub, for a synchronous
abort from \texttt{EL0}. This exception class includes system calls,
breakpoints, and page faults on both code and data. The effect of the loss of
store multiple instructions is again visible, for example on lines 27--32.
Although not as severe as in the case of the user-level thread restore in
\autoref{s:threads}, the extra instructions required do constrain us somewhat,
as each trap handler is constrained to 128 bytes, or 32 instructions, before
branching to another code block.

We were able to squeeze the necessary code into the space available, including
the optimised test for a disabled dispatcher at lines 10--14, but only by
splitting the page fault handler (\texttt{el0\_abort\_enabled}) into a
separate subroutine, incurring an unnecessary branch. A more significant
annoyance is that system calls (\texttt{svc} and \texttt{hvc}) are routed to
the same exception vector as page faults (aborts).  The effect of this is that
we are forced to spill registers to the stack (\texttt{x9}--\texttt{x12} on
lines 4--5), even on the system call fast path, as we need at least one
register to check the exception syndrome (\texttt{esr\_el1}) to distinguish
aborts (where we must preserve all registers) from system calls (where we
could immediately begin using the caller-saved registers). Note that the code
on lines 27--32 only needs to stack the callee-saved registers, and leaves the
system call arguments in \texttt{x0}--\texttt{x7}, to be read as required by
\texttt{sys\_syscall} (in C).

This sort of mismatch between the exception-handling interface of the CPU
architecture, and what is required for really high-performance systems code is
unfortunately extremely common. Unnecessary overheads, such as the additional
stacked registers here hurt the performance of highly-componentised systems,
such as Barrelfish, which rely on frequently crossing protection domains.

The relatively well-compressed boolean arithmetic on lines 10--14 demonstrates
that, even with the loss of ARM's fully-conditional instructions, the
conditional compares which remain are still relatively powerful.

\section{Cache Coherence}

One aspect of the ARM architecture that is of particular interest for the
Barrelfish project, but which we have not yet explored in depth, is the
configurable cache coherency and fine-grained cache management operations
available. Any virtual mapping on a recent ARM architecture, including both
ARMv7 and ARMv8, can be tagged with various cacheability properties: inner
(L1), outer (L2+, usually), write-back or write-through. Combined with the
explicit flush operations at cache-line granularity, able to target either PoU
(point of unification, where data and instruction caches merge) or PoC (point
of coherency, typically RAM), a multi-core, multi-socket ARMv8 system would
make a very interesting testbed for investigating efficient cache management
and communication primitives for future partially-coherent architectures.
Indeed, the latest revision of the ARMv8 specification, ARMv8.2, introduced
flush to PoP, or \emph{point of persistence} --- perhaps in response to
interest from well-known systems integration firms investigating large
persistent memories.

The design presented in this report is intended to expose as much control over
the caching hierarchy as possible to user-level code, to provide a platform
for future research.

\bibliographystyle{plainnat}
\bibliography{defs,barrelfish}

\end{document}