1\documentclass[a4paper,11pt,twoside]{report} 2\usepackage{bftn} 3\usepackage{calc} 4\usepackage{verbatim} 5\usepackage{xspace} 6\usepackage{pifont} 7\usepackage{pxfonts} 8\usepackage{textcomp} 9\usepackage{amsmath} 10\usepackage{multirow} 11\usepackage{listings} 12\usepackage[framemethod=default]{mdframed} 13\usepackage[shortlabels]{enumitem} 14\usepackage{parskip} 15\usepackage{xparse} 16 17\newcommand{\todo}[1]{[\textcolor{red}{\emph{#1}}]} 18 19\title{Coreboot in Barrelfish} 20\author{Barrelfish project} 21\tnnumber{23} 22\tnkey{Coreboot} 23 24\begin{document} 25\maketitle % Uncomment for final draft 26 27\begin{versionhistory} 28\vhEntry{0.1}{28.02.2017}{GZ}{Initial Version} 29\vhEntry{0.1}{02.03.2017}{RA}{Adding basic structure} 30\end{versionhistory} 31 32% \intro{Abstract} % Insert abstract here 33% \intro{Acknowledgements} % Uncomment (if needed) for acknowledgements 34\tableofcontents % Uncomment (if needed) for final draft 35% \listoffigures % Uncomment (if needed) for final draft 36% \listoftables % Uncomment (if needed) for final draft 37\cleardoublepage 38\setcounter{secnumdepth}{2} 39 40\newcommand{\fnname}[1]{\textit{\texttt{#1}}}% 41\newcommand{\datatype}[1]{\textit{\texttt{#1}}}% 42\newcommand{\varname}[1]{\texttt{#1}}% 43\newcommand{\keywname}[1]{\textbf{\texttt{#1}}}% 44\newcommand{\pathname}[1]{\texttt{#1}}% 45\newcommand{\tabindent}{\hspace*{3ex}}% 46\newcommand{\sockeye}{\lstinline[language=sockeye]} 47\newcommand{\ccode}{\lstinline[language=C]} 48 49\lstset{ 50 language=C, 51 basicstyle=\ttfamily \small, 52 keywordstyle=\bfseries, 53 flexiblecolumns=false, 54 basewidth={0.5em,0.45em}, 55 boxpos=t, 56 captionpos=b 57} 58 59\chapter{Introduction} 60\label{chap:introduction} 61 62This document describes the way Barrelfish boots cores on supported 63architectures. We first explain the terminology used throughout this document. 64Then, we will give an overview of supported operations and finally, explain in 65detail how the implementation of these operations on differ for various 66supported architectures. 67 68\section{Boot drivers} 69 70Barrelfish uses \textit{boot drivers}, which is a piece of code running on a 71``home core'' and manages a ``target core''. It encapsulates the hardware 72functionality to boot, suspend, resume, and power-down the latter. Boot drivers 73run as processes, but closely resemble device drivers and could equally run as 74software objects within another process. The boot driver abstraction treats CPU 75cores much like peripheral devices, and allows us to reuse the OS's existing 76device and hot-plug management infrastructure to handle new cores and select 77drivers and kernels for them. The code for the Barrelfish boot driver can be 78found in \texttt{usr/drivers/cpuboot/}. From the source folder, a binary called 79``corectrl'' is generated at build time and can be used to do target core 80management. The ``corectrl'' program is a command line tool that supports 81different operations as described below. 82 83\section{CPU driver} 84 85Is the binary that runs on a core and executes in privileged mode. CPU drivers 86are the Barrelfish equivalent of a kernel, except that there is one per core, 87and they share no state or synchronization. CPU drivers are typically 88non-preemptible, single-threaded, and mostly stateless. The CPU driver binary is 89selected and loaded by a \textit{boot driver}, which among other things, is 90responsible of making sure the CPU driver is executed on a core. TN-021 contains 91a more detailed explanation of CPU drivers and the provided functionality. 92 93The code for the Barrelfish CPU drivers can be found in \texttt{kernel/}. 94 95\section{Kernel Control Block (KCB)} 96 97The Kernel Control Block (or KCB) is a memory region that can only be accessed 98by kernel-space. However, a program, especially boot drivers, can create and 99reference KCBs in user-space through the capability system. The Kernel Control 100Block's purpose is to hold or keep pointers to all per-core state. All per-KCB 101state maintained by a kernel must be reachable from the KCB itself or 102reconstructible by reading the KCB. 103 104Usually, a KCB is created for each core we start and passed to the new core on 105start-up. On boot-up, the kernel checks if the KCB it received is initialized. 106In that case, it will use the KCB and start to dispatch the applications it 107contains. If the KCB is uninitialized, the kernel sets all the entries of the 108KCB to its default values and continues bootstrapping the core (and the KCB 109block) by starting the monitor application, which is critical for communication 110across cores in user-space. 111 112Every kernel has the possibility to maintain multiple KCBs. The kernel maintains 113a ring structure of KCBs it owns. A CPU driver supports switching to another KCB 114on demand. In case a CPU driver is running multiple KCBs it divides the 115time-slices evenly among different KCBs and switches KCBs after a given amount 116of time has passed. Currently, the CPU driver uses round-robin scheduling for 117having multiple KCBs on one core but other models are possible as well. 118 119 120\chapter{Generic Operations} 121\label{chap:generic} 122 123The boot driver (currently called ``corectrl'') has support for a range of 124operations, which we describe in more detail in this chapter. As a summary the 125operations are listed here: 126 127\begin{itemize} 128\item boot \textless target coreid\textgreater: Boots a new core with a KCB. 129\item stop \textless target coreid\textgreater: Stop execution on an existing core. 130\item update \textless target coreid\textgreater: Update the CPU driver on an existing core. 131\item give \textless from kcbid\textgreater \textless to kcbid\textgreater: Give kcb from one core to another. 132\item rmkcb \textless kcbid\textgreater: Remove a running KCB from a core. 133%\item park \textless kcbid to stop\textgreater \textless recv kcbid\textgreater: Stop execution on an existing core and park KCB on another core. 134%\item unpark \textless kcbid to unpark\textgreater: Reestablish parked KCB on its original core. 135\item lscpu: List current status of all cores. 136\item lskcb: List current KCBs. 137\end{itemize} 138 139Note that the implementation of ``corectrl'' in it's current from is essentially 140a command line tool that must be invoked for every operation. This means that 141the boot driver is currently state-less which is fine for just executing the 142operations since all information required can be reconstructed from the SKB. 143 144There are a few optional flags that can be passed to the different operations: 145\begin{itemize} 146\item -d, --debug: Print debug information 147\item -k, --kernel \textless binary\textgreater: Overwrite default kernel binary. 148\item -x, --monitor \textless binary\textgreater: Overwrite default monitor binary. 149\item -a, --kargs \textless args\textgreater: Overwrite default kernel command line arguments. 150\item -n, --newkcb: Create a new KCB even if there is already one for that core. 151\item -m, --nomsg: Don't wait for a monitor message. 152\end{itemize} 153 154 155\section{boot \textless target coreid\textgreater} 156 157Starts a CPU driver on a core. If the new core that has never been started 158before, it is brought online as follows: 159 160\begin{penumerate} 161 162\item The new core is detected by some platform-specific mechanism (e.g., ACPI) 163and its appearance registered with the device management subsystem. This is done 164by adding an octopus record to the SKB. The record name is of the format 165\texttt{hw.processor.ID} where ID is an identifier guaranteed to be unique by the 166SKB/Octopus. The record has the following mandatory, architecture independent 167fields: 168 169\begin{itemize} 170 171\item \texttt{enabled}: A boolean field that says if the core is usable or not 172(for example, on x86 if hyper-threads are disabled, this flag would be set to 173false for hyper-threads -- the resulting core would not be booted by default). 174 175\item \texttt{barrelfish\_id}: A unique integer identifier used by Barrelfish to 176refer to this core. 177 178\item \texttt{hw\_id}: An identifier assigned by the hardware/platform to this 179core. 180 181\item \texttt{type}: A number that corresponds to that cores architecture as 182described in \texttt{enum cpu\_type} (defined in barrelfish\_kpi/cpu.h). 183 184\end{itemize} 185 186\item Barrelfish selects and starts an appropriate boot driver for the new core. 187This is done by propagating the added octopus record to kaluga which then 188invokes ``corectrl'' with the boot command. 189 190\item The boot driver selects a kernel binary and arguments for the new core, 191and directs the boot driver to boot the kernel on the core. By default 192``corectrl'' uses the \texttt{type} field in the octopus record to decide which 193binary to spawn on the target core. However, this can be overridden by manually 194selecting a kernel with the --kernel flag. Alternatively, the default 195parameters passed to the kernel can be overridden with the --kargs option. 196 197\item The boot driver loads and relocates the kernel, and executes the hardware 198protocol to start the new core. This is done by using the file system to load 199the binaries and additionally a series of system calls to invoke protected 200operations (starting a core, sending an IPI etc.). In this process, also a new 201KCB is created for the core and initialized with default arguments. 202 203\item The new kernel initializes and uses existing Barrelfish protocols for 204integrating into the running OS. This involves spawning the first user-space 205program (the monitor) on the new core. The default choice 206for the first user-space program loaded on the new core can be overridden 207by using the --monitor flag. 208 209\item The monitor will inform the other boot driver about its existence by 210sending a message back to it. Then, the boot driver will tell its local monitor 211about the newly available and initialized core. In some cases (for example if 212something else is spawned than the default monitor binary) we may not want to 213wait for such a message. If the --nomsg flag is passed to ``corectrl'' this 214step is skipped. 215 216\item The SKB is updated with information about the new core: The core is marked 217online and booted. The KCB that was created during the boot process by 218``corectrl'' is stored in the capability storage in case it needs to be 219retrieved at a different point in time. 220 221\end{penumerate} 222 223At least three special cases need to be considered: 224 225\begin{itemize} 226 227 \item The core id is invalid: If we don't find the core in the SKB, 228 the operation is aborted. 229 230 \item The core is already running: In that case we abort with an error. 231 232 \item The core has been booted before: By default ``corectrl'' will check if 233 an existing KCB for the given core already exists. If yes, the capability 234 for the KCB will be retrieved from the SKB and passed to the new core. This 235 behavior can be overridden by passing the --newkcb flag which makes sure 236 that an existing KCB is not reused. 237 238\end{itemize} 239 240 241\section{stop \textless target coreid\textgreater} 242 243Stops a CPU driver on a core. If the core is already stopped, nothing is done. 244The KCB running on this core will stop dispatching until the core is either 245restarted, or the KCB is given to another core. 246 247In general the operations executed is follow: 248 249\begin{itemize} 250 251\item Invoke hardware specific protocol to stop the core. 252 253\item Optional: Update meta-data in SKB to mark core as stopped. 254 255\end{itemize} 256 257This operations does not take any optional flags or additional arguments. 258 259Note that while this is deliberately very simple and low-overhead, just having 260the KCB stop to execute is generally a bad idea as there may be messages waiting 261to be received on that core etc. which can cause the whole system to lock-up. It 262makes sense for example to park the KCB after stopping (see the give operation 263later). 264 265\section{update \textless target coreid\textgreater} 266 267Update is implemented as a sequence of executing the \texttt{stop} and 268\texttt{start} command. It takes the same arguments and flags as described in 269the start section. 270 271\section{give \textless target kcbid\textgreater \textless destination kcbid\textgreater} 272 273Removes a KCB on the core it is currently executing and transfers it to 274the core that is currently dispatching the destination KCB. 275This command involves several steps: 276 277\begin{penumerate} 278 279\item The boot driver instructs its local monitor to send a message to the 280monitor of the target KCB. The message instructs the receiving monitor to inform 281its local CPU driver to stop dispatching the KCB (or OSNode). 282 283\item The boot driver sends message to the monitor of the destination KCB, 284passing along the cap reference of the target KCB. Upon receipt, the destination 285monitor will hand the reference to its CPU driver. 286 287\item The destination CPU driver will do some minimal initialization, resetting 288timers and scheduling state, check for registered interrupts and up-call drivers 289to re-register them, and then start dispatching the target KCB. 290 291\item The boot driver will update the SKB to change the core assigned to the 292target kcbid. 293 294\end{penumerate} 295 296 297The following special cases need to be considered: 298 299\begin{itemize} 300 301 \item The target or destination KCB ID is invalid: The operation is aborted. 302 303 \item The destination KCB is not currently assigned a core: The operation is 304 aborted. 305 306 \item The target KCB is not currently running on a core: The step where we 307 first send a message to the target KCB is skipped. 308 309\end{itemize} 310 311 312\section{rmkcb \textless kcbid\textgreater} 313 314\begin{penumerate} 315 316\item The boot driver instructs its local monitor to send a message to the 317monitor of the target KCB. 318 319\item Upon receipt, the monitor of the target KCB will instruct the CPU driver 320to remove the KCB from the scheduling queue. If queue is now empty, the CPU 321driver will stop itself, otherwise it will switch to the next KCB in the list. 322 323\end{penumerate} 324 325The following special cases need to be considered: 326 327\begin{itemize} 328 329 \item The KCB id is invalid: If we don't find the KCB capability in the SKB, 330 the operation is aborted. 331 332 \item The KCB is not dispatched on any core: If the KCB is not currently 333 associated with any core, it can't be removed and therefore the operation 334 is a NOP. 335 336\end{itemize} 337 338\todo{Don't remember exactly if we need to send this to KCBID we want to 339remove or the core/KCB that is currently hosting us...} 340 341\chapter{Booting x86 Cores} 342\label{chap:x86} 343 344\section{Core discovery and identification} 345 346\section{Booting a new core} 347 348 349 350\chapter{Booting ARMv7 Cores} 351\label{chap:armv7} 352\todo{explain ARMv7 specific boot arguements / protocols} 353 354\section{Core discovery and identification} 355 356\section{Booting a new core} 357 358\chapter{Booting ARMv8 Cores} 359\label{chap:armv8} 360 361\todo{explain ARMv8 specific boot arguements / protocols} 362 363\section{Core discovery and identification} 364 365\section{Booting a new core} 366% 367%http://infocenter.arm.com/help/topic/com.arm.doc.den0044b/DEN0044B_Server_Base_Boot_Requirements.pdf 368 369\subsection{Power State Coordination Interface (PSCI)} 370 371% 372%http://infocenter.arm.com/help/topic/com.arm.doc.den0022c/DEN0022C_Power_State_Coordination_Interface.pdf 373 374\subsection{Parking Protocol} 375%https://acpica.org/sites/acpica/files/MP%20Startup%20for%20ARM%20platforms.doc 376 377\subsection{All at once} 378Raspberry Pi 379 380 381%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 382\bibliographystyle{abbrv} 383\bibliography{barrelfish} 384 385\end{document} 386