12: HOW THE DEVELOPMENT PROCESS WORKS 2 3Linux kernel development in the early 1990's was a pretty loose affair, 4with relatively small numbers of users and developers involved. With a 5user base in the millions and with some 2,000 developers involved over the 6course of one year, the kernel has since had to evolve a number of 7processes to keep development happening smoothly. A solid understanding of 8how the process works is required in order to be an effective part of it. 9 10 112.1: THE BIG PICTURE 12 13The kernel developers use a loosely time-based release process, with a new 14major kernel release happening every two or three months. The recent 15release history looks like this: 16 17 2.6.26 July 13, 2008 18 2.6.25 April 16, 2008 19 2.6.24 January 24, 2008 20 2.6.23 October 9, 2007 21 2.6.22 July 8, 2007 22 2.6.21 April 25, 2007 23 2.6.20 February 4, 2007 24 25Every 2.6.x release is a major kernel release with new features, internal 26API changes, and more. A typical 2.6 release can contain over 10,000 27changesets with changes to several hundred thousand lines of code. 2.6 is 28thus the leading edge of Linux kernel development; the kernel uses a 29rolling development model which is continually integrating major changes. 30 31A relatively straightforward discipline is followed with regard to the 32merging of patches for each release. At the beginning of each development 33cycle, the "merge window" is said to be open. At that time, code which is 34deemed to be sufficiently stable (and which is accepted by the development 35community) is merged into the mainline kernel. The bulk of changes for a 36new development cycle (and all of the major changes) will be merged during 37this time, at a rate approaching 1,000 changes ("patches," or "changesets") 38per day. 39 40(As an aside, it is worth noting that the changes integrated during the 41merge window do not come out of thin air; they have been collected, tested, 42and staged ahead of time. How that process works will be described in 43detail later on). 44 45The merge window lasts for two weeks. At the end of this time, Linus 46Torvalds will declare that the window is closed and release the first of 47the "rc" kernels. For the kernel which is destined to be 2.6.26, for 48example, the release which happens at the end of the merge window will be 49called 2.6.26-rc1. The -rc1 release is the signal that the time to merge 50new features has passed, and that the time to stabilize the next kernel has 51begun. 52 53Over the next six to ten weeks, only patches which fix problems should be 54submitted to the mainline. On occasion a more significant change will be 55allowed, but such occasions are rare; developers who try to merge new 56features outside of the merge window tend to get an unfriendly reception. 57As a general rule, if you miss the merge window for a given feature, the 58best thing to do is to wait for the next development cycle. (An occasional 59exception is made for drivers for previously-unsupported hardware; if they 60touch no in-tree code, they cannot cause regressions and should be safe to 61add at any time). 62 63As fixes make their way into the mainline, the patch rate will slow over 64time. Linus releases new -rc kernels about once a week; a normal series 65will get up to somewhere between -rc6 and -rc9 before the kernel is 66considered to be sufficiently stable and the final 2.6.x release is made. 67At that point the whole process starts over again. 68 69As an example, here is how the 2.6.25 development cycle went (all dates in 702008): 71 72 January 24 2.6.24 stable release 73 February 10 2.6.25-rc1, merge window closes 74 February 15 2.6.25-rc2 75 February 24 2.6.25-rc3 76 March 4 2.6.25-rc4 77 March 9 2.6.25-rc5 78 March 16 2.6.25-rc6 79 March 25 2.6.25-rc7 80 April 1 2.6.25-rc8 81 April 11 2.6.25-rc9 82 April 16 2.6.25 stable release 83 84How do the developers decide when to close the development cycle and create 85the stable release? The most significant metric used is the list of 86regressions from previous releases. No bugs are welcome, but those which 87break systems which worked in the past are considered to be especially 88serious. For this reason, patches which cause regressions are looked upon 89unfavorably and are quite likely to be reverted during the stabilization 90period. 91 92The developers' goal is to fix all known regressions before the stable 93release is made. In the real world, this kind of perfection is hard to 94achieve; there are just too many variables in a project of this size. 95There comes a point where delaying the final release just makes the problem 96worse; the pile of changes waiting for the next merge window will grow 97larger, creating even more regressions the next time around. So most 2.6.x 98kernels go out with a handful of known regressions though, hopefully, none 99of them are serious. 100 101Once a stable release is made, its ongoing maintenance is passed off to the 102"stable team," currently comprised of Greg Kroah-Hartman and Chris Wright. 103The stable team will release occasional updates to the stable release using 104the 2.6.x.y numbering scheme. To be considered for an update release, a 105patch must (1) fix a significant bug, and (2) already be merged into the 106mainline for the next development kernel. Continuing our 2.6.25 example, 107the history (as of this writing) is: 108 109 May 1 2.6.25.1 110 May 6 2.6.25.2 111 May 9 2.6.25.3 112 May 15 2.6.25.4 113 June 7 2.6.25.5 114 June 9 2.6.25.6 115 June 16 2.6.25.7 116 June 21 2.6.25.8 117 June 24 2.6.25.9 118 119Stable updates for a given kernel are made for approximately six months; 120after that, the maintenance of stable releases is solely the responsibility 121of the distributors which have shipped that particular kernel. 122 123 1242.2: THE LIFECYCLE OF A PATCH 125 126Patches do not go directly from the developer's keyboard into the mainline 127kernel. There is, instead, a somewhat involved (if somewhat informal) 128process designed to ensure that each patch is reviewed for quality and that 129each patch implements a change which is desirable to have in the mainline. 130This process can happen quickly for minor fixes, or, in the case of large 131and controversial changes, go on for years. Much developer frustration 132comes from a lack of understanding of this process or from attempts to 133circumvent it. 134 135In the hopes of reducing that frustration, this document will describe how 136a patch gets into the kernel. What follows below is an introduction which 137describes the process in a somewhat idealized way. A much more detailed 138treatment will come in later sections. 139 140The stages that a patch goes through are, generally: 141 142 - Design. This is where the real requirements for the patch - and the way 143 those requirements will be met - are laid out. Design work is often 144 done without involving the community, but it is better to do this work 145 in the open if at all possible; it can save a lot of time redesigning 146 things later. 147 148 - Early review. Patches are posted to the relevant mailing list, and 149 developers on that list reply with any comments they may have. This 150 process should turn up any major problems with a patch if all goes 151 well. 152 153 - Wider review. When the patch is getting close to ready for mainline 154 inclusion, it should be accepted by a relevant subsystem maintainer - 155 though this acceptance is not a guarantee that the patch will make it 156 all the way to the mainline. The patch will show up in the maintainer's 157 subsystem tree and into the staging trees (described below). When the 158 process works, this step leads to more extensive review of the patch and 159 the discovery of any problems resulting from the integration of this 160 patch with work being done by others. 161 162- Please note that most maintainers also have day jobs, so merging 163 your patch may not be their highest priority. If your patch is 164 getting feedback about changes that are needed, you should either 165 make those changes or justify why they should not be made. If your 166 patch has no review complaints but is not being merged by its 167 appropriate subsystem or driver maintainer, you should be persistent 168 in updating the patch to the current kernel so that it applies cleanly 169 and keep sending it for review and merging. 170 171 - Merging into the mainline. Eventually, a successful patch will be 172 merged into the mainline repository managed by Linus Torvalds. More 173 comments and/or problems may surface at this time; it is important that 174 the developer be responsive to these and fix any issues which arise. 175 176 - Stable release. The number of users potentially affected by the patch 177 is now large, so, once again, new problems may arise. 178 179 - Long-term maintenance. While it is certainly possible for a developer 180 to forget about code after merging it, that sort of behavior tends to 181 leave a poor impression in the development community. Merging code 182 eliminates some of the maintenance burden, in that others will fix 183 problems caused by API changes. But the original developer should 184 continue to take responsibility for the code if it is to remain useful 185 in the longer term. 186 187One of the largest mistakes made by kernel developers (or their employers) 188is to try to cut the process down to a single "merging into the mainline" 189step. This approach invariably leads to frustration for everybody 190involved. 191 192 1932.3: HOW PATCHES GET INTO THE KERNEL 194 195There is exactly one person who can merge patches into the mainline kernel 196repository: Linus Torvalds. But, of the over 12,000 patches which went 197into the 2.6.25 kernel, only 250 (around 2%) were directly chosen by Linus 198himself. The kernel project has long since grown to a size where no single 199developer could possibly inspect and select every patch unassisted. The 200way the kernel developers have addressed this growth is through the use of 201a lieutenant system built around a chain of trust. 202 203The kernel code base is logically broken down into a set of subsystems: 204networking, specific architecture support, memory management, video 205devices, etc. Most subsystems have a designated maintainer, a developer 206who has overall responsibility for the code within that subsystem. These 207subsystem maintainers are the gatekeepers (in a loose way) for the portion 208of the kernel they manage; they are the ones who will (usually) accept a 209patch for inclusion into the mainline kernel. 210 211Subsystem maintainers each manage their own version of the kernel source 212tree, usually (but certainly not always) using the git source management 213tool. Tools like git (and related tools like quilt or mercurial) allow 214maintainers to track a list of patches, including authorship information 215and other metadata. At any given time, the maintainer can identify which 216patches in his or her repository are not found in the mainline. 217 218When the merge window opens, top-level maintainers will ask Linus to "pull" 219the patches they have selected for merging from their repositories. If 220Linus agrees, the stream of patches will flow up into his repository, 221becoming part of the mainline kernel. The amount of attention that Linus 222pays to specific patches received in a pull operation varies. It is clear 223that, sometimes, he looks quite closely. But, as a general rule, Linus 224trusts the subsystem maintainers to not send bad patches upstream. 225 226Subsystem maintainers, in turn, can pull patches from other maintainers. 227For example, the networking tree is built from patches which accumulated 228first in trees dedicated to network device drivers, wireless networking, 229etc. This chain of repositories can be arbitrarily long, though it rarely 230exceeds two or three links. Since each maintainer in the chain trusts 231those managing lower-level trees, this process is known as the "chain of 232trust." 233 234Clearly, in a system like this, getting patches into the kernel depends on 235finding the right maintainer. Sending patches directly to Linus is not 236normally the right way to go. 237 238 2392.4: STAGING TREES 240 241The chain of subsystem trees guides the flow of patches into the kernel, 242but it also raises an interesting question: what if somebody wants to look 243at all of the patches which are being prepared for the next merge window? 244Developers will be interested in what other changes are pending to see 245whether there are any conflicts to worry about; a patch which changes a 246core kernel function prototype, for example, will conflict with any other 247patches which use the older form of that function. Reviewers and testers 248want access to the changes in their integrated form before all of those 249changes land in the mainline kernel. One could pull changes from all of 250the interesting subsystem trees, but that would be a big and error-prone 251job. 252 253The answer comes in the form of staging trees, where subsystem trees are 254collected for testing and review. The older of these trees, maintained by 255Andrew Morton, is called "-mm" (for memory management, which is how it got 256started). The -mm tree integrates patches from a long list of subsystem 257trees; it also has some patches aimed at helping with debugging. 258 259Beyond that, -mm contains a significant collection of patches which have 260been selected by Andrew directly. These patches may have been posted on a 261mailing list, or they may apply to a part of the kernel for which there is 262no designated subsystem tree. As a result, -mm operates as a sort of 263subsystem tree of last resort; if there is no other obvious path for a 264patch into the mainline, it is likely to end up in -mm. Miscellaneous 265patches which accumulate in -mm will eventually either be forwarded on to 266an appropriate subsystem tree or be sent directly to Linus. In a typical 267development cycle, approximately 10% of the patches going into the mainline 268get there via -mm. 269 270The current -mm patch is available in the "mmotm" (-mm of the moment) 271directory at: 272 273 http://userweb.kernel.org/~akpm/mmotm/ 274 275Use of the MMOTM tree is likely to be a frustrating experience, though; 276there is a definite chance that it will not even compile. 277 278The other staging tree, started more recently, is linux-next, maintained by 279Stephen Rothwell. The linux-next tree is, by design, a snapshot of what 280the mainline is expected to look like after the next merge window closes. 281Linux-next trees are announced on the linux-kernel and linux-next mailing 282lists when they are assembled; they can be downloaded from: 283 284 http://www.kernel.org/pub/linux/kernel/people/sfr/linux-next/ 285 286Some information about linux-next has been gathered at: 287 288 http://linux.f-seidel.de/linux-next/pmwiki/ 289 290How the linux-next tree will fit into the development process is still 291changing. As of this writing, the first full development cycle involving 292linux-next (2.6.26) is coming to an end; thus far, it has proved to be a 293valuable resource for finding and fixing integration problems before the 294beginning of the merge window. See http://lwn.net/Articles/287155/ for 295more information on how linux-next has worked to set up the 2.6.27 merge 296window. 297 298Some developers have begun to suggest that linux-next should be used as the 299target for future development as well. The linux-next tree does tend to be 300far ahead of the mainline and is more representative of the tree into which 301any new work will be merged. The downside to this idea is that the 302volatility of linux-next tends to make it a difficult development target. 303See http://lwn.net/Articles/289013/ for more information on this topic, and 304stay tuned; much is still in flux where linux-next is involved. 305 306Besides the mmotm and linux-next trees, the kernel source tree now contains 307the drivers/staging/ directory and many sub-directories for drivers or 308filesystems that are on their way to being added to the kernel tree 309proper, but they remain in drivers/staging/ while they still need more 310work. 311 312 3132.5: TOOLS 314 315As can be seen from the above text, the kernel development process depends 316heavily on the ability to herd collections of patches in various 317directions. The whole thing would not work anywhere near as well as it 318does without suitably powerful tools. Tutorials on how to use these tools 319are well beyond the scope of this document, but there is space for a few 320pointers. 321 322By far the dominant source code management system used by the kernel 323community is git. Git is one of a number of distributed version control 324systems being developed in the free software community. It is well tuned 325for kernel development, in that it performs quite well when dealing with 326large repositories and large numbers of patches. It also has a reputation 327for being difficult to learn and use, though it has gotten better over 328time. Some sort of familiarity with git is almost a requirement for kernel 329developers; even if they do not use it for their own work, they'll need git 330to keep up with what other developers (and the mainline) are doing. 331 332Git is now packaged by almost all Linux distributions. There is a home 333page at: 334 335 http://git-scm.com/ 336 337That page has pointers to documentation and tutorials. One should be 338aware, in particular, of the Kernel Hacker's Guide to git, which has 339information specific to kernel development: 340 341 http://linux.yyz.us/git-howto.html 342 343Among the kernel developers who do not use git, the most popular choice is 344almost certainly Mercurial: 345 346 http://www.selenic.com/mercurial/ 347 348Mercurial shares many features with git, but it provides an interface which 349many find easier to use. 350 351The other tool worth knowing about is Quilt: 352 353 http://savannah.nongnu.org/projects/quilt/ 354 355Quilt is a patch management system, rather than a source code management 356system. It does not track history over time; it is, instead, oriented 357toward tracking a specific set of changes against an evolving code base. 358Some major subsystem maintainers use quilt to manage patches intended to go 359upstream. For the management of certain kinds of trees (-mm, for example), 360quilt is the best tool for the job. 361 362 3632.6: MAILING LISTS 364 365A great deal of Linux kernel development work is done by way of mailing 366lists. It is hard to be a fully-functioning member of the community 367without joining at least one list somewhere. But Linux mailing lists also 368represent a potential hazard to developers, who risk getting buried under a 369load of electronic mail, running afoul of the conventions used on the Linux 370lists, or both. 371 372Most kernel mailing lists are run on vger.kernel.org; the master list can 373be found at: 374 375 http://vger.kernel.org/vger-lists.html 376 377There are lists hosted elsewhere, though; a number of them are at 378lists.redhat.com. 379 380The core mailing list for kernel development is, of course, linux-kernel. 381This list is an intimidating place to be; volume can reach 500 messages per 382day, the amount of noise is high, the conversation can be severely 383technical, and participants are not always concerned with showing a high 384degree of politeness. But there is no other place where the kernel 385development community comes together as a whole; developers who avoid this 386list will miss important information. 387 388There are a few hints which can help with linux-kernel survival: 389 390- Have the list delivered to a separate folder, rather than your main 391 mailbox. One must be able to ignore the stream for sustained periods of 392 time. 393 394- Do not try to follow every conversation - nobody else does. It is 395 important to filter on both the topic of interest (though note that 396 long-running conversations can drift away from the original subject 397 without changing the email subject line) and the people who are 398 participating. 399 400- Do not feed the trolls. If somebody is trying to stir up an angry 401 response, ignore them. 402 403- When responding to linux-kernel email (or that on other lists) preserve 404 the Cc: header for all involved. In the absence of a strong reason (such 405 as an explicit request), you should never remove recipients. Always make 406 sure that the person you are responding to is in the Cc: list. This 407 convention also makes it unnecessary to explicitly ask to be copied on 408 replies to your postings. 409 410- Search the list archives (and the net as a whole) before asking 411 questions. Some developers can get impatient with people who clearly 412 have not done their homework. 413 414- Avoid top-posting (the practice of putting your answer above the quoted 415 text you are responding to). It makes your response harder to read and 416 makes a poor impression. 417 418- Ask on the correct mailing list. Linux-kernel may be the general meeting 419 point, but it is not the best place to find developers from all 420 subsystems. 421 422The last point - finding the correct mailing list - is a common place for 423beginning developers to go wrong. Somebody who asks a networking-related 424question on linux-kernel will almost certainly receive a polite suggestion 425to ask on the netdev list instead, as that is the list frequented by most 426networking developers. Other lists exist for the SCSI, video4linux, IDE, 427filesystem, etc. subsystems. The best place to look for mailing lists is 428in the MAINTAINERS file packaged with the kernel source. 429 430 4312.7: GETTING STARTED WITH KERNEL DEVELOPMENT 432 433Questions about how to get started with the kernel development process are 434common - from both individuals and companies. Equally common are missteps 435which make the beginning of the relationship harder than it has to be. 436 437Companies often look to hire well-known developers to get a development 438group started. This can, in fact, be an effective technique. But it also 439tends to be expensive and does not do much to grow the pool of experienced 440kernel developers. It is possible to bring in-house developers up to speed 441on Linux kernel development, given the investment of a bit of time. Taking 442this time can endow an employer with a group of developers who understand 443the kernel and the company both, and who can help to train others as well. 444Over the medium term, this is often the more profitable approach. 445 446Individual developers are often, understandably, at a loss for a place to 447start. Beginning with a large project can be intimidating; one often wants 448to test the waters with something smaller first. This is the point where 449some developers jump into the creation of patches fixing spelling errors or 450minor coding style issues. Unfortunately, such patches create a level of 451noise which is distracting for the development community as a whole, so, 452increasingly, they are looked down upon. New developers wishing to 453introduce themselves to the community will not get the sort of reception 454they wish for by these means. 455 456Andrew Morton gives this advice for aspiring kernel developers 457 458 The #1 project for all kernel beginners should surely be "make sure 459 that the kernel runs perfectly at all times on all machines which 460 you can lay your hands on". Usually the way to do this is to work 461 with others on getting things fixed up (this can require 462 persistence!) but that's fine - it's a part of kernel development. 463 464(http://lwn.net/Articles/283982/). 465 466In the absence of obvious problems to fix, developers are advised to look 467at the current lists of regressions and open bugs in general. There is 468never any shortage of issues in need of fixing; by addressing these issues, 469developers will gain experience with the process while, at the same time, 470building respect with the rest of the development community. 471