1.. _embargoed_hardware_issues: 2 3Embargoed hardware issues 4========================= 5 6Scope 7----- 8 9Hardware issues which result in security problems are a different category 10of security bugs than pure software bugs which only affect the Linux 11kernel. 12 13Hardware issues like Meltdown, Spectre, L1TF etc. must be treated 14differently because they usually affect all Operating Systems ("OS") and 15therefore need coordination across different OS vendors, distributions, 16hardware vendors and other parties. For some of the issues, software 17mitigations can depend on microcode or firmware updates, which need further 18coordination. 19 20.. _Contact: 21 22Contact 23------- 24 25The Linux kernel hardware security team is separate from the regular Linux 26kernel security team. 27 28The team only handles developing fixes for embargoed hardware security 29issues. Reports of pure software security bugs in the Linux kernel are not 30handled by this team and the reporter will be guided to contact the regular 31Linux kernel security team (:ref:`Documentation/admin-guide/ 32<securitybugs>`) instead. 33 34The team can be contacted by email at <hardware-security@kernel.org>. This 35is a private list of security officers who will help you to coordinate a 36fix according to our documented process. 37 38The list is encrypted and email to the list can be sent by either PGP or 39S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME 40certificate. The list's PGP key and S/MIME certificate are available from 41the following URLs: 42 43 - PGP: https://www.kernel.org/static/files/hardware-security.asc 44 - S/MIME: https://www.kernel.org/static/files/hardware-security.crt 45 46While hardware security issues are often handled by the affected hardware 47vendor, we welcome contact from researchers or individuals who have 48identified a potential hardware flaw. 49 50Hardware security officers 51^^^^^^^^^^^^^^^^^^^^^^^^^^ 52 53The current team of hardware security officers: 54 55 - Linus Torvalds (Linux Foundation Fellow) 56 - Greg Kroah-Hartman (Linux Foundation Fellow) 57 - Thomas Gleixner (Linux Foundation Fellow) 58 59Operation of mailing-lists 60^^^^^^^^^^^^^^^^^^^^^^^^^^ 61 62The encrypted mailing-lists which are used in our process are hosted on 63Linux Foundation's IT infrastructure. By providing this service, members 64of Linux Foundation's IT operations personnel technically have the 65ability to access the embargoed information, but are obliged to 66confidentiality by their employment contract. Linux Foundation IT 67personnel are also responsible for operating and managing the rest of 68kernel.org infrastructure. 69 70The Linux Foundation's current director of IT Project infrastructure is 71Konstantin Ryabitsev. 72 73 74Non-disclosure agreements 75------------------------- 76 77The Linux kernel hardware security team is not a formal body and therefore 78unable to enter into any non-disclosure agreements. The kernel community 79is aware of the sensitive nature of such issues and offers a Memorandum of 80Understanding instead. 81 82 83Memorandum of Understanding 84--------------------------- 85 86The Linux kernel community has a deep understanding of the requirement to 87keep hardware security issues under embargo for coordination between 88different OS vendors, distributors, hardware vendors and other parties. 89 90The Linux kernel community has successfully handled hardware security 91issues in the past and has the necessary mechanisms in place to allow 92community compliant development under embargo restrictions. 93 94The Linux kernel community has a dedicated hardware security team for 95initial contact, which oversees the process of handling such issues under 96embargo rules. 97 98The hardware security team identifies the developers (domain experts) who 99will form the initial response team for a particular issue. The initial 100response team can bring in further developers (domain experts) to address 101the issue in the best technical way. 102 103All involved developers pledge to adhere to the embargo rules and to keep 104the received information confidential. Violation of the pledge will lead to 105immediate exclusion from the current issue and removal from all related 106mailing-lists. In addition, the hardware security team will also exclude 107the offender from future issues. The impact of this consequence is a highly 108effective deterrent in our community. In case a violation happens the 109hardware security team will inform the involved parties immediately. If you 110or anyone becomes aware of a potential violation, please report it 111immediately to the Hardware security officers. 112 113 114Process 115^^^^^^^ 116 117Due to the globally distributed nature of Linux kernel development, 118face-to-face meetings are almost impossible to address hardware security 119issues. Phone conferences are hard to coordinate due to time zones and 120other factors and should be only used when absolutely necessary. Encrypted 121email has been proven to be the most effective and secure communication 122method for these types of issues. 123 124Start of Disclosure 125""""""""""""""""""" 126 127Disclosure starts by contacting the Linux kernel hardware security team by 128email. This initial contact should contain a description of the problem and 129a list of any known affected hardware. If your organization builds or 130distributes the affected hardware, we encourage you to also consider what 131other hardware could be affected. 132 133The hardware security team will provide an incident-specific encrypted 134mailing-list which will be used for initial discussion with the reporter, 135further disclosure, and coordination of fixes. 136 137The hardware security team will provide the disclosing party a list of 138developers (domain experts) who should be informed initially about the 139issue after confirming with the developers that they will adhere to this 140Memorandum of Understanding and the documented process. These developers 141form the initial response team and will be responsible for handling the 142issue after initial contact. The hardware security team is supporting the 143response team, but is not necessarily involved in the mitigation 144development process. 145 146While individual developers might be covered by a non-disclosure agreement 147via their employer, they cannot enter individual non-disclosure agreements 148in their role as Linux kernel developers. They will, however, agree to 149adhere to this documented process and the Memorandum of Understanding. 150 151The disclosing party should provide a list of contacts for all other 152entities who have already been, or should be, informed about the issue. 153This serves several purposes: 154 155 - The list of disclosed entities allows communication across the 156 industry, e.g. other OS vendors, HW vendors, etc. 157 158 - The disclosed entities can be contacted to name experts who should 159 participate in the mitigation development. 160 161 - If an expert which is required to handle an issue is employed by an 162 listed entity or member of an listed entity, then the response teams can 163 request the disclosure of that expert from that entity. This ensures 164 that the expert is also part of the entity's response team. 165 166Disclosure 167"""""""""" 168 169The disclosing party provides detailed information to the initial response 170team via the specific encrypted mailing-list. 171 172From our experience the technical documentation of these issues is usually 173a sufficient starting point and further technical clarification is best 174done via email. 175 176Mitigation development 177"""""""""""""""""""""" 178 179The initial response team sets up an encrypted mailing-list or repurposes 180an existing one if appropriate. 181 182Using a mailing-list is close to the normal Linux development process and 183has been successfully used in developing mitigations for various hardware 184security issues in the past. 185 186The mailing-list operates in the same way as normal Linux development. 187Patches are posted, discussed and reviewed and if agreed on applied to a 188non-public git repository which is only accessible to the participating 189developers via a secure connection. The repository contains the main 190development branch against the mainline kernel and backport branches for 191stable kernel versions as necessary. 192 193The initial response team will identify further experts from the Linux 194kernel developer community as needed. Bringing in experts can happen at any 195time of the development process and needs to be handled in a timely manner. 196 197If an expert is employed by or member of an entity on the disclosure list 198provided by the disclosing party, then participation will be requested from 199the relevant entity. 200 201If not, then the disclosing party will be informed about the experts 202participation. The experts are covered by the Memorandum of Understanding 203and the disclosing party is requested to acknowledge the participation. In 204case that the disclosing party has a compelling reason to object, then this 205objection has to be raised within five work days and resolved with the 206incident team immediately. If the disclosing party does not react within 207five work days this is taken as silent acknowledgement. 208 209After acknowledgement or resolution of an objection the expert is disclosed 210by the incident team and brought into the development process. 211 212List participants may not communicate about the issue outside of the 213private mailing list. List participants may not use any shared resources 214(e.g. employer build farms, CI systems, etc) when working on patches. 215 216 217Coordinated release 218""""""""""""""""""" 219 220The involved parties will negotiate the date and time where the embargo 221ends. At that point the prepared mitigations are integrated into the 222relevant kernel trees and published. There is no pre-notification process: 223fixes are published in public and available to everyone at the same time. 224 225While we understand that hardware security issues need coordinated embargo 226time, the embargo time should be constrained to the minimum time which is 227required for all involved parties to develop, test and prepare the 228mitigations. Extending embargo time artificially to meet conference talk 229dates or other non-technical reasons is creating more work and burden for 230the involved developers and response teams as the patches need to be kept 231up to date in order to follow the ongoing upstream kernel development, 232which might create conflicting changes. 233 234CVE assignment 235"""""""""""""" 236 237Neither the hardware security team nor the initial response team assign 238CVEs, nor are CVEs required for the development process. If CVEs are 239provided by the disclosing party they can be used for documentation 240purposes. 241 242Process ambassadors 243------------------- 244 245For assistance with this process we have established ambassadors in various 246organizations, who can answer questions about or provide guidance on the 247reporting process and further handling. Ambassadors are not involved in the 248disclosure of a particular issue, unless requested by a response team or by 249an involved disclosed party. The current ambassadors list: 250 251 ============= ======================================================== 252 AMD Tom Lendacky <thomas.lendacky@amd.com> 253 Ampere Darren Hart <darren@os.amperecomputing.com> 254 ARM Catalin Marinas <catalin.marinas@arm.com> 255 IBM Power Michael Ellerman <ellerman@au.ibm.com> 256 IBM Z Christian Borntraeger <borntraeger@de.ibm.com> 257 Intel Tony Luck <tony.luck@intel.com> 258 Qualcomm Trilok Soni <quic_tsoni@quicinc.com> 259 RISC-V Palmer Dabbelt <palmer@dabbelt.com> 260 Samsung Javier Gonz��lez <javier.gonz@samsung.com> 261 262 Microsoft James Morris <jamorris@linux.microsoft.com> 263 Xen Andrew Cooper <andrew.cooper3@citrix.com> 264 265 Canonical John Johansen <john.johansen@canonical.com> 266 Debian Ben Hutchings <ben@decadent.org.uk> 267 Oracle Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> 268 Red Hat Josh Poimboeuf <jpoimboe@redhat.com> 269 SUSE Jiri Kosina <jkosina@suse.cz> 270 271 Google Kees Cook <keescook@chromium.org> 272 273 LLVM Nick Desaulniers <ndesaulniers@google.com> 274 ============= ======================================================== 275 276If you want your organization to be added to the ambassadors list, please 277contact the hardware security team. The nominated ambassador has to 278understand and support our process fully and is ideally well connected in 279the Linux kernel community. 280 281Encrypted mailing-lists 282----------------------- 283 284We use encrypted mailing-lists for communication. The operating principle 285of these lists is that email sent to the list is encrypted either with the 286list's PGP key or with the list's S/MIME certificate. The mailing-list 287software decrypts the email and re-encrypts it individually for each 288subscriber with the subscriber's PGP key or S/MIME certificate. Details 289about the mailing-list software and the setup which is used to ensure the 290security of the lists and protection of the data can be found here: 291https://korg.wiki.kernel.org/userdoc/remail. 292 293List keys 294^^^^^^^^^ 295 296For initial contact see :ref:`Contact`. For incident specific mailing-lists 297the key and S/MIME certificate are conveyed to the subscribers by email 298sent from the specific list. 299 300Subscription to incident specific lists 301^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 302 303Subscription is handled by the response teams. Disclosed parties who want 304to participate in the communication send a list of potential subscribers to 305the response team so the response team can validate subscription requests. 306 307Each subscriber needs to send a subscription request to the response team 308by email. The email must be signed with the subscriber's PGP key or S/MIME 309certificate. If a PGP key is used, it must be available from a public key 310server and is ideally connected to the Linux kernel's PGP web of trust. See 311also: https://www.kernel.org/signature.html. 312 313The response team verifies that the subscriber request is valid and adds 314the subscriber to the list. After subscription the subscriber will receive 315email from the mailing-list which is signed either with the list's PGP key 316or the list's S/MIME certificate. The subscriber's email client can extract 317the PGP key or the S/MIME certificate from the signature so the subscriber 318can send encrypted email to the list. 319 320