[PATCH] Documentation/admin-guide: Embargoed hardware security issues

[PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

From: Thomas Gleixner <tglx@linutronix.de>

To address the requirements of embargoed hardware issues, like Meltdown,
Spectre, L1TF, etc. it is necessary to define and document a process for
handling embargoed hardware security issues.

Following the discussion at the maintainer summit 2018 in Edinburgh
(https://lwn.net/Articles/769417/) the volunteered people have worked
out a process and a Memorandum of Understanding.  The latter addresses
the fact that the Linux kernel community cannot sign NDAs for various
reasons.

The initial contact point for hardware security issues is different from
the regular kernel security contact to provide a known and neutral
interface for hardware vendors and researchers.  The initial primary
contact team is proposed to be staffed by Linux Foundation Fellows, who
are not associated to a vendor or a distribution and are well connected
in the industry as a whole.

The process is designed with the experience of the past incidents in
mind and tries to address the remaining gaps, so future (hopefully rare)
incidents can be handled more efficiently.  It won’t remove the fact,
that most of this has to be done behind closed doors, but it is set up
to avoid big bureaucratic hurdles for individual developers.

The process is solely for handling hardware security issues and cannot
be used for regular kernel (software only) security bugs.

The process can help with hardware companies who, and I quote, "[my
manager] doesn't want to bet his job on the list keeping things secret."
This despite numerous leaks directly from that company over the years,
and none ever so far from the kernel security team.  Cognitive
dissidence seems to be a requirement to be a "successful" manager.

To accelerate the adoption of this process, we introduce the concept of
ambassadors in participating companies.  The ambassadors are there to
guide people to comply with the process, but are not automatically
involved in the disclosure of a particular incident.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Jiri Kosina <jkosina@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 Documentation/admin-guide/embargoed-hardware-issues.rst |  281 ++++++++++++++++
 Documentation/admin-guide/index.rst                     |    1 
 2 files changed, 282 insertions(+)

Note, this document has gone through numerous reviews by a number of
kernel developers, developers at some of the Linux distros, as well as
all of the lawyers from almost all open source-related companies.  It's
been sitting on my local drive with no comments for a few months now,
and it's about time to get this out and merged properly.

If anyone has any final comments, please let me know.

If anyone from any company listed below wishes to add their name to the
document, please send a follow-on patch and I will be glad to add it to
the series.  I had a number of "I'll sign up" type comments from
different people, but I want something with a "s-o-b" to keep people on
the hook for this, so I did not add their name to the file without that.

thanks,

greg k-h



--- /dev/null
+++ b/Documentation/admin-guide/embargoed-hardware-issues.rst
@@ -0,0 +1,281 @@
+.. _embargoedhardwareissues:
+
+Embargoed hardware issues
+=========================
+
+Scope
+-----
+
+Hardware issues which result in security problems are a different category
+of security bugs than pure software bugs which  only affect the Linux
+kernel.
+
+Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
+differently because they usually affect all Operating Systems (“OS“) and
+therefore need coordination across different OS vendors, distributions,
+hardware vendors and other parties. For some of the issues, software
+mitigations can depend on microcode or firmware updates, which need further
+coordination.
+
+.. _Contact:
+
+Contact
+-------
+
+The Linux kernel hardware security team is separate from the regular Linux
+kernel security team.
+
+The team is only handling the coordination of embargoed hardware security
+issues. Reports of pure software security bugs in the Linux kernel are not
+handled by this team and the reporter will be guided to contact the regular
+Linux kernel security team (:ref:`Documentation/admin-guide/
+<securitybugs>`) instead.
+
+The team can be contacted by email at <hardware-security@kernel.org>. This
+is a private list of security officers who will help you to coordinate an
+issue according to our documented process.
+
+The list is encrypted and email to the list can be sent by either PGP or
+S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
+certificate. The list's PGP key and S/MIME certificate are available from
+https://www.kernel.org/....
+
+While hardware security issues are often handled by the affected hardware
+vendor, we welcome contact from researchers or individuals who identified a
+potential hardware flaw.
+
+Hardware security officers
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current team of hardware security officers:
+
+  - Linus Torvalds (Linux Foundation Fellow)
+  - Greg Kroah-Hartman (Linux Foundation Fellow)
+  - Thomas Gleixner (Linux Foundation Fellow)
+
+Operation of mailing-lists
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The encrypted mailing-lists which are used in our process are hosted on
+Linux Foundation's IT infrastructure. By providing this service Linux
+Foundation's director of IT Infrastructure security technically has the
+ability to access the embargoed information, but is obliged to
+confidentiality by his employment contract. Linux Foundation's director of
+IT Infrastructure security is also responsible for the kernel.org
+infrastructure.
+
+The Linux Foundation's current director of IT Infrastructure security is
+Konstantin Ryabitsev.
+
+
+Non-disclosure agreements
+-------------------------
+
+The Linux kernel hardware security team is not a formal body and therefore
+unable to enter into any non-disclosure agreements.  The kernel community
+is aware of the sensitive nature of such issues and offers a Memorandum of
+Understanding instead.
+
+
+Memorandum of Understanding
+---------------------------
+
+The Linux kernel community has a deep understanding of the requirement to
+keep hardware security issues under embargo for coordination between
+different OS vendors, distributors, hardware vendors and other parties.
+
+The Linux kernel community has successfully handled hardware security
+issues in the past and has the necessary mechanisms in place to allow
+community compliant development under embargo restrictions.
+
+The Linux kernel community has a dedicated hardware security team for
+initial contact, which oversees the process of handling such issues under
+embargo rules.
+
+The hardware security team identifies the developers (domain experts) which
+form the initial response team for a particular issue. The initial response
+team can bring in further developers (domain experts) to address the issue
+in the best technical way.
+
+All involved developers pledge to adhere to the embargo rules and to keep
+the received information confidential. Violation of the pledge will lead to
+immediate exclusion from the current issue and removal from all related
+mailing-lists. In addition, the hardware security team will also exclude
+the offender from future issues. The impact of this consequence is a highly
+effective deterrent in our community. In case a violation happens the
+hardware security team will inform the involved parties immediately. If you
+or anyone becomes aware of a potential violation, please report it
+immediately to the Hardware security officers.
+
+
+Process
+^^^^^^^
+
+Due to the globally distributed nature of Linux kernel development, face to
+face meetings are almost impossible to address hardware security issues.
+Phone conferences are hard to coordinate due to time zones and other
+factors and should be only used when absolutely necessary. Encrypted email
+has been proven to be the most effective and secure communication method
+for these types of issues.
+
+Start of Disclosure
+"""""""""""""""""""
+
+Disclosure starts by contacting the Linux kernel hardware security team by
+email. This initial contact should contain a description of the problem and
+a list of any known affected hardware. If your organization builds or
+distributes the affected hardware, we encourage you to also consider what
+other hardware could be affected.
+
+The hardware security team will provide a per incident specific encrypted
+mailing-list which will be used for initial discussion with the reporter,
+further disclosure and coordination.
+
+The hardware security team will provide the disclosing party a list of
+developers (domain experts) who should be informed initially about the
+issue after confirming with the developers  that they will adhere to this
+Memorandum of Understanding and the documented process. These developers
+form the initial response team and will be responsible for handling the
+issue after initial contact. The hardware security team is supporting the
+response team, but is not necessarily involved in the mitigation
+development process.
+
+While individual developers might be covered by a non-disclosure agreement
+via their employer, they cannot enter individual non-disclosure agreements
+in their role as Linux kernel developers. They will, however, adhere to
+this documented process and the Memorandum of Understanding.
+
+
+Disclosure
+""""""""""
+
+The disclosing party provides detailed information to the initial response
+team via the specific encrypted mailing-list.
+
+From our experience the technical documentation of these issues is usually
+a sufficient starting point and further technical clarification is best
+done via email.
+
+Mitigation development
+""""""""""""""""""""""
+
+The initial response team sets up an encrypted mailing-list or repurposes
+an existing one if appropriate. The disclosing party should provide a list
+of contacts for all other parties who have already been, or should be
+informed about the issue. The response team contacts these parties so they
+can name experts who should be subscribed to the mailing-list.
+
+Using a mailing-list is close to the normal Linux development process and
+has been successfully used in developing mitigations for various hardware
+security issues in the past.
+
+The mailing-list operates in the same way as normal Linux development.
+Patches are posted, discussed and reviewed and if agreed on applied to a
+non-public git repository which is only accessible to the participating
+developers via a secure connection. The repository contains the main
+development branch against the mainline kernel and backport branches for
+stable kernel versions as necessary.
+
+The initial response team will identify further experts from the Linux
+kernel developer community as needed and inform the disclosing party about
+their participation. Bringing in experts can happen at any time of the
+development process and often needs to be handled in a timely manner.
+
+Coordinated release
+"""""""""""""""""""
+
+The involved parties will negotiate the date and time where the embargo
+ends. At that point the prepared mitigations are integrated into the
+relevant kernel trees and published.
+
+While we understand that hardware security issues need coordinated embargo
+time, the embargo time should be constrained to the minimum time which is
+required for all involved parties to develop, test and prepare the
+mitigations. Extending embargo time artificially to meet conference talk
+dates or other non-technical reasons is creating more work and burden for
+the involved developers and response teams as the patches need to be kept
+up to date in order to follow the ongoing upstream kernel development,
+which might create conflicting changes.
+
+CVE assignment
+""""""""""""""
+
+Neither the hardware security team nor the initial response team assign
+CVEs, nor are CVEs required for the development process. If CVEs are
+provided by the disclosing party they can be used for documentation
+purposes.
+
+Process ambassadors
+-------------------
+
+For assistance with this process we have established ambassadors in various
+organizations, who can answer questions about or provide guidance on the
+reporting process and further handling. Ambassadors are not involved in the
+disclosure of a particular issue, unless requested by a response team or by
+an involved disclosed party. The current ambassadors list:
+
+  ============== ========================================================
+  ARM
+  AMD
+  IBM
+  Intel
+  Qualcomm
+
+  Microsoft
+  VMware
+  XEN
+
+  Canonical
+  Debian
+  Oracle
+  Redhat
+  Suse           Jiri Kosina <jkosina@suse.com>
+
+  Amazon
+  Google
+  ============== ========================================================
+
+If you want your organization to be added to the ambassadors list, please
+contact the hardware security team. The nominated ambassador has to
+understand and support our process fully and is ideally well connected in
+the Linux kernel community.
+
+Encrypted mailing-lists
+-----------------------
+
+We use encrypted mailing-lists for communication. The operating principle
+of these lists is that email sent to the list is encrypted either with the
+list's PGP key or with the list's S/MIME certificate. The mailing-list
+software decrypts the email and re-encrypts it individually for each
+subscriber with the subscriber's PGP key or S/MIME certificate. Details
+about the mailing-list software and the setup which is used to ensure the
+security of the lists and protection of the data can be found here:
+https://www.kernel.org/....
+
+List keys
+^^^^^^^^^
+
+For initial contact see :ref:`Contact`. For incident specific mailing-lists
+the key and S/MIME certificate are conveyed to the subscribers by email
+sent from the specific list.
+
+Subscription to incident specific lists
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Subscription is handled by the response teams. Disclosed parties who want
+to participate in the communication send a list of potential subscribers to
+the response team so the response team can validate subscription requests.
+
+Each subscriber needs to send a subscription request to the response team
+by email. The email must be signed with the subscriber's PGP key or S/MIME
+certificate. If a PGP key is used, it must be available from a public key
+server and is ideally connected to the Linux kernel's PGP web of trust. See
+also: https://www.kernel.org/signature.html.
+
+The response team verifies that the subscriber request is valid and adds
+the subscriber to the list. After subscription the subscriber will receive
+email from the mailing-list which is signed either with the list's PGP key
+or the list's S/MIME certificate. The subscriber's email client can extract
+the PGP key or the S/MIME certificate from the signature so the subscriber
+can send encrypted email to the list.
+
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -33,6 +33,7 @@ problems and bugs in particular.
 
    reporting-bugs
    security-bugs
+   embargoed-hardware-issues
    bug-hunting
    bug-bisect
    tainted-kernels

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Jonathan Corbet]

On Thu, 25 Jul 2019 15:01:13 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:

> From: Thomas Gleixner <tglx@linutronix.de>
> 
> To address the requirements of embargoed hardware issues, like Meltdown,
> Spectre, L1TF, etc. it is necessary to define and document a process for
> handling embargoed hardware security issues.
> 
> Following the discussion at the maintainer summit 2018 in Edinburgh
> (https://lwn.net/Articles/769417/) the volunteered people have worked
> out a process and a Memorandum of Understanding.  The latter addresses
> the fact that the Linux kernel community cannot sign NDAs for various
> reasons.
> 

[...]

>  Documentation/admin-guide/embargoed-hardware-issues.rst |  281 ++++++++++++++++
>  Documentation/admin-guide/index.rst                     |    1 
>  2 files changed, 282 insertions(+)

So I would argue that the admin guide (which is aimed at sysadmins) is the
wrong place for this document.  It's process information and is best placed
in the process manual (Documentation/process) IMO.  (Yes, I know
security-bugs.rst is in the admin guide; I remember there was a discussion
at the time and it ended up there, but I'm not really sure that's right
either).

> Note, this document has gone through numerous reviews by a number of
> kernel developers, developers at some of the Linux distros, as well as
> all of the lawyers from almost all open source-related companies.  It's
> been sitting on my local drive with no comments for a few months now,
> and it's about time to get this out and merged properly.
> 
> If anyone has any final comments, please let me know.

I do think it could benefit from a pass for basic language issues; I can do
that if such an effort would be welcome.

> If anyone from any company listed below wishes to add their name to the
> document, please send a follow-on patch and I will be glad to add it to
> the series.  I had a number of "I'll sign up" type comments from
> different people, but I want something with a "s-o-b" to keep people on
> the hook for this, so I did not add their name to the file without that.
> 
> thanks,
> 
> greg k-h
> 
> 
> 
> --- /dev/null
> +++ b/Documentation/admin-guide/embargoed-hardware-issues.rst
> @@ -0,0 +1,281 @@
> +.. _embargoedhardwareissues:

This label isn't used anywhere.

> +Embargoed hardware issues
> +=========================
> +
> +Scope
> +-----
> +
> +Hardware issues which result in security problems are a different category
> +of security bugs than pure software bugs which  only affect the Linux
> +kernel.
> +
> +Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
> +differently because they usually affect all Operating Systems (“OS“) and

Somebody may well complain about the "smart quotes" here; non-ascii stuff
has led to unhappiness in the past.

> +therefore need coordination across different OS vendors, distributions,
> +hardware vendors and other parties. For some of the issues, software
> +mitigations can depend on microcode or firmware updates, which need further
> +coordination.
> +
> +.. _Contact:
> +
> +Contact
> +-------
> +
> +The Linux kernel hardware security team is separate from the regular Linux
> +kernel security team.
> +
> +The team is only handling the coordination of embargoed hardware security

s/is only handling/only handles/

> +issues. Reports of pure software security bugs in the Linux kernel are not
> +handled by this team and the reporter will be guided to contact the regular
> +Linux kernel security team (:ref:`Documentation/admin-guide/
> +<securitybugs>`) instead.
> +
> +The team can be contacted by email at <hardware-security@kernel.org>. This
> +is a private list of security officers who will help you to coordinate an
> +issue according to our documented process.
> +
> +The list is encrypted and email to the list can be sent by either PGP or
> +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> +certificate. The list's PGP key and S/MIME certificate are available from
> +https://www.kernel.org/....

Somebody needs to fill in some dots there...:)

> +While hardware security issues are often handled by the affected hardware
> +vendor, we welcome contact from researchers or individuals who identified a

who *have* identified

> +potential hardware flaw.
> +
> +Hardware security officers
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The current team of hardware security officers:
> +
> +  - Linus Torvalds (Linux Foundation Fellow)
> +  - Greg Kroah-Hartman (Linux Foundation Fellow)
> +  - Thomas Gleixner (Linux Foundation Fellow)
> +
> +Operation of mailing-lists
> +^^^^^^^^^^^^^^^^^^^^^^^^^^

I would de-hyphenate "mailing list" throughout.  But that's me.

> +The encrypted mailing-lists which are used in our process are hosted on
> +Linux Foundation's IT infrastructure. By providing this service Linux
> +Foundation's director of IT Infrastructure security technically has the
> +ability to access the embargoed information, but is obliged to
> +confidentiality by his employment contract. Linux Foundation's director of
> +IT Infrastructure security is also responsible for the kernel.org
> +infrastructure.
> +
> +The Linux Foundation's current director of IT Infrastructure security is
> +Konstantin Ryabitsev.
> +
> +
> +Non-disclosure agreements
> +-------------------------
> +
> +The Linux kernel hardware security team is not a formal body and therefore
> +unable to enter into any non-disclosure agreements.  The kernel community
> +is aware of the sensitive nature of such issues and offers a Memorandum of
> +Understanding instead.
> +
> +
> +Memorandum of Understanding
> +---------------------------
> +
> +The Linux kernel community has a deep understanding of the requirement to
> +keep hardware security issues under embargo for coordination between
> +different OS vendors, distributors, hardware vendors and other parties.
> +
> +The Linux kernel community has successfully handled hardware security
> +issues in the past and has the necessary mechanisms in place to allow
> +community compliant development under embargo restrictions.
> +
> +The Linux kernel community has a dedicated hardware security team for
> +initial contact, which oversees the process of handling such issues under
> +embargo rules.
> +
> +The hardware security team identifies the developers (domain experts) which
> +form the initial response team for a particular issue. The initial response

s/which form/who will form/

> +team can bring in further developers (domain experts) to address the issue
> +in the best technical way.

Does the reporter get any say in who can be in this group?  That should
probably be made explicit either way.

> +All involved developers pledge to adhere to the embargo rules and to keep
> +the received information confidential. Violation of the pledge will lead to
> +immediate exclusion from the current issue and removal from all related
> +mailing-lists. In addition, the hardware security team will also exclude
> +the offender from future issues. The impact of this consequence is a highly
> +effective deterrent in our community. In case a violation happens the
> +hardware security team will inform the involved parties immediately. If you
> +or anyone becomes aware of a potential violation, please report it
> +immediately to the Hardware security officers.
> +
> +
> +Process
> +^^^^^^^
> +
> +Due to the globally distributed nature of Linux kernel development, face to
> +face meetings are almost impossible to address hardware security issues.

face-to-face

> +Phone conferences are hard to coordinate due to time zones and other
> +factors and should be only used when absolutely necessary. Encrypted email
> +has been proven to be the most effective and secure communication method
> +for these types of issues.
> +
> +Start of Disclosure
> +"""""""""""""""""""
> +
> +Disclosure starts by contacting the Linux kernel hardware security team by
> +email. This initial contact should contain a description of the problem and
> +a list of any known affected hardware. If your organization builds or
> +distributes the affected hardware, we encourage you to also consider what
> +other hardware could be affected.
> +
> +The hardware security team will provide a per incident specific encrypted

s/per incident specific/incident-specific/

> +mailing-list which will be used for initial discussion with the reporter,
> +further disclosure and coordination.
> +
> +The hardware security team will provide the disclosing party a list of
> +developers (domain experts) who should be informed initially about the
> +issue after confirming with the developers  that they will adhere to this
> +Memorandum of Understanding and the documented process. These developers
> +form the initial response team and will be responsible for handling the
> +issue after initial contact. The hardware security team is supporting the
> +response team, but is not necessarily involved in the mitigation
> +development process.

Again, "should be informed" is conditional, suggesting that the reporter
might have some sort of veto power.  But the actual policy is not clear.

> +While individual developers might be covered by a non-disclosure agreement
> +via their employer, they cannot enter individual non-disclosure agreements
> +in their role as Linux kernel developers. They will, however, adhere to
> +this documented process and the Memorandum of Understanding.

They will *agree to* adhere ...  We expect that actual adherence will be
the case but there is no way (even if an NDA were involved) to guarantee
that.

> +Disclosure
> +""""""""""
> +
> +The disclosing party provides detailed information to the initial response
> +team via the specific encrypted mailing-list.
> +
> +From our experience the technical documentation of these issues is usually
> +a sufficient starting point and further technical clarification is best
> +done via email.
> +
> +Mitigation development
> +""""""""""""""""""""""
> +
> +The initial response team sets up an encrypted mailing-list or repurposes
> +an existing one if appropriate. The disclosing party should provide a list
> +of contacts for all other parties who have already been, or should be
> +informed about the issue. The response team contacts these parties so they
> +can name experts who should be subscribed to the mailing-list.
> +
> +Using a mailing-list is close to the normal Linux development process and
> +has been successfully used in developing mitigations for various hardware
> +security issues in the past.
> +
> +The mailing-list operates in the same way as normal Linux development.
> +Patches are posted, discussed and reviewed and if agreed on applied to a
> +non-public git repository which is only accessible to the participating
> +developers via a secure connection. The repository contains the main
> +development branch against the mainline kernel and backport branches for
> +stable kernel versions as necessary.

Do we want to envision a KPTI-like situation where the mitigation can be
developed publicly?  Or perhaps just handle any such case if and when it
ever arises?

> +The initial response team will identify further experts from the Linux
> +kernel developer community as needed and inform the disclosing party about
> +their participation. Bringing in experts can happen at any time of the
> +development process and often needs to be handled in a timely manner.
> +
> +Coordinated release
> +"""""""""""""""""""
> +
> +The involved parties will negotiate the date and time where the embargo
> +ends. At that point the prepared mitigations are integrated into the
> +relevant kernel trees and published.
> +
> +While we understand that hardware security issues need coordinated embargo
> +time, the embargo time should be constrained to the minimum time which is
> +required for all involved parties to develop, test and prepare the
> +mitigations. Extending embargo time artificially to meet conference talk
> +dates or other non-technical reasons is creating more work and burden for
> +the involved developers and response teams as the patches need to be kept
> +up to date in order to follow the ongoing upstream kernel development,
> +which might create conflicting changes.
> +
> +CVE assignment
> +""""""""""""""
> +
> +Neither the hardware security team nor the initial response team assign
> +CVEs, nor are CVEs required for the development process. If CVEs are
> +provided by the disclosing party they can be used for documentation
> +purposes.
> +
> +Process ambassadors
> +-------------------
> +
> +For assistance with this process we have established ambassadors in various
> +organizations, who can answer questions about or provide guidance on the
> +reporting process and further handling. Ambassadors are not involved in the
> +disclosure of a particular issue, unless requested by a response team or by
> +an involved disclosed party. The current ambassadors list:
> +
> +  ============== ========================================================
> +  ARM
> +  AMD
> +  IBM
> +  Intel
> +  Qualcomm
> +
> +  Microsoft
> +  VMware
> +  XEN
> +
> +  Canonical
> +  Debian
> +  Oracle
> +  Redhat
> +  Suse           Jiri Kosina <jkosina@suse.com>
> +
> +  Amazon
> +  Google
> +  ============== ========================================================

Having companies without names seems a little weird.  Unless perhaps you
have people teed up to add their names in follow-on patches?

> +If you want your organization to be added to the ambassadors list, please
> +contact the hardware security team. The nominated ambassador has to
> +understand and support our process fully and is ideally well connected in
> +the Linux kernel community.
> +
> +Encrypted mailing-lists
> +-----------------------
> +
> +We use encrypted mailing-lists for communication. The operating principle
> +of these lists is that email sent to the list is encrypted either with the
> +list's PGP key or with the list's S/MIME certificate. The mailing-list
> +software decrypts the email and re-encrypts it individually for each
> +subscriber with the subscriber's PGP key or S/MIME certificate. Details
> +about the mailing-list software and the setup which is used to ensure the
> +security of the lists and protection of the data can be found here:
> +https://www.kernel.org/....

That URL is also in need of completion.

The topic of encrypted mailing lists is visited several times in this
document; I wonder if that could be coalesced somehow?

> +List keys
> +^^^^^^^^^
> +
> +For initial contact see :ref:`Contact`. For incident specific mailing-lists
> +the key and S/MIME certificate are conveyed to the subscribers by email
> +sent from the specific list.
> +
> +Subscription to incident specific lists
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Subscription is handled by the response teams. Disclosed parties who want
> +to participate in the communication send a list of potential subscribers to
> +the response team so the response team can validate subscription requests.
> +
> +Each subscriber needs to send a subscription request to the response team
> +by email. The email must be signed with the subscriber's PGP key or S/MIME
> +certificate. If a PGP key is used, it must be available from a public key
> +server and is ideally connected to the Linux kernel's PGP web of trust. See
> +also: https://www.kernel.org/signature.html.

The "public key server" thing isn't working quite as well as it was; does
this requirement need to be revisited?

> +The response team verifies that the subscriber request is valid and adds
> +the subscriber to the list. After subscription the subscriber will receive
> +email from the mailing-list which is signed either with the list's PGP key
> +or the list's S/MIME certificate. The subscriber's email client can extract
> +the PGP key or the S/MIME certificate from the signature so the subscriber
> +can send encrypted email to the list.
> +
> --- a/Documentation/admin-guide/index.rst
> +++ b/Documentation/admin-guide/index.rst
> @@ -33,6 +33,7 @@ problems and bugs in particular.
>  
>     reporting-bugs
>     security-bugs
> +   embargoed-hardware-issues
>     bug-hunting
>     bug-bisect
>     tainted-kernels

jon

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Thomas Gleixner]

On Thu, 25 Jul 2019, Jonathan Corbet wrote:
> > Note, this document has gone through numerous reviews by a number of
> > kernel developers, developers at some of the Linux distros, as well as
> > all of the lawyers from almost all open source-related companies.  It's
> > been sitting on my local drive with no comments for a few months now,
> > and it's about time to get this out and merged properly.
> > 
> > If anyone has any final comments, please let me know.
> 
> I do think it could benefit from a pass for basic language issues; I can do
> that if such an effort would be welcome.

Definitely!

> > +
> > +The list is encrypted and email to the list can be sent by either PGP or
> > +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> > +certificate. The list's PGP key and S/MIME certificate are available from
> > +https://www.kernel.org/....
> 
> Somebody needs to fill in some dots there...:)

Yes. I need to sort that out with Konstantin before the thing gets merged,
but we wanted to give it a wider audience in general.
 
> > +The hardware security team identifies the developers (domain experts) which
> > +form the initial response team for a particular issue. The initial response
> 
> s/which form/who will form/
> 
> > +team can bring in further developers (domain experts) to address the issue
> > +in the best technical way.
> 
> Does the reporter get any say in who can be in this group?  That should
> probably be made explicit either way.

See below.

> > +The hardware security team will provide the disclosing party a list of
> > +developers (domain experts) who should be informed initially about the
> > +issue after confirming with the developers  that they will adhere to this
> > +Memorandum of Understanding and the documented process. These developers
> > +form the initial response team and will be responsible for handling the
> > +issue after initial contact. The hardware security team is supporting the
> > +response team, but is not necessarily involved in the mitigation
> > +development process.
> 
> Again, "should be informed" is conditional, suggesting that the reporter
> might have some sort of veto power.  But the actual policy is not clear.

Yes and no. That's a tricky field. We surely need some agreement with the
reporter/owner, but of course we want as much freedom here as we can
get. The past issues were always a pain when we had the need to get a
particular expert into the group.

> > +While individual developers might be covered by a non-disclosure agreement
> > +via their employer, they cannot enter individual non-disclosure agreements
> > +in their role as Linux kernel developers. They will, however, adhere to
> > +this documented process and the Memorandum of Understanding.
> 
> They will *agree to* adhere ...  We expect that actual adherence will be
> the case but there is no way (even if an NDA were involved) to guarantee
> that.

Correct.
 
> > +Disclosure
> > +""""""""""
> > +
> > +The disclosing party provides detailed information to the initial response
> > +team via the specific encrypted mailing-list.
> > +
> > +From our experience the technical documentation of these issues is usually
> > +a sufficient starting point and further technical clarification is best
> > +done via email.
> > +
> > +Mitigation development
> > +""""""""""""""""""""""
> > +
> > +The initial response team sets up an encrypted mailing-list or repurposes
> > +an existing one if appropriate. The disclosing party should provide a list
> > +of contacts for all other parties who have already been, or should be
> > +informed about the issue. The response team contacts these parties so they
> > +can name experts who should be subscribed to the mailing-list.
> > +
> > +Using a mailing-list is close to the normal Linux development process and
> > +has been successfully used in developing mitigations for various hardware
> > +security issues in the past.
> > +
> > +The mailing-list operates in the same way as normal Linux development.
> > +Patches are posted, discussed and reviewed and if agreed on applied to a
> > +non-public git repository which is only accessible to the participating
> > +developers via a secure connection. The repository contains the main
> > +development branch against the mainline kernel and backport branches for
> > +stable kernel versions as necessary.
> 
> Do we want to envision a KPTI-like situation where the mitigation can be
> developed publicly?  Or perhaps just handle any such case if and when it
> ever arises?

Yes, we handle that when it happens which is hopefully never.

> > +Process ambassadors
> > +-------------------
> > +
> > +For assistance with this process we have established ambassadors in various
> > +organizations, who can answer questions about or provide guidance on the
> > +reporting process and further handling. Ambassadors are not involved in the
> > +disclosure of a particular issue, unless requested by a response team or by
> > +an involved disclosed party. The current ambassadors list:
> > +
> > +  ============== ========================================================
> > +  ARM
> > +  AMD
> > +  IBM
> > +  Intel
> > +  Qualcomm
> > +
> > +  Microsoft
> > +  VMware
> > +  XEN
> > +
> > +  Canonical
> > +  Debian
> > +  Oracle
> > +  Redhat
> > +  Suse           Jiri Kosina <jkosina@suse.com>
> > +
> > +  Amazon
> > +  Google
> > +  ============== ========================================================
> 
> Having companies without names seems a little weird.  Unless perhaps you
> have people teed up to add their names in follow-on patches?

We already talked to companies and the names should come forth before this
is finished.
 
> > +Encrypted mailing-lists
> > +-----------------------
> > +
> > +We use encrypted mailing-lists for communication. The operating principle
> > +of these lists is that email sent to the list is encrypted either with the
> > +list's PGP key or with the list's S/MIME certificate. The mailing-list
> > +software decrypts the email and re-encrypts it individually for each
> > +subscriber with the subscriber's PGP key or S/MIME certificate. Details
> > +about the mailing-list software and the setup which is used to ensure the
> > +security of the lists and protection of the data can be found here:
> > +https://www.kernel.org/....
> 
> That URL is also in need of completion.
> 
> The topic of encrypted mailing lists is visited several times in this
> document; I wonder if that could be coalesced somehow?

Suggestions welcome.

> > +Each subscriber needs to send a subscription request to the response team
> > +by email. The email must be signed with the subscriber's PGP key or S/MIME
> > +certificate. If a PGP key is used, it must be available from a public key
> > +server and is ideally connected to the Linux kernel's PGP web of trust. See
> > +also: https://www.kernel.org/signature.html.
> 
> The "public key server" thing isn't working quite as well as it was; does
> this requirement need to be revisited?

I think so. That was written way before that mess happened.
 
Thanks,

	tglx

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Willy Tarreau]

Hi Greg, Thomas,

On Thu, Jul 25, 2019 at 03:01:13PM +0200, Greg Kroah-Hartman wrote:
> +The list is encrypted and email to the list can be sent by either PGP or
> +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> +certificate. The list's PGP key and S/MIME certificate are available from
> +https://www.kernel.org/....

Just thinking, wouldn't it be useful to strongly encourage that the
document should be in plain text format ? Otherwise the door remains open
for sending you a self-extractable EXE file which contains an encrypted
Word doc, which is not the most useful to handle especially to copy-paste
mitigation code nor to comment on. Even some occasional PDFs we've seen
on the sec@k.o list were sometimes quite detailed but less convenient
than the vast majority of plain text ones, particularly when it comes
to quoting some parts.

Just my two cents,
Willy

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Fri, Aug 02, 2019 at 06:49:08AM +0200, Willy Tarreau wrote:
> Hi Greg, Thomas,
> 
> On Thu, Jul 25, 2019 at 03:01:13PM +0200, Greg Kroah-Hartman wrote:
> > +The list is encrypted and email to the list can be sent by either PGP or
> > +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> > +certificate. The list's PGP key and S/MIME certificate are available from
> > +https://www.kernel.org/....
> 
> Just thinking, wouldn't it be useful to strongly encourage that the
> document should be in plain text format ? Otherwise the door remains open
> for sending you a self-extractable EXE file which contains an encrypted
> Word doc, which is not the most useful to handle especially to copy-paste
> mitigation code nor to comment on. Even some occasional PDFs we've seen
> on the sec@k.o list were sometimes quite detailed but less convenient
> than the vast majority of plain text ones, particularly when it comes
> to quoting some parts.

What document are you referring to here?  This just describes how the
encrypted mailing list is going to work, not anything else.

But yes, we have had some "encrypted pdfs" be sent to us recently that
no one can decrypt unless they run Windows or do some really crazy hacks
with the gstreamer pipeline.  But that's separate from this specific
mailing list, we can always just tell people to not do foolish things if
that happens again (like we did in this case.)

thanks,

greg k-h

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Willy Tarreau]

On Fri, Aug 02, 2019 at 08:57:29AM +0200, Greg Kroah-Hartman wrote:
> On Fri, Aug 02, 2019 at 06:49:08AM +0200, Willy Tarreau wrote:
> > Hi Greg, Thomas,
> > 
> > On Thu, Jul 25, 2019 at 03:01:13PM +0200, Greg Kroah-Hartman wrote:
> > > +The list is encrypted and email to the list can be sent by either PGP or
> > > +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> > > +certificate. The list's PGP key and S/MIME certificate are available from
> > > +https://www.kernel.org/....
> > 
> > Just thinking, wouldn't it be useful to strongly encourage that the
> > document should be in plain text format ? Otherwise the door remains open
> > for sending you a self-extractable EXE file which contains an encrypted
> > Word doc, which is not the most useful to handle especially to copy-paste
> > mitigation code nor to comment on. Even some occasional PDFs we've seen
> > on the sec@k.o list were sometimes quite detailed but less convenient
> > than the vast majority of plain text ones, particularly when it comes
> > to quoting some parts.
> 
> What document are you referring to here?  This just describes how the
> encrypted mailing list is going to work, not anything else.

I mean the document describing the issue that the reporter is going to
send to the mailing list.

> But yes, we have had some "encrypted pdfs" be sent to us recently that
> no one can decrypt unless they run Windows or do some really crazy hacks
> with the gstreamer pipeline.  But that's separate from this specific
> mailing list, we can always just tell people to not do foolish things if
> that happens again (like we did in this case.)

That was exactly my point :-)  Just like the process indicates what list
to contact to report an issue, it can also indicate the preferred way to
efficiently report these issues.

Willy

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Jiri Kosina]

On Thu, 25 Jul 2019, Greg Kroah-Hartman wrote:

> To address the requirements of embargoed hardware issues, like Meltdown,
> Spectre, L1TF, etc. it is necessary to define and document a process for
> handling embargoed hardware security issues.

I don't know what exactly went wrong, but there is a much more up-to-date 
version of that document (especially when it comes to vendor contacts), 
which I sent around on Thu, 2 May 2019 20:23:48 +0200 (CEST) already. 
Please find it below.



From: Jiri Kosina <jkosina@suse.cz>
Subject: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

To address the requirements of embargoed hardware issues, like Meltdown, 
Spectre, L1TF etc. it is necessary to define and document a process for 
handling embargoed hardware security issues.

Following the discussion at the maintainer summit 2018 in Edinburgh
(https://lwn.net/Articles/769417/) the volunteered people have worked
out a process and a Memorandum of Understanding. The latter addresses
the fact that the Linux kernel community cannot sign NDAs for various
reasons.

The initial contact point for hardware security issues is different from
the regular kernel security contact to provide a known and neutral
interface for hardware vendors and researchers. The initial primary
contact team is proposed to be staffed by Linux Foundation Fellows, who
are not associated to a vendor or a distribution and are well connected
in the industry as a whole.

The process is designed with the experience of the past incidents in mind 
and tries to address the remaining gaps, so future (hopefully rare) 
incidents can be handled more efficiently. It won't remove the fact, that 
most of this has to be done behind closed doors, but it is set up to avoid 
big bureaucratic hurdles for individual developers.

The process is solely for handling hardware security  issues and cannot
be used for regular kernel (software only) security bugs.

To accelerate the adoption of this  process, we introduce the concept of
ambassadors in participating companies. The ambassadors are there to
guide people to comply with the process, but are not automatically
involved in the disclosure of a particular incident.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Acked-by: Laura Abbott <labbott@redhat.com>
Acked-by: Ben Hutchings <ben@decadent.org.uk>
Reviewed-by: Tyler Hicks <tyhicks@canonical.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Jiri Kosina <jkosina@suse.cz>
---

v6 -> v7: added contacts (and Acks/Reviewed-bys) for distro people
	  fixed spelling of Red Hat
	  fixed spelling of SUSE
v5 -> v6: legal review and minor wording and line-wrapping changes
          Fixed Jiri's email address
V4 -> V5: Fix the last bits (LF and space/tab)
V3 -> V4: Addressed review comments
          Added changelog
          Added Google and Amazon to the ambassador list. Is there
          any company missing?


 .../admin-guide/embargoed-hardware-issues.rst      | 281 +++++++++++++++++++++
 Documentation/admin-guide/index.rst                |   1 +
 2 files changed, 282 insertions(+)
 create mode 100644 Documentation/admin-guide/embargoed-hardware-issues.rst

diff --git a/Documentation/admin-guide/embargoed-hardware-issues.rst b/Documentation/admin-guide/embargoed-hardware-issues.rst
new file mode 100644
index 000000000000..0bc4d01e13dd
--- /dev/null
+++ b/Documentation/admin-guide/embargoed-hardware-issues.rst
@@ -0,0 +1,281 @@
+.. _embargoedhardwareissues:
+
+Embargoed hardware issues
+=========================
+
+Scope
+-----
+
+Hardware issues which result in security problems are a different category
+of security bugs than pure software bugs which  only affect the Linux
+kernel.
+
+Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
+differently because they usually affect all Operating Systems (???OS???) and
+therefore need coordination across different OS vendors, distributions,
+hardware vendors and other parties. For some of the issues, software
+mitigations can depend on microcode or firmware updates, which need further
+coordination.
+
+.. _Contact:
+
+Contact
+-------
+
+The Linux kernel hardware security team is separate from the regular Linux
+kernel security team.
+
+The team is only handling the coordination of embargoed hardware security
+issues. Reports of pure software security bugs in the Linux kernel are not
+handled by this team and the reporter will be guided to contact the regular
+Linux kernel security team (:ref:`Documentation/admin-guide/
+<securitybugs>`) instead.
+
+The team can be contacted by email at <hardware-security@kernel.org>. This
+is a private list of security officers who will help you to coordinate an
+issue according to our documented process.
+
+The list is encrypted and email to the list can be sent by either PGP or
+S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
+certificate. The list's PGP key and S/MIME certificate are available from
+https://www.kernel.org/....
+
+While hardware security issues are often handled by the affected hardware
+vendor, we welcome contact from researchers or individuals who identified a
+potential hardware flaw.
+
+Hardware security officers
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current team of hardware security officers:
+
+  - Linus Torvalds (Linux Foundation Fellow)
+  - Greg Kroah-Hartman (Linux Foundation Fellow)
+  - Thomas Gleixner (Linux Foundation Fellow)
+
+Operation of mailing-lists
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The encrypted mailing-lists which are used in our process are hosted on
+Linux Foundation's IT infrastructure. By providing this service Linux
+Foundation's director of IT Infrastructure security technically has the
+ability to access the embargoed information, but is obliged to
+confidentiality by his employment contract. Linux Foundation's director of
+IT Infrastructure security is also responsible for the kernel.org
+infrastructure.
+
+The Linux Foundation's current director of IT Infrastructure security is
+Konstantin Ryabitsev.
+
+
+Non-disclosure agreements
+-------------------------
+
+The Linux kernel hardware security team is not a formal body and therefore
+unable to enter into any non-disclosure agreements.  The kernel community
+is aware of the sensitive nature of such issues and offers a Memorandum of
+Understanding instead.
+
+
+Memorandum of Understanding
+---------------------------
+
+The Linux kernel community has a deep understanding of the requirement to
+keep hardware security issues under embargo for coordination between
+different OS vendors, distributors, hardware vendors and other parties.
+
+The Linux kernel community has successfully handled hardware security
+issues in the past and has the necessary mechanisms in place to allow
+community compliant development under embargo restrictions.
+
+The Linux kernel community has a dedicated hardware security team for
+initial contact, which oversees the process of handling such issues under
+embargo rules.
+
+The hardware security team identifies the developers (domain experts) which
+form the initial response team for a particular issue. The initial response
+team can bring in further developers (domain experts) to address the issue
+in the best technical way.
+
+All involved developers pledge to adhere to the embargo rules and to keep
+the received information confidential. Violation of the pledge will lead to
+immediate exclusion from the current issue and removal from all related
+mailing-lists. In addition, the hardware security team will also exclude
+the offender from future issues. The impact of this consequence is a highly
+effective deterrent in our community. In case a violation happens the
+hardware security team will inform the involved parties immediately. If you
+or anyone becomes aware of a potential violation, please report it
+immediately to the Hardware security officers.
+
+
+Process
+^^^^^^^
+
+Due to the globally distributed nature of Linux kernel development, face to
+face meetings are almost impossible to address hardware security issues.
+Phone conferences are hard to coordinate due to time zones and other
+factors and should be only used when absolutely necessary. Encrypted email
+has been proven to be the most effective and secure communication method
+for these types of issues.
+
+Start of Disclosure
+"""""""""""""""""""
+
+Disclosure starts by contacting the Linux kernel hardware security team by
+email. This initial contact should contain a description of the problem and
+a list of any known affected hardware. If your organization builds or
+distributes the affected hardware, we encourage you to also consider what
+other hardware could be affected.
+
+The hardware security team will provide a per incident specific encrypted
+mailing-list which will be used for initial discussion with the reporter,
+further disclosure and coordination.
+
+The hardware security team will provide the disclosing party a list of
+developers (domain experts) who should be informed initially about the
+issue after confirming with the developers  that they will adhere to this
+Memorandum of Understanding and the documented process. These developers
+form the initial response team and will be responsible for handling the
+issue after initial contact. The hardware security team is supporting the
+response team, but is not necessarily involved in the mitigation
+development process.
+
+While individual developers might be covered by a non-disclosure agreement
+via their employer, they cannot enter individual non-disclosure agreements
+in their role as Linux kernel developers. They will, however, adhere to
+this documented process and the Memorandum of Understanding.
+
+
+Disclosure
+""""""""""
+
+The disclosing party provides detailed information to the initial response
+team via the specific encrypted mailing-list.
+
+From our experience the technical documentation of these issues is usually
+a sufficient starting point and further technical clarification is best
+done via email.
+
+Mitigation development
+""""""""""""""""""""""
+
+The initial response team sets up an encrypted mailing-list or repurposes
+an existing one if appropriate. The disclosing party should provide a list
+of contacts for all other parties who have already been, or should be
+informed about the issue. The response team contacts these parties so they
+can name experts who should be subscribed to the mailing-list.
+
+Using a mailing-list is close to the normal Linux development process and
+has been successfully used in developing mitigations for various hardware
+security issues in the past.
+
+The mailing-list operates in the same way as normal Linux development.
+Patches are posted, discussed and reviewed and if agreed on applied to a
+non-public git repository which is only accessible to the participating
+developers via a secure connection. The repository contains the main
+development branch against the mainline kernel and backport branches for
+stable kernel versions as necessary.
+
+The initial response team will identify further experts from the Linux
+kernel developer community as needed and inform the disclosing party about
+their participation. Bringing in experts can happen at any time of the
+development process and often needs to be handled in a timely manner.
+
+Coordinated release
+"""""""""""""""""""
+
+The involved parties will negotiate the date and time where the embargo
+ends. At that point the prepared mitigations are integrated into the
+relevant kernel trees and published.
+
+While we understand that hardware security issues need coordinated embargo
+time, the embargo time should be constrained to the minimum time which is
+required for all involved parties to develop, test and prepare the
+mitigations. Extending embargo time artificially to meet conference talk
+dates or other non-technical reasons is creating more work and burden for
+the involved developers and response teams as the patches need to be kept
+up to date in order to follow the ongoing upstream kernel development,
+which might create conflicting changes.
+
+CVE assignment
+""""""""""""""
+
+Neither the hardware security team nor the initial response team assign
+CVEs, nor are CVEs required for the development process. If CVEs are
+provided by the disclosing party they can be used for documentation
+purposes.
+
+Process ambassadors
+-------------------
+
+For assistance with this process we have established ambassadors in various
+organizations, who can answer questions about or provide guidance on the
+reporting process and further handling. Ambassadors are not involved in the
+disclosure of a particular issue, unless requested by a response team or by
+an involved disclosed party. The current ambassadors list:
+
+  ============= ========================================================
+  ARM
+  AMD
+  IBM
+  Intel
+  Qualcomm
+
+  Microsoft
+  VMware
+  XEN
+
+  Canonical	Tyler Hicks <tyhicks@canonical.com>
+  Debian	Ben Hutchings <ben@decadent.org.uk>
+  Oracle	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
+  Red Hat	Josh Poimboeuf <jpoimboe@redhat.com>
+  SUSE		Jiri Kosina <jkosina@suse.cz>
+
+  Amazon
+  Google
+  ============== ========================================================
+
+If you want your organization to be added to the ambassadors list, please
+contact the hardware security team. The nominated ambassador has to
+understand and support our process fully and is ideally well connected in
+the Linux kernel community.
+
+Encrypted mailing-lists
+-----------------------
+
+We use encrypted mailing-lists for communication. The operating principle
+of these lists is that email sent to the list is encrypted either with the
+list's PGP key or with the list's S/MIME certificate. The mailing-list
+software decrypts the email and re-encrypts it individually for each
+subscriber with the subscriber's PGP key or S/MIME certificate. Details
+about the mailing-list software and the setup which is used to ensure the
+security of the lists and protection of the data can be found here:
+https://www.kernel.org/....
+
+List keys
+^^^^^^^^^
+
+For initial contact see :ref:`Contact`. For incident specific mailing-lists
+the key and S/MIME certificate are conveyed to the subscribers by email
+sent from the specific list.
+
+Subscription to incident specific lists
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Subscription is handled by the response teams. Disclosed parties who want
+to participate in the communication send a list of potential subscribers to
+the response team so the response team can validate subscription requests.
+
+Each subscriber needs to send a subscription request to the response team
+by email. The email must be signed with the subscriber's PGP key or S/MIME
+certificate. If a PGP key is used, it must be available from a public key
+server and is ideally connected to the Linux kernel's PGP web of trust. See
+also: https://www.kernel.org/signature.html.
+
+The response team verifies that the subscriber request is valid and adds
+the subscriber to the list. After subscription the subscriber will receive
+email from the mailing-list which is signed either with the list's PGP key
+or the list's S/MIME certificate. The subscriber's email client can extract
+the PGP key or the S/MIME certificate from the signature so the subscriber
+can send encrypted email to the list.
+
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 0a491676685e..003585de3816 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -34,6 +34,7 @@ problems and bugs in particular.
 
    reporting-bugs
    security-bugs
+   embargoed-hardware-issues
    bug-hunting
    bug-bisect
    tainted-kernels

-- 
Jiri Kosina
SUSE Labs

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Jiri Kosina]

On Sun, 4 Aug 2019, Jiri Kosina wrote:

> On Thu, 25 Jul 2019, Greg Kroah-Hartman wrote:
> 
> > To address the requirements of embargoed hardware issues, like Meltdown,
> > Spectre, L1TF, etc. it is necessary to define and document a process for
> > handling embargoed hardware security issues.
> 
> I don't know what exactly went wrong, but there is a much more up-to-date 
> version of that document (especially when it comes to vendor contacts), 
> which I sent around on Thu, 2 May 2019 20:23:48 +0200 (CEST) already. 
> Please find it below.
> 
> 
> 
> From: Jiri Kosina <jkosina@suse.cz>

And this should've been

	From: Thomas Gleixner <tglx@linutronix.de>

as Thomas wrote most part of the text of course.

Sorry for the noise,

-- 
Jiri Kosina
SUSE Labs

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Eric W. Biederman]

I skimmed this and a couple things jumped out at me.

1) PGP and S/MIME because of their use of long term keys do not provide
   forward secrecy.  Which can makes it worth while to cryptographically
   factor a key or to obtain knowledge of a private key without the key
   holders knowledge.  As the keys will be used again and again over a
   long period of time.

   More recent protocol's such as Signal's Double Ratchet Protocol
   enable forward secrecy for store and foward communications, and
   remove the problem of long term keys.

2) The existence of such a process with encrypted communications to
   ensure long term confidentiality is going to make our contact people
   the targets of people who want access to knolwedge about hardware
   bugs like meltdown, before they become public.

I am just mentioning these things in case they are not immediately
obvious to everyone else involved, so that people can be certain
they are comfortable with the tradeoffs being made.

Eric

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Mon, Aug 05, 2019 at 09:40:21AM -0500, Eric W. Biederman wrote:
> 
> I skimmed this and a couple things jumped out at me.
> 
> 1) PGP and S/MIME because of their use of long term keys do not provide
>    forward secrecy.  Which can makes it worth while to cryptographically
>    factor a key or to obtain knowledge of a private key without the key
>    holders knowledge.  As the keys will be used again and again over a
>    long period of time.

Secrecy over a "long period of time" is not what is needed here.  6
months max is what I have seen, why would you need longer?

>    More recent protocol's such as Signal's Double Ratchet Protocol
>    enable forward secrecy for store and foward communications, and
>    remove the problem of long term keys.

And how does that work with email?  We need something that actually
works with a tool that everyone can use for development (i.e. email)

> 2) The existence of such a process with encrypted communications to
>    ensure long term confidentiality is going to make our contact people
>    the targets of people who want access to knolwedge about hardware
>    bugs like meltdown, before they become public.

Why are those same people not "targets" today?

And again, it's not long-term.

> I am just mentioning these things in case they are not immediately
> obvious to everyone else involved, so that people can be certain
> they are comfortable with the tradeoffs being made.

I know of no other thing that actually works (and lots of people can't
even get PGP to work as they use foolish email clients.)  Do you?

thanks,

greg k-h

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Sun, Aug 04, 2019 at 02:17:00AM +0200, Jiri Kosina wrote:
> On Thu, 25 Jul 2019, Greg Kroah-Hartman wrote:
> 
> > To address the requirements of embargoed hardware issues, like Meltdown,
> > Spectre, L1TF, etc. it is necessary to define and document a process for
> > handling embargoed hardware security issues.
> 
> I don't know what exactly went wrong, but there is a much more up-to-date 
> version of that document (especially when it comes to vendor contacts), 
> which I sent around on Thu, 2 May 2019 20:23:48 +0200 (CEST) already. 
> Please find it below.

Ah, sorry, don't know what happened here, we had too many different
versions floating around.

I'll take your version, make the edits recommended and send out a new
one in a few days, thanks!

greg k-h

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Mon, Aug 05, 2019 at 05:12:44PM +0200, Greg Kroah-Hartman wrote:
> On Sun, Aug 04, 2019 at 02:17:00AM +0200, Jiri Kosina wrote:
> > On Thu, 25 Jul 2019, Greg Kroah-Hartman wrote:
> > 
> > > To address the requirements of embargoed hardware issues, like Meltdown,
> > > Spectre, L1TF, etc. it is necessary to define and document a process for
> > > handling embargoed hardware security issues.
> > 
> > I don't know what exactly went wrong, but there is a much more up-to-date 
> > version of that document (especially when it comes to vendor contacts), 
> > which I sent around on Thu, 2 May 2019 20:23:48 +0200 (CEST) already. 
> > Please find it below.
> 
> Ah, sorry, don't know what happened here, we had too many different
> versions floating around.
> 
> I'll take your version, make the edits recommended and send out a new
> one in a few days, thanks!

Looks like your version only had the difference being the list of
ambassadors and a bunch of people who reviewed the document.  No
text changes in the document itself, which was good to see we all agreed
on the correct wording  :)

thanks,

greg k-h

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Thu, Jul 25, 2019 at 03:13:02PM -0600, Jonathan Corbet wrote:
> On Thu, 25 Jul 2019 15:01:13 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> 
> > From: Thomas Gleixner <tglx@linutronix.de>
> > 
> > To address the requirements of embargoed hardware issues, like Meltdown,
> > Spectre, L1TF, etc. it is necessary to define and document a process for
> > handling embargoed hardware security issues.
> > 
> > Following the discussion at the maintainer summit 2018 in Edinburgh
> > (https://lwn.net/Articles/769417/) the volunteered people have worked
> > out a process and a Memorandum of Understanding.  The latter addresses
> > the fact that the Linux kernel community cannot sign NDAs for various
> > reasons.
> > 
> 
> [...]
> 
> >  Documentation/admin-guide/embargoed-hardware-issues.rst |  281 ++++++++++++++++
> >  Documentation/admin-guide/index.rst                     |    1 
> >  2 files changed, 282 insertions(+)
> 
> So I would argue that the admin guide (which is aimed at sysadmins) is the
> wrong place for this document.  It's process information and is best placed
> in the process manual (Documentation/process) IMO.  (Yes, I know
> security-bugs.rst is in the admin guide; I remember there was a discussion
> at the time and it ended up there, but I'm not really sure that's right
> either).

Ok, will move it to the process directory.

> > Note, this document has gone through numerous reviews by a number of
> > kernel developers, developers at some of the Linux distros, as well as
> > all of the lawyers from almost all open source-related companies.  It's
> > been sitting on my local drive with no comments for a few months now,
> > and it's about time to get this out and merged properly.
> > 
> > If anyone has any final comments, please let me know.
> 
> I do think it could benefit from a pass for basic language issues; I can do
> that if such an effort would be welcome.

It would be!

> > If anyone from any company listed below wishes to add their name to the
> > document, please send a follow-on patch and I will be glad to add it to
> > the series.  I had a number of "I'll sign up" type comments from
> > different people, but I want something with a "s-o-b" to keep people on
> > the hook for this, so I did not add their name to the file without that.
> > 
> > thanks,
> > 
> > greg k-h
> > 
> > 
> > 
> > --- /dev/null
> > +++ b/Documentation/admin-guide/embargoed-hardware-issues.rst
> > @@ -0,0 +1,281 @@
> > +.. _embargoedhardwareissues:
> 
> This label isn't used anywhere.

Odd, ok, dropped.

> 
> > +Embargoed hardware issues
> > +=========================
> > +
> > +Scope
> > +-----
> > +
> > +Hardware issues which result in security problems are a different category
> > +of security bugs than pure software bugs which  only affect the Linux
> > +kernel.
> > +
> > +Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
> > +differently because they usually affect all Operating Systems (“OS“) and
> 
> Somebody may well complain about the "smart quotes" here; non-ascii stuff
> has led to unhappiness in the past.

Now fixed, thanks.

> > +therefore need coordination across different OS vendors, distributions,
> > +hardware vendors and other parties. For some of the issues, software
> > +mitigations can depend on microcode or firmware updates, which need further
> > +coordination.
> > +
> > +.. _Contact:
> > +
> > +Contact
> > +-------
> > +
> > +The Linux kernel hardware security team is separate from the regular Linux
> > +kernel security team.
> > +
> > +The team is only handling the coordination of embargoed hardware security
> 
> s/is only handling/only handles/

Fixed.

> > +issues. Reports of pure software security bugs in the Linux kernel are not
> > +handled by this team and the reporter will be guided to contact the regular
> > +Linux kernel security team (:ref:`Documentation/admin-guide/
> > +<securitybugs>`) instead.
> > +
> > +The team can be contacted by email at <hardware-security@kernel.org>. This
> > +is a private list of security officers who will help you to coordinate an
> > +issue according to our documented process.
> > +
> > +The list is encrypted and email to the list can be sent by either PGP or
> > +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> > +certificate. The list's PGP key and S/MIME certificate are available from
> > +https://www.kernel.org/....
> 
> Somebody needs to fill in some dots there...:)

Will work on that...

> > +While hardware security issues are often handled by the affected hardware
> > +vendor, we welcome contact from researchers or individuals who identified a
> 
> who *have* identified

fixed.

> > +potential hardware flaw.
> > +
> > +Hardware security officers
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +The current team of hardware security officers:
> > +
> > +  - Linus Torvalds (Linux Foundation Fellow)
> > +  - Greg Kroah-Hartman (Linux Foundation Fellow)
> > +  - Thomas Gleixner (Linux Foundation Fellow)
> > +
> > +Operation of mailing-lists
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> I would de-hyphenate "mailing list" throughout.  But that's me.

I'll let the original author have it his way :)

> > +The encrypted mailing-lists which are used in our process are hosted on
> > +Linux Foundation's IT infrastructure. By providing this service Linux
> > +Foundation's director of IT Infrastructure security technically has the
> > +ability to access the embargoed information, but is obliged to
> > +confidentiality by his employment contract. Linux Foundation's director of
> > +IT Infrastructure security is also responsible for the kernel.org
> > +infrastructure.
> > +
> > +The Linux Foundation's current director of IT Infrastructure security is
> > +Konstantin Ryabitsev.
> > +
> > +
> > +Non-disclosure agreements
> > +-------------------------
> > +
> > +The Linux kernel hardware security team is not a formal body and therefore
> > +unable to enter into any non-disclosure agreements.  The kernel community
> > +is aware of the sensitive nature of such issues and offers a Memorandum of
> > +Understanding instead.
> > +
> > +
> > +Memorandum of Understanding
> > +---------------------------
> > +
> > +The Linux kernel community has a deep understanding of the requirement to
> > +keep hardware security issues under embargo for coordination between
> > +different OS vendors, distributors, hardware vendors and other parties.
> > +
> > +The Linux kernel community has successfully handled hardware security
> > +issues in the past and has the necessary mechanisms in place to allow
> > +community compliant development under embargo restrictions.
> > +
> > +The Linux kernel community has a dedicated hardware security team for
> > +initial contact, which oversees the process of handling such issues under
> > +embargo rules.
> > +
> > +The hardware security team identifies the developers (domain experts) which
> > +form the initial response team for a particular issue. The initial response
> 
> s/which form/who will form/

fixed.

> > +team can bring in further developers (domain experts) to address the issue
> > +in the best technical way.
> 
> Does the reporter get any say in who can be in this group?  That should
> probably be made explicit either way.

That is discussed later in the document, and the reporter can offer up
people, but that's about it.  It's up to us running the list to be the
ones in charge of this, and that's one of the main reasons this document
exists.

I'll leave the wording as-is for now, given all of the lawyers who have
already agreed with it.  If we find people objecting to how it all is
written now in the future, we can revisit it.

> > +All involved developers pledge to adhere to the embargo rules and to keep
> > +the received information confidential. Violation of the pledge will lead to
> > +immediate exclusion from the current issue and removal from all related
> > +mailing-lists. In addition, the hardware security team will also exclude
> > +the offender from future issues. The impact of this consequence is a highly
> > +effective deterrent in our community. In case a violation happens the
> > +hardware security team will inform the involved parties immediately. If you
> > +or anyone becomes aware of a potential violation, please report it
> > +immediately to the Hardware security officers.
> > +
> > +
> > +Process
> > +^^^^^^^
> > +
> > +Due to the globally distributed nature of Linux kernel development, face to
> > +face meetings are almost impossible to address hardware security issues.
> 
> face-to-face

fixed.

> > +Phone conferences are hard to coordinate due to time zones and other
> > +factors and should be only used when absolutely necessary. Encrypted email
> > +has been proven to be the most effective and secure communication method
> > +for these types of issues.
> > +
> > +Start of Disclosure
> > +"""""""""""""""""""
> > +
> > +Disclosure starts by contacting the Linux kernel hardware security team by
> > +email. This initial contact should contain a description of the problem and
> > +a list of any known affected hardware. If your organization builds or
> > +distributes the affected hardware, we encourage you to also consider what
> > +other hardware could be affected.
> > +
> > +The hardware security team will provide a per incident specific encrypted
> 
> s/per incident specific/incident-specific/

Fixed.  And changed /a/ to /an/

> > +mailing-list which will be used for initial discussion with the reporter,
> > +further disclosure and coordination.
> > +
> > +The hardware security team will provide the disclosing party a list of
> > +developers (domain experts) who should be informed initially about the
> > +issue after confirming with the developers  that they will adhere to this
> > +Memorandum of Understanding and the documented process. These developers
> > +form the initial response team and will be responsible for handling the
> > +issue after initial contact. The hardware security team is supporting the
> > +response team, but is not necessarily involved in the mitigation
> > +development process.
> 
> Again, "should be informed" is conditional, suggesting that the reporter
> might have some sort of veto power.  But the actual policy is not clear.

Embrace the grey, it is much better here :)

> > +While individual developers might be covered by a non-disclosure agreement
> > +via their employer, they cannot enter individual non-disclosure agreements
> > +in their role as Linux kernel developers. They will, however, adhere to
> > +this documented process and the Memorandum of Understanding.
> 
> They will *agree to* adhere ...  We expect that actual adherence will be
> the case but there is no way (even if an NDA were involved) to guarantee
> that.

True, fixed up now.

> > +Disclosure
> > +""""""""""
> > +
> > +The disclosing party provides detailed information to the initial response
> > +team via the specific encrypted mailing-list.
> > +
> > +From our experience the technical documentation of these issues is usually
> > +a sufficient starting point and further technical clarification is best
> > +done via email.
> > +
> > +Mitigation development
> > +""""""""""""""""""""""
> > +
> > +The initial response team sets up an encrypted mailing-list or repurposes
> > +an existing one if appropriate. The disclosing party should provide a list
> > +of contacts for all other parties who have already been, or should be
> > +informed about the issue. The response team contacts these parties so they
> > +can name experts who should be subscribed to the mailing-list.
> > +
> > +Using a mailing-list is close to the normal Linux development process and
> > +has been successfully used in developing mitigations for various hardware
> > +security issues in the past.
> > +
> > +The mailing-list operates in the same way as normal Linux development.
> > +Patches are posted, discussed and reviewed and if agreed on applied to a
> > +non-public git repository which is only accessible to the participating
> > +developers via a secure connection. The repository contains the main
> > +development branch against the mainline kernel and backport branches for
> > +stable kernel versions as necessary.
> 
> Do we want to envision a KPTI-like situation where the mitigation can be
> developed publicly?  Or perhaps just handle any such case if and when it
> ever arises?

We can handle such cases if and when it arises.  And who's to say it
already hasn't?  :)

> > +The initial response team will identify further experts from the Linux
> > +kernel developer community as needed and inform the disclosing party about
> > +their participation. Bringing in experts can happen at any time of the
> > +development process and often needs to be handled in a timely manner.
> > +
> > +Coordinated release
> > +"""""""""""""""""""
> > +
> > +The involved parties will negotiate the date and time where the embargo
> > +ends. At that point the prepared mitigations are integrated into the
> > +relevant kernel trees and published.
> > +
> > +While we understand that hardware security issues need coordinated embargo
> > +time, the embargo time should be constrained to the minimum time which is
> > +required for all involved parties to develop, test and prepare the
> > +mitigations. Extending embargo time artificially to meet conference talk
> > +dates or other non-technical reasons is creating more work and burden for
> > +the involved developers and response teams as the patches need to be kept
> > +up to date in order to follow the ongoing upstream kernel development,
> > +which might create conflicting changes.
> > +
> > +CVE assignment
> > +""""""""""""""
> > +
> > +Neither the hardware security team nor the initial response team assign
> > +CVEs, nor are CVEs required for the development process. If CVEs are
> > +provided by the disclosing party they can be used for documentation
> > +purposes.
> > +
> > +Process ambassadors
> > +-------------------
> > +
> > +For assistance with this process we have established ambassadors in various
> > +organizations, who can answer questions about or provide guidance on the
> > +reporting process and further handling. Ambassadors are not involved in the
> > +disclosure of a particular issue, unless requested by a response team or by
> > +an involved disclosed party. The current ambassadors list:
> > +
> > +  ============== ========================================================
> > +  ARM
> > +  AMD
> > +  IBM
> > +  Intel
> > +  Qualcomm
> > +
> > +  Microsoft
> > +  VMware
> > +  XEN
> > +
> > +  Canonical
> > +  Debian
> > +  Oracle
> > +  Redhat
> > +  Suse           Jiri Kosina <jkosina@suse.com>
> > +
> > +  Amazon
> > +  Google
> > +  ============== ========================================================
> 
> Having companies without names seems a little weird.  Unless perhaps you
> have people teed up to add their names in follow-on patches?

Jiri posted an update with more names.

The blank lines are there for where we want names.  I have a bunch of
people who have agreed, but I want them sending add-on patches for the
document, with their s-o-b on it, so that everyone "knows" they really
agree with this, and that someone didn't just add their name to the
document without them realizing.

> > +If you want your organization to be added to the ambassadors list, please
> > +contact the hardware security team. The nominated ambassador has to
> > +understand and support our process fully and is ideally well connected in
> > +the Linux kernel community.
> > +
> > +Encrypted mailing-lists
> > +-----------------------
> > +
> > +We use encrypted mailing-lists for communication. The operating principle
> > +of these lists is that email sent to the list is encrypted either with the
> > +list's PGP key or with the list's S/MIME certificate. The mailing-list
> > +software decrypts the email and re-encrypts it individually for each
> > +subscriber with the subscriber's PGP key or S/MIME certificate. Details
> > +about the mailing-list software and the setup which is used to ensure the
> > +security of the lists and protection of the data can be found here:
> > +https://www.kernel.org/....
> 
> That URL is also in need of completion.

Yeah, we will work on that next...

> 
> The topic of encrypted mailing lists is visited several times in this
> document; I wonder if that could be coalesced somehow?

Edits are welcome :)

> > +List keys
> > +^^^^^^^^^
> > +
> > +For initial contact see :ref:`Contact`. For incident specific mailing-lists
> > +the key and S/MIME certificate are conveyed to the subscribers by email
> > +sent from the specific list.
> > +
> > +Subscription to incident specific lists
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Subscription is handled by the response teams. Disclosed parties who want
> > +to participate in the communication send a list of potential subscribers to
> > +the response team so the response team can validate subscription requests.
> > +
> > +Each subscriber needs to send a subscription request to the response team
> > +by email. The email must be signed with the subscriber's PGP key or S/MIME
> > +certificate. If a PGP key is used, it must be available from a public key
> > +server and is ideally connected to the Linux kernel's PGP web of trust. See
> > +also: https://www.kernel.org/signature.html.
> 
> The "public key server" thing isn't working quite as well as it was; does
> this requirement need to be revisited?

We have a private web right now, and it's the best we have at the
moment.  So until we get something else, let's stick with it.

Many thanks for the edits, I'll send out a new version soon.

greg k-h

[PATCH v2] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

From: Thomas Gleixner <tglx@linutronix.de>

To address the requirements of embargoed hardware issues, like Meltdown,
Spectre, L1TF etc. it is necessary to define and document a process for
handling embargoed hardware security issues.

Following the discussion at the maintainer summit 2018 in Edinburgh
(https://lwn.net/Articles/769417/) the volunteered people have worked
out a process and a Memorandum of Understanding.  The latter addresses
the fact that the Linux kernel community cannot sign NDAs for various
reasons.

The initial contact point for hardware security issues is different from
the regular kernel security contact to provide a known and neutral
interface for hardware vendors and researchers. The initial primary
contact team is proposed to be staffed by Linux Foundation Fellows, who
are not associated to a vendor or a distribution and are well connected
in the industry as a whole.

The process is designed with the experience of the past incidents in
mind and tries to address the remaining gaps, so future (hopefully rare)
incidents can be handled more efficiently.  It won't remove the fact,
that most of this has to be done behind closed doors, but it is set up
to avoid big bureaucratic hurdles for individual developers.

The process is solely for handling hardware security issues and cannot
be used for regular kernel (software only) security bugs.

This memo can help with hardware companies who, and I quote, "[my
manager] doesn't want to bet his job on the list keeping things secret."
This despite numerous leaks directly from that company over the years,
and none ever so far from the kernel security team.  Cognitive
dissidence seems to be a requirement to be a good manager.

To accelerate the adoption of this  process, we introduce the concept of
ambassadors in participating companies. The ambassadors are there to
guide people to comply with the process, but are not automatically
involved in the disclosure of a particular incident.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Acked-by: Laura Abbott <labbott@redhat.com>
Acked-by: Ben Hutchings <ben@decadent.org.uk>
Reviewed-by: Tyler Hicks <tyhicks@canonical.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Jiri Kosina <jkosina@suse.cz>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
v2: updated list of people with document from Jiri as I had the old one
    grammer tweaks based on Jon's review
    moved document to Documentation/process/

 Documentation/process/embargoed-hardware-issues.rst |  279 ++++++++++++++++++++
 Documentation/process/index.rst                     |    1 
 2 files changed, 280 insertions(+)

--- /dev/null
+++ b/Documentation/process/embargoed-hardware-issues.rst
@@ -0,0 +1,279 @@
+Embargoed hardware issues
+=========================
+
+Scope
+-----
+
+Hardware issues which result in security problems are a different category
+of security bugs than pure software bugs which  only affect the Linux
+kernel.
+
+Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
+differently because they usually affect all Operating Systems ("OS") and
+therefore need coordination across different OS vendors, distributions,
+hardware vendors and other parties. For some of the issues, software
+mitigations can depend on microcode or firmware updates, which need further
+coordination.
+
+.. _Contact:
+
+Contact
+-------
+
+The Linux kernel hardware security team is separate from the regular Linux
+kernel security team.
+
+The team only handles the coordination of embargoed hardware security
+issues.  Reports of pure software security bugs in the Linux kernel are not
+handled by this team and the reporter will be guided to contact the regular
+Linux kernel security team (:ref:`Documentation/admin-guide/
+<securitybugs>`) instead.
+
+The team can be contacted by email at <hardware-security@kernel.org>. This
+is a private list of security officers who will help you to coordinate an
+issue according to our documented process.
+
+The list is encrypted and email to the list can be sent by either PGP or
+S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
+certificate. The list's PGP key and S/MIME certificate are available from
+https://www.kernel.org/....
+
+While hardware security issues are often handled by the affected hardware
+vendor, we welcome contact from researchers or individuals who have
+identified a potential hardware flaw.
+
+Hardware security officers
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current team of hardware security officers:
+
+  - Linus Torvalds (Linux Foundation Fellow)
+  - Greg Kroah-Hartman (Linux Foundation Fellow)
+  - Thomas Gleixner (Linux Foundation Fellow)
+
+Operation of mailing-lists
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The encrypted mailing-lists which are used in our process are hosted on
+Linux Foundation's IT infrastructure. By providing this service Linux
+Foundation's director of IT Infrastructure security technically has the
+ability to access the embargoed information, but is obliged to
+confidentiality by his employment contract. Linux Foundation's director of
+IT Infrastructure security is also responsible for the kernel.org
+infrastructure.
+
+The Linux Foundation's current director of IT Infrastructure security is
+Konstantin Ryabitsev.
+
+
+Non-disclosure agreements
+-------------------------
+
+The Linux kernel hardware security team is not a formal body and therefore
+unable to enter into any non-disclosure agreements.  The kernel community
+is aware of the sensitive nature of such issues and offers a Memorandum of
+Understanding instead.
+
+
+Memorandum of Understanding
+---------------------------
+
+The Linux kernel community has a deep understanding of the requirement to
+keep hardware security issues under embargo for coordination between
+different OS vendors, distributors, hardware vendors and other parties.
+
+The Linux kernel community has successfully handled hardware security
+issues in the past and has the necessary mechanisms in place to allow
+community compliant development under embargo restrictions.
+
+The Linux kernel community has a dedicated hardware security team for
+initial contact, which oversees the process of handling such issues under
+embargo rules.
+
+The hardware security team identifies the developers (domain experts) who
+will form the initial response team for a particular issue. The initial
+response team can bring in further developers (domain experts) to address
+the issue in the best technical way.
+
+All involved developers pledge to adhere to the embargo rules and to keep
+the received information confidential. Violation of the pledge will lead to
+immediate exclusion from the current issue and removal from all related
+mailing-lists. In addition, the hardware security team will also exclude
+the offender from future issues. The impact of this consequence is a highly
+effective deterrent in our community. In case a violation happens the
+hardware security team will inform the involved parties immediately. If you
+or anyone becomes aware of a potential violation, please report it
+immediately to the Hardware security officers.
+
+
+Process
+^^^^^^^
+
+Due to the globally distributed nature of Linux kernel development,
+face-to-face meetings are almost impossible to address hardware security
+issues.  Phone conferences are hard to coordinate due to time zones and
+other factors and should be only used when absolutely necessary. Encrypted
+email has been proven to be the most effective and secure communication
+method for these types of issues.
+
+Start of Disclosure
+"""""""""""""""""""
+
+Disclosure starts by contacting the Linux kernel hardware security team by
+email. This initial contact should contain a description of the problem and
+a list of any known affected hardware. If your organization builds or
+distributes the affected hardware, we encourage you to also consider what
+other hardware could be affected.
+
+The hardware security team will provide an incident-specific encrypted
+mailing-list which will be used for initial discussion with the reporter,
+further disclosure and coordination.
+
+The hardware security team will provide the disclosing party a list of
+developers (domain experts) who should be informed initially about the
+issue after confirming with the developers  that they will adhere to this
+Memorandum of Understanding and the documented process. These developers
+form the initial response team and will be responsible for handling the
+issue after initial contact. The hardware security team is supporting the
+response team, but is not necessarily involved in the mitigation
+development process.
+
+While individual developers might be covered by a non-disclosure agreement
+via their employer, they cannot enter individual non-disclosure agreements
+in their role as Linux kernel developers. They will, however, agree to
+adhere to this documented process and the Memorandum of Understanding.
+
+
+Disclosure
+""""""""""
+
+The disclosing party provides detailed information to the initial response
+team via the specific encrypted mailing-list.
+
+From our experience the technical documentation of these issues is usually
+a sufficient starting point and further technical clarification is best
+done via email.
+
+Mitigation development
+""""""""""""""""""""""
+
+The initial response team sets up an encrypted mailing-list or repurposes
+an existing one if appropriate. The disclosing party should provide a list
+of contacts for all other parties who have already been, or should be
+informed about the issue. The response team contacts these parties so they
+can name experts who should be subscribed to the mailing-list.
+
+Using a mailing-list is close to the normal Linux development process and
+has been successfully used in developing mitigations for various hardware
+security issues in the past.
+
+The mailing-list operates in the same way as normal Linux development.
+Patches are posted, discussed and reviewed and if agreed on applied to a
+non-public git repository which is only accessible to the participating
+developers via a secure connection. The repository contains the main
+development branch against the mainline kernel and backport branches for
+stable kernel versions as necessary.
+
+The initial response team will identify further experts from the Linux
+kernel developer community as needed and inform the disclosing party about
+their participation. Bringing in experts can happen at any time of the
+development process and often needs to be handled in a timely manner.
+
+Coordinated release
+"""""""""""""""""""
+
+The involved parties will negotiate the date and time where the embargo
+ends. At that point the prepared mitigations are integrated into the
+relevant kernel trees and published.
+
+While we understand that hardware security issues need coordinated embargo
+time, the embargo time should be constrained to the minimum time which is
+required for all involved parties to develop, test and prepare the
+mitigations. Extending embargo time artificially to meet conference talk
+dates or other non-technical reasons is creating more work and burden for
+the involved developers and response teams as the patches need to be kept
+up to date in order to follow the ongoing upstream kernel development,
+which might create conflicting changes.
+
+CVE assignment
+""""""""""""""
+
+Neither the hardware security team nor the initial response team assign
+CVEs, nor are CVEs required for the development process. If CVEs are
+provided by the disclosing party they can be used for documentation
+purposes.
+
+Process ambassadors
+-------------------
+
+For assistance with this process we have established ambassadors in various
+organizations, who can answer questions about or provide guidance on the
+reporting process and further handling. Ambassadors are not involved in the
+disclosure of a particular issue, unless requested by a response team or by
+an involved disclosed party. The current ambassadors list:
+
+  ============= ========================================================
+  ARM
+  AMD
+  IBM
+  Intel
+  Qualcomm
+
+  Microsoft
+  VMware
+  XEN
+
+  Canonical	Tyler Hicks <tyhicks@canonical.com>
+  Debian	Ben Hutchings <ben@decadent.org.uk>
+  Oracle	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
+  Red Hat	Josh Poimboeuf <jpoimboe@redhat.com>
+  SUSE		Jiri Kosina <jkosina@suse.cz>
+
+  Amazon
+  Google
+  ============== ========================================================
+
+If you want your organization to be added to the ambassadors list, please
+contact the hardware security team. The nominated ambassador has to
+understand and support our process fully and is ideally well connected in
+the Linux kernel community.
+
+Encrypted mailing-lists
+-----------------------
+
+We use encrypted mailing-lists for communication. The operating principle
+of these lists is that email sent to the list is encrypted either with the
+list's PGP key or with the list's S/MIME certificate. The mailing-list
+software decrypts the email and re-encrypts it individually for each
+subscriber with the subscriber's PGP key or S/MIME certificate. Details
+about the mailing-list software and the setup which is used to ensure the
+security of the lists and protection of the data can be found here:
+https://www.kernel.org/....
+
+List keys
+^^^^^^^^^
+
+For initial contact see :ref:`Contact`. For incident specific mailing-lists
+the key and S/MIME certificate are conveyed to the subscribers by email
+sent from the specific list.
+
+Subscription to incident specific lists
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Subscription is handled by the response teams. Disclosed parties who want
+to participate in the communication send a list of potential subscribers to
+the response team so the response team can validate subscription requests.
+
+Each subscriber needs to send a subscription request to the response team
+by email. The email must be signed with the subscriber's PGP key or S/MIME
+certificate. If a PGP key is used, it must be available from a public key
+server and is ideally connected to the Linux kernel's PGP web of trust. See
+also: https://www.kernel.org/signature.html.
+
+The response team verifies that the subscriber request is valid and adds
+the subscriber to the list. After subscription the subscriber will receive
+email from the mailing-list which is signed either with the list's PGP key
+or the list's S/MIME certificate. The subscriber's email client can extract
+the PGP key or the S/MIME certificate from the signature so the subscriber
+can send encrypted email to the list.
+
--- a/Documentation/process/index.rst
+++ b/Documentation/process/index.rst
@@ -45,6 +45,7 @@ Other guides to the community that are o
    submit-checklist
    kernel-docs
    deprecated
+   embargoed-hardware-issues
 
 These are some overall technical guides that have been put here for now for
 lack of a better place.

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Randy Dunlap]

On 8/15/19 2:20 PM, Greg Kroah-Hartman wrote:
>>> +The hardware security team will provide a per incident specific encrypted
>> s/per incident specific/incident-specific/
> Fixed.  And changed /a/ to /an/

eh?  still should be /a per incident/

-- 
~Randy

Re: [PATCH v2] Documentation/admin-guide: Embargoed hardware security issues

[Randy Dunlap]

On 8/15/19 2:25 PM, Greg Kroah-Hartman wrote:
> v2: updated list of people with document from Jiri as I had the old one
>     grammer tweaks based on Jon's review
>     moved document to Documentation/process/

If you get to do a v3, you can change the $Subject also.

-- 
~Randy

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Thu, Aug 15, 2019 at 03:12:34PM -0700, Randy Dunlap wrote:
> On 8/15/19 2:20 PM, Greg Kroah-Hartman wrote:
> >>> +The hardware security team will provide a per incident specific encrypted
> >> s/per incident specific/incident-specific/
> > Fixed.  And changed /a/ to /an/
> 
> eh?  still should be /a per incident/

The sentence was changed to:
	The hardware security team will provide an incident-specific
	encrypted...

is not "an" correct here?

thanks,

greg k-h

Re: [PATCH v2] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Thu, Aug 15, 2019 at 03:15:11PM -0700, Randy Dunlap wrote:
> On 8/15/19 2:25 PM, Greg Kroah-Hartman wrote:
> > v2: updated list of people with document from Jiri as I had the old one
> >     grammer tweaks based on Jon's review
> >     moved document to Documentation/process/
> 
> If you get to do a v3, you can change the $Subject also.

Doh!

Re: [PATCH] Documentation/admin-guide: Embargoed hardware security issues

[Randy Dunlap]

On 8/15/19 3:31 PM, Greg Kroah-Hartman wrote:
> On Thu, Aug 15, 2019 at 03:12:34PM -0700, Randy Dunlap wrote:
>> On 8/15/19 2:20 PM, Greg Kroah-Hartman wrote:
>>>>> +The hardware security team will provide a per incident specific encrypted
>>>> s/per incident specific/incident-specific/
>>> Fixed.  And changed /a/ to /an/
>>
>> eh?  still should be /a per incident/
> 
> The sentence was changed to:
> 	The hardware security team will provide an incident-specific
> 	encrypted...
> 
> is not "an" correct here?

That's good then.  I didn't see the complete sentence.

-- 
~Randy

Re: [PATCH v2] Documentation/admin-guide: Embargoed hardware security issues

[Josh Poimboeuf]

On Thu, Aug 15, 2019 at 11:25:05PM +0200, Greg Kroah-Hartman wrote:
> +Contact
> +-------
> +
> +The Linux kernel hardware security team is separate from the regular Linux
> +kernel security team.
> +
> +The team only handles the coordination of embargoed hardware security
> +issues.  Reports of pure software security bugs in the Linux kernel are not
> +handled by this team and the reporter will be guided to contact the regular
> +Linux kernel security team (:ref:`Documentation/admin-guide/
> +<securitybugs>`) instead.
> +
> +The team can be contacted by email at <hardware-security@kernel.org>. This
> +is a private list of security officers who will help you to coordinate an
> +issue according to our documented process.
> +
> +The list is encrypted and email to the list can be sent by either PGP or
> +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> +certificate. The list's PGP key and S/MIME certificate are available from
> +https://www.kernel.org/....

This link needs to be filled in?

> +Encrypted mailing-lists
> +-----------------------
> +
> +We use encrypted mailing-lists for communication. The operating principle
> +of these lists is that email sent to the list is encrypted either with the
> +list's PGP key or with the list's S/MIME certificate. The mailing-list
> +software decrypts the email and re-encrypts it individually for each
> +subscriber with the subscriber's PGP key or S/MIME certificate. Details
> +about the mailing-list software and the setup which is used to ensure the
> +security of the lists and protection of the data can be found here:
> +https://www.kernel.org/....

Ditto?

-- 
Josh

Re: [PATCH v2] Documentation/admin-guide: Embargoed hardware security issues

[Greg Kroah-Hartman]

On Tue, Aug 20, 2019 at 09:58:50AM -0500, Josh Poimboeuf wrote:
> On Thu, Aug 15, 2019 at 11:25:05PM +0200, Greg Kroah-Hartman wrote:
> > +Contact
> > +-------
> > +
> > +The Linux kernel hardware security team is separate from the regular Linux
> > +kernel security team.
> > +
> > +The team only handles the coordination of embargoed hardware security
> > +issues.  Reports of pure software security bugs in the Linux kernel are not
> > +handled by this team and the reporter will be guided to contact the regular
> > +Linux kernel security team (:ref:`Documentation/admin-guide/
> > +<securitybugs>`) instead.
> > +
> > +The team can be contacted by email at <hardware-security@kernel.org>. This
> > +is a private list of security officers who will help you to coordinate an
> > +issue according to our documented process.
> > +
> > +The list is encrypted and email to the list can be sent by either PGP or
> > +S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
> > +certificate. The list's PGP key and S/MIME certificate are available from
> > +https://www.kernel.org/....
> 
> This link needs to be filled in?
> 
> > +Encrypted mailing-lists
> > +-----------------------
> > +
> > +We use encrypted mailing-lists for communication. The operating principle
> > +of these lists is that email sent to the list is encrypted either with the
> > +list's PGP key or with the list's S/MIME certificate. The mailing-list
> > +software decrypts the email and re-encrypts it individually for each
> > +subscriber with the subscriber's PGP key or S/MIME certificate. Details
> > +about the mailing-list software and the setup which is used to ensure the
> > +security of the lists and protection of the data can be found here:
> > +https://www.kernel.org/....
> 
> Ditto?

Yes, they will once the links are up and running :)

thanks,

greg k-h