Security vulnerability process, and CVE-2012-0217

Security vulnerability process, and CVE-2012-0217

[Ian Jackson]

Introduction
------------

The Xen.org security response team is charged with implementing the
Xen security response process, the current version of which can be
found here:

    http://www.xen.org/projects/security_vulnerability_process.html

Over the past two months we on that team have been involved with
XSA-7 / CVE-2012-0217 and its various fallout.

During this exercise we have encountered some problems with the
process.  Also, we have had to make some difficult decisions.  We feel
it is essential for keeping us honest that we explain to the community
what we did, and when.  There's a detailed timeline at the bottom of
this mail, listing the important events; reading it will give you a
much clearer picture of the problems we encountered and why we are
asking for various issues to be clarified.

In this message I'll go through some of the problems we encountered.
Where there are uncontroversial improvements for the process we
propose them here.  For the more controversial issues, in this message
we will raise the questions and discuss the issues in a neutral way,
and make our individual responses separately.

The outcome of this discussion will be a set of changes to be agreed
on and/or voted on using the existing Xen.org governance processes:
    http://www.xen.org/projects/governance.html

This discussion will take place on the xen-devel mailing list.
We expect it to take some weeks.


Summary of the most important difficulties we encountered
---------------------------------------------------------

One member of the predisclosure list, only a few days before the
embargo date, mounted a sustained and eventually successful lobbying
campaign to get the embargo extened.

There were leaks during the embargo period, after it had been
extended, which we experienced as enquiries from community members.

We were forced to made several ad-hoc decisions with insufficient
guidance from the process document.

We discovered too late in the process that the set of vulnerable
operating systems was substantially wider than we thought.



The big issues
--------------

1. Purpose of the process

The first point is that we think the security vulnerability process
document should have an explanation of what its goals are.  This would
have greatly assisted us when making some of the more difficult
decisions.

In particular, if the process explained its purpose, we would be able
to refer to the spirit of the document when making interpretation
decisions or dealing with issues not clearly resolved by the document.

In this context we need to decide whether:
 (a) the predisclosure arrangements are just there so that
     organisations can backport patches, develop and test updated
     versions, and so forth;
 (b) the arrangements are also intended to allow operators to deploy
     fixed versions.

Of course this needs to deal clearly with the common stituation of an
organisation running Xen which is a direct consumer of Xen.org source
code.


2. Extension of embargo dates

The most controversial decision was whether the embargo date might be
extended after it had originally been set.  We resisted these
suggestions, referring to the process document, which does not
contemplate extending the disclosure period.  However, when a
predisclosure list member pressured the discoverer into requesting an
extension, we felt the best interpretation of the process document
required us to acquiesce.

Specifically, of course, we as a team would like clearer guidance from
the process about whether, and if so under what circumstances, a
predisclosure period should be extended.


3. Decisionmaking

It was suggested to us several times, and indeed seemed to be regarded
by some as an underlying principle, that the predisclosure list
members should be making the decisions about disclosure schedule etc.

The question of who is to make the decisions, and on what basis, needs
to be addressed.


4. Length of (default) predisclosure period

Several members of the predisclosure list suggested that the
predisclosure period was too short.  Others were ready within the
predisclosure period's timeframe, and were disappointed to see it
extended and to have to delay their fixes.


5. Criteria for predisclosure list membership

We had one request from a public Xen cloud provider to be provided
with predisclosure information.  However it appeared to us that they
didn't meet the size threshold in the process document.

The size threshold is of course open to discussion.

We need to clarify whether upstreams and hardware vendors should be on
the predisclosure list.  Currently we have one notable upstream vendor
on that list.

Our preliminary view is that the predisclosure list should be used for
downstream notifications.  Upstreams, including hardware
manufacturers, should be brought in to the disclosure process as
needed.  And as noted, our process for making sure we do that properly
needs to be formalised.

It is probably time for a review of the membership of the
predisclosure list, in any case, after we have incorporated any
changes to the criteria which are agreed as a result of this
discussion.


6. Sharing of information and code between predisclosure participants

We need the process document to be clearer about what kinds of
communications are contemplated _between_ members of the predisclosure
list.

One particular issue here which also relates to the predisclosure
membership criteria, is whether large indirect consumers of Xen should
be on the predisclosure list in their own right.  That would allow
them to deploy the fix before the embargo date.  It would also allow
them to prepare for testing and deployment, before the fix is
available from their vendor (who would in this scenario also be
entitled to be a predisclosure list member).

In this context, the process needs to clarify whether vendors may
release fixed binaries during the predisclosure period.  Certainly
these should not be released other than to predisclosure list members,
since the nature of a bug can often by discovered by reverse
engineering.  But can they be released to predisclousre list members ?



More minor policy questions
---------------------------

7. Public communications during the embargo period

We need to clarify what individual vendors are allowed to say during
the embargo period.  It was brought to our attention that one public
cloud provider told its users that their VMs would be migrated due to
"a vendor issue" which would be revealed in "a few weeks".  Our view
was that that was fine.

As a starting point, it seems to us that it is OK to disclose that
there is an issue, including its XSA and CVE numbers and its planned
disclosure date; but that it is not OK to disclose the impact, scope,
set of vulnerable systems, and certainly not the nature of the
vulnerability.

And we need to clarify what information the security team should give
out when we get an enquiry from a community member about an embargoed
problem.


8. Predisclosure subscription process, and email address criteria

We certainly need to clarify the subscription process to the list, to
make it clear that the organisation's security team should request all
subscriptions and that self-subscriptions via the mailing list
interface will be rejected out of hand.  The current arrangements have
caused quite a lot of admin work, to confirm various change requests.

The predisclosure list contains a number of individuals at various
organisations.  One vendor seemed to get into difficulty because it
had only one individual on the predisclosure list, which seems to have
unfortunately kept their established team in the dark during the early
stages of the process.  The process document requires that downstreams
on the list have established security teams.  Should we require that
such response teams' contact details be publicly advertised ?  Should
we insist that only response teams' role addresses may be included on
the list ?


9. Vulnerability process scope

We need to clarify the scope of the process document.  Currently it
looks like it covers every Xen.org project.

While it would be a nice ideal to support every Xen.org project this
way, in practice the team behind security@xen do not have the
expertise or resources to fix problems in (say) XCP, or the ARM PV
port.  Our capability is concentrated on the "Upstream Xen" codebase
(xen-*.hg and its sub-repositories) and the Xen support code in Linux.

Perhaps this could be dealt with by making it clear that problems are
handled on a best effort basis.


10. Exploits

We need a clear policy about releasing proof of concept exploits -
whether, when and who to.


11. Transparency

We think the process document should explicitly state that any
potentially controversial private decisions made by the Xen.org
security response team should be disclosed after the vulnerability
itself is disclosed.


Lacunae in the process
----------------------

12. Holidays

The original planned disclosure date, 2012-05-01, was a public holiday
in many places.  We should try to avoid holidays, and Fridays.  We
need to discuss which territories' holidays should be checked and make
a list for inclusion in the process.


13. Patch development and review

The patch development process is too closed and not robust enough.

We need to be much clearer both about the need for security patches to
fix things in the smallest, simplest and most reliable way, and not
fix or "improve" matters in unrelated ways.  Our internal code review
needs to be much better.

We need to clarify that other issues discovered during the course of
investigating a security disclosure will not be publicly discussed
without first consulting the Xen.org security team, regardless of
their actual relationship to the vulnerability at hand or expected
security impact.  Otherwise there is a substantial risk that such
changes will draw attention to embargoed problems.

It may be that it would be a better idea to send out earlier
advisories without patches to the predisclosure list, to enable those
predisclosure list members who would like to do so to help with
development and testing of the patches.  If so this needs to be
catered for.


14. Early consideration of which other organisations to bring in

Our process needs to have, very early on, an explicit step to of
deciding which other projects/organisations may also be vulnerable and
may therefore need to be part of the same disclosure process.  It also
needs to make sure that we ask for any help (for example from
upstreams or hardware vendors) as soon as possible.

We also need to explicitly determine the availability of various key
players over the disclosure timeline as an early step.  One of the
difficulties we encountered was that one of the more important team
members was unexpectedly out of their office at a critical point.


15. Process automation

Many of the advisory versions sent to the predisclosure list contained
clerical errors.  Our processes for constructing these need to be
made more automatic.


Thanks for your attention,
Ian.
on behalf of the Xen.org security response team


Timeline (here "us" is the Xen.org security team)
-------------------------------------------------

 2012-04-09  Xen.org and FreeBSD notified of the problem (which
             will become XSA-7 / CVE-2012-0217) by the discoverer.

 2012-04-11  Discoverer tells us that they are happy with following the
             Xen.org process for this vulnerability, and asks for
             objections from FreeBSD.

 2012-04-11  We ask the Debian security team to allocate us a CVE
             candidate number.  (We have had difficulty with getting
             these from Mitre.)

 2012-04-12  We confirm in response to a question from Debian
             that we think Debian is vulnerable, giving some more
             information but not enough to find the problem.

 2012-04-12  We contact AMD to ask them to confirm whether any
             AMD processors are vulnerable.

 2012-04-12  Debian allocates us CVE-2012-0217.  We make sure that
             everyone currently aware of the issue is told.

 2012-04-13  Prompted by the discoverer, we contacts NetBSD
             so that they can check if they also are vulnerable,
             given that we believe that NetBSD may share the relevant
             code with FreeBSD.

 2012-04-16  Near-final draft of the predisclosure advisory is ready,
             including initial versions of patches for xen-unstable,
             Xen 4.1 and Xen 4.0.

 2012-04-17  The problem which will later become XSA-8 /
             CVE-2012-0218 is spotted and a fix committed to
             xen-unstable.hg, without mentioning that this bug is a
             security vulnerability.  In our view this was a mistake.

 2012-04-17  XSA-7 v1 is sent out, with an embargo end date of
             2012-05-01.

 2012-04-17  We become aware of the earlier commit and assign it
             XSA-8.  XSA-7 v2 is sent out, with a slightly improved
             patch and mentioning that XSA-8 will follow.

 2012-04-18  We ask Mitre and Debian for a CVE for XSA-8.  We send
             XSA-7 v3, and XSA-8 to the predisclosure list.

 2012-04-20  We send XSA-8 v2, which contains backported patches and
             corrections to the advisory text widening the set of
             possibly affected systems.

 2012-04-20  We send XSA-7 v4, which contains backported patches and
             the information that AMD CPUs are not vulnerable
             (following confirmation from AMD).

 2012-04-20  A member of the predisclosure list complains to us that
             the disclosure schedule is too short, and making various
             other points about the process.  We reply with a
             reference to the process document; we suggest that if the
             organisation feels that the process needs improving this
             should be discussed in public.

 2012-04-20  We are asked by a member of the predisclosure list whether
             it would be OK to release fixed binaries to a customer of
             theirs who is also a member of the predisclosure list.
             On the 2012-04-23, after internal discussions and
             consulting with the discoverer, we reply to say that we
             think that would be OK.

 2012-04-20  We are asked by a member of the predisclousre list
             whether there is any way they can verify whether their
             systems are vulnerable, or perhaps whether we can share
             any proof of concept exploit code.  Again, we consulted
             with the discoverer.

 2012-04-21  The discoverer reproduces the problem on a widely-deployed
             proprietary operating system that runs on x86, and
             suggests to us that the embargo period should therefore
             be extended.  We reply:

       Our process doesn't contemplate the extension of the embargo
       after the release of information to our predisclosure list.

       It would in theory be possible for us to email the
       predisclosure list ask them to extend the embargo date.
       However, from a practical point of view, if anyone didn't get
       the email they might go ahead and disclose on the original
       date, leaving us all scrambling madly with an unexpected
       disclosure.  And from a process point of view, that would be
       departing substantially from our published process, so I'm very
       reluctant.

       Certainly you are entitled to notify [proprietary OS vendor]
       right away and it would be sensible to do so.

             With hindsight it was an error not to consider, as part
             of the formal process, which other operating systems
             might be vulnerable.

 2012-04-26  We obtain the number CVE-2012-0218 from Debian for XSA-8.

 2012-04-26  We send XSA-7 v5 with improved wording.
             We send XSA-8 v3, with the CVE number.

 2012-04-28  A member of the predisclosure list requests an extension
             of the disclosure timeframe, and in the same message
             requests a change to the email address subscribed to the
             predisclosure list.  We reply declining, referring to our
             process document, and once again inviting public
             discussion of the process itself.

 2012-04-28  The member of the predisclosure list asking for an
             extension asks to be put in touch with all the other
             members of the predisclosure list and proposes a
             conference call.

 2012-04-30  We decline to give further details of the predisclosure
             list members, other than the organisational identities
             listed.  We again decline to extend the disclosure date.

 2012-04-30  The member asking for an extension reiterates their
             request, citing support from various other named
             predisclosure list members - in at least some cases,
             exaggerating the level of such support.

 2012-04-30  Various other members of the predisclosure list report to
             us having been asked to support the member asking for a
             delay.  CEOs of several companies were involved.  In some
             cases the other members support the request.  We again
             decline.

 2012-04-30  The member requesting an embargo suggests holding a straw
             poll of predisclosure list members on the question of a
             delay.  Again, we decline.

 2012-04-30  The member requesting an embargo places intense pressure
             on the employers of some members of the Xen.org security
             team to try to get the decision changed, without success.

 2012-04-30 23:06 Z  (13h before planned disclosure)
             The member requesting an embargo contacts the employer
             of the discoverer to apply pressure.  The discoverer
             notifies Xen.org that "on behalf of the organization who
             discovered this issue" they request a delay, giving
             a putative but unconfirmed new disclosure date of the
             2012-05-31.

 2012-05-01 08:08 Z  (3h52 before planned disclosure)
             Xen.org team decides, after internal discussions, that
             as the policy gives the discoverer control of the
             disclosure date, and the question is not otherwise
             addressed, we must agree to this request.

 2012-05-10 08:48 Z  (3h12 before planned disclosure)
             We send an ad-hoc message to the predisclosure list
             requesting a disclosure delay for XSA-7 / XSA-8.
             We do not include a revised disclosure date as we don't
             have one confirmed.

 2012-05-01  Backport of XSA-7/XSA-8 patches to 3.4 are prepared.

 2012-05-02  We become aware, indirectly, that US-CERT have become
             involved.

 2012-05-03  US-CERT suggest to us a disclosure date of 2012-06-12.
             We make a formal response, saying that we think this is
             far too late.

 2012-05-03  We receive the first enquiry from a Xen community member
             not on the predisclosure list, saying they had heard
             somerthing was wrong, and asking about it.  This is the
             first of several such enquiries.

             We quote the number CVE-2012-0217 and say that we expect
             it to be disclosed on the 12th of June, but that we
             aren't allowed to say more.

 2012-05-08  We press the discoverers' employer for a confirmed
             disclosure date.

 2012-05-08  Given that no disclosure date is forthcoming, we send
             XSA-7 v6 and XSA-8 v4, with an indefinite embargo and the
             backport patch for Xen 3.4.

 2012-05-14  Given that still no firm date is forthcoming, we (the
             Xen.org team) point out to the discoverers and to CERT
             that our policy requires us to honour reasonable
             requests, but that the continued lack of any date is
             unreasonable, and demanding that a date be set by the
             2012-05-18.

 2012-05-14  The discover confirms 2012-06-12 as the disclosure
             date.

 2012-05-15  We send XSA-7 v7 and XSA-8 v5 with the disclosure date of
             2012-06-12.

 2012-05-15  We receive a complaint about the delay from a member of
             the predisclosure list.

 2012-05-15  A member of the predisclosure list discovers what will
             become XSA-9.  The context was testing the fix for
             XSA-7.

 2012-05-16  We conclude that this is due to AMD erratum #121, and
             involve AMD.

 2012-05-16  After discussions we conclude that there is no reasonable
             software workaround for Xen and that instead Xen should
             refuse to boot on affected systems.

 2012-05-18  We decide that releasing this as a separate advisory with
             the same embargo date will be less confusing than trying
             to fold it into XSA-7.

 2012-05-18  While investigating this we come to the conclusion that
             our previous patch to fix XSA-7 was incomplete.

 2012-05-21  First draft of XSA-9 is available.

 2012-05-22  After intense internal discussions we decide that the
             original fix for XSA-7 is too fragile and conclude that
             it should be replaced by something more obvious.

 2012-05-23  We request a CVE for XSA-9 from Mitre and Debian.

 2012-05-24  We send XSA-7 v8, now with a cross-reference to XSA-9,
             and with updated simpler patch, and XSA-8 v6, and
             XSA-9 v1 with its own separate patch, to the
             predisclosure list.

 2012-05-24  Mitre assign CVE-2012-2934 for XSA-9.

 2012-06-11  We send XSA-9 v2, containing the CVE and a clear
             statement that no affected processors support SVM.
             (The delay to this message was a mistake by the Xen.org
             team.)

 2012-06-12  Public disclosure.

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 19.06.12 at 20:16, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
Representing only personal (i.e. not necessarily my employer's)
opinions below.

> 1. Purpose of the process
> 
> The first point is that we think the security vulnerability process
> document should have an explanation of what its goals are.  This would
> have greatly assisted us when making some of the more difficult
> decisions.
> 
> In particular, if the process explained its purpose, we would be able
> to refer to the spirit of the document when making interpretation
> decisions or dealing with issues not clearly resolved by the document.
> 
> In this context we need to decide whether:
>  (a) the predisclosure arrangements are just there so that
>      organisations can backport patches, develop and test updated
>      versions, and so forth;
>  (b) the arrangements are also intended to allow operators to deploy
>      fixed versions.
> 
> Of course this needs to deal clearly with the common stituation of an
> organisation running Xen which is a direct consumer of Xen.org source
> code.

(a). Anything beyond may give (and apparently gave in the case
at hand) unfair advantages to certain downstreams over others.

> 2. Extension of embargo dates
> 
> The most controversial decision was whether the embargo date might be
> extended after it had originally been set.  We resisted these
> suggestions, referring to the process document, which does not
> contemplate extending the disclosure period.  However, when a
> predisclosure list member pressured the discoverer into requesting an
> extension, we felt the best interpretation of the process document
> required us to acquiesce.
> 
> Specifically, of course, we as a team would like clearer guidance from
> the process about whether, and if so under what circumstances, a
> predisclosure period should be extended.

I think this should be done via (perhaps silent) consensus on the
pre-discosure list. Leaving the embargo time line determination
entirely to the discoverer isn't really suitable. He might provide an
initial suggestion, but we ought to set a period of time (say, a
week or less) during which adjustment proposals can be made.

Extensions to the embargo period should be limited exclusively to
cases where problems addressing the problem in a timely manner
(including eventual later _necessary_ adjustments to the fixes)
arise.

> 3. Decisionmaking
> 
> It was suggested to us several times, and indeed seemed to be regarded
> by some as an underlying principle, that the predisclosure list
> members should be making the decisions about disclosure schedule etc.
> 
> The question of who is to make the decisions, and on what basis, needs
> to be addressed.

See above. Leaving this to the discretion of the discoverer is
neither open nor fair.

> 4. Length of (default) predisclosure period
> 
> Several members of the predisclosure list suggested that the
> predisclosure period was too short.  Others were ready within the
> predisclosure period's timeframe, and were disappointed to see it
> extended and to have to delay their fixes.

Setting a default period might be counter productive. There may
be cases (for example had we wanted to fix XSA-9 properly)
where developing a fix could take quite a bit of time. Not having
a fix ready shouldn't, however, prevent initial announcements of
a problem.

> 5. Criteria for predisclosure list membership
> 
> We had one request from a public Xen cloud provider to be provided
> with predisclosure information.  However it appeared to us that they
> didn't meet the size threshold in the process document.
> 
> The size threshold is of course open to discussion.
> 
> We need to clarify whether upstreams and hardware vendors should be on
> the predisclosure list.  Currently we have one notable upstream vendor
> on that list.
> 
> Our preliminary view is that the predisclosure list should be used for
> downstream notifications.  Upstreams, including hardware
> manufacturers, should be brought in to the disclosure process as
> needed.  And as noted, our process for making sure we do that properly
> needs to be formalised.
> 
> It is probably time for a review of the membership of the
> predisclosure list, in any case, after we have incorporated any
> changes to the criteria which are agreed as a result of this
> discussion.

I think having hardware vendors involved is really only necessary
when hardware related issues need to be dealt with.

As to downstreams, I think only direct ones should be involved in
any decision making processes. Indirect ones might be allowed on
the list, but exclusively as observers. Mis-use of the observer role
(e.g. as in influencing those participating in decision making in
undue ways), should it become known, should result in removal of
list membership.

> 6. Sharing of information and code between predisclosure participants
> 
> We need the process document to be clearer about what kinds of
> communications are contemplated _between_ members of the predisclosure
> list.
> 
> One particular issue here which also relates to the predisclosure
> membership criteria, is whether large indirect consumers of Xen should
> be on the predisclosure list in their own right.  That would allow
> them to deploy the fix before the embargo date.  It would also allow
> them to prepare for testing and deployment, before the fix is
> available from their vendor (who would in this scenario also be
> entitled to be a predisclosure list member).
> 
> In this context, the process needs to clarify whether vendors may
> release fixed binaries during the predisclosure period.  Certainly
> these should not be released other than to predisclosure list members,
> since the nature of a bug can often by discovered by reverse
> engineering.  But can they be released to predisclousre list members ?

As indicated above, this should not be permitted, due to
fairness considerations. Otherwise we should not place
restrictions on who might be on the list at least as an observer.

> 7. Public communications during the embargo period
> 
> We need to clarify what individual vendors are allowed to say during
> the embargo period.  It was brought to our attention that one public
> cloud provider told its users that their VMs would be migrated due to
> "a vendor issue" which would be revealed in "a few weeks".  Our view
> was that that was fine.
> 
> As a starting point, it seems to us that it is OK to disclose that
> there is an issue, including its XSA and CVE numbers and its planned
> disclosure date; but that it is not OK to disclose the impact, scope,
> set of vulnerable systems, and certainly not the nature of the
> vulnerability.
> 
> And we need to clarify what information the security team should give
> out when we get an enquiry from a community member about an embargoed
> problem.

Just the same we allow individual vendors to communicate -
acknowledge the fact that there is a vulnerability, identifiers, and
expected embargo deadline.

The alternative would be to not do and allow any communication,
which I think is not realistic to enforce.

> 8. Predisclosure subscription process, and email address criteria
> 
> We certainly need to clarify the subscription process to the list, to
> make it clear that the organisation's security team should request all
> subscriptions and that self-subscriptions via the mailing list
> interface will be rejected out of hand.  The current arrangements have
> caused quite a lot of admin work, to confirm various change requests.
> 
> The predisclosure list contains a number of individuals at various
> organisations.  One vendor seemed to get into difficulty because it
> had only one individual on the predisclosure list, which seems to have
> unfortunately kept their established team in the dark during the early
> stages of the process.  The process document requires that downstreams
> on the list have established security teams.  Should we require that
> such response teams' contact details be publicly advertised ?  Should
> we insist that only response teams' role addresses may be included on
> the list ?

Yes.

> 9. Vulnerability process scope
> 
> We need to clarify the scope of the process document.  Currently it
> looks like it covers every Xen.org project.
> 
> While it would be a nice ideal to support every Xen.org project this
> way, in practice the team behind security@xen do not have the
> expertise or resources to fix problems in (say) XCP, or the ARM PV
> port.  Our capability is concentrated on the "Upstream Xen" codebase
> (xen-*.hg and its sub-repositories) and the Xen support code in Linux.
> 
> Perhaps this could be dealt with by making it clear that problems are
> handled on a best effort basis.

Agreed.

> 10. Exploits
> 
> We need a clear policy about releasing proof of concept exploits -
> whether, when and who to.

This I think could (and perhaps should) be really be left to the
discoverer, as this may be considered intellectual property.

> 11. Transparency
> 
> We think the process document should explicitly state that any
> potentially controversial private decisions made by the Xen.org
> security response team should be disclosed after the vulnerability
> itself is disclosed.

While fine from a theoretical pov, this places extra work on the
security response team, and I'm therefore not sure it's worth it.

Perhaps, if anything, such information (when regarding details of
the implementation of a fix) could go into the commit message.

> 13. Patch development and review
> 
> The patch development process is too closed and not robust enough.
> 
> We need to be much clearer both about the need for security patches to
> fix things in the smallest, simplest and most reliable way, and not
> fix or "improve" matters in unrelated ways.  Our internal code review
> needs to be much better.

I think this shouldn't be as strict. Selecting e.g. between larger,
less performance intrusive changes over smaller changes with a
larger impact on performance should be done on a case by case
basis. (I'm still of the opinion that the original XSA-7 fix, which
now became c/s 25485:5b6a857411ba [minus the now pointless
adjustment to the hypercall path], would have been the better
one, despite it having required quite a bit of thinking to verify
that it is sufficient in that final form. The main problem here was
that no-one, including me, took the time early on to document all
the affected code paths for others to verify, and patch review
wasn't concise enough either.)

If in doubt, room should be provided to involve individuals not
on the security response team (of course requiring them to not
disclose the knowledge gained).

> We need to clarify that other issues discovered during the course of
> investigating a security disclosure will not be publicly discussed
> without first consulting the Xen.org security team, regardless of
> their actual relationship to the vulnerability at hand or expected
> security impact.  Otherwise there is a substantial risk that such
> changes will draw attention to embargoed problems.

I don't think this is strictly necessary, nor a problem. This has
become a problem in the recent process mainly due to the
extraordinary length of the embargo period.

> It may be that it would be a better idea to send out earlier
> advisories without patches to the predisclosure list, to enable those
> predisclosure list members who would like to do so to help with
> development and testing of the patches.  If so this needs to be
> catered for.

As said earlier, if a fix is available, it of course should be included.
But there shouldn't be any requirement for that.

> 14. Early consideration of which other organisations to bring in
> 
> Our process needs to have, very early on, an explicit step to of
> deciding which other projects/organisations may also be vulnerable and
> may therefore need to be part of the same disclosure process.  It also
> needs to make sure that we ask for any help (for example from
> upstreams or hardware vendors) as soon as possible.

Our security response team should only take our projects into
consideration. Cross project vulnerabilities should be managed
by entities set up to deal with this. (Not that I'm generally in
favor of copying bad behavior, but note how the problem in
XSA-7 was not brought to our or other OS vendors' attention
when dealt with years ago in Linux.)

Jan

> We also need to explicitly determine the availability of various key
> players over the disclosure timeline as an early step.  One of the
> difficulties we encountered was that one of the more important team
> members was unexpectedly out of their office at a critical point.
> 
> 
> 15. Process automation
> 
> Many of the advisory versions sent to the predisclosure list contained
> clerical errors.  Our processes for constructing these need to be
> made more automatic.

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

On Wed, Jun 20, 2012 at 9:49 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 19.06.12 at 20:16, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>> The most controversial decision was whether the embargo date might be
>> extended after it had originally been set.  We resisted these
>> suggestions, referring to the process document, which does not
>> contemplate extending the disclosure period.  However, when a
>> predisclosure list member pressured the discoverer into requesting an
>> extension, we felt the best interpretation of the process document
>> required us to acquiesce.
>>
>> Specifically, of course, we as a team would like clearer guidance from
>> the process about whether, and if so under what circumstances, a
>> predisclosure period should be extended.
>
> I think this should be done via (perhaps silent) consensus on the
> pre-discosure list. Leaving the embargo time line determination
> entirely to the discoverer isn't really suitable. He might provide an
> initial suggestion, but we ought to set a period of time (say, a
> week or less) during which adjustment proposals can be made.

The problem with this is that extending the embargo helps providers
who are on the predisclosure list (a minority), but hurts those not on
the list (the vast majority).  So there's a moral hazard with what you
suggest: it is just way too easy for the people on the list to vote to
make their own lives easier, not considering the plight of those who
are not big enough or connected enough to be on the list.  Doing so
also favors large players and incumbents against small players; doing
so may well be considered anti-competitive and illegal in many
jurisdictions.

The only way this would work is if the predisclosure list consisted
exclusively of software providers, and specifically excluded service
providers.  It should be possible to include all software providers on
the list, since it's a relatively small number of entities.
Furthermore, since software providers presumably care about the
security of their customers, it would provide the incentive to make
the embargo as short as is reasonable.

>> 3. Decisionmaking
>>
>> It was suggested to us several times, and indeed seemed to be regarded
>> by some as an underlying principle, that the predisclosure list
>> members should be making the decisions about disclosure schedule etc.
>>
>> The question of who is to make the decisions, and on what basis, needs
>> to be addressed.
>
> See above. Leaving this to the discretion of the discoverer is
> neither open nor fair.

But there's a practical matter to consider here: if it's known that we
won't cooperate with the discoverer wrt disclosure times, discoverers
may simply not share their information with us.  I think that's the
main reason for the "do what the discoverer wants" rule.  I think
there needs to be some kind of a balance though: extending the embargo
less than 12 hours before the deadline is clearly not reasonable.

>> Several members of the predisclosure list suggested that the
>> predisclosure period was too short.  Others were ready within the
>> predisclosure period's timeframe, and were disappointed to see it
>> extended and to have to delay their fixes.
>
> Setting a default period might be counter productive. There may
> be cases (for example had we wanted to fix XSA-9 properly)
> where developing a fix could take quite a bit of time. Not having
> a fix ready shouldn't, however, prevent initial announcements of
> a problem.

A default period can be considered a guideline.  In the case of a
particularly tricky fix, saying, "Normally we would set 2 weeks, but I
think in this case we should go for 3 or 4" is perfectly reasonable.

Since the embargo is really for software providers to test fixes,
perhaps we should suggest it should be "X business days after a fix is
released", rather than "X days after the vulnerability is disclosed"?

> As to downstreams, I think only direct ones should be involved in
> any decision making processes. Indirect ones might be allowed on
> the list, but exclusively as observers. Mis-use of the observer role
> (e.g. as in influencing those participating in decision making in
> undue ways), should it become known, should result in removal of
> list membership.

What do you mean by "direct" and "indirect"?  If by "direct" you mean,
"sell/distribute software", and "indirect" you mean, "use the software
to provide a service", I think we're probably in agreement. :-)

-George

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 20.06.12 at 11:45, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> On Wed, Jun 20, 2012 at 9:49 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 19.06.12 at 20:16, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>>> The most controversial decision was whether the embargo date might be
>>> extended after it had originally been set.  We resisted these
>>> suggestions, referring to the process document, which does not
>>> contemplate extending the disclosure period.  However, when a
>>> predisclosure list member pressured the discoverer into requesting an
>>> extension, we felt the best interpretation of the process document
>>> required us to acquiesce.
>>>
>>> Specifically, of course, we as a team would like clearer guidance from
>>> the process about whether, and if so under what circumstances, a
>>> predisclosure period should be extended.
>>
>> I think this should be done via (perhaps silent) consensus on the
>> pre-discosure list. Leaving the embargo time line determination
>> entirely to the discoverer isn't really suitable. He might provide an
>> initial suggestion, but we ought to set a period of time (say, a
>> week or less) during which adjustment proposals can be made.
> 
> The problem with this is that extending the embargo helps providers
> who are on the predisclosure list (a minority), but hurts those not on
> the list (the vast majority).  So there's a moral hazard with what you
> suggest: it is just way too easy for the people on the list to vote to
> make their own lives easier, not considering the plight of those who
> are not big enough or connected enough to be on the list.  Doing so
> also favors large players and incumbents against small players; doing
> so may well be considered anti-competitive and illegal in many
> jurisdictions.
> 
> The only way this would work is if the predisclosure list consisted
> exclusively of software providers, and specifically excluded service
> providers.  It should be possible to include all software providers on
> the list, since it's a relatively small number of entities.
> Furthermore, since software providers presumably care about the
> security of their customers, it would provide the incentive to make
> the embargo as short as is reasonable.

You need to take this in context with my proposal to have only
software vendors have a vote here, and to allow service
providers at most an observing (maybe recommending to a
limited degree) role.

>>> 3. Decisionmaking
>>>
>>> It was suggested to us several times, and indeed seemed to be regarded
>>> by some as an underlying principle, that the predisclosure list
>>> members should be making the decisions about disclosure schedule etc.
>>>
>>> The question of who is to make the decisions, and on what basis, needs
>>> to be addressed.
>>
>> See above. Leaving this to the discretion of the discoverer is
>> neither open nor fair.
> 
> But there's a practical matter to consider here: if it's known that we
> won't cooperate with the discoverer wrt disclosure times, discoverers
> may simply not share their information with us.  I think that's the
> main reason for the "do what the discoverer wants" rule.  I think
> there needs to be some kind of a balance though: extending the embargo
> less than 12 hours before the deadline is clearly not reasonable.

I still suggested that the discoverer gets a first shot at the
embargo deadline. But if everyone else disagrees (i.e. the
deadline was unreasonable), then it should be possible to ignore
the discoverer's preference.

>>> Several members of the predisclosure list suggested that the
>>> predisclosure period was too short.  Others were ready within the
>>> predisclosure period's timeframe, and were disappointed to see it
>>> extended and to have to delay their fixes.
>>
>> Setting a default period might be counter productive. There may
>> be cases (for example had we wanted to fix XSA-9 properly)
>> where developing a fix could take quite a bit of time. Not having
>> a fix ready shouldn't, however, prevent initial announcements of
>> a problem.
> 
> A default period can be considered a guideline.  In the case of a
> particularly tricky fix, saying, "Normally we would set 2 weeks, but I
> think in this case we should go for 3 or 4" is perfectly reasonable.
> 
> Since the embargo is really for software providers to test fixes,
> perhaps we should suggest it should be "X business days after a fix is
> released", rather than "X days after the vulnerability is disclosed"?

Yes, that sounds like a reasonable thing.

>> As to downstreams, I think only direct ones should be involved in
>> any decision making processes. Indirect ones might be allowed on
>> the list, but exclusively as observers. Mis-use of the observer role
>> (e.g. as in influencing those participating in decision making in
>> undue ways), should it become known, should result in removal of
>> list membership.
> 
> What do you mean by "direct" and "indirect"?  If by "direct" you mean,
> "sell/distribute software", and "indirect" you mean, "use the software
> to provide a service", I think we're probably in agreement. :-)

That's what I mean, and so we are apparently.

Jan

Re: Security vulnerability process, and CVE-2012-0217

[Matt Wilson]

On Tue, Jun 19, 2012 at 11:16:05AM -0700, Ian Jackson wrote:
> [...]

Thank you for writing all this up, and for all of your efforts as part
of Xen.org and the security response team, Ian. You have already
covered many of my concerns, such as avoiding holidays.

> The big issues
> --------------
>
> 1. Purpose of the process
>
> The first point is that we think the security vulnerability process
> document should have an explanation of what its goals are.  This would
> have greatly assisted us when making some of the more difficult
> decisions.

The security vulnerability process document should most definitely
encapsulate both explicit guidance and broad tenets that can be used
to make tough calls. I think that these should be explicitly called
out in front matter as an evolutionary part of the process. Tenets
should always be open to being refined or redefined as Xen.org
projects grow and usage shifts.

> In particular, if the process explained its purpose, we would be able
> to refer to the spirit of the document when making interpretation
> decisions or dealing with issues not clearly resolved by the document.
>
> In this context we need to decide whether:
>  (a) the predisclosure arrangements are just there so that
>      organisations can backport patches, develop and test updated
>      versions, and so forth;
>  (b) the arrangements are also intended to allow operators to deploy
>      fixed versions.

I think that there may be other aspects of the predisclosure period
that deserve explicit callouts, so I don't think that it's helpful to
frame this discussion in particular context that only gives us choices
(a) or (b). For example, we might want to explicitly call out the
purpose of the predisclosure period to include further testing and
validation of a fix from a group more broad than the security response
team.

But taking a step back, I propose that a core tenet of the security
response process should be to reduce days-of-risk for all consumers of
Xen.org projects, whether direct or indirect, to zero. Days-of-risk
can be fuzzy to measure, so we could define this as the number of days
between when a security problem is publicly known (e.g. through
evidence of exploitation in the wild or a public announcement) and
when the problem has been addressed such that there is no longer any
risk (e.g., through a Xen consumer deploying a fixed version from a
vendor or an infrastructure provider doing the same on a consumer’s
behalf.)

This measure might not match up perfectly with typical days-of-risk
assessments that some software vendors make. For example, Red Hat
measures days-of-risk from the date of public disclosure to the date
that a fixed package is made available. They publish metrics on this,
e.g.:
  http://www.redhat.com/security/data/metrics/summary-rhel6-critical.html

> Of course this needs to deal clearly with the common stituation of an
> organisation running Xen which is a direct consumer of Xen.org source
> code.

Indeed. If a goal of the process is to reduce days-of-risk for all
consumers of Xen.org projects to zero, coordinating package updates
only from software vendors will not achieve it.

[...]
>
> 2. Extension of embargo dates
>
> The most controversial decision was whether the embargo date might be
> extended after it had originally been set.  We resisted these
> suggestions, referring to the process document, which does not
> contemplate extending the disclosure period.  However, when a
> predisclosure list member pressured the discoverer into requesting an
> extension, we felt the best interpretation of the process document
> required us to acquiesce.
>
> Specifically, of course, we as a team would like clearer guidance from
> the process about whether, and if so under what circumstances, a
> predisclosure period should be extended.

I think that the issue of establishing and changing the embargo end
date, whether extending or shorting, deserves a lot of discussion.
It's good that we're not limiting the discussion only to extending
extending disclosure dates, but also establishing the default timeline
as you call out in 4 below.  I think that some guidelines for
establishing an appropriate timeline can address problems that cause
extension requests.

The default disclosure period, if agreeable to the discoverer, is
currently three weeks:
   1. One working week between notification arriving at security@xen
      and the issue of our own advisory to our predisclosure list. We
      will use this time to gather information and prepare our
      advisory, including required patches.
   2. Two working weeks between issue of our advisory to our
      predisclosure list and publication.

This may be a reasonable starting point for general issues, but I
think that the overall impact and complexity of an issue needs to be
considered when establishing a timeline.

For example, the oCERT Disclosure Policy gives this guidance:
  https://www.ocert.org/disclosure_policy.html
"""
   The following time frames regulate oCERT embargo proposals:

   - 7 days, in case the issue is already well narrowed down and
     tested, requiring trivial configuration and/or code change

   - 14 days, standard embargo for most cases

   - 30 days, in case of critical and complex vulnerabilities
     (example, trivial exploitation of administrative privileges on a
     static library affecting a large number of packages), and with
     the agreement of all parties

   - under extremely exceptional circumstances, if the oCERT Team and
     all the parties involved feel the need for longer time, a 2
     months embargo can be applied, in this case we would clearly
     document the decision for public review

   - in any circumstance reporter preference will always be honoured
     in case a joint agreement is not reached, as oCERT would be
     anyway unable to force its embargo
"""

The CERT process defaults to 45 days, with guidance to shorten the
release schedule if there's evidence of active exploitation:
 http://www.cert.org/kb/vul_disclosure.html
"""
   Q: Will all vulnerabilities be disclosed within 45 days?

   A: No. There may often be circumstances that will cause us to
      adjust our publication schedule. Threats that are especially
      serious or for which we have evidence of exploitation will
      likely cause us to shorten our release schedule. Threats that
      require "hard" changes (changes to standards, changes to core
      operating system components) will cause us to extend our
      publication schedule.
"""

For another take, the WebKit default embargo period, found here:
http://www.webkit.org/security/, is 60 days unless all vendors have
addressed an issue, in which case an embargo may be lifted sooner.

> 3. Decisionmaking
>
> It was suggested to us several times, and indeed seemed to be regarded
> by some as an underlying principle, that the predisclosure list
> members should be making the decisions about disclosure schedule etc.
>
> The question of who is to make the decisions, and on what basis, needs
> to be addressed.

I do not think that it is wise for the policy to establish decision
making by committee. All concerned parties should voice any concerns
regarding the security response process so that the resulting document
will guide those making decisions.

To put it another way, predisclosure list members (and any other
interested party) should make decisions on how they'd like for
disclosure schedules to be established now, and the ratified process
should act as a proxy when specific vulnerabilities are handled.

In the interest of transparency, those making decisions, i.e. the
members of the security response team and any other person or entity
on the security@xen.org alias, should be disclosed and documented in
the process document.

Ultimately I think that, as is common practice in coordinated
vulnerability disclosure, the discoverer is the ultimate
decision-maker as to what information is disclosed and when it is
disclosed. The xen@security team makes decisions per the guidance in
the process and the facts at hand.

> 4. Length of (default) predisclosure period
>
[...]

See comments above.

> 5. Criteria for predisclosure list membership
>
[...]
> We need to clarify whether upstreams and hardware vendors should be on
> the predisclosure list.  Currently we have one notable upstream vendor
> on that list.
>
> Our preliminary view is that the predisclosure list should be used for
> downstream notifications.  Upstreams, including hardware
> manufacturers, should be brought in to the disclosure process as
> needed.  And as noted, our process for making sure we do that properly
> needs to be formalised.

Overall I think that, in a coordinated disclosure response,
information should be distributed on a need-to-know basis.

[...]
> 6. Sharing of information and code between predisclosure participants
>
> We need the process document to be clearer about what kinds of
> communications are contemplated _between_ members of the predisclosure
> list.
>
> One particular issue here which also relates to the predisclosure
> membership criteria, is whether large indirect consumers of Xen should
> be on the predisclosure list in their own right.  That would allow
> them to deploy the fix before the embargo date.  It would also allow
> them to prepare for testing and deployment, before the fix is
> available from their vendor (who would in this scenario also be
> entitled to be a predisclosure list member).
>
> In this context, the process needs to clarify whether vendors may
> release fixed binaries during the predisclosure period.  Certainly
> these should not be released other than to predisclosure list members,
> since the nature of a bug can often by discovered by reverse
> engineering.  But can they be released to predisclousre list members ?

In my opinion, there is less distinction between "software vendors"
and "large indirect consumers of Xen" than you suggest.

> More minor policy questions
> ---------------------------
>
[...]
> 9. Vulnerability process scope
>
> We need to clarify the scope of the process document.  Currently it
> looks like it covers every Xen.org project.
>
> While it would be a nice ideal to support every Xen.org project this
> way, in practice the team behind security@xen do not have the
> expertise or resources to fix problems in (say) XCP, or the ARM PV
> port.  Our capability is concentrated on the "Upstream Xen" codebase
> (xen-*.hg and its sub-repositories) and the Xen support code in Linux.
>
> Perhaps this could be dealt with by making it clear that problems are
> handled on a best effort basis.

It would seem that in cases with mature projects, appropriate
maintainers and committers should be engaged by security@xen.org to
address problems and prepare fixes. security@xen.org should, I think,
continue to coordinate the overall response.

In order to graduate, a project should maintain sufficient sponsorship
to address security problems in a timely manner. That should be part
of the "basic conditions" of being labeled a "mature" project. If a
mature project does not have sufficient sponsorship to address
security concerns, it should be called out and archiving the project
should be considered.

The Apache Incubator process has some good advice to projects that are
graduating: http://incubator.apache.org/guides/graduation.html#security

> 10. Exploits
>
> We need a clear policy about releasing proof of concept exploits -
> whether, when and who to.

Let's not limit the discussion only to PoC exploit code. The policy
should speak to who gets access to specific technical details of a
vulnerability, and when they gain access.

Parties that are *active* in the coordinated disclosure, for example
by building patched versions of products or services built on Xen,
need to have access to as much technical information as possible in
order to validate the fix and their ports. Those that are passive, who
may be deploying a pre-release fixed version from a vendor, do not
need access to these materials.

[...]
>
> Lacunae in the process
> ----------------------
>
> 12. Holidays
>
> The original planned disclosure date, 2012-05-01, was a public holiday
> in many places.  We should try to avoid holidays, and Fridays.  We
> need to discuss which territories' holidays should be checked and make
> a list for inclusion in the process.

This makes sense, and is similar to other coordinated response groups
like linux-distros@vs.openwall.org.

> 13. Patch development and review
>
> The patch development process is too closed and not robust enough.
>
> We need to be much clearer both about the need for security patches to
> fix things in the smallest, simplest and most reliable way, and not
> fix or "improve" matters in unrelated ways.  Our internal code review
> needs to be much better.
>
> We need to clarify that other issues discovered during the course of
> investigating a security disclosure will not be publicly discussed
> without first consulting the Xen.org security team, regardless of
> their actual relationship to the vulnerability at hand or expected
> security impact.  Otherwise there is a substantial risk that such
> changes will draw attention to embargoed problems.

All of this makes sense to me.

> It may be that it would be a better idea to send out earlier
> advisories without patches to the predisclosure list, to enable those
> predisclosure list members who would like to do so to help with
> development and testing of the patches.  If so this needs to be
> catered for.

This would be tremendously helpful, so long as adequate technical
information is provided.

> 14. Early consideration of which other organisations to bring in
>
> Our process needs to have, very early on, an explicit step to of
> deciding which other projects/organisations may also be vulnerable and
> may therefore need to be part of the same disclosure process.  It also
> needs to make sure that we ask for any help (for example from
> upstreams or hardware vendors) as soon as possible.

The existing process had a step for this:
"""
  d. If we think other software systems (for example, competing
     hypervisor systems) are likely to be affected by the same
     vulnerability, we will try to make those other projects aware of
     the problem and include them in the advisory preparation process.
"""

> [...]

Thank you again for acting as the facilitator for this discussion, and
many thanks to everyone who works tirelessly as part of the Xen.org
community on issues like these.

Matt

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Security vulnerability process, and CVE-2012-0217

[Thomas Goirand]

Hi Ian,

Thanks for discussing this in a public way!

On 06/20/2012 02:16 AM, Ian Jackson wrote:
> We had one request from a public Xen cloud provider to be provided
> with predisclosure information.  However it appeared to us that they
> didn't meet the size threshold in the process document.
>
> The size threshold is of course open to discussion.
>   
I find the concept of "Xen Cloud provider size threshold"
quite anti competitive. Why would a bigger provider, would
be offered a substantial advantage over the smaller one?

On 06/20/2012 02:16 AM, Ian Jackson wrote:
> One particular issue here which also relates to the predisclosure
> membership criteria, is whether large indirect consumers of Xen should
> be on the predisclosure list in their own right.  That would allow
> them to deploy the fix before the embargo date.  It would also allow
> them to prepare for testing and deployment, before the fix is
> available from their vendor (who would in this scenario also be
> entitled to be a predisclosure list member).
>   

And other hosting providers not in the list? They can be hacked and die,
while the big ones are safe?

Why wouldn't a smaller company know? Can *I* be in the predisclosure list?
If you reject me from such list, why? What's the procedure to be on such
list?

On 06/20/2012 05:45 PM, George Dunlap wrote:
> The only way this would work is if the predisclosure list consisted
> exclusively of software providers, and specifically excluded service
> providers.
I agree, though you might have corner cases.

What if you are *both* software and service provider (eg: I'm working on
Debian and XCP, and my small company provides a hosted Xen service)?

Cheers,

Thomas

Re: Security vulnerability process, and CVE-2012-0217

[Alan Cox]

> I find the concept of "Xen Cloud provider size threshold"
> quite anti competitive. Why would a bigger provider, would
> be offered a substantial advantage over the smaller one?

This came up with end providers for Linux disclosure years back and in the
end we said no for all sorts of reasons. The main one being that it is
nigh on impossible to set a "fair" policy. The fact you'd have to keep
adding/removing people as their size changes or importance changes, and
also the fact that the more people you tell the more leaks you get (note
not 'it might leak', that's a given, the question is how much).

It was also readily apparent that you'd have to have a policy that was
written, approved by lawyers and clear and strong enough to stand a
potential judicial review by an aggreived major vendor, or someone wanting
a cheap PR point against their rivals.

It also creates problem incentives. Cloud provider A who is not in the
cartel will not report bugs that only their rivals who are cartel members
get to see first. They'll instead report it to their friends, fix in
private and then go public. So you fracture the reporting environment, and
you get lots of zero day "fixes" from providers that may well be
inadequate or just badly designed.

There are people who get early disclosures on the kernel, but they don't
get them because the community policy covers them but because they have
three letter initials and undoubtedly intercept and monitor the relevant
security lists 8). They wouldn't be doing their job if they didn't.

> On 06/20/2012 02:16 AM, Ian Jackson wrote:
> > One particular issue here which also relates to the predisclosure
> > membership criteria, is whether large indirect consumers of Xen should
> > be on the predisclosure list in their own right.  That would allow
> > them to deploy the fix before the embargo date.  It would also allow
> > them to prepare for testing and deployment, before the fix is
> > available from their vendor (who would in this scenario also be
> > entitled to be a predisclosure list member).
> >   
> 
> And other hosting providers not in the list? They can be hacked and die,
> while the big ones are safe?
> 
> Why wouldn't a smaller company know? Can *I* be in the predisclosure list?
> If you reject me from such list, why? What's the procedure to be on such
> list?

Your lawyers sue them, or you set up a rival "fair reporting" list and
exclude members of the other predisclosure group, do your own private
fixing and then zero day hand them over to Xen.

Providing your group is well publicised most people will tell both
anyway and given the PR battle will probably favour the 'small oppressed'
division you'll probably get more of the "one group only" bugs. Quite
frankly I'd also expect the small guys to find most of the holes.

Alan

Re: Security vulnerability process, and CVE-2012-0217

[Sander Eikelenboom]

Hello Thomas,

Wednesday, June 27, 2012, 8:07:25 PM, you wrote:

> Hi Ian,

> Thanks for discussing this in a public way!

> On 06/20/2012 02:16 AM, Ian Jackson wrote:
>> We had one request from a public Xen cloud provider to be provided
>> with predisclosure information.  However it appeared to us that they
>> didn't meet the size threshold in the process document.
>>
>> The size threshold is of course open to discussion.
>>   
> I find the concept of "Xen Cloud provider size threshold"
> quite anti competitive. Why would a bigger provider, would
> be offered a substantial advantage over the smaller one?

> On 06/20/2012 02:16 AM, Ian Jackson wrote:
>> One particular issue here which also relates to the predisclosure
>> membership criteria, is whether large indirect consumers of Xen should
>> be on the predisclosure list in their own right.  That would allow
>> them to deploy the fix before the embargo date.  It would also allow
>> them to prepare for testing and deployment, before the fix is
>> available from their vendor (who would in this scenario also be
>> entitled to be a predisclosure list member).
>>   

> And other hosting providers not in the list? They can be hacked and die,
> while the big ones are safe?

> Why wouldn't a smaller company know? Can *I* be in the predisclosure list?
> If you reject me from such list, why? What's the procedure to be on such
> list?

I think it's all a trade-off between:
A) Informing as much stakeholders as possible about the threat
B) Not informing anyone with malicious intentions

And the assumptions that have been made are:
C) Employees of large companies have a small chance of having malicious intentions
D) The risk of informing someone with a malicious intention rises when more people are included on the list

Well while D would probably be true .. C) is indeed more questionable

It's also the question if larger companies don't break the embargo by informing their customers who aren't on the list, with as end result a rumor spreading in public.

On the other hand i see no ways to circumvent any of these, except by fixing the threat ASAP and keeping the embargo time as short as possible.


> On 06/20/2012 05:45 PM, George Dunlap wrote:
>> The only way this would work is if the predisclosure list consisted
>> exclusively of software providers, and specifically excluded service
>> providers.
> I agree, though you might have corner cases.

> What if you are *both* software and service provider (eg: I'm working on
> Debian and XCP, and my small company provides a hosted Xen service)?

> Cheers,

> Thomas






-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: Security vulnerability process, and CVE-2012-0217

[Lars Kurth]

>>/  1. Purpose of the process/
>>
>>/  The first point is that we think the security vulnerability process/
>>/  document should have an explanation of what its goals are.  This would/
>>/  have greatly assisted us when making some of the more difficult/
>>/  decisions./
>
> The security vulnerability process document should most definitely
> encapsulate both explicit guidance and broad tenets that can be used
> to make tough calls. I think that these should be explicitly called
> out in front matter as an evolutionary part of the process. Tenets
> should always be open to being refined or redefined as Xen.org
> projects grow and usage shifts.

I wanted to dive a little bit into the purpose of the process as the
discussion will otherwise get stuck on detail. Also into the actors
in a security vulnerability and their interest.

1.1: The Xen.org Security team, whose task is to administer the process
and act as referee if needed: the process should provide clarity and
ensure that the team can be seen as referee. The process needs to be
clear enough to avoid a chance that members of the team are accused of
being partial.

1.2: Discoverer of the vulnerability: The process must provide an
incentive for the discover to report the issue to the project. If we
get the process wrong, the consequence will be that vulnerabilities
will not be reported to us. That would be detrimental to all
stake-holders as it will increase days-of-risk for everybody else.
E.g. if the embargo period is too long. Not sure what other factors
motivate a discoverer.

1.3: Developers within the project, who will be task to find a fix
as quickly as possible.

1.4: 3rd parties that may need to be contacted in order to develop a
fix to understand the issue and advise the security team (in the case
of CVE-2012-0217 hardware vendors).

1.5: Developers of downstream projects/distros consuming Xen and
distributing it (FOSS or commercial), who will apply the fix to
their project/distro.

1.6: Other direct consumers of Xen (e.g. service providers). Their
main goal of the process would be to reduce days-of-risk for
themselves. The issue being that deploying a fix can take a long time.
Another issue is that providing a fix before somebody else is a
competitive advantage.

1.7: Indirect consumers of Xen. With all the same motivations than
direct consumers.

A couple of observations:
1) The further you get down the list, the more people are involved,
the greater the risk that information will leak.
2) Groups 1.1 - 1.4 necessarily need to be involved to create a fix,
as otherwise there won't be a fix for the other groups.
3) Groups 1.6 - 1.7 are directly impacted by days-of-risk
4) Groups 1.1, 1.3 & 1.5 are impacted insofar that the reputation
of their organisation may be damaged if the situation is not handled
well.
  
Of course the ultimate goal of the process should be to minimize
days-of-risk for everyone. To do this, the process has to balance the
following factors to reduce days-of-risk

A) Provide incentive for vulnerabilities to be reported to Xen.org
security team in the first place
B) Time it takes to develop a fix
C) Time it takes downstream projects/distros to apply it
D) Time it takes to deploy fixes for consumers (1.6 & 1.7)
E) Reduce the possibility of a leak (keep pre-disclosure list small)
F) Avoid the perception that members of the pre-disclosure list have
an unfair advantage
  
Looking at the existing pre-disclosure list shows that it contains
parties from all groups. This opens Xen.org up to criticism that some
members of the pre-disclosure have an uncertain advantage, which has
already been highlighted earlier in this discussion.

To be honest, I don't really see a way to satisfy all needs:
- Restrict membership of disclosure lists to stake-holders 1.1, 1.2,
   1.3 and 1.5 with members of 1.4 drawn in as needed and full public
   disclosure afterwards as to who was consulted.
- Of course it may be desirable to get advice from members of groups
   1.6 and 1.7. Or is it? If it is, a different approach may be to have
   a merit rather than size based selection criteria for members of 1.6
   and 1.7 (e.g. something along the lines of "contributions" to the
   community, but that would need to be measurable - or even some sort
   of elections). But it is hard to see how this would work in practice.
   Of course such an would have the advantage that a member of that group
   that used mbership to gain an unfair advantage over others could be
   removed from the pre-disclosure list.
- Another possible approach may be a two-stage pre-disclosure
   - Stage 1: Groups 1.1, 1.2, 1.3 and 1.5 with 1.4 as needed
   - Stage 2: Groups 1.6 and 1.7 (with a relatively weak membership
     criterion such as being a service provider - but then how does
     this differ from being public and who would administer?)
- Another option may be to delegate more difficult issues on
   principle to an independent organisation such as OCERT.

Best Regards
Lars
P.S.: I just wanted to make clear that these are my personal views and reflect
in no way any position of Xen.org or my employer.

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

On Sat, Jun 23, 2012 at 8:42 PM, Matt Wilson <msw@amazon.com> wrote:
> But taking a step back, I propose that a core tenet of the security
> response process should be to reduce days-of-risk for all consumers of
> Xen.org projects, whether direct or indirect, to zero. Days-of-risk
> can be fuzzy to measure, so we could define this as the number of days
> between when a security problem is publicly known (e.g. through
> evidence of exploitation in the wild or a public announcement) and
> when the problem has been addressed such that there is no longer any
> risk (e.g., through a Xen consumer deploying a fixed version from a
> vendor or an infrastructure provider doing the same on a consumer’s
> behalf.)

Minimizing overall risk for all users of Xen certainly is the end goal
of the "responsible disclosure" process.  However, there seem to be
two flaws in the way you are defining "days-of-risk" (from how you
seem to be using and arguing about the term):
* It assumes risk on any given day is 0 or 1
* It assumes that, in the absence of evidence to the contrary, risk is
0 until the vulnerability is published by members of the
pre-disclosure list.

If those two things were true, then your policy recommendations below
might make sense.

Unfortunately, neither are true.  The real risk begins the date
software with that vulnerability is *deployed*.  From that time
software with a vulnerability is first *deployed* until the time
software with the fix is deployed, every user of that software is at
risk.  Any black hat could have been looking at the code and wondering
the same thing that the original reporter was wondering.  In this case
even more so, because Linux introduced a patch to fix the
vulnerability for them in 2006.  There must have been any number of
black hats who looked at that vulnerability and wondered if other
operating systems were vulnerable, and discovered that they were.  So
having "0 days-of-risk" for a vulnerability disclosure process is, by
definition, impossible.

Now clearly, the risk greatly increases when the vulnerability is
announced.  But any discussion of whether, and which service providers
may be included in the vulnerability disclosure process must
acknowledge that:
1. *All* service providers, and indeed all users whether they provide
service or not, are at risk every hour that a fix is not deployed on
their systems
2. Any disclosure which allows some users to patch before others gives
an advantage to those users (whether by allowing them to actually
patch their systems before the embargo period, or allowing them to
prepare the patching system so that they can be down within an hour)

And I'm going to suggest something else which I think is true, and
relevant to the discussion:
3. A large majority of deployed Xen instances are run by people not on
the pre-disclosure list.

>> Of course this needs to deal clearly with the common stituation of an
>> organisation running Xen which is a direct consumer of Xen.org source
>> code.
>
> Indeed. If a goal of the process is to reduce days-of-risk for all
> consumers of Xen.org projects to zero, coordinating package updates
> only from software vendors will not achieve it.

But "coordinating package updates only" may minimize the total risk to
all users, by allowing more users to deploy sooner.

> This may be a reasonable starting point for general issues, but I
> think that the overall impact and complexity of an issue needs to be
> considered when establishing a timeline.

I guess in part it means on what you mean by "complexity of the issue":
* In this particular case, we ended up having to coordinate with other
vulnerable projects; that is rather an exception to the rule, and
should of course have special consideration.  (Although, I might argue
that we should have told the other projects, but simply not mentioned
in our disclosure that they may be vulnerable -- i.e., make the
announcement similar to Linux's 2006 reporting of the same
vulnerability.)
* You may mean "difficulty in coming up with a fix".  I think this can
be addressed by setting the disclosure period from the time that a
*fix* is available, not the time the vulnerability is known.
* You may mean, "difficulty in deploying the fix".  In this case, it
was a simple case of recompiling the hypervisor and rebooting* -- as
such, the deployment is probably as simple as it will ever get.

(* The complexity of doing unsupported things, like hot-patching the
hypervisor, shouldn't be considered IMHO.  If hot-patching is
valuable, work should be done to make that a supported feature.)

> Parties that are *active* in the coordinated disclosure, for example
> by building patched versions of products or services built on Xen,
> need to have access to as much technical information as possible in
> order to validate the fix and their ports. Those that are passive, who
> may be deploying a pre-release fixed version from a vendor, do not
> need access to these materials.

I think this distinction is important to consider.

This is not to you, but to everyone:  Can I suggest that as much as
possible, we don't use poorly-defined terms like "direct" and
"indirect"?  As far as I know we have the people listed below; we
should try to come up with clear terms for each of them.

* Software vendors, who take upstream Xen and package it into
something else and provide that software to users
* Users
 - "Private" users, who take Xen in some form and use it for themselves
 - Service providers, who use Xen to provide
infrastructure-as-a-service to others. (I think that's the right
*AAS...)
 - Service clients, who buy IAAS services from service providers.

The "Users" group can also be divided another way:
* Those that only consume software vendors' updates
* Those that apply their own patches to xen.org (or software vendors' updates)
I don't think "passive" and "active" users really conveys the right
idea, but it will have to do for now.

So the question is: given that we can't have all active users on the
pre-disclosure list, what justification is there for having some of
them on the list?

 -George

(I think we should pretty much assume that unless explicitly stated,
all opinions are the opinions of the individuals, not the
organizations they work for.)

Re: Security vulnerability process, and CVE-2012-0217

[Alan Cox]

> I think this should be done via (perhaps silent) consensus on the
> pre-discosure list. Leaving the embargo time line determination
> entirely to the discoverer isn't really suitable. 

Reality check. If the reporter decides to give you a date you are given a
date, if they decide to not negotiate that date then you can either meet
it or leave your customers vulnerable when it is published.

A lot of reporters are hostile to extensions and slowness because there
is a long history of process abuse elsewhere.

> I think having hardware vendors involved is really only necessary
> when hardware related issues need to be dealt with.

Which can be done on a case by case basis.

> As indicated above, this should not be permitted, due to
> fairness considerations. Otherwise we should not place
> restrictions on who might be on the list at least as an observer.

For any GPL code elements you cannot create a contractual basis for
preventing code release. It's a violation of the GPL "additional
restrictions" rule. At best you can trust everyone to be sensible.

> Just the same we allow individual vendors to communicate -
> acknowledge the fact that there is a vulnerability, identifiers, and
> expected embargo deadline.

You also need to end it the moment a third party publishes any info, or
you'll get into stupid situations where only those who signed up to it
can't talk about it (eg the infamous pentium lockup bug)

> > 8. Predisclosure subscription process, and email address criteria

Email is not a trustworthy medium. The linux security list  was in the
past intercepted.

> > We need a clear policy about releasing proof of concept exploits -
> > whether, when and who to.
> 
> This I think could (and perhaps should) be really be left to the
> discoverer, as this may be considered intellectual property.

They ought to be in the regression suite if possible.

On the fixes side also remember some vendors may choose to ship fixes
that differ from the "official" one.

Alan

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

On Wed, Jun 27, 2012 at 7:07 PM, Thomas Goirand <thomas@goirand.fr> wrote:
> On 06/20/2012 05:45 PM, George Dunlap wrote:
>> The only way this would work is if the predisclosure list consisted
>> exclusively of software providers, and specifically excluded service
>> providers.
> I agree, though you might have corner cases.
>
> What if you are *both* software and service provider (eg: I'm working on
> Debian and XCP, and my small company provides a hosted Xen service)?

If we do make a rule that only software providers can be on the list,
and not service providers, then ideally you should try to separate the
roles.  If you are on the list as a software provider, you should use
that information only to prepare patches; but not deploy them on your
own systems until the embargo date.

In a way, the question is very similar to asking, "I'm working on
Debian and XCP, and my best friend owns a small company that provides
a hosted Xen service."  If you told your friend about the
vulnerability, you would be breaking the security embargo (and giving
your friend an unfair advantage over other hosting services), and
would be at risk of being removed from the list if someone found out.
If you wear two "hats", as it were, the same would be true if your
developer "hat" told your service provider "hat": actually updating
your systems before the embargo would (I think) be considered breaking
the embargo, and would be giving yourself an unfair advantage over
other hosting services.

(All of the above discussion is, of course, only valid in the
hypothetical situation that we don't allow service providers to be on
the list.)

 -George

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

On Wed, Jun 20, 2012 at 9:49 AM, Jan Beulich <JBeulich@suse.com> wrote:
> As to downstreams, I think only direct ones should be involved in
> any decision making processes. Indirect ones might be allowed on
> the list, but exclusively as observers. Mis-use of the observer role
> (e.g. as in influencing those participating in decision making in
> undue ways), should it become known, should result in removal of
> list membership.

Do we have an outline of who should decide that someone has mis-used
their role, what kind of appeal process there is (if any), and when
and under what conditions a person or organization can be reinstated
to the list?

 -George

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 29.06.12 at 12:26, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> On Wed, Jun 20, 2012 at 9:49 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> As to downstreams, I think only direct ones should be involved in
>> any decision making processes. Indirect ones might be allowed on
>> the list, but exclusively as observers. Mis-use of the observer role
>> (e.g. as in influencing those participating in decision making in
>> undue ways), should it become known, should result in removal of
>> list membership.
> 
> Do we have an outline of who should decide that someone has mis-used
> their role, what kind of appeal process there is (if any), and when
> and under what conditions a person or organization can be reinstated
> to the list?

If common sense can't be applied here (and from how subsequent
discussion went I'm afraid it can't be), I don't think we can come
up with a definition that can't be put under attack by someone for
what would appear to be not completely illegitimate reasons.

Jan

Re: Security vulnerability process, and CVE-2012-0217

[Thomas Goirand]

On 06/29/2012 06:01 PM, George Dunlap wrote:
> On Wed, Jun 27, 2012 at 7:07 PM, Thomas Goirand <thomas@goirand.fr> wrote:
>> On 06/20/2012 05:45 PM, George Dunlap wrote:
>>> The only way this would work is if the predisclosure list consisted
>>> exclusively of software providers, and specifically excluded service
>>> providers.
>> I agree, though you might have corner cases.
>>
>> What if you are *both* software and service provider (eg: I'm working on
>> Debian and XCP, and my small company provides a hosted Xen service)?
> 
> If we do make a rule that only software providers can be on the list,
> and not service providers, then ideally you should try to separate the
> roles.  If you are on the list as a software provider, you should use
> that information only to prepare patches; but not deploy them on your
> own systems until the embargo date.
> 
> In a way, the question is very similar to asking, "I'm working on
> Debian and XCP, and my best friend owns a small company that provides
> a hosted Xen service."  If you told your friend about the
> vulnerability, you would be breaking the security embargo (and giving
> your friend an unfair advantage over other hosting services), and
> would be at risk of being removed from the list if someone found out.
> If you wear two "hats", as it were, the same would be true if your
> developer "hat" told your service provider "hat": actually updating
> your systems before the embargo would (I think) be considered breaking
> the embargo, and would be giving yourself an unfair advantage over
> other hosting services.
> 
> (All of the above discussion is, of course, only valid in the
> hypothetical situation that we don't allow service providers to be on
> the list.)
> 
>  -George

Exactly what I think as well. I'm happy you wrote the above.

Thomas

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

(speaking for myself)

On Thu, 2012-06-28 at 10:28 +0100, Lars Kurth wrote:
> >>/  1. Purpose of the process/
> >>
> >>/  The first point is that we think the security vulnerability process/
> >>/  document should have an explanation of what its goals are.  This would/
> >>/  have greatly assisted us when making some of the more difficult/
> >>/  decisions./
> >
> > The security vulnerability process document should most definitely
> > encapsulate both explicit guidance and broad tenets that can be used
> > to make tough calls. I think that these should be explicitly called
> > out in front matter as an evolutionary part of the process. Tenets
> > should always be open to being refined or redefined as Xen.org
> > projects grow and usage shifts.
> 
> I wanted to dive a little bit into the purpose of the process as the
> discussion will otherwise get stuck on detail.

Thanks for doing this, I think it was useful.

[...]

> Looking at the existing pre-disclosure list shows that it contains
> parties from all groups. This opens Xen.org up to criticism that some
> members of the pre-disclosure have an uncertain advantage, which has
> already been highlighted earlier in this discussion.
> 
> To be honest, I don't really see a way to satisfy all needs:

[...]

I think this is the crux of the issue.

Previously I was in favour of the pre-disclosure method but the more I
consider this the more I come to think that it is not the one best
suited to Xen or of the greatest overall benefit to all Xen users.

Pre-disclosure might be appropriate for projects whose downstreams are
generally software providers (e.g. Linux distros) but the high
proportion of Xen's immediate downstreams who are service providers
makes the balance somewhat different. In the case where you have a high
proportion of downstreams who are service providers the inherent
unfairness of pre-disclosure lists amplified since membership of the
pre-disclosure list allows those service providers to begin deploying
the fix without breaching the embargo, which is even more of an
advantage than just knowing about the issue and being able to prepare an
update for your users.

With that in mind I'm inclined to suggest that Xen.org would be better
off publishing security vulnerabilities publicly immediately once we
have a fix ready and tested/validated. This should be done ASAP and
ideally within a couple of weeks of being informed by the discloser (but
if we need longer we should take it to be sure we get it right). Before
that point only persons who need to know in order to contribute to the
fix should be given knowledge of the vulnerability.

This is basically the only way which is properly fair, since everyone
starts out on the same footing.

As soon as you have a pre-disclosure list you have to start dealing with
issues of fairness, who gets to be on the list, anti-competitiveness,
dealings between pre-disclosure members, issues of revoking and
reinstating privilege at which point you end up with complex policies
and associated bureaucracy. And at the end what you have still won't be
fair, because it can't be unless you let everyone in.

I don't think doing things this way any particular impact on "days of
risk", vulnerabilities basically fall into two sets wrt Blackhat's
knowledge about them AFAICS:

a) they are already known to the blackhats, who are either exploiting
them or not. Either way as soon the vulnerability is published they will
throw it away and start using some other zero day. Even if they do
continue to use it for a little while this is no different to them
continuing to exploit it during the embargo period. Either way our going
public without an embargo period hasn't helped them.

b) the don't know about it, but as soon as the vulnerability is
published it becomes uninteresting to them, they can either race with
the ongoing deployments to develop an exploit or they can continue to
use some other zero day exploit which we don't know about yet. This does
open a small potential window but the motivation for blackhats to start
developing an exploit for something which is already being actively
closed down seems to me to be quite small. This window is not
appreciably different to the window after the embargo period ends.

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Fri, 2012-06-29 at 11:01 +0100, George Dunlap wrote:
> On Wed, Jun 27, 2012 at 7:07 PM, Thomas Goirand <thomas@goirand.fr> wrote:
> > On 06/20/2012 05:45 PM, George Dunlap wrote:
> >> The only way this would work is if the predisclosure list consisted
> >> exclusively of software providers, and specifically excluded service
> >> providers.
> > I agree, though you might have corner cases.
> >
> > What if you are *both* software and service provider (eg: I'm working on
> > Debian and XCP, and my small company provides a hosted Xen service)?
> 
> If we do make a rule that only software providers can be on the list,
> and not service providers, then ideally you should try to separate the
> roles.  If you are on the list as a software provider, you should use
> that information only to prepare patches; but not deploy them on your
> own systems until the embargo date.
> 
> In a way, the question is very similar to asking, "I'm working on
> Debian and XCP, and my best friend owns a small company that provides
> a hosted Xen service."  If you told your friend about the
> vulnerability, you would be breaking the security embargo (and giving
> your friend an unfair advantage over other hosting services), and
> would be at risk of being removed from the list if someone found out.
> If you wear two "hats", as it were, the same would be true if your
> developer "hat" told your service provider "hat": actually updating
> your systems before the embargo would (I think) be considered breaking
> the embargo, and would be giving yourself an unfair advantage over
> other hosting services.
> 
> (All of the above discussion is, of course, only valid in the
> hypothetical situation that we don't allow service providers to be on
> the list.)

It also supposes that there would be some way to police this separation
-- how could you tell if a software vendor had given unfair advantage to
their friends, and how could you tell which one it was in order to
"punish" them?

The same problem exists if you allow service providers but insist that a
condition of membership is that they use the pre-disclosure period to
"prepare but not deploy" (i.e. to keep their hats separate). Other than
a suspicious wave of reboots across that providers infrastructure
(attributed to "routine maintenance") how would you know?

> 
>  -George
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

(speaking for myself) 

On Wed, 2012-06-20 at 11:32 +0100, Jan Beulich wrote:
> >>> On 20.06.12 at 11:45, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> > On Wed, Jun 20, 2012 at 9:49 AM, Jan Beulich <JBeulich@suse.com> wrote:
> >>>>> On 19.06.12 at 20:16, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> >>> The most controversial decision was whether the embargo date might be
> >>> extended after it had originally been set.  We resisted these
> >>> suggestions, referring to the process document, which does not
> >>> contemplate extending the disclosure period.  However, when a
> >>> predisclosure list member pressured the discoverer into requesting an
> >>> extension, we felt the best interpretation of the process document
> >>> required us to acquiesce.
> >>>
> >>> Specifically, of course, we as a team would like clearer guidance from
> >>> the process about whether, and if so under what circumstances, a
> >>> predisclosure period should be extended.
> >>
> >> I think this should be done via (perhaps silent) consensus on the
> >> pre-discosure list. Leaving the embargo time line determination
> >> entirely to the discoverer isn't really suitable. He might provide an
> >> initial suggestion, but we ought to set a period of time (say, a
> >> week or less) during which adjustment proposals can be made.
> > 
> > The problem with this is that extending the embargo helps providers
> > who are on the predisclosure list (a minority), but hurts those not on
> > the list (the vast majority).  So there's a moral hazard with what you
> > suggest: it is just way too easy for the people on the list to vote to
> > make their own lives easier, not considering the plight of those who
> > are not big enough or connected enough to be on the list.  Doing so
> > also favors large players and incumbents against small players; doing
> > so may well be considered anti-competitive and illegal in many
> > jurisdictions.
> > 
> > The only way this would work is if the predisclosure list consisted
> > exclusively of software providers, and specifically excluded service
> > providers.  It should be possible to include all software providers on
> > the list, since it's a relatively small number of entities.
> > Furthermore, since software providers presumably care about the
> > security of their customers, it would provide the incentive to make
> > the embargo as short as is reasonable.
> 
> You need to take this in context with my proposal to have only
> software vendors have a vote here, and to allow service
> providers at most an observing (maybe recommending to a
> limited degree) role.

(I'm going to reply elsewhere about my opinions on the existence of the
pre-disclosure list at all, but for the purposes of responding here I'll
assume we are going to keep it)

I think allowing the pre-disclosure list to have any "authority" in any
case would be a mistake. There are certainly scenarios where even a
vendor might want to delay the release for their own purposes. Perhaps
their QA team is too slow, or they want to delay the security update
until they have a new version of their s/w offering ready.

IMHO the predisclosure list should be purely about information flowing
out of Xen.org to the list participants and requests for clarification
going the other way. Membership of the predisclosure list should not
infer any other special privilege or control over the security embargo
process and we need to be explicit about that. Members of the list
*already* have the privilege of being on the list, they should not
expect to also have the privilege of having control over the timelines
etc of individual security vulnerabilities. Influence over the process
should happen during the formulation of the policy -- IOW in public
where everyone, not some select group of downstreams who happen to be on
the list, can have a say.

> 
> >>> 3. Decisionmaking
> >>>
> >>> It was suggested to us several times, and indeed seemed to be regarded
> >>> by some as an underlying principle, that the predisclosure list
> >>> members should be making the decisions about disclosure schedule etc.
> >>>
> >>> The question of who is to make the decisions, and on what basis, needs
> >>> to be addressed.
> >>
> >> See above. Leaving this to the discretion of the discoverer is
> >> neither open nor fair.
> > 
> > But there's a practical matter to consider here: if it's known that we
> > won't cooperate with the discoverer wrt disclosure times, discoverers
> > may simply not share their information with us.  I think that's the
> > main reason for the "do what the discoverer wants" rule.  I think
> > there needs to be some kind of a balance though: extending the embargo
> > less than 12 hours before the deadline is clearly not reasonable.
> 
> I still suggested that the discoverer gets a first shot at the
> embargo deadline. But if everyone else disagrees (i.e. the
> deadline was unreasonable), then it should be possible to ignore
> the discoverer's preference.

We already have wording abut "unreasonableness" but it's not entirely
clear what this means and/or when we would apply it.

I think the default of accepting the disclosers position is a good one.
We want to encourage people to report such bugs to us and taking control
away from them is a good way to discourage them.

Anyhow I'd expect that disclosers wanting to extend the period will be
the uncommon case. More normal would be a discloser who wanted to go
much faster than we do by default (which we can't do anything about and
in any case should respect).

> >>> Several members of the predisclosure list suggested that the
> >>> predisclosure period was too short.  Others were ready within the
> >>> predisclosure period's timeframe, and were disappointed to see it
> >>> extended and to have to delay their fixes.
> >>
> >> Setting a default period might be counter productive. There may
> >> be cases (for example had we wanted to fix XSA-9 properly)
> >> where developing a fix could take quite a bit of time. Not having
> >> a fix ready shouldn't, however, prevent initial announcements of
> >> a problem.
> > 
> > A default period can be considered a guideline.  In the case of a
> > particularly tricky fix, saying, "Normally we would set 2 weeks, but I
> > think in this case we should go for 3 or 4" is perfectly reasonable.
> > 
> > Since the embargo is really for software providers to test fixes,
> > perhaps we should suggest it should be "X business days after a fix is
> > released", rather than "X days after the vulnerability is disclosed"?
> 
> Yes, that sounds like a reasonable thing.

This is probably better, but also ties into the question of public
holidays in various territories. i.e. business day where...

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Thu, 2012-06-28 at 18:45 +0100, George Dunlap wrote:
[...]
> And I'm going to suggest something else which I think is true, and
> relevant to the discussion:
> 3. A large majority of deployed Xen instances are run by people not on
> the pre-disclosure list.

I agree. Although there may be a vendors with massive Xen deployments my
instinct is that they will be dwarfed by the sum of all the small
individual deployments.

> (I think we should pretty much assume that unless explicitly stated,
> all opinions are the opinions of the individuals, not the
> organizations they work for.)

Agreed (on my own behalf ;-)).

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

(speaking for myself) 
On Wed, 2012-06-20 at 09:49 +0100, Jan Beulich wrote:
> > 14. Early consideration of which other organisations to bring in
> >
> > Our process needs to have, very early on, an explicit step to of
> > deciding which other projects/organisations may also be vulnerable and
> > may therefore need to be part of the same disclosure process.  It also
> > needs to make sure that we ask for any help (for example from
> > upstreams or hardware vendors) as soon as possible.
> 
> Our security response team should only take our projects into
> consideration. Cross project vulnerabilities should be managed
> by entities set up to deal with this.

I'm generally of the same opinion -- if/when we discover that the impact
of an issue is wider than Xen instead of continuing to drive the process
forward ourselves we should "escalate" to an entity which is setup to
deal with such cross-project vulnerabilities.

What are the options here? CERT would be one, I'm sure there must be
others. We should probably pick one which has policies we are happy
with.

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 02.07.12 at 15:58, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> Pre-disclosure might be appropriate for projects whose downstreams are
> generally software providers (e.g. Linux distros) but the high
> proportion of Xen's immediate downstreams who are service providers
> makes the balance somewhat different. In the case where you have a high
> proportion of downstreams who are service providers the inherent
> unfairness of pre-disclosure lists amplified since membership of the
> pre-disclosure list allows those service providers to begin deploying
> the fix without breaching the embargo, which is even more of an
> advantage than just knowing about the issue and being able to prepare an
> update for your users.

But if a service provider takes on the extra effort to be an
immediate downstream, wouldn't it be fair to give it the
advantage over those who consume distros? (Of course, I'd
personally still want to give less of an advantage to those who
don't contribute back, but I realize that this is impossible to
implement in a reasonable way.)

Jan

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Mon, 2012-07-02 at 15:51 +0100, Jan Beulich wrote:
> >>> On 02.07.12 at 15:58, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > Pre-disclosure might be appropriate for projects whose downstreams are
> > generally software providers (e.g. Linux distros) but the high
> > proportion of Xen's immediate downstreams who are service providers
> > makes the balance somewhat different. In the case where you have a high
> > proportion of downstreams who are service providers the inherent
> > unfairness of pre-disclosure lists amplified since membership of the
> > pre-disclosure list allows those service providers to begin deploying
> > the fix without breaching the embargo, which is even more of an
> > advantage than just knowing about the issue and being able to prepare an
> > update for your users.
> 
> But if a service provider takes on the extra effort to be an
> immediate downstream, wouldn't it be fair to give it the
> advantage over those who consume distros?

I'm not sure why it would be. I can't see any link between the effort
taken to install Xen and level of security support one should expect.

Consuming Xen via a distro is a completely rational and reasonable thing
to do.

>  (Of course, I'd
> personally still want to give less of an advantage to those who
> don't contribute back, but I realize that this is impossible to
> implement in a reasonable way.)

While I can appreciate the sentiment I think that even if we could
achieve this we should not. The provision of security updates should not
be used as either a carrot or a stick.

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 02.07.12 at 15:59, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2012-06-20 at 11:32 +0100, Jan Beulich wrote:
>> >>> On 20.06.12 at 11:45, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>> > On Wed, Jun 20, 2012 at 9:49 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> >>>>> On 19.06.12 at 20:16, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>> >>> The most controversial decision was whether the embargo date might be
>> >>> extended after it had originally been set.  We resisted these
>> >>> suggestions, referring to the process document, which does not
>> >>> contemplate extending the disclosure period.  However, when a
>> >>> predisclosure list member pressured the discoverer into requesting an
>> >>> extension, we felt the best interpretation of the process document
>> >>> required us to acquiesce.
>> >>>
>> >>> Specifically, of course, we as a team would like clearer guidance from
>> >>> the process about whether, and if so under what circumstances, a
>> >>> predisclosure period should be extended.
>> >>
>> >> I think this should be done via (perhaps silent) consensus on the
>> >> pre-discosure list. Leaving the embargo time line determination
>> >> entirely to the discoverer isn't really suitable. He might provide an
>> >> initial suggestion, but we ought to set a period of time (say, a
>> >> week or less) during which adjustment proposals can be made.
>> > 
>> > The problem with this is that extending the embargo helps providers
>> > who are on the predisclosure list (a minority), but hurts those not on
>> > the list (the vast majority).  So there's a moral hazard with what you
>> > suggest: it is just way too easy for the people on the list to vote to
>> > make their own lives easier, not considering the plight of those who
>> > are not big enough or connected enough to be on the list.  Doing so
>> > also favors large players and incumbents against small players; doing
>> > so may well be considered anti-competitive and illegal in many
>> > jurisdictions.
>> > 
>> > The only way this would work is if the predisclosure list consisted
>> > exclusively of software providers, and specifically excluded service
>> > providers.  It should be possible to include all software providers on
>> > the list, since it's a relatively small number of entities.
>> > Furthermore, since software providers presumably care about the
>> > security of their customers, it would provide the incentive to make
>> > the embargo as short as is reasonable.
>> 
>> You need to take this in context with my proposal to have only
>> software vendors have a vote here, and to allow service
>> providers at most an observing (maybe recommending to a
>> limited degree) role.
> 
> (I'm going to reply elsewhere about my opinions on the existence of the
> pre-disclosure list at all, but for the purposes of responding here I'll
> assume we are going to keep it)
> 
> I think allowing the pre-disclosure list to have any "authority" in any
> case would be a mistake. There are certainly scenarios where even a
> vendor might want to delay the release for their own purposes. Perhaps
> their QA team is too slow, or they want to delay the security update
> until they have a new version of their s/w offering ready.
> 
> IMHO the predisclosure list should be purely about information flowing
> out of Xen.org to the list participants and requests for clarification
> going the other way. Membership of the predisclosure list should not
> infer any other special privilege or control over the security embargo
> process and we need to be explicit about that. Members of the list
> *already* have the privilege of being on the list, they should not
> expect to also have the privilege of having control over the timelines
> etc of individual security vulnerabilities. Influence over the process
> should happen during the formulation of the policy -- IOW in public
> where everyone, not some select group of downstreams who happen to be on
> the list, can have a say.

But in that case, negotiation of an embargo period becomes a
dead thing altogether - the discoverer saying something, it
would need to be followed, and in the absence of him stating
anything defaults would need to be applied. No other option
would exist, as no-one is given authority. Giving the security
team members authority in this respect would already make them
susceptible to pressure put onto them by other list members.

Jan

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Mon, 2012-07-02 at 15:58 +0100, Jan Beulich wrote:
> >>> On 02.07.12 at 15:59, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Wed, 2012-06-20 at 11:32 +0100, Jan Beulich wrote:
> >> >>> On 20.06.12 at 11:45, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> >> > On Wed, Jun 20, 2012 at 9:49 AM, Jan Beulich <JBeulich@suse.com> wrote:
> >> >>>>> On 19.06.12 at 20:16, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> >> >>> The most controversial decision was whether the embargo date might be
> >> >>> extended after it had originally been set.  We resisted these
> >> >>> suggestions, referring to the process document, which does not
> >> >>> contemplate extending the disclosure period.  However, when a
> >> >>> predisclosure list member pressured the discoverer into requesting an
> >> >>> extension, we felt the best interpretation of the process document
> >> >>> required us to acquiesce.
> >> >>>
> >> >>> Specifically, of course, we as a team would like clearer guidance from
> >> >>> the process about whether, and if so under what circumstances, a
> >> >>> predisclosure period should be extended.
> >> >>
> >> >> I think this should be done via (perhaps silent) consensus on the
> >> >> pre-discosure list. Leaving the embargo time line determination
> >> >> entirely to the discoverer isn't really suitable. He might provide an
> >> >> initial suggestion, but we ought to set a period of time (say, a
> >> >> week or less) during which adjustment proposals can be made.
> >> > 
> >> > The problem with this is that extending the embargo helps providers
> >> > who are on the predisclosure list (a minority), but hurts those not on
> >> > the list (the vast majority).  So there's a moral hazard with what you
> >> > suggest: it is just way too easy for the people on the list to vote to
> >> > make their own lives easier, not considering the plight of those who
> >> > are not big enough or connected enough to be on the list.  Doing so
> >> > also favors large players and incumbents against small players; doing
> >> > so may well be considered anti-competitive and illegal in many
> >> > jurisdictions.
> >> > 
> >> > The only way this would work is if the predisclosure list consisted
> >> > exclusively of software providers, and specifically excluded service
> >> > providers.  It should be possible to include all software providers on
> >> > the list, since it's a relatively small number of entities.
> >> > Furthermore, since software providers presumably care about the
> >> > security of their customers, it would provide the incentive to make
> >> > the embargo as short as is reasonable.
> >> 
> >> You need to take this in context with my proposal to have only
> >> software vendors have a vote here, and to allow service
> >> providers at most an observing (maybe recommending to a
> >> limited degree) role.
> > 
> > (I'm going to reply elsewhere about my opinions on the existence of the
> > pre-disclosure list at all, but for the purposes of responding here I'll
> > assume we are going to keep it)
> > 
> > I think allowing the pre-disclosure list to have any "authority" in any
> > case would be a mistake. There are certainly scenarios where even a
> > vendor might want to delay the release for their own purposes. Perhaps
> > their QA team is too slow, or they want to delay the security update
> > until they have a new version of their s/w offering ready.
> > 
> > IMHO the predisclosure list should be purely about information flowing
> > out of Xen.org to the list participants and requests for clarification
> > going the other way. Membership of the predisclosure list should not
> > infer any other special privilege or control over the security embargo
> > process and we need to be explicit about that. Members of the list
> > *already* have the privilege of being on the list, they should not
> > expect to also have the privilege of having control over the timelines
> > etc of individual security vulnerabilities. Influence over the process
> > should happen during the formulation of the policy -- IOW in public
> > where everyone, not some select group of downstreams who happen to be on
> > the list, can have a say.
> 
> But in that case, negotiation of an embargo period becomes a
> dead thing altogether - the discoverer saying something, it
> would need to be followed, and in the absence of him stating
> anything defaults would need to be applied. No other option
> would exist, as no-one is given authority.

Hrmm yes. We should certainly make it explicit that in the absence of an
opinion from the discloser the security team can set the time line it
thinks is appropriate, suing the default as a guideline only and
endevouring to keep it as short as possible.

Matt previously posted a link to
https://www.ocert.org/disclosure_policy.html which contains 

"""
   The following time frames regulate oCERT embargo proposals:

   - 7 days, in case the issue is already well narrowed down and
     tested, requiring trivial configuration and/or code change

   - 14 days, standard embargo for most cases

   - 30 days, in case of critical and complex vulnerabilities
     (example, trivial exploitation of administrative privileges on a
     static library affecting a large number of packages), and with
     the agreement of all parties

   - under extremely exceptional circumstances, if the oCERT Team and
     all the parties involved feel the need for longer time, a 2
     months embargo can be applied, in this case we would clearly
     document the decision for public review

   - in any circumstance reporter preference will always be honoured
     in case a joint agreement is not reached, as oCERT would be
     anyway unable to force its embargo
"""

which seems like a reasonable set of rules to me, modulo discussion
about the specific time lines.

>  Giving the security
> team members authority in this respect would already make them
> susceptible to pressure put onto them by other list members.

I think we can a) make it clear that list members don't have that
authority and b) rely on the security team to resist such pressure
better than disclosers and/or other list members.

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[Alan Cox]

> It also supposes that there would be some way to police this separation
> -- how could you tell if a software vendor had given unfair advantage to
> their friends, and how could you tell which one it was in order to
> "punish" them?

You've equally got to deal with the vendors whose employees decide to
tell their friends. Common sense will always apply.

> The same problem exists if you allow service providers but insist that a
> condition of membership is that they use the pre-disclosure period to
> "prepare but not deploy" (i.e. to keep their hats separate). Other than
> a suspicious wave of reboots across that providers infrastructure
> (attributed to "routine maintenance") how would you know?

The bad guys monitor this.

The other way you'll know is when blackhats observe a given provider is
immune to an exploit infeasibly fast. At which point that will slowly
become general knowledge.

Alan

Re: Security vulnerability process, and CVE-2012-0217

[Alan Cox]

> I think the default of accepting the disclosers position is a good one.
> We want to encourage people to report such bugs to us and taking control
> away from them is a good way to discourage them.

You do need a standard answer for when they don't.

> This is probably better, but also ties into the question of public
> holidays in various territories. i.e. business day where...

On a global basis you can't win. Saturday/Sunday are out, a chunk of the
middle of summer the French are all away, then Chinese have golden week
and so on and by the time you've blocked them all in your calendar is
basically full.

It's a global community so the counterpoint is that while someone is
always on holiday, someone else is always at work.

Alan

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Mon, 2012-07-02 at 16:17 +0100, Alan Cox wrote:
> > I think the default of accepting the disclosers position is a good one.
> > We want to encourage people to report such bugs to us and taking control
> > away from them is a good way to discourage them.
> 
> You do need a standard answer for when they don't.

Agreed.

> > This is probably better, but also ties into the question of public
> > holidays in various territories. i.e. business day where...
> 
> On a global basis you can't win. Saturday/Sunday are out, a chunk of the
> middle of summer the French are all away, then Chinese have golden week
> and so on and by the time you've blocked them all in your calendar is
> basically full.
> 
> It's a global community so the counterpoint is that while someone is
> always on holiday, someone else is always at work.

This is true. Perhaps rather than consider all consumers we just need to
give consideration to those actually involved in creating / sending out
the advisory i.e. the security@ team since having one of them be away at
a critical juncture can throw a bit of a spanner into the works.

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

On Mon, Jul 2, 2012 at 2:59 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2012-06-29 at 11:01 +0100, George Dunlap wrote:
>> On Wed, Jun 27, 2012 at 7:07 PM, Thomas Goirand <thomas@goirand.fr> wrote:
>> > On 06/20/2012 05:45 PM, George Dunlap wrote:
>> >> The only way this would work is if the predisclosure list consisted
>> >> exclusively of software providers, and specifically excluded service
>> >> providers.
>> > I agree, though you might have corner cases.
>> >
>> > What if you are *both* software and service provider (eg: I'm working on
>> > Debian and XCP, and my small company provides a hosted Xen service)?
>>
>> If we do make a rule that only software providers can be on the list,
>> and not service providers, then ideally you should try to separate the
>> roles.  If you are on the list as a software provider, you should use
>> that information only to prepare patches; but not deploy them on your
>> own systems until the embargo date.
>>
>> In a way, the question is very similar to asking, "I'm working on
>> Debian and XCP, and my best friend owns a small company that provides
>> a hosted Xen service."  If you told your friend about the
>> vulnerability, you would be breaking the security embargo (and giving
>> your friend an unfair advantage over other hosting services), and
>> would be at risk of being removed from the list if someone found out.
>> If you wear two "hats", as it were, the same would be true if your
>> developer "hat" told your service provider "hat": actually updating
>> your systems before the embargo would (I think) be considered breaking
>> the embargo, and would be giving yourself an unfair advantage over
>> other hosting services.
>>
>> (All of the above discussion is, of course, only valid in the
>> hypothetical situation that we don't allow service providers to be on
>> the list.)
>
> It also supposes that there would be some way to police this separation
> -- how could you tell if a software vendor had given unfair advantage to
> their friends, and how could you tell which one it was in order to
> "punish" them?
>
> The same problem exists if you allow service providers but insist that a
> condition of membership is that they use the pre-disclosure period to
> "prepare but not deploy" (i.e. to keep their hats separate). Other than
> a suspicious wave of reboots across that providers infrastructure
> (attributed to "routine maintenance") how would you know?

I took Thomas' question as, "What is the right thing to do?"  There's
certainly no ability for us to try to police everyone.  In a sense the
pre-disclosure list is already built on trust.  As Alan said, there's
not much we can do legally to enforce a restriction on releasing a
GPL'ed patch; all we can do is remove the offender from the list
post-facto.  And as someone else said, the bigger the company, the
more people may end up hearing about the vulnerability, which
increases the chance that one of them will be using the information
for unapproved purposes (such as writing an exploit for it).

When we have to rely on trust like this:
* I think most people will follow the guidelines if they think everyone else is
* A few "cheaters" are probably inevitable; as long as they're a tiny
minority, things will still work fine
* For those for whom doing the right thing is not a motivating factor,
the risk of being discovered and being excluded , or the risk of a
complete collapse of trust (which in this case would probably make the
pre-disclosure list go away) can make it in their self-interest to
follow the rules.

So I think as long as the expectations are made clear, there is a
chance that an "honor system" could work.

That said, it still introduces unfairness; and will certainly cause
the xen.org security team more headache.  And if your analysis of
blackhat behavior is correct, that releasing a vulnerability by and
large only *reduces* risk, then there's no real need for one.

 -George

Re: Security vulnerability process, and CVE-2012-0217

[Stefano Stabellini]

On Tue, 3 Jul 2012, George Dunlap wrote:
> That said, it still introduces unfairness

Considering that the members of the security disclosure list are public
(http://www.xen.org/projects/security_vulnerability_process.html) and
considering that some of them are service providers, if I am a costumer,
why would I ever choose a provider that is not in that list?

Having that list on the website is like writing: "please choose one of
the providers in the list below as they have a better security
response".

Re: Security vulnerability process, and CVE-2012-0217

[Matt Wilson]

On Thu, Jun 28, 2012 at 02:28:23AM -0700, Lars Kurth wrote:
> On Sat, 23 Jun 2012 12:42:14 -0700, Matt Wilson <msw@amazon.com> wrote:
> > On Tue, Jun 19, 2012 at 11:16:05AM -0700, Ian Jackson wrote: 
> > >/  1. Purpose of the process/
> > >
> > >/  The first point is that we think the security vulnerability process/
> > >/  document should have an explanation of what its goals are.  This would/
> > >/  have greatly assisted us when making some of the more difficult/
> > >/  decisions./
> >
> > The security vulnerability process document should most definitely
> > encapsulate both explicit guidance and broad tenets that can be used
> > to make tough calls. I think that these should be explicitly called
> > out in front matter as an evolutionary part of the process. Tenets
> > should always be open to being refined or redefined as Xen.org
> > projects grow and usage shifts.
> 
> I wanted to dive a little bit into the purpose of the process as the
> discussion will otherwise get stuck on detail. Also into the actors
> in a security vulnerability and their interest.
> 
> 1.1: The Xen.org Security team, whose task is to administer the process
> and act as referee if needed: the process should provide clarity and
> ensure that the team can be seen as referee. The process needs to be
> clear enough to avoid a chance that members of the team are accused of
> being partial.
> 
> 1.2: Discoverer of the vulnerability: The process must provide an
> incentive for the discover to report the issue to the project. If we
> get the process wrong, the consequence will be that vulnerabilities
> will not be reported to us. That would be detrimental to all
> stake-holders as it will increase days-of-risk for everybody else.
> E.g. if the embargo period is too long. Not sure what other factors
> motivate a discoverer.

I agree. The process should align with the common goal, shared among
all the groups that you've identified, of improving security. We can
only expect to attract disclosures from discoverers that agree that
"coordinated disclosure" (or "responsible disclosure" if the term is
acceptable) improves security for everyone.

We can't expect to attract discoverers that feel that "full
disclosure" is the best way to improve security, nor can we expect to
attract discoverers that want to be compensated through either black
markets or white markets for zero-day exploits.

I think that 1) calling out the important role of the discoverer in
the process, and 2) stating that proposed disclosure time lines are
negotiable should be sufficient in retaining discoverers that choose
to participate in coordinated disclosure. Also, a track record of
rapidly addressing reported vulnerabilities should help.

For further reading, a fairly detailed exploration of the motives of
discoverers is covered in a paper by Stefan Frei et al., "Modelling
the Security Ecosystem - The Dynamics of (In)Security". [1]

> 1.3: Developers within the project, who will be task to find a fix
> as quickly as possible.
> 
> 1.4: 3rd parties that may need to be contacted in order to develop a
> fix to understand the issue and advise the security team (in the case
> of CVE-2012-0217 hardware vendors).
> 
> 1.5: Developers of downstream projects/distros consuming Xen and
> distributing it (FOSS or commercial), who will apply the fix to
> their project/distro.

The distinction between 1.5 and 1.6 isn't always clear. For example,
Amazon Web Services builds a Linux image for use in EC2 called the
Amazon Linux AMI [2]. Some of the issues coordinated through
security@xen.org may involve the domU side of PV drivers. The fix will
need to be applied to the kernel in the AMI, just as it would need to
be applied to Debian, Ubuntu, Fedora, RHEL, SLES, etc.

Additionally, members of group 1.6 may have the same amount of
porting, building and validation work to do as a traditional Linux
"distro" when addressing an issue in the hypervisor, dom0 or related
sub-component.

> 1.6: Other direct consumers of Xen (e.g. service providers). Their
> main goal of the process would be to reduce days-of-risk for
> themselves. The issue being that deploying a fix can take a long time.
> Another issue is that providing a fix before somebody else is a
> competitive advantage.

The notion that service providers are only concerned with reducing
risk for themselves seems short sighted. In my view, the real concern
is to reduce days-of-risk for everyone that relies, in any way, on the
infrastructure supplied by IaaS providers. That includes a service
provider's direct customer, and that customer's users or customers,
and so on.

> 1.7: Indirect consumers of Xen. With all the same motivations than
> direct consumers.
> 
> A couple of observations:
> 1) The further you get down the list, the more people are involved,
> the greater the risk that information will leak.

I mostly agree, so long as 1.1 (security@xen.org) and 1.2 (discoverer)
are reversed.

> 2) Groups 1.1 - 1.4 necessarily need to be involved to create a fix,
> as otherwise there won't be a fix for the other groups.
> 3) Groups 1.6 - 1.7 are directly impacted by days-of-risk
> 4) Groups 1.1, 1.3 & 1.5 are impacted insofar that the reputation
> of their organisation may be damaged if the situation is not handled
> well.

Group 1.6 may also have reputation impacts.

> Of course the ultimate goal of the process should be to minimize
> days-of-risk for everyone. To do this, the process has to balance the
> following factors to reduce days-of-risk
> 
> A) Provide incentive for vulnerabilities to be reported to Xen.org
> security team in the first place
> B) Time it takes to develop a fix
> C) Time it takes downstream projects/distros to apply it
> D) Time it takes to deploy fixes for consumers (1.6 & 1.7)

See my comment above about fix integration for group 1.6.

> E) Reduce the possibility of a leak (keep pre-disclosure list small)
> F) Avoid the perception that members of the pre-disclosure list have
> an unfair advantage
>   
> Looking at the existing pre-disclosure list shows that it contains
> parties from all groups. This opens Xen.org up to criticism that some
> members of the pre-disclosure have an uncertain advantage, which has
> already been highlighted earlier in this discussion.

I think that reworking the membership criteria and a transparent
membership request process, similar to how subscribe / unsubscribe
requests to the "distros" and "linux-distros" mailing lists [3], can
solve this. Or, address it as well as the distro lists have.

> To be honest, I don't really see a way to satisfy all needs:
> - Restrict membership of disclosure lists to stake-holders 1.1, 1.2,
>    1.3 and 1.5 with members of 1.4 drawn in as needed and full public
>    disclosure afterwards as to who was consulted.

Indeed, this won't satisfy the needs of group 1.6 if my comments above
are generally accepted.

> - Of course it may be desirable to get advice from members of groups
>    1.6 and 1.7. Or is it? If it is, a different approach may be to have
>    a merit rather than size based selection criteria for members of 1.6
>    and 1.7 (e.g. something along the lines of "contributions" to the
>    community, but that would need to be measurable - or even some sort
>    of elections). But it is hard to see how this would work in practice.
>    Of course such an would have the advantage that a member of that group
>    that used mbership to gain an unfair advantage over others could be
>    removed from the pre-disclosure list.

I think that feedback and testing from members of the pre-disclosure
list has been beneficial in the past and is desirable.

> - Another possible approach may be a two-stage pre-disclosure
>    - Stage 1: Groups 1.1, 1.2, 1.3 and 1.5 with 1.4 as needed
>    - Stage 2: Groups 1.6 and 1.7 (with a relatively weak membership
>      criterion such as being a service provider - but then how does
>      this differ from being public and who would administer?)

I don't think that this is substantially different than the first
approach.

> - Another option may be to delegate more difficult issues on
>    principle to an independent organisation such as OCERT.
> 
> Best Regards
> Lars
> P.S.: I just wanted to make clear that these are my personal views and reflect
> in no way any position of Xen.org or my employer.

Thanks for raising the discussion up a level, Lars. I think we should
be able to come to a consensus on these broad topics fairly quickly,
then start to address some of the finer details.

Matt

[1] http://www.techzoom.net/papers/weis_security_ecosystem_2009.pdf
[2] http://aws.amazon.com/amazon-linux-ami/
[3] http://oss-security.openwall.org/wiki/mailing-lists/distros

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Thu, 2012-06-28 at 19:30 +0100, Alan Cox wrote:
> 
> > > 8. Predisclosure subscription process, and email address criteria
> 
> Email is not a trustworthy medium. The linux security list  was in the
> past intercepted. 

I think it would be wise to add encryption (and the requirement to
provide a key) to the pre-disclosure list. I wonder if mailman has
per-subscriber encryption capabilities.

If not then we should consider moving this particular list to a list
manager which can. Apparently whatever the linux-distros list uses can
do this (judging from
http://oss-security.openwall.org/wiki/mailing-lists/distros)

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[John Haxby]

On 04/07/12 10:27, Ian Campbell wrote:
> On Thu, 2012-06-28 at 19:30 +0100, Alan Cox wrote:
>> > 
>>>> > > > 8. Predisclosure subscription process, and email address criteria
>> > 
>> > Email is not a trustworthy medium. The linux security list  was in the
>> > past intercepted. 
> I think it would be wise to add encryption (and the requirement to
> provide a key) to the pre-disclosure list. I wonder if mailman has
> per-subscriber encryption capabilities.
>
> If not then we should consider moving this particular list to a list
> manager which can. Apparently whatever the linux-distros list uses can
> do this (judging from
> http://oss-security.openwall.org/wiki/mailing-lists/distros)

That's correct.   linux-distros (and distros) also precludes having
exploder lists in organisations.   That's not generally going to be a
problem -- you're not likely to have more than a handful of people on
the list even in largish organisations.

The linux-distros and distros lists are, I believe, based around
something the list manager maintains.  You'd have to ask him about how
it works.

jch

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Tue, 2012-07-03 at 23:03 +0100, Matt Wilson wrote:
> On Thu, Jun 28, 2012 at 02:28:23AM -0700, Lars Kurth wrote:
> > On Sat, 23 Jun 2012 12:42:14 -0700, Matt Wilson <msw@amazon.com> wrote:
> > > On Tue, Jun 19, 2012 at 11:16:05AM -0700, Ian Jackson wrote: 
> > > >/  1. Purpose of the process/
> > > >
> > > >/  The first point is that we think the security vulnerability process/
> > > >/  document should have an explanation of what its goals are.  This would/
> > > >/  have greatly assisted us when making some of the more difficult/
> > > >/  decisions./
> > >
> > > The security vulnerability process document should most definitely
> > > encapsulate both explicit guidance and broad tenets that can be used
> > > to make tough calls. I think that these should be explicitly called
> > > out in front matter as an evolutionary part of the process. Tenets
> > > should always be open to being refined or redefined as Xen.org
> > > projects grow and usage shifts.
> > 
> > I wanted to dive a little bit into the purpose of the process as the
> > discussion will otherwise get stuck on detail. Also into the actors
> > in a security vulnerability and their interest.
> > 
> > 1.1: The Xen.org Security team, whose task is to administer the process
> > and act as referee if needed: the process should provide clarity and
> > ensure that the team can be seen as referee. The process needs to be
> > clear enough to avoid a chance that members of the team are accused of
> > being partial.
> > 
> > 1.2: Discoverer of the vulnerability: The process must provide an
> > incentive for the discover to report the issue to the project. If we
> > get the process wrong, the consequence will be that vulnerabilities
> > will not be reported to us. That would be detrimental to all
> > stake-holders as it will increase days-of-risk for everybody else.
> > E.g. if the embargo period is too long. Not sure what other factors
> > motivate a discoverer.
> 
> I agree. The process should align with the common goal, shared among
> all the groups that you've identified, of improving security. We can
> only expect to attract disclosures from discoverers that agree that
> "coordinated disclosure" (or "responsible disclosure" if the term is
> acceptable) improves security for everyone.
> 
> We can't expect to attract discoverers that feel that "full
> disclosure" is the best way to improve security,

I don't think this is necessarily true. There is a spectrum between
"full disclosure" and "coordinated". Part of the purpose of giving the
discloser the final say on timelines is precisely to allow people who
hold views towards the full end of the spectrum to feel comfortable
coming to us. If they want to do immediate disclosure then that is what
we will do, per the current policy at least, and I think this property
needs to be retained.

Having such full disclosure come through the proper Xen communication
channels in a timely manner is clearly preferable to having it be
published on some security researchers blog or in a paper which many of
our users may never read and which we ourselves may not find out about
until some (hopefully short, but non-zero) time later.

Now maybe not all such people will want to come to us, some will
obviously just publish without any warning, but we should make it clear
that if they come to us we will respect their wishes here so that some
may feel comfortable giving us a heads up. Even if they say "I'm
publishing this in 1 hour" that will allow us to at least do
*something*, e.g. repost their disclosure and make people aware while we
develop the fix.

>  nor can we expect to
> attract discoverers that want to be compensated through either black
> markets or white markets for zero-day exploits.
> 
> I think that 1) calling out the important role of the discoverer in
> the process, and 2) stating that proposed disclosure time lines are
> negotiable should be sufficient in retaining discoverers that choose
> to participate in coordinated disclosure. Also, a track record of
> rapidly addressing reported vulnerabilities should help.
> 
> For further reading, a fairly detailed exploration of the motives of
> discoverers is covered in a paper by Stefan Frei et al., "Modelling
> the Security Ecosystem - The Dynamics of (In)Security". [1]
> 
> > 1.3: Developers within the project, who will be task to find a fix
> > as quickly as possible.
> > 
> > 1.4: 3rd parties that may need to be contacted in order to develop a
> > fix to understand the issue and advise the security team (in the case
> > of CVE-2012-0217 hardware vendors).
> > 
> > 1.5: Developers of downstream projects/distros consuming Xen and
> > distributing it (FOSS or commercial), who will apply the fix to
> > their project/distro.
> 
> The distinction between 1.5 and 1.6 isn't always clear. For example,
> Amazon Web Services builds a Linux image for use in EC2 called the
> Amazon Linux AMI [2]. Some of the issues coordinated through
> security@xen.org may involve the domU side of PV drivers. The fix will
> need to be applied to the kernel in the AMI, just as it would need to
> be applied to Debian, Ubuntu, Fedora, RHEL, SLES, etc.

Fixes to Linux (even ones relating to Xen) should IMHO go via the Linux
security process, not the Xen.org one, since Linux is not a Xen project.
Presumably all those distros etc are set up to deal with that channel in
the appropriate manner.

Similarly we would not expect to release e.g. *BSD updates via this
process -- those would be done by the relevant OS vendor following their
normal processes.

We could (and probably should) consider writing into the policy that we
will communicate Xen related security updates which come out of other
projects (*BSD, Linux, qemu etc) to our users. i.e. forwarding them to
our relevant lists.

> Additionally, members of group 1.6 may have the same amount of
> porting, building and validation work to do as a traditional Linux
> "distro" when addressing an issue in the hypervisor, dom0 or related
> sub-component.
> 
> > 1.6: Other direct consumers of Xen (e.g. service providers). Their
> > main goal of the process would be to reduce days-of-risk for
> > themselves. The issue being that deploying a fix can take a long time.
> > Another issue is that providing a fix before somebody else is a
> > competitive advantage.
> 
> The notion that service providers are only concerned with reducing
> risk for themselves seems short sighted. In my view, the real concern
> is to reduce days-of-risk for everyone that relies, in any way, on the
> infrastructure supplied by IaaS providers. That includes a service
> provider's direct customer, and that customer's users or customers,
> and so on.

I think that by themselves Lars was suggesting that IaaS providers are
general most interested in their own direct customers, that customer's
user or customers etc. They aren't necessarily interested in days of
risk for some other IaaS providers customers, their customer's customers
etc.

This is especially relevant when some IaaS providers are not eligible to
be on the predisclosure list. As we have seen in this most recent
embargo members of the predisclosure list are quite willing to lobby for
extensions to the embargo for their own benefit (or at best the benefit
of only the pre-disclosure members) despite the fact that this leaves
everyone else (the "silent majority") vulnerable, and unknowingly so,
for longer.

[...]

> > Looking at the existing pre-disclosure list shows that it contains
> > parties from all groups. This opens Xen.org up to criticism that some
> > members of the pre-disclosure have an uncertain advantage, which has
> > already been highlighted earlier in this discussion.
> 
> I think that reworking the membership criteria and a transparent
> membership request process, similar to how subscribe / unsubscribe
> requests to the "distros" and "linux-distros" mailing lists [3], can
> solve this. Or, address it as well as the distro lists have.

Do you have any specific criteria in mind for membership?

linux-distros is "membership limited to operating system distribution
security contacts".

Which of groups 1.5, 1.6 and 1.7 should be allowed? I think that unless
we open this list to anyone in 1.6 and 1.7, regardless of size or other
factors, the pre-disclosure concept is unworkable.

Ian.

Re: Security vulnerability process, and CVE-2012-0217

[Stefano Stabellini]

On Tue, 3 Jul 2012, Matt Wilson wrote:
> > Looking at the existing pre-disclosure list shows that it contains
> > parties from all groups. This opens Xen.org up to criticism that some
> > members of the pre-disclosure have an uncertain advantage, which has
> > already been highlighted earlier in this discussion.
> 
> I think that reworking the membership criteria and a transparent
> membership request process, similar to how subscribe / unsubscribe
> requests to the "distros" and "linux-distros" mailing lists [3], can
> solve this. Or, address it as well as the distro lists have.

I agree.

As we can see from the list at

http://oss-security.openwall.org/wiki/mailing-lists/distros

both big companies like Oracle and very small, entirely not-for-profit,
groups like Frugalware are present.
Therefore I think that the size of the company or the entity that is
applying for subscription should NOT be the criteria to based the
acceptance upon.

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

Let me toss another possibility out there.  So far this discussion has
assumed that we can't have all interested parties on a list.  Is that
true?  Could we have a list that either anyone can join, or limited by
some easily verifiable criteria (e.g., has a website, a company e-mail
in the same domain, and can provide a scan of some official document)?

Such a list would definitely be lower security than a more restricted
list.  So there would be two questions:
1. What would a reasonable criteria for this kind of list be?
2. How would disclosing to this list fit within the embargo period,
and with the discloser's wishes (if any)?

I think #2 would probably be:
* Make sure the disclosure knows about the open nature of the list,
and abide by their wishes.  If the discloser considers the list to be
a public disclosure, they may ask us not to announce to the list until
the end of the embargo period, or until some period of time before the
end (say, 1 week).
* By default, suggest disclosing to the list as soon as we have a fix
available, and then making a public announcement (on blogs / press
releases / whatever) some time afterwards (say, 1 or 2 weeks).

This is certainly more fair than options which have a list but limit
the membership artificially by size.  For those who think a public
announcement does not significantly increase the risk, this is will be
very similar to what they would have if we decided not to have a list
at all.  However, for those who believe that publicly announcing the
vulnerability greatly increases the risk of exploitation, it will give
them some extra time to patch their systems until that happens.

There will be administrative work on the part of someone at xen.org to
determine who is on the list or not; but it shouldn't require too much
extra effort on the part of the security team.

The only caveat I can think of is that it may increase the risk,
during the time between the predisclosure and the public announcement,
for those not on the list.  We can basically assume that the list will
have some blackhats.  If the timeframe is anywhere near what some
people have asked for (e.g., 3-4 weeks), then it might become
worthwhile for people to develop an exploit to take advantage of
people during that timeframe.  This might be an acceptable cost, since
those people *could* be on the list of they wanted.

Thoughts?

 -George

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 04.07.12 at 14:36, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> The only caveat I can think of is that it may increase the risk,
> during the time between the predisclosure and the public announcement,
> for those not on the list.  We can basically assume that the list will
> have some blackhats.  If the timeframe is anywhere near what some
> people have asked for (e.g., 3-4 weeks), then it might become
> worthwhile for people to develop an exploit to take advantage of
> people during that timeframe.  This might be an acceptable cost, since
> those people *could* be on the list of they wanted.

Being on the list doesn't make you non-susceptible. Such an
approach, imo, would need to imply permission to anyone on
the list to deploy a fix as soon as it is available. But since
distros can't ship binaries without also making sources available,
that's a contradiction by itself.

Jan

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

On Wed, Jul 4, 2012 at 1:52 PM, Jan Beulich <JBeulich@suse.com> wrote:
> Being on the list doesn't make you non-susceptible. Such an
> approach, imo, would need to imply permission to anyone on
> the list to deploy a fix as soon as it is available. But since
> distros can't ship binaries without also making sources available,
> that's a contradiction by itself.

Yes, preventing vendors from shipping until the public disclosure date
would discriminates against "vendor-supplied" users in favor of
"self-supplied" users (i.e., those who download and build their own
directly from xen.org).

Would it work to say that vendors can ship to anyone on the list?  In
theory that could work, but in practice I think most distros would
rather just release once and be done with it, rather than dealing with
a 2-stage process.

 -George

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 04.07.12 at 14:56, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> On Wed, Jul 4, 2012 at 1:52 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> Being on the list doesn't make you non-susceptible. Such an
>> approach, imo, would need to imply permission to anyone on
>> the list to deploy a fix as soon as it is available. But since
>> distros can't ship binaries without also making sources available,
>> that's a contradiction by itself.
> 
> Yes, preventing vendors from shipping until the public disclosure date
> would discriminates against "vendor-supplied" users in favor of
> "self-supplied" users (i.e., those who download and build their own
> directly from xen.org).
> 
> Would it work to say that vendors can ship to anyone on the list?  In
> theory that could work, but in practice I think most distros would
> rather just release once and be done with it, rather than dealing with
> a 2-stage process.

So would I think.

Jan

Re: Security vulnerability process, and CVE-2012-0217

[Stefano Stabellini]

On Wed, 4 Jul 2012, George Dunlap wrote:
> On Wed, Jul 4, 2012 at 1:52 PM, Jan Beulich <JBeulich@suse.com> wrote:
> > Being on the list doesn't make you non-susceptible. Such an
> > approach, imo, would need to imply permission to anyone on
> > the list to deploy a fix as soon as it is available. But since
> > distros can't ship binaries without also making sources available,
> > that's a contradiction by itself.
> 
> Yes, preventing vendors from shipping until the public disclosure date
> would discriminates against "vendor-supplied" users in favor of
> "self-supplied" users (i.e., those who download and build their own
> directly from xen.org).
> 
> Would it work to say that vendors can ship to anyone on the list?  In
> theory that could work, but in practice I think most distros would
> rather just release once and be done with it, rather than dealing with
> a 2-stage process.

I don't think that software vendors will be very happy to release
twice. Also how would Debian make an update available only to the list
members? It would need to setup a strange apt-get repository system that
ships different packages (and sources) depending on who's asking?

Can we just avoid all this and use the security list to communicate that
a fix is going to be available on a particular hour of a particular day?
This way all the software vendors and service providers can ready
themselves to deploy it as soon as they can.
The fix would be released to the security list and xen-devel at the same
time.

In practice, given the terms of the GPL, we cannot restrict anybody on
the list from releasing the source of the fix before the embargo ends.

Re: Security vulnerability process, and CVE-2012-0217

[Jan Beulich]

>>> On 04.07.12 at 15:30, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> Can we just avoid all this and use the security list to communicate that
> a fix is going to be available on a particular hour of a particular day?
> This way all the software vendors and service providers can ready
> themselves to deploy it as soon as they can.
> The fix would be released to the security list and xen-devel at the same
> time.

That would only call for each party trying to create and deliver
their fix themselves and up front. You'd then also have to hide
the issue description. Which would render the security list
redundant.

> In practice, given the terms of the GPL, we cannot restrict anybody on
> the list from releasing the source of the fix before the embargo ends.

Of course. It's an agreement between the list members to not
disclose anything.

Jan

Re: Security vulnerability process, and CVE-2012-0217

[Stefano Stabellini]

On Wed, 4 Jul 2012, Jan Beulich wrote:
> >>> On 04.07.12 at 15:30, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> > Can we just avoid all this and use the security list to communicate that
> > a fix is going to be available on a particular hour of a particular day?
> > This way all the software vendors and service providers can ready
> > themselves to deploy it as soon as they can.
> > The fix would be released to the security list and xen-devel at the same
> > time.
> 
> That would only call for each party trying to create and deliver
> their fix themselves and up front. You'd then also have to hide
> the issue description.

Yes, we would have to hide the issue description.


> Which would render the security list redundant.

It would be a very different kind of security list.


> > In practice, given the terms of the GPL, we cannot restrict anybody on
> > the list from releasing the source of the fix before the embargo ends.
> 
> Of course. It's an agreement between the list members to not
> disclose anything.

Yes, but an agreement that cannot be legally enforced.

Re: Security vulnerability process, and CVE-2012-0217

[John Haxby]

On 04/07/12 16:09, Stefano Stabellini wrote:
>>> > > In practice, given the terms of the GPL, we cannot restrict anybody on
>>> > > the list from releasing the source of the fix before the embargo ends.
>> > 
>> > Of course. It's an agreement between the list members to not
>> > disclose anything.
> Yes, but an agreement that cannot be legally enforced.

I don't see that that is an issue.

Taking linux-distros as an example, an embargo date cannot be enforced
as there is no legal framework in which to enforce it.   Everyone
involved agrees to respect the embargo dates.   If an individual or
organisation repeatedly flaunted the embargo dates they would likely
find themselves removed from the list although, to my knowledge, this
has not happened.

For the list to work, the members need to cooperate: it is in their own
interest to cooperate, legal frameworks aren't required.

jch

Re: Security vulnerability process, and CVE-2012-0217

[Matthew Allen]

(Full disclosure: I am part of the Citrix security team on the present pre-disclosure list)

On 06/20/2012 02:16 AM, Ian Jackson wrote:
> The first point is that we think the security vulnerability process 
> document should have an explanation of what its goals are.
Then we need consensus on the goals before we get too far into the details of the process.
Otherwise we don't have anything against which to measure the options.

and on 06/28/2012 10:28 AM, Lars Kurth wrote:
> Of course the ultimate goal of the process should be to minimize 
> days-of-risk for everyone. To do this, the process has to balance the 
> following factors to reduce days-of-risk [...]

This also seems an obvious goal to me; is there consensus that this _is_ a goal, and are
there suggestions for other goals?

One question I've got is how we measure days-of-risk for everyone in order to minimise
them?  From a Citrix point of view we'd see that as (time between it being probable that
an exploit is used and a fix being deployed) x (number of affected individuals).  We'd
then expect that a pre-disclosure list would reduce that number.

I'd also like to add to Lars' list of factors (incentive to disclose, times to fix/apply/deploy,
reducing leaks and being fair) with another issue: the quality of the fix.  ISTM that the
Xen.org security team did a superb job in getting out an initial fix for a complex set of
problems within the one week target of the present policy, but (again from a Citrix pov)
I'd rather have more time spent to get a greater assurance of a complete fix.  That means
that not only has the identified issue been comprehensively fixed but similar issues in the
code have been looked for (fixing a single unchecked memcpy and publicising it probably
_increases_ actual risk if there are more in the code).
Having a good quality fix can be done by allowing more time or (as has been suggested)
having more people involved; I don't think we'd have a strong view on which route was
chosen but if it's the latter, Citrix would be willing to help where requested and appropriate.

My other concern is the time available to apply the fix once available.  It really does take
time to test a fix to a level that companies will trust their business, particularly if you're
supporting multiple versions.  You can disregard this as "not an open-source problem",
but I would claim that a viable commercial use of Xen also benefits the open-source
community.  I'd therefore particularly endorse the idea that, rather than having a single
fixed timescale, the timing should depend on the severity particular issue(s).    We also
need to be clear where backporting happens in the timescales; I suspect that a significant
proportion of Xen users aren't on xen-unstable and rather than that being a bad thing I'd
claim that it's evidence of Xen's widespread adoption and success!

Matthew

Re: Security vulnerability process, and CVE-2012-0217

[George Dunlap]

On 06/07/12 17:39, Matthew Allen wrote:
> One question I've got is how we measure days-of-risk for everyone in order to minimise
> them?  From a Citrix point of view we'd see that as (time between it being probable that
> an exploit is used and a fix being deployed) x (number of affected individuals).  We'd
> then expect that a pre-disclosure list would reduce that number.

I've just sent an e-mail with a discussion of risk (among other 
factors).  I think a more accurate formula would be SUM[over 
days](probability of exploit * number of affected individuals); or, to 
simplify things, the total risk is the sum of "private vulnerability" 
(time before the public disclosure) and "public vulnerability" (after 
the disclosure but before users can roll out fixes).

As I say in that e-mail, one might expect that pre-disclosure would 
reduce that number.  But I'm of the opinion that in fact it will not.  
Most of the people advocating pre-disclosure seem to think that 
"probability of exploit" is very low before public disclosure, and very 
high afterwards.  I don't think that's true; at very least, the 
"probability of exploit" during the "private vulnerability" period is a 
serious risk.  Any pre-disclosure period will extend that time, thus 
extending the risk.

> I'd also like to add to Lars' list of factors (incentive to disclose, times to fix/apply/deploy,
> reducing leaks and being fair) with another issue: the quality of the fix.  ISTM that the
> Xen.org security team did a superb job in getting out an initial fix for a complex set of
> problems within the one week target of the present policy, but (again from a Citrix pov)
> I'd rather have more time spent to get a greater assurance of a complete fix.  That means
> that not only has the identified issue been comprehensively fixed but similar issues in the
> code have been looked for (fixing a single unchecked memcpy and publicising it probably
> _increases_ actual risk if there are more in the code).
> Having a good quality fix can be done by allowing more time or (as has been suggested)
> having more people involved; I don't think we'd have a strong view on which route was
> chosen but if it's the latter, Citrix would be willing to help where requested and appropriate.
>
> My other concern is the time available to apply the fix once available.  It really does take
> time to test a fix to a level that companies will trust their business, particularly if you're
> supporting multiple versions.
This is certainly true.  Not having a predisclosure list does mean that 
customers of "high value" software vendors, such as SUSE, Oracle, and 
Citrix, may spend an amount of time "publicly vulnerable" while those 
vendors prepare fixes (but don't forget, with pre-disclosure, they will 
have at least that much time "privately vulnerable").  Since these "high 
value" software vendors contribute nearly all the resources for xen.org, 
I think it's fair to take their needs into consideration.

It may be true, as you say, that because (say) Oracle will do more 
testing than (say) Debian, that Oracle's users will be publicly 
vulnerable for longer than Debian users, and both may be vulnerable 
longer than users who download directly from xen.org.  However, to a 
certain degree that's probably fair -- the cost of being publicly 
vulnerable for longer is paid for by the confidence that the update will 
not cause problems when they roll it out.  Having a pre-disclosure 
period means that users who download directly from xen.org have 
significantly longer "private vulnerability", but without getting any 
advantages in terms of reliability of the patch.
> I'd therefore particularly endorse the idea that, rather than having a single
> fixed timescale, the timing should depend on the severity particular issue(s).    We also
> need to be clear where backporting happens in the timescales; I suspect that a significant
> proportion of Xen users aren't on xen-unstable and rather than that being a bad thing I'd
> claim that it's evidence of Xen's widespread adoption and success!
I would expect that if we chose to do away with the pre-disclosure 
period, the xen.org security team would also release backports to 
still-maintained releases; in this case, at least 4.1, 4.0, and 3.4.  
This will hopefully reduce most vendors' time creating a fix to testing.

  -George

Re: Security vulnerability process, and CVE-2012-0217

[Ian Campbell]

On Tue, 2012-06-19 at 19:16 +0100, Ian Jackson wrote:
[...]
> More minor policy questions
> ---------------------------
[...]
> Lacunae in the process
> ----------------------

I've prepared a series of patches to the policy[0] which implement most
(but not all) of what was suggested in these two sections. I think they
are largely uncontroversial, but please do holler if you disagree or
want to suggest further improvements..

I'm not at all sure that patches are the best way to present this but there
we are.

Ian.

[0] http://www.xen.org/projects/security_vulnerability_process.html

[PATCH 1/6] Clarify what info predisclosure list members may share during an embargo

[Ian Campbell]

See <20448.49637.38489.246434@mariner.uk.xensource.com>, section
  "7. Public communications during the embargo period"
---
 security_vulnerability_process.html |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/security_vulnerability_process.html b/security_vulnerability_process.html
index d1a6629..eff108a 100644
--- a/security_vulnerability_process.html
+++ b/security_vulnerability_process.html
@@ -195,9 +195,17 @@ if(ns4)_d.write("<scr"+"ipt type=text/javascript src=/globals/mmenuns4.js><\/scr
     should not make available, even to their own customers and partners:<ul>
        <li>the Xen.org advisory</li>
        <li>their own advisory</li>
+       <li>the impact, scope, set of vulnerable systems or the nature
+       of the vulnerability</li>
        <li>revision control commits which are a fix for the problem</li>
        <li>patched software (even in binary form) without prior consultation with security@xen and/or the discoverer.</li>
     </ul></p>    
+    <p>List members are allowed to make available to their users only the following:<ul>
+       <li>The existance of an issue</li>
+       <li>The assigned XSA and CVE numbers</li>
+       <li>The planned disclosure date</li>
+    </ul></p>
+
     <p>Organisations who meet the criteria should contact security@xen if they wish to receive pre-disclosure of advisories.</p>    
     <p>The pre-disclosure list will also receive copies of public advisories when they are first issued or updated.</p>
     
-- 
1.7.10.4

[PATCH 2/6] Clarifications to predisclosure list subscription instructions

[Ian Campbell]

Specially:
  * Mention that subscriptions via the webterface do not work / are
    not honoured.
  * Mention the preference for role addresses only.

See <20448.49637.38489.246434@mariner.uk.xensource.com>, section
    "8. Predisclosure subscription process, and email address
        criteria"
---
 security_vulnerability_process.html |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/security_vulnerability_process.html b/security_vulnerability_process.html
index eff108a..ee42402 100644
--- a/security_vulnerability_process.html
+++ b/security_vulnerability_process.html
@@ -206,7 +206,9 @@ if(ns4)_d.write("<scr"+"ipt type=text/javascript src=/globals/mmenuns4.js><\/scr
        <li>The planned disclosure date</li>
     </ul></p>
 
-    <p>Organisations who meet the criteria should contact security@xen if they wish to receive pre-disclosure of advisories.</p>    
+    <p>Organisations who meet the criteria should contact security@xen if they wish to receive pre-disclosure of advisories. Organisations should not request subscription via the mailing list web interface, any such subscription requests will be rejected and ignored.</p>
+    <p>Normally we would prefer that a role address be used for each organisation, rather than one or more individual's direct email address. This helps to ensure that changes of personel do not end up effectively dropping an organisation from the list</p>
+
     <p>The pre-disclosure list will also receive copies of public advisories when they are first issued or updated.</p>
     
     <h3>Organizations on the pre-disclosure list:</h3>
-- 
1.7.10.4

[PATCH 3/6] Clarify the scope of the process to just the hypervisor project

[Ian Campbell]

Other projects are handled on a best effort basis by the project lead
with the assistance of the security team.

See <20448.49637.38489.246434@mariner.uk.xensource.com>, section
    "9. Vulnerability process scope"
---
 security_vulnerability_process.html |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/security_vulnerability_process.html b/security_vulnerability_process.html
index ee42402..ddd88a1 100644
--- a/security_vulnerability_process.html
+++ b/security_vulnerability_process.html
@@ -77,6 +77,9 @@ if(ns4)_d.write("<scr"+"ipt type=text/javascript src=/globals/mmenuns4.js><\/scr
     will treat with respect the requests of discoverers, or other vendors, who
     report problems to us.</p>
 
+    <h2>Scope of this process</h2>
+    <p>This process primarily covers the <a href="http://www.xen.org/products/xenhyp.html">Xen Hypervisor Project</a>. Vulnerabilties reported against other Xen.org projects will be handled on a best effort basis by the relevant Project Lead together with the security team.</p>
+
     <h2>Specific process</h2>
     <ol type="1">
     <li><p>We request that anyone who discovers a vulnerability in xen.org
-- 
1.7.10.4

[PATCH 4/6] Discuss post-embargo disclosure of potentially controversial private decisions

[Ian Campbell]

See <20448.49637.38489.246434@mariner.uk.xensource.com>, section
    "11. Transparency"
---
 security_vulnerability_process.html |   12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/security_vulnerability_process.html b/security_vulnerability_process.html
index ddd88a1..687e452 100644
--- a/security_vulnerability_process.html
+++ b/security_vulnerability_process.html
@@ -147,6 +147,18 @@ if(ns4)_d.write("<scr"+"ipt type=text/javascript src=/globals/mmenuns4.js><\/scr
        public advisory.  This will also be sent to the pre-disclosure
        list.</p>
     </li>
+
+    <li><p><b>Post embargo transparency:</b></p>
+    <p>During an embargo period the Xen.org security response team may
+    be required to make potentially controverial decisions in private,
+    since they cannot confer with the community without breaking the
+    embargo. The security team will attempt to make such decisions
+    following the guidance of this document and where necessary their
+    own best judgement. Following the embargo period any such
+    decisions will be disclosed to the community in the interests of
+    transperency and to help provide guidance should a similar
+    decision be required in the future.</p>
+    </li>
     </ol>
     
     <h2>Embargo and disclosure schedule</h2>    
-- 
1.7.10.4

[PATCH 5/6] Patch review, expert advice and targetted fixes

[Ian Campbell]

See <20448.49637.38489.246434@mariner.uk.xensource.com>, section
    "Patch development and review"
---
 security_vulnerability_process.html |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/security_vulnerability_process.html b/security_vulnerability_process.html
index 687e452..c830a04 100644
--- a/security_vulnerability_process.html
+++ b/security_vulnerability_process.html
@@ -109,8 +109,13 @@ if(ns4)_d.write("<scr"+"ipt type=text/javascript src=/globals/mmenuns4.js><\/scr
        process.</p></li>
        <p>(This may rely on the other project(s) having
        documented and responsive security contact points)</p>
-    <li><p>We will prepare or check patch(es) which fix the vulnerability.
-       This would ideally include all relevant backports.</p></li>
+    <li><p>We will prepare or check patch(es) which fix the
+       vulnerability.  This would ideally include all relevant
+       backports.  Patches will be tightly targeted on fixing the
+       specific security vulnerability in the smallest, simplest and
+       most reliable way.  Where necessary domain specific experts
+       within the community will be brought in to help with patch
+       preparation.</p></li>
     <li><p>We will determine which systems/configurations/versions are
        vulnerable, and what the impact of the vulnerability is.
        Depending on the nature of the vulnerability this may involve
-- 
1.7.10.4

[PATCH 6/6] Declare version 1.3

[Ian Campbell]

---
 security_vulnerability_process.html |    1 +
 1 file changed, 1 insertion(+)

diff --git a/security_vulnerability_process.html b/security_vulnerability_process.html
index c830a04..c6c0d1d 100644
--- a/security_vulnerability_process.html
+++ b/security_vulnerability_process.html
@@ -251,6 +251,7 @@ if(ns4)_d.write("<scr"+"ipt type=text/javascript src=/globals/mmenuns4.js><\/scr
     
     <h2>Change History</h2>
     <ul>
+      <li><b>v1.3 Aug 2012:</b> Various minor updates</li>
       <li><b>v1.2 Apr 2012:</b> Added pre-disclosure list</li>
       <li><b>v1.1 Feb 2012:</b> Added link to Security Announcements wiki page</li>
       <li><b>v1.0 Dec 2011:</b> Intial document published after review</li>
-- 
1.7.10.4

Re: Security vulnerability process, and CVE-2012-0217 [vote?]

[Lars Kurth]

Hi,

Does anybody have any objections to Ian's changes in this patch series? 
None of these appear significant changes. In line with the decision 
making process, I would be happy to apply. But, I am also happy to run 
this through a vote.

The patch series was
[PATCH 1/6] Clarify what info predisclosure list members may share 
during an embargo
[PATCH 2/6] Clarifications to predisclosure list subscription instructions
[PATCH 3/6] Clarify the scope of the process to just the hypervisor project
[PATCH 4/6] Discuss post-embargo disclosure of potentially controversial 
private decisions
[PATCH 5/6] Patch review, expert advice and targetted fixes
[PATCH 6/6] Declare version 1.3 ... that would have changed to 1.4 as we 
added CentOS to the pre-disclosure list in 1.3

Best Regards
Lars

On 23/08/2012 11:37, Ian Campbell wrote:
> On Tue, 2012-06-19 at 19:16 +0100, Ian Jackson wrote:
> [...]
>> More minor policy questions
>> ---------------------------
> [...]
>> Lacunae in the process
>> ----------------------
> I've prepared a series of patches to the policy[0] which implement most
> (but not all) of what was suggested in these two sections. I think they
> are largely uncontroversial, but please do holler if you disagree or
> want to suggest further improvements..
>
> I'm not at all sure that patches are the best way to present this but there
> we are.
>
> Ian.
>
> [0] http://www.xen.org/projects/security_vulnerability_process.html
>
>
>
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Security vulnerability process, and CVE-2012-0217 [vote?]

[Ian Jackson]

Lars Kurth writes ("Re: [Xen-devel] Security vulnerability process, and CVE-2012-0217 [vote?]"):
> Does anybody have any objections to Ian's changes in this patch series? 
> None of these appear significant changes. In line with the decision 
> making process, I would be happy to apply. But, I am also happy to run 
> this through a vote.

They all seem uncontroversial to me and no-one has objected.  You
should go ahead and make the changes, I think.

Ian.

Re: Security vulnerability process, and CVE-2012-0217 [vote?]

[Lars Kurth]

Agreed. I will apply and ACK when done
Lars

On 01/10/2012 17:38, Ian Jackson wrote:
> Lars Kurth writes ("Re: [Xen-devel] Security vulnerability process, and CVE-2012-0217 [vote?]"):
>> Does anybody have any objections to Ian's changes in this patch series?
>> None of these appear significant changes. In line with the decision
>> making process, I would be happy to apply. But, I am also happy to run
>> this through a vote.
> They all seem uncontroversial to me and no-one has objected.  You
> should go ahead and make the changes, I think.
>
> Ian.

Re: Security vulnerability process, and CVE-2012-0217 [vote?]

[Lars Kurth]

On 01/10/2012 17:38, Ian Jackson wrote:
> They all seem uncontroversial to me and no-one has objected. You 
> should go ahead and make the changes, I think. Ian. 
All published at 
http://www.xen.org/projects/security_vulnerability_process.html
Lars

Re: Security vulnerability process, and CVE-2012-0217

[John Creol]

Just a quick perspective. I'm part of a smaller downstream "Service provider" -- albeit with "Many thousands" of Xen instances running vs. "Hundreds of thousands."

About a month before disclosure we were aware of patching/rebooting/notices that Linode was performing, based on word our own customers and our ear to the ground. From what we could understand we were concerned. We asked around. Unfortunately, as a true consumer of Xen we had no real "in" to confirm the scope of the issue. 

So we did what we could to prepare - we ensured we were up to date, that our rapid patch deployment process worked, and ramped up internal processes to look for problems. We've always asked ourselves, what would happen if a hypervisor was compromised - a doomsday scenario - and prepared as best we could. We'd heard enough to suspect this was what was in the wild and spent quite a few sleepless nights thinking about it.

Once the embargo was lifted we had the unique perspective of watching as the bug was updated, corrected, and the various CVE's and details rolled out. We were on top of it that quickly - we had to be. It was the worst case scenario for us - we're intel based and allow extremely custom levels of access to Xen so customers can control their environment. 

Not knowing if a vulnerability was in the wild, we took a few precautionary steps - for example, disabling 64bit images from new deployments, and pushing existing ones to HVM if a customer initiated a reboot. Luckily these are switches we can flip in our platform.

On the hypervisor, since we build our own packages we were able to quickly build, test, and deploy to our hosts. This was accomplished in most cases without customer interruption due to live migration or save/restores, but it was still a tense 24-72 hours with hundreds of physical servers to patch in multiple worldwide locations.

As we were working through the process, we saw Linode's self congratulatory blog post disclosing the vulnerability, and the fact that their customers didn't have to worry - they'd been on it for weeks. Wow, we sure wish we could be on the pre-disclosure list. We could market that. Unfortunately when it comes to the Xen.org disclosure process, size matters.

Do we blame Linode or the Xen.org team? No. Do we think the process was fair? No. Would we do anything differently if we were in your position? Maybe not. That said, it's clear that personal, business, and other decisions played far too great a part in the conversation here. When you have CEO's calling in, and jobs on the line, for what amounts to a community project (and a vulnerability disclosed by a community member) with hundreds of thousands of downstream users that's a real shame -- and a very real advantage to the /service/ providers on the list.

The net for us is that we were already doing the right things - our processes and Infrastructure are set up in such a way that /when/ the next critical Xen vulnerability is released - maybe directly into the wild as an easily runable exploit for everyone to get their hands on - we'll be able to respond quickly and on multiple fronts.

We can appreciate the process aspects of the discussion, and the scope and scale of such large Xen based providers. That said, I would also hope there is some renewed thought going into hot patching of the dom0 / hypervisor / etc. 

As someone else mentioned, the actual disclosure could be far better off being served by an outside organization like CERT that has established processes in place and can move forward without undue influence from a small group of players.

Thanks
John