From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ian Jackson Subject: Security vulnerability process, and CVE-2012-0217 Date: Tue, 19 Jun 2012 19:16:05 +0100 Message-ID: <20448.49637.38489.246434@mariner.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org Introduction ------------ The Xen.org security response team is charged with implementing the Xen security response process, the current version of which can be found here: http://www.xen.org/projects/security_vulnerability_process.html Over the past two months we on that team have been involved with XSA-7 / CVE-2012-0217 and its various fallout. During this exercise we have encountered some problems with the process. Also, we have had to make some difficult decisions. We feel it is essential for keeping us honest that we explain to the community what we did, and when. There's a detailed timeline at the bottom of this mail, listing the important events; reading it will give you a much clearer picture of the problems we encountered and why we are asking for various issues to be clarified. In this message I'll go through some of the problems we encountered. Where there are uncontroversial improvements for the process we propose them here. For the more controversial issues, in this message we will raise the questions and discuss the issues in a neutral way, and make our individual responses separately. The outcome of this discussion will be a set of changes to be agreed on and/or voted on using the existing Xen.org governance processes: http://www.xen.org/projects/governance.html This discussion will take place on the xen-devel mailing list. We expect it to take some weeks. Summary of the most important difficulties we encountered --------------------------------------------------------- One member of the predisclosure list, only a few days before the embargo date, mounted a sustained and eventually successful lobbying campaign to get the embargo extened. There were leaks during the embargo period, after it had been extended, which we experienced as enquiries from community members. We were forced to made several ad-hoc decisions with insufficient guidance from the process document. We discovered too late in the process that the set of vulnerable operating systems was substantially wider than we thought. The big issues -------------- 1. Purpose of the process The first point is that we think the security vulnerability process document should have an explanation of what its goals are. This would have greatly assisted us when making some of the more difficult decisions. In particular, if the process explained its purpose, we would be able to refer to the spirit of the document when making interpretation decisions or dealing with issues not clearly resolved by the document. In this context we need to decide whether: (a) the predisclosure arrangements are just there so that organisations can backport patches, develop and test updated versions, and so forth; (b) the arrangements are also intended to allow operators to deploy fixed versions. Of course this needs to deal clearly with the common stituation of an organisation running Xen which is a direct consumer of Xen.org source code. 2. Extension of embargo dates The most controversial decision was whether the embargo date might be extended after it had originally been set. We resisted these suggestions, referring to the process document, which does not contemplate extending the disclosure period. However, when a predisclosure list member pressured the discoverer into requesting an extension, we felt the best interpretation of the process document required us to acquiesce. Specifically, of course, we as a team would like clearer guidance from the process about whether, and if so under what circumstances, a predisclosure period should be extended. 3. Decisionmaking It was suggested to us several times, and indeed seemed to be regarded by some as an underlying principle, that the predisclosure list members should be making the decisions about disclosure schedule etc. The question of who is to make the decisions, and on what basis, needs to be addressed. 4. Length of (default) predisclosure period Several members of the predisclosure list suggested that the predisclosure period was too short. Others were ready within the predisclosure period's timeframe, and were disappointed to see it extended and to have to delay their fixes. 5. Criteria for predisclosure list membership We had one request from a public Xen cloud provider to be provided with predisclosure information. However it appeared to us that they didn't meet the size threshold in the process document. The size threshold is of course open to discussion. We need to clarify whether upstreams and hardware vendors should be on the predisclosure list. Currently we have one notable upstream vendor on that list. Our preliminary view is that the predisclosure list should be used for downstream notifications. Upstreams, including hardware manufacturers, should be brought in to the disclosure process as needed. And as noted, our process for making sure we do that properly needs to be formalised. It is probably time for a review of the membership of the predisclosure list, in any case, after we have incorporated any changes to the criteria which are agreed as a result of this discussion. 6. Sharing of information and code between predisclosure participants We need the process document to be clearer about what kinds of communications are contemplated _between_ members of the predisclosure list. One particular issue here which also relates to the predisclosure membership criteria, is whether large indirect consumers of Xen should be on the predisclosure list in their own right. That would allow them to deploy the fix before the embargo date. It would also allow them to prepare for testing and deployment, before the fix is available from their vendor (who would in this scenario also be entitled to be a predisclosure list member). In this context, the process needs to clarify whether vendors may release fixed binaries during the predisclosure period. Certainly these should not be released other than to predisclosure list members, since the nature of a bug can often by discovered by reverse engineering. But can they be released to predisclousre list members ? More minor policy questions --------------------------- 7. Public communications during the embargo period We need to clarify what individual vendors are allowed to say during the embargo period. It was brought to our attention that one public cloud provider told its users that their VMs would be migrated due to "a vendor issue" which would be revealed in "a few weeks". Our view was that that was fine. As a starting point, it seems to us that it is OK to disclose that there is an issue, including its XSA and CVE numbers and its planned disclosure date; but that it is not OK to disclose the impact, scope, set of vulnerable systems, and certainly not the nature of the vulnerability. And we need to clarify what information the security team should give out when we get an enquiry from a community member about an embargoed problem. 8. Predisclosure subscription process, and email address criteria We certainly need to clarify the subscription process to the list, to make it clear that the organisation's security team should request all subscriptions and that self-subscriptions via the mailing list interface will be rejected out of hand. The current arrangements have caused quite a lot of admin work, to confirm various change requests. The predisclosure list contains a number of individuals at various organisations. One vendor seemed to get into difficulty because it had only one individual on the predisclosure list, which seems to have unfortunately kept their established team in the dark during the early stages of the process. The process document requires that downstreams on the list have established security teams. Should we require that such response teams' contact details be publicly advertised ? Should we insist that only response teams' role addresses may be included on the list ? 9. Vulnerability process scope We need to clarify the scope of the process document. Currently it looks like it covers every Xen.org project. While it would be a nice ideal to support every Xen.org project this way, in practice the team behind security@xen do not have the expertise or resources to fix problems in (say) XCP, or the ARM PV port. Our capability is concentrated on the "Upstream Xen" codebase (xen-*.hg and its sub-repositories) and the Xen support code in Linux. Perhaps this could be dealt with by making it clear that problems are handled on a best effort basis. 10. Exploits We need a clear policy about releasing proof of concept exploits - whether, when and who to. 11. Transparency We think the process document should explicitly state that any potentially controversial private decisions made by the Xen.org security response team should be disclosed after the vulnerability itself is disclosed. Lacunae in the process ---------------------- 12. Holidays The original planned disclosure date, 2012-05-01, was a public holiday in many places. We should try to avoid holidays, and Fridays. We need to discuss which territories' holidays should be checked and make a list for inclusion in the process. 13. Patch development and review The patch development process is too closed and not robust enough. We need to be much clearer both about the need for security patches to fix things in the smallest, simplest and most reliable way, and not fix or "improve" matters in unrelated ways. Our internal code review needs to be much better. We need to clarify that other issues discovered during the course of investigating a security disclosure will not be publicly discussed without first consulting the Xen.org security team, regardless of their actual relationship to the vulnerability at hand or expected security impact. Otherwise there is a substantial risk that such changes will draw attention to embargoed problems. It may be that it would be a better idea to send out earlier advisories without patches to the predisclosure list, to enable those predisclosure list members who would like to do so to help with development and testing of the patches. If so this needs to be catered for. 14. Early consideration of which other organisations to bring in Our process needs to have, very early on, an explicit step to of deciding which other projects/organisations may also be vulnerable and may therefore need to be part of the same disclosure process. It also needs to make sure that we ask for any help (for example from upstreams or hardware vendors) as soon as possible. We also need to explicitly determine the availability of various key players over the disclosure timeline as an early step. One of the difficulties we encountered was that one of the more important team members was unexpectedly out of their office at a critical point. 15. Process automation Many of the advisory versions sent to the predisclosure list contained clerical errors. Our processes for constructing these need to be made more automatic. Thanks for your attention, Ian. on behalf of the Xen.org security response team Timeline (here "us" is the Xen.org security team) ------------------------------------------------- 2012-04-09 Xen.org and FreeBSD notified of the problem (which will become XSA-7 / CVE-2012-0217) by the discoverer. 2012-04-11 Discoverer tells us that they are happy with following the Xen.org process for this vulnerability, and asks for objections from FreeBSD. 2012-04-11 We ask the Debian security team to allocate us a CVE candidate number. (We have had difficulty with getting these from Mitre.) 2012-04-12 We confirm in response to a question from Debian that we think Debian is vulnerable, giving some more information but not enough to find the problem. 2012-04-12 We contact AMD to ask them to confirm whether any AMD processors are vulnerable. 2012-04-12 Debian allocates us CVE-2012-0217. We make sure that everyone currently aware of the issue is told. 2012-04-13 Prompted by the discoverer, we contacts NetBSD so that they can check if they also are vulnerable, given that we believe that NetBSD may share the relevant code with FreeBSD. 2012-04-16 Near-final draft of the predisclosure advisory is ready, including initial versions of patches for xen-unstable, Xen 4.1 and Xen 4.0. 2012-04-17 The problem which will later become XSA-8 / CVE-2012-0218 is spotted and a fix committed to xen-unstable.hg, without mentioning that this bug is a security vulnerability. In our view this was a mistake. 2012-04-17 XSA-7 v1 is sent out, with an embargo end date of 2012-05-01. 2012-04-17 We become aware of the earlier commit and assign it XSA-8. XSA-7 v2 is sent out, with a slightly improved patch and mentioning that XSA-8 will follow. 2012-04-18 We ask Mitre and Debian for a CVE for XSA-8. We send XSA-7 v3, and XSA-8 to the predisclosure list. 2012-04-20 We send XSA-8 v2, which contains backported patches and corrections to the advisory text widening the set of possibly affected systems. 2012-04-20 We send XSA-7 v4, which contains backported patches and the information that AMD CPUs are not vulnerable (following confirmation from AMD). 2012-04-20 A member of the predisclosure list complains to us that the disclosure schedule is too short, and making various other points about the process. We reply with a reference to the process document; we suggest that if the organisation feels that the process needs improving this should be discussed in public. 2012-04-20 We are asked by a member of the predisclosure list whether it would be OK to release fixed binaries to a customer of theirs who is also a member of the predisclosure list. On the 2012-04-23, after internal discussions and consulting with the discoverer, we reply to say that we think that would be OK. 2012-04-20 We are asked by a member of the predisclousre list whether there is any way they can verify whether their systems are vulnerable, or perhaps whether we can share any proof of concept exploit code. Again, we consulted with the discoverer. 2012-04-21 The discoverer reproduces the problem on a widely-deployed proprietary operating system that runs on x86, and suggests to us that the embargo period should therefore be extended. We reply: Our process doesn't contemplate the extension of the embargo after the release of information to our predisclosure list. It would in theory be possible for us to email the predisclosure list ask them to extend the embargo date. However, from a practical point of view, if anyone didn't get the email they might go ahead and disclose on the original date, leaving us all scrambling madly with an unexpected disclosure. And from a process point of view, that would be departing substantially from our published process, so I'm very reluctant. Certainly you are entitled to notify [proprietary OS vendor] right away and it would be sensible to do so. With hindsight it was an error not to consider, as part of the formal process, which other operating systems might be vulnerable. 2012-04-26 We obtain the number CVE-2012-0218 from Debian for XSA-8. 2012-04-26 We send XSA-7 v5 with improved wording. We send XSA-8 v3, with the CVE number. 2012-04-28 A member of the predisclosure list requests an extension of the disclosure timeframe, and in the same message requests a change to the email address subscribed to the predisclosure list. We reply declining, referring to our process document, and once again inviting public discussion of the process itself. 2012-04-28 The member of the predisclosure list asking for an extension asks to be put in touch with all the other members of the predisclosure list and proposes a conference call. 2012-04-30 We decline to give further details of the predisclosure list members, other than the organisational identities listed. We again decline to extend the disclosure date. 2012-04-30 The member asking for an extension reiterates their request, citing support from various other named predisclosure list members - in at least some cases, exaggerating the level of such support. 2012-04-30 Various other members of the predisclosure list report to us having been asked to support the member asking for a delay. CEOs of several companies were involved. In some cases the other members support the request. We again decline. 2012-04-30 The member requesting an embargo suggests holding a straw poll of predisclosure list members on the question of a delay. Again, we decline. 2012-04-30 The member requesting an embargo places intense pressure on the employers of some members of the Xen.org security team to try to get the decision changed, without success. 2012-04-30 23:06 Z (13h before planned disclosure) The member requesting an embargo contacts the employer of the discoverer to apply pressure. The discoverer notifies Xen.org that "on behalf of the organization who discovered this issue" they request a delay, giving a putative but unconfirmed new disclosure date of the 2012-05-31. 2012-05-01 08:08 Z (3h52 before planned disclosure) Xen.org team decides, after internal discussions, that as the policy gives the discoverer control of the disclosure date, and the question is not otherwise addressed, we must agree to this request. 2012-05-10 08:48 Z (3h12 before planned disclosure) We send an ad-hoc message to the predisclosure list requesting a disclosure delay for XSA-7 / XSA-8. We do not include a revised disclosure date as we don't have one confirmed. 2012-05-01 Backport of XSA-7/XSA-8 patches to 3.4 are prepared. 2012-05-02 We become aware, indirectly, that US-CERT have become involved. 2012-05-03 US-CERT suggest to us a disclosure date of 2012-06-12. We make a formal response, saying that we think this is far too late. 2012-05-03 We receive the first enquiry from a Xen community member not on the predisclosure list, saying they had heard somerthing was wrong, and asking about it. This is the first of several such enquiries. We quote the number CVE-2012-0217 and say that we expect it to be disclosed on the 12th of June, but that we aren't allowed to say more. 2012-05-08 We press the discoverers' employer for a confirmed disclosure date. 2012-05-08 Given that no disclosure date is forthcoming, we send XSA-7 v6 and XSA-8 v4, with an indefinite embargo and the backport patch for Xen 3.4. 2012-05-14 Given that still no firm date is forthcoming, we (the Xen.org team) point out to the discoverers and to CERT that our policy requires us to honour reasonable requests, but that the continued lack of any date is unreasonable, and demanding that a date be set by the 2012-05-18. 2012-05-14 The discover confirms 2012-06-12 as the disclosure date. 2012-05-15 We send XSA-7 v7 and XSA-8 v5 with the disclosure date of 2012-06-12. 2012-05-15 We receive a complaint about the delay from a member of the predisclosure list. 2012-05-15 A member of the predisclosure list discovers what will become XSA-9. The context was testing the fix for XSA-7. 2012-05-16 We conclude that this is due to AMD erratum #121, and involve AMD. 2012-05-16 After discussions we conclude that there is no reasonable software workaround for Xen and that instead Xen should refuse to boot on affected systems. 2012-05-18 We decide that releasing this as a separate advisory with the same embargo date will be less confusing than trying to fold it into XSA-7. 2012-05-18 While investigating this we come to the conclusion that our previous patch to fix XSA-7 was incomplete. 2012-05-21 First draft of XSA-9 is available. 2012-05-22 After intense internal discussions we decide that the original fix for XSA-7 is too fragile and conclude that it should be replaced by something more obvious. 2012-05-23 We request a CVE for XSA-9 from Mitre and Debian. 2012-05-24 We send XSA-7 v8, now with a cross-reference to XSA-9, and with updated simpler patch, and XSA-8 v6, and XSA-9 v1 with its own separate patch, to the predisclosure list. 2012-05-24 Mitre assign CVE-2012-2934 for XSA-9. 2012-06-11 We send XSA-9 v2, containing the CVE and a clear statement that no affected processors support SVM. (The delay to this message was a mistake by the Xen.org team.) 2012-06-12 Public disclosure.