From mboxrd@z Thu Jan 1 00:00:00 1970 From: John Creol Subject: Re: Security vulnerability process, and CVE-2012-0217 Date: Mon, 2 Jul 2012 08:24:43 -0700 (PDT) Message-ID: <1341242683.79217.YahooMailNeo@web39404.mail.mud.yahoo.com> Reply-To: John Creol Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org Just a quick perspective. I'm part of a smaller downstream "Service provide= r" -- albeit with "Many thousands" of Xen instances running vs. "Hundreds o= f thousands." About a month before disclosure we were aware of patching/rebooting/notices= that Linode was performing, based on word our own customers and our ear to= the ground. From what we could understand we were concerned. We asked arou= nd. Unfortunately, as a true consumer of Xen we had no real "in" to confirm= the scope of the issue.=A0 So we did what we could to prepare - we ensured we were up to date, that ou= r rapid patch deployment process worked, and ramped up internal processes t= o look for problems. We've always asked ourselves, what would happen if a h= ypervisor was compromised - a doomsday=A0scenario=A0- and prepared as best = we could. We'd heard enough to suspect this was what was in the wild and sp= ent quite a few sleepless nights thinking about it. Once the embargo was lifted we had the unique perspective of watching as th= e bug was updated, corrected, and the various CVE's and details rolled out.= We were on top of it that quickly - we had to be. It was the worst case=A0= scenario=A0for us - we're intel based and allow extremely custom levels of = access to Xen so customers can control their environment.=A0 Not knowing if a vulnerability was in the wild, we took a few=A0precautiona= ry=A0steps - for example, disabling 64bit images from new deployments, and = pushing existing ones to HVM if a customer initiated a reboot. Luckily thes= e are switches we can flip in our platform. On the hypervisor, since we build our own packages we were able to quickly = build, test, and deploy to our hosts. This was accomplished in most cases w= ithout customer interruption due to live migration or save/restores, but it= was still a tense 24-72 hours with hundreds of physical servers to patch i= n multiple worldwide locations. As we were working through the process, we saw Linode's self congratulatory= blog post disclosing the vulnerability, and the fact that their customers = didn't have to worry - they'd been on it for weeks. Wow, we sure wish we co= uld be on the pre-disclosure list. We could market that. Unfortunately when= it comes to the Xen.org disclosure process, size matters. Do we blame Linode or the Xen.org team? No. Do we think the process was fai= r? No. Would we do anything differently if we were in your position? Maybe = not. That said, it's clear that personal, business, and other decisions pla= yed far too great a part in the conversation here. When you have CEO's call= ing in, and jobs on the line, for what amounts to a community project (and = a=A0vulnerability=A0disclosed by a community member) with hundreds of thous= ands of downstream users that's a real shame -- and a very real advantage t= o the /service/ providers on the list. The net for us is that we were already doing the right things - our process= es and Infrastructure are set up in such a way that /when/ the next critica= l Xen vulnerability is released - maybe directly into the wild as an easily= runable exploit for everyone to get their hands on - we'll be able to resp= ond quickly and on multiple fronts. We can appreciate the process aspects of the discussion, and the scope and = scale of such large Xen based providers. That said, I would also hope there= is some renewed thought going into hot patching of the dom0 / hypervisor /= etc.=A0 As someone else mentioned, the actual disclosure could be far better off be= ing served by an outside organization like CERT that has established proces= ses in place and can move forward without undue=A0influence=A0from a small = group of players. Thanks John