[PATCH for-4.9] livepatch: Declare live patching as a supported feature

[PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Ross Lagerwall]

Xen Live Patching has been available as tech preview feature since Xen
4.7 and has now had a couple of releases to stabilize. Xen Live patching
has been used by multiple vendors to fix several real-world security
issues without any severe bugs encountered. Additionally, there are now
tests in OSSTest that test live patching to ensure that no regressions
are introduced.

Based on the amount of testing and usage it has had, we are ready to
declare live patching as a 'Supported' feature.

Live patching is slightly peculiar when it comes to support because it
allows the host administrator to break their system rather easily
depending on the content of the live patch.
Because of this, it is worth detailing out the scope of security
support:

* Unprivileged access to live patching operations:
    Live patching operations should only be accessible to privileged
    guests and it shall be treated as a security issue if this is not
    the case.

* Bugs in the patch-application code such that vulnerabilities exist
  after application:
    If a correct live patch is loaded but it is not applied correctly
    such that it might result in an insecure system (e.g. not all
    functions are patched), it shall be treated as a security issue.

* Bugs in livepatch-build-tools creating incorrect live patch that
  results in an insecure host:
    If livepatch-build-tools creates an incorrect live patch that
    results in an insecure host, this shall not be considered a security
    issue. There are too many OSes and toolchains to consider supporting
    this. A live patch should be checked to verify that it is valid
    before loading.

* Loading an incorrect live patch that results in an insecure host or
  host crash:
    If a live patch (whether created using livepatch-build-tools or some
    alternative) is loaded and it results in an insecure host or host
    crash due to the content of the live patch being incorrect or the
    issue being inappropriate to live patch, this is not considered as a
    security issue.

* Bugs in the live patch parsing code (the ELF loader):
    Bugs in the live patch parsing code such as out-of-bounds reads
    caused by invalid ELF files are not considered to be security issues
    because the it can only be triggered by a privileged domain.

* Bugs which allow a guest to prevent the application of a livepatch:
    A guest should not be able to prevent the application of a live
    patch. If an unprivileged guest can prevent the application of a
    live patch, it shall be treated as a security issue.

There are also some generic security questions which it is worth asking:

1) Is guest->host privilege escalation possible?

The new live patching sysctl subops are only accessible to privileged
domains and this is tested by OSSTest with an XTF test.
There is a caveat -- an incorrect live patch can introduce a guest->host
privilege escalation.

2) Is guest user->guest kernel escalation possible?

No, although an incorrect live patch can introduce a guest user->guest
kernel privilege escalation.

3) Is there any information leakage?

The new live patching sysctl subops are only accessible to privileged
domains so it is not possible for an unprivileged guest to access the
list of loaded live patches. This is tested by OSSTest with an XTF test.
There is a caveat -- an incorrect live patch can introduce an
information leakage.

4) Can a Denial-of-Service be triggered?

There are no known ways that an unprivileged guest can prevent a live
patch from being loaded.
Once again, there is a caveat that an incorrect live patch can introduce
an arbitrary denial of service.

Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
---
 xen/common/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index dc8e876..876086c 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -226,7 +226,7 @@ config CRYPTO
 	bool
 
 config LIVEPATCH
-	bool "Live patching support (TECH PREVIEW)"
+	bool "Live patching support"
 	default n
 	depends on HAS_BUILD_ID = "y"
 	---help---
-- 
2.9.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Andrew Cooper]

On 26/06/17 16:36, Ross Lagerwall wrote:
> Xen Live Patching has been available as tech preview feature since Xen
> 4.7 and has now had a couple of releases to stabilize. Xen Live patching
> has been used by multiple vendors to fix several real-world security
> issues without any severe bugs encountered. Additionally, there are now
> tests in OSSTest that test live patching to ensure that no regressions
> are introduced.
>
> Based on the amount of testing and usage it has had, we are ready to
> declare live patching as a 'Supported' feature.
>
> Live patching is slightly peculiar when it comes to support because it
> allows the host administrator to break their system rather easily
> depending on the content of the live patch.
> Because of this, it is worth detailing out the scope of security
> support:
>
> * Unprivileged access to live patching operations:
>     Live patching operations should only be accessible to privileged
>     guests and it shall be treated as a security issue if this is not
>     the case.
>
> * Bugs in the patch-application code such that vulnerabilities exist
>   after application:
>     If a correct live patch is loaded but it is not applied correctly
>     such that it might result in an insecure system (e.g. not all
>     functions are patched), it shall be treated as a security issue.
>
> * Bugs in livepatch-build-tools creating incorrect live patch that
>   results in an insecure host:
>     If livepatch-build-tools creates an incorrect live patch that
>     results in an insecure host, this shall not be considered a security
>     issue. There are too many OSes and toolchains to consider supporting
>     this. A live patch should be checked to verify that it is valid
>     before loading.
>
> * Loading an incorrect live patch that results in an insecure host or
>   host crash:
>     If a live patch (whether created using livepatch-build-tools or some
>     alternative) is loaded and it results in an insecure host or host
>     crash due to the content of the live patch being incorrect or the
>     issue being inappropriate to live patch, this is not considered as a
>     security issue.
>
> * Bugs in the live patch parsing code (the ELF loader):
>     Bugs in the live patch parsing code such as out-of-bounds reads
>     caused by invalid ELF files are not considered to be security issues
>     because the it can only be triggered by a privileged domain.

For these last points, I think it is worth stating that people using
livepatching are expected to test their patches in a test environment first.

>
> * Bugs which allow a guest to prevent the application of a livepatch:
>     A guest should not be able to prevent the application of a live
>     patch. If an unprivileged guest can prevent the application of a
>     live patch, it shall be treated as a security issue.

This one is harder to say.  We know that enough concurrent live
migrations can, which extends to "lots of activity in the guest".  Its
perhaps worth noting the potential workaround of `xl pause $DOM;
xen-livepatch ...; xl unpause`.

I'd prefer that we excluded situations like this from being within
security support.  "guest having heavy workloads" is normal for end
users, so shouldn't constitute a security vulnerability, as there is
nothing we can do about it.

>
> There are also some generic security questions which it is worth asking:
>
> 1) Is guest->host privilege escalation possible?
>
> The new live patching sysctl subops are only accessible to privileged
> domains and this is tested by OSSTest with an XTF test.
> There is a caveat -- an incorrect live patch can introduce a guest->host
> privilege escalation.
>
> 2) Is guest user->guest kernel escalation possible?
>
> No, although an incorrect live patch can introduce a guest user->guest
> kernel privilege escalation.
>
> 3) Is there any information leakage?
>
> The new live patching sysctl subops are only accessible to privileged
> domains so it is not possible for an unprivileged guest to access the
> list of loaded live patches. This is tested by OSSTest with an XTF test.
> There is a caveat -- an incorrect live patch can introduce an
> information leakage.
>
> 4) Can a Denial-of-Service be triggered?
>
> There are no known ways that an unprivileged guest can prevent a live
> patch from being loaded.
> Once again, there is a caveat that an incorrect live patch can introduce
> an arbitrary denial of service.
>
> Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>

This is all good, but this information needs to be in a file in
docs/features/, most probably livepatching.pandoc

> ---
>  xen/common/Kconfig | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/xen/common/Kconfig b/xen/common/Kconfig
> index dc8e876..876086c 100644
> --- a/xen/common/Kconfig
> +++ b/xen/common/Kconfig
> @@ -226,7 +226,7 @@ config CRYPTO
>  	bool
>  
>  config LIVEPATCH
> -	bool "Live patching support (TECH PREVIEW)"
> +	bool "Live patching support"
>  	default n

This default should flip as well.

~Andrew

>  	depends on HAS_BUILD_ID = "y"
>  	---help---


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 26/06/17 17:39, Andrew Cooper wrote:
>> * Bugs which allow a guest to prevent the application of a livepatch:
>>     A guest should not be able to prevent the application of a live
>>     patch. If an unprivileged guest can prevent the application of a
>>     live patch, it shall be treated as a security issue.
> 
> This one is harder to say.  We know that enough concurrent live
> migrations can, which extends to "lots of activity in the guest".  Its
> perhaps worth noting the potential workaround of `xl pause $DOM;
> xen-livepatch ...; xl unpause`.

And what if the guest can prevent itself from being paused?

Or, what if the guest can trigger some other persistent state change
such that livepatching will fail even if the domain is paused (or
destroyed)?

I agree that as long as the patch can be applied after "xl pause", then
the domain cannot be said to be preventing the application of the
livepatch.  But if either 'xl pause' doesn't work, or if livepatching
fails due to a malicious domain's actions after 'xl pause' (or 'xl
destroy'), then it should be treated as a security issue.

> This is all good, but this information needs to be in a file in
> docs/features/, most probably livepatching.pandoc

+1

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Ross Lagerwall]

On 06/26/2017 05:39 PM, Andrew Cooper wrote:
> On 26/06/17 16:36, Ross Lagerwall wrote:
snip
>> * Unprivileged access to live patching operations:
>>      Live patching operations should only be accessible to privileged
>>      guests and it shall be treated as a security issue if this is not
>>      the case.
>>
>> * Bugs in the patch-application code such that vulnerabilities exist
>>    after application:
>>      If a correct live patch is loaded but it is not applied correctly
>>      such that it might result in an insecure system (e.g. not all
>>      functions are patched), it shall be treated as a security issue.
>>
>> * Bugs in livepatch-build-tools creating incorrect live patch that
>>    results in an insecure host:
>>      If livepatch-build-tools creates an incorrect live patch that
>>      results in an insecure host, this shall not be considered a security
>>      issue. There are too many OSes and toolchains to consider supporting
>>      this. A live patch should be checked to verify that it is valid
>>      before loading.
>>
>> * Loading an incorrect live patch that results in an insecure host or
>>    host crash:
>>      If a live patch (whether created using livepatch-build-tools or some
>>      alternative) is loaded and it results in an insecure host or host
>>      crash due to the content of the live patch being incorrect or the
>>      issue being inappropriate to live patch, this is not considered as a
>>      security issue.
>>
>> * Bugs in the live patch parsing code (the ELF loader):
>>      Bugs in the live patch parsing code such as out-of-bounds reads
>>      caused by invalid ELF files are not considered to be security issues
>>      because the it can only be triggered by a privileged domain.
> 
> For these last points, I think it is worth stating that people using
> livepatching are expected to test their patches in a test environment first.

OK.

> 
>>
>> * Bugs which allow a guest to prevent the application of a livepatch:
>>      A guest should not be able to prevent the application of a live
>>      patch. If an unprivileged guest can prevent the application of a
>>      live patch, it shall be treated as a security issue.
> 
> This one is harder to say.  We know that enough concurrent live
> migrations can, which extends to "lots of activity in the guest".  Its
> perhaps worth noting the potential workaround of `xl pause $DOM;
> xen-livepatch ...; xl unpause`.
> 
> I'd prefer that we excluded situations like this from being within
> security support.  "guest having heavy workloads" is normal for end
> users, so shouldn't constitute a security vulnerability, as there is
> nothing we can do about it.

But surely live migrations cannot be triggered by the guest, only the 
host administrator? I don't know of any way of triggering the timeout 
from within an unprivileged guest.

> 
>>
>> There are also some generic security questions which it is worth asking:
>>
>> 1) Is guest->host privilege escalation possible?
>>
>> The new live patching sysctl subops are only accessible to privileged
>> domains and this is tested by OSSTest with an XTF test.
>> There is a caveat -- an incorrect live patch can introduce a guest->host
>> privilege escalation.
>>
>> 2) Is guest user->guest kernel escalation possible?
>>
>> No, although an incorrect live patch can introduce a guest user->guest
>> kernel privilege escalation.
>>
>> 3) Is there any information leakage?
>>
>> The new live patching sysctl subops are only accessible to privileged
>> domains so it is not possible for an unprivileged guest to access the
>> list of loaded live patches. This is tested by OSSTest with an XTF test.
>> There is a caveat -- an incorrect live patch can introduce an
>> information leakage.
>>
>> 4) Can a Denial-of-Service be triggered?
>>
>> There are no known ways that an unprivileged guest can prevent a live
>> patch from being loaded.
>> Once again, there is a caveat that an incorrect live patch can introduce
>> an arbitrary denial of service.
>>
>> Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
> 
> This is all good, but this information needs to be in a file in
> docs/features/, most probably livepatching.pandoc

OK.

> 
>> ---
>>   xen/common/Kconfig | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/xen/common/Kconfig b/xen/common/Kconfig
>> index dc8e876..876086c 100644
>> --- a/xen/common/Kconfig
>> +++ b/xen/common/Kconfig
>> @@ -226,7 +226,7 @@ config CRYPTO
>>   	bool
>>   
>>   config LIVEPATCH
>> -	bool "Live patching support (TECH PREVIEW)"
>> +	bool "Live patching support"
>>   	default n
> 
> This default should flip as well.
> 

OK.

-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Ian Jackson]

George Dunlap writes ("Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature"):
> I agree that as long as the patch can be applied after "xl pause", then
> the domain cannot be said to be preventing the application of the
> livepatch.  But if either 'xl pause' doesn't work, or if livepatching
> fails due to a malicious domain's actions after 'xl pause' (or 'xl
> destroy'), then it should be treated as a security issue.

+1

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 26/06/17 16:36, Ross Lagerwall wrote:
> Xen Live Patching has been available as tech preview feature since Xen
> 4.7 and has now had a couple of releases to stabilize. Xen Live patching
> has been used by multiple vendors to fix several real-world security
> issues without any severe bugs encountered. Additionally, there are now
> tests in OSSTest that test live patching to ensure that no regressions
> are introduced.
> 
> Based on the amount of testing and usage it has had, we are ready to
> declare live patching as a 'Supported' feature.

Great write-up, Ross, thanks.  I more or less agree with everything
except...

> * Bugs in livepatch-build-tools creating incorrect live patch that
>   results in an insecure host:
>     If livepatch-build-tools creates an incorrect live patch that
>     results in an insecure host, this shall not be considered a security
>     issue. There are too many OSes and toolchains to consider supporting
>     this. A live patch should be checked to verify that it is valid
>     before loading.

I'm not sure I follow the argument here.  Suppose in one months' time it
is discovered that livepatch-build-tools, under some circumstances,
creates patches that open up a side vulnerability.  Do you really think
we should just post a fix to the mailing list, without alerting anybody
who may be affected by it?

Rememeber, "security support" doesn't mean, "We promise there are no
bugs".  It means, "If bugs are discovered, we will notify people
according to the XenProject Security Response Process"; and this is not
only for people on the pre-disclosure list, but for everyone *not* on
the list as well, to have one place to find all security-related issues
relevant to Xen.

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Andrew Cooper]

On 26/06/17 17:50, Ross Lagerwall wrote:
> On 06/26/2017 05:39 PM, Andrew Cooper wrote:
>> On 26/06/17 16:36, Ross Lagerwall wrote:
>>
>>>
>>> * Bugs which allow a guest to prevent the application of a livepatch:
>>>      A guest should not be able to prevent the application of a live
>>>      patch. If an unprivileged guest can prevent the application of a
>>>      live patch, it shall be treated as a security issue.
>>
>> This one is harder to say.  We know that enough concurrent live
>> migrations can, which extends to "lots of activity in the guest".  Its
>> perhaps worth noting the potential workaround of `xl pause $DOM;
>> xen-livepatch ...; xl unpause`.
>>
>> I'd prefer that we excluded situations like this from being within
>> security support.  "guest having heavy workloads" is normal for end
>> users, so shouldn't constitute a security vulnerability, as there is
>> nothing we can do about it.
>
> But surely live migrations cannot be triggered by the guest, only the
> host administrator? I don't know of any way of triggering the timeout
> from within an unprivileged guest.

Every VCPU issuing a loop of decrease/increase reservation on a single
gfn will cause a similar quantity of p2m lock contention.

On older AMD hardware, we have to hold the p2m read lock to service
hypercalls, which is why XSA-114 was issued.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Andrew Cooper]

On 26/06/17 17:50, George Dunlap wrote:
> On 26/06/17 17:39, Andrew Cooper wrote:
>>> * Bugs which allow a guest to prevent the application of a livepatch:
>>>     A guest should not be able to prevent the application of a live
>>>     patch. If an unprivileged guest can prevent the application of a
>>>     live patch, it shall be treated as a security issue.
>> This one is harder to say.  We know that enough concurrent live
>> migrations can, which extends to "lots of activity in the guest".  Its
>> perhaps worth noting the potential workaround of `xl pause $DOM;
>> xen-livepatch ...; xl unpause`.
> And what if the guest can prevent itself from being paused?

In which case, that is an XSA in its own right.

The underlying implementation uses XEN_DOMCTL_{,un}pausedomain which
call straight into domain_{un,}pause().  We have very big problems if
the guest has any influence in this...

>
> Or, what if the guest can trigger some other persistent state change
> such that livepatching will fail even if the domain is paused (or
> destroyed)?

Such as?

The guest being able to cause damaging mutative state change in Xen is
clearly a security issue, irrespective of any livepatch involvement.

However, livepatch content (hook function for example) which trips over
state as found in the hypervisor at the point of application is a bad
livepatch, not a vulnerability in livepatching.

> I agree that as long as the patch can be applied after "xl pause", then
> the domain cannot be said to be preventing the application of the
> livepatch.  But if either 'xl pause' doesn't work, or if livepatching
> fails due to a malicious domain's actions after 'xl pause' (or 'xl
> destroy'), then it should be treated as a security issue.

I broadly agree, but these bugs feel like they would be self-standing,
perhaps with an impact to applying a livepatch, rather than XSAs in
livepatching itself.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Andrew Cooper]

On 26/06/17 18:00, George Dunlap wrote:
> On 26/06/17 16:36, Ross Lagerwall wrote:
>> Xen Live Patching has been available as tech preview feature since Xen
>> 4.7 and has now had a couple of releases to stabilize. Xen Live patching
>> has been used by multiple vendors to fix several real-world security
>> issues without any severe bugs encountered. Additionally, there are now
>> tests in OSSTest that test live patching to ensure that no regressions
>> are introduced.
>>
>> Based on the amount of testing and usage it has had, we are ready to
>> declare live patching as a 'Supported' feature.
> Great write-up, Ross, thanks.  I more or less agree with everything
> except...
>
>> * Bugs in livepatch-build-tools creating incorrect live patch that
>>   results in an insecure host:
>>     If livepatch-build-tools creates an incorrect live patch that
>>     results in an insecure host, this shall not be considered a security
>>     issue. There are too many OSes and toolchains to consider supporting
>>     this. A live patch should be checked to verify that it is valid
>>     before loading.
> I'm not sure I follow the argument here.  Suppose in one months' time it
> is discovered that livepatch-build-tools, under some circumstances,
> creates patches that open up a side vulnerability.  Do you really think
> we should just post a fix to the mailing list, without alerting anybody
> who may be affected by it?

There are a million ways this could happen, starting from the simple
cases of accidentally building the livepatch from a non-clean working
tree, or accidentally using a compiler other than the one used to build
the running hypervisor.

We absolutely cannot be in the position of issuing XSAs for situations
like this, because there are too many ways where it definitely will go
wrong, and we'd end up issuing XSAs saying "remember to clean your
working tree before building a livepatch".  This is of course absurd.

IMO, The only viable option is to exclude livepatch-build-tools entirely
from security scope.  It is already the case that people producing
livepatches need to check the resulting livepatch binary for sanity, and
test it suitably in a development environment before use in production.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Julien Grall]

Hi,

On 06/26/2017 04:36 PM, Ross Lagerwall wrote:
> Xen Live Patching has been available as tech preview feature since Xen
> 4.7 and has now had a couple of releases to stabilize. Xen Live patching
> has been used by multiple vendors to fix several real-world security
> issues without any severe bugs encountered. Additionally, there are now
> tests in OSSTest that test live patching to ensure that no regressions
> are introduced.
> 
> Based on the amount of testing and usage it has had, we are ready to
> declare live patching as a 'Supported' feature.

There are only test for x86 and amd64. We likely want to have those test 
enabled for all architectures by default.

Also, I am not aware of anyone using in production livepatch on ARM64 
and ARM32. So did anyone give a good kick at the ARM implementation?

If not, then we should  do it before even considering as a supported 
feature for ARM.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Konrad Rzeszutek Wilk]

On Mon, Jun 26, 2017 at 07:29:22PM +0100, Julien Grall wrote:
> Hi,
> 
> On 06/26/2017 04:36 PM, Ross Lagerwall wrote:
> > Xen Live Patching has been available as tech preview feature since Xen
> > 4.7 and has now had a couple of releases to stabilize. Xen Live patching
> > has been used by multiple vendors to fix several real-world security
> > issues without any severe bugs encountered. Additionally, there are now
> > tests in OSSTest that test live patching to ensure that no regressions
> > are introduced.
> > 
> > Based on the amount of testing and usage it has had, we are ready to
> > declare live patching as a 'Supported' feature.
> 
> There are only test for x86 and amd64. We likely want to have those test

The test-cases are also for ARM32.

> enabled for all architectures by default.

And the OSSTest can test all of those.
> 
> Also, I am not aware of anyone using in production livepatch on ARM64 and
> ARM32. So did anyone give a good kick at the ARM implementation?

I am not aware of anybody using it on production on ARM32 or ARM64.

The test-cases are there, the code is there, but yes nobody has kicked
the tires on ARM32/ARM64 extensively with it. I would be excited to
see vendors that use it and their reports but I am not aware of any.

> 
> If not, then we should  do it before even considering as a supported feature
> for ARM.

OK. Perhaps then only for x86 until ARM operational users pipe up?

> 
> Cheers,
> 
> -- 
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Jan Beulich]

>>> Andrew Cooper <andrew.cooper3@citrix.com> 06/26/17 6:40 PM >>>
>> --- a/xen/common/Kconfig
>> +++ b/xen/common/Kconfig
>> @@ -226,7 +226,7 @@ config CRYPTO
>>  	bool
>>  
>>  config LIVEPATCH
>> -	bool "Live patching support (TECH PREVIEW)"
>> +	bool "Live patching support"
>>  	default n
>
>This default should flip as well.

For unstable maybe. But please not for 4.9 - people shouldn't be caught by
surprise after 9 RCs that a config option's default changes. Furthermore, if
we default it to y, will there be much of a difference to simply removing the
config option?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Julien Grall]

Hi Jan,

On 06/27/2017 07:04 AM, Jan Beulich wrote:
>>>> Andrew Cooper <andrew.cooper3@citrix.com> 06/26/17 6:40 PM >>>
>>> --- a/xen/common/Kconfig
>>> +++ b/xen/common/Kconfig
>>> @@ -226,7 +226,7 @@ config CRYPTO
>>>   	bool
>>>   
>>>   config LIVEPATCH
>>> -	bool "Live patching support (TECH PREVIEW)"
>>> +	bool "Live patching support"
>>>   	default n
>>
>> This default should flip as well.
> 
> For unstable maybe. But please not for 4.9 - people shouldn't be caught by
> surprise after 9 RCs that a config option's default changes. Furthermore, if
> we default it to y, will there be much of a difference to simply removing the
> config option?

I think we should allow user to disable livepatch at build time. I have 
mind any people looking at using Xen in constraint environment and 
require some certifications. They will likely want to disable some part 
of the Xen.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Julien Grall]

On 06/26/2017 10:07 PM, Konrad Rzeszutek Wilk wrote:
> On Mon, Jun 26, 2017 at 07:29:22PM +0100, Julien Grall wrote:
>> Hi,
>>
>> On 06/26/2017 04:36 PM, Ross Lagerwall wrote:
>>> Xen Live Patching has been available as tech preview feature since Xen
>>> 4.7 and has now had a couple of releases to stabilize. Xen Live patching
>>> has been used by multiple vendors to fix several real-world security
>>> issues without any severe bugs encountered. Additionally, there are now
>>> tests in OSSTest that test live patching to ensure that no regressions
>>> are introduced.
>>>
>>> Based on the amount of testing and usage it has had, we are ready to
>>> declare live patching as a 'Supported' feature.
>>
>> There are only test for x86 and amd64. We likely want to have those test
> 
> The test-cases are also for ARM32.
> 
>> enabled for all architectures by default.
> 
> And the OSSTest can test all of those.

Can we enable them by default? I know that we limited the number of 
tests for ARM64 due to limited bandwidth. But I don't think we have 
anything preventing it on ARM32.

>>
>> Also, I am not aware of anyone using in production livepatch on ARM64 and
>> ARM32. So did anyone give a good kick at the ARM implementaton?
> 
> I am not aware of anybody using it on production on ARM32 or ARM64.
> 
> The test-cases are there, the code is there, but yes nobody has kicked
> the tires on ARM32/ARM64 extensively with it. I would be excited to
> see vendors that use it and their reports but I am not aware of any.
> 
>>
>> If not, then we should  do it before even considering as a supported feature
>> for ARM.
> 
> OK. Perhaps then only for x86 until ARM operational users pipe up?

That would be my preference. My main concern is to handle security issue 
afterwards because we didn't give any kick at the code.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Lars Kurth]

Hi all,

there was also a discussion on IRC, which Ian said we should formally
summarise in e-mail, just so there is no doubt. So here is my go at it. As
far as I can tell - besides the technical discussion in this thread, there
are several issues which need to be clarified:

* For Xen 4.9 we can declare live patching supported, without spinning
another RC to update the in-tree documentation: in other words, we would
apply the documentation/policy changes + to the 4.9 tree sometimes after
this discussion has been concluded. In effect this means that
docs/features/livepatching.pandoc (or similar) and associated changes to
KCONFIG options would not show up until Xen 4.9.1 is spun, but the
security team would treat live patching as supported for 4.9. In other
words for now, we can update the table in the wiki
(https://wiki.xenproject.org/wiki/Xen_Project_Release_Features) and live
with in-tree artefacts being out-of-sync with the support status for a few
months. We need to fix this anyway in-tree and there is a concrete
proposal which should be discussed at the summit.

* There was a proposal to declare live patching supported for older
releases (aka "back port" docs/features/livepatching.pandoc), but Royger
pointed out that the toolstack in question needs to support buildid. If
so, we should include back-porting requests and d

* Julien pointed out that maybe we shouldn't declare live patching as
supported for ARM32/64. I don't see an issue to declare it supported for
x86/amd64 only for now. But it is obviously up to committers to make that
call.

I think that covers the ghist of the IRC discussion

Regards
Lars

On 27/06/2017, 08:24, "Julien Grall" <julien.grall@arm.com> wrote:

>
>
>On 06/26/2017 10:07 PM, Konrad Rzeszutek Wilk wrote:
>> On Mon, Jun 26, 2017 at 07:29:22PM +0100, Julien Grall wrote:
>>> Hi,
>>>
>>> On 06/26/2017 04:36 PM, Ross Lagerwall wrote:
>>>> Xen Live Patching has been available as tech preview feature since Xen
>>>> 4.7 and has now had a couple of releases to stabilize. Xen Live
>>>>patching
>>>> has been used by multiple vendors to fix several real-world security
>>>> issues without any severe bugs encountered. Additionally, there are
>>>>now
>>>> tests in OSSTest that test live patching to ensure that no regressions
>>>> are introduced.
>>>>
>>>> Based on the amount of testing and usage it has had, we are ready to
>>>> declare live patching as a 'Supported' feature.
>>>
>>> There are only test for x86 and amd64. We likely want to have those
>>>test
>> 
>> The test-cases are also for ARM32.
>> 
>>> enabled for all architectures by default.
>> 
>> And the OSSTest can test all of those.
>
>Can we enable them by default? I know that we limited the number of
>tests for ARM64 due to limited bandwidth. But I don't think we have
>anything preventing it on ARM32.
>
>>>
>>> Also, I am not aware of anyone using in production livepatch on ARM64
>>>and
>>> ARM32. So did anyone give a good kick at the ARM implementaton?
>> 
>> I am not aware of anybody using it on production on ARM32 or ARM64.
>> 
>> The test-cases are there, the code is there, but yes nobody has kicked
>> the tires on ARM32/ARM64 extensively with it. I would be excited to
>> see vendors that use it and their reports but I am not aware of any.
>> 
>>>
>>> If not, then we should  do it before even considering as a supported
>>>feature
>>> for ARM.
>> 
>> OK. Perhaps then only for x86 until ARM operational users pipe up?
>
>That would be my preference. My main concern is to handle security issue
>afterwards because we didn't give any kick at the code.
>
>Cheers,
>
>-- 
>Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 26/06/17 18:18, Andrew Cooper wrote:
> On 26/06/17 17:50, George Dunlap wrote:
>> On 26/06/17 17:39, Andrew Cooper wrote:
>>>> * Bugs which allow a guest to prevent the application of a livepatch:
>>>>     A guest should not be able to prevent the application of a live
>>>>     patch. If an unprivileged guest can prevent the application of a
>>>>     live patch, it shall be treated as a security issue.
>>> This one is harder to say.  We know that enough concurrent live
>>> migrations can, which extends to "lots of activity in the guest".  Its
>>> perhaps worth noting the potential workaround of `xl pause $DOM;
>>> xen-livepatch ...; xl unpause`.
>> And what if the guest can prevent itself from being paused?
> 
> In which case, that is an XSA in its own right.
> 
> The underlying implementation uses XEN_DOMCTL_{,un}pausedomain which
> call straight into domain_{un,}pause().  We have very big problems if
> the guest has any influence in this...
> 
>>
>> Or, what if the guest can trigger some other persistent state change
>> such that livepatching will fail even if the domain is paused (or
>> destroyed)?
> 
> Such as?
> 
> The guest being able to cause damaging mutative state change in Xen is
> clearly a security issue, irrespective of any livepatch involvement.
> 
> However, livepatch content (hook function for example) which trips over
> state as found in the hypervisor at the point of application is a bad
> livepatch, not a vulnerability in livepatching.
> 
>> I agree that as long as the patch can be applied after "xl pause", then
>> the domain cannot be said to be preventing the application of the
>> livepatch.  But if either 'xl pause' doesn't work, or if livepatching
>> fails due to a malicious domain's actions after 'xl pause' (or 'xl
>> destroy'), then it should be treated as a security issue.
> 
> I broadly agree, but these bugs feel like they would be self-standing,
> perhaps with an impact to applying a livepatch, rather than XSAs in
> livepatching itself.

So let me get this right.

You think that all possible cases in which a guest can persistently
prevent a livepatch from being applied would already be a security issue
for other reasons.

Therefore, you think we should include a paragraph in our security
support statement specifically stating that we do not provide security
support if the guest can prevent a livepatch.

Is that correct?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 26/06/17 18:30, Andrew Cooper wrote:
> On 26/06/17 18:00, George Dunlap wrote:
>> On 26/06/17 16:36, Ross Lagerwall wrote:
>>> Xen Live Patching has been available as tech preview feature since Xen
>>> 4.7 and has now had a couple of releases to stabilize. Xen Live patching
>>> has been used by multiple vendors to fix several real-world security
>>> issues without any severe bugs encountered. Additionally, there are now
>>> tests in OSSTest that test live patching to ensure that no regressions
>>> are introduced.
>>>
>>> Based on the amount of testing and usage it has had, we are ready to
>>> declare live patching as a 'Supported' feature.
>> Great write-up, Ross, thanks.  I more or less agree with everything
>> except...
>>
>>> * Bugs in livepatch-build-tools creating incorrect live patch that
>>>   results in an insecure host:
>>>     If livepatch-build-tools creates an incorrect live patch that
>>>     results in an insecure host, this shall not be considered a security
>>>     issue. There are too many OSes and toolchains to consider supporting
>>>     this. A live patch should be checked to verify that it is valid
>>>     before loading.
>> I'm not sure I follow the argument here.  Suppose in one months' time it
>> is discovered that livepatch-build-tools, under some circumstances,
>> creates patches that open up a side vulnerability.  Do you really think
>> we should just post a fix to the mailing list, without alerting anybody
>> who may be affected by it?
> 
> There are a million ways this could happen, starting from the simple
> cases of accidentally building the livepatch from a non-clean working
> tree, or accidentally using a compiler other than the one used to build
> the running hypervisor.
> 
> We absolutely cannot be in the position of issuing XSAs for situations
> like this, because there are too many ways where it definitely will go
> wrong, and we'd end up issuing XSAs saying "remember to clean your
> working tree before building a livepatch".  This is of course absurd.

Your argument is that because we do not issue XSAs for *user mistakes*,
that therefore we should not issue XSAs for *bugs in the tool*.

That is of course absurd.  We do not issue XSAs for user mistakes in
building the hypervisor either (for instance, switching gcc versions
without cleaning the hypervisor tree), and yet we still issue XSAs for
bugs in the hypervisor itself.

> IMO, The only viable option is to exclude livepatch-build-tools entirely
> from security scope.  It is already the case that people producing
> livepatches need to check the resulting livepatch binary for sanity, and
> test it suitably in a development environment before use in production.

Look, it sounds like right now you are going through all the livepatches
with a fine-tooth comb *because* the tools are (or recently have been)
unreliable.  But at some point in the future, the patch generation
mechanism will become more reliable.  After 20 XSAs over six months in
which the livepatch tool created the correct patch, you will become more
complacent.  You won't look as closely; it's human nature.

You seem to be simply refusing to use your imagination.  Step back.
Imagine yourself in one year.  You come to the office and find an e-mail
on security@ which says, "Livepatch tools open a security hole when
compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
gcc x.yy, so you take a closer look at that livepatch, only to discover
that the livepatches generated actually do contain the bug, but you
missed it because ${LATEST-[0,1]} were perfectly fine (since they used
newer versions of gcc), the difference was subtle, and it passed all the
functional tests.

Now all of the customers that have applied those patches are vulnerable.

Do you:

1. Tell the reporter to post it publicly to xen-devel immediately, since
livepatch tools are not security supported -- thus "zero-day"-ing all
your customers (as well as anyone else who happens to have used x.yy to
build a hypervisor)?

2. Secretly take advantage of Citrix' privileged position on the
security list, and try to get an update out to your customers before it
gets announced (but allowing everyone *else* using gcc x.yy to
experience a zero-day)?

3. Issue an XSA so that everyone has the opportunity to fix things up
before making a public announcement, and so that anyone not on the
embargo list gets an alert, so they know to either update their own
livepatches, or look for updates from their software provider?

I think #3 is the only possible choice.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Ian Jackson]

Julien Grall writes ("Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature"):
> On 06/26/2017 10:07 PM, Konrad Rzeszutek Wilk wrote:
> > On Mon, Jun 26, 2017 at 07:29:22PM +0100, Julien Grall wrote:
> >> enabled for all architectures by default.
> > 
> > And the OSSTest can test all of those.
> 
> Can we enable them by default? I know that we limited the number of 
> tests for ARM64 due to limited bandwidth. But I don't think we have 
> anything preventing it on ARM32.

We're slightly short of armhf bandwidth too and are about to become
more so, with the expansion of the facility (and our difficulty
getting more armhf hardware).

> > OK. Perhaps then only for x86 until ARM operational users pipe up?
> 
> That would be my preference. My main concern is to handle security issue 
> afterwards because we didn't give any kick at the code.

SGTM

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Andrew Cooper]

On 27/06/17 09:37, George Dunlap wrote:
> On 26/06/17 18:18, Andrew Cooper wrote:
>> On 26/06/17 17:50, George Dunlap wrote:
>>> On 26/06/17 17:39, Andrew Cooper wrote:
>>>>> * Bugs which allow a guest to prevent the application of a livepatch:
>>>>>     A guest should not be able to prevent the application of a live
>>>>>     patch. If an unprivileged guest can prevent the application of a
>>>>>     live patch, it shall be treated as a security issue.
>>>> This one is harder to say.  We know that enough concurrent live
>>>> migrations can, which extends to "lots of activity in the guest".  Its
>>>> perhaps worth noting the potential workaround of `xl pause $DOM;
>>>> xen-livepatch ...; xl unpause`.
>>> And what if the guest can prevent itself from being paused?
>> In which case, that is an XSA in its own right.
>>
>> The underlying implementation uses XEN_DOMCTL_{,un}pausedomain which
>> call straight into domain_{un,}pause().  We have very big problems if
>> the guest has any influence in this...
>>
>>> Or, what if the guest can trigger some other persistent state change
>>> such that livepatching will fail even if the domain is paused (or
>>> destroyed)?
>> Such as?
>>
>> The guest being able to cause damaging mutative state change in Xen is
>> clearly a security issue, irrespective of any livepatch involvement.
>>
>> However, livepatch content (hook function for example) which trips over
>> state as found in the hypervisor at the point of application is a bad
>> livepatch, not a vulnerability in livepatching.
>>
>>> I agree that as long as the patch can be applied after "xl pause", then
>>> the domain cannot be said to be preventing the application of the
>>> livepatch.  But if either 'xl pause' doesn't work, or if livepatching
>>> fails due to a malicious domain's actions after 'xl pause' (or 'xl
>>> destroy'), then it should be treated as a security issue.
>> I broadly agree, but these bugs feel like they would be self-standing,
>> perhaps with an impact to applying a livepatch, rather than XSAs in
>> livepatching itself.
> So let me get this right.
>
> You think that all possible cases in which a guest can persistently
> prevent a livepatch from being applied would already be a security issue
> for other reasons.

Yes.  The only possible ways a guest (potentially) has of preventing
livepatching from functioning (in the case that it is paused) is by
mechanisms such as causing memory corruption, mis-refcounting or by
having already achieved code injection.

> Therefore, you think we should include a paragraph in our security
> support statement specifically stating that we do not provide security
> support if the guest can prevent a livepatch.
>
> Is that correct?

Incorrect.  I don't think it is worth mentioning at all.

By calling it out, you are adding confusion to the area, as it is
redundant with the rest of our policy.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Ian Jackson]

Lars Kurth writes ("Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature"):
> * For Xen 4.9 we can declare live patching supported, without spinning
> another RC to update the in-tree documentation: in other words, we would
> apply the documentation/policy changes + to the 4.9 tree sometimes after
> this discussion has been concluded. In effect this means that
> docs/features/livepatching.pandoc (or similar) and associated changes to
> KCONFIG options would not show up until Xen 4.9.1 is spun,

I think that the effective version of these documents is not the one
in the most recent release, but the one in (one of the) git branches.
In this case, I think xenbits:xen.git#stable-4.9.

(This is an essential way to look at it because we have releases which
are still in security support, and might need updates to their support
status, but which are out of functional support and do not get
releases of updates.)

> * There was a proposal to declare live patching supported for older
> releases (aka "back port" docs/features/livepatching.pandoc), but Royger
> pointed out that the toolstack in question needs to support buildid. If
> so, we should include back-porting requests and d

("and do it then" or something?)  Yes.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 27/06/17 11:47, Andrew Cooper wrote:
> On 27/06/17 09:37, George Dunlap wrote:
>> On 26/06/17 18:18, Andrew Cooper wrote:
>>> On 26/06/17 17:50, George Dunlap wrote:
>>>> On 26/06/17 17:39, Andrew Cooper wrote:
>>>>>> * Bugs which allow a guest to prevent the application of a livepatch:
>>>>>>     A guest should not be able to prevent the application of a live
>>>>>>     patch. If an unprivileged guest can prevent the application of a
>>>>>>     live patch, it shall be treated as a security issue.
>>>>> This one is harder to say.  We know that enough concurrent live
>>>>> migrations can, which extends to "lots of activity in the guest".  Its
>>>>> perhaps worth noting the potential workaround of `xl pause $DOM;
>>>>> xen-livepatch ...; xl unpause`.
>>>> And what if the guest can prevent itself from being paused?
>>> In which case, that is an XSA in its own right.
>>>
>>> The underlying implementation uses XEN_DOMCTL_{,un}pausedomain which
>>> call straight into domain_{un,}pause().  We have very big problems if
>>> the guest has any influence in this...
>>>
>>>> Or, what if the guest can trigger some other persistent state change
>>>> such that livepatching will fail even if the domain is paused (or
>>>> destroyed)?
>>> Such as?
>>>
>>> The guest being able to cause damaging mutative state change in Xen is
>>> clearly a security issue, irrespective of any livepatch involvement.
>>>
>>> However, livepatch content (hook function for example) which trips over
>>> state as found in the hypervisor at the point of application is a bad
>>> livepatch, not a vulnerability in livepatching.
>>>
>>>> I agree that as long as the patch can be applied after "xl pause", then
>>>> the domain cannot be said to be preventing the application of the
>>>> livepatch.  But if either 'xl pause' doesn't work, or if livepatching
>>>> fails due to a malicious domain's actions after 'xl pause' (or 'xl
>>>> destroy'), then it should be treated as a security issue.
>>> I broadly agree, but these bugs feel like they would be self-standing,
>>> perhaps with an impact to applying a livepatch, rather than XSAs in
>>> livepatching itself.
>> So let me get this right.
>>
>> You think that all possible cases in which a guest can persistently
>> prevent a livepatch from being applied would already be a security issue
>> for other reasons.
> 
> Yes.  The only possible ways a guest (potentially) has of preventing
> livepatching from functioning (in the case that it is paused) is by
> mechanisms such as causing memory corruption, mis-refcounting or by
> having already achieved code injection.
> 
>> Therefore, you think we should include a paragraph in our security
>> support statement specifically stating that we do not provide security
>> support if the guest can prevent a livepatch.
>>
>> Is that correct?
> 
> Incorrect.  I don't think it is worth mentioning at all.
> 
> By calling it out, you are adding confusion to the area, as it is
> redundant with the rest of our policy.

So we shouldn't tell people we'll issue an XSA, because they should
already be able to infer from other things in our policy that an XSA
will be issued?  And if we clearly state that we'll issue an XSA in
those circumstances, they'll be confused?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Lars Kurth]

On 27/06/2017, 11:49, "Ian Jackson" <ian.jackson@eu.citrix.com> wrote:


>> * There was a proposal to declare live patching supported for older
>> releases (aka "back port" docs/features/livepatching.pandoc), but Royger
>> pointed out that the toolstack in question needs to support buildid. If
>> so, we should include back-porting requests and d
>
>("and do it then" or something?)  Yes.

Correct: "and do it then" ... Must have deleted the text when hovering
over it. 
Lars

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Jan Beulich]

>>> Julien Grall <julien.grall@arm.com> 06/27/17 9:20 AM >>>
>On 06/27/2017 07:04 AM, Jan Beulich wrote:
>>>>> Andrew Cooper <andrew.cooper3@citrix.com> 06/26/17 6:40 PM >>>
>>>> --- a/xen/common/Kconfig
>>>> +++ b/xen/common/Kconfig
>>>> @@ -226,7 +226,7 @@ config CRYPTO
>>>>   	bool
>>>>   
>>>>   config LIVEPATCH
>>>> -	bool "Live patching support (TECH PREVIEW)"
>>>> +	bool "Live patching support"
>>>>   	default n
>>>
>>> This default should flip as well.
>> 
>> For unstable maybe. But please not for 4.9 - people shouldn't be caught by
>> surprise after 9 RCs that a config option's default changes. Furthermore, if
>> we default it to y, will there be much of a difference to simply removing the
>> config option?
>
>I think we should allow user to disable livepatch at build time. I have 
>mind any people looking at using Xen in constraint environment and 
>require some certifications. They will likely want to disable some part 
>of the Xen.

The at least I'd like to see it gain an EXPERT dependency. For now our
overall goal still is to limit the number of possible different configurations.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On Tue, Jun 27, 2017 at 7:04 AM, Jan Beulich <jbeulich@suse.com> wrote:
>>>> Andrew Cooper <andrew.cooper3@citrix.com> 06/26/17 6:40 PM >>>
>>> --- a/xen/common/Kconfig
>>> +++ b/xen/common/Kconfig
>>> @@ -226,7 +226,7 @@ config CRYPTO
>>>      bool
>>>
>>>  config LIVEPATCH
>>> -    bool "Live patching support (TECH PREVIEW)"
>>> +    bool "Live patching support"
>>>      default n
>>
>>This default should flip as well.
>
> For unstable maybe. But please not for 4.9 - people shouldn't be caught by
> surprise after 9 RCs that a config option's default changes.

Leaving it the same as the RCs makes sense to me.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Ross Lagerwall]

On 06/27/2017 10:17 AM, George Dunlap wrote:
> On 26/06/17 18:30, Andrew Cooper wrote:
>> On 26/06/17 18:00, George Dunlap wrote:
>>> On 26/06/17 16:36, Ross Lagerwall wrote:
...
>>
>> We absolutely cannot be in the position of issuing XSAs for situations
>> like this, because there are too many ways where it definitely will go
>> wrong, and we'd end up issuing XSAs saying "remember to clean your
>> working tree before building a livepatch".  This is of course absurd.
> 
> Your argument is that because we do not issue XSAs for *user mistakes*,
> that therefore we should not issue XSAs for *bugs in the tool*.
> 
> That is of course absurd.  We do not issue XSAs for user mistakes in
> building the hypervisor either (for instance, switching gcc versions
> without cleaning the hypervisor tree), and yet we still issue XSAs for
> bugs in the hypervisor itself.
> 
>> IMO, The only viable option is to exclude livepatch-build-tools entirely
>> from security scope.  It is already the case that people producing
>> livepatches need to check the resulting livepatch binary for sanity, and
>> test it suitably in a development environment before use in production.
> 
> Look, it sounds like right now you are going through all the livepatches
> with a fine-tooth comb *because* the tools are (or recently have been)
> unreliable.  But at some point in the future, the patch generation
> mechanism will become more reliable.  After 20 XSAs over six months in
> which the livepatch tool created the correct patch, you will become more
> complacent.  You won't look as closely; it's human nature.
> 
> You seem to be simply refusing to use your imagination.  Step back.
> Imagine yourself in one year.  You come to the office and find an e-mail
> on security@ which says, "Livepatch tools open a security hole when
> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
> gcc x.yy, so you take a closer look at that livepatch, only to discover
> that the livepatches generated actually do contain the bug, but you
> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
> newer versions of gcc), the difference was subtle, and it passed all the
> functional tests.
> 
> Now all of the customers that have applied those patches are vulnerable.
> 
> Do you:
> 
> 1. Tell the reporter to post it publicly to xen-devel immediately, since
> livepatch tools are not security supported -- thus "zero-day"-ing all
> your customers (as well as anyone else who happens to have used x.yy to
> build a hypervisor)?
> 
> 2. Secretly take advantage of Citrix' privileged position on the
> security list, and try to get an update out to your customers before it
> gets announced (but allowing everyone *else* using gcc x.yy to
> experience a zero-day)?
> 
> 3. Issue an XSA so that everyone has the opportunity to fix things up
> before making a public announcement, and so that anyone not on the
> embargo list gets an alert, so they know to either update their own
> livepatches, or look for updates from their software provider?
> 
> I think #3 is the only possible choice.
> 
>   -George
> 

The issue here is that any bug in livepatch-build-tools which still 
results in output being generated would be a security issue, because 
someone might have used it to patch a security issue. 
livepatch-build-tools is certainly not stable enough yet (ever?) to be 
treated in this fashion.

-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Konrad Rzeszutek Wilk]

. snip..
> > Now all of the customers that have applied those patches are vulnerable.
> > 
> > Do you:
> > 
> > 1. Tell the reporter to post it publicly to xen-devel immediately, since
> > livepatch tools are not security supported -- thus "zero-day"-ing all
> > your customers (as well as anyone else who happens to have used x.yy to
> > build a hypervisor)?
> > 
> > 2. Secretly take advantage of Citrix' privileged position on the
> > security list, and try to get an update out to your customers before it
> > gets announced (but allowing everyone *else* using gcc x.yy to
> > experience a zero-day)?
> > 
> > 3. Issue an XSA so that everyone has the opportunity to fix things up
> > before making a public announcement, and so that anyone not on the
> > embargo list gets an alert, so they know to either update their own
> > livepatches, or look for updates from their software provider?
> > 
> > I think #3 is the only possible choice.
> > 
> >   -George
> > 
> 
> The issue here is that any bug in livepatch-build-tools which still results
> in output being generated would be a security issue, because someone might
> have used it to patch a security issue. livepatch-build-tools is certainly
> not stable enough yet (ever?) to be treated in this fashion.
> 

To add a bit. The livepatch-build-tools does not have to be used to 
create the livepatches. One can use other tools to create the livepatches
(like for example the test-cases).

And there is a nice design http://xenbits.xen.org/docs/unstable/misc/livepatch.html
(see "Design of payload format") which describes what the format of this livepatch
MUST be.

> -- 
> Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
> On 06/27/2017 10:17 AM, George Dunlap wrote:
>> On 26/06/17 18:30, Andrew Cooper wrote:
>>> On 26/06/17 18:00, George Dunlap wrote:
>>>> On 26/06/17 16:36, Ross Lagerwall wrote:
> ...
>>>
>>> We absolutely cannot be in the position of issuing XSAs for situations
>>> like this, because there are too many ways where it definitely will go
>>> wrong, and we'd end up issuing XSAs saying "remember to clean your
>>> working tree before building a livepatch".  This is of course absurd.
>>
>> Your argument is that because we do not issue XSAs for *user mistakes*,
>> that therefore we should not issue XSAs for *bugs in the tool*.
>>
>> That is of course absurd.  We do not issue XSAs for user mistakes in
>> building the hypervisor either (for instance, switching gcc versions
>> without cleaning the hypervisor tree), and yet we still issue XSAs for
>> bugs in the hypervisor itself.
>>
>>> IMO, The only viable option is to exclude livepatch-build-tools entirely
>>> from security scope.  It is already the case that people producing
>>> livepatches need to check the resulting livepatch binary for sanity, and
>>> test it suitably in a development environment before use in production.
>>
>> Look, it sounds like right now you are going through all the livepatches
>> with a fine-tooth comb *because* the tools are (or recently have been)
>> unreliable.  But at some point in the future, the patch generation
>> mechanism will become more reliable.  After 20 XSAs over six months in
>> which the livepatch tool created the correct patch, you will become more
>> complacent.  You won't look as closely; it's human nature.
>>
>> You seem to be simply refusing to use your imagination.  Step back.
>> Imagine yourself in one year.  You come to the office and find an e-mail
>> on security@ which says, "Livepatch tools open a security hole when
>> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
>> gcc x.yy, so you take a closer look at that livepatch, only to discover
>> that the livepatches generated actually do contain the bug, but you
>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
>> newer versions of gcc), the difference was subtle, and it passed all the
>> functional tests.
>>
>> Now all of the customers that have applied those patches are vulnerable.
>>
>> Do you:
>>
>> 1. Tell the reporter to post it publicly to xen-devel immediately, since
>> livepatch tools are not security supported -- thus "zero-day"-ing all
>> your customers (as well as anyone else who happens to have used x.yy to
>> build a hypervisor)?
>>
>> 2. Secretly take advantage of Citrix' privileged position on the
>> security list, and try to get an update out to your customers before it
>> gets announced (but allowing everyone *else* using gcc x.yy to
>> experience a zero-day)?
>>
>> 3. Issue an XSA so that everyone has the opportunity to fix things up
>> before making a public announcement, and so that anyone not on the
>> embargo list gets an alert, so they know to either update their own
>> livepatches, or look for updates from their software provider?
>>
>> I think #3 is the only possible choice.
>>
>>   -George
>>
> 
> The issue here is that any bug in livepatch-build-tools which still
> results in output being generated would be a security issue, because
> someone might have used it to patch a security issue.
> livepatch-build-tools is certainly not stable enough yet (ever?) to be
> treated in this fashion.

You didn't answer my question.  If the situation described happens, what
position do you want Andrew to be put in?  (If I missed a potential
action, let me know.)

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Ross Lagerwall]

On 06/30/2017 02:42 PM, George Dunlap wrote:
> On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
>> On 06/27/2017 10:17 AM, George Dunlap wrote:
>>> On 26/06/17 18:30, Andrew Cooper wrote:
>>>> On 26/06/17 18:00, George Dunlap wrote:
>>>>> On 26/06/17 16:36, Ross Lagerwall wrote:
>> ...
>>>>
>>>> We absolutely cannot be in the position of issuing XSAs for situations
>>>> like this, because there are too many ways where it definitely will go
>>>> wrong, and we'd end up issuing XSAs saying "remember to clean your
>>>> working tree before building a livepatch".  This is of course absurd.
>>>
>>> Your argument is that because we do not issue XSAs for *user mistakes*,
>>> that therefore we should not issue XSAs for *bugs in the tool*.
>>>
>>> That is of course absurd.  We do not issue XSAs for user mistakes in
>>> building the hypervisor either (for instance, switching gcc versions
>>> without cleaning the hypervisor tree), and yet we still issue XSAs for
>>> bugs in the hypervisor itself.
>>>
>>>> IMO, The only viable option is to exclude livepatch-build-tools entirely
>>>> from security scope.  It is already the case that people producing
>>>> livepatches need to check the resulting livepatch binary for sanity, and
>>>> test it suitably in a development environment before use in production.
>>>
>>> Look, it sounds like right now you are going through all the livepatches
>>> with a fine-tooth comb *because* the tools are (or recently have been)
>>> unreliable.  But at some point in the future, the patch generation
>>> mechanism will become more reliable.  After 20 XSAs over six months in
>>> which the livepatch tool created the correct patch, you will become more
>>> complacent.  You won't look as closely; it's human nature.
>>>
>>> You seem to be simply refusing to use your imagination.  Step back.
>>> Imagine yourself in one year.  You come to the office and find an e-mail
>>> on security@ which says, "Livepatch tools open a security hole when
>>> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
>>> gcc x.yy, so you take a closer look at that livepatch, only to discover
>>> that the livepatches generated actually do contain the bug, but you
>>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
>>> newer versions of gcc), the difference was subtle, and it passed all the
>>> functional tests.
>>>
>>> Now all of the customers that have applied those patches are vulnerable.
>>>
>>> Do you:
>>>
>>> 1. Tell the reporter to post it publicly to xen-devel immediately, since
>>> livepatch tools are not security supported -- thus "zero-day"-ing all
>>> your customers (as well as anyone else who happens to have used x.yy to
>>> build a hypervisor)?
>>>
>>> 2. Secretly take advantage of Citrix' privileged position on the
>>> security list, and try to get an update out to your customers before it
>>> gets announced (but allowing everyone *else* using gcc x.yy to
>>> experience a zero-day)?
>>>
>>> 3. Issue an XSA so that everyone has the opportunity to fix things up
>>> before making a public announcement, and so that anyone not on the
>>> embargo list gets an alert, so they know to either update their own
>>> livepatches, or look for updates from their software provider?
>>>
>>> I think #3 is the only possible choice.
>>>
>>>    -George
>>>
>>
>> The issue here is that any bug in livepatch-build-tools which still
>> results in output being generated would be a security issue, because
>> someone might have used it to patch a security issue.
>> livepatch-build-tools is certainly not stable enough yet (ever?) to be
>> treated in this fashion.
> 
> You didn't answer my question.  If the situation described happens, what
> position do you want Andrew to be put in?  (If I missed a potential
> action, let me know.)
> 

I would choose #3 as it is the obvious choice. But I still don't think 
it is a sensible idea to have security support for the build tools, at 
least at this point. The same scenario could be posed for a nasty bug 
that affects Xen 4.4 only, but it is now just out of security support. 
IMO something being not supported doesn't preclude it from having an XSA 
released if there is a particularly nasty vulnerability found.

-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Roger Pau Monné]

On Mon, Jul 03, 2017 at 03:53:42PM +0100, Ross Lagerwall wrote:
> On 06/30/2017 02:42 PM, George Dunlap wrote:
> > On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
> > > On 06/27/2017 10:17 AM, George Dunlap wrote:
> > > > On 26/06/17 18:30, Andrew Cooper wrote:
> > > > > On 26/06/17 18:00, George Dunlap wrote:
> > > > > > On 26/06/17 16:36, Ross Lagerwall wrote:
> > > ...
> > > > > 
> > > > > We absolutely cannot be in the position of issuing XSAs for situations
> > > > > like this, because there are too many ways where it definitely will go
> > > > > wrong, and we'd end up issuing XSAs saying "remember to clean your
> > > > > working tree before building a livepatch".  This is of course absurd.
> > > > 
> > > > Your argument is that because we do not issue XSAs for *user mistakes*,
> > > > that therefore we should not issue XSAs for *bugs in the tool*.
> > > > 
> > > > That is of course absurd.  We do not issue XSAs for user mistakes in
> > > > building the hypervisor either (for instance, switching gcc versions
> > > > without cleaning the hypervisor tree), and yet we still issue XSAs for
> > > > bugs in the hypervisor itself.
> > > > 
> > > > > IMO, The only viable option is to exclude livepatch-build-tools entirely
> > > > > from security scope.  It is already the case that people producing
> > > > > livepatches need to check the resulting livepatch binary for sanity, and
> > > > > test it suitably in a development environment before use in production.
> > > > 
> > > > Look, it sounds like right now you are going through all the livepatches
> > > > with a fine-tooth comb *because* the tools are (or recently have been)
> > > > unreliable.  But at some point in the future, the patch generation
> > > > mechanism will become more reliable.  After 20 XSAs over six months in
> > > > which the livepatch tool created the correct patch, you will become more
> > > > complacent.  You won't look as closely; it's human nature.
> > > > 
> > > > You seem to be simply refusing to use your imagination.  Step back.
> > > > Imagine yourself in one year.  You come to the office and find an e-mail
> > > > on security@ which says, "Livepatch tools open a security hole when
> > > > compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
> > > > gcc x.yy, so you take a closer look at that livepatch, only to discover
> > > > that the livepatches generated actually do contain the bug, but you
> > > > missed it because ${LATEST-[0,1]} were perfectly fine (since they used
> > > > newer versions of gcc), the difference was subtle, and it passed all the
> > > > functional tests.
> > > > 
> > > > Now all of the customers that have applied those patches are vulnerable.
> > > > 
> > > > Do you:
> > > > 
> > > > 1. Tell the reporter to post it publicly to xen-devel immediately, since
> > > > livepatch tools are not security supported -- thus "zero-day"-ing all
> > > > your customers (as well as anyone else who happens to have used x.yy to
> > > > build a hypervisor)?
> > > > 
> > > > 2. Secretly take advantage of Citrix' privileged position on the
> > > > security list, and try to get an update out to your customers before it
> > > > gets announced (but allowing everyone *else* using gcc x.yy to
> > > > experience a zero-day)?
> > > > 
> > > > 3. Issue an XSA so that everyone has the opportunity to fix things up
> > > > before making a public announcement, and so that anyone not on the
> > > > embargo list gets an alert, so they know to either update their own
> > > > livepatches, or look for updates from their software provider?
> > > > 
> > > > I think #3 is the only possible choice.
> > > > 
> > > >    -George
> > > > 
> > > 
> > > The issue here is that any bug in livepatch-build-tools which still
> > > results in output being generated would be a security issue, because
> > > someone might have used it to patch a security issue.
> > > livepatch-build-tools is certainly not stable enough yet (ever?) to be
> > > treated in this fashion.
> > 
> > You didn't answer my question.  If the situation described happens, what
> > position do you want Andrew to be put in?  (If I missed a potential
> > action, let me know.)
> > 
> 
> I would choose #3 as it is the obvious choice. But I still don't think it is
> a sensible idea to have security support for the build tools, at least at
> this point. The same scenario could be posed for a nasty bug that affects
> Xen 4.4 only, but it is now just out of security support. IMO something
> being not supported doesn't preclude it from having an XSA released if there
> is a particularly nasty vulnerability found.

I think this is a grey area that we should try to avoid as much as
possible. What constitutes a XSA should be clearly defined, so that
there's no room for speculation or subjective decisions.

Following from the example above (and I'm really not doubting Andrew's
objective criteria here) but what might not seem as a relevant issue
to Andrew (or the security team) in general might be severe to other
parties IMHO, and since not everyone has a stake into what
constitutes a XSA the process would become unfair to them.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 07/03/2017 03:53 PM, Ross Lagerwall wrote:
> On 06/30/2017 02:42 PM, George Dunlap wrote:
>> On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
>>> On 06/27/2017 10:17 AM, George Dunlap wrote:
>>>> On 26/06/17 18:30, Andrew Cooper wrote:
>>>>> On 26/06/17 18:00, George Dunlap wrote:
>>>>>> On 26/06/17 16:36, Ross Lagerwall wrote:
>>> ...
>>>> You seem to be simply refusing to use your imagination.  Step back.
>>>> Imagine yourself in one year.  You come to the office and find an
>>>> e-mail
>>>> on security@ which says, "Livepatch tools open a security hole when
>>>> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
>>>> gcc x.yy, so you take a closer look at that livepatch, only to discover
>>>> that the livepatches generated actually do contain the bug, but you
>>>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
>>>> newer versions of gcc), the difference was subtle, and it passed all
>>>> the
>>>> functional tests.
>>>>
>>>> Now all of the customers that have applied those patches are
>>>> vulnerable.
>>>>
>>>> Do you:
>>>>
>>>> 1. Tell the reporter to post it publicly to xen-devel immediately,
>>>> since
>>>> livepatch tools are not security supported -- thus "zero-day"-ing all
>>>> your customers (as well as anyone else who happens to have used x.yy to
>>>> build a hypervisor)?
>>>>
>>>> 2. Secretly take advantage of Citrix' privileged position on the
>>>> security list, and try to get an update out to your customers before it
>>>> gets announced (but allowing everyone *else* using gcc x.yy to
>>>> experience a zero-day)?
>>>>
>>>> 3. Issue an XSA so that everyone has the opportunity to fix things up
>>>> before making a public announcement, and so that anyone not on the
>>>> embargo list gets an alert, so they know to either update their own
>>>> livepatches, or look for updates from their software provider?
>>>>
>>>> I think #3 is the only possible choice.
>>>>
>>>>    -George
>>>>
>>>
>>> The issue here is that any bug in livepatch-build-tools which still
>>> results in output being generated would be a security issue, because
>>> someone might have used it to patch a security issue.
>>> livepatch-build-tools is certainly not stable enough yet (ever?) to be
>>> treated in this fashion.
>>
>> You didn't answer my question.  If the situation described happens, what
>> position do you want Andrew to be put in?  (If I missed a potential
>> action, let me know.)
>>
> 
> I would choose #3 as it is the obvious choice. But I still don't think
> it is a sensible idea to have security support for the build tools, at
> least at this point. The same scenario could be posed for a nasty bug
> that affects Xen 4.4 only, but it is now just out of security support.
> IMO something being not supported doesn't preclude it from having an XSA
> released if there is a particularly nasty vulnerability found.

Well basically I think we agree, but we're using different terms.  You
want to say, "This isn't security supported, but if important bug is
actually found then we'll issue an XSA".  I want to say, "This is
security supported, because if an important bug is actually found we'll
issue an XSA."

So it seems to me there are likely two things that make you resistant to
calling it "security supported":

1. The fear that we'll be issuing XSAs over trivial things that don't matter

2. The fear that people will not do due diligence when creating patches
with the tools.

I think #1 is just a misconception.  *Every* bug reported to us about
any part of the code we go through the process of trying to determine
its impact and whether we need to issue an XSA or not.  All of the
examples put forward of things we don't want to issue an XSA for are
things that I'm sure we would not issue an XSA for.

For #2, that is a reasonable fear, but we can deal with that in a
different way than calling the tools "unsupported".  We can, for
instance, mention that in the documents.  We can add a warning message
that the build tools output saying that the result should be manually
inspected for correctness.

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 08/03/2017 06:20 PM, George Dunlap wrote:
> On 07/03/2017 03:53 PM, Ross Lagerwall wrote:
>> On 06/30/2017 02:42 PM, George Dunlap wrote:
>>> On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
>>>> On 06/27/2017 10:17 AM, George Dunlap wrote:
>>>>> On 26/06/17 18:30, Andrew Cooper wrote:
>>>>>> On 26/06/17 18:00, George Dunlap wrote:
>>>>>>> On 26/06/17 16:36, Ross Lagerwall wrote:
>>>> ...
>>>>> You seem to be simply refusing to use your imagination.  Step back.
>>>>> Imagine yourself in one year.  You come to the office and find an
>>>>> e-mail
>>>>> on security@ which says, "Livepatch tools open a security hole when
>>>>> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
>>>>> gcc x.yy, so you take a closer look at that livepatch, only to discover
>>>>> that the livepatches generated actually do contain the bug, but you
>>>>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
>>>>> newer versions of gcc), the difference was subtle, and it passed all
>>>>> the
>>>>> functional tests.
>>>>>
>>>>> Now all of the customers that have applied those patches are
>>>>> vulnerable.
>>>>>
>>>>> Do you:
>>>>>
>>>>> 1. Tell the reporter to post it publicly to xen-devel immediately,
>>>>> since
>>>>> livepatch tools are not security supported -- thus "zero-day"-ing all
>>>>> your customers (as well as anyone else who happens to have used x.yy to
>>>>> build a hypervisor)?
>>>>>
>>>>> 2. Secretly take advantage of Citrix' privileged position on the
>>>>> security list, and try to get an update out to your customers before it
>>>>> gets announced (but allowing everyone *else* using gcc x.yy to
>>>>> experience a zero-day)?
>>>>>
>>>>> 3. Issue an XSA so that everyone has the opportunity to fix things up
>>>>> before making a public announcement, and so that anyone not on the
>>>>> embargo list gets an alert, so they know to either update their own
>>>>> livepatches, or look for updates from their software provider?
>>>>>
>>>>> I think #3 is the only possible choice.
>>>>>
>>>>>    -George
>>>>>
>>>>
>>>> The issue here is that any bug in livepatch-build-tools which still
>>>> results in output being generated would be a security issue, because
>>>> someone might have used it to patch a security issue.
>>>> livepatch-build-tools is certainly not stable enough yet (ever?) to be
>>>> treated in this fashion.
>>>
>>> You didn't answer my question.  If the situation described happens, what
>>> position do you want Andrew to be put in?  (If I missed a potential
>>> action, let me know.)
>>>
>>
>> I would choose #3 as it is the obvious choice. But I still don't think
>> it is a sensible idea to have security support for the build tools, at
>> least at this point. The same scenario could be posed for a nasty bug
>> that affects Xen 4.4 only, but it is now just out of security support.
>> IMO something being not supported doesn't preclude it from having an XSA
>> released if there is a particularly nasty vulnerability found.
> 
> Well basically I think we agree, but we're using different terms.  You
> want to say, "This isn't security supported, but if important bug is
> actually found then we'll issue an XSA".  I want to say, "This is
> security supported, because if an important bug is actually found we'll
> issue an XSA."
> 
> So it seems to me there are likely two things that make you resistant to
> calling it "security supported":
> 
> 1. The fear that we'll be issuing XSAs over trivial things that don't matter
> 
> 2. The fear that people will not do due diligence when creating patches
> with the tools.
> 
> I think #1 is just a misconception.  *Every* bug reported to us about
> any part of the code we go through the process of trying to determine
> its impact and whether we need to issue an XSA or not.  All of the
> examples put forward of things we don't want to issue an XSA for are
> things that I'm sure we would not issue an XSA for.
> 
> For #2, that is a reasonable fear, but we can deal with that in a
> different way than calling the tools "unsupported".  We can, for
> instance, mention that in the documents.  We can add a warning message
> that the build tools output saying that the result should be manually
> inspected for correctness.

We need to get a resolution on this.  Anyone else (particarly
committers) want to give their opinion?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Konrad Rzeszutek Wilk]

On Thu, Aug 03, 2017 at 06:21:30PM +0100, George Dunlap wrote:
> On 08/03/2017 06:20 PM, George Dunlap wrote:
> > On 07/03/2017 03:53 PM, Ross Lagerwall wrote:
> >> On 06/30/2017 02:42 PM, George Dunlap wrote:
> >>> On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
> >>>> On 06/27/2017 10:17 AM, George Dunlap wrote:
> >>>>> On 26/06/17 18:30, Andrew Cooper wrote:
> >>>>>> On 26/06/17 18:00, George Dunlap wrote:
> >>>>>>> On 26/06/17 16:36, Ross Lagerwall wrote:
> >>>> ...
> >>>>> You seem to be simply refusing to use your imagination.  Step back.
> >>>>> Imagine yourself in one year.  You come to the office and find an
> >>>>> e-mail
> >>>>> on security@ which says, "Livepatch tools open a security hole when
> >>>>> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
> >>>>> gcc x.yy, so you take a closer look at that livepatch, only to discover
> >>>>> that the livepatches generated actually do contain the bug, but you
> >>>>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
> >>>>> newer versions of gcc), the difference was subtle, and it passed all
> >>>>> the
> >>>>> functional tests.
> >>>>>
> >>>>> Now all of the customers that have applied those patches are
> >>>>> vulnerable.
> >>>>>
> >>>>> Do you:
> >>>>>
> >>>>> 1. Tell the reporter to post it publicly to xen-devel immediately,
> >>>>> since
> >>>>> livepatch tools are not security supported -- thus "zero-day"-ing all
> >>>>> your customers (as well as anyone else who happens to have used x.yy to
> >>>>> build a hypervisor)?
> >>>>>
> >>>>> 2. Secretly take advantage of Citrix' privileged position on the
> >>>>> security list, and try to get an update out to your customers before it
> >>>>> gets announced (but allowing everyone *else* using gcc x.yy to
> >>>>> experience a zero-day)?
> >>>>>
> >>>>> 3. Issue an XSA so that everyone has the opportunity to fix things up
> >>>>> before making a public announcement, and so that anyone not on the
> >>>>> embargo list gets an alert, so they know to either update their own
> >>>>> livepatches, or look for updates from their software provider?
> >>>>>
> >>>>> I think #3 is the only possible choice.
> >>>>>
> >>>>>    -George
> >>>>>
> >>>>
> >>>> The issue here is that any bug in livepatch-build-tools which still
> >>>> results in output being generated would be a security issue, because
> >>>> someone might have used it to patch a security issue.
> >>>> livepatch-build-tools is certainly not stable enough yet (ever?) to be
> >>>> treated in this fashion.
> >>>
> >>> You didn't answer my question.  If the situation described happens, what
> >>> position do you want Andrew to be put in?  (If I missed a potential
> >>> action, let me know.)
> >>>
> >>
> >> I would choose #3 as it is the obvious choice. But I still don't think
> >> it is a sensible idea to have security support for the build tools, at
> >> least at this point. The same scenario could be posed for a nasty bug
> >> that affects Xen 4.4 only, but it is now just out of security support.
> >> IMO something being not supported doesn't preclude it from having an XSA
> >> released if there is a particularly nasty vulnerability found.
> > 
> > Well basically I think we agree, but we're using different terms.  You
> > want to say, "This isn't security supported, but if important bug is
> > actually found then we'll issue an XSA".  I want to say, "This is
> > security supported, because if an important bug is actually found we'll
> > issue an XSA."
> > 
> > So it seems to me there are likely two things that make you resistant to
> > calling it "security supported":
> > 
> > 1. The fear that we'll be issuing XSAs over trivial things that don't matter
> > 
> > 2. The fear that people will not do due diligence when creating patches
> > with the tools.
> > 
> > I think #1 is just a misconception.  *Every* bug reported to us about
> > any part of the code we go through the process of trying to determine
> > its impact and whether we need to issue an XSA or not.  All of the
> > examples put forward of things we don't want to issue an XSA for are
> > things that I'm sure we would not issue an XSA for.
> > 
> > For #2, that is a reasonable fear, but we can deal with that in a
> > different way than calling the tools "unsupported".  We can, for
> > instance, mention that in the documents.  We can add a warning message
> > that the build tools output saying that the result should be manually
> > inspected for correctness.
> 
> We need to get a resolution on this.  Anyone else (particarly
> committers) want to give their opinion?

Changing title as this is all about now livepatch-build-tools.

The livepatch-build-tools get a lot of usage around XSA times.
And that is when the corner cases are being found. The three of them:
0c10457 Remove section alignment requirement
b30d34c Ignore .discard sections
6327ab9 create-diff-object: Update fixup offsets in .rela.ex_table

where thanks to generating XSAs. Now the folks who use these tools
are also the ones that do pre-disclosures. And the folks who
work on these tools also are the ones who have to get the livepatches out.

It is a stressful time and in the past the issues were off:
'oh, livepatch-build-tools won't generate the livepatch' 

which I don't even know how to classify - is it an XSA that it could not
create an livepatch?

And if the livepatch-build-tools does generate something mighty wrong
then the folks on the XSA pre-disclosure list should be let know
(and that has been happening).

But I am not really a fan of 'Oh, and one more XSA'


The second argument is that livepatch-build-tools is like the GCC compiler.
In fact it takes the binary blob of what the compiler has produced
and checks it against the other one. If the compiler adds extra instructions
or changes the instructions slightly we will classify that as
code needing to be patched (and yes that has come up).

This is very similar to what XSA-155 was - the GCC compiler optimizations
added a nice jump table that was accessed twice. And the offset was
retrieved from the shared ring.

But we didn't do an XSA-155 for the GCC compiler. That is we didn't
file a ticket with GCC saying 'Hey, your compiler can create an race
on shared memory. Could you make your compiler be smarter in these cases'
We instead wrote code with this optimization in mind with more
barriers.

I think livepatch-build-tools is in the same category as GCC or linkers.
> 
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 08/06/2017 01:07 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Aug 03, 2017 at 06:21:30PM +0100, George Dunlap wrote:
>> On 08/03/2017 06:20 PM, George Dunlap wrote:
>>> On 07/03/2017 03:53 PM, Ross Lagerwall wrote:
>>>> On 06/30/2017 02:42 PM, George Dunlap wrote:
>>>>> On 06/28/2017 05:18 PM, Ross Lagerwall wrote:
>>>>>> On 06/27/2017 10:17 AM, George Dunlap wrote:
>>>>>>> On 26/06/17 18:30, Andrew Cooper wrote:
>>>>>>>> On 26/06/17 18:00, George Dunlap wrote:
>>>>>>>>> On 26/06/17 16:36, Ross Lagerwall wrote:
>>>>>> ...
>>>>>>> You seem to be simply refusing to use your imagination.  Step back.
>>>>>>> Imagine yourself in one year.  You come to the office and find an
>>>>>>> e-mail
>>>>>>> on security@ which says, "Livepatch tools open a security hole when
>>>>>>> compiling with gcc x.yy".  You realize that XenVerson ${LATEST-2} uses
>>>>>>> gcc x.yy, so you take a closer look at that livepatch, only to discover
>>>>>>> that the livepatches generated actually do contain the bug, but you
>>>>>>> missed it because ${LATEST-[0,1]} were perfectly fine (since they used
>>>>>>> newer versions of gcc), the difference was subtle, and it passed all
>>>>>>> the
>>>>>>> functional tests.
>>>>>>>
>>>>>>> Now all of the customers that have applied those patches are
>>>>>>> vulnerable.
>>>>>>>
>>>>>>> Do you:
>>>>>>>
>>>>>>> 1. Tell the reporter to post it publicly to xen-devel immediately,
>>>>>>> since
>>>>>>> livepatch tools are not security supported -- thus "zero-day"-ing all
>>>>>>> your customers (as well as anyone else who happens to have used x.yy to
>>>>>>> build a hypervisor)?
>>>>>>>
>>>>>>> 2. Secretly take advantage of Citrix' privileged position on the
>>>>>>> security list, and try to get an update out to your customers before it
>>>>>>> gets announced (but allowing everyone *else* using gcc x.yy to
>>>>>>> experience a zero-day)?
>>>>>>>
>>>>>>> 3. Issue an XSA so that everyone has the opportunity to fix things up
>>>>>>> before making a public announcement, and so that anyone not on the
>>>>>>> embargo list gets an alert, so they know to either update their own
>>>>>>> livepatches, or look for updates from their software provider?
>>>>>>>
>>>>>>> I think #3 is the only possible choice.
>>>>>>>
>>>>>>>    -George
>>>>>>>
>>>>>>
>>>>>> The issue here is that any bug in livepatch-build-tools which still
>>>>>> results in output being generated would be a security issue, because
>>>>>> someone might have used it to patch a security issue.
>>>>>> livepatch-build-tools is certainly not stable enough yet (ever?) to be
>>>>>> treated in this fashion.
>>>>>
>>>>> You didn't answer my question.  If the situation described happens, what
>>>>> position do you want Andrew to be put in?  (If I missed a potential
>>>>> action, let me know.)
>>>>>
>>>>
>>>> I would choose #3 as it is the obvious choice. But I still don't think
>>>> it is a sensible idea to have security support for the build tools, at
>>>> least at this point. The same scenario could be posed for a nasty bug
>>>> that affects Xen 4.4 only, but it is now just out of security support.
>>>> IMO something being not supported doesn't preclude it from having an XSA
>>>> released if there is a particularly nasty vulnerability found.
>>>
>>> Well basically I think we agree, but we're using different terms.  You
>>> want to say, "This isn't security supported, but if important bug is
>>> actually found then we'll issue an XSA".  I want to say, "This is
>>> security supported, because if an important bug is actually found we'll
>>> issue an XSA."
>>>
>>> So it seems to me there are likely two things that make you resistant to
>>> calling it "security supported":
>>>
>>> 1. The fear that we'll be issuing XSAs over trivial things that don't matter
>>>
>>> 2. The fear that people will not do due diligence when creating patches
>>> with the tools.
>>>
>>> I think #1 is just a misconception.  *Every* bug reported to us about
>>> any part of the code we go through the process of trying to determine
>>> its impact and whether we need to issue an XSA or not.  All of the
>>> examples put forward of things we don't want to issue an XSA for are
>>> things that I'm sure we would not issue an XSA for.
>>>
>>> For #2, that is a reasonable fear, but we can deal with that in a
>>> different way than calling the tools "unsupported".  We can, for
>>> instance, mention that in the documents.  We can add a warning message
>>> that the build tools output saying that the result should be manually
>>> inspected for correctness.
>>
>> We need to get a resolution on this.  Anyone else (particarly
>> committers) want to give their opinion?
> 
> Changing title as this is all about now livepatch-build-tools.
> 
> The livepatch-build-tools get a lot of usage around XSA times.
> And that is when the corner cases are being found. The three of them:
> 0c10457 Remove section alignment requirement
> b30d34c Ignore .discard sections
> 6327ab9 create-diff-object: Update fixup offsets in .rela.ex_table
> 
> where thanks to generating XSAs. Now the folks who use these tools
> are also the ones that do pre-disclosures. And the folks who
> work on these tools also are the ones who have to get the livepatches out.
> 
> It is a stressful time and in the past the issues were off:
> 'oh, livepatch-build-tools won't generate the livepatch' 
> 
> which I don't even know how to classify - is it an XSA that it could not
> create an livepatch?
> 
> And if the livepatch-build-tools does generate something mighty wrong
> then the folks on the XSA pre-disclosure list should be let know
> (and that has been happening).
> 
> But I am not really a fan of 'Oh, and one more XSA'

Thanks for weighing in, Konrad.

So it seems that people are still not quite clear about what I'm proposing.

In general, a bug is a security issue if a human told the computer to do
safe thing X, and the computer appeared to do safe thing X, but in fact
did unsafe thing Y.

Consider the question: "Is it an XSA that domain creation will fail for
a kernel with feature A enabled?"  No, that's not a security issue: The
human told the computer to do safe thing X ("boot with kernel A"), and
it did safe thing Z ("not boot").

Consider this question: "Is it an XSA that if you detach a block device
from one domain, and then attach it to another block device, that the
second user can read potentially sensitive data from the first domain?"
No, that's not a security issue: The human told the computer to do
unsafe thing Y ("expose block device from domain B to domain C"), and
the computer did unsafe thing Y.

So.  Suppose a developer / package maintainer / sysadmin tries to build
a livepatch, and the livepatch throws an error.  Is this a security
issue?  No -- the human told it to do safe thing X ("build me a
livepatch"), and it did safe thing Z ("print an error").

Suppose someone builds a livepatch that, when loaded, immediately
crashes the hypervisor in all cases.  Is this a security issue?  No --
the human told it to do safe thing X ("patch this vulnerability") and it
did safe thing Z ("crash the hypervisor immediately").  (This is safe
because we assume you do a reasonable amount of testing before deploying
a livepatch.)

According to my understanding, the above three fixes that happened as
part of devloping XSAs were like one of the above two examples: The tool
either just didn't make a livepatch at all, or it made one that clearly
didn't work.  As such, they would not be considered XSAs.

Suppose someone builds the livepatch with two different versions of the
compiler; or against stale versions of the binaries; and the livepatch
tool doesn't notice, and generates a livepatch which opens up a security
hole.  Is this a security issue?  No -- the human told it to do unsafe
thing Y, and it did unsafe thing Y.

Suppose someone builds a livepatch that with buggy fix-up code that
inadvertently gives PV guests access to hypervisor memory.  Is this a
security issue?  No -- the human told it to do unsafe thing Y ("give PV
guest access to hypervisor memory") and it did unsafe thing Y.

Suppose someone builds a livepatch with the correct compiler, with a
correct patch (that would fix the bug if rebooted into a new
hypervisor), with correct fix-up code.  Suppose that the bug passes all
reasonable testing; but that, *due to a bug in the tools*, the patch
also gives PV guests access to hypervisor memory.  Is this a security
issue?  Yes -- the human told it to do safe thing X ("build a livepatch
based on correct inputs to fix this bug") and it did unsafe thing Y
("build a livepatch that opens up a new security hole").

We could even place more restrictions on the scope if we wanted to.  We
could say that we only support the livepatch tools generating patches
for XSAs.

> This is very similar to what XSA-155 was - the GCC compiler optimizations
> added a nice jump table that was accessed twice. And the offset was
> retrieved from the shared ring.
>
> But we didn't do an XSA-155 for the GCC compiler. That is we didn't
> file a ticket with GCC saying 'Hey, your compiler can create an race
> on shared memory. Could you make your compiler be smarter in these cases'
> We instead wrote code with this optimization in mind with more
> barriers.

Right -- so the gcc compiler guys are using a specification that allows
that behavior.  So from their perspective, we told the compiler to do
unsafe thing Y (or at least, said that we were OK with it doing unsafe
thing Y), and it did unsafe thing Y -- a security issue for Xen, but not
for gcc.  If gcc had *violated* the spec when causing the security
issue, then we certainly would have called that a security issue in gcc.

> I think livepatch-build-tools is in the same category as GCC or linkers.

Indeed; gcc is "security-supported" in the way I am proposing
livepatching be security supported.

I don't think this is a hill worth dying on; if everyone agrees we
should call them "security un-supported", we can go with that.  I'm only
still talking because people seem to be confused about what I'm proposing.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Jan Beulich]

>>> George Dunlap <george.dunlap@citrix.com> 08/07/17 12:27 PM >>>
>So it seems that people are still not quite clear about what I'm proposing.

And indeed your examples helped me understand better what you mean
(or at least I hope they did).

>Suppose someone builds a livepatch with the correct compiler, with a
>correct patch (that would fix the bug if rebooted into a new
>hypervisor), with correct fix-up code.  Suppose that the bug passes all
>reasonable testing; but that, *due to a bug in the tools*, the patch
>also gives PV guests access to hypervisor memory.  Is this a security
>issue?  Yes -- the human told it to do safe thing X ("build a livepatch
>based on correct inputs to fix this bug") and it did unsafe thing Y
>("build a livepatch that opens up a new security hole").

There's one more factor here: The livepatch tools may behave properly
with one version of the compiler, and improperly with another. Or they
may behave properly with the Oracle patched hypervisor sources, but
improperly with the Citrix ones (assuming every distro carries a different
set of patches on top of an upstream stable release or branch).

>We could even place more restrictions on the scope if we wanted to.  We
>could say that we only support the livepatch tools generating patches
>for XSAs.

For me, much depends on how tight such restrictions would be. I.e.
with the examples given above, how would we determine a canonical
livepatch-tools / hypervisor pair (or set of pairs)? After all tools
mis-behavior may be a result of some custom patch in someone's
derived tree.

>> This is very similar to what XSA-155 was - the GCC compiler optimizations
>> added a nice jump table that was accessed twice. And the offset was
>> retrieved from the shared ring.
>>
>> But we didn't do an XSA-155 for the GCC compiler. That is we didn't
>> file a ticket with GCC saying 'Hey, your compiler can create an race
>> on shared memory. Could you make your compiler be smarter in these cases'
>> We instead wrote code with this optimization in mind with more
>> barriers.
>
>Right -- so the gcc compiler guys are using a specification that allows
>that behavior.  So from their perspective, we told the compiler to do
>unsafe thing Y (or at least, said that we were OK with it doing unsafe
>thing Y), and it did unsafe thing Y -- a security issue for Xen, but not
>for gcc.  If gcc had *violated* the spec when causing the security
>issue, then we certainly would have called that a security issue in gcc.

But would we have issued an XSA? Wouldn't that rather be a CVE
against gcc then?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 08/07/2017 04:59 PM, Jan Beulich wrote:
>>>> George Dunlap <george.dunlap@citrix.com> 08/07/17 12:27 PM >>>
>> So it seems that people are still not quite clear about what I'm proposing.
> 
> And indeed your examples helped me understand better what you mean
> (or at least I hope they did).
> 
>> Suppose someone builds a livepatch with the correct compiler, with a
>> correct patch (that would fix the bug if rebooted into a new
>> hypervisor), with correct fix-up code.  Suppose that the bug passes all
>> reasonable testing; but that, *due to a bug in the tools*, the patch
>> also gives PV guests access to hypervisor memory.  Is this a security
>> issue?  Yes -- the human told it to do safe thing X ("build a livepatch
>> based on correct inputs to fix this bug") and it did unsafe thing Y
>> ("build a livepatch that opens up a new security hole").
> 
> There's one more factor here: The livepatch tools may behave properly
> with one version of the compiler, and improperly with another. 

I don't really understand the reasoning here.  Is this your argument:
"One can imagine a security-critical livepatch bug that only affects
say, gcc 6.x and not gcc 5.x or 7.x.  Therefore, we should never issue
XSAs for any security-critical livepatch bugs."

If we found that livepatching tools make an incorrect patch only when
using gcc 5.x, and we have reason to believe that some people may be
using gcc 5.x, then I think we should issue an XSA and say that it only
affects people compiling xen with gcc 5.x.

It probably would make sense to specify some range of compiler versions
for which we will issue XSAs for the livepatch tools.  A good baseline
would be what versions of gcc Xen uses, and then we can restrict it
further if we need to (for instance, if some versions of gcc are missing
requisite features, or if they are just known to be buggy).

And remember, this is not "We have tested all compiler versions and
promise you there are no bugs."  It's, "If someone finds a bug for this
set of compilers, we will tell you about it so you can do something
about it."

>> We could even place more restrictions on the scope if we wanted to.  We
>> could say that we only support the livepatch tools generating patches
>> for XSAs.
> 
> For me, much depends on how tight such restrictions would be. I.e.
> with the examples given above, how would we determine a canonical
> livepatch-tools / hypervisor pair (or set of pairs)? After all tools
> mis-behavior may be a result of some custom patch in someone's
> derived tree.

Well, suppose that we issued an XSA with a patch, and suppose it was
later discovered that the patch opened up a different security hole when
applied on the upstream tree.  Would we issue another XSA and/or an
update to the existing XSA?  I think obviously yes we would.

Suppose instead we issued an XSA with a patch, and that it was later
discovered that the patch opened up a different security hole when
applied on top of XenServer's patchqueue, but not on the baseline
XenProject.  Would we issue another XSA and/or an update to an existing XSA?

The obvious *default* answer to that is "No; it's not practical for us
to deal with software that is not inside the XenProject's control."  One
could imagine circumstances in which we issue statements or an XSA
anyway, but that would the exception and not the rule.

I think the same kind of thing would apply to the livepatch tools: *by
default*, we only issue XSAs for the livepatch tools if they create
security issues when generating blobs based on security patches issued
by the XenProject, and on top of XenProject-released software.  As
always, if there's some unforeseen circumstance then someone could argue
for an exception.

>>> This is very similar to what XSA-155 was - the GCC compiler optimizations
>>> added a nice jump table that was accessed twice. And the offset was
>>> retrieved from the shared ring.
>>>
>>> But we didn't do an XSA-155 for the GCC compiler. That is we didn't
>>> file a ticket with GCC saying 'Hey, your compiler can create an race
>>> on shared memory. Could you make your compiler be smarter in these cases'
>>> We instead wrote code with this optimization in mind with more
>>> barriers.
>>
>> Right -- so the gcc compiler guys are using a specification that allows
>> that behavior.  So from their perspective, we told the compiler to do
>> unsafe thing Y (or at least, said that we were OK with it doing unsafe
>> thing Y), and it did unsafe thing Y -- a security issue for Xen, but not
>> for gcc.  If gcc had *violated* the spec when causing the security
>> issue, then we certainly would have called that a security issue in gcc.
> 
> But would we have issued an XSA? Wouldn't that rather be a CVE
> against gcc then?

This is changing the question slightly, from "Should X have security
support", to "If X is to be security supported, what organization and
process should be used to support them?"

Obviously in the case of gcc, we would primarily handle the security
issue the way the gcc project handles security issues (which may be
nothing at all for all I know).  (Although depending on the bug and the
circumstances, we might still issue an advisory to raise awareness for
downstreams who might have compiled Xen with a particular version of gcc.)

If livepatch-tools were an external project run by somebody else, with
their own security process, then we would report the issue to them and
let them handle it.  Since livepatch-tools is developed by Xen
developers and for Xen downstreams and users, if it is to be security
supported, then it seems to me that the obvious thing to do is to
support it within the XenProject security response process.

I mean, if someone *wants* to set up an independent organization with an
independent security team and security process to handle the livepatch
project, then I guess that would be OK with me -- I don't care so much
*who* does the security support, as long as it gets done.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Jan Beulich]

>>> On 08.08.17 at 13:16, <george.dunlap@citrix.com> wrote:
> On 08/07/2017 04:59 PM, Jan Beulich wrote:
>>>>> George Dunlap <george.dunlap@citrix.com> 08/07/17 12:27 PM >>>
>>> So it seems that people are still not quite clear about what I'm proposing.
>> 
>> And indeed your examples helped me understand better what you mean
>> (or at least I hope they did).
>> 
>>> Suppose someone builds a livepatch with the correct compiler, with a
>>> correct patch (that would fix the bug if rebooted into a new
>>> hypervisor), with correct fix-up code.  Suppose that the bug passes all
>>> reasonable testing; but that, *due to a bug in the tools*, the patch
>>> also gives PV guests access to hypervisor memory.  Is this a security
>>> issue?  Yes -- the human told it to do safe thing X ("build a livepatch
>>> based on correct inputs to fix this bug") and it did unsafe thing Y
>>> ("build a livepatch that opens up a new security hole").
>> 
>> There's one more factor here: The livepatch tools may behave properly
>> with one version of the compiler, and improperly with another. 
> 
> I don't really understand the reasoning here.  Is this your argument:
> "One can imagine a security-critical livepatch bug that only affects
> say, gcc 6.x and not gcc 5.x or 7.x.  Therefore, we should never issue
> XSAs for any security-critical livepatch bugs."
> 
> If we found that livepatching tools make an incorrect patch only when
> using gcc 5.x, and we have reason to believe that some people may be
> using gcc 5.x, then I think we should issue an XSA and say that it only
> affects people compiling xen with gcc 5.x.
> 
> It probably would make sense to specify some range of compiler versions
> for which we will issue XSAs for the livepatch tools.  A good baseline
> would be what versions of gcc Xen uses, and then we can restrict it
> further if we need to (for instance, if some versions of gcc are missing
> requisite features, or if they are just known to be buggy).
> 
> And remember, this is not "We have tested all compiler versions and
> promise you there are no bugs."  It's, "If someone finds a bug for this
> set of compilers, we will tell you about it so you can do something
> about it."

I can see and understand all of what you say; my argument,
however was more towards the matrix of what needs supporting
possibly becoming unreasonably large (no matter whether we
specify a range of compilers, as once again distros tend to not
ship plain unpatched upstream compiler versions).

>>> We could even place more restrictions on the scope if we wanted to.  We
>>> could say that we only support the livepatch tools generating patches
>>> for XSAs.
>> 
>> For me, much depends on how tight such restrictions would be. I.e.
>> with the examples given above, how would we determine a canonical
>> livepatch-tools / hypervisor pair (or set of pairs)? After all tools
>> mis-behavior may be a result of some custom patch in someone's
>> derived tree.
> 
> Well, suppose that we issued an XSA with a patch, and suppose it was
> later discovered that the patch opened up a different security hole when
> applied on the upstream tree.  Would we issue another XSA and/or an
> update to the existing XSA?  I think obviously yes we would.

Yes (this has happened in the past already).

> Suppose instead we issued an XSA with a patch, and that it was later
> discovered that the patch opened up a different security hole when
> applied on top of XenServer's patchqueue, but not on the baseline
> XenProject.  Would we issue another XSA and/or an update to an existing XSA?
> 
> The obvious *default* answer to that is "No; it's not practical for us
> to deal with software that is not inside the XenProject's control."  One
> could imagine circumstances in which we issue statements or an XSA
> anyway, but that would the exception and not the rule.
> 
> I think the same kind of thing would apply to the livepatch tools: *by
> default*, we only issue XSAs for the livepatch tools if they create
> security issues when generating blobs based on security patches issued
> by the XenProject, and on top of XenProject-released software.  As
> always, if there's some unforeseen circumstance then someone could argue
> for an exception.

Not sure here - if analysis showed that the same issue could happen
elsewhere, and others were just lucky so far, I think we'd have to
alter the default (and I'm hesitant to call this an exception). Plus
analysis may, the more different components are involved
(specifically the compiler, which perhaps none of us has deep enough
knowledge about), become more and more difficult.

Bottom line - while technically I agree it would be good for the tools
to be security supported, from a practical perspective I see too
much complexity for this to be reasonably manageable.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On Wed, Aug 9, 2017 at 8:36 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 08.08.17 at 13:16, <george.dunlap@citrix.com> wrote:
>> On 08/07/2017 04:59 PM, Jan Beulich wrote:
>>>>>> George Dunlap <george.dunlap@citrix.com> 08/07/17 12:27 PM >>>
>>>> So it seems that people are still not quite clear about what I'm proposing.
>>>
>>> And indeed your examples helped me understand better what you mean
>>> (or at least I hope they did).
>>>
>>>> Suppose someone builds a livepatch with the correct compiler, with a
>>>> correct patch (that would fix the bug if rebooted into a new
>>>> hypervisor), with correct fix-up code.  Suppose that the bug passes all
>>>> reasonable testing; but that, *due to a bug in the tools*, the patch
>>>> also gives PV guests access to hypervisor memory.  Is this a security
>>>> issue?  Yes -- the human told it to do safe thing X ("build a livepatch
>>>> based on correct inputs to fix this bug") and it did unsafe thing Y
>>>> ("build a livepatch that opens up a new security hole").
>>>
>>> There's one more factor here: The livepatch tools may behave properly
>>> with one version of the compiler, and improperly with another.
>>
>> I don't really understand the reasoning here.  Is this your argument:
>> "One can imagine a security-critical livepatch bug that only affects
>> say, gcc 6.x and not gcc 5.x or 7.x.  Therefore, we should never issue
>> XSAs for any security-critical livepatch bugs."
>>
>> If we found that livepatching tools make an incorrect patch only when
>> using gcc 5.x, and we have reason to believe that some people may be
>> using gcc 5.x, then I think we should issue an XSA and say that it only
>> affects people compiling xen with gcc 5.x.
>>
>> It probably would make sense to specify some range of compiler versions
>> for which we will issue XSAs for the livepatch tools.  A good baseline
>> would be what versions of gcc Xen uses, and then we can restrict it
>> further if we need to (for instance, if some versions of gcc are missing
>> requisite features, or if they are just known to be buggy).
>>
>> And remember, this is not "We have tested all compiler versions and
>> promise you there are no bugs."  It's, "If someone finds a bug for this
>> set of compilers, we will tell you about it so you can do something
>> about it."
>
> I can see and understand all of what you say; my argument,
> however was more towards the matrix of what needs supporting
> possibly becoming unreasonably large (no matter whether we
> specify a range of compilers, as once again distros tend to not
> ship plain unpatched upstream compiler versions).

What do you mean, "The matrix of what needs supporting [may possibly
become] increasingly large"?   What is the problem with having a large
(implicit) "supported" matrix?  How is supporting a "large matrix" for
livepatch tools different than the current "large matrix" we support
for just building Xen at all?


>> Suppose instead we issued an XSA with a patch, and that it was later
>> discovered that the patch opened up a different security hole when
>> applied on top of XenServer's patchqueue, but not on the baseline
>> XenProject.  Would we issue another XSA and/or an update to an existing XSA?
>>
>> The obvious *default* answer to that is "No; it's not practical for us
>> to deal with software that is not inside the XenProject's control."  One
>> could imagine circumstances in which we issue statements or an XSA
>> anyway, but that would the exception and not the rule.
>>
>> I think the same kind of thing would apply to the livepatch tools: *by
>> default*, we only issue XSAs for the livepatch tools if they create
>> security issues when generating blobs based on security patches issued
>> by the XenProject, and on top of XenProject-released software.  As
>> always, if there's some unforeseen circumstance then someone could argue
>> for an exception.
>
> Not sure here - if analysis showed that the same issue could happen
> elsewhere, and others were just lucky so far, I think we'd have to
> alter the default (and I'm hesitant to call this an exception). Plus
> analysis may, the more different components are involved
> (specifically the compiler, which perhaps none of us has deep enough
> knowledge about), become more and more difficult.
>
> Bottom line - while technically I agree it would be good for the tools
> to be security supported, from a practical perspective I see too
> much complexity for this to be reasonably manageable.

But I still don't understand why you think so.  Every single objection
or question about what would or would  not be supported that has been
raised so far has analogs in what we already support.  It is no more
complex to support livepatch-tools than to support HVM USB
passthrough, or credit2.

I have elsewhere described a hypothetical scenario where I think we
should issue an XSA for livepatch-tools.  Are you really seriously
suggesting that in that scenario we should simply publish the
vulnerability onto xen-devel with no predisclosure?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Jan Beulich]

>>> On 21.08.17 at 12:59, <george.dunlap@citrix.com> wrote:
> On Wed, Aug 9, 2017 at 8:36 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 08.08.17 at 13:16, <george.dunlap@citrix.com> wrote:
>>> On 08/07/2017 04:59 PM, Jan Beulich wrote:
>>>>>>> George Dunlap <george.dunlap@citrix.com> 08/07/17 12:27 PM >>>
>>>>> So it seems that people are still not quite clear about what I'm proposing.
>>>>
>>>> And indeed your examples helped me understand better what you mean
>>>> (or at least I hope they did).
>>>>
>>>>> Suppose someone builds a livepatch with the correct compiler, with a
>>>>> correct patch (that would fix the bug if rebooted into a new
>>>>> hypervisor), with correct fix-up code.  Suppose that the bug passes all
>>>>> reasonable testing; but that, *due to a bug in the tools*, the patch
>>>>> also gives PV guests access to hypervisor memory.  Is this a security
>>>>> issue?  Yes -- the human told it to do safe thing X ("build a livepatch
>>>>> based on correct inputs to fix this bug") and it did unsafe thing Y
>>>>> ("build a livepatch that opens up a new security hole").
>>>>
>>>> There's one more factor here: The livepatch tools may behave properly
>>>> with one version of the compiler, and improperly with another.
>>>
>>> I don't really understand the reasoning here.  Is this your argument:
>>> "One can imagine a security-critical livepatch bug that only affects
>>> say, gcc 6.x and not gcc 5.x or 7.x.  Therefore, we should never issue
>>> XSAs for any security-critical livepatch bugs."
>>>
>>> If we found that livepatching tools make an incorrect patch only when
>>> using gcc 5.x, and we have reason to believe that some people may be
>>> using gcc 5.x, then I think we should issue an XSA and say that it only
>>> affects people compiling xen with gcc 5.x.
>>>
>>> It probably would make sense to specify some range of compiler versions
>>> for which we will issue XSAs for the livepatch tools.  A good baseline
>>> would be what versions of gcc Xen uses, and then we can restrict it
>>> further if we need to (for instance, if some versions of gcc are missing
>>> requisite features, or if they are just known to be buggy).
>>>
>>> And remember, this is not "We have tested all compiler versions and
>>> promise you there are no bugs."  It's, "If someone finds a bug for this
>>> set of compilers, we will tell you about it so you can do something
>>> about it."
>>
>> I can see and understand all of what you say; my argument,
>> however was more towards the matrix of what needs supporting
>> possibly becoming unreasonably large (no matter whether we
>> specify a range of compilers, as once again distros tend to not
>> ship plain unpatched upstream compiler versions).
> 
> What do you mean, "The matrix of what needs supporting [may possibly
> become] increasingly large"?   What is the problem with having a large
> (implicit) "supported" matrix?  How is supporting a "large matrix" for
> livepatch tools different than the current "large matrix" we support
> for just building Xen at all?

The matrix of Xen only has just a single dimension. Since livepatch
tools and Xen are independent, any pair of them would need
building/testing in order to be sure things work in all supported
combinations.

>>> Suppose instead we issued an XSA with a patch, and that it was later
>>> discovered that the patch opened up a different security hole when
>>> applied on top of XenServer's patchqueue, but not on the baseline
>>> XenProject.  Would we issue another XSA and/or an update to an existing XSA?
>>>
>>> The obvious *default* answer to that is "No; it's not practical for us
>>> to deal with software that is not inside the XenProject's control."  One
>>> could imagine circumstances in which we issue statements or an XSA
>>> anyway, but that would the exception and not the rule.
>>>
>>> I think the same kind of thing would apply to the livepatch tools: *by
>>> default*, we only issue XSAs for the livepatch tools if they create
>>> security issues when generating blobs based on security patches issued
>>> by the XenProject, and on top of XenProject-released software.  As
>>> always, if there's some unforeseen circumstance then someone could argue
>>> for an exception.
>>
>> Not sure here - if analysis showed that the same issue could happen
>> elsewhere, and others were just lucky so far, I think we'd have to
>> alter the default (and I'm hesitant to call this an exception). Plus
>> analysis may, the more different components are involved
>> (specifically the compiler, which perhaps none of us has deep enough
>> knowledge about), become more and more difficult.
>>
>> Bottom line - while technically I agree it would be good for the tools
>> to be security supported, from a practical perspective I see too
>> much complexity for this to be reasonably manageable.
> 
> But I still don't understand why you think so.  Every single objection
> or question about what would or would  not be supported that has been
> raised so far has analogs in what we already support.  It is no more
> complex to support livepatch-tools than to support HVM USB
> passthrough, or credit2.

I don't think we currently have any scenario where it might be
required (rather than just being optional) to do in-depth analysis
of compiler behavior. If we ran into a compiler induced vulnerability
in Xen, I'm sure we'd forward this to the compiler folks. If, however,
there's an issue with a livepatch, and it's unclear whether the root
cause are the tool chain or the livepatch tools, both would need to
be analyzed to find the culprit. I don't think forwarding this to the
compiler folks would be appropriate until we're sure it's in their code
rather than ours.

> I have elsewhere described a hypothetical scenario where I think we
> should issue an XSA for livepatch-tools.  Are you really seriously
> suggesting that in that scenario we should simply publish the
> vulnerability onto xen-devel with no predisclosure?

Well, at least I'm not 100% convinced issuing an XSA in this case
would be appropriate.

Anyway - since it feels like we're moving in circles (which in part
may be because I can't express well enough the reasons for my
hesitation to go to the full XSA extent with the livepatch tools)
I'd like to just conclude my part here with saying that I'm not
going to stand in the way whichever decision is taken. I've
voiced my reservations, and that will have to do. I'd therefore
prefer to leave the discussion to those more familiar with those
tools (and their possible limitations and issues).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 08/21/2017 01:07 PM, Jan Beulich wrote:
>>>> And remember, this is not "We have tested all compiler versions and
>>>> promise you there are no bugs."  It's, "If someone finds a bug for this
>>>> set of compilers, we will tell you about it so you can do something
>>>> about it."
>>>
>>> I can see and understand all of what you say; my argument,
>>> however was more towards the matrix of what needs supporting
>>> possibly becoming unreasonably large (no matter whether we
>>> specify a range of compilers, as once again distros tend to not
>>> ship plain unpatched upstream compiler versions).
>>
>> What do you mean, "The matrix of what needs supporting [may possibly
>> become] increasingly large"?   What is the problem with having a large
>> (implicit) "supported" matrix?  How is supporting a "large matrix" for
>> livepatch tools different than the current "large matrix" we support
>> for just building Xen at all?
> 
> The matrix of Xen only has just a single dimension. Since livepatch
> tools and Xen are independent, any pair of them would need
> building/testing in order to be sure things work in all supported
> combinations.

So your argument seems to be:

1. We can only provide security support in situations where we can test
all possible combinations in the support matrix.

2. We cannot test the entire matrix of combinations for Xen x livepatch
tools x compilers

3. Therefore, we cannot provide security support for livepatching tools.

Put this way, I hope you can see what the flaw in the argument is: #1 is
false.  Xen has {Xen version} x {Linux version} x {Compiler} x
{Hardware}.  Hardware of course includes not only the chip itself, but
the BIOS / firmware, and the particular devices (and device firmware).
If we wanted we could add in {Python version} for people using pygrub,
and {Ocaml compiler version} for people running Ocaml, versions of
systemd -- I'm sure with effort I could find more dimensions to add to
the matrix.

We do not, and never have, *tested* the entire matrix of possible
combinations considered "security supported" to make sure they work.
Such a matrix is completely impossible to even consider, and even if we
did some sort of testing, that could not guarantee that they are bug free.

What we do for security support is:

1. Test a *representative sample* of combinations (via osstest, product
testing, user testing, &c)

2. Promise to issue XSAs if anyone *happens to discover* a combination
in the rest of the support matrix that has a security issue

That is the requirment for normal Xen, and it would be the same
requirement for livepatch-tools: That between osstest, product, and the
community, we get regular testing of *a representative sample* of {Xen,
livepatch-tools, compiler}, and (what primarily concerns me) issue an
XSA if anyone discovers a security issue somewhere in that matrix.

I'm not frustrated, but I am baffled by the fact that this "support
matrix" objection is so persistent.  Nearly everyone has brought it up,
as though "test every combination" was a necessary requirement, in spite
of the fact that 1) there is *no* piece of software for which we test
the entire matrix of possible combinations 2) I have said over and over
again (in fact, I specifically said a few replies ago -- it's there at
the top of this email) that we do not test all possible combinations.

>> I have elsewhere described a hypothetical scenario where I think we
>> should issue an XSA for livepatch-tools.  Are you really seriously
>> suggesting that in that scenario we should simply publish the
>> vulnerability onto xen-devel with no predisclosure?
> 
> Well, at least I'm not 100% convinced issuing an XSA in this case
> would be appropriate.
> 
> Anyway - since it feels like we're moving in circles (which in part
> may be because I can't express well enough the reasons for my
> hesitation to go to the full XSA extent with the livepatch tools)
> I'd like to just conclude my part here with saying that I'm not
> going to stand in the way whichever decision is taken. I've
> voiced my reservations, and that will have to do. I'd therefore
> prefer to leave the discussion to those more familiar with those
> tools (and their possible limitations and issues).

Indeed; and as I think I said before, I think we need to move forward
with getting a statement on livepatching in, and since most of the
voices involved in this conversation seem to be in favor of saying
livepatch-tools are *not* supported, I won't object. I'm only still
continuing this thread because people seem to be confused about what I
am asking people to do.

I think the likelihood of an XSA-worthy bug being found in the livepatch
tools is very low.  I'm happy to defer the argument about whether we
should issue an XSA for such a bug until such time as one becomes known.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Jan Beulich]

>>> On 21.08.17 at 17:28, <george.dunlap@citrix.com> wrote:
> So your argument seems to be:
> 
> 1. We can only provide security support in situations where we can test
> all possible combinations in the support matrix.
> 
> 2. We cannot test the entire matrix of combinations for Xen x livepatch
> tools x compilers
> 
> 3. Therefore, we cannot provide security support for livepatching tools.
> 
> Put this way, I hope you can see what the flaw in the argument is: #1 is
> false.  Xen has {Xen version} x {Linux version} x {Compiler} x
> {Hardware}.  Hardware of course includes not only the chip itself, but
> the BIOS / firmware, and the particular devices (and device firmware).
> If we wanted we could add in {Python version} for people using pygrub,
> and {Ocaml compiler version} for people running Ocaml, versions of
> systemd -- I'm sure with effort I could find more dimensions to add to
> the matrix.
> 
> We do not, and never have, *tested* the entire matrix of possible
> combinations considered "security supported" to make sure they work.
> Such a matrix is completely impossible to even consider, and even if we
> did some sort of testing, that could not guarantee that they are bug free.
> 
> What we do for security support is:
> 
> 1. Test a *representative sample* of combinations (via osstest, product
> testing, user testing, &c)
> 
> 2. Promise to issue XSAs if anyone *happens to discover* a combination
> in the rest of the support matrix that has a security issue
> 
> That is the requirment for normal Xen, and it would be the same
> requirement for livepatch-tools: That between osstest, product, and the
> community, we get regular testing of *a representative sample* of {Xen,
> livepatch-tools, compiler}, and (what primarily concerns me) issue an
> XSA if anyone discovers a security issue somewhere in that matrix.
> 
> I'm not frustrated, but I am baffled by the fact that this "support
> matrix" objection is so persistent.  Nearly everyone has brought it up,
> as though "test every combination" was a necessary requirement, in spite
> of the fact that 1) there is *no* piece of software for which we test
> the entire matrix of possible combinations 2) I have said over and over
> again (in fact, I specifically said a few replies ago -- it's there at
> the top of this email) that we do not test all possible combinations.

Well, part of it may be that the other components involved in the
test matrix you suggest are external, i.e. we're just their consumers.
If we consider just our own portions, the matrix is - as said - one
dimensional. With the livepatching tools, a dimension is being added.
Even us issuing Linux XSAs is, with the current upstream status of
the Xen pieces in there, questionable imo. This is also considering
the fact that iirc we've never issued an XSA for another Xen guest
OS, yet it is hard to believe that only Linux would ever had any
vulnerability.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On Tue, Aug 22, 2017 at 7:37 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 21.08.17 at 17:28, <george.dunlap@citrix.com> wrote:
>> So your argument seems to be:
>>
>> 1. We can only provide security support in situations where we can test
>> all possible combinations in the support matrix.
>>
>> 2. We cannot test the entire matrix of combinations for Xen x livepatch
>> tools x compilers
>>
>> 3. Therefore, we cannot provide security support for livepatching tools.
>>
>> Put this way, I hope you can see what the flaw in the argument is: #1 is
>> false.  Xen has {Xen version} x {Linux version} x {Compiler} x
>> {Hardware}.  Hardware of course includes not only the chip itself, but
>> the BIOS / firmware, and the particular devices (and device firmware).
>> If we wanted we could add in {Python version} for people using pygrub,
>> and {Ocaml compiler version} for people running Ocaml, versions of
>> systemd -- I'm sure with effort I could find more dimensions to add to
>> the matrix.
>>
>> We do not, and never have, *tested* the entire matrix of possible
>> combinations considered "security supported" to make sure they work.
>> Such a matrix is completely impossible to even consider, and even if we
>> did some sort of testing, that could not guarantee that they are bug free.
>>
>> What we do for security support is:
>>
>> 1. Test a *representative sample* of combinations (via osstest, product
>> testing, user testing, &c)
>>
>> 2. Promise to issue XSAs if anyone *happens to discover* a combination
>> in the rest of the support matrix that has a security issue
>>
>> That is the requirment for normal Xen, and it would be the same
>> requirement for livepatch-tools: That between osstest, product, and the
>> community, we get regular testing of *a representative sample* of {Xen,
>> livepatch-tools, compiler}, and (what primarily concerns me) issue an
>> XSA if anyone discovers a security issue somewhere in that matrix.
>>
>> I'm not frustrated, but I am baffled by the fact that this "support
>> matrix" objection is so persistent.  Nearly everyone has brought it up,
>> as though "test every combination" was a necessary requirement, in spite
>> of the fact that 1) there is *no* piece of software for which we test
>> the entire matrix of possible combinations 2) I have said over and over
>> again (in fact, I specifically said a few replies ago -- it's there at
>> the top of this email) that we do not test all possible combinations.
>
> Well, part of it may be that the other components involved in the
> test matrix you suggest are external, i.e. we're just their consumers.

But our response to me, you mentioned distros having patched compilers
as a reason that the matrix is untenably large.

> If we consider just our own portions, the matrix is - as said - one
> dimensional. With the livepatching tools, a dimension is being added.
> Even us issuing Linux XSAs is, with the current upstream status of
> the Xen pieces in there, questionable imo. This is also considering
> the fact that iirc we've never issued an XSA for another Xen guest
> OS, yet it is hard to believe that only Linux would ever had any
> vulnerability.

Well, no -- we have at very least {Linux} x {Xen}, for which we end up
testing *many* different configurations, but certainly not all.

I think guest OS support is actually a pretty good analog.  I can't
imagine not issuing XSAs for bugs in Linux, just as I can't imagine
not issuing XSAs for actual security issues that get found in the
livepatch tools.  If you think we shouldn't give security support for
Linux, it makes sense that you would feel the same way for
livepatch-tools (although I don't really understand why you think that
way about either).

We issue more XSAs for Linux than for other guests, in part because of
the complexity of the code inside Linux compared to other OSes; but
also in part due to the fact that that is the most tested and
looked-at.  There probably *are* more bugs in Linux than in NetBSD or
FreeBSD; but also more of them are found because more people are
testing and looking.

But in any case, in my mind, the promise never was "we test all
versions of Linux with all versions of Xen", much less "We test all
versions of all operating systems with all versions of Xen".  The
promise was, "We test a representative sample of Linux and Xen
combinations, and we promise to report issues if they are found."
That's what I thing is the right thing to do for livepatch-tools as
well.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Roger Pau Monné]

On Tue, Aug 22, 2017 at 11:58:57AM +0100, George Dunlap wrote:
> I think guest OS support is actually a pretty good analog.  I can't
> imagine not issuing XSAs for bugs in Linux, just as I can't imagine
> not issuing XSAs for actual security issues that get found in the
> livepatch tools.  If you think we shouldn't give security support for
> Linux, it makes sense that you would feel the same way for
> livepatch-tools (although I don't really understand why you think that
> way about either).
> 
> We issue more XSAs for Linux than for other guests, in part because of
> the complexity of the code inside Linux compared to other OSes; but
> also in part due to the fact that that is the most tested and
> looked-at.  There probably *are* more bugs in Linux than in NetBSD or
> FreeBSD; but also more of them are found because more people are
> testing and looking.

IMHO, we issue XSA for Linux because Linux lacks a security process.
If a bug was found in the BSDs, it should be handled using the normal
security process that each BSD has, and a SA would be issued by the
security officer:

https://www.freebsd.org/security/advisories.html

For example NetBSD has recently released a SA for a Xen-specific
PV vulnerability in their implementation:

ftp://ftp.nl.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2017-003.txt.asc

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Konrad Rzeszutek Wilk]

.giant snip..
> Indeed; and as I think I said before, I think we need to move forward
> with getting a statement on livepatching in, and since most of the
> voices involved in this conversation seem to be in favor of saying
> livepatch-tools are *not* supported, I won't object. I'm only still

Thank you.

As such, here is the patch. Would folks like me to repost it, or
OK with Acking/Reviewing it as such?

I think the point 3) succinctly explains the position that has been so hotly
debated. I can of course expand it, but not sure if it makes sense?



From f968f003d36ac2b9d1b670b34bbe2a1222830138 Mon Sep 17 00:00:00 2001
From: Ross Lagerwall <ross.lagerwall@citrix.com>
Date: Wed, 28 Jun 2017 17:13:44 +0100
Subject: [PATCH v3] livepatch: Declare live patching as a supported feature

See docs/features/livepatch.pandoc for the details.

Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

--
v2:
 - Moved it into a feature document.
 - Clarified a few bits and pieces based on feedback.
v3:
 - default X86
---
 docs/features/livepatch.pandoc | 103 +++++++++++++++++++++++++++++++++++++++++
 xen/common/Kconfig             |   4 +-
 2 files changed, 105 insertions(+), 2 deletions(-)
 create mode 100644 docs/features/livepatch.pandoc

diff --git a/docs/features/livepatch.pandoc b/docs/features/livepatch.pandoc
new file mode 100644
index 0000000000..faaf2d1e77
--- /dev/null
+++ b/docs/features/livepatch.pandoc
@@ -0,0 +1,103 @@
+% Live Patching
+% Revision 1
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+         Status: **Supported**
+
+   Architecture: x86
+
+      Component: Hypervisor, toolstack
+---------------- ----------------------------------------------------
+
+
+# Details
+
+Xen Live Patching has been available as tech preview feature since Xen
+4.7 and has now had a couple of releases to stabilize. Xen Live patching
+has been used by multiple vendors to fix several real-world security
+issues without any severe bugs encountered. Additionally, there are now
+tests in OSSTest that test live patching to ensure that no regressions
+are introduced.
+
+Based on the amount of testing and usage it has had, we are ready to
+declare live patching as a 'Supported' feature on x86.
+
+Live patching is slightly peculiar when it comes to support because it
+allows the host administrator to break their system rather easily
+depending on the content of the live patch. Because of this, it is
+worth detailing the scope of security support:
+
+1) Unprivileged access to live patching operations:
+   Live patching operations should only be accessible to privileged
+   guests and it shall be treated as a security issue if this is not
+   the case.
+
+2) Bugs in the patch-application code such that vulnerabilities exist
+   after application:
+   If a correct live patch is loaded but it is not applied correctly
+   such that it might result in an insecure system (e.g. not all
+   functions are patched), it shall be treated as a security issue.
+
+3) Bugs in livepatch-build-tools creating an incorrect live patch that
+   results in an insecure host:
+   If livepatch-build-tools creates an incorrect live patch that
+   results in an insecure host, this shall not be considered a security
+   issue. There are too many OSes and toolchains to consider supporting
+   this. A live patch should be checked to verify that it is valid
+   before loading.
+
+4) Loading an incorrect live patch that results in an insecure host or
+   host crash:
+   If a live patch (whether created using livepatch-build-tools or some
+   alternative) is loaded and it results in an insecure host or host
+   crash due to the content of the live patch being incorrect or the
+   issue being inappropriate to live patch, this is not considered as a
+   security issue.
+
+5) Bugs in the live patch parsing code (the ELF loader):
+   Bugs in the live patch parsing code such as out-of-bounds reads
+   caused by invalid ELF files are not considered to be security issues
+   because the it can only be triggered by a privileged domain.
+
+6) Bugs which allow a guest to prevent the application of a livepatch:
+   A guest should not be able to prevent the application of a live
+   patch. If an unprivileged guest can somehow prevent the application
+   of a live patch despite pausing it (xl pause ...), it shall be
+   treated as a security issue.
+
+Note: It is expected that live patches are tested in a test environment
+before being used in production to avoid unexpected issues. In
+particular, to avoid the issues described by (3), (4), & (5).
+
+There are also some generic security questions which are worth asking:
+
+1) Is guest->host privilege escalation possible?
+
+The new live patching sysctl subops are only accessible to privileged
+domains and this is tested by OSSTest with an XTF test.
+There is a caveat -- an incorrect live patch can introduce a guest->host
+privilege escalation.
+
+2) Is guest user->guest kernel escalation possible?
+
+No, although an incorrect live patch can introduce a guest user->guest
+kernel privilege escalation.
+
+3) Is there any information leakage?
+
+The new live patching sysctl subops are only accessible to privileged
+domains so it is not possible for an unprivileged guest to access the
+list of loaded live patches. This is tested by OSSTest with an XTF test.
+There is a caveat -- an incorrect live patch can introduce an
+information leakage.
+
+4) Can a Denial-of-Service be triggered?
+
+There are no known ways that an unprivileged guest can prevent a live
+patch from being loaded.
+Once again, there is a caveat that an incorrect live patch can introduce
+an arbitrary denial of service.
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index dc8e876439..e9bb849298 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -226,8 +226,8 @@ config CRYPTO
 	bool
 
 config LIVEPATCH
-	bool "Live patching support (TECH PREVIEW)"
-	default n
+	bool "Live patching support"
+	default X86
 	depends on HAS_BUILD_ID = "y"
 	---help---
 	  Allows a running Xen hypervisor to be dynamically patched using
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[Andrew Cooper]

On 29/08/17 15:44, Konrad Rzeszutek Wilk wrote:
> .giant snip..
>> Indeed; and as I think I said before, I think we need to move forward
>> with getting a statement on livepatching in, and since most of the
>> voices involved in this conversation seem to be in favor of saying
>> livepatch-tools are *not* supported, I won't object. I'm only still
> Thank you.
>
> As such, here is the patch. Would folks like me to repost it, or
> OK with Acking/Reviewing it as such?
>
> I think the point 3) succinctly explains the position that has been so hotly
> debated. I can of course expand it, but not sure if it makes sense?
>
>
>
> From f968f003d36ac2b9d1b670b34bbe2a1222830138 Mon Sep 17 00:00:00 2001
> From: Ross Lagerwall <ross.lagerwall@citrix.com>
> Date: Wed, 28 Jun 2017 17:13:44 +0100
> Subject: [PATCH v3] livepatch: Declare live patching as a supported feature
>
> See docs/features/livepatch.pandoc for the details.
>
> Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>
> --
> v2:
>  - Moved it into a feature document.
>  - Clarified a few bits and pieces based on feedback.
> v3:
>  - default X86
> ---
>  docs/features/livepatch.pandoc | 103 +++++++++++++++++++++++++++++++++++++++++
>  xen/common/Kconfig             |   4 +-
>  2 files changed, 105 insertions(+), 2 deletions(-)
>  create mode 100644 docs/features/livepatch.pandoc
>
> diff --git a/docs/features/livepatch.pandoc b/docs/features/livepatch.pandoc
> new file mode 100644
> index 0000000000..faaf2d1e77
> --- /dev/null
> +++ b/docs/features/livepatch.pandoc
> @@ -0,0 +1,103 @@
> +% Live Patching
> +% Revision 1
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> +         Status: **Supported**
> +
> +   Architecture: x86

Sorry to only be noticing this now.

This should read Supported for x86, Tech Preview/Experimental (as
appropriate) for ARM.

~Andrew

> +
> +      Component: Hypervisor, toolstack
> +---------------- ----------------------------------------------------


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap]

On 08/29/2017 03:44 PM, Konrad Rzeszutek Wilk wrote:
> .giant snip..
>> Indeed; and as I think I said before, I think we need to move forward
>> with getting a statement on livepatching in, and since most of the
>> voices involved in this conversation seem to be in favor of saying
>> livepatch-tools are *not* supported, I won't object. I'm only still
> 
> Thank you.
> 
> As such, here is the patch. Would folks like me to repost it, or
> OK with Acking/Reviewing it as such?
> 
> I think the point 3) succinctly explains the position that has been so hotly
> debated. I can of course expand it, but not sure if it makes sense?

I'd prefer to remove a justification I believe to be unreasonable, and
just say what the support status is:

> +3) Bugs in livepatch-build-tools creating an incorrect live patch that
> +   results in an insecure host:
> +   If livepatch-build-tools creates an incorrect live patch that
> +   results in an insecure host, this shall not be considered a security
> +   issue. A live patch should be checked to verify that it is valid
> +   before loading.

Is that OK with everyone?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel