[Xen-devel] Design session report: Xen on Distros

[Xen-devel] Design session report: Xen on Distros

[George Dunlap]

Much of this covered things discussed elsewhere:
* Allowing multiple versions of the tools to be installed at the same time
* Getting rid of external builds

There was a long discussion about security patches, with the general
proposal being that we should cut a point release for every security issue.

One random thing was that xenstored apparently has an 'in-memory-only'
option.  Since xenstored can't actually be restarted ATM, and most
distros seemed to put xenstored in a tmpfs for performance reasons, this
should probably be the default.

https://hackmd.io/vmacVBYbQiORJ9H4_a9Ivw

# Xen on Distros design session

* qemu / libxc dependency loop
* build system needs "extras" turned off
* xenstored / tmpfs / memory-only option?
* Disabling auto-download in build system
    * WGET=/bin/false
* Multiple version of Xen / tools?
    * Debian has co-install
        * Change some installation paths
        * /usr/lib/xen/4.11/...
        * /usr/bin/xl is a shell script
        * libfsimage is special
        * Don't need to downgrade to older tools
    * Gentoo has a ~~dumpster fire~~ something
        * A hack which stops the package manager to allow you to reboot
the box halfway through
* Security issues
    * Building from stable branch / staging branch
    * Doing a "point release" every XSA?
    * "Release from staging" is effectively a low-quality release
    * Idea: Always immediately release from staging?


# Actions
* [ ] Ian: Post a git branch of Debian co-install to xen-devel
* [ ] George: Post systemd / selinux / xenstored patch
* [ ] George, Ian: private osstest runs
* [ ] VOLUNTEER: Propose / argue for a point release per XSA
* [ ] VOLUNTEER: Improve release automation

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[Jan Beulich]

On 15.07.2019 16:11, George Dunlap wrote:
> There was a long discussion about security patches, with the general
> proposal being that we should cut a point release for every security issue.

Interesting. Looks like in politics that until a decision fits people
they keep re-raising the point. Iirc on a prior meeting (Budapest?)
we had settled on continuing with the current scheme. Were there any
new arguments towards this alternative model?

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[George Dunlap]

On 7/15/19 3:23 PM, Jan Beulich wrote:
> On 15.07.2019 16:11, George Dunlap wrote:
>> There was a long discussion about security patches, with the general
>> proposal being that we should cut a point release for every security issue.
> 
> Interesting. Looks like in politics that until a decision fits people
> they keep re-raising the point. Iirc on a prior meeting (Budapest?)
> we had settled on continuing with the current scheme. Were there any
> new arguments towards this alternative model?

Well I don't know if there were any new arguments because I don't
immediately remember the old discussion.  Do we have a summary of the
discussion in Budapest, with its conclusions, anywhere?

The basic idea was that:

1. Most distros / packagers are going to want to do an immediate release
anyway.

2. Distros generally seemed to be rebasing on top of staging as soon as
the XSA went out anyway (and ISTR this being the recommeneded course of
action)

So for all intents and purposes, we have something which is, in fact, a
release; all it's missing is a signed tag and a tarball.

Obviously there are testing implications that would need to be sorted
out before this could become a reality.

In any case, the ball is in the court of "VOLUNTEER" to write up a
concrete proposal which could be discussed.  You'll be able to raise all
your concerns at that point if you want (although having a sketch would
of course be helpful for whoever is writing such a proposal).

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[George Dunlap]

On 7/15/19 3:42 PM, George Dunlap wrote:
> On 7/15/19 3:23 PM, Jan Beulich wrote:
>> On 15.07.2019 16:11, George Dunlap wrote:
>>> There was a long discussion about security patches, with the general
>>> proposal being that we should cut a point release for every security issue.
>>
>> Interesting. Looks like in politics that until a decision fits people
>> they keep re-raising the point. Iirc on a prior meeting (Budapest?)
>> we had settled on continuing with the current scheme. Were there any
>> new arguments towards this alternative model?

If we end up deciding against doing this again, it might be worth
creating a ticket somewhere to try to capture the arguments on each
side, so that we don't end up re-hashing the same issues in another few
years.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[Jan Beulich]

On 15.07.2019 16:42, George Dunlap wrote:
> On 7/15/19 3:23 PM, Jan Beulich wrote:
>> On 15.07.2019 16:11, George Dunlap wrote:
>>> There was a long discussion about security patches, with the general
>>> proposal being that we should cut a point release for every security issue.
>>
>> Interesting. Looks like in politics that until a decision fits people
>> they keep re-raising the point. Iirc on a prior meeting (Budapest?)
>> we had settled on continuing with the current scheme. Were there any
>> new arguments towards this alternative model?
> 
> Well I don't know if there were any new arguments because I don't
> immediately remember the old discussion.  Do we have a summary of the
> discussion in Budapest, with its conclusions, anywhere?

I don't recall if suitable notes were taken back then; as indicated
I'm not even sure which meeting it was at.

> The basic idea was that:
> 
> 1. Most distros / packagers are going to want to do an immediate release
> anyway.
> 
> 2. Distros generally seemed to be rebasing on top of staging as soon as
> the XSA went out anyway (and ISTR this being the recommeneded course of
> action)
> 
> So for all intents and purposes, we have something which is, in fact, a
> release; all it's missing is a signed tag and a tarball.
> 
> Obviously there are testing implications that would need to be sorted
> out before this could become a reality.
> 
> In any case, the ball is in the court of "VOLUNTEER" to write up a
> concrete proposal which could be discussed.  You'll be able to raise all
> your concerns at that point if you want (although having a sketch would
> of course be helpful for whoever is writing such a proposal).

Sure - I realized soon after having sent the initial reply that perhaps
this was the wrong context in the first place to raise my question.

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[Amit Shah]

On Mon, 2019-07-15 at 14:52 +0000, Jan Beulich wrote:
> On 15.07.2019 16:42, George Dunlap wrote:
> > On 7/15/19 3:23 PM, Jan Beulich wrote:
> > > On 15.07.2019 16:11, George Dunlap wrote:
> > > > There was a long discussion about security patches, with the
> > > > general
> > > > proposal being that we should cut a point release for every
> > > > security issue.
> > > 
> > > Interesting. Looks like in politics that until a decision fits
> > > people
> > > they keep re-raising the point. Iirc on a prior meeting
> > > (Budapest?)
> > > we had settled on continuing with the current scheme. Were there
> > > any
> > > new arguments towards this alternative model?
> > 
> > Well I don't know if there were any new arguments because I don't
> > immediately remember the old discussion.  Do we have a summary of
> > the
> > discussion in Budapest, with its conclusions, anywhere?
> 
> I don't recall if suitable notes were taken back then; as indicated
> I'm not even sure which meeting it was at.
> 
> > The basic idea was that:
> > 
> > 1. Most distros / packagers are going to want to do an immediate
> > release
> > anyway.
> > 
> > 2. Distros generally seemed to be rebasing on top of staging as
> > soon as
> > the XSA went out anyway (and ISTR this being the recommeneded
> > course of
> > action)
> > 
> > So for all intents and purposes, we have something which is, in
> > fact, a
> > release; all it's missing is a signed tag and a tarball.
> > 
> > Obviously there are testing implications that would need to be
> > sorted
> > out before this could become a reality.
> > 
> > In any case, the ball is in the court of "VOLUNTEER" to write up a
> > concrete proposal which could be discussed.  You'll be able to
> > raise all
> > your concerns at that point if you want (although having a sketch
> > would
> > of course be helpful for whoever is writing such a proposal).
> 
> Sure - I realized soon after having sent the initial reply that
> perhaps
> this was the wrong context in the first place to raise my question.

In any case, I'd like to know why it doesn't make sense for Xen to have
a point release frequently, and not have a point release after an XSA
above some severity level (pick one - high/critical/important).  As
George mentioned, distros have to do it anyway, and the upstream
project not doing it only makes it more difficult for all distros
involved.

Not sure of the politics involved though, and what can of worms this
opens.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[Andrew Cooper]

On 15/07/2019 18:52, Amit Shah wrote:
> On Mon, 2019-07-15 at 14:52 +0000, Jan Beulich wrote:
>> On 15.07.2019 16:42, George Dunlap wrote:
>>> On 7/15/19 3:23 PM, Jan Beulich wrote:
>>>> On 15.07.2019 16:11, George Dunlap wrote:
>>>>> There was a long discussion about security patches, with the
>>>>> general
>>>>> proposal being that we should cut a point release for every
>>>>> security issue.
>>>> Interesting. Looks like in politics that until a decision fits
>>>> people
>>>> they keep re-raising the point. Iirc on a prior meeting
>>>> (Budapest?)
>>>> we had settled on continuing with the current scheme. Were there
>>>> any
>>>> new arguments towards this alternative model?
>>> Well I don't know if there were any new arguments because I don't
>>> immediately remember the old discussion.  Do we have a summary of
>>> the
>>> discussion in Budapest, with its conclusions, anywhere?
>> I don't recall if suitable notes were taken back then; as indicated
>> I'm not even sure which meeting it was at.
>>
>>> The basic idea was that:
>>>
>>> 1. Most distros / packagers are going to want to do an immediate
>>> release
>>> anyway.
>>>
>>> 2. Distros generally seemed to be rebasing on top of staging as
>>> soon as
>>> the XSA went out anyway (and ISTR this being the recommeneded
>>> course of
>>> action)
>>>
>>> So for all intents and purposes, we have something which is, in
>>> fact, a
>>> release; all it's missing is a signed tag and a tarball.
>>>
>>> Obviously there are testing implications that would need to be
>>> sorted
>>> out before this could become a reality.
>>>
>>> In any case, the ball is in the court of "VOLUNTEER" to write up a
>>> concrete proposal which could be discussed.  You'll be able to
>>> raise all
>>> your concerns at that point if you want (although having a sketch
>>> would
>>> of course be helpful for whoever is writing such a proposal).
>> Sure - I realized soon after having sent the initial reply that
>> perhaps
>> this was the wrong context in the first place to raise my question.
> In any case, I'd like to know why it doesn't make sense for Xen to have
> a point release frequently, and not have a point release after an XSA
> above some severity level (pick one - high/critical/important).

We specifically do not rate XSAs like that.  One persons apple is a
different persons orange, considering how varied the environments which
Xen runs in are.

If in doubt, people should take all the XSA fixes.

> As George mentioned, distros have to do it anyway, and the upstream
> project not doing it only makes it more difficult for all distros
> involved.
>
> Not sure of the politics involved though, and what can of worms this
> opens.

Its a perfectly valid discussion to have, and comes down (in large part)
to the overhead of doing releases, which comes largely from other areas
of our infrastructure which are currently under question.

A full point release includes tagging a load of repos (2x qemu, seabios,
ovmf iirc), then waiting for generally 2-3 weeks for OSSTest to
stabilise enough to declare the patches good, even for ones which were
tested (elsewhere) to various downstream's satisfaction during the
embargo period.

We could trivially throw the fixes into the branch, tag it, sign it and
throw it out into the open, but doing that on the embargo date itself
would result in an official release of Xen which has had 0 testing in
the incumbent test system.

One aspect which would ease things is to not include 3rd party packages
in releases.  This was discussed as part of the build system gripes
session iirc, and would reduce the number of in-flight moving parts to
just xen.git.

Another aspect is that of the test system.  At the moment, nothing can
be released on the older security branches, because they still haven't
recovered from the carnage of the Debian upgrade.  Irrespective of that
particular issue, the general delay between patches appearing and
OSSTest saying yes is a concerning factor in how much effort it takes to
get a release out.


What a number of people want is for the patches in an XSA advisory to
apply cleanly to whatever random thing they have in their local/distro
patch queue.  This is entirely impossible for the security to arrange,
and furthermore, we have exactly one location where the patches we
produce will be committed.

As a personal view here, I don't think blindly taking the latest
staging-$X.$Y is a viable substitute for at least understanding the
patches well enough to work around trivial offsets/fuzz due to minor
variations.

However, if the overhead of doing a micro release became substantially
less, and producing a micro release after every XSA would help a decent
chunk of our downstreams consume security fixes more easily, then it
would be worth considering.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[George Dunlap]

> On Jul 16, 2019, at 11:03 PM, Andrew Cooper 
> 
> We could trivially throw the fixes into the branch, tag it, sign it and
> throw it out into the open, but doing that on the embargo date itself
> would result in an official release of Xen which has had 0 testing in
> the incumbent test system.

The point is that anyone who ships / deploys a fix on the disclosure date is pretty much shipping exactly that.  If it’s not good enough to sign, why is it good enough to deploy?

Probably the best answer is that it’s _not_ really good enough to deploy; and so it’s worth thinking about how we can improve that, perhaps by having embargoed osstest runs or something.

> What a number of people want is for the patches in an XSA advisory to
> apply cleanly to whatever random thing they have in their local/distro
> patch queue.  This is entirely impossible for the security to arrange,
> and furthermore, we have exactly one location where the patches we
> produce will be committed.

I’m not sure people want to apply “wherever”; it’s more that there wasn’t a clear place that they *could* apply.  Without any prior knowledge, I think the most natural expectation would be that you could take the most recent point release and just stack on all the outstanding XSAs since then.  There are reasons why we don’t do that, but I wouldn’t expect users to guess that.

This is the sort of area where maybe a document in your sphinx system would be helpful, just to lay out the issues.

> As a personal view here, I don't think blindly taking the latest
> staging-$X.$Y is a viable substitute for at least understanding the
> patches well enough to work around trivial offsets/fuzz due to minor
> variations.

I think this is fine for downstreams that have full-fledged hypervisor development teams (like XenServer or Amazon), but is a bit too high a barrier for "mom-and-pop" cloud providers or community-maintained distributions.

 -George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[Jan Beulich]

On 17.07.2019 12:33, George Dunlap wrote:
> 
>> On Jul 16, 2019, at 11:03 PM, Andrew Cooper
>>
>> We could trivially throw the fixes into the branch, tag it, sign it and
>> throw it out into the open, but doing that on the embargo date itself
>> would result in an official release of Xen which has had 0 testing in
>> the incumbent test system.
> 
> The point is that anyone who ships / deploys a fix on the disclosure date
> is pretty much shipping exactly that.  If it’s not good enough to sign,
> why is it good enough to deploy?

I think the security fixes themselves are good enough to deploy, perhaps
with the assumption that consumers of our pre-disclosure list have done
some testing on their own. The stable trees, however, contain more than
just security fixes, and can have regressions (most likely due to
backporting mistakes).

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[Andrew Cooper]

On 17/07/2019 13:37, Jan Beulich wrote:
> On 17.07.2019 12:33, George Dunlap wrote:
>>> On Jul 16, 2019, at 11:03 PM, Andrew Cooper
>>>
>>> We could trivially throw the fixes into the branch, tag it, sign it and
>>> throw it out into the open, but doing that on the embargo date itself
>>> would result in an official release of Xen which has had 0 testing in
>>> the incumbent test system.
>> The point is that anyone who ships / deploys a fix on the disclosure date
>> is pretty much shipping exactly that.  If it’s not good enough to sign,
>> why is it good enough to deploy?
> I think the security fixes themselves are good enough to deploy, perhaps
> with the assumption that consumers of our pre-disclosure list have done
> some testing on their own. The stable trees, however, contain more than
> just security fixes, and can have regressions (most likely due to
> backporting mistakes).

Right, but e.g. proposed changing the commit/push model whereby all
changes must pass CI before they get merged would reduce the likelyhood
of bad backports getting into staging-* in the first place.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Design session report: Xen on Distros

[Hans van Kranenburg]

Hi,

On 7/15/19 4:42 PM, George Dunlap wrote:
> On 7/15/19 3:23 PM, Jan Beulich wrote:
>> On 15.07.2019 16:11, George Dunlap wrote:
>>> There was a long discussion about security patches, with the general
>>> proposal being that we should cut a point release for every security issue.
>>
>> Interesting. Looks like in politics that until a decision fits people
>> they keep re-raising the point. Iirc on a prior meeting (Budapest?)
>> we had settled on continuing with the current scheme. Were there any
>> new arguments towards this alternative model?
> 
> Well I don't know if there were any new arguments because I don't
> immediately remember the old discussion.  Do we have a summary of the
> discussion in Budapest, with its conclusions, anywhere?
> 
> The basic idea was that:
> 
> 1. Most distros / packagers are going to want to do an immediate release
> anyway.
> 
> 2. Distros generally seemed to be rebasing on top of staging as soon as
> the XSA went out anyway (and ISTR this being the recommeneded course of
> action)

FYI, In Debian, we only ship the stable branch, not the staging branch.
Better safe than sorry.

And yes, this means there's a delay, and it's not immediate, etc.

Well, at least since I have been involved. In the past security update
packages were made by applying patches manually on top of older (like,
4.4.1 instead of 4.4.x) frozen versions, but we decided that "trying to
assemble our own subset of the patches is riskier than taking upstream's
collection".

The Debian Release Team allows us to upload newer upstream versions
during a security upload for Xen, which is officially not allowed in
Debian stable. One of the things that obviously help with being able to
keep doing this is the "track record" of the quality (expecially
regression testing) of the stable-4.x branches.

Currently, our package versions mostly look like e.g.
4.11.1+92-g6c33308a8d-1, instead of a nice simple version number. For me
this is fine, but I can understand that for an end user it would just
look nicer (and feel better perception-wise) to get a 4.11.x package.

So just for that reason I would be all +1 for more tagged releases.

> So for all intents and purposes, we have something which is, in fact, a
> release; all it's missing is a signed tag and a tarball.
> 
> Obviously there are testing implications that would need to be sorted
> out before this could become a reality.
> 
> In any case, the ball is in the court of "VOLUNTEER" to write up a
> concrete proposal which could be discussed.  You'll be able to raise all
> your concerns at that point if you want (although having a sketch would
> of course be helpful for whoever is writing such a proposal).

Hans

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel