RFC: Automatically making a PCI device assignable in the config file

RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

I've been doing some work to try to make driver domains easier to set
up and use.  At the moment, in order to pass a device through to a
guest, you first need to assign it to pciback.  This involves doing
one of three things:
* Running xl pci-assignable-add for the device
* Specifying the device to be grabbed on the dom0 Linux command-line
* Doing some hackery in /etc/modules.d

None of these are very satisfying.  What I think would be better is if
there was a way to specify in the guest config file, "If device X is
not assignable, try to make it assignable".  That way you can have a
driver domain grab the appropriate device just by running "xl create
domnet"; and once we have the xendomains script up and running with
xl, you can simply configure your domnet appropriately, and then put
it in /etc/xen/auto, to be started automatically on boot.

My initial idea was to add a parameter to the pci argument in the
config file; for example:

pci = ['08:04.1,permissive=1,seize=1']

The 'seize=1' would indicate that if bdf 08:04.1 is not already
assignable, that xl should try to make is assignable.

The problem here is that this would need to be parsed by
xlu_pci_parse_bdf(), which only takes an argumen tof type
libxl_device_pci.

Now it seems to me that the right place to do this "seizing" is in xl,
not inside libxl -- the functions for doing assignment exist already,
and are simple and straightforward.  But doing it in xl, but as a
parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
pass something else back, which begins to get awkward.

So it seems to me we have a couple of options:
1. Create a new argument, "pci_seize" or something like that, which
would be processed separately from pci
2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
arguments directed at xl rather than libxl
3. Add "seize" to libxl_device_pci, but have it only used by xl
4. Add "seize" to libxl_device_pci, and have libxl do the seizing.

Any preference -- or any other ideas?

 -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Andrew Cooper]

On 05/07/13 12:01, George Dunlap wrote:
> I've been doing some work to try to make driver domains easier to set
> up and use.  At the moment, in order to pass a device through to a
> guest, you first need to assign it to pciback.  This involves doing
> one of three things:
> * Running xl pci-assignable-add for the device
> * Specifying the device to be grabbed on the dom0 Linux command-line
> * Doing some hackery in /etc/modules.d
>
> None of these are very satisfying.  What I think would be better is if
> there was a way to specify in the guest config file, "If device X is
> not assignable, try to make it assignable".  That way you can have a
> driver domain grab the appropriate device just by running "xl create
> domnet"; and once we have the xendomains script up and running with
> xl, you can simply configure your domnet appropriately, and then put
> it in /etc/xen/auto, to be started automatically on boot.
>
> My initial idea was to add a parameter to the pci argument in the
> config file; for example:
>
> pci = ['08:04.1,permissive=1,seize=1']
>
> The 'seize=1' would indicate that if bdf 08:04.1 is not already
> assignable, that xl should try to make is assignable.
>
> The problem here is that this would need to be parsed by
> xlu_pci_parse_bdf(), which only takes an argumen tof type
> libxl_device_pci.
>
> Now it seems to me that the right place to do this "seizing" is in xl,
> not inside libxl -- the functions for doing assignment exist already,
> and are simple and straightforward.  But doing it in xl, but as a
> parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
> pass something else back, which begins to get awkward.
>
> So it seems to me we have a couple of options:
> 1. Create a new argument, "pci_seize" or something like that, which
> would be processed separately from pci
> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
> arguments directed at xl rather than libxl
> 3. Add "seize" to libxl_device_pci, but have it only used by xl
> 4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
>
> Any preference -- or any other ideas?
>
>  -George

How about a setting in xl.conf of "auto-seize pci devices" ?  That way
the seizing is entirely part of xl

~Andrew

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 05/07/13 14:39, Andrew Cooper wrote:
> On 05/07/13 12:01, George Dunlap wrote:
>> I've been doing some work to try to make driver domains easier to set
>> up and use.  At the moment, in order to pass a device through to a
>> guest, you first need to assign it to pciback.  This involves doing
>> one of three things:
>> * Running xl pci-assignable-add for the device
>> * Specifying the device to be grabbed on the dom0 Linux command-line
>> * Doing some hackery in /etc/modules.d
>>
>> None of these are very satisfying.  What I think would be better is if
>> there was a way to specify in the guest config file, "If device X is
>> not assignable, try to make it assignable".  That way you can have a
>> driver domain grab the appropriate device just by running "xl create
>> domnet"; and once we have the xendomains script up and running with
>> xl, you can simply configure your domnet appropriately, and then put
>> it in /etc/xen/auto, to be started automatically on boot.
>>
>> My initial idea was to add a parameter to the pci argument in the
>> config file; for example:
>>
>> pci = ['08:04.1,permissive=1,seize=1']
>>
>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>> assignable, that xl should try to make is assignable.
>>
>> The problem here is that this would need to be parsed by
>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>> libxl_device_pci.
>>
>> Now it seems to me that the right place to do this "seizing" is in xl,
>> not inside libxl -- the functions for doing assignment exist already,
>> and are simple and straightforward.  But doing it in xl, but as a
>> parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
>> pass something else back, which begins to get awkward.
>>
>> So it seems to me we have a couple of options:
>> 1. Create a new argument, "pci_seize" or something like that, which
>> would be processed separately from pci
>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
>> arguments directed at xl rather than libxl
>> 3. Add "seize" to libxl_device_pci, but have it only used by xl
>> 4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
>>
>> Any preference -- or any other ideas?
>>
>>   -George
> How about a setting in xl.conf of "auto-seize pci devices" ?  That way
> the seizing is entirely part of xl

Auto-seizing is fairly dangerous; you could easily accidentally yank out 
the ethernet card, or even the disk that dom0 is using.  I really think 
it should have to be enabled on a device-by-device basis.

I suppose another option would be to be able to set, in xl.conf, a list 
of auto-seizeable devices.  I don't really like that option as well, 
though.  I'd rather be able to keep all the configuration in one place.

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Andrew Cooper]

On 05/07/13 14:45, George Dunlap wrote:
> On 05/07/13 14:39, Andrew Cooper wrote:
>> On 05/07/13 12:01, George Dunlap wrote:
>>> I've been doing some work to try to make driver domains easier to set
>>> up and use.  At the moment, in order to pass a device through to a
>>> guest, you first need to assign it to pciback.  This involves doing
>>> one of three things:
>>> * Running xl pci-assignable-add for the device
>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>> * Doing some hackery in /etc/modules.d
>>>
>>> None of these are very satisfying.  What I think would be better is if
>>> there was a way to specify in the guest config file, "If device X is
>>> not assignable, try to make it assignable".  That way you can have a
>>> driver domain grab the appropriate device just by running "xl create
>>> domnet"; and once we have the xendomains script up and running with
>>> xl, you can simply configure your domnet appropriately, and then put
>>> it in /etc/xen/auto, to be started automatically on boot.
>>>
>>> My initial idea was to add a parameter to the pci argument in the
>>> config file; for example:
>>>
>>> pci = ['08:04.1,permissive=1,seize=1']
>>>
>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>> assignable, that xl should try to make is assignable.
>>>
>>> The problem here is that this would need to be parsed by
>>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>>> libxl_device_pci.
>>>
>>> Now it seems to me that the right place to do this "seizing" is in xl,
>>> not inside libxl -- the functions for doing assignment exist already,
>>> and are simple and straightforward.  But doing it in xl, but as a
>>> parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
>>> pass something else back, which begins to get awkward.
>>>
>>> So it seems to me we have a couple of options:
>>> 1. Create a new argument, "pci_seize" or something like that, which
>>> would be processed separately from pci
>>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
>>> arguments directed at xl rather than libxl
>>> 3. Add "seize" to libxl_device_pci, but have it only used by xl
>>> 4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
>>>
>>> Any preference -- or any other ideas?
>>>
>>>   -George
>> How about a setting in xl.conf of "auto-seize pci devices" ?  That way
>> the seizing is entirely part of xl
>
> Auto-seizing is fairly dangerous; you could easily accidentally yank
> out the ethernet card, or even the disk that dom0 is using.  I really
> think it should have to be enabled on a device-by-device basis.
>
> I suppose another option would be to be able to set, in xl.conf, a
> list of auto-seizeable devices.  I don't really like that option as
> well, though.  I'd rather be able to keep all the configuration in one
> place.
>
>  -George

Or a slight less extreme version.

If xl sees that it would need seize a device, it could ask "You are
trying to create a domain with device $FOO.  Would you like to seize it
from dom0 ?"

I do think that libxl is not the correct place to have any logic like this.

~Andrew

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 05/07/13 14:48, Andrew Cooper wrote:
> On 05/07/13 14:45, George Dunlap wrote:
>> On 05/07/13 14:39, Andrew Cooper wrote:
>>> On 05/07/13 12:01, George Dunlap wrote:
>>>> I've been doing some work to try to make driver domains easier to set
>>>> up and use.  At the moment, in order to pass a device through to a
>>>> guest, you first need to assign it to pciback.  This involves doing
>>>> one of three things:
>>>> * Running xl pci-assignable-add for the device
>>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>>> * Doing some hackery in /etc/modules.d
>>>>
>>>> None of these are very satisfying.  What I think would be better is if
>>>> there was a way to specify in the guest config file, "If device X is
>>>> not assignable, try to make it assignable".  That way you can have a
>>>> driver domain grab the appropriate device just by running "xl create
>>>> domnet"; and once we have the xendomains script up and running with
>>>> xl, you can simply configure your domnet appropriately, and then put
>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>
>>>> My initial idea was to add a parameter to the pci argument in the
>>>> config file; for example:
>>>>
>>>> pci = ['08:04.1,permissive=1,seize=1']
>>>>
>>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>>> assignable, that xl should try to make is assignable.
>>>>
>>>> The problem here is that this would need to be parsed by
>>>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>>>> libxl_device_pci.
>>>>
>>>> Now it seems to me that the right place to do this "seizing" is in xl,
>>>> not inside libxl -- the functions for doing assignment exist already,
>>>> and are simple and straightforward.  But doing it in xl, but as a
>>>> parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
>>>> pass something else back, which begins to get awkward.
>>>>
>>>> So it seems to me we have a couple of options:
>>>> 1. Create a new argument, "pci_seize" or something like that, which
>>>> would be processed separately from pci
>>>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
>>>> arguments directed at xl rather than libxl
>>>> 3. Add "seize" to libxl_device_pci, but have it only used by xl
>>>> 4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
>>>>
>>>> Any preference -- or any other ideas?
>>>>
>>>>    -George
>>> How about a setting in xl.conf of "auto-seize pci devices" ?  That way
>>> the seizing is entirely part of xl
>> Auto-seizing is fairly dangerous; you could easily accidentally yank
>> out the ethernet card, or even the disk that dom0 is using.  I really
>> think it should have to be enabled on a device-by-device basis.
>>
>> I suppose another option would be to be able to set, in xl.conf, a
>> list of auto-seizeable devices.  I don't really like that option as
>> well, though.  I'd rather be able to keep all the configuration in one
>> place.
>>
>>   -George
> Or a slight less extreme version.
>
> If xl sees that it would need seize a device, it could ask "You are
> trying to create a domain with device $FOO.  Would you like to seize it
> from dom0 ?"

That won't work for driver domains, as we want it all to happen 
automatically when the host is booting. :-)

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Konrad Rzeszutek Wilk]

On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
> On 05/07/13 14:48, Andrew Cooper wrote:
> >On 05/07/13 14:45, George Dunlap wrote:
> >>On 05/07/13 14:39, Andrew Cooper wrote:
> >>>On 05/07/13 12:01, George Dunlap wrote:
> >>>>I've been doing some work to try to make driver domains easier to set
> >>>>up and use.  At the moment, in order to pass a device through to a
> >>>>guest, you first need to assign it to pciback.  This involves doing
> >>>>one of three things:
> >>>>* Running xl pci-assignable-add for the device
> >>>>* Specifying the device to be grabbed on the dom0 Linux command-line
> >>>>* Doing some hackery in /etc/modules.d
> >>>>
> >>>>None of these are very satisfying.  What I think would be better is if
> >>>>there was a way to specify in the guest config file, "If device X is
> >>>>not assignable, try to make it assignable".  That way you can have a
> >>>>driver domain grab the appropriate device just by running "xl create
> >>>>domnet"; and once we have the xendomains script up and running with
> >>>>xl, you can simply configure your domnet appropriately, and then put
> >>>>it in /etc/xen/auto, to be started automatically on boot.
> >>>>
> >>>>My initial idea was to add a parameter to the pci argument in the
> >>>>config file; for example:
> >>>>
> >>>>pci = ['08:04.1,permissive=1,seize=1']
> >>>>
> >>>>The 'seize=1' would indicate that if bdf 08:04.1 is not already
> >>>>assignable, that xl should try to make is assignable.
> >>>>
> >>>>The problem here is that this would need to be parsed by
> >>>>xlu_pci_parse_bdf(), which only takes an argumen tof type
> >>>>libxl_device_pci.
> >>>>
> >>>>Now it seems to me that the right place to do this "seizing" is in xl,
> >>>>not inside libxl -- the functions for doing assignment exist already,
> >>>>and are simple and straightforward.  But doing it in xl, but as a
> >>>>parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
> >>>>pass something else back, which begins to get awkward.
> >>>>
> >>>>So it seems to me we have a couple of options:
> >>>>1. Create a new argument, "pci_seize" or something like that, which
> >>>>would be processed separately from pci
> >>>>2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
> >>>>arguments directed at xl rather than libxl
> >>>>3. Add "seize" to libxl_device_pci, but have it only used by xl
> >>>>4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
> >>>>
> >>>>Any preference -- or any other ideas?
> >>>>
> >>>>   -George
> >>>How about a setting in xl.conf of "auto-seize pci devices" ?  That way
> >>>the seizing is entirely part of xl
> >>Auto-seizing is fairly dangerous; you could easily accidentally yank
> >>out the ethernet card, or even the disk that dom0 is using.  I really
> >>think it should have to be enabled on a device-by-device basis.
> >>
> >>I suppose another option would be to be able to set, in xl.conf, a
> >>list of auto-seizeable devices.  I don't really like that option as
> >>well, though.  I'd rather be able to keep all the configuration in one
> >>place.
> >>
> >>  -George
> >Or a slight less extreme version.
> >
> >If xl sees that it would need seize a device, it could ask "You are
> >trying to create a domain with device $FOO.  Would you like to seize it
> >from dom0 ?"
> 
> That won't work for driver domains, as we want it all to happen
> automatically when the host is booting. :-)

The high-level goal is that we want to put the network devices with a
network backend and storage devices with storage backend. Ignorning
that for network devices you might want seperate backends for each
device (say one backend for Wireless, one for Ethernet, etc).

Perhaps the logic ought to do grouping - so you say:
 a) "backends:all-network" (which would created one backend with all of the
   wireless, ethernet, etc PCI devices), or
 b) "backends:all-network,seperate-storage", which  create one backend with
  all of the wireless, ethernet in one backend; and one backend domain for each
  storage device?

Naturally the user gets to chose which grouping they would like?
> 
>  -George
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
>> On 05/07/13 14:48, Andrew Cooper wrote:
>>> On 05/07/13 14:45, George Dunlap wrote:
>>>> On 05/07/13 14:39, Andrew Cooper wrote:
>>>>> On 05/07/13 12:01, George Dunlap wrote:
>>>>>> I've been doing some work to try to make driver domains easier to set
>>>>>> up and use.  At the moment, in order to pass a device through to a
>>>>>> guest, you first need to assign it to pciback.  This involves doing
>>>>>> one of three things:
>>>>>> * Running xl pci-assignable-add for the device
>>>>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>>>>> * Doing some hackery in /etc/modules.d
>>>>>>
>>>>>> None of these are very satisfying.  What I think would be better is if
>>>>>> there was a way to specify in the guest config file, "If device X is
>>>>>> not assignable, try to make it assignable".  That way you can have a
>>>>>> driver domain grab the appropriate device just by running "xl create
>>>>>> domnet"; and once we have the xendomains script up and running with
>>>>>> xl, you can simply configure your domnet appropriately, and then put
>>>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>>>
>>>>>> My initial idea was to add a parameter to the pci argument in the
>>>>>> config file; for example:
>>>>>>
>>>>>> pci = ['08:04.1,permissive=1,seize=1']
>>>>>>
>>>>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>>>>> assignable, that xl should try to make is assignable.
>>>>>>
>>>>>> The problem here is that this would need to be parsed by
>>>>>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>>>>>> libxl_device_pci.
>>>>>>
>>>>>> Now it seems to me that the right place to do this "seizing" is in xl,
>>>>>> not inside libxl -- the functions for doing assignment exist already,
>>>>>> and are simple and straightforward.  But doing it in xl, but as a
>>>>>> parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
>>>>>> pass something else back, which begins to get awkward.
>>>>>>
>>>>>> So it seems to me we have a couple of options:
>>>>>> 1. Create a new argument, "pci_seize" or something like that, which
>>>>>> would be processed separately from pci
>>>>>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
>>>>>> arguments directed at xl rather than libxl
>>>>>> 3. Add "seize" to libxl_device_pci, but have it only used by xl
>>>>>> 4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
>>>>>>
>>>>>> Any preference -- or any other ideas?
>>>>>>
>>>>>>    -George
>>>>> How about a setting in xl.conf of "auto-seize pci devices" ?  That way
>>>>> the seizing is entirely part of xl
>>>> Auto-seizing is fairly dangerous; you could easily accidentally yank
>>>> out the ethernet card, or even the disk that dom0 is using.  I really
>>>> think it should have to be enabled on a device-by-device basis.
>>>>
>>>> I suppose another option would be to be able to set, in xl.conf, a
>>>> list of auto-seizeable devices.  I don't really like that option as
>>>> well, though.  I'd rather be able to keep all the configuration in one
>>>> place.
>>>>
>>>>   -George
>>> Or a slight less extreme version.
>>>
>>> If xl sees that it would need seize a device, it could ask "You are
>>> trying to create a domain with device $FOO.  Would you like to seize it
>> >from dom0 ?"
>>
>> That won't work for driver domains, as we want it all to happen
>> automatically when the host is booting. :-)
>
> The high-level goal is that we want to put the network devices with a
> network backend and storage devices with storage backend. Ignorning
> that for network devices you might want seperate backends for each
> device (say one backend for Wireless, one for Ethernet, etc).
>
> Perhaps the logic ought to do grouping - so you say:
>   a) "backends:all-network" (which would created one backend with all of the
>     wireless, ethernet, etc PCI devices), or
>   b) "backends:all-network,seperate-storage", which  create one backend with
>    all of the wireless, ethernet in one backend; and one backend domain for each
>    storage device?
>
> Naturally the user gets to chose which grouping they would like?

We seem to be talking about different things.  You seem to be talking 
about automatically starting some pre-made VMs and assigning devices and 
backends to them?  But I'm not really sure.

I was assuming that the user was going to be installing and configuring 
their own driver domains.  The user already has to specify 
"pci=['$BDF']" in their config file to get specific devices passed 
through -- this would just be making it easy to have the device assigned 
to pciback as well.

I suspect that a lot of people will want to have one network card 
assigned to domain 0 as a "management network", and only have other 
devices assigned to driver domains.  I think that having one device per 
domain is probably the best recommendation; although we obviously want 
to support someone who wants a single "manage all the devices" domain, 
we should assume that people are going to have one device per driver domain.

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Konrad Rzeszutek Wilk]

On Tue, Jul 09, 2013 at 01:52:38PM +0100, George Dunlap wrote:
> On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
> >On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
> >>On 05/07/13 14:48, Andrew Cooper wrote:
> >>>On 05/07/13 14:45, George Dunlap wrote:
> >>>>On 05/07/13 14:39, Andrew Cooper wrote:
> >>>>>On 05/07/13 12:01, George Dunlap wrote:
> >>>>>>I've been doing some work to try to make driver domains easier to set
> >>>>>>up and use.  At the moment, in order to pass a device through to a
> >>>>>>guest, you first need to assign it to pciback.  This involves doing
> >>>>>>one of three things:
> >>>>>>* Running xl pci-assignable-add for the device
> >>>>>>* Specifying the device to be grabbed on the dom0 Linux command-line
> >>>>>>* Doing some hackery in /etc/modules.d
> >>>>>>
> >>>>>>None of these are very satisfying.  What I think would be better is if
> >>>>>>there was a way to specify in the guest config file, "If device X is
> >>>>>>not assignable, try to make it assignable".  That way you can have a
> >>>>>>driver domain grab the appropriate device just by running "xl create
> >>>>>>domnet"; and once we have the xendomains script up and running with
> >>>>>>xl, you can simply configure your domnet appropriately, and then put
> >>>>>>it in /etc/xen/auto, to be started automatically on boot.
> >>>>>>
> >>>>>>My initial idea was to add a parameter to the pci argument in the
> >>>>>>config file; for example:
> >>>>>>
> >>>>>>pci = ['08:04.1,permissive=1,seize=1']
> >>>>>>
> >>>>>>The 'seize=1' would indicate that if bdf 08:04.1 is not already
> >>>>>>assignable, that xl should try to make is assignable.
> >>>>>>
> >>>>>>The problem here is that this would need to be parsed by
> >>>>>>xlu_pci_parse_bdf(), which only takes an argumen tof type
> >>>>>>libxl_device_pci.
> >>>>>>
> >>>>>>Now it seems to me that the right place to do this "seizing" is in xl,
> >>>>>>not inside libxl -- the functions for doing assignment exist already,
> >>>>>>and are simple and straightforward.  But doing it in xl, but as a
> >>>>>>parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
> >>>>>>pass something else back, which begins to get awkward.
> >>>>>>
> >>>>>>So it seems to me we have a couple of options:
> >>>>>>1. Create a new argument, "pci_seize" or something like that, which
> >>>>>>would be processed separately from pci
> >>>>>>2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
> >>>>>>arguments directed at xl rather than libxl
> >>>>>>3. Add "seize" to libxl_device_pci, but have it only used by xl
> >>>>>>4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
> >>>>>>
> >>>>>>Any preference -- or any other ideas?
> >>>>>>
> >>>>>>   -George
> >>>>>How about a setting in xl.conf of "auto-seize pci devices" ?  That way
> >>>>>the seizing is entirely part of xl
> >>>>Auto-seizing is fairly dangerous; you could easily accidentally yank
> >>>>out the ethernet card, or even the disk that dom0 is using.  I really
> >>>>think it should have to be enabled on a device-by-device basis.
> >>>>
> >>>>I suppose another option would be to be able to set, in xl.conf, a
> >>>>list of auto-seizeable devices.  I don't really like that option as
> >>>>well, though.  I'd rather be able to keep all the configuration in one
> >>>>place.
> >>>>
> >>>>  -George
> >>>Or a slight less extreme version.
> >>>
> >>>If xl sees that it would need seize a device, it could ask "You are
> >>>trying to create a domain with device $FOO.  Would you like to seize it
> >>>from dom0 ?"
> >>
> >>That won't work for driver domains, as we want it all to happen
> >>automatically when the host is booting. :-)
> >
> >The high-level goal is that we want to put the network devices with a
> >network backend and storage devices with storage backend. Ignorning
> >that for network devices you might want seperate backends for each
> >device (say one backend for Wireless, one for Ethernet, etc).
> >
> >Perhaps the logic ought to do grouping - so you say:
> >  a) "backends:all-network" (which would created one backend with all of the
> >    wireless, ethernet, etc PCI devices), or
> >  b) "backends:all-network,seperate-storage", which  create one backend with
> >   all of the wireless, ethernet in one backend; and one backend domain for each
> >   storage device?
> >
> >Naturally the user gets to chose which grouping they would like?
> 
> We seem to be talking about different things.  You seem to be
> talking about automatically starting some pre-made VMs and assigning
> devices and backends to them?  But I'm not really sure.

I am trying to look at it from a high perspective to see whether we can
make this automated for 99% of people out of the box. Hence the
idea of grouping. And yes to '..assigning devices and backends to them'.
> 
> I was assuming that the user was going to be installing and
> configuring their own driver domains.  The user already has to
> specify "pci=['$BDF']" in their config file to get specific devices
> passed through -- this would just be making it easy to have the
> device assigned to pciback as well.

I think the technical bits what libxl is doing and yanking devices
around is driven either by the admin or a policy. If the policy
is this idea of grouping (that is a terrible name now that I think
of it), then perhaps we should think how to make that work and then
the details (such as this automatic yanking of devices to pci-back)
can be filled in.


> 
> I suspect that a lot of people will want to have one network card
> assigned to domain 0 as a "management network", and only have other
> devices assigned to driver domains.  I think that having one device
> per domain is probably the best recommendation; although we
> obviously want to support someone who wants a single "manage all the
> devices" domain, we should assume that people are going to have one
> device per driver domain.

I don't know. My feeble idea was that we would have at minimum _two_
guests on bootup. One is a control one that has no devices - but is
the one that launches the guests.

Then there is the dom1 which would have all (or some) of the storage
and network devices plugged in along with the backends. Then a dom2
which would be the old-style-dom0 - so it would have the graphic card
and the rest of the PCI devices.

In other words, when I boot I would have two tiny domains launch
right before "old-style-dom0" is started. But I am getting in specifics
here.

Perhaps you could explain to me how you envisioned how the device
driver domains idea would work? How would you want it to work on your
laptop?

Or are we right now just thinking of the small pieces of making the
code be able to yank the devices around and assign them?

Thanks.

> 
>  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 07/09/2013 03:25 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Jul 09, 2013 at 01:52:38PM +0100, George Dunlap wrote:
>> On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
>>>> On 05/07/13 14:48, Andrew Cooper wrote:
>>>>> On 05/07/13 14:45, George Dunlap wrote:
>>>>>> On 05/07/13 14:39, Andrew Cooper wrote:
>>>>>>> On 05/07/13 12:01, George Dunlap wrote:
>>>>>>>> I've been doing some work to try to make driver domains easier to set
>>>>>>>> up and use.  At the moment, in order to pass a device through to a
>>>>>>>> guest, you first need to assign it to pciback.  This involves doing
>>>>>>>> one of three things:
>>>>>>>> * Running xl pci-assignable-add for the device
>>>>>>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>>>>>>> * Doing some hackery in /etc/modules.d
>>>>>>>>
>>>>>>>> None of these are very satisfying.  What I think would be better is if
>>>>>>>> there was a way to specify in the guest config file, "If device X is
>>>>>>>> not assignable, try to make it assignable".  That way you can have a
>>>>>>>> driver domain grab the appropriate device just by running "xl create
>>>>>>>> domnet"; and once we have the xendomains script up and running with
>>>>>>>> xl, you can simply configure your domnet appropriately, and then put
>>>>>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>>>>>
>>>>>>>> My initial idea was to add a parameter to the pci argument in the
>>>>>>>> config file; for example:
>>>>>>>>
>>>>>>>> pci = ['08:04.1,permissive=1,seize=1']
>>>>>>>>
>>>>>>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>>>>>>> assignable, that xl should try to make is assignable.
>>>>>>>>
>>>>>>>> The problem here is that this would need to be parsed by
>>>>>>>> xlu_pci_parse_bdf(), which only takes an argumen tof type
>>>>>>>> libxl_device_pci.
>>>>>>>>
>>>>>>>> Now it seems to me that the right place to do this "seizing" is in xl,
>>>>>>>> not inside libxl -- the functions for doing assignment exist already,
>>>>>>>> and are simple and straightforward.  But doing it in xl, but as a
>>>>>>>> parameter of the "pci" setting, means changing xlu_pci_parse_bdf() to
>>>>>>>> pass something else back, which begins to get awkward.
>>>>>>>>
>>>>>>>> So it seems to me we have a couple of options:
>>>>>>>> 1. Create a new argument, "pci_seize" or something like that, which
>>>>>>>> would be processed separately from pci
>>>>>>>> 2. Change xlu_pci_parse_bdf to take a pointer to an extra struct, for
>>>>>>>> arguments directed at xl rather than libxl
>>>>>>>> 3. Add "seize" to libxl_device_pci, but have it only used by xl
>>>>>>>> 4. Add "seize" to libxl_device_pci, and have libxl do the seizing.
>>>>>>>>
>>>>>>>> Any preference -- or any other ideas?
>>>>>>>>
>>>>>>>>    -George
>>>>>>> How about a setting in xl.conf of "auto-seize pci devices" ?  That way
>>>>>>> the seizing is entirely part of xl
>>>>>> Auto-seizing is fairly dangerous; you could easily accidentally yank
>>>>>> out the ethernet card, or even the disk that dom0 is using.  I really
>>>>>> think it should have to be enabled on a device-by-device basis.
>>>>>>
>>>>>> I suppose another option would be to be able to set, in xl.conf, a
>>>>>> list of auto-seizeable devices.  I don't really like that option as
>>>>>> well, though.  I'd rather be able to keep all the configuration in one
>>>>>> place.
>>>>>>
>>>>>>   -George
>>>>> Or a slight less extreme version.
>>>>>
>>>>> If xl sees that it would need seize a device, it could ask "You are
>>>>> trying to create a domain with device $FOO.  Would you like to seize it
>>>> >from dom0 ?"
>>>>
>>>> That won't work for driver domains, as we want it all to happen
>>>> automatically when the host is booting. :-)
>>>
>>> The high-level goal is that we want to put the network devices with a
>>> network backend and storage devices with storage backend. Ignorning
>>> that for network devices you might want seperate backends for each
>>> device (say one backend for Wireless, one for Ethernet, etc).
>>>
>>> Perhaps the logic ought to do grouping - so you say:
>>>   a) "backends:all-network" (which would created one backend with all of the
>>>     wireless, ethernet, etc PCI devices), or
>>>   b) "backends:all-network,seperate-storage", which  create one backend with
>>>    all of the wireless, ethernet in one backend; and one backend domain for each
>>>    storage device?
>>>
>>> Naturally the user gets to chose which grouping they would like?
>>
>> We seem to be talking about different things.  You seem to be
>> talking about automatically starting some pre-made VMs and assigning
>> devices and backends to them?  But I'm not really sure.
>
> I am trying to look at it from a high perspective to see whether we can
> make this automated for 99% of people out of the box. Hence the
> idea of grouping. And yes to '..assigning devices and backends to them'.
>>
>> I was assuming that the user was going to be installing and
>> configuring their own driver domains.  The user already has to
>> specify "pci=['$BDF']" in their config file to get specific devices
>> passed through -- this would just be making it easy to have the
>> device assigned to pciback as well.
>
> I think the technical bits what libxl is doing and yanking devices
> around is driven either by the admin or a policy. If the policy
> is this idea of grouping (that is a terrible name now that I think
> of it), then perhaps we should think how to make that work and then
> the details (such as this automatic yanking of devices to pci-back)
> can be filled in.
>
>
>>
>> I suspect that a lot of people will want to have one network card
>> assigned to domain 0 as a "management network", and only have other
>> devices assigned to driver domains.  I think that having one device
>> per domain is probably the best recommendation; although we
>> obviously want to support someone who wants a single "manage all the
>> devices" domain, we should assume that people are going to have one
>> device per driver domain.
>
> I don't know. My feeble idea was that we would have at minimum _two_
> guests on bootup. One is a control one that has no devices - but is
> the one that launches the guests.
>
> Then there is the dom1 which would have all (or some) of the storage
> and network devices plugged in along with the backends. Then a dom2
> which would be the old-style-dom0 - so it would have the graphic card
> and the rest of the PCI devices.
>
> In other words, when I boot I would have two tiny domains launch
> right before "old-style-dom0" is started. But I am getting in specifics
> here.
>
> Perhaps you could explain to me how you envisioned how the device
> driver domains idea would work? How would you want it to work on your
> laptop?
>
> Or are we right now just thinking of the small pieces of making the
> code be able to yank the devices around and assign them?

I was thinking for now just making the "manually configure it" case 
easier.  I decided to switch one of my test boxen to using a network 
driver domain by default, and although the core is there, there are a 
bunch of things that are unnecessarily crufty.

I do agree that long term it would be nice to make it easy to make 
driver domains the default, but that's not what I had in mind for this 
conversation. :-)

The hard part for making it really automated, it seems to me, comes from 
two things.  O

One, you have to make sure your driver domain has the appropriate 
hardware drivers for your system as well.  We don't want to be in the 
business of maintaining a distro; most people will probably want the 
driver domain to be from the same distro they're using for dom0, which 
means that setting up such a domain will need to be done differently on 
a distro-by-distro basis.

Two, you have the configuration problem.  In Debian, for instance, if 
you wanted to switch a device from being owned by dom0 to being in a 
driver domain, you'd have to:
* Copy over the udev rules recognizing the mac address, so it got the 
same ethN
* copy over the eth and bridge info from dom0's /etc/network/interfaces 
into the guest /etc/network/interfaces

I'm not sure exactly what you have to do in Fedora, but I bet it's 
something similar.

It might be nice to work with distros to make the process of making 
driver domains / stub domains easier, and to make it easy to configure 
driver domain networking options from the distro's network scripts; but 
that's kind of another level of functionality.

I think first things first: make manually-set-up driver domains actually 
easy to use.

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Stefano Stabellini]

On Tue, 9 Jul 2013, George Dunlap wrote:
> On 07/09/2013 03:25 PM, Konrad Rzeszutek Wilk wrote:
> > On Tue, Jul 09, 2013 at 01:52:38PM +0100, George Dunlap wrote:
> > > On 07/08/2013 08:23 PM, Konrad Rzeszutek Wilk wrote:
> > > > On Fri, Jul 05, 2013 at 02:52:08PM +0100, George Dunlap wrote:
> > > > > On 05/07/13 14:48, Andrew Cooper wrote:
> > > > > > On 05/07/13 14:45, George Dunlap wrote:
> > > > > > > On 05/07/13 14:39, Andrew Cooper wrote:
> > > > > > > > On 05/07/13 12:01, George Dunlap wrote:
> > > > > > > > > I've been doing some work to try to make driver domains easier
> > > > > > > > > to set
> > > > > > > > > up and use.  At the moment, in order to pass a device through
> > > > > > > > > to a
> > > > > > > > > guest, you first need to assign it to pciback.  This involves
> > > > > > > > > doing
> > > > > > > > > one of three things:
> > > > > > > > > * Running xl pci-assignable-add for the device
> > > > > > > > > * Specifying the device to be grabbed on the dom0 Linux
> > > > > > > > > command-line
> > > > > > > > > * Doing some hackery in /etc/modules.d
> > > > > > > > > 
> > > > > > > > > None of these are very satisfying.  What I think would be
> > > > > > > > > better is if
> > > > > > > > > there was a way to specify in the guest config file, "If
> > > > > > > > > device X is
> > > > > > > > > not assignable, try to make it assignable".  That way you can
> > > > > > > > > have a
> > > > > > > > > driver domain grab the appropriate device just by running "xl
> > > > > > > > > create
> > > > > > > > > domnet"; and once we have the xendomains script up and running
> > > > > > > > > with
> > > > > > > > > xl, you can simply configure your domnet appropriately, and
> > > > > > > > > then put
> > > > > > > > > it in /etc/xen/auto, to be started automatically on boot.
> > > > > > > > > 
> > > > > > > > > My initial idea was to add a parameter to the pci argument in
> > > > > > > > > the
> > > > > > > > > config file; for example:
> > > > > > > > > 
> > > > > > > > > pci = ['08:04.1,permissive=1,seize=1']
> > > > > > > > > 
> > > > > > > > > The 'seize=1' would indicate that if bdf 08:04.1 is not
> > > > > > > > > already
> > > > > > > > > assignable, that xl should try to make is assignable.
> > > > > > > > > 
> > > > > > > > > The problem here is that this would need to be parsed by
> > > > > > > > > xlu_pci_parse_bdf(), which only takes an argumen tof type
> > > > > > > > > libxl_device_pci.
> > > > > > > > > 
> > > > > > > > > Now it seems to me that the right place to do this "seizing"
> > > > > > > > > is in xl,
> > > > > > > > > not inside libxl -- the functions for doing assignment exist
> > > > > > > > > already,
> > > > > > > > > and are simple and straightforward.  But doing it in xl, but
> > > > > > > > > as a
> > > > > > > > > parameter of the "pci" setting, means changing
> > > > > > > > > xlu_pci_parse_bdf() to
> > > > > > > > > pass something else back, which begins to get awkward.
> > > > > > > > > 
> > > > > > > > > So it seems to me we have a couple of options:
> > > > > > > > > 1. Create a new argument, "pci_seize" or something like that,
> > > > > > > > > which
> > > > > > > > > would be processed separately from pci
> > > > > > > > > 2. Change xlu_pci_parse_bdf to take a pointer to an extra
> > > > > > > > > struct, for
> > > > > > > > > arguments directed at xl rather than libxl
> > > > > > > > > 3. Add "seize" to libxl_device_pci, but have it only used by
> > > > > > > > > xl
> > > > > > > > > 4. Add "seize" to libxl_device_pci, and have libxl do the
> > > > > > > > > seizing.
> > > > > > > > > 
> > > > > > > > > Any preference -- or any other ideas?
> > > > > > > > > 
> > > > > > > > >    -George
> > > > > > > > How about a setting in xl.conf of "auto-seize pci devices" ?
> > > > > > > > That way
> > > > > > > > the seizing is entirely part of xl
> > > > > > > Auto-seizing is fairly dangerous; you could easily accidentally
> > > > > > > yank
> > > > > > > out the ethernet card, or even the disk that dom0 is using.  I
> > > > > > > really
> > > > > > > think it should have to be enabled on a device-by-device basis.
> > > > > > > 
> > > > > > > I suppose another option would be to be able to set, in xl.conf, a
> > > > > > > list of auto-seizeable devices.  I don't really like that option
> > > > > > > as
> > > > > > > well, though.  I'd rather be able to keep all the configuration in
> > > > > > > one
> > > > > > > place.
> > > > > > > 
> > > > > > >   -George
> > > > > > Or a slight less extreme version.
> > > > > > 
> > > > > > If xl sees that it would need seize a device, it could ask "You are
> > > > > > trying to create a domain with device $FOO.  Would you like to seize
> > > > > > it
> > > > > >from dom0 ?"
> > > > > 
> > > > > That won't work for driver domains, as we want it all to happen
> > > > > automatically when the host is booting. :-)
> > > > 
> > > > The high-level goal is that we want to put the network devices with a
> > > > network backend and storage devices with storage backend. Ignorning
> > > > that for network devices you might want seperate backends for each
> > > > device (say one backend for Wireless, one for Ethernet, etc).
> > > > 
> > > > Perhaps the logic ought to do grouping - so you say:
> > > >   a) "backends:all-network" (which would created one backend with all of
> > > > the
> > > >     wireless, ethernet, etc PCI devices), or
> > > >   b) "backends:all-network,seperate-storage", which  create one backend
> > > > with
> > > >    all of the wireless, ethernet in one backend; and one backend domain
> > > > for each
> > > >    storage device?
> > > > 
> > > > Naturally the user gets to chose which grouping they would like?
> > > 
> > > We seem to be talking about different things.  You seem to be
> > > talking about automatically starting some pre-made VMs and assigning
> > > devices and backends to them?  But I'm not really sure.
> > 
> > I am trying to look at it from a high perspective to see whether we can
> > make this automated for 99% of people out of the box. Hence the
> > idea of grouping. And yes to '..assigning devices and backends to them'.
> > > 
> > > I was assuming that the user was going to be installing and
> > > configuring their own driver domains.  The user already has to
> > > specify "pci=['$BDF']" in their config file to get specific devices
> > > passed through -- this would just be making it easy to have the
> > > device assigned to pciback as well.
> > 
> > I think the technical bits what libxl is doing and yanking devices
> > around is driven either by the admin or a policy. If the policy
> > is this idea of grouping (that is a terrible name now that I think
> > of it), then perhaps we should think how to make that work and then
> > the details (such as this automatic yanking of devices to pci-back)
> > can be filled in.
> > 
> > 
> > > 
> > > I suspect that a lot of people will want to have one network card
> > > assigned to domain 0 as a "management network", and only have other
> > > devices assigned to driver domains.  I think that having one device
> > > per domain is probably the best recommendation; although we
> > > obviously want to support someone who wants a single "manage all the
> > > devices" domain, we should assume that people are going to have one
> > > device per driver domain.
> > 
> > I don't know. My feeble idea was that we would have at minimum _two_
> > guests on bootup. One is a control one that has no devices - but is
> > the one that launches the guests.
> > 
> > Then there is the dom1 which would have all (or some) of the storage
> > and network devices plugged in along with the backends. Then a dom2
> > which would be the old-style-dom0 - so it would have the graphic card
> > and the rest of the PCI devices.
> > 
> > In other words, when I boot I would have two tiny domains launch
> > right before "old-style-dom0" is started. But I am getting in specifics
> > here.
> > 
> > Perhaps you could explain to me how you envisioned how the device
> > driver domains idea would work? How would you want it to work on your
> > laptop?
> > 
> > Or are we right now just thinking of the small pieces of making the
> > code be able to yank the devices around and assign them?
> 
> I was thinking for now just making the "manually configure it" case easier.  I
> decided to switch one of my test boxen to using a network driver domain by
> default, and although the core is there, there are a bunch of things that are
> unnecessarily crufty.
> 
> I do agree that long term it would be nice to make it easy to make driver
> domains the default, but that's not what I had in mind for this conversation.
> :-)
> 
> The hard part for making it really automated, it seems to me, comes from two
> things.  O
> 
> One, you have to make sure your driver domain has the appropriate hardware
> drivers for your system as well.  We don't want to be in the business of
> maintaining a distro; most people will probably want the driver domain to be
> from the same distro they're using for dom0, which means that setting up such
> a domain will need to be done differently on a distro-by-distro basis.

Can we just use the same kernel used in dom0 as a starting point? That
way we would be sure that has everything we need.


> Two, you have the configuration problem.  In Debian, for instance, if you
> wanted to switch a device from being owned by dom0 to being in a driver
> domain, you'd have to:
> * Copy over the udev rules recognizing the mac address, so it got the same
> ethN
> * copy over the eth and bridge info from dom0's /etc/network/interfaces into
> the guest /etc/network/interfaces
> 
> I'm not sure exactly what you have to do in Fedora, but I bet it's something
> similar.
> 
> It might be nice to work with distros to make the process of making driver
> domains / stub domains easier, and to make it easy to configure driver domain
> networking options from the distro's network scripts; but that's kind of
> another level of functionality.

We could host some example per-distro scripts on the xen-unstable
repository to show distros how to setup driver domains by default.  Then
we can go to Fedora, Ubuntu, etc and tell them: "look, if you start this
at boot you get driver domains!".


> I think first things first: make manually-set-up driver domains actually easy
> to use.

Right

Re: RFC: Automatically making a PCI device assignable in the config file

[Stefano Stabellini]

On Tue, 9 Jul 2013, Konrad Rzeszutek Wilk wrote:
> > I suspect that a lot of people will want to have one network card
> > assigned to domain 0 as a "management network", and only have other
> > devices assigned to driver domains.  I think that having one device
> > per domain is probably the best recommendation; although we
> > obviously want to support someone who wants a single "manage all the
> > devices" domain, we should assume that people are going to have one
> > device per driver domain.
> 
> I don't know. My feeble idea was that we would have at minimum _two_
> guests on bootup. One is a control one that has no devices - but is
> the one that launches the guests.

So this would be "dom0".


> Then there is the dom1 which would have all (or some) of the storage
> and network devices plugged in along with the backends. Then a dom2
> which would be the old-style-dom0 - so it would have the graphic card
> and the rest of the PCI devices.

If "dom2" has the graphic card, then it's the domain the user is
interacting with. Therefore the user would be starting domains by using
a tool in "dom2" then it would need to communicate with "dom0" to do
the actual domain creation.

I am not arguing against this solution, but this sounds very much like a
"stage 2" thing to have.
I would keep "dom0" and "dom2" collapsed in the same domain for the
moment.

Re: RFC: Automatically making a PCI device assignable in the config file

[Ian Jackson]

George Dunlap writes ("RFC: Automatically making a PCI device assignable in the config file"):
> I've been doing some work to try to make driver domains easier to set
> up and use.  At the moment, in order to pass a device through to a
> guest, you first need to assign it to pciback.  This involves doing
> one of three things:
> * Running xl pci-assignable-add for the device
> * Specifying the device to be grabbed on the dom0 Linux command-line
> * Doing some hackery in /etc/modules.d
>
> None of these are very satisfying.  What I think would be better is if
> there was a way to specify in the guest config file, "If device X is
> not assignable, try to make it assignable".  That way you can have a
> driver domain grab the appropriate device just by running "xl create
> domnet"; and once we have the xendomains script up and running with
> xl, you can simply configure your domnet appropriately, and then put
> it in /etc/xen/auto, to be started automatically on boot.
> 
> My initial idea was to add a parameter to the pci argument in the
> config file; for example:
> 
> pci = ['08:04.1,permissive=1,seize=1']
> 
> The 'seize=1' would indicate that if bdf 08:04.1 is not already
> assignable, that xl should try to make is assignable.

I think it's a design error that this isn't done automatically by
default.

It would be nice if there was a safety check that the device isn't
currently in use by dom0, but I don't think it's essential.  We could
just put a note in the docs saying "if you specify your dom0 nic it
will go away, duh" or something.

And I think this should be done by libxl, under the control of the
usual kinds of xl configuration.

Ian.

Re: RFC: Automatically making a PCI device assignable in the config file

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
> Auto-seizing is fairly dangerous; you could easily accidentally yank out 
> the ethernet card, or even the disk that dom0 is using.  I really think 
> it should have to be enabled on a device-by-device basis.

I don't think this makes any sense.

In practice you are saying that in order to avoid mistakes, the local
admin must provide the BDF of the device to be passed through in two
separate places.

But that doesn't actually help.  If this is all properly documented
and so forth (ie, if the admin has a smooth experience and doesn't
trip over having dailed to "seize" the device), they will just
cut-and-paste the same value into both places in the config.

They will also mutter under their breath to ask why this is
necessary...

Ian.

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 10/07/13 14:55, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
>> Auto-seizing is fairly dangerous; you could easily accidentally yank out
>> the ethernet card, or even the disk that dom0 is using.  I really think
>> it should have to be enabled on a device-by-device basis.
> I don't think this makes any sense.
>
> In practice you are saying that in order to avoid mistakes, the local
> admin must provide the BDF of the device to be passed through in two
> separate places.

That's not what I had in mind; what I had in mind was something like this:

pci = [ '08:04.1,seize=1' ]

Or alternately:

xl pci-attach h0 08:04.1,seize=1

One could also imagine having something in xl.conf like the following:

pci.autoseize = [ '08:04.1','01:00.0' ]

In this case you wouldn't be simply copy and pasting, because you'd 
probably have different domains handling each device.  But in any case, 
this was just exploring the alternatives -- I don't think that's the 
best thing to do.

> But that doesn't actually help.  If this is all properly documented
> and so forth (ie, if the admin has a smooth experience and doesn't
> trip over having dailed to "seize" the device), they will just
> cut-and-paste the same value into both places in the config.
>
> They will also mutter under their breath to ask why this is
> necessary...

If someone can accidentally type "xl pci-attach 08:04.0" instead of "xl 
pci-attach 08.04.1" and suddenly completely lose their network 
connectivity (or yank their hard drive out), then I think they'll 
appreciate it.

In any case, at the moment it's much worse: you have to either 
scriptwise run "xl pci-assignable-add" a device, or add an exclusion on 
the Linux commandline in grub.  So far I'm the only person I know who 
has complained about it. :-)

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 10/07/13 14:53, Ian Jackson wrote:
> George Dunlap writes ("RFC: Automatically making a PCI device assignable in the config file"):
>> I've been doing some work to try to make driver domains easier to set
>> up and use.  At the moment, in order to pass a device through to a
>> guest, you first need to assign it to pciback.  This involves doing
>> one of three things:
>> * Running xl pci-assignable-add for the device
>> * Specifying the device to be grabbed on the dom0 Linux command-line
>> * Doing some hackery in /etc/modules.d
>>
>> None of these are very satisfying.  What I think would be better is if
>> there was a way to specify in the guest config file, "If device X is
>> not assignable, try to make it assignable".  That way you can have a
>> driver domain grab the appropriate device just by running "xl create
>> domnet"; and once we have the xendomains script up and running with
>> xl, you can simply configure your domnet appropriately, and then put
>> it in /etc/xen/auto, to be started automatically on boot.
>>
>> My initial idea was to add a parameter to the pci argument in the
>> config file; for example:
>>
>> pci = ['08:04.1,permissive=1,seize=1']
>>
>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>> assignable, that xl should try to make is assignable.
> I think it's a design error that this isn't done automatically by
> default.
>
> It would be nice if there was a safety check that the device isn't
> currently in use by dom0, but I don't think it's essential.  We could
> just put a note in the docs saying "if you specify your dom0 nic it
> will go away, duh" or something.

I think it's a really bad interface decision if a simple typo might 
result in you yanking out your disk.

But in any case, if you think libxl_pci_attach() should automatically 
grab the device, that makes things a lot easier.  I'll write up a patch 
to add a "seize" flag and that will be that.

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Gordan Bobic]

 On Wed, 10 Jul 2013 15:45:49 +0100, George Dunlap 
 <george.dunlap@eu.citrix.com> wrote:
> On 10/07/13 14:55, Ian Jackson wrote:
>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a 
>> PCI device assignable in the config file"):
>>> Auto-seizing is fairly dangerous; you could easily accidentally 
>>> yank out
>>> the ethernet card, or even the disk that dom0 is using.  I really 
>>> think
>>> it should have to be enabled on a device-by-device basis.
>> I don't think this makes any sense.
>>
>> In practice you are saying that in order to avoid mistakes, the 
>> local
>> admin must provide the BDF of the device to be passed through in two
>> separate places.
>
> That's not what I had in mind; what I had in mind was something like 
> this:
>
> pci = [ '08:04.1,seize=1' ]
>
> Or alternately:
>
> xl pci-attach h0 08:04.1,seize=1
>
> One could also imagine having something in xl.conf like the 
> following:
>
> pci.autoseize = [ '08:04.1','01:00.0' ]
>
> In this case you wouldn't be simply copy and pasting, because you'd
> probably have different domains handling each device.  But in any
> case, this was just exploring the alternatives -- I don't think 
> that's
> the best thing to do.
>
>> But that doesn't actually help.  If this is all properly documented
>> and so forth (ie, if the admin has a smooth experience and doesn't
>> trip over having dailed to "seize" the device), they will just
>> cut-and-paste the same value into both places in the config.
>>
>> They will also mutter under their breath to ask why this is
>> necessary...
>
> If someone can accidentally type "xl pci-attach 08:04.0" instead of
> "xl pci-attach 08.04.1" and suddenly completely lose their network
> connectivity (or yank their hard drive out), then I think they'll
> appreciate it.
>
> In any case, at the moment it's much worse: you have to either
> scriptwise run "xl pci-assignable-add" a device, or add an exclusion
> on the Linux commandline in grub.  So far I'm the only person I know
> who has complained about it. :-)

 This would be an awesome feature, but success can be mixed. My
 experience is that Nvidia drivers (as one example, I'm sure there
 are others) don't handle device detaching very gracefully. ACS
 might help but I have no means of testing
 that at the moment. So ultimately, you still have to either
 exclude at least some devices from grub (if
 xen-pciback is built in) or in modprobe.d (attach device to
 pciback in driver pre-install). I'm not sure what can be
 done about this considering we are talking about 3rd
 party binary drivers. :(

 Gordan

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 10/07/13 16:12, Gordan Bobic wrote:
> On Wed, 10 Jul 2013 15:45:49 +0100, George Dunlap 
> <george.dunlap@eu.citrix.com> wrote:
>> On 10/07/13 14:55, Ian Jackson wrote:
>>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a 
>>> PCI device assignable in the config file"):
>>>> Auto-seizing is fairly dangerous; you could easily accidentally 
>>>> yank out
>>>> the ethernet card, or even the disk that dom0 is using.  I really 
>>>> think
>>>> it should have to be enabled on a device-by-device basis.
>>> I don't think this makes any sense.
>>>
>>> In practice you are saying that in order to avoid mistakes, the local
>>> admin must provide the BDF of the device to be passed through in two
>>> separate places.
>>
>> That's not what I had in mind; what I had in mind was something like 
>> this:
>>
>> pci = [ '08:04.1,seize=1' ]
>>
>> Or alternately:
>>
>> xl pci-attach h0 08:04.1,seize=1
>>
>> One could also imagine having something in xl.conf like the following:
>>
>> pci.autoseize = [ '08:04.1','01:00.0' ]
>>
>> In this case you wouldn't be simply copy and pasting, because you'd
>> probably have different domains handling each device.  But in any
>> case, this was just exploring the alternatives -- I don't think that's
>> the best thing to do.
>>
>>> But that doesn't actually help.  If this is all properly documented
>>> and so forth (ie, if the admin has a smooth experience and doesn't
>>> trip over having dailed to "seize" the device), they will just
>>> cut-and-paste the same value into both places in the config.
>>>
>>> They will also mutter under their breath to ask why this is
>>> necessary...
>>
>> If someone can accidentally type "xl pci-attach 08:04.0" instead of
>> "xl pci-attach 08.04.1" and suddenly completely lose their network
>> connectivity (or yank their hard drive out), then I think they'll
>> appreciate it.
>>
>> In any case, at the moment it's much worse: you have to either
>> scriptwise run "xl pci-assignable-add" a device, or add an exclusion
>> on the Linux commandline in grub.  So far I'm the only person I know
>> who has complained about it. :-)
>
> This would be an awesome feature, but success can be mixed. My
> experience is that Nvidia drivers (as one example, I'm sure there
> are others) don't handle device detaching very gracefully. ACS
> might help but I have no means of testing
> that at the moment. So ultimately, you still have to either
> exclude at least some devices from grub (if
> xen-pciback is built in) or in modprobe.d (attach device to
> pciback in driver pre-install). I'm not sure what can be
> done about this considering we are talking about 3rd
> party binary drivers. :(

Yes, unfortunately we'll always have to have the fallback for devices 
that don't play well with being re-assigned.  But I my very limited 
experience with network cards gives me some hope that at least for many 
devices for which driver domains might be in use, the auto-reassign 
thing will be useful in a number of cases.

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Gordan Bobic]

 On Wed, 10 Jul 2013 16:29:33 +0100, George Dunlap 
 <george.dunlap@eu.citrix.com> wrote:
> On 10/07/13 16:12, Gordan Bobic wrote:
>> On Wed, 10 Jul 2013 15:45:49 +0100, George Dunlap 
>> <george.dunlap@eu.citrix.com> wrote:
>>> On 10/07/13 14:55, Ian Jackson wrote:
>>>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a 
>>>> PCI device assignable in the config file"):
>>>>> Auto-seizing is fairly dangerous; you could easily accidentally 
>>>>> yank out
>>>>> the ethernet card, or even the disk that dom0 is using.  I really 
>>>>> think
>>>>> it should have to be enabled on a device-by-device basis.
>>>> I don't think this makes any sense.
>>>>
>>>> In practice you are saying that in order to avoid mistakes, the 
>>>> local
>>>> admin must provide the BDF of the device to be passed through in 
>>>> two
>>>> separate places.
>>>
>>> That's not what I had in mind; what I had in mind was something 
>>> like this:
>>>
>>> pci = [ '08:04.1,seize=1' ]
>>>
>>> Or alternately:
>>>
>>> xl pci-attach h0 08:04.1,seize=1
>>>
>>> One could also imagine having something in xl.conf like the 
>>> following:
>>>
>>> pci.autoseize = [ '08:04.1','01:00.0' ]
>>>
>>> In this case you wouldn't be simply copy and pasting, because you'd
>>> probably have different domains handling each device.  But in any
>>> case, this was just exploring the alternatives -- I don't think 
>>> that's
>>> the best thing to do.
>>>
>>>> But that doesn't actually help.  If this is all properly 
>>>> documented
>>>> and so forth (ie, if the admin has a smooth experience and doesn't
>>>> trip over having dailed to "seize" the device), they will just
>>>> cut-and-paste the same value into both places in the config.
>>>>
>>>> They will also mutter under their breath to ask why this is
>>>> necessary...
>>>
>>> If someone can accidentally type "xl pci-attach 08:04.0" instead of
>>> "xl pci-attach 08.04.1" and suddenly completely lose their network
>>> connectivity (or yank their hard drive out), then I think they'll
>>> appreciate it.
>>>
>>> In any case, at the moment it's much worse: you have to either
>>> scriptwise run "xl pci-assignable-add" a device, or add an 
>>> exclusion
>>> on the Linux commandline in grub.  So far I'm the only person I 
>>> know
>>> who has complained about it. :-)
>>
>> This would be an awesome feature, but success can be mixed. My
>> experience is that Nvidia drivers (as one example, I'm sure there
>> are others) don't handle device detaching very gracefully. ACS
>> might help but I have no means of testing
>> that at the moment. So ultimately, you still have to either
>> exclude at least some devices from grub (if
>> xen-pciback is built in) or in modprobe.d (attach device to
>> pciback in driver pre-install). I'm not sure what can be
>> done about this considering we are talking about 3rd
>> party binary drivers. :(
>
> Yes, unfortunately we'll always have to have the fallback for devices
> that don't play well with being re-assigned.  But I my very limited
> experience with network cards gives me some hope that at least for
> many devices for which driver domains might be in use, the
> auto-reassign thing will be useful in a number of cases.

 What about having a feature that upon parsing the config file modifies
 the grub.conf or modprobe.d/xen-pciback.conf (depending on whether
 xen-pciback is module or built in) and adds the required entries
 for the device(s)?

 Gordan

Re: RFC: Automatically making a PCI device assignable in the config file

[David Vrabel]

On 10/07/13 15:48, George Dunlap wrote:
> On 10/07/13 14:53, Ian Jackson wrote:
>> George Dunlap writes ("RFC: Automatically making a PCI device
>> assignable in the config file"):
>>> I've been doing some work to try to make driver domains easier to set
>>> up and use.  At the moment, in order to pass a device through to a
>>> guest, you first need to assign it to pciback.  This involves doing
>>> one of three things:
>>> * Running xl pci-assignable-add for the device
>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>> * Doing some hackery in /etc/modules.d
>>>
>>> None of these are very satisfying.  What I think would be better is if
>>> there was a way to specify in the guest config file, "If device X is
>>> not assignable, try to make it assignable".  That way you can have a
>>> driver domain grab the appropriate device just by running "xl create
>>> domnet"; and once we have the xendomains script up and running with
>>> xl, you can simply configure your domnet appropriately, and then put
>>> it in /etc/xen/auto, to be started automatically on boot.
>>>
>>> My initial idea was to add a parameter to the pci argument in the
>>> config file; for example:
>>>
>>> pci = ['08:04.1,permissive=1,seize=1']
>>>
>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>> assignable, that xl should try to make is assignable.
>> I think it's a design error that this isn't done automatically by
>> default.
>>
>> It would be nice if there was a safety check that the device isn't
>> currently in use by dom0, but I don't think it's essential.  We could
>> just put a note in the docs saying "if you specify your dom0 nic it
>> will go away, duh" or something.
> 
> I think it's a really bad interface decision if a simple typo might
> result in you yanking out your disk.

I don't think this proposal really helps with avoiding this.  I think
most people will end up always adding 'seize=1' because to avoid having
to do so means altering config files elsewhere and rebooting.

David

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel <david.vrabel@citrix.com> wrote:
> On 10/07/13 15:48, George Dunlap wrote:
>> On 10/07/13 14:53, Ian Jackson wrote:
>>> George Dunlap writes ("RFC: Automatically making a PCI device
>>> assignable in the config file"):
>>>> I've been doing some work to try to make driver domains easier to set
>>>> up and use.  At the moment, in order to pass a device through to a
>>>> guest, you first need to assign it to pciback.  This involves doing
>>>> one of three things:
>>>> * Running xl pci-assignable-add for the device
>>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>>> * Doing some hackery in /etc/modules.d
>>>>
>>>> None of these are very satisfying.  What I think would be better is if
>>>> there was a way to specify in the guest config file, "If device X is
>>>> not assignable, try to make it assignable".  That way you can have a
>>>> driver domain grab the appropriate device just by running "xl create
>>>> domnet"; and once we have the xendomains script up and running with
>>>> xl, you can simply configure your domnet appropriately, and then put
>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>
>>>> My initial idea was to add a parameter to the pci argument in the
>>>> config file; for example:
>>>>
>>>> pci = ['08:04.1,permissive=1,seize=1']
>>>>
>>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>>> assignable, that xl should try to make is assignable.
>>> I think it's a design error that this isn't done automatically by
>>> default.
>>>
>>> It would be nice if there was a safety check that the device isn't
>>> currently in use by dom0, but I don't think it's essential.  We could
>>> just put a note in the docs saying "if you specify your dom0 nic it
>>> will go away, duh" or something.
>>
>> I think it's a really bad interface decision if a simple typo might
>> result in you yanking out your disk.
>
> I don't think this proposal really helps with avoiding this.  I think
> most people will end up always adding 'seize=1' because to avoid having
> to do so means altering config files elsewhere and rebooting.

I guess what I'm worried about is the fact that we would be changing
things that are now "safe" to things that are not safe.  At the
moment, "xl pci-assignable-add" might yank out a system device if you
make a typo; but it was introduced that way, so people always had to
be careful.  But currently, "pci=[]" and "xl pci-attach" do *not*
behave that way; you have to make the device assignable first.  So you
don't need to be particularly careful.  Adding "seize" at least should
flag up to people that they need to double-check.

If people really object to the extra flag, I can write up a patch
without it, but I'd prefer to have something...

 -George

Re: RFC: Automatically making a PCI device assignable in the config file

[David Vrabel]

On 12/07/13 10:36, George Dunlap wrote:
> On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel <david.vrabel@citrix.com> wrote:
>> On 10/07/13 15:48, George Dunlap wrote:
>>> On 10/07/13 14:53, Ian Jackson wrote:
>>>> George Dunlap writes ("RFC: Automatically making a PCI device
>>>> assignable in the config file"):
>>>>> I've been doing some work to try to make driver domains easier to set
>>>>> up and use.  At the moment, in order to pass a device through to a
>>>>> guest, you first need to assign it to pciback.  This involves doing
>>>>> one of three things:
>>>>> * Running xl pci-assignable-add for the device
>>>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>>>> * Doing some hackery in /etc/modules.d
>>>>>
>>>>> None of these are very satisfying.  What I think would be better is if
>>>>> there was a way to specify in the guest config file, "If device X is
>>>>> not assignable, try to make it assignable".  That way you can have a
>>>>> driver domain grab the appropriate device just by running "xl create
>>>>> domnet"; and once we have the xendomains script up and running with
>>>>> xl, you can simply configure your domnet appropriately, and then put
>>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>>
>>>>> My initial idea was to add a parameter to the pci argument in the
>>>>> config file; for example:
>>>>>
>>>>> pci = ['08:04.1,permissive=1,seize=1']
>>>>>
>>>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>>>> assignable, that xl should try to make is assignable.
>>>> I think it's a design error that this isn't done automatically by
>>>> default.
>>>>
>>>> It would be nice if there was a safety check that the device isn't
>>>> currently in use by dom0, but I don't think it's essential.  We could
>>>> just put a note in the docs saying "if you specify your dom0 nic it
>>>> will go away, duh" or something.
>>>
>>> I think it's a really bad interface decision if a simple typo might
>>> result in you yanking out your disk.
>>
>> I don't think this proposal really helps with avoiding this.  I think
>> most people will end up always adding 'seize=1' because to avoid having
>> to do so means altering config files elsewhere and rebooting.
> 
> I guess what I'm worried about is the fact that we would be changing
> things that are now "safe" to things that are not safe.  At the
> moment, "xl pci-assignable-add" might yank out a system device if you
> make a typo; but it was introduced that way, so people always had to
> be careful.  But currently, "pci=[]" and "xl pci-attach" do *not*
> behave that way; you have to make the device assignable first.  So you
> don't need to be particularly careful.  Adding "seize" at least should
> flag up to people that they need to double-check.

You could see if you can determine if a PCI device provides either a
block device or a network device and then check whether the block device
is mounted or the network device has a IP address configured.

These checks should probably be handled by a script that xl calls.

David

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 12/07/13 10:55, David Vrabel wrote:
> On 12/07/13 10:36, George Dunlap wrote:
>> On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel <david.vrabel@citrix.com> wrote:
>>> On 10/07/13 15:48, George Dunlap wrote:
>>>> On 10/07/13 14:53, Ian Jackson wrote:
>>>>> George Dunlap writes ("RFC: Automatically making a PCI device
>>>>> assignable in the config file"):
>>>>>> I've been doing some work to try to make driver domains easier to set
>>>>>> up and use.  At the moment, in order to pass a device through to a
>>>>>> guest, you first need to assign it to pciback.  This involves doing
>>>>>> one of three things:
>>>>>> * Running xl pci-assignable-add for the device
>>>>>> * Specifying the device to be grabbed on the dom0 Linux command-line
>>>>>> * Doing some hackery in /etc/modules.d
>>>>>>
>>>>>> None of these are very satisfying.  What I think would be better is if
>>>>>> there was a way to specify in the guest config file, "If device X is
>>>>>> not assignable, try to make it assignable".  That way you can have a
>>>>>> driver domain grab the appropriate device just by running "xl create
>>>>>> domnet"; and once we have the xendomains script up and running with
>>>>>> xl, you can simply configure your domnet appropriately, and then put
>>>>>> it in /etc/xen/auto, to be started automatically on boot.
>>>>>>
>>>>>> My initial idea was to add a parameter to the pci argument in the
>>>>>> config file; for example:
>>>>>>
>>>>>> pci = ['08:04.1,permissive=1,seize=1']
>>>>>>
>>>>>> The 'seize=1' would indicate that if bdf 08:04.1 is not already
>>>>>> assignable, that xl should try to make is assignable.
>>>>> I think it's a design error that this isn't done automatically by
>>>>> default.
>>>>>
>>>>> It would be nice if there was a safety check that the device isn't
>>>>> currently in use by dom0, but I don't think it's essential.  We could
>>>>> just put a note in the docs saying "if you specify your dom0 nic it
>>>>> will go away, duh" or something.
>>>> I think it's a really bad interface decision if a simple typo might
>>>> result in you yanking out your disk.
>>> I don't think this proposal really helps with avoiding this.  I think
>>> most people will end up always adding 'seize=1' because to avoid having
>>> to do so means altering config files elsewhere and rebooting.
>> I guess what I'm worried about is the fact that we would be changing
>> things that are now "safe" to things that are not safe.  At the
>> moment, "xl pci-assignable-add" might yank out a system device if you
>> make a typo; but it was introduced that way, so people always had to
>> be careful.  But currently, "pci=[]" and "xl pci-attach" do *not*
>> behave that way; you have to make the device assignable first.  So you
>> don't need to be particularly careful.  Adding "seize" at least should
>> flag up to people that they need to double-check.
> You could see if you can determine if a PCI device provides either a
> block device or a network device and then check whether the block device
> is mounted or the network device has a IP address configured.
>
> These checks should probably be handled by a script that xl calls.

I think we'd be opening a can of worms.  For example, if you do ifconfig 
on an eth device that's being used by a bridge, it comes up empty; so 
the script would have to be smart enough to find the associated bridge 
as well.  That seems to be the case both for Linux bridging and 
openvswitch.  This might become arbitrarily complicated for various 
devices: what about a USB root switch with an external hub that has a 
mounted disk?

I think the only maintainable option is to have it just done with no 
checks; the only question is if we should require an option to enable 
the auto-assigning or having it happen by default.

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
> On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel <david.vrabel@citrix.com> wrot> > I don't think this proposal really helps with avoiding this.  I think
> > most people will end up always adding 'seize=1' because to avoid having
> > to do so means altering config files elsewhere and rebooting.
> 
> I guess what I'm worried about is the fact that we would be changing
> things that are now "safe" to things that are not safe.  At the
> moment, "xl pci-assignable-add" might yank out a system device if you
> make a typo; but it was introduced that way, so people always had to
> be careful.  But currently, "pci=[]" and "xl pci-attach" do *not*
> behave that way; you have to make the device assignable first.  So you
> don't need to be particularly careful.  Adding "seize" at least should
> flag up to people that they need to double-check.
> 
> If people really object to the extra flag, I can write up a patch
> without it, but I'd prefer to have something...

I think at the very least there should be the ability to have it as a
global config option.

Ian.

Re: RFC: Automatically making a PCI device assignable in the config file

[Konrad Rzeszutek Wilk]

On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
> > On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel <david.vrabel@citrix.com> wrot> > I don't think this proposal really helps with avoiding this.  I think
> > > most people will end up always adding 'seize=1' because to avoid having
> > > to do so means altering config files elsewhere and rebooting.
> > 
> > I guess what I'm worried about is the fact that we would be changing
> > things that are now "safe" to things that are not safe.  At the
> > moment, "xl pci-assignable-add" might yank out a system device if you
> > make a typo; but it was introduced that way, so people always had to
> > be careful.  But currently, "pci=[]" and "xl pci-attach" do *not*
> > behave that way; you have to make the device assignable first.  So you
> > don't need to be particularly careful.  Adding "seize" at least should
> > flag up to people that they need to double-check.
> > 
> > If people really object to the extra flag, I can write up a patch
> > without it, but I'd prefer to have something...
> 
> I think at the very least there should be the ability to have it as a
> global config option.

Like the seatbelt option or perhaps a better name 'expert' in the /etc/xl.conf?

> 
> Ian.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: RFC: Automatically making a PCI device assignable in the config file

[Ian Jackson]

Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
> > I think at the very least there should be the ability to have it as a
> > global config option.
> 
> Like the seatbelt option or perhaps a better name 'expert' in the /etc/xl.conf?

I was thinking "default.pci.seize" or something.

Ian.

Re: RFC: Automatically making a PCI device assignable in the config file

[Sander Eikelenboom]

Friday, July 12, 2013, 3:48:34 PM, you wrote:

> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
>> George Dunlap writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
>> > On Thu, Jul 11, 2013 at 12:35 PM, David Vrabel <david.vrabel@citrix.com> wrot> > I don't think this proposal really helps with avoiding this.  I think
>> > > most people will end up always adding 'seize=1' because to avoid having
>> > > to do so means altering config files elsewhere and rebooting.
>> > 
>> > I guess what I'm worried about is the fact that we would be changing
>> > things that are now "safe" to things that are not safe.  At the
>> > moment, "xl pci-assignable-add" might yank out a system device if you
>> > make a typo; but it was introduced that way, so people always had to
>> > be careful.  But currently, "pci=[]" and "xl pci-attach" do *not*
>> > behave that way; you have to make the device assignable first.  So you
>> > don't need to be particularly careful.  Adding "seize" at least should
>> > flag up to people that they need to double-check.
>> > 
>> > If people really object to the extra flag, I can write up a patch
>> > without it, but I'd prefer to have something...
>> 
>> I think at the very least there should be the ability to have it as a
>> global config option.

> Like the seatbelt option or perhaps a better name 'expert' in the /etc/xl.conf?

Another way could be to make it possible to run a script onbefore/onafter any hotplug event.
That would be flexible as well as inline with what currently is done for networking/vifs.

--
Sander

>> 
>> Ian.
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

Re: RFC: Automatically making a PCI device assignable in the config file

[Konrad Rzeszutek Wilk]

On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson wrote:
> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
> > On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
> > > I think at the very least there should be the ability to have it as a
> > > global config option.
> > 
> > Like the seatbelt option or perhaps a better name 'expert' in the /etc/xl.conf?
> 
> I was thinking "default.pci.seize" or something.

Why not combine this with the other - the one where you can call
'xl vpcu-set <guest> <huge number of VCPUs>' where the number of
VCPUs is greater than PCPUs. And if the 'expert' option is turned
on it will let you. If it is not, it will notify you about this
being a bad idea.

Which reminds me, I need to send a patch for that..
> 
> Ian.

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 12/07/13 16:01, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson wrote:
>> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
>>> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
>>>> I think at the very least there should be the ability to have it as a
>>>> global config option.
>>> Like the seatbelt option or perhaps a better name 'expert' in the /etc/xl.conf?
>> I was thinking "default.pci.seize" or something.
> Why not combine this with the other - the one where you can call
> 'xl vpcu-set <guest> <huge number of VCPUs>' where the number of
> VCPUs is greater than PCPUs. And if the 'expert' option is turned
> on it will let you. If it is not, it will notify you about this
> being a bad idea.

...because you may want to have one and not the other?

  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[Konrad Rzeszutek Wilk]

On Fri, Jul 12, 2013 at 04:09:29PM +0100, George Dunlap wrote:
> On 12/07/13 16:01, Konrad Rzeszutek Wilk wrote:
> >On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson wrote:
> >>Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
> >>>On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
> >>>>I think at the very least there should be the ability to have it as a
> >>>>global config option.
> >>>Like the seatbelt option or perhaps a better name 'expert' in the /etc/xl.conf?
> >>I was thinking "default.pci.seize" or something.
> >Why not combine this with the other - the one where you can call
> >'xl vpcu-set <guest> <huge number of VCPUs>' where the number of
> >VCPUs is greater than PCPUs. And if the 'expert' option is turned
> >on it will let you. If it is not, it will notify you about this
> >being a bad idea.
> 
> ...because you may want to have one and not the other?

Seatbelts are seatbelts. They have many parts - but nonethless
at the end of the day you describe them by just one word.

I am not a big fan of many knobs - even thought I did work at some
point for IBM which was known for its knob-for-everything products.

> 
>  -George

Re: RFC: Automatically making a PCI device assignable in the config file

[George Dunlap]

On 12/07/13 17:02, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 12, 2013 at 04:09:29PM +0100, George Dunlap wrote:
>> On 12/07/13 16:01, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Jul 12, 2013 at 03:43:32PM +0100, Ian Jackson wrote:
>>>> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] RFC: Automatically making a PCI device assignable in the config file"):
>>>>> On Fri, Jul 12, 2013 at 02:10:41PM +0100, Ian Jackson wrote:
>>>>>> I think at the very least there should be the ability to have it as a
>>>>>> global config option.
>>>>> Like the seatbelt option or perhaps a better name 'expert' in the /etc/xl.conf?
>>>> I was thinking "default.pci.seize" or something.
>>> Why not combine this with the other - the one where you can call
>>> 'xl vpcu-set <guest> <huge number of VCPUs>' where the number of
>>> VCPUs is greater than PCPUs. And if the 'expert' option is turned
>>> on it will let you. If it is not, it will notify you about this
>>> being a bad idea.
>> ...because you may want to have one and not the other?
> Seatbelts are seatbelts. They have many parts - but nonethless
> at the end of the day you describe them by just one word.
>
> I am not a big fan of many knobs - even thought I did work at some
> point for IBM which was known for its knob-for-everything products.

Well remember that my main concern here is with the *changing* of 
behavior, not the *existence* of behavior.  (i.e., xl pci-assignable-add 
already behaves in the way in which xl pci-attach would behave with the 
seatbelt off).  If we had already had a single "expert" option in 
xl.conf, then we would be in nearly the same situation -- add xl 
pci-attach to the "expert" option and have it change from safe to unsafe 
without any warning, or add yet another parameter.

  -George