[virtio-dev] RFC: Use of bridge devices to store pairing information...

[virtio-dev] RFC: Use of bridge devices to store pairing information...

[Venu Busireddy]

I looked at the discussion in the threads [1] and [2], where it was
suggested placing the passthrough device behind one bridge, and the virtio
device behind another bridge, and storing in those bridges' configuration
space some unique identifier that can be used to pair the two devices.

After some discussions with Si-Wei Liu and others, we believe that the
following scheme may be a viable approach. Please take a look at this
proposal and provide your thoughts.

1. Enhance the QEMU CLI to include a "group_id" option to the bridge
   devices for Q35 as well as i440FX models. I have already made changes
   for the Q35 model (ioh3420 bridge).

2. When the guest is created, the operator creates two bridge devices
   (for example, using '-device ioh3420,group_id="string"'), and specifies
   a unique identifier string for both bridges. This identifier can be
   the UUID generated by 'uuidgen' command.

3. QEMU places this unique identifier in the PCI configuration space of
   the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
   not to be confused with a specific organization. Instead, the vendor
   of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).

4. The operator places the passthrough device behind one of the bridges,
   and the virtio device behind the other bridge.

5. Patch 4 in patch series [3] should be modified to use the unique
   identifier string stored in the bridges' configuration space instead
   of the MAC address for pairing the devices.

If it is desirable to create only one bridge instead of two (to conserve
the  number of devices in the system), then the passthrough device can be
attached to that single bridge (with the identifier), and the identifier
for the virtio device can be stored in the virtio device's configuration
space itself. To do that, we need to update the virtio specification,
and I have sent a proposal [4] to the OASIS team to update the virtio
specification. If that proposal is accepted, then we can modify QEMU to
use the virtio device's configuration space instead of the second bridge
to store the unique identifier.

Thank you for sparing the time.

Venu

[1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
[2] https://www.spinics.net/lists/netdev/msg499011.html
[3] https://patchwork.ozlabs.org/cover/920005/
[4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Samudrala, Sridhar]

On 5/31/2018 6:28 PM, Venu Busireddy wrote:
> I looked at the discussion in the threads [1] and [2], where it was
> suggested placing the passthrough device behind one bridge, and the virtio
> device behind another bridge, and storing in those bridges' configuration
> space some unique identifier that can be used to pair the two devices.
>
> After some discussions with Si-Wei Liu and others, we believe that the
> following scheme may be a viable approach. Please take a look at this
> proposal and provide your thoughts.
>
> 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
>     devices for Q35 as well as i440FX models. I have already made changes
>     for the Q35 model (ioh3420 bridge).
>
> 2. When the guest is created, the operator creates two bridge devices
>     (for example, using '-device ioh3420,group_id="string"'), and specifies
>     a unique identifier string for both bridges. This identifier can be
>     the UUID generated by 'uuidgen' command.
>
> 3. QEMU places this unique identifier in the PCI configuration space of
>     the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
>     not to be confused with a specific organization. Instead, the vendor
>     of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
>
> 4. The operator places the passthrough device behind one of the bridges,
>     and the virtio device behind the other bridge.
>
> 5. Patch 4 in patch series [3] should be modified to use the unique
>     identifier string stored in the bridges' configuration space instead
>     of the MAC address for pairing the devices.

This should be an alternate option that allows failover slaves to be registered
based on MAC and ID.


>
> If it is desirable to create only one bridge instead of two (to conserve
> the  number of devices in the system), then the passthrough device can be
> attached to that single bridge (with the identifier), and the identifier
> for the virtio device can be stored in the virtio device's configuration
> space itself. To do that, we need to update the virtio specification,
> and I have sent a proposal [4] to the OASIS team to update the virtio
> specification. If that proposal is accepted, then we can modify QEMU to
> use the virtio device's configuration space instead of the second bridge
> to store the unique identifier.

I think one bridge solution is much cleaner than having to use 2 bridges.


>
> Thank you for sparing the time.
>
> Venu
>
> [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> [2] https://www.spinics.net/lists/netdev/msg499011.html
> [3] https://patchwork.ozlabs.org/cover/920005/
> [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RFC: Use of bridge devices to store pairing information...

[Michael S. Tsirkin]

On Thu, May 31, 2018 at 08:28:32PM -0500, Venu Busireddy wrote:
> 
> I looked at the discussion in the threads [1] and [2], where it was
> suggested placing the passthrough device behind one bridge, and the virtio
> device behind another bridge, and storing in those bridges' configuration
> space some unique identifier that can be used to pair the two devices.
> 
> After some discussions with Si-Wei Liu and others, we believe that the
> following scheme may be a viable approach.

Thanks for writing this up!  I mostly agree which isn't surprising since
I proposed this scheme originally ;)
https://patchwork.ozlabs.org/patch/901596/#1901665

Some minor comments though:

> Please take a look at this
> proposal and provide your thoughts.
> 
> 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
>    devices for Q35 as well as i440FX models. I have already made changes
>    for the Q35 model (ioh3420 bridge).
> 
> 2. When the guest is created, the operator creates two bridge devices
>    (for example, using '-device ioh3420,group_id="string"'), and specifies
>    a unique identifier string for both bridges. This identifier can be
>    the UUID generated by 'uuidgen' command.

I suggest we limit this to a bridge with QEMU vendor/device ID.

We have pci-bridge and pci-bridge-seat.

This could be similar: pci-bridge-group.
Less of a chance there's a conflict.

Accordingly bridge is the group, not the id :)

> 3. QEMU places this unique identifier in the PCI configuration space of
>    the bridge as Vendor-Specific capability (0x09).

Why not use the standard UUID capability? Accordingly name property
uuid?

> The "Vendor" here is
>    not to be confused with a specific organization. Instead, the vendor
>    of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).

So that has a small chance of conflicting with a vendor
specific capability used by some driver of this bridge.
We are better off with an actual vendor/device ID IMHO.

> 4. The operator places the passthrough device behind one of the bridges,
>    and the virtio device behind the other bridge.
> 
> 5. Patch 4 in patch series [3] should be modified to use the unique
>    identifier string stored in the bridges' configuration space instead
>    of the MAC address for pairing the devices.
> 
> If it is desirable to create only one bridge instead of two (to conserve
> the  number of devices in the system), then the passthrough device can be
> attached to that single bridge (with the identifier), and the identifier
> for the virtio device can be stored in the virtio device's configuration
> space itself. To do that, we need to update the virtio specification,

I think we can use the standard PCI UUID. No need to update the spec
for this.

> and I have sent a proposal [4] to the OASIS team to update the virtio
> specification. If that proposal is accepted, then we can modify QEMU to
> use the virtio device's configuration space instead of the second bridge
> to store the unique identifier.
> 
> Thank you for sparing the time.
> 
> Venu
> 
> [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> [2] https://www.spinics.net/lists/netdev/msg499011.html
> [3] https://patchwork.ozlabs.org/cover/920005/
> [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RFC: Use of bridge devices to store pairing information...

[Michael S. Tsirkin]

On Fri, Jun 01, 2018 at 06:43:41PM +0300, Michael S. Tsirkin wrote:
> On Thu, May 31, 2018 at 08:28:32PM -0500, Venu Busireddy wrote:
> > 
> > I looked at the discussion in the threads [1] and [2], where it was
> > suggested placing the passthrough device behind one bridge, and the virtio
> > device behind another bridge, and storing in those bridges' configuration
> > space some unique identifier that can be used to pair the two devices.
> > 
> > After some discussions with Si-Wei Liu and others, we believe that the
> > following scheme may be a viable approach.
> 
> Thanks for writing this up!  I mostly agree which isn't surprising since
> I proposed this scheme originally ;)
> https://patchwork.ozlabs.org/patch/901596/#1901665
> 
> Some minor comments though:
> 
> > Please take a look at this
> > proposal and provide your thoughts.
> > 
> > 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
> >    devices for Q35 as well as i440FX models. I have already made changes
> >    for the Q35 model (ioh3420 bridge).
> > 
> > 2. When the guest is created, the operator creates two bridge devices
> >    (for example, using '-device ioh3420,group_id="string"'), and specifies
> >    a unique identifier string for both bridges. This identifier can be
> >    the UUID generated by 'uuidgen' command.
> 
> I suggest we limit this to a bridge with QEMU vendor/device ID.
> 
> We have pci-bridge and pci-bridge-seat.
> 
> This could be similar: pci-bridge-group.
> Less of a chance there's a conflict.
> 
> Accordingly bridge is the group, not the id :)
> 
> > 3. QEMU places this unique identifier in the PCI configuration space of
> >    the bridge as Vendor-Specific capability (0x09).
> 
> Why not use the standard UUID capability? Accordingly name property
> uuid?
> 
> > The "Vendor" here is
> >    not to be confused with a specific organization. Instead, the vendor
> >    of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
> 
> So that has a small chance of conflicting with a vendor
> specific capability used by some driver of this bridge.
> We are better off with an actual vendor/device ID IMHO.
> 
> > 4. The operator places the passthrough device behind one of the bridges,
> >    and the virtio device behind the other bridge.
> > 
> > 5. Patch 4 in patch series [3] should be modified to use the unique
> >    identifier string stored in the bridges' configuration space instead
> >    of the MAC address for pairing the devices.
> > 
> > If it is desirable to create only one bridge instead of two (to conserve
> > the  number of devices in the system), then the passthrough device can be
> > attached to that single bridge (with the identifier), and the identifier
> > for the virtio device can be stored in the virtio device's configuration
> > space itself. To do that, we need to update the virtio specification,
> 
> I think we can use the standard PCI UUID. No need to update the spec
> for this.

Thinking some more we might want to say in the spec that
PCI UUID is used for pairing so I guess yes, we do need to
update the spec after all.

> > and I have sent a proposal [4] to the OASIS team to update the virtio
> > specification. If that proposal is accepted, then we can modify QEMU to
> > use the virtio device's configuration space instead of the second bridge
> > to store the unique identifier.
> > 
> > Thank you for sparing the time.
> > 
> > Venu
> > 
> > [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> > [2] https://www.spinics.net/lists/netdev/msg499011.html
> > [3] https://patchwork.ozlabs.org/cover/920005/
> > [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Michael S. Tsirkin]

On Fri, Jun 01, 2018 at 08:26:03AM -0700, Samudrala, Sridhar wrote:
> On 5/31/2018 6:28 PM, Venu Busireddy wrote:
> > I looked at the discussion in the threads [1] and [2], where it was
> > suggested placing the passthrough device behind one bridge, and the virtio
> > device behind another bridge, and storing in those bridges' configuration
> > space some unique identifier that can be used to pair the two devices.
> > 
> > After some discussions with Si-Wei Liu and others, we believe that the
> > following scheme may be a viable approach. Please take a look at this
> > proposal and provide your thoughts.
> > 
> > 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
> >     devices for Q35 as well as i440FX models. I have already made changes
> >     for the Q35 model (ioh3420 bridge).
> > 
> > 2. When the guest is created, the operator creates two bridge devices
> >     (for example, using '-device ioh3420,group_id="string"'), and specifies
> >     a unique identifier string for both bridges. This identifier can be
> >     the UUID generated by 'uuidgen' command.
> > 
> > 3. QEMU places this unique identifier in the PCI configuration space of
> >     the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
> >     not to be confused with a specific organization. Instead, the vendor
> >     of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
> > 
> > 4. The operator places the passthrough device behind one of the bridges,
> >     and the virtio device behind the other bridge.
> > 
> > 5. Patch 4 in patch series [3] should be modified to use the unique
> >     identifier string stored in the bridges' configuration space instead
> >     of the MAC address for pairing the devices.
> 
> This should be an alternate option that allows failover slaves to be registered
> based on MAC and ID.

I wonder whether we ever want to pair based solely on ID and not on MAC.
There used to be devices which randomized VF MAC on each reset but I
think most of them have been changed to get MAC from the PF. Not sure
whether any are left.

If we want the flexibility, we can use a separate feature bit for
matching by ID. If not we can just extend the meaning of the
existing one.

> 
> > 
> > If it is desirable to create only one bridge instead of two (to conserve
> > the  number of devices in the system), then the passthrough device can be
> > attached to that single bridge (with the identifier), and the identifier
> > for the virtio device can be stored in the virtio device's configuration
> > space itself. To do that, we need to update the virtio specification,
> > and I have sent a proposal [4] to the OASIS team to update the virtio
> > specification. If that proposal is accepted, then we can modify QEMU to
> > use the virtio device's configuration space instead of the second bridge
> > to store the unique identifier.
> 
> I think one bridge solution is much cleaner than having to use 2 bridges.
> 

So for one bridge, we'd have a bridge with a special device/vendor ID,
if virtio is behind that, we use that for pairing.


> > 
> > Thank you for sparing the time.
> > 
> > Venu
> > 
> > [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> > [2] https://www.spinics.net/lists/netdev/msg499011.html
> > [3] https://patchwork.ozlabs.org/cover/920005/
> > [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Samudrala, Sridhar]

On 6/1/2018 8:48 AM, Michael S. Tsirkin wrote:
> On Fri, Jun 01, 2018 at 08:26:03AM -0700, Samudrala, Sridhar wrote:
>> On 5/31/2018 6:28 PM, Venu Busireddy wrote:
>>> I looked at the discussion in the threads [1] and [2], where it was
>>> suggested placing the passthrough device behind one bridge, and the virtio
>>> device behind another bridge, and storing in those bridges' configuration
>>> space some unique identifier that can be used to pair the two devices.
>>>
>>> After some discussions with Si-Wei Liu and others, we believe that the
>>> following scheme may be a viable approach. Please take a look at this
>>> proposal and provide your thoughts.
>>>
>>> 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
>>>      devices for Q35 as well as i440FX models. I have already made changes
>>>      for the Q35 model (ioh3420 bridge).
>>>
>>> 2. When the guest is created, the operator creates two bridge devices
>>>      (for example, using '-device ioh3420,group_id="string"'), and specifies
>>>      a unique identifier string for both bridges. This identifier can be
>>>      the UUID generated by 'uuidgen' command.
>>>
>>> 3. QEMU places this unique identifier in the PCI configuration space of
>>>      the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
>>>      not to be confused with a specific organization. Instead, the vendor
>>>      of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
>>>
>>> 4. The operator places the passthrough device behind one of the bridges,
>>>      and the virtio device behind the other bridge.
>>>
>>> 5. Patch 4 in patch series [3] should be modified to use the unique
>>>      identifier string stored in the bridges' configuration space instead
>>>      of the MAC address for pairing the devices.
>> This should be an alternate option that allows failover slaves to be registered
>> based on MAC and ID.
> I wonder whether we ever want to pair based solely on ID and not on MAC.
> There used to be devices which randomized VF MAC on each reset but I
> think most of them have been changed to get MAC from the PF. Not sure
> whether any are left.

Yes. I also don't think we need to support pairing based solely on ID.  It has be based
on MAC OR  MAC+ID combination.

>
> If we want the flexibility, we can use a separate feature bit for
> matching by ID. If not we can just extend the meaning of the
> existing one.

Sure. If we can use 0 ID as MAC based pairing, it may be possible to extend
this without any new feature bit.


>
>>> If it is desirable to create only one bridge instead of two (to conserve
>>> the  number of devices in the system), then the passthrough device can be
>>> attached to that single bridge (with the identifier), and the identifier
>>> for the virtio device can be stored in the virtio device's configuration
>>> space itself. To do that, we need to update the virtio specification,
>>> and I have sent a proposal [4] to the OASIS team to update the virtio
>>> specification. If that proposal is accepted, then we can modify QEMU to
>>> use the virtio device's configuration space instead of the second bridge
>>> to store the unique identifier.
>> I think one bridge solution is much cleaner than having to use 2 bridges.
>>
> So for one bridge, we'd have a bridge with a special device/vendor ID,
> if virtio is behind that, we use that for pairing.
>
>
>>> Thank you for sparing the time.
>>>
>>> Venu
>>>
>>> [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
>>> [2] https://www.spinics.net/lists/netdev/msg499011.html
>>> [3] https://patchwork.ozlabs.org/cover/920005/
>>> [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Michael S. Tsirkin]

On Fri, Jun 01, 2018 at 10:22:59AM -0700, Samudrala, Sridhar wrote:
> 
> 
> On 6/1/2018 8:48 AM, Michael S. Tsirkin wrote:
> > On Fri, Jun 01, 2018 at 08:26:03AM -0700, Samudrala, Sridhar wrote:
> > > On 5/31/2018 6:28 PM, Venu Busireddy wrote:
> > > > I looked at the discussion in the threads [1] and [2], where it was
> > > > suggested placing the passthrough device behind one bridge, and the virtio
> > > > device behind another bridge, and storing in those bridges' configuration
> > > > space some unique identifier that can be used to pair the two devices.
> > > > 
> > > > After some discussions with Si-Wei Liu and others, we believe that the
> > > > following scheme may be a viable approach. Please take a look at this
> > > > proposal and provide your thoughts.
> > > > 
> > > > 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
> > > >      devices for Q35 as well as i440FX models. I have already made changes
> > > >      for the Q35 model (ioh3420 bridge).
> > > > 
> > > > 2. When the guest is created, the operator creates two bridge devices
> > > >      (for example, using '-device ioh3420,group_id="string"'), and specifies
> > > >      a unique identifier string for both bridges. This identifier can be
> > > >      the UUID generated by 'uuidgen' command.
> > > > 
> > > > 3. QEMU places this unique identifier in the PCI configuration space of
> > > >      the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
> > > >      not to be confused with a specific organization. Instead, the vendor
> > > >      of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
> > > > 
> > > > 4. The operator places the passthrough device behind one of the bridges,
> > > >      and the virtio device behind the other bridge.
> > > > 
> > > > 5. Patch 4 in patch series [3] should be modified to use the unique
> > > >      identifier string stored in the bridges' configuration space instead
> > > >      of the MAC address for pairing the devices.
> > > This should be an alternate option that allows failover slaves to be registered
> > > based on MAC and ID.
> > I wonder whether we ever want to pair based solely on ID and not on MAC.
> > There used to be devices which randomized VF MAC on each reset but I
> > think most of them have been changed to get MAC from the PF. Not sure
> > whether any are left.
> 
> Yes. I also don't think we need to support pairing based solely on ID.  It has be based
> on MAC OR  MAC+ID combination.
> 
> > 
> > If we want the flexibility, we can use a separate feature bit for
> > matching by ID. If not we can just extend the meaning of the
> > existing one.
> 
> Sure. If we can use 0 ID as MAC based pairing, it may be possible to extend
> this without any new feature bit.

Not sure what does 0 ID mean. If it's in a capability, you just don't
add the capability.

> 
> > 
> > > > If it is desirable to create only one bridge instead of two (to conserve
> > > > the  number of devices in the system), then the passthrough device can be
> > > > attached to that single bridge (with the identifier), and the identifier
> > > > for the virtio device can be stored in the virtio device's configuration
> > > > space itself. To do that, we need to update the virtio specification,
> > > > and I have sent a proposal [4] to the OASIS team to update the virtio
> > > > specification. If that proposal is accepted, then we can modify QEMU to
> > > > use the virtio device's configuration space instead of the second bridge
> > > > to store the unique identifier.
> > > I think one bridge solution is much cleaner than having to use 2 bridges.
> > > 
> > So for one bridge, we'd have a bridge with a special device/vendor ID,
> > if virtio is behind that, we use that for pairing.
> > 
> > 
> > > > Thank you for sparing the time.
> > > > 
> > > > Venu
> > > > 
> > > > [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> > > > [2] https://www.spinics.net/lists/netdev/msg499011.html
> > > > [3] https://patchwork.ozlabs.org/cover/920005/
> > > > [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
> > > > 
> > > > 
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > > > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Siwei Liu]

On Fri, Jun 1, 2018 at 8:48 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Fri, Jun 01, 2018 at 08:26:03AM -0700, Samudrala, Sridhar wrote:
>> On 5/31/2018 6:28 PM, Venu Busireddy wrote:
>> > I looked at the discussion in the threads [1] and [2], where it was
>> > suggested placing the passthrough device behind one bridge, and the virtio
>> > device behind another bridge, and storing in those bridges' configuration
>> > space some unique identifier that can be used to pair the two devices.
>> >
>> > After some discussions with Si-Wei Liu and others, we believe that the
>> > following scheme may be a viable approach. Please take a look at this
>> > proposal and provide your thoughts.
>> >
>> > 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
>> >     devices for Q35 as well as i440FX models. I have already made changes
>> >     for the Q35 model (ioh3420 bridge).
>> >
>> > 2. When the guest is created, the operator creates two bridge devices
>> >     (for example, using '-device ioh3420,group_id="string"'), and specifies
>> >     a unique identifier string for both bridges. This identifier can be
>> >     the UUID generated by 'uuidgen' command.
>> >
>> > 3. QEMU places this unique identifier in the PCI configuration space of
>> >     the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
>> >     not to be confused with a specific organization. Instead, the vendor
>> >     of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
>> >
>> > 4. The operator places the passthrough device behind one of the bridges,
>> >     and the virtio device behind the other bridge.
>> >
>> > 5. Patch 4 in patch series [3] should be modified to use the unique
>> >     identifier string stored in the bridges' configuration space instead
>> >     of the MAC address for pairing the devices.
>>
>> This should be an alternate option that allows failover slaves to be registered
>> based on MAC and ID.
>
> I wonder whether we ever want to pair based solely on ID and not on MAC.
> There used to be devices which randomized VF MAC on each reset but I
> think most of them have been changed to get MAC from the PF. Not sure
> whether any are left.
>
> If we want the flexibility, we can use a separate feature bit for
> matching by ID. If not we can just extend the meaning of the
> existing one.
>
>>
>> >
>> > If it is desirable to create only one bridge instead of two (to conserve
>> > the  number of devices in the system), then the passthrough device can be
>> > attached to that single bridge (with the identifier), and the identifier
>> > for the virtio device can be stored in the virtio device's configuration
>> > space itself. To do that, we need to update the virtio specification,
>> > and I have sent a proposal [4] to the OASIS team to update the virtio
>> > specification. If that proposal is accepted, then we can modify QEMU to
>> > use the virtio device's configuration space instead of the second bridge
>> > to store the unique identifier.
>>
>> I think one bridge solution is much cleaner than having to use 2 bridges.
>>
>
> So for one bridge, we'd have a bridge with a special device/vendor ID,
> if virtio is behind that, we use that for pairing.

What does that mean? The virtio deivce needs to be in a same bridge
for pairing? I think Venu's 1-bridge proposal will be able to support
the case when virtio and passthrough devices are placed in separate
bridges.

-Siwei

>
>
>> >
>> > Thank you for sparing the time.
>> >
>> > Venu
>> >
>> > [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
>> > [2] https://www.spinics.net/lists/netdev/msg499011.html
>> > [3] https://patchwork.ozlabs.org/cover/920005/
>> > [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Venu Busireddy]

On 2018-06-01 13:31:24 -0700, Siwei Liu wrote:
> On Fri, Jun 1, 2018 at 8:48 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Fri, Jun 01, 2018 at 08:26:03AM -0700, Samudrala, Sridhar wrote:
> >> On 5/31/2018 6:28 PM, Venu Busireddy wrote:
> >> > I looked at the discussion in the threads [1] and [2], where it was
> >> > suggested placing the passthrough device behind one bridge, and the virtio
> >> > device behind another bridge, and storing in those bridges' configuration
> >> > space some unique identifier that can be used to pair the two devices.
> >> >
> >> > After some discussions with Si-Wei Liu and others, we believe that the
> >> > following scheme may be a viable approach. Please take a look at this
> >> > proposal and provide your thoughts.
> >> >
> >> > 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
> >> >     devices for Q35 as well as i440FX models. I have already made changes
> >> >     for the Q35 model (ioh3420 bridge).
> >> >
> >> > 2. When the guest is created, the operator creates two bridge devices
> >> >     (for example, using '-device ioh3420,group_id="string"'), and specifies
> >> >     a unique identifier string for both bridges. This identifier can be
> >> >     the UUID generated by 'uuidgen' command.
> >> >
> >> > 3. QEMU places this unique identifier in the PCI configuration space of
> >> >     the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
> >> >     not to be confused with a specific organization. Instead, the vendor
> >> >     of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
> >> >
> >> > 4. The operator places the passthrough device behind one of the bridges,
> >> >     and the virtio device behind the other bridge.
> >> >
> >> > 5. Patch 4 in patch series [3] should be modified to use the unique
> >> >     identifier string stored in the bridges' configuration space instead
> >> >     of the MAC address for pairing the devices.
> >>
> >> This should be an alternate option that allows failover slaves to be registered
> >> based on MAC and ID.
> >
> > I wonder whether we ever want to pair based solely on ID and not on MAC.
> > There used to be devices which randomized VF MAC on each reset but I
> > think most of them have been changed to get MAC from the PF. Not sure
> > whether any are left.
> >
> > If we want the flexibility, we can use a separate feature bit for
> > matching by ID. If not we can just extend the meaning of the
> > existing one.
> >
> >>
> >> >
> >> > If it is desirable to create only one bridge instead of two (to conserve
> >> > the  number of devices in the system), then the passthrough device can be
> >> > attached to that single bridge (with the identifier), and the identifier
> >> > for the virtio device can be stored in the virtio device's configuration
> >> > space itself. To do that, we need to update the virtio specification,
> >> > and I have sent a proposal [4] to the OASIS team to update the virtio
> >> > specification. If that proposal is accepted, then we can modify QEMU to
> >> > use the virtio device's configuration space instead of the second bridge
> >> > to store the unique identifier.
> >>
> >> I think one bridge solution is much cleaner than having to use 2 bridges.
> >>
> >
> > So for one bridge, we'd have a bridge with a special device/vendor ID,
> > if virtio is behind that, we use that for pairing.
> 
> What does that mean? The virtio deivce needs to be in a same bridge

I too had the same question.

> for pairing? I think Venu's 1-bridge proposal will be able to support
> the case when virtio and passthrough devices are placed in separate
> bridges.

However, in my 1-bridge proposal, I am suggesting that we put the
passthrough device behind that single bridge and store the UUID in that
bridge, and for the virtio device, store the UUID in the configuration
space of the virtio device itself (no second bridge).

Venu

> >> >
> >> > Thank you for sparing the time.
> >> >
> >> > Venu
> >> >
> >> > [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> >> > [2] https://www.spinics.net/lists/netdev/msg499011.html
> >> > [3] https://patchwork.ozlabs.org/cover/920005/
> >> > [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Michael S. Tsirkin]

On Fri, Jun 01, 2018 at 01:31:24PM -0700, Siwei Liu wrote:
> On Fri, Jun 1, 2018 at 8:48 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Fri, Jun 01, 2018 at 08:26:03AM -0700, Samudrala, Sridhar wrote:
> >> On 5/31/2018 6:28 PM, Venu Busireddy wrote:
> >> > I looked at the discussion in the threads [1] and [2], where it was
> >> > suggested placing the passthrough device behind one bridge, and the virtio
> >> > device behind another bridge, and storing in those bridges' configuration
> >> > space some unique identifier that can be used to pair the two devices.
> >> >
> >> > After some discussions with Si-Wei Liu and others, we believe that the
> >> > following scheme may be a viable approach. Please take a look at this
> >> > proposal and provide your thoughts.
> >> >
> >> > 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
> >> >     devices for Q35 as well as i440FX models. I have already made changes
> >> >     for the Q35 model (ioh3420 bridge).
> >> >
> >> > 2. When the guest is created, the operator creates two bridge devices
> >> >     (for example, using '-device ioh3420,group_id="string"'), and specifies
> >> >     a unique identifier string for both bridges. This identifier can be
> >> >     the UUID generated by 'uuidgen' command.
> >> >
> >> > 3. QEMU places this unique identifier in the PCI configuration space of
> >> >     the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
> >> >     not to be confused with a specific organization. Instead, the vendor
> >> >     of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
> >> >
> >> > 4. The operator places the passthrough device behind one of the bridges,
> >> >     and the virtio device behind the other bridge.
> >> >
> >> > 5. Patch 4 in patch series [3] should be modified to use the unique
> >> >     identifier string stored in the bridges' configuration space instead
> >> >     of the MAC address for pairing the devices.
> >>
> >> This should be an alternate option that allows failover slaves to be registered
> >> based on MAC and ID.
> >
> > I wonder whether we ever want to pair based solely on ID and not on MAC.
> > There used to be devices which randomized VF MAC on each reset but I
> > think most of them have been changed to get MAC from the PF. Not sure
> > whether any are left.
> >
> > If we want the flexibility, we can use a separate feature bit for
> > matching by ID. If not we can just extend the meaning of the
> > existing one.
> >
> >>
> >> >
> >> > If it is desirable to create only one bridge instead of two (to conserve
> >> > the  number of devices in the system), then the passthrough device can be
> >> > attached to that single bridge (with the identifier), and the identifier
> >> > for the virtio device can be stored in the virtio device's configuration
> >> > space itself. To do that, we need to update the virtio specification,
> >> > and I have sent a proposal [4] to the OASIS team to update the virtio
> >> > specification. If that proposal is accepted, then we can modify QEMU to
> >> > use the virtio device's configuration space instead of the second bridge
> >> > to store the unique identifier.
> >>
> >> I think one bridge solution is much cleaner than having to use 2 bridges.
> >>
> >
> > So for one bridge, we'd have a bridge with a special device/vendor ID,
> > if virtio is behind that, we use that for pairing.
> 
> What does that mean? The virtio deivce needs to be in a same bridge
> for pairing? I think Venu's 1-bridge proposal will be able to support
> the case when virtio and passthrough devices are placed in separate
> bridges.
> 
> -Siwei

Reusing intel's device/vendor ID and adding qemu vendor specific
capability is really a no-go.

You can refer to my original proposal for covering all the options,
including Venu's and Sridhar's:
https://patchwork.ozlabs.org/patch/901596/#1901665


what it boils down to is that if two devices
with same mac are behind the same bridge, they pair
without need for an ID.
If they need to be behind separate bridges we can look up
bridge based on ID.

> >
> >
> >> >
> >> > Thank you for sparing the time.
> >> >
> >> > Venu
> >> >
> >> > [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> >> > [2] https://www.spinics.net/lists/netdev/msg499011.html
> >> > [3] https://patchwork.ozlabs.org/cover/920005/
> >> > [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] RFC: Use of bridge devices to store pairing information...

[Michael S. Tsirkin]

On Fri, Jun 01, 2018 at 03:37:09PM -0500, Venu Busireddy wrote:
> On 2018-06-01 13:31:24 -0700, Siwei Liu wrote:
> > On Fri, Jun 1, 2018 at 8:48 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > On Fri, Jun 01, 2018 at 08:26:03AM -0700, Samudrala, Sridhar wrote:
> > >> On 5/31/2018 6:28 PM, Venu Busireddy wrote:
> > >> > I looked at the discussion in the threads [1] and [2], where it was
> > >> > suggested placing the passthrough device behind one bridge, and the virtio
> > >> > device behind another bridge, and storing in those bridges' configuration
> > >> > space some unique identifier that can be used to pair the two devices.
> > >> >
> > >> > After some discussions with Si-Wei Liu and others, we believe that the
> > >> > following scheme may be a viable approach. Please take a look at this
> > >> > proposal and provide your thoughts.
> > >> >
> > >> > 1. Enhance the QEMU CLI to include a "group_id" option to the bridge
> > >> >     devices for Q35 as well as i440FX models. I have already made changes
> > >> >     for the Q35 model (ioh3420 bridge).
> > >> >
> > >> > 2. When the guest is created, the operator creates two bridge devices
> > >> >     (for example, using '-device ioh3420,group_id="string"'), and specifies
> > >> >     a unique identifier string for both bridges. This identifier can be
> > >> >     the UUID generated by 'uuidgen' command.
> > >> >
> > >> > 3. QEMU places this unique identifier in the PCI configuration space of
> > >> >     the bridge as Vendor-Specific capability (0x09). The "Vendor" here is
> > >> >     not to be confused with a specific organization. Instead, the vendor
> > >> >     of the bridge is QEMU (with vendor ID 0x8086 and device ID 0x3420).
> > >> >
> > >> > 4. The operator places the passthrough device behind one of the bridges,
> > >> >     and the virtio device behind the other bridge.
> > >> >
> > >> > 5. Patch 4 in patch series [3] should be modified to use the unique
> > >> >     identifier string stored in the bridges' configuration space instead
> > >> >     of the MAC address for pairing the devices.
> > >>
> > >> This should be an alternate option that allows failover slaves to be registered
> > >> based on MAC and ID.
> > >
> > > I wonder whether we ever want to pair based solely on ID and not on MAC.
> > > There used to be devices which randomized VF MAC on each reset but I
> > > think most of them have been changed to get MAC from the PF. Not sure
> > > whether any are left.
> > >
> > > If we want the flexibility, we can use a separate feature bit for
> > > matching by ID. If not we can just extend the meaning of the
> > > existing one.
> > >
> > >>
> > >> >
> > >> > If it is desirable to create only one bridge instead of two (to conserve
> > >> > the  number of devices in the system), then the passthrough device can be
> > >> > attached to that single bridge (with the identifier), and the identifier
> > >> > for the virtio device can be stored in the virtio device's configuration
> > >> > space itself. To do that, we need to update the virtio specification,
> > >> > and I have sent a proposal [4] to the OASIS team to update the virtio
> > >> > specification. If that proposal is accepted, then we can modify QEMU to
> > >> > use the virtio device's configuration space instead of the second bridge
> > >> > to store the unique identifier.
> > >>
> > >> I think one bridge solution is much cleaner than having to use 2 bridges.
> > >>
> > >
> > > So for one bridge, we'd have a bridge with a special device/vendor ID,
> > > if virtio is behind that, we use that for pairing.
> > 
> > What does that mean? The virtio deivce needs to be in a same bridge
> 
> I too had the same question.
> 
> > for pairing? I think Venu's 1-bridge proposal will be able to support
> > the case when virtio and passthrough devices are placed in separate
> > bridges.
> 
> However, in my 1-bridge proposal, I am suggesting that we put the
> passthrough device behind that single bridge and store the UUID in that
> bridge, and for the virtio device, store the UUID in the configuration
> space of the virtio device itself (no second bridge).
> 
> Venu

If the virtio device is behind the same bridge, then we don't even
need to add capability to the virtio device.
It might not be appropriate for everyone, but it's an option.

> > >> >
> > >> > Thank you for sparing the time.
> > >> >
> > >> > Venu
> > >> >
> > >> > [1] https://www.spinics.net/lists/linux-virtualization/msg33518.html
> > >> > [2] https://www.spinics.net/lists/netdev/msg499011.html
> > >> > [3] https://patchwork.ozlabs.org/cover/920005/
> > >> > [4] https://lists.oasis-open.org/archives/virtio-dev/201805/msg00118.html
> > >> >
> > >> >
> > >> > ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > >> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > >> >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > >
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org