[Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

We had a call with those interested and/or working on vfio
for platform devices.

Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
              Santosh Shukla, Alex Williamson, Alexander Graf,
              Antonios Motakis, Christoffer Dall, Kim Phillips,
              Stuart Yoder

Several aspects to vfio for platform devices:

1. IOMMU groups

 -iommu driver needs to register a bus notifier for the platform bus
  and create groups for relevant platform devices
 -Antonios is looking at this for several ARM IOMMUs
 -PAMU (Freescale) driver already does this

2. unbinding device from host

 PCI:
   echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
 Platform:
   echo ffe101300.dma > /sys/bus/platform/devices/ffe101300.dma/driver/unbind

 -don't believe there are issues or work to do here

3. binding device to vfio-platform driver

 PCI:
   echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id

 -this is probably the least understood issue-- platform drivers
  register themselves with the bus for a specific "name"
  string.  That is matched with device tree "compatible" strings
  later to bind a device to a driver
 -we want is to have the vfio-platform driver to dynamically bind
  to a variety of platform devices previously unknown to
  vfio-platform
 -ideally unbinding and binding could be an atomic operation
 -Alex W pointed out that x86 could leverage this work so
  keep that in mind in what we design
 -Kim Phillips (Linaro) will start working on this

4. vfio kernel interface

 -exposes regions and interrupts to user space via FDs
 -there are 'info' ioctls that allow getting info about
  regions/interrupts such as size and type of interrupt
 -there is a proposed extension to the 'info' ioctls that
  provides device tree paths, allowing user space to coorelate
  resources with the device tree
  (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-July/006237.html)

5. QEMU
 -some key tasks
   -interacts with vfio kernel interface
   -registers memslots
   -needs to dynamically get device hooked up to VM's
    platform bus, including IRQs
   -needs to generate device tree node
 -a key point: we don't believe that platform device passthru
  in QEMU can be solved in a completely generic way.  There will
  need to be device specific code for each device type being passed
  through...to do things like generate device tree nodes
 -in general we expect a relatively small number of device types
  to be passed through to VMs
 -Alex Graf is working on dynamic creation of platform devices
  for the PPC e500 paravirt machine
 -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-07/msg03614.html
 -first step is dynamically generating a virtual UART
 -that sets the stage to create and create vfio devices backed
  by real hardware

There is a session at Linux Plumbers in a couple of weeks
and further discussions will happen there.

Regards,
Stuart

vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

We had a call with those interested and/or working on vfio
for platform devices.

Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
              Santosh Shukla, Alex Williamson, Alexander Graf,
              Antonios Motakis, Christoffer Dall, Kim Phillips,
              Stuart Yoder

Several aspects to vfio for platform devices:

1. IOMMU groups

 -iommu driver needs to register a bus notifier for the platform bus
  and create groups for relevant platform devices
 -Antonios is looking at this for several ARM IOMMUs
 -PAMU (Freescale) driver already does this

2. unbinding device from host

 PCI:
   echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
 Platform:
   echo ffe101300.dma > /sys/bus/platform/devices/ffe101300.dma/driver/unbind

 -don't believe there are issues or work to do here

3. binding device to vfio-platform driver

 PCI:
   echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id

 -this is probably the least understood issue-- platform drivers
  register themselves with the bus for a specific "name"
  string.  That is matched with device tree "compatible" strings
  later to bind a device to a driver
 -we want is to have the vfio-platform driver to dynamically bind
  to a variety of platform devices previously unknown to
  vfio-platform
 -ideally unbinding and binding could be an atomic operation
 -Alex W pointed out that x86 could leverage this work so
  keep that in mind in what we design
 -Kim Phillips (Linaro) will start working on this

4. vfio kernel interface

 -exposes regions and interrupts to user space via FDs
 -there are 'info' ioctls that allow getting info about
  regions/interrupts such as size and type of interrupt
 -there is a proposed extension to the 'info' ioctls that
  provides device tree paths, allowing user space to coorelate
  resources with the device tree
  (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-July/006237.html)

5. QEMU
 -some key tasks
   -interacts with vfio kernel interface
   -registers memslots
   -needs to dynamically get device hooked up to VM's
    platform bus, including IRQs
   -needs to generate device tree node
 -a key point: we don't believe that platform device passthru
  in QEMU can be solved in a completely generic way.  There will
  need to be device specific code for each device type being passed
  through...to do things like generate device tree nodes
 -in general we expect a relatively small number of device types
  to be passed through to VMs
 -Alex Graf is working on dynamic creation of platform devices
  for the PPC e500 paravirt machine
 -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-07/msg03614.html
 -first step is dynamically generating a virtual UART
 -that sets the stage to create and create vfio devices backed
  by real hardware

There is a session at Linux Plumbers in a couple of weeks
and further discussions will happen there.

Regards,
Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Sethi Varun-B16395]

I have a query about the ARM SMMU driver. In the ARM smmu driver I see, that bus notifiers are registered for both amba and platform bus. Amba is the I/O interconnect, right? Why is bus notifier required for the amba bus?
 
-Varun

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Thursday, September 05, 2013 11:21 PM
> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: vfio for platform devices - 9/5/2012 - minutes
> 
> We had a call with those interested and/or working on vfio for platform
> devices.
> 
> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>               Santosh Shukla, Alex Williamson, Alexander Graf,
>               Antonios Motakis, Christoffer Dall, Kim Phillips,
>               Stuart Yoder
> 
> Several aspects to vfio for platform devices:
> 
> 1. IOMMU groups
> 
>  -iommu driver needs to register a bus notifier for the platform bus
>   and create groups for relevant platform devices  -Antonios is looking
> at this for several ARM IOMMUs  -PAMU (Freescale) driver already does
> this
> 
> 2. unbinding device from host
> 
>  PCI:
>    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>  Platform:
>    echo ffe101300.dma >
> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> 
>  -don't believe there are issues or work to do here
> 
> 3. binding device to vfio-platform driver
> 
>  PCI:
>    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
>  -this is probably the least understood issue-- platform drivers
>   register themselves with the bus for a specific "name"
>   string.  That is matched with device tree "compatible" strings
>   later to bind a device to a driver
>  -we want is to have the vfio-platform driver to dynamically bind
>   to a variety of platform devices previously unknown to
>   vfio-platform
>  -ideally unbinding and binding could be an atomic operation  -Alex W
> pointed out that x86 could leverage this work so
>   keep that in mind in what we design
>  -Kim Phillips (Linaro) will start working on this
> 
> 4. vfio kernel interface
> 
>  -exposes regions and interrupts to user space via FDs  -there are 'info'
> ioctls that allow getting info about
>   regions/interrupts such as size and type of interrupt  -there is a
> proposed extension to the 'info' ioctls that
>   provides device tree paths, allowing user space to coorelate
>   resources with the device tree
>   (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-July/006237.html)
> 
> 5. QEMU
>  -some key tasks
>    -interacts with vfio kernel interface
>    -registers memslots
>    -needs to dynamically get device hooked up to VM's
>     platform bus, including IRQs
>    -needs to generate device tree node
>  -a key point: we don't believe that platform device passthru
>   in QEMU can be solved in a completely generic way.  There will
>   need to be device specific code for each device type being passed
>   through...to do things like generate device tree nodes  -in general we
> expect a relatively small number of device types
>   to be passed through to VMs
>  -Alex Graf is working on dynamic creation of platform devices
>   for the PPC e500 paravirt machine
>  -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-
> 07/msg03614.html
>  -first step is dynamically generating a virtual UART  -that sets the
> stage to create and create vfio devices backed
>   by real hardware
> 
> There is a session at Linux Plumbers in a couple of weeks and further
> discussions will happen there.
> 
> Regards,
> Stuart

RE: vfio for platform devices - 9/5/2012 - minutes

[Sethi Varun-B16395]

I have a query about the ARM SMMU driver. In the ARM smmu driver I see, that bus notifiers are registered for both amba and platform bus. Amba is the I/O interconnect, right? Why is bus notifier required for the amba bus?
 
-Varun

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Thursday, September 05, 2013 11:21 PM
> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: vfio for platform devices - 9/5/2012 - minutes
> 
> We had a call with those interested and/or working on vfio for platform
> devices.
> 
> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>               Santosh Shukla, Alex Williamson, Alexander Graf,
>               Antonios Motakis, Christoffer Dall, Kim Phillips,
>               Stuart Yoder
> 
> Several aspects to vfio for platform devices:
> 
> 1. IOMMU groups
> 
>  -iommu driver needs to register a bus notifier for the platform bus
>   and create groups for relevant platform devices  -Antonios is looking
> at this for several ARM IOMMUs  -PAMU (Freescale) driver already does
> this
> 
> 2. unbinding device from host
> 
>  PCI:
>    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>  Platform:
>    echo ffe101300.dma >
> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> 
>  -don't believe there are issues or work to do here
> 
> 3. binding device to vfio-platform driver
> 
>  PCI:
>    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
>  -this is probably the least understood issue-- platform drivers
>   register themselves with the bus for a specific "name"
>   string.  That is matched with device tree "compatible" strings
>   later to bind a device to a driver
>  -we want is to have the vfio-platform driver to dynamically bind
>   to a variety of platform devices previously unknown to
>   vfio-platform
>  -ideally unbinding and binding could be an atomic operation  -Alex W
> pointed out that x86 could leverage this work so
>   keep that in mind in what we design
>  -Kim Phillips (Linaro) will start working on this
> 
> 4. vfio kernel interface
> 
>  -exposes regions and interrupts to user space via FDs  -there are 'info'
> ioctls that allow getting info about
>   regions/interrupts such as size and type of interrupt  -there is a
> proposed extension to the 'info' ioctls that
>   provides device tree paths, allowing user space to coorelate
>   resources with the device tree
>   (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-July/006237.html)
> 
> 5. QEMU
>  -some key tasks
>    -interacts with vfio kernel interface
>    -registers memslots
>    -needs to dynamically get device hooked up to VM's
>     platform bus, including IRQs
>    -needs to generate device tree node
>  -a key point: we don't believe that platform device passthru
>   in QEMU can be solved in a completely generic way.  There will
>   need to be device specific code for each device type being passed
>   through...to do things like generate device tree nodes  -in general we
> expect a relatively small number of device types
>   to be passed through to VMs
>  -Alex Graf is working on dynamic creation of platform devices
>   for the PPC e500 paravirt machine
>  -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-
> 07/msg03614.html
>  -first step is dynamically generating a virtual UART  -that sets the
> stage to create and create vfio devices backed
>   by real hardware
> 
> There is a session at Linux Plumbers in a couple of weeks and further
> discussions will happen there.
> 
> Regards,
> Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

Adding Will...

> -----Original Message-----
> From: Sethi Varun-B16395
> Sent: Friday, September 06, 2013 11:56 AM
> To: Yoder Stuart-B08248; Wood Scott-B07421; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: RE: vfio for platform devices - 9/5/2012 - minutes
> 
> I have a query about the ARM SMMU driver. In the ARM smmu driver I see,
> that bus notifiers are registered for both amba and platform bus. Amba is
> the I/O interconnect, right? Why is bus notifier required for the amba
> bus?
> 
> -Varun
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Thursday, September 05, 2013 11:21 PM
> > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> > 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> > devel@nongnu.org
> > Subject: vfio for platform devices - 9/5/2012 - minutes
> >
> > We had a call with those interested and/or working on vfio for platform
> > devices.
> >
> > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> >               Santosh Shukla, Alex Williamson, Alexander Graf,
> >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> >               Stuart Yoder
> >
> > Several aspects to vfio for platform devices:
> >
> > 1. IOMMU groups
> >
> >  -iommu driver needs to register a bus notifier for the platform bus
> >   and create groups for relevant platform devices  -Antonios is looking
> > at this for several ARM IOMMUs  -PAMU (Freescale) driver already does
> > this
> >
> > 2. unbinding device from host
> >
> >  PCI:
> >    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> >  Platform:
> >    echo ffe101300.dma >
> > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> >
> >  -don't believe there are issues or work to do here
> >
> > 3. binding device to vfio-platform driver
> >
> >  PCI:
> >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> >
> >  -this is probably the least understood issue-- platform drivers
> >   register themselves with the bus for a specific "name"
> >   string.  That is matched with device tree "compatible" strings
> >   later to bind a device to a driver
> >  -we want is to have the vfio-platform driver to dynamically bind
> >   to a variety of platform devices previously unknown to
> >   vfio-platform
> >  -ideally unbinding and binding could be an atomic operation  -Alex W
> > pointed out that x86 could leverage this work so
> >   keep that in mind in what we design
> >  -Kim Phillips (Linaro) will start working on this
> >
> > 4. vfio kernel interface
> >
> >  -exposes regions and interrupts to user space via FDs  -there are
> 'info'
> > ioctls that allow getting info about
> >   regions/interrupts such as size and type of interrupt  -there is a
> > proposed extension to the 'info' ioctls that
> >   provides device tree paths, allowing user space to coorelate
> >   resources with the device tree
> >   (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-
> July/006237.html)
> >
> > 5. QEMU
> >  -some key tasks
> >    -interacts with vfio kernel interface
> >    -registers memslots
> >    -needs to dynamically get device hooked up to VM's
> >     platform bus, including IRQs
> >    -needs to generate device tree node
> >  -a key point: we don't believe that platform device passthru
> >   in QEMU can be solved in a completely generic way.  There will
> >   need to be device specific code for each device type being passed
> >   through...to do things like generate device tree nodes  -in general
> we
> > expect a relatively small number of device types
> >   to be passed through to VMs
> >  -Alex Graf is working on dynamic creation of platform devices
> >   for the PPC e500 paravirt machine
> >  -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-
> > 07/msg03614.html
> >  -first step is dynamically generating a virtual UART  -that sets the
> > stage to create and create vfio devices backed
> >   by real hardware
> >
> > There is a session at Linux Plumbers in a couple of weeks and further
> > discussions will happen there.
> >
> > Regards,
> > Stuart

RE: vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

Adding Will...

> -----Original Message-----
> From: Sethi Varun-B16395
> Sent: Friday, September 06, 2013 11:56 AM
> To: Yoder Stuart-B08248; Wood Scott-B07421; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: RE: vfio for platform devices - 9/5/2012 - minutes
> 
> I have a query about the ARM SMMU driver. In the ARM smmu driver I see,
> that bus notifiers are registered for both amba and platform bus. Amba is
> the I/O interconnect, right? Why is bus notifier required for the amba
> bus?
> 
> -Varun
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Thursday, September 05, 2013 11:21 PM
> > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> > 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > Cc: kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> > devel@nongnu.org
> > Subject: vfio for platform devices - 9/5/2012 - minutes
> >
> > We had a call with those interested and/or working on vfio for platform
> > devices.
> >
> > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> >               Santosh Shukla, Alex Williamson, Alexander Graf,
> >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> >               Stuart Yoder
> >
> > Several aspects to vfio for platform devices:
> >
> > 1. IOMMU groups
> >
> >  -iommu driver needs to register a bus notifier for the platform bus
> >   and create groups for relevant platform devices  -Antonios is looking
> > at this for several ARM IOMMUs  -PAMU (Freescale) driver already does
> > this
> >
> > 2. unbinding device from host
> >
> >  PCI:
> >    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> >  Platform:
> >    echo ffe101300.dma >
> > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> >
> >  -don't believe there are issues or work to do here
> >
> > 3. binding device to vfio-platform driver
> >
> >  PCI:
> >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> >
> >  -this is probably the least understood issue-- platform drivers
> >   register themselves with the bus for a specific "name"
> >   string.  That is matched with device tree "compatible" strings
> >   later to bind a device to a driver
> >  -we want is to have the vfio-platform driver to dynamically bind
> >   to a variety of platform devices previously unknown to
> >   vfio-platform
> >  -ideally unbinding and binding could be an atomic operation  -Alex W
> > pointed out that x86 could leverage this work so
> >   keep that in mind in what we design
> >  -Kim Phillips (Linaro) will start working on this
> >
> > 4. vfio kernel interface
> >
> >  -exposes regions and interrupts to user space via FDs  -there are
> 'info'
> > ioctls that allow getting info about
> >   regions/interrupts such as size and type of interrupt  -there is a
> > proposed extension to the 'info' ioctls that
> >   provides device tree paths, allowing user space to coorelate
> >   resources with the device tree
> >   (https://lists.cs.columbia.edu/pipermail/kvmarm/2013-
> July/006237.html)
> >
> > 5. QEMU
> >  -some key tasks
> >    -interacts with vfio kernel interface
> >    -registers memslots
> >    -needs to dynamically get device hooked up to VM's
> >     platform bus, including IRQs
> >    -needs to generate device tree node
> >  -a key point: we don't believe that platform device passthru
> >   in QEMU can be solved in a completely generic way.  There will
> >   need to be device specific code for each device type being passed
> >   through...to do things like generate device tree nodes  -in general
> we
> > expect a relatively small number of device types
> >   to be passed through to VMs
> >  -Alex Graf is working on dynamic creation of platform devices
> >   for the PPC e500 paravirt machine
> >  -see: http://lists.nongnu.org/archive/html/qemu-devel/2013-
> > 07/msg03614.html
> >  -first step is dynamically generating a virtual UART  -that sets the
> > stage to create and create vfio devices backed
> >   by real hardware
> >
> > There is a session at Linux Plumbers in a couple of weeks and further
> > discussions will happen there.
> >
> > Regards,
> > Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Will Deacon]

On Fri, Sep 06, 2013 at 07:20:19PM +0100, Yoder Stuart-B08248 wrote:
> Adding Will...

[...]

> > I have a query about the ARM SMMU driver. In the ARM smmu driver I see,
> > that bus notifiers are registered for both amba and platform bus. Amba is
> > the I/O interconnect, right? Why is bus notifier required for the amba
> > bus?

Not sure I follow the question, really. If you have a DMA master registered
as an AMBA device (e.g. PL330) and it wants to use an SMMU, then you need to
be registered on that bus.

What would you prefer instead?

Will

Re: vfio for platform devices - 9/5/2012 - minutes

[Will Deacon]

On Fri, Sep 06, 2013 at 07:20:19PM +0100, Yoder Stuart-B08248 wrote:
> Adding Will...

[...]

> > I have a query about the ARM SMMU driver. In the ARM smmu driver I see,
> > that bus notifiers are registered for both amba and platform bus. Amba is
> > the I/O interconnect, right? Why is bus notifier required for the amba
> > bus?

Not sure I follow the question, really. If you have a DMA master registered
as an AMBA device (e.g. PL330) and it wants to use an SMMU, then you need to
be registered on that bus.

What would you prefer instead?

Will

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Sethi Varun-B16395]

Hi Will,
Just trying to understand the scope of platform device assignment to guest on ARM. So, are the AMBA devices also represented in the device tree?

Regards
Varun

> -----Original Message-----
> From: kvm-ppc-owner@vger.kernel.org [mailto:kvm-ppc-
> owner@vger.kernel.org] On Behalf Of Will Deacon
> Sent: Monday, September 09, 2013 2:23 PM
> To: Yoder Stuart-B08248
> Cc: Sethi Varun-B16395; Wood Scott-B07421; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org';
> kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> 
> On Fri, Sep 06, 2013 at 07:20:19PM +0100, Yoder Stuart-B08248 wrote:
> > Adding Will...
> 
> [...]
> 
> > > I have a query about the ARM SMMU driver. In the ARM smmu driver I
> > > see, that bus notifiers are registered for both amba and platform
> > > bus. Amba is the I/O interconnect, right? Why is bus notifier
> > > required for the amba bus?
> 
> Not sure I follow the question, really. If you have a DMA master
> registered as an AMBA device (e.g. PL330) and it wants to use an SMMU,
> then you need to be registered on that bus.
> 
> What would you prefer instead?
> 
> Will
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

RE: vfio for platform devices - 9/5/2012 - minutes

[Sethi Varun-B16395]

Hi Will,
Just trying to understand the scope of platform device assignment to guest on ARM. So, are the AMBA devices also represented in the device tree?

Regards
Varun

> -----Original Message-----
> From: kvm-ppc-owner@vger.kernel.org [mailto:kvm-ppc-
> owner@vger.kernel.org] On Behalf Of Will Deacon
> Sent: Monday, September 09, 2013 2:23 PM
> To: Yoder Stuart-B08248
> Cc: Sethi Varun-B16395; Wood Scott-B07421; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org';
> kvmarm@lists.cs.columbia.edu; kvm-ppc@vger.kernel.org; qemu-
> devel@nongnu.org
> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> 
> On Fri, Sep 06, 2013 at 07:20:19PM +0100, Yoder Stuart-B08248 wrote:
> > Adding Will...
> 
> [...]
> 
> > > I have a query about the ARM SMMU driver. In the ARM smmu driver I
> > > see, that bus notifiers are registered for both amba and platform
> > > bus. Amba is the I/O interconnect, right? Why is bus notifier
> > > required for the amba bus?
> 
> Not sure I follow the question, really. If you have a DMA master
> registered as an AMBA device (e.g. PL330) and it wants to use an SMMU,
> then you need to be registered on that bus.
> 
> What would you prefer instead?
> 
> Will
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Thursday, September 05, 2013 12:51 PM
> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> devel@nongnu.org'
> Subject: vfio for platform devices - 9/5/2012 - minutes
> 
> We had a call with those interested and/or working on vfio
> for platform devices.
> 
> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>               Santosh Shukla, Alex Williamson, Alexander Graf,
>               Antonios Motakis, Christoffer Dall, Kim Phillips,
>               Stuart Yoder
> 
> Several aspects to vfio for platform devices:
> 
> 1. IOMMU groups
> 
>  -iommu driver needs to register a bus notifier for the platform bus
>   and create groups for relevant platform devices
>  -Antonios is looking at this for several ARM IOMMUs
>  -PAMU (Freescale) driver already does this
> 
> 2. unbinding device from host
> 
>  PCI:
>    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>  Platform:
>    echo ffe101300.dma >
> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> 
>  -don't believe there are issues or work to do here
> 
> 3. binding device to vfio-platform driver
> 
>  PCI:
>    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
>  -this is probably the least understood issue-- platform drivers
>   register themselves with the bus for a specific "name"
>   string.  That is matched with device tree "compatible" strings
>   later to bind a device to a driver
>  -we want is to have the vfio-platform driver to dynamically bind
>   to a variety of platform devices previously unknown to
>   vfio-platform
>  -ideally unbinding and binding could be an atomic operation
>  -Alex W pointed out that x86 could leverage this work so
>   keep that in mind in what we design
>  -Kim Phillips (Linaro) will start working on this

One thing we didn't discuss needs to be considered (probably by
Kim who is looking at the 'binding device' issue) is around
returning a passthru device back to the host.

After a platform device has been bound to vfio and is in use by
user space or a virtual machine, we also need to be able
to unwind all that and return the device back to the host
in a sane state.

What happens when user space exits and the vfio file
descriptors are closed?

What if the device is still active and doing bus 
mastering?   (e.g. a VM crashed causing a QEMU
exit)

How can the vfio-platform layer in the host kernel
get a specific device in a sane state?

When a plaform device is 'unbound' from vfio, what
specifically happens to the device?

Platform devices don't have generic mechanisms like on PCI
to disable bus mastering or even disable or reset a
device.

Haven't thought through all this yet, but just raising
some issues I see.

Regards,
Stuart

RE: vfio for platform devices - 9/5/2012 - minutes

[Yoder Stuart-B08248]

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Thursday, September 05, 2013 12:51 PM
> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> devel@nongnu.org'
> Subject: vfio for platform devices - 9/5/2012 - minutes
> 
> We had a call with those interested and/or working on vfio
> for platform devices.
> 
> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>               Santosh Shukla, Alex Williamson, Alexander Graf,
>               Antonios Motakis, Christoffer Dall, Kim Phillips,
>               Stuart Yoder
> 
> Several aspects to vfio for platform devices:
> 
> 1. IOMMU groups
> 
>  -iommu driver needs to register a bus notifier for the platform bus
>   and create groups for relevant platform devices
>  -Antonios is looking at this for several ARM IOMMUs
>  -PAMU (Freescale) driver already does this
> 
> 2. unbinding device from host
> 
>  PCI:
>    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>  Platform:
>    echo ffe101300.dma >
> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> 
>  -don't believe there are issues or work to do here
> 
> 3. binding device to vfio-platform driver
> 
>  PCI:
>    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
>  -this is probably the least understood issue-- platform drivers
>   register themselves with the bus for a specific "name"
>   string.  That is matched with device tree "compatible" strings
>   later to bind a device to a driver
>  -we want is to have the vfio-platform driver to dynamically bind
>   to a variety of platform devices previously unknown to
>   vfio-platform
>  -ideally unbinding and binding could be an atomic operation
>  -Alex W pointed out that x86 could leverage this work so
>   keep that in mind in what we design
>  -Kim Phillips (Linaro) will start working on this

One thing we didn't discuss needs to be considered (probably by
Kim who is looking at the 'binding device' issue) is around
returning a passthru device back to the host.

After a platform device has been bound to vfio and is in use by
user space or a virtual machine, we also need to be able
to unwind all that and return the device back to the host
in a sane state.

What happens when user space exits and the vfio file
descriptors are closed?

What if the device is still active and doing bus 
mastering?   (e.g. a VM crashed causing a QEMU
exit)

How can the vfio-platform layer in the host kernel
get a specific device in a sane state?

When a plaform device is 'unbound' from vfio, what
specifically happens to the device?

Platform devices don't have generic mechanisms like on PCI
to disable bus mastering or even disable or reset a
device.

Haven't thought through all this yet, but just raising
some issues I see.

Regards,
Stuart

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Wed, 2013-09-11 at 11:42 -0500, Yoder Stuart-B08248 wrote:
> One thing we didn't discuss needs to be considered (probably by
> Kim who is looking at the 'binding device' issue) is around
> returning a passthru device back to the host.
> 
> After a platform device has been bound to vfio and is in use by
> user space or a virtual machine, we also need to be able
> to unwind all that and return the device back to the host
> in a sane state.
> 
> What happens when user space exits and the vfio file
> descriptors are closed?
> 
> What if the device is still active and doing bus 
> mastering?   (e.g. a VM crashed causing a QEMU
> exit)
> 
> How can the vfio-platform layer in the host kernel
> get a specific device in a sane state?
> 
> When a plaform device is 'unbound' from vfio, what
> specifically happens to the device?
> 
> Platform devices don't have generic mechanisms like on PCI
> to disable bus mastering or even disable or reset a
> device.
> 
> Haven't thought through all this yet, but just raising
> some issues I see.

In the absence of some code in the host kernel that knows how to reset
the device, I think the model needs to be that devices start out with
the IOMMU blocking all transactions, and the user must ensure that the
device has been quiesced prior to enabling the device at the IOMMU.
This may require a hypercall or similar to let QEMU know when it's safe
to enable the device's IOMMU mappings (assuming that there's no virtual
IOMMU visible to the VM).

-Scott

Re: vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Wed, 2013-09-11 at 11:42 -0500, Yoder Stuart-B08248 wrote:
> One thing we didn't discuss needs to be considered (probably by
> Kim who is looking at the 'binding device' issue) is around
> returning a passthru device back to the host.
> 
> After a platform device has been bound to vfio and is in use by
> user space or a virtual machine, we also need to be able
> to unwind all that and return the device back to the host
> in a sane state.
> 
> What happens when user space exits and the vfio file
> descriptors are closed?
> 
> What if the device is still active and doing bus 
> mastering?   (e.g. a VM crashed causing a QEMU
> exit)
> 
> How can the vfio-platform layer in the host kernel
> get a specific device in a sane state?
> 
> When a plaform device is 'unbound' from vfio, what
> specifically happens to the device?
> 
> Platform devices don't have generic mechanisms like on PCI
> to disable bus mastering or even disable or reset a
> device.
> 
> Haven't thought through all this yet, but just raising
> some issues I see.

In the absence of some code in the host kernel that knows how to reset
the device, I think the model needs to be that devices start out with
the IOMMU blocking all transactions, and the user must ensure that the
device has been quiesced prior to enabling the device at the IOMMU.
This may require a hypercall or similar to let QEMU know when it's safe
to enable the device's IOMMU mappings (assuming that there's no virtual
IOMMU visible to the VM).

-Scott

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Alex Williamson]

On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Thursday, September 05, 2013 12:51 PM
> > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> > 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > devel@nongnu.org'
> > Subject: vfio for platform devices - 9/5/2012 - minutes
> > 
> > We had a call with those interested and/or working on vfio
> > for platform devices.
> > 
> > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> >               Santosh Shukla, Alex Williamson, Alexander Graf,
> >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> >               Stuart Yoder
> > 
> > Several aspects to vfio for platform devices:
> > 
> > 1. IOMMU groups
> > 
> >  -iommu driver needs to register a bus notifier for the platform bus
> >   and create groups for relevant platform devices
> >  -Antonios is looking at this for several ARM IOMMUs
> >  -PAMU (Freescale) driver already does this
> > 
> > 2. unbinding device from host
> > 
> >  PCI:
> >    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> >  Platform:
> >    echo ffe101300.dma >
> > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > 
> >  -don't believe there are issues or work to do here
> > 
> > 3. binding device to vfio-platform driver
> > 
> >  PCI:
> >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > 
> >  -this is probably the least understood issue-- platform drivers
> >   register themselves with the bus for a specific "name"
> >   string.  That is matched with device tree "compatible" strings
> >   later to bind a device to a driver
> >  -we want is to have the vfio-platform driver to dynamically bind
> >   to a variety of platform devices previously unknown to
> >   vfio-platform
> >  -ideally unbinding and binding could be an atomic operation
> >  -Alex W pointed out that x86 could leverage this work so
> >   keep that in mind in what we design
> >  -Kim Phillips (Linaro) will start working on this
> 
> One thing we didn't discuss needs to be considered (probably by
> Kim who is looking at the 'binding device' issue) is around
> returning a passthru device back to the host.
> 
> After a platform device has been bound to vfio and is in use by
> user space or a virtual machine, we also need to be able
> to unwind all that and return the device back to the host
> in a sane state.
> 
> What happens when user space exits and the vfio file
> descriptors are closed?

For reference, expectations of how vfio-pci handles these situations:

For vfio-pci, when the reference count on the device fd drops to zero we
call a device disable function that includes disabling the bus master
bit in config space stop ongoing DMA.

> What if the device is still active and doing bus 
> mastering?   (e.g. a VM crashed causing a QEMU
> exit)

If the VM crashes the vfio fds get released resulting in the above
opportunity for the vfio device driver to quiesce the device.

> How can the vfio-platform layer in the host kernel
> get a specific device in a sane state?

It's not easy on pci either.  We save config space prior to exposing the
device and restore config space later, but it's not complete.  We mostly
rely on device (function) resets, to put things in a sane state, but
those aren't always supported.  I just introduced patches for v3.12 that
enable a PCI bus reset interface, but it's mostly useful for userspace,
since on PCI it's often the case that a bus contains multiple devices
which don't necessarily align to iommu group boundaries.

> When a plaform device is 'unbound' from vfio, what
> specifically happens to the device?

The driver remove function is called, which needs to call
vfio_del_group_dev(), which will block if the device is in use.  If it's
unused, the release function should have already been called and the
device should be in a quiesced state.  On x86/pci, there is no auto
probe of devices when unbound, so the device sits without a driver until
manually bound or until something causes an auto probe.  Thanks,

Alex

> Platform devices don't have generic mechanisms like on PCI
> to disable bus mastering or even disable or reset a
> device.
> 
> Haven't thought through all this yet, but just raising
> some issues I see.
> 
> Regards,
> Stuart
> 
> 
> 
> 

Re: vfio for platform devices - 9/5/2012 - minutes

[Alex Williamson]

On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Thursday, September 05, 2013 12:51 PM
> > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander Graf';
> > 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > devel@nongnu.org'
> > Subject: vfio for platform devices - 9/5/2012 - minutes
> > 
> > We had a call with those interested and/or working on vfio
> > for platform devices.
> > 
> > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> >               Santosh Shukla, Alex Williamson, Alexander Graf,
> >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> >               Stuart Yoder
> > 
> > Several aspects to vfio for platform devices:
> > 
> > 1. IOMMU groups
> > 
> >  -iommu driver needs to register a bus notifier for the platform bus
> >   and create groups for relevant platform devices
> >  -Antonios is looking at this for several ARM IOMMUs
> >  -PAMU (Freescale) driver already does this
> > 
> > 2. unbinding device from host
> > 
> >  PCI:
> >    echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> >  Platform:
> >    echo ffe101300.dma >
> > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > 
> >  -don't believe there are issues or work to do here
> > 
> > 3. binding device to vfio-platform driver
> > 
> >  PCI:
> >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > 
> >  -this is probably the least understood issue-- platform drivers
> >   register themselves with the bus for a specific "name"
> >   string.  That is matched with device tree "compatible" strings
> >   later to bind a device to a driver
> >  -we want is to have the vfio-platform driver to dynamically bind
> >   to a variety of platform devices previously unknown to
> >   vfio-platform
> >  -ideally unbinding and binding could be an atomic operation
> >  -Alex W pointed out that x86 could leverage this work so
> >   keep that in mind in what we design
> >  -Kim Phillips (Linaro) will start working on this
> 
> One thing we didn't discuss needs to be considered (probably by
> Kim who is looking at the 'binding device' issue) is around
> returning a passthru device back to the host.
> 
> After a platform device has been bound to vfio and is in use by
> user space or a virtual machine, we also need to be able
> to unwind all that and return the device back to the host
> in a sane state.
> 
> What happens when user space exits and the vfio file
> descriptors are closed?

For reference, expectations of how vfio-pci handles these situations:

For vfio-pci, when the reference count on the device fd drops to zero we
call a device disable function that includes disabling the bus master
bit in config space stop ongoing DMA.

> What if the device is still active and doing bus 
> mastering?   (e.g. a VM crashed causing a QEMU
> exit)

If the VM crashes the vfio fds get released resulting in the above
opportunity for the vfio device driver to quiesce the device.

> How can the vfio-platform layer in the host kernel
> get a specific device in a sane state?

It's not easy on pci either.  We save config space prior to exposing the
device and restore config space later, but it's not complete.  We mostly
rely on device (function) resets, to put things in a sane state, but
those aren't always supported.  I just introduced patches for v3.12 that
enable a PCI bus reset interface, but it's mostly useful for userspace,
since on PCI it's often the case that a bus contains multiple devices
which don't necessarily align to iommu group boundaries.

> When a plaform device is 'unbound' from vfio, what
> specifically happens to the device?

The driver remove function is called, which needs to call
vfio_del_group_dev(), which will block if the device is in use.  If it's
unused, the release function should have already been called and the
device should be in a quiesced state.  On x86/pci, there is no auto
probe of devices when unbound, so the device sits without a driver until
manually bound or until something causes an auto probe.  Thanks,

Alex

> Platform devices don't have generic mechanisms like on PCI
> to disable bus mastering or even disable or reset a
> device.
> 
> Haven't thought through all this yet, but just raising
> some issues I see.
> 
> Regards,
> Stuart
> 
> 
> 
> 

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Bhushan Bharat-R65777]

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Wednesday, September 11, 2013 10:45 PM
> To: Yoder Stuart-B08248
> Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
> Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
> ppc@vger.kernel.org; qemu-devel@nongnu.org
> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> 
> On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> >
> > > -----Original Message-----
> > > From: Yoder Stuart-B08248
> > > Sent: Thursday, September 05, 2013 12:51 PM
> > > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> > > 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
> > > Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > > devel@nongnu.org'
> > > Subject: vfio for platform devices - 9/5/2012 - minutes
> > >
> > > We had a call with those interested and/or working on vfio for
> > > platform devices.
> > >
> > > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> > >               Santosh Shukla, Alex Williamson, Alexander Graf,
> > >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> > >               Stuart Yoder
> > >
> > > Several aspects to vfio for platform devices:
> > >
> > > 1. IOMMU groups
> > >
> > >  -iommu driver needs to register a bus notifier for the platform bus
> > >   and create groups for relevant platform devices  -Antonios is
> > > looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
> > > already does this
> > >
> > > 2. unbinding device from host
> > >
> > >  PCI:
> > >    echo 0000:06:0d.0 >
> > > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> > >  Platform:
> > >    echo ffe101300.dma >
> > > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > >
> > >  -don't believe there are issues or work to do here
> > >
> > > 3. binding device to vfio-platform driver
> > >
> > >  PCI:
> > >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > >
> > >  -this is probably the least understood issue-- platform drivers
> > >   register themselves with the bus for a specific "name"
> > >   string.  That is matched with device tree "compatible" strings
> > >   later to bind a device to a driver  -we want is to have the
> > > vfio-platform driver to dynamically bind
> > >   to a variety of platform devices previously unknown to
> > >   vfio-platform
> > >  -ideally unbinding and binding could be an atomic operation  -Alex
> > > W pointed out that x86 could leverage this work so
> > >   keep that in mind in what we design  -Kim Phillips (Linaro) will
> > > start working on this
> >
> > One thing we didn't discuss needs to be considered (probably by Kim
> > who is looking at the 'binding device' issue) is around returning a
> > passthru device back to the host.
> >
> > After a platform device has been bound to vfio and is in use by user
> > space or a virtual machine, we also need to be able to unwind all that
> > and return the device back to the host in a sane state.
> >
> > What happens when user space exits and the vfio file descriptors are
> > closed?
> 
> For reference, expectations of how vfio-pci handles these situations:
> 
> For vfio-pci, when the reference count on the device fd drops to zero we call a
> device disable function that includes disabling the bus master bit in config
> space stop ongoing DMA.

There is no bus mastering for platform devices and device disabling is not a standard like PCI. Do you think that we might need to do some device specific handling. For example for DMA controller, we need to atleast disable the DMA controller and mask its interrupts (may be ensure that there is no pending/running dma transaction, release irqs etc). As we are not yet having any linkage to respective kernel driver then I am not sure how we will do that specific handling?

> 
> > What if the device is still active and doing bus
> > mastering?   (e.g. a VM crashed causing a QEMU
> > exit)
> 
> If the VM crashes the vfio fds get released resulting in the above opportunity
> for the vfio device driver to quiesce the device.

I think the quiesing of devices with we device specific, so the generic vfio=platform driver may not be able to handle that, right?

> 
> > How can the vfio-platform layer in the host kernel get a specific
> > device in a sane state?
> 
> It's not easy on pci either.  We save config space prior to exposing the device
> and restore config space later, but it's not complete.  We mostly rely on device
> (function) resets, to put things in a sane state, but those aren't always
> supported.

All platform devices also may not have reset capability (and if any then not generic way for all devices).

>  I just introduced patches for v3.12 that enable a PCI bus reset
> interface, but it's mostly useful for userspace, since on PCI it's often the
> case that a bus contains multiple devices which don't necessarily align to iommu
> group boundaries.
> 
> > When a plaform device is 'unbound' from vfio, what specifically
> > happens to the device?
> 
> The driver remove function is called, which needs to call vfio_del_group_dev(),
> which will block if the device is in use.  If it's unused, the release function
> should have already been called and the device should be in a quiesced state.

What does it mean by quiesced state here? Does this mean:
 - Device does not take more work from software; This will done if device is not attached to any driver.
 - Device does not create work for software (no interrupt etc). This can achieved if interrupts masked at device/PIC and device in disabled.

Thanks
-Bharat

> On x86/pci, there is no auto probe of devices when unbound, so the device sits
> without a driver until manually bound or until something causes an auto probe.
> Thanks,
> 
> Alex
> 
> > Platform devices don't have generic mechanisms like on PCI to disable
> > bus mastering or even disable or reset a device.
> >
> > Haven't thought through all this yet, but just raising some issues I
> > see.
> >
> > Regards,
> > Stuart
> >
> >
> >
> >
> 
> 
> 

RE: vfio for platform devices - 9/5/2012 - minutes

[Bhushan Bharat-R65777]

DQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogQWxleCBXaWxsaWFtc29u
IFttYWlsdG86YWxleC53aWxsaWFtc29uQHJlZGhhdC5jb21dDQo+IFNlbnQ6IFdlZG5lc2RheSwg
U2VwdGVtYmVyIDExLCAyMDEzIDEwOjQ1IFBNDQo+IFRvOiBZb2RlciBTdHVhcnQtQjA4MjQ4DQo+
IENjOiBXb29kIFNjb3R0LUIwNzQyMTsgU2V0aGkgVmFydW4tQjE2Mzk1OyBCaHVzaGFuIEJoYXJh
dC1SNjU3Nzc7ICdQZXRlcg0KPiBNYXlkZWxsJzsgJ1NhbnRvc2ggU2h1a2xhJzsgJ0FsZXhhbmRl
ciBHcmFmJzsgJ0FudG9uaW9zIE1vdGFraXMnOyAnQ2hyaXN0b2ZmZXINCj4gRGFsbCc7ICdraW0u
cGhpbGxpcHNAbGluYXJvLm9yZyc7IGt2bWFybUBsaXN0cy5jcy5jb2x1bWJpYS5lZHU7IGt2bS0N
Cj4gcHBjQHZnZXIua2VybmVsLm9yZzsgcWVtdS1kZXZlbEBub25nbnUub3JnDQo+IFN1YmplY3Q6
IFJlOiB2ZmlvIGZvciBwbGF0Zm9ybSBkZXZpY2VzIC0gOS81LzIwMTIgLSBtaW51dGVzDQo+IA0K
PiBPbiBXZWQsIDIwMTMtMDktMTEgYXQgMTY6NDIgKzAwMDAsIFlvZGVyIFN0dWFydC1CMDgyNDgg
d3JvdGU6DQo+ID4NCj4gPiA+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gPiBGcm9t
OiBZb2RlciBTdHVhcnQtQjA4MjQ4DQo+ID4gPiBTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDA1
LCAyMDEzIDEyOjUxIFBNDQo+ID4gPiBUbzogV29vZCBTY290dC1CMDc0MjE7IFNldGhpIFZhcnVu
LUIxNjM5NTsgQmh1c2hhbiBCaGFyYXQtUjY1Nzc3Ow0KPiA+ID4gJ1BldGVyIE1heWRlbGwnOyAn
U2FudG9zaCBTaHVrbGEnOyAnQWxleCBXaWxsaWFtc29uJzsgJ0FsZXhhbmRlcg0KPiA+ID4gR3Jh
Zic7ICdBbnRvbmlvcyBNb3Rha2lzJzsgJ0NocmlzdG9mZmVyIERhbGwnOyAna2ltLnBoaWxsaXBz
QGxpbmFyby5vcmcnDQo+ID4gPiBDYzoga3ZtYXJtQGxpc3RzLmNzLmNvbHVtYmlhLmVkdTsgJ2t2
bS1wcGNAdmdlci5rZXJuZWwub3JnJzsgJ3FlbXUtDQo+ID4gPiBkZXZlbEBub25nbnUub3JnJw0K
PiA+ID4gU3ViamVjdDogdmZpbyBmb3IgcGxhdGZvcm0gZGV2aWNlcyAtIDkvNS8yMDEyIC0gbWlu
dXRlcw0KPiA+ID4NCj4gPiA+IFdlIGhhZCBhIGNhbGwgd2l0aCB0aG9zZSBpbnRlcmVzdGVkIGFu
ZC9vciB3b3JraW5nIG9uIHZmaW8gZm9yDQo+ID4gPiBwbGF0Zm9ybSBkZXZpY2VzLg0KPiA+ID4N
Cj4gPiA+IFBhcnRpY2lwYW50czogU2NvdHQgV29vZCwgVmFydW4gU2V0aGksIEJoYXJhdCBCaHVz
aGFuLCBQZXRlciBNYXlkZWxsLA0KPiA+ID4gICAgICAgICAgICAgICBTYW50b3NoIFNodWtsYSwg
QWxleCBXaWxsaWFtc29uLCBBbGV4YW5kZXIgR3JhZiwNCj4gPiA+ICAgICAgICAgICAgICAgQW50
b25pb3MgTW90YWtpcywgQ2hyaXN0b2ZmZXIgRGFsbCwgS2ltIFBoaWxsaXBzLA0KPiA+ID4gICAg
ICAgICAgICAgICBTdHVhcnQgWW9kZXINCj4gPiA+DQo+ID4gPiBTZXZlcmFsIGFzcGVjdHMgdG8g
dmZpbyBmb3IgcGxhdGZvcm0gZGV2aWNlczoNCj4gPiA+DQo+ID4gPiAxLiBJT01NVSBncm91cHMN
Cj4gPiA+DQo+ID4gPiAgLWlvbW11IGRyaXZlciBuZWVkcyB0byByZWdpc3RlciBhIGJ1cyBub3Rp
ZmllciBmb3IgdGhlIHBsYXRmb3JtIGJ1cw0KPiA+ID4gICBhbmQgY3JlYXRlIGdyb3VwcyBmb3Ig
cmVsZXZhbnQgcGxhdGZvcm0gZGV2aWNlcyAgLUFudG9uaW9zIGlzDQo+ID4gPiBsb29raW5nIGF0
IHRoaXMgZm9yIHNldmVyYWwgQVJNIElPTU1VcyAgLVBBTVUgKEZyZWVzY2FsZSkgZHJpdmVyDQo+
ID4gPiBhbHJlYWR5IGRvZXMgdGhpcw0KPiA+ID4NCj4gPiA+IDIuIHVuYmluZGluZyBkZXZpY2Ug
ZnJvbSBob3N0DQo+ID4gPg0KPiA+ID4gIFBDSToNCj4gPiA+ICAgIGVjaG8gMDAwMDowNjowZC4w
ID4NCj4gPiA+IC9zeXMvYnVzL3BjaS9kZXZpY2VzLzAwMDA6MDY6MGQuMC9kcml2ZXIvdW5iaW5k
DQo+ID4gPiAgUGxhdGZvcm06DQo+ID4gPiAgICBlY2hvIGZmZTEwMTMwMC5kbWEgPg0KPiA+ID4g
L3N5cy9idXMvcGxhdGZvcm0vZGV2aWNlcy9mZmUxMDEzMDAuZG1hL2RyaXZlci91bmJpbmQNCj4g
PiA+DQo+ID4gPiAgLWRvbid0IGJlbGlldmUgdGhlcmUgYXJlIGlzc3VlcyBvciB3b3JrIHRvIGRv
IGhlcmUNCj4gPiA+DQo+ID4gPiAzLiBiaW5kaW5nIGRldmljZSB0byB2ZmlvLXBsYXRmb3JtIGRy
aXZlcg0KPiA+ID4NCj4gPiA+ICBQQ0k6DQo+ID4gPiAgICBlY2hvIDExMDIgMDAwMiA+IC9zeXMv
YnVzL3BjaS9kcml2ZXJzL3ZmaW8tcGNpL25ld19pZA0KPiA+ID4NCj4gPiA+ICAtdGhpcyBpcyBw
cm9iYWJseSB0aGUgbGVhc3QgdW5kZXJzdG9vZCBpc3N1ZS0tIHBsYXRmb3JtIGRyaXZlcnMNCj4g
PiA+ICAgcmVnaXN0ZXIgdGhlbXNlbHZlcyB3aXRoIHRoZSBidXMgZm9yIGEgc3BlY2lmaWMgIm5h
bWUiDQo+ID4gPiAgIHN0cmluZy4gIFRoYXQgaXMgbWF0Y2hlZCB3aXRoIGRldmljZSB0cmVlICJj
b21wYXRpYmxlIiBzdHJpbmdzDQo+ID4gPiAgIGxhdGVyIHRvIGJpbmQgYSBkZXZpY2UgdG8gYSBk
cml2ZXIgIC13ZSB3YW50IGlzIHRvIGhhdmUgdGhlDQo+ID4gPiB2ZmlvLXBsYXRmb3JtIGRyaXZl
ciB0byBkeW5hbWljYWxseSBiaW5kDQo+ID4gPiAgIHRvIGEgdmFyaWV0eSBvZiBwbGF0Zm9ybSBk
ZXZpY2VzIHByZXZpb3VzbHkgdW5rbm93biB0bw0KPiA+ID4gICB2ZmlvLXBsYXRmb3JtDQo+ID4g
PiAgLWlkZWFsbHkgdW5iaW5kaW5nIGFuZCBiaW5kaW5nIGNvdWxkIGJlIGFuIGF0b21pYyBvcGVy
YXRpb24gIC1BbGV4DQo+ID4gPiBXIHBvaW50ZWQgb3V0IHRoYXQgeDg2IGNvdWxkIGxldmVyYWdl
IHRoaXMgd29yayBzbw0KPiA+ID4gICBrZWVwIHRoYXQgaW4gbWluZCBpbiB3aGF0IHdlIGRlc2ln
biAgLUtpbSBQaGlsbGlwcyAoTGluYXJvKSB3aWxsDQo+ID4gPiBzdGFydCB3b3JraW5nIG9uIHRo
aXMNCj4gPg0KPiA+IE9uZSB0aGluZyB3ZSBkaWRuJ3QgZGlzY3VzcyBuZWVkcyB0byBiZSBjb25z
aWRlcmVkIChwcm9iYWJseSBieSBLaW0NCj4gPiB3aG8gaXMgbG9va2luZyBhdCB0aGUgJ2JpbmRp
bmcgZGV2aWNlJyBpc3N1ZSkgaXMgYXJvdW5kIHJldHVybmluZyBhDQo+ID4gcGFzc3RocnUgZGV2
aWNlIGJhY2sgdG8gdGhlIGhvc3QuDQo+ID4NCj4gPiBBZnRlciBhIHBsYXRmb3JtIGRldmljZSBo
YXMgYmVlbiBib3VuZCB0byB2ZmlvIGFuZCBpcyBpbiB1c2UgYnkgdXNlcg0KPiA+IHNwYWNlIG9y
IGEgdmlydHVhbCBtYWNoaW5lLCB3ZSBhbHNvIG5lZWQgdG8gYmUgYWJsZSB0byB1bndpbmQgYWxs
IHRoYXQNCj4gPiBhbmQgcmV0dXJuIHRoZSBkZXZpY2UgYmFjayB0byB0aGUgaG9zdCBpbiBhIHNh
bmUgc3RhdGUuDQo+ID4NCj4gPiBXaGF0IGhhcHBlbnMgd2hlbiB1c2VyIHNwYWNlIGV4aXRzIGFu
ZCB0aGUgdmZpbyBmaWxlIGRlc2NyaXB0b3JzIGFyZQ0KPiA+IGNsb3NlZD8NCj4gDQo+IEZvciBy
ZWZlcmVuY2UsIGV4cGVjdGF0aW9ucyBvZiBob3cgdmZpby1wY2kgaGFuZGxlcyB0aGVzZSBzaXR1
YXRpb25zOg0KPiANCj4gRm9yIHZmaW8tcGNpLCB3aGVuIHRoZSByZWZlcmVuY2UgY291bnQgb24g
dGhlIGRldmljZSBmZCBkcm9wcyB0byB6ZXJvIHdlIGNhbGwgYQ0KPiBkZXZpY2UgZGlzYWJsZSBm
dW5jdGlvbiB0aGF0IGluY2x1ZGVzIGRpc2FibGluZyB0aGUgYnVzIG1hc3RlciBiaXQgaW4gY29u
ZmlnDQo+IHNwYWNlIHN0b3Agb25nb2luZyBETUEuDQoNClRoZXJlIGlzIG5vIGJ1cyBtYXN0ZXJp
bmcgZm9yIHBsYXRmb3JtIGRldmljZXMgYW5kIGRldmljZSBkaXNhYmxpbmcgaXMgbm90IGEgc3Rh
bmRhcmQgbGlrZSBQQ0kuIERvIHlvdSB0aGluayB0aGF0IHdlIG1pZ2h0IG5lZWQgdG8gZG8gc29t
ZSBkZXZpY2Ugc3BlY2lmaWMgaGFuZGxpbmcuIEZvciBleGFtcGxlIGZvciBETUEgY29udHJvbGxl
ciwgd2UgbmVlZCB0byBhdGxlYXN0IGRpc2FibGUgdGhlIERNQSBjb250cm9sbGVyIGFuZCBtYXNr
IGl0cyBpbnRlcnJ1cHRzIChtYXkgYmUgZW5zdXJlIHRoYXQgdGhlcmUgaXMgbm8gcGVuZGluZy9y
dW5uaW5nIGRtYSB0cmFuc2FjdGlvbiwgcmVsZWFzZSBpcnFzIGV0YykuIEFzIHdlIGFyZSBub3Qg
eWV0IGhhdmluZyBhbnkgbGlua2FnZSB0byByZXNwZWN0aXZlIGtlcm5lbCBkcml2ZXIgdGhlbiBJ
IGFtIG5vdCBzdXJlIGhvdyB3ZSB3aWxsIGRvIHRoYXQgc3BlY2lmaWMgaGFuZGxpbmc/DQoNCj4g
DQo+ID4gV2hhdCBpZiB0aGUgZGV2aWNlIGlzIHN0aWxsIGFjdGl2ZSBhbmQgZG9pbmcgYnVzDQo+
ID4gbWFzdGVyaW5nPyAgIChlLmcuIGEgVk0gY3Jhc2hlZCBjYXVzaW5nIGEgUUVNVQ0KPiA+IGV4
aXQpDQo+IA0KPiBJZiB0aGUgVk0gY3Jhc2hlcyB0aGUgdmZpbyBmZHMgZ2V0IHJlbGVhc2VkIHJl
c3VsdGluZyBpbiB0aGUgYWJvdmUgb3Bwb3J0dW5pdHkNCj4gZm9yIHRoZSB2ZmlvIGRldmljZSBk
cml2ZXIgdG8gcXVpZXNjZSB0aGUgZGV2aWNlLg0KDQpJIHRoaW5rIHRoZSBxdWllc2luZyBvZiBk
ZXZpY2VzIHdpdGggd2UgZGV2aWNlIHNwZWNpZmljLCBzbyB0aGUgZ2VuZXJpYyB2ZmlvPXBsYXRm
b3JtIGRyaXZlciBtYXkgbm90IGJlIGFibGUgdG8gaGFuZGxlIHRoYXQsIHJpZ2h0Pw0KDQo+IA0K
PiA+IEhvdyBjYW4gdGhlIHZmaW8tcGxhdGZvcm0gbGF5ZXIgaW4gdGhlIGhvc3Qga2VybmVsIGdl
dCBhIHNwZWNpZmljDQo+ID4gZGV2aWNlIGluIGEgc2FuZSBzdGF0ZT8NCj4gDQo+IEl0J3Mgbm90
IGVhc3kgb24gcGNpIGVpdGhlci4gIFdlIHNhdmUgY29uZmlnIHNwYWNlIHByaW9yIHRvIGV4cG9z
aW5nIHRoZSBkZXZpY2UNCj4gYW5kIHJlc3RvcmUgY29uZmlnIHNwYWNlIGxhdGVyLCBidXQgaXQn
cyBub3QgY29tcGxldGUuICBXZSBtb3N0bHkgcmVseSBvbiBkZXZpY2UNCj4gKGZ1bmN0aW9uKSBy
ZXNldHMsIHRvIHB1dCB0aGluZ3MgaW4gYSBzYW5lIHN0YXRlLCBidXQgdGhvc2UgYXJlbid0IGFs
d2F5cw0KPiBzdXBwb3J0ZWQuDQoNCkFsbCBwbGF0Zm9ybSBkZXZpY2VzIGFsc28gbWF5IG5vdCBo
YXZlIHJlc2V0IGNhcGFiaWxpdHkgKGFuZCBpZiBhbnkgdGhlbiBub3QgZ2VuZXJpYyB3YXkgZm9y
IGFsbCBkZXZpY2VzKS4NCg0KPiAgSSBqdXN0IGludHJvZHVjZWQgcGF0Y2hlcyBmb3IgdjMuMTIg
dGhhdCBlbmFibGUgYSBQQ0kgYnVzIHJlc2V0DQo+IGludGVyZmFjZSwgYnV0IGl0J3MgbW9zdGx5
IHVzZWZ1bCBmb3IgdXNlcnNwYWNlLCBzaW5jZSBvbiBQQ0kgaXQncyBvZnRlbiB0aGUNCj4gY2Fz
ZSB0aGF0IGEgYnVzIGNvbnRhaW5zIG11bHRpcGxlIGRldmljZXMgd2hpY2ggZG9uJ3QgbmVjZXNz
YXJpbHkgYWxpZ24gdG8gaW9tbXUNCj4gZ3JvdXAgYm91bmRhcmllcy4NCj4gDQo+ID4gV2hlbiBh
IHBsYWZvcm0gZGV2aWNlIGlzICd1bmJvdW5kJyBmcm9tIHZmaW8sIHdoYXQgc3BlY2lmaWNhbGx5
DQo+ID4gaGFwcGVucyB0byB0aGUgZGV2aWNlPw0KPiANCj4gVGhlIGRyaXZlciByZW1vdmUgZnVu
Y3Rpb24gaXMgY2FsbGVkLCB3aGljaCBuZWVkcyB0byBjYWxsIHZmaW9fZGVsX2dyb3VwX2Rldigp
LA0KPiB3aGljaCB3aWxsIGJsb2NrIGlmIHRoZSBkZXZpY2UgaXMgaW4gdXNlLiAgSWYgaXQncyB1
bnVzZWQsIHRoZSByZWxlYXNlIGZ1bmN0aW9uDQo+IHNob3VsZCBoYXZlIGFscmVhZHkgYmVlbiBj
YWxsZWQgYW5kIHRoZSBkZXZpY2Ugc2hvdWxkIGJlIGluIGEgcXVpZXNjZWQgc3RhdGUuDQoNCldo
YXQgZG9lcyBpdCBtZWFuIGJ5IHF1aWVzY2VkIHN0YXRlIGhlcmU/IERvZXMgdGhpcyBtZWFuOg0K
IC0gRGV2aWNlIGRvZXMgbm90IHRha2UgbW9yZSB3b3JrIGZyb20gc29mdHdhcmU7IFRoaXMgd2ls
bCBkb25lIGlmIGRldmljZSBpcyBub3QgYXR0YWNoZWQgdG8gYW55IGRyaXZlci4NCiAtIERldmlj
ZSBkb2VzIG5vdCBjcmVhdGUgd29yayBmb3Igc29mdHdhcmUgKG5vIGludGVycnVwdCBldGMpLiBU
aGlzIGNhbiBhY2hpZXZlZCBpZiBpbnRlcnJ1cHRzIG1hc2tlZCBhdCBkZXZpY2UvUElDIGFuZCBk
ZXZpY2UgaW4gZGlzYWJsZWQuDQoNClRoYW5rcw0KLUJoYXJhdA0KDQo+IE9uIHg4Ni9wY2ksIHRo
ZXJlIGlzIG5vIGF1dG8gcHJvYmUgb2YgZGV2aWNlcyB3aGVuIHVuYm91bmQsIHNvIHRoZSBkZXZp
Y2Ugc2l0cw0KPiB3aXRob3V0IGEgZHJpdmVyIHVudGlsIG1hbnVhbGx5IGJvdW5kIG9yIHVudGls
IHNvbWV0aGluZyBjYXVzZXMgYW4gYXV0byBwcm9iZS4NCj4gVGhhbmtzLA0KPiANCj4gQWxleA0K
PiANCj4gPiBQbGF0Zm9ybSBkZXZpY2VzIGRvbid0IGhhdmUgZ2VuZXJpYyBtZWNoYW5pc21zIGxp
a2Ugb24gUENJIHRvIGRpc2FibGUNCj4gPiBidXMgbWFzdGVyaW5nIG9yIGV2ZW4gZGlzYWJsZSBv
ciByZXNldCBhIGRldmljZS4NCj4gPg0KPiA+IEhhdmVuJ3QgdGhvdWdodCB0aHJvdWdoIGFsbCB0
aGlzIHlldCwgYnV0IGp1c3QgcmFpc2luZyBzb21lIGlzc3VlcyBJDQo+ID4gc2VlLg0KPiA+DQo+
ID4gUmVnYXJkcywNCj4gPiBTdHVhcnQNCj4gPg0KPiA+DQo+ID4NCj4gPg0KPiANCj4gDQo+IA0K
DQo

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Wednesday, September 11, 2013 10:45 PM
> > To: Yoder Stuart-B08248
> > Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
> > Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
> > ppc@vger.kernel.org; qemu-devel@nongnu.org
> > Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> > 
> > On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> > >
> > > > -----Original Message-----
> > > > From: Yoder Stuart-B08248
> > > > Sent: Thursday, September 05, 2013 12:51 PM
> > > > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> > > > 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
> > > > Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > > > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > > > devel@nongnu.org'
> > > > Subject: vfio for platform devices - 9/5/2012 - minutes
> > > >
> > > > We had a call with those interested and/or working on vfio for
> > > > platform devices.
> > > >
> > > > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> > > >               Santosh Shukla, Alex Williamson, Alexander Graf,
> > > >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> > > >               Stuart Yoder
> > > >
> > > > Several aspects to vfio for platform devices:
> > > >
> > > > 1. IOMMU groups
> > > >
> > > >  -iommu driver needs to register a bus notifier for the platform bus
> > > >   and create groups for relevant platform devices  -Antonios is
> > > > looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
> > > > already does this
> > > >
> > > > 2. unbinding device from host
> > > >
> > > >  PCI:
> > > >    echo 0000:06:0d.0 >
> > > > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> > > >  Platform:
> > > >    echo ffe101300.dma >
> > > > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > > >
> > > >  -don't believe there are issues or work to do here
> > > >
> > > > 3. binding device to vfio-platform driver
> > > >
> > > >  PCI:
> > > >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > >
> > > >  -this is probably the least understood issue-- platform drivers
> > > >   register themselves with the bus for a specific "name"
> > > >   string.  That is matched with device tree "compatible" strings
> > > >   later to bind a device to a driver  -we want is to have the
> > > > vfio-platform driver to dynamically bind
> > > >   to a variety of platform devices previously unknown to
> > > >   vfio-platform
> > > >  -ideally unbinding and binding could be an atomic operation  -Alex
> > > > W pointed out that x86 could leverage this work so
> > > >   keep that in mind in what we design  -Kim Phillips (Linaro) will
> > > > start working on this
> > >
> > > One thing we didn't discuss needs to be considered (probably by Kim
> > > who is looking at the 'binding device' issue) is around returning a
> > > passthru device back to the host.
> > >
> > > After a platform device has been bound to vfio and is in use by user
> > > space or a virtual machine, we also need to be able to unwind all that
> > > and return the device back to the host in a sane state.
> > >
> > > What happens when user space exits and the vfio file descriptors are
> > > closed?
> > 
> > For reference, expectations of how vfio-pci handles these situations:
> > 
> > For vfio-pci, when the reference count on the device fd drops to zero we call a
> > device disable function that includes disabling the bus master bit in config
> > space stop ongoing DMA.
> 
> There is no bus mastering for platform devices

Sure there is -- that's just PCI's name for DMA.

> and device disabling is not a standard like PCI. Do you think that we might need to do some
> device specific handling.

It would be better if we could avoid the need for device specific
handling.  DMA can be disabled via the IOMMU.

The tricky part is how to enable the device in the next user; as I wrote
elsewhere in the thread I think this needs to be something the user is
aware of, so that the user quiesces the device prior to enabling IOMMU
mappings.

> >  I just introduced patches for v3.12 that enable a PCI bus reset
> > interface, but it's mostly useful for userspace, since on PCI it's often the
> > case that a bus contains multiple devices which don't necessarily align to iommu
> > group boundaries.
> > 
> > > When a plaform device is 'unbound' from vfio, what specifically
> > > happens to the device?
> > 
> > The driver remove function is called, which needs to call vfio_del_group_dev(),
> > which will block if the device is in use.  If it's unused, the release function
> > should have already been called and the device should be in a quiesced state.
> 
> What does it mean by quiesced state here? Does this mean:
>  - Device does not take more work from software; This will done if device is not attached to any driver.
>  - Device does not create work for software (no interrupt etc). This can achieved if interrupts masked at device/PIC and device in disabled.

It at least means that the device will not do any DMA or generate any
interrupts.

-Scott

Re: vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Wednesday, September 11, 2013 10:45 PM
> > To: Yoder Stuart-B08248
> > Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> > Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
> > Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
> > ppc@vger.kernel.org; qemu-devel@nongnu.org
> > Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> > 
> > On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> > >
> > > > -----Original Message-----
> > > > From: Yoder Stuart-B08248
> > > > Sent: Thursday, September 05, 2013 12:51 PM
> > > > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> > > > 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
> > > > Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
> > > > Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
> > > > devel@nongnu.org'
> > > > Subject: vfio for platform devices - 9/5/2012 - minutes
> > > >
> > > > We had a call with those interested and/or working on vfio for
> > > > platform devices.
> > > >
> > > > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> > > >               Santosh Shukla, Alex Williamson, Alexander Graf,
> > > >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> > > >               Stuart Yoder
> > > >
> > > > Several aspects to vfio for platform devices:
> > > >
> > > > 1. IOMMU groups
> > > >
> > > >  -iommu driver needs to register a bus notifier for the platform bus
> > > >   and create groups for relevant platform devices  -Antonios is
> > > > looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
> > > > already does this
> > > >
> > > > 2. unbinding device from host
> > > >
> > > >  PCI:
> > > >    echo 0000:06:0d.0 >
> > > > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> > > >  Platform:
> > > >    echo ffe101300.dma >
> > > > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > > >
> > > >  -don't believe there are issues or work to do here
> > > >
> > > > 3. binding device to vfio-platform driver
> > > >
> > > >  PCI:
> > > >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > > >
> > > >  -this is probably the least understood issue-- platform drivers
> > > >   register themselves with the bus for a specific "name"
> > > >   string.  That is matched with device tree "compatible" strings
> > > >   later to bind a device to a driver  -we want is to have the
> > > > vfio-platform driver to dynamically bind
> > > >   to a variety of platform devices previously unknown to
> > > >   vfio-platform
> > > >  -ideally unbinding and binding could be an atomic operation  -Alex
> > > > W pointed out that x86 could leverage this work so
> > > >   keep that in mind in what we design  -Kim Phillips (Linaro) will
> > > > start working on this
> > >
> > > One thing we didn't discuss needs to be considered (probably by Kim
> > > who is looking at the 'binding device' issue) is around returning a
> > > passthru device back to the host.
> > >
> > > After a platform device has been bound to vfio and is in use by user
> > > space or a virtual machine, we also need to be able to unwind all that
> > > and return the device back to the host in a sane state.
> > >
> > > What happens when user space exits and the vfio file descriptors are
> > > closed?
> > 
> > For reference, expectations of how vfio-pci handles these situations:
> > 
> > For vfio-pci, when the reference count on the device fd drops to zero we call a
> > device disable function that includes disabling the bus master bit in config
> > space stop ongoing DMA.
> 
> There is no bus mastering for platform devices

Sure there is -- that's just PCI's name for DMA.

> and device disabling is not a standard like PCI. Do you think that we might need to do some
> device specific handling.

It would be better if we could avoid the need for device specific
handling.  DMA can be disabled via the IOMMU.

The tricky part is how to enable the device in the next user; as I wrote
elsewhere in the thread I think this needs to be something the user is
aware of, so that the user quiesces the device prior to enabling IOMMU
mappings.

> >  I just introduced patches for v3.12 that enable a PCI bus reset
> > interface, but it's mostly useful for userspace, since on PCI it's often the
> > case that a bus contains multiple devices which don't necessarily align to iommu
> > group boundaries.
> > 
> > > When a plaform device is 'unbound' from vfio, what specifically
> > > happens to the device?
> > 
> > The driver remove function is called, which needs to call vfio_del_group_dev(),
> > which will block if the device is in use.  If it's unused, the release function
> > should have already been called and the device should be in a quiesced state.
> 
> What does it mean by quiesced state here? Does this mean:
>  - Device does not take more work from software; This will done if device is not attached to any driver.
>  - Device does not create work for software (no interrupt etc). This can achieved if interrupts masked at device/PIC and device in disabled.

It at least means that the device will not do any DMA or generate any
interrupts.

-Scott

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Alexander Graf]

On 12.09.2013, at 13:10, Scott Wood wrote:

> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
>> 
>>> -----Original Message-----
>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> Sent: Wednesday, September 11, 2013 10:45 PM
>>> To: Yoder Stuart-B08248
>>> Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
>>> Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
>>> Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
>>> ppc@vger.kernel.org; qemu-devel@nongnu.org
>>> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
>>> 
>>> On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
>>>> 
>>>>> -----Original Message-----
>>>>> From: Yoder Stuart-B08248
>>>>> Sent: Thursday, September 05, 2013 12:51 PM
>>>>> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
>>>>> 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
>>>>> Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
>>>>> Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
>>>>> devel@nongnu.org'
>>>>> Subject: vfio for platform devices - 9/5/2012 - minutes
>>>>> 
>>>>> We had a call with those interested and/or working on vfio for
>>>>> platform devices.
>>>>> 
>>>>> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>>>>>              Santosh Shukla, Alex Williamson, Alexander Graf,
>>>>>              Antonios Motakis, Christoffer Dall, Kim Phillips,
>>>>>              Stuart Yoder
>>>>> 
>>>>> Several aspects to vfio for platform devices:
>>>>> 
>>>>> 1. IOMMU groups
>>>>> 
>>>>> -iommu driver needs to register a bus notifier for the platform bus
>>>>>  and create groups for relevant platform devices  -Antonios is
>>>>> looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
>>>>> already does this
>>>>> 
>>>>> 2. unbinding device from host
>>>>> 
>>>>> PCI:
>>>>>   echo 0000:06:0d.0 >
>>>>> /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>>>>> Platform:
>>>>>   echo ffe101300.dma >
>>>>> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
>>>>> 
>>>>> -don't believe there are issues or work to do here
>>>>> 
>>>>> 3. binding device to vfio-platform driver
>>>>> 
>>>>> PCI:
>>>>>   echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
>>>>> 
>>>>> -this is probably the least understood issue-- platform drivers
>>>>>  register themselves with the bus for a specific "name"
>>>>>  string.  That is matched with device tree "compatible" strings
>>>>>  later to bind a device to a driver  -we want is to have the
>>>>> vfio-platform driver to dynamically bind
>>>>>  to a variety of platform devices previously unknown to
>>>>>  vfio-platform
>>>>> -ideally unbinding and binding could be an atomic operation  -Alex
>>>>> W pointed out that x86 could leverage this work so
>>>>>  keep that in mind in what we design  -Kim Phillips (Linaro) will
>>>>> start working on this
>>>> 
>>>> One thing we didn't discuss needs to be considered (probably by Kim
>>>> who is looking at the 'binding device' issue) is around returning a
>>>> passthru device back to the host.
>>>> 
>>>> After a platform device has been bound to vfio and is in use by user
>>>> space or a virtual machine, we also need to be able to unwind all that
>>>> and return the device back to the host in a sane state.
>>>> 
>>>> What happens when user space exits and the vfio file descriptors are
>>>> closed?
>>> 
>>> For reference, expectations of how vfio-pci handles these situations:
>>> 
>>> For vfio-pci, when the reference count on the device fd drops to zero we call a
>>> device disable function that includes disabling the bus master bit in config
>>> space stop ongoing DMA.
>> 
>> There is no bus mastering for platform devices
> 
> Sure there is -- that's just PCI's name for DMA.
> 
>> and device disabling is not a standard like PCI. Do you think that we might need to do some
>> device specific handling.
> 
> It would be better if we could avoid the need for device specific
> handling.  DMA can be disabled via the IOMMU.
> 
> The tricky part is how to enable the device in the next user; as I wrote
> elsewhere in the thread I think this needs to be something the user is
> aware of, so that the user quiesces the device prior to enabling IOMMU
> mappings.

Or resets it. I think we can make it mandatory to VFIO users (and Linux drivers) to reset the device prior to mapping it. QEMU needs to have device specific code to assemble its device tree fragment anyways, so it can just as well reset the device before it enables it to DMA.

Alex

Re: vfio for platform devices - 9/5/2012 - minutes

[Alexander Graf]

On 12.09.2013, at 13:10, Scott Wood wrote:

> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
>> 
>>> -----Original Message-----
>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> Sent: Wednesday, September 11, 2013 10:45 PM
>>> To: Yoder Stuart-B08248
>>> Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
>>> Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
>>> Dall'; 'kim.phillips@linaro.org'; kvmarm@lists.cs.columbia.edu; kvm-
>>> ppc@vger.kernel.org; qemu-devel@nongnu.org
>>> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
>>> 
>>> On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
>>>> 
>>>>> -----Original Message-----
>>>>> From: Yoder Stuart-B08248
>>>>> Sent: Thursday, September 05, 2013 12:51 PM
>>>>> To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
>>>>> 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
>>>>> Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'kim.phillips@linaro.org'
>>>>> Cc: kvmarm@lists.cs.columbia.edu; 'kvm-ppc@vger.kernel.org'; 'qemu-
>>>>> devel@nongnu.org'
>>>>> Subject: vfio for platform devices - 9/5/2012 - minutes
>>>>> 
>>>>> We had a call with those interested and/or working on vfio for
>>>>> platform devices.
>>>>> 
>>>>> Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
>>>>>              Santosh Shukla, Alex Williamson, Alexander Graf,
>>>>>              Antonios Motakis, Christoffer Dall, Kim Phillips,
>>>>>              Stuart Yoder
>>>>> 
>>>>> Several aspects to vfio for platform devices:
>>>>> 
>>>>> 1. IOMMU groups
>>>>> 
>>>>> -iommu driver needs to register a bus notifier for the platform bus
>>>>>  and create groups for relevant platform devices  -Antonios is
>>>>> looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
>>>>> already does this
>>>>> 
>>>>> 2. unbinding device from host
>>>>> 
>>>>> PCI:
>>>>>   echo 0000:06:0d.0 >
>>>>> /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
>>>>> Platform:
>>>>>   echo ffe101300.dma >
>>>>> /sys/bus/platform/devices/ffe101300.dma/driver/unbind
>>>>> 
>>>>> -don't believe there are issues or work to do here
>>>>> 
>>>>> 3. binding device to vfio-platform driver
>>>>> 
>>>>> PCI:
>>>>>   echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
>>>>> 
>>>>> -this is probably the least understood issue-- platform drivers
>>>>>  register themselves with the bus for a specific "name"
>>>>>  string.  That is matched with device tree "compatible" strings
>>>>>  later to bind a device to a driver  -we want is to have the
>>>>> vfio-platform driver to dynamically bind
>>>>>  to a variety of platform devices previously unknown to
>>>>>  vfio-platform
>>>>> -ideally unbinding and binding could be an atomic operation  -Alex
>>>>> W pointed out that x86 could leverage this work so
>>>>>  keep that in mind in what we design  -Kim Phillips (Linaro) will
>>>>> start working on this
>>>> 
>>>> One thing we didn't discuss needs to be considered (probably by Kim
>>>> who is looking at the 'binding device' issue) is around returning a
>>>> passthru device back to the host.
>>>> 
>>>> After a platform device has been bound to vfio and is in use by user
>>>> space or a virtual machine, we also need to be able to unwind all that
>>>> and return the device back to the host in a sane state.
>>>> 
>>>> What happens when user space exits and the vfio file descriptors are
>>>> closed?
>>> 
>>> For reference, expectations of how vfio-pci handles these situations:
>>> 
>>> For vfio-pci, when the reference count on the device fd drops to zero we call a
>>> device disable function that includes disabling the bus master bit in config
>>> space stop ongoing DMA.
>> 
>> There is no bus mastering for platform devices
> 
> Sure there is -- that's just PCI's name for DMA.
> 
>> and device disabling is not a standard like PCI. Do you think that we might need to do some
>> device specific handling.
> 
> It would be better if we could avoid the need for device specific
> handling.  DMA can be disabled via the IOMMU.
> 
> The tricky part is how to enable the device in the next user; as I wrote
> elsewhere in the thread I think this needs to be something the user is
> aware of, so that the user quiesces the device prior to enabling IOMMU
> mappings.

Or resets it. I think we can make it mandatory to VFIO users (and Linux drivers) to reset the device prior to mapping it. QEMU needs to have device specific code to assemble its device tree fragment anyways, so it can just as well reset the device before it enables it to DMA.

Alex

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
> On 12.09.2013, at 13:10, Scott Wood wrote:
> 
> > On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> >> and device disabling is not a standard like PCI. Do you think that we might need to do some
> >> device specific handling.
> > 
> > It would be better if we could avoid the need for device specific
> > handling.  DMA can be disabled via the IOMMU.
> > 
> > The tricky part is how to enable the device in the next user; as I wrote
> > elsewhere in the thread I think this needs to be something the user is
> > aware of, so that the user quiesces the device prior to enabling IOMMU
> > mappings.
> 
> Or resets it. I think we can make it mandatory to VFIO users (and Linux
> drivers) to reset the device prior to mapping it.

Reset is one way to quiesce it, but as Bharat pointed out not all
devices make that possible (even with device specific knowledge).  In
particular, Freescale datapath stuff has problems with this.

> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
> it can just as well reset the device before it enables it to DMA.

Now you're adding to the depth of knowledge QEMU needs to have about a
device.  It's not that big of a deal if it's just "set this bit and wait
for it to clear", but it could be a pain with a device that can't be
reset and has a complicated sequence needed to quiesce -- and as you
noted above, Linux drivers would need to take care of this anyway.
Doing it in QEMU would still help with non-Linux guests and avoid the
need for a hypercall to indicate when the IOMMU can be opened, though.

-Scott

Re: vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
> On 12.09.2013, at 13:10, Scott Wood wrote:
> 
> > On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> >> and device disabling is not a standard like PCI. Do you think that we might need to do some
> >> device specific handling.
> > 
> > It would be better if we could avoid the need for device specific
> > handling.  DMA can be disabled via the IOMMU.
> > 
> > The tricky part is how to enable the device in the next user; as I wrote
> > elsewhere in the thread I think this needs to be something the user is
> > aware of, so that the user quiesces the device prior to enabling IOMMU
> > mappings.
> 
> Or resets it. I think we can make it mandatory to VFIO users (and Linux
> drivers) to reset the device prior to mapping it.

Reset is one way to quiesce it, but as Bharat pointed out not all
devices make that possible (even with device specific knowledge).  In
particular, Freescale datapath stuff has problems with this.

> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
> it can just as well reset the device before it enables it to DMA.

Now you're adding to the depth of knowledge QEMU needs to have about a
device.  It's not that big of a deal if it's just "set this bit and wait
for it to clear", but it could be a pain with a device that can't be
reset and has a complicated sequence needed to quiesce -- and as you
noted above, Linux drivers would need to take care of this anyway.
Doing it in QEMU would still help with non-Linux guests and avoid the
need for a hypercall to indicate when the IOMMU can be opened, though.

-Scott

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Alexander Graf]

On 12.09.2013, at 16:45, Scott Wood wrote:

> On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
>> On 12.09.2013, at 13:10, Scott Wood wrote:
>> 
>>> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
>>>> and device disabling is not a standard like PCI. Do you think that we might need to do some
>>>> device specific handling.
>>> 
>>> It would be better if we could avoid the need for device specific
>>> handling.  DMA can be disabled via the IOMMU.
>>> 
>>> The tricky part is how to enable the device in the next user; as I wrote
>>> elsewhere in the thread I think this needs to be something the user is
>>> aware of, so that the user quiesces the device prior to enabling IOMMU
>>> mappings.
>> 
>> Or resets it. I think we can make it mandatory to VFIO users (and Linux
>> drivers) to reset the device prior to mapping it.
> 
> Reset is one way to quiesce it, but as Bharat pointed out not all
> devices make that possible (even with device specific knowledge).  In
> particular, Freescale datapath stuff has problems with this.

I'm sure once Freescale datapath drivers are upstream in Linux we can also get that code upstream in QEMU :).

>> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
>> it can just as well reset the device before it enables it to DMA.
> 
> Now you're adding to the depth of knowledge QEMU needs to have about a
> device.  It's not that big of a deal if it's just "set this bit and wait
> for it to clear", but it could be a pain with a device that can't be
> reset and has a complicated sequence needed to quiesce -- and as you
> noted above, Linux drivers would need to take care of this anyway.
> Doing it in QEMU would still help with non-Linux guests and avoid the
> need for a hypercall to indicate when the IOMMU can be opened, though.

Yes, and you don't want to burden the Linux driver with a situation it doesn't face in non-virtualized environments :).


Alex

Re: vfio for platform devices - 9/5/2012 - minutes

[Alexander Graf]

On 12.09.2013, at 16:45, Scott Wood wrote:

> On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
>> On 12.09.2013, at 13:10, Scott Wood wrote:
>> 
>>> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
>>>> and device disabling is not a standard like PCI. Do you think that we might need to do some
>>>> device specific handling.
>>> 
>>> It would be better if we could avoid the need for device specific
>>> handling.  DMA can be disabled via the IOMMU.
>>> 
>>> The tricky part is how to enable the device in the next user; as I wrote
>>> elsewhere in the thread I think this needs to be something the user is
>>> aware of, so that the user quiesces the device prior to enabling IOMMU
>>> mappings.
>> 
>> Or resets it. I think we can make it mandatory to VFIO users (and Linux
>> drivers) to reset the device prior to mapping it.
> 
> Reset is one way to quiesce it, but as Bharat pointed out not all
> devices make that possible (even with device specific knowledge).  In
> particular, Freescale datapath stuff has problems with this.

I'm sure once Freescale datapath drivers are upstream in Linux we can also get that code upstream in QEMU :).

>> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
>> it can just as well reset the device before it enables it to DMA.
> 
> Now you're adding to the depth of knowledge QEMU needs to have about a
> device.  It's not that big of a deal if it's just "set this bit and wait
> for it to clear", but it could be a pain with a device that can't be
> reset and has a complicated sequence needed to quiesce -- and as you
> noted above, Linux drivers would need to take care of this anyway.
> Doing it in QEMU would still help with non-Linux guests and avoid the
> need for a hypercall to indicate when the IOMMU can be opened, though.

Yes, and you don't want to burden the Linux driver with a situation it doesn't face in non-virtualized environments :).


Alex

Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 16:48 -0500, Alexander Graf wrote:
> On 12.09.2013, at 16:45, Scott Wood wrote:
> 
> > On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
> >> On 12.09.2013, at 13:10, Scott Wood wrote:
> >> 
> >>> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> >>>> and device disabling is not a standard like PCI. Do you think that we might need to do some
> >>>> device specific handling.
> >>> 
> >>> It would be better if we could avoid the need for device specific
> >>> handling.  DMA can be disabled via the IOMMU.
> >>> 
> >>> The tricky part is how to enable the device in the next user; as I wrote
> >>> elsewhere in the thread I think this needs to be something the user is
> >>> aware of, so that the user quiesces the device prior to enabling IOMMU
> >>> mappings.
> >> 
> >> Or resets it. I think we can make it mandatory to VFIO users (and Linux
> >> drivers) to reset the device prior to mapping it.
> > 
> > Reset is one way to quiesce it, but as Bharat pointed out not all
> > devices make that possible (even with device specific knowledge).  In
> > particular, Freescale datapath stuff has problems with this.
> 
> I'm sure once Freescale datapath drivers are upstream in Linux we can also get that code upstream in QEMU :).

Yeah, but the fewer places it lives, the better. :-)

> >> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
> >> it can just as well reset the device before it enables it to DMA.
> > 
> > Now you're adding to the depth of knowledge QEMU needs to have about a
> > device.  It's not that big of a deal if it's just "set this bit and wait
> > for it to clear", but it could be a pain with a device that can't be
> > reset and has a complicated sequence needed to quiesce -- and as you
> > noted above, Linux drivers would need to take care of this anyway.
> > Doing it in QEMU would still help with non-Linux guests and avoid the
> > need for a hypercall to indicate when the IOMMU can be opened, though.
> 
> Yes, and you don't want to burden the Linux driver with a situation it doesn't face in non-virtualized environments :).

The driver does need to handle it in non-virtualized environments, if
you want to be able to return a device to the host driver after a VFIO
user has had it.  It will also be needed for kexec.

-Scott

Re: vfio for platform devices - 9/5/2012 - minutes

[Scott Wood]

On Thu, 2013-09-12 at 16:48 -0500, Alexander Graf wrote:
> On 12.09.2013, at 16:45, Scott Wood wrote:
> 
> > On Thu, 2013-09-12 at 16:23 -0500, Alexander Graf wrote:
> >> On 12.09.2013, at 13:10, Scott Wood wrote:
> >> 
> >>> On Thu, 2013-09-12 at 04:18 -0500, Bhushan Bharat-R65777 wrote:
> >>>> and device disabling is not a standard like PCI. Do you think that we might need to do some
> >>>> device specific handling.
> >>> 
> >>> It would be better if we could avoid the need for device specific
> >>> handling.  DMA can be disabled via the IOMMU.
> >>> 
> >>> The tricky part is how to enable the device in the next user; as I wrote
> >>> elsewhere in the thread I think this needs to be something the user is
> >>> aware of, so that the user quiesces the device prior to enabling IOMMU
> >>> mappings.
> >> 
> >> Or resets it. I think we can make it mandatory to VFIO users (and Linux
> >> drivers) to reset the device prior to mapping it.
> > 
> > Reset is one way to quiesce it, but as Bharat pointed out not all
> > devices make that possible (even with device specific knowledge).  In
> > particular, Freescale datapath stuff has problems with this.
> 
> I'm sure once Freescale datapath drivers are upstream in Linux we can also get that code upstream in QEMU :).

Yeah, but the fewer places it lives, the better. :-)

> >> QEMU needs to have device specific code to assemble its device tree fragment anyways, so
> >> it can just as well reset the device before it enables it to DMA.
> > 
> > Now you're adding to the depth of knowledge QEMU needs to have about a
> > device.  It's not that big of a deal if it's just "set this bit and wait
> > for it to clear", but it could be a pain with a device that can't be
> > reset and has a complicated sequence needed to quiesce -- and as you
> > noted above, Linux drivers would need to take care of this anyway.
> > Doing it in QEMU would still help with non-Linux guests and avoid the
> > need for a hypercall to indicate when the IOMMU can be opened, though.
> 
> Yes, and you don't want to burden the Linux driver with a situation it doesn't face in non-virtualized environments :).

The driver does need to handle it in non-virtualized environments, if
you want to be able to return a device to the host driver after a VFIO
user has had it.  It will also be needed for kexec.

-Scott