All of lore.kernel.org
 help / color / mirror / Atom feed
From: Roger Pau Monne <roger.pau@citrix.com>
To: Paul Durrant <Paul.Durrant@citrix.com>
Cc: "Edgar Iglesias (edgar.iglesias@xilinx.com)"
	<edgar.iglesias@xilinx.com>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Anshul Makkar <anshul.makkar@citrix.com>,
	Wei Chen <Wei.Chen@arm.com>, Steve Capper <Steve.Capper@arm.com>,
	Andrew Cooper <Andrew.Cooper3@citrix.com>,
	Jiandi An <anjiandi@codeaurora.org>,
	Punit Agrawal <punit.agrawal@arm.com>,
	Julien Grall <julien.grall@linaro.org>,
	"alistair.francis@xilinx.com" <alistair.francis@xilinx.com>,
	Shanker Donthineni <shankerd@codeaurora.org>,
	xen-devel <xen-devel@lists.xenproject.org>,
	"manish.jaggi@caviumnetworks.com"
	<manish.jaggi@caviumnetworks.com>,
	Campbell Sean <scampbel@codeaurora.org>
Subject: Re: [early RFC] ARM PCI Passthrough design document
Date: Fri, 10 Feb 2017 12:57:48 +0000	[thread overview]
Message-ID: <20170210125748.2636p25agqld5kg4@dhcp-3-221.uk.xensource.com> (raw)
In-Reply-To: <d631440252da4fd8b6e03af8da4cf552@AMSPEX02CL03.citrite.net>

On Fri, Feb 10, 2017 at 10:11:53AM +0000, Paul Durrant wrote:
> > -----Original Message-----
> > From: Roger Pau Monne
> > Sent: 10 February 2017 09:49
> > To: Stefano Stabellini <sstabellini@kernel.org>
> > Cc: Julien Grall <julien.grall@linaro.org>; xen-devel <xen-
> > devel@lists.xenproject.org>; Edgar Iglesias (edgar.iglesias@xilinx.com)
> > <edgar.iglesias@xilinx.com>; Steve Capper <Steve.Capper@arm.com>; Punit
> > Agrawal <punit.agrawal@arm.com>; Wei Chen <Wei.Chen@arm.com>;
> > Campbell Sean <scampbel@codeaurora.org>; Shanker Donthineni
> > <shankerd@codeaurora.org>; Jiandi An <anjiandi@codeaurora.org>;
> > manish.jaggi@caviumnetworks.com; alistair.francis@xilinx.com; Andrew
> > Cooper <Andrew.Cooper3@citrix.com>; Anshul Makkar
> > <anshul.makkar@citrix.com>; Paul Durrant <Paul.Durrant@citrix.com>
> > Subject: Re: [early RFC] ARM PCI Passthrough design document
> > 
> > On Wed, Feb 01, 2017 at 10:50:49AM -0800, Stefano Stabellini wrote:
> > > On Wed, 1 Feb 2017, Roger Pau Monné wrote:
> > > > On Wed, Jan 25, 2017 at 06:53:20PM +0000, Julien Grall wrote:
> > > > > Hi Stefano,
> > > > >
> > > > > On 24/01/17 20:07, Stefano Stabellini wrote:
> > > > > > On Tue, 24 Jan 2017, Julien Grall wrote:
> > > > > When using ECAM like host bridge, I don't think it will be an issue to
> > have
> > > > > both DOM0 and Xen accessing configuration space at the same time.
> > Although,
> > > > > we need to define who is doing what. In general case, DOM0 should
> > not
> > > > > touched an assigned PCI device. The only possible interaction would be
> > > > > resetting a device (see my answer below).
> > > >
> > > > Iff Xen is really going to perform the reset of passthrough devices, then I
> > > > don't see any reason to expose those devices to Dom0 at all, IMHO you
> > should
> > > > hide them from ACPI and ideally prevent Dom0 from interacting with
> > them using
> > > > the PCI configuration space (although that would require trapping on
> > accesses
> > > > to the PCI config space, which AFAIK you would like to avoid).
> > >
> > > Right! A much cleaner solution! If we are going to have Xen handle ECAM
> > > and emulating PCI host bridges, then we should go all the way and have
> > > Xen do everything about PCI.
> > 
> > Replying here because this thread has become so long that's hard to find a
> > good
> > place to put this information.
> > 
> > I've recently been told (f2f), that more complex passthrough (like Nvidia
> > vGPU
> > or Intel XenGT) work in a slightly different way, which seems to be a bit
> > incompatible with what we are proposing. I've been told that Nvidia vGPU
> > passthrough requires a driver in Dom0 (closed-source Nvidia code AFAIK),
> > and
> > that upon loading this driver a bunch of virtual functions appear out of the
> > blue in the PCI bus.
> > 
> > Now, if we completely hide passed-through devices from Dom0, it would be
> > impossible to load this driver, and thus to make the virtual functions appear.
> > I would like someone that's more familiar with this to comment, so I'm
> > adding
> > Paul and Anshul to the conversation.
> > 
> > To give some context to them, we were currently discussing to completely
> > hide
> > passthrough PCI devices from Dom0, and have Xen perform the reset of the
> > device. This would apply to PVH and ARM. Can you comment on whether
> > such
> > approach would work with things like vGPU passthrough?
> 
> Neither NVIDIA vGPU nor Intel GVT-g are pass-through. They both use emulation to synthesize GPU devices for guests and then use the actual GPU to service the commands sent by the guest driver to the virtual GPU. So, I think they fall outside the discussion here.

So in this case those devices would simply be assigned to Dom0, and everything
would be trapped/emulated there? (by QEMU or whatever dm we are using)

> AMD MxGPU is somewhat different in that it is an almost-SRIOV solution. I say 'almost' because the VF's are not truly independent and so some interception of accesses to certain registers is required, so that arbitration can be applied, or they can be blocked. In this case a dedicated driver in dom0 is required, and I believe it needs access to both the PF and all the VFs to function correctly. However, once initial set-up is done, I think the VFs could then be hidden from dom0. The PF is never passed-through and so there should be no issue in leaving it visible to dom0.

The approach we where thinking of is hiding everything from Dom0 when it
boots, so that Dom0 would never really see those devices. This would be done by
Xen scanning the PCI bus and any ECAM areas. DEvices that first need to be
assigned to Dom0 and then hidden where not part of the approach here.

> There is a further complication with GVT-d (Intel's term for GPU pass-through) also because I believe there is also some initial set-up required and some supporting emulation (e.g. Intel's guest driver expects there to be an ISA bridge along with the GPU) which may need access to the real GPU. It is also possible that, once this set-up is done, the GPU can then be hidden from dom0 but I'm not sure because I was not involved with that code.

And then I guess some MMIO regions are assigned to the guest, and some dm
performs the trapping of the accesses to the configuration space?

> Full pass-through of NVIDIA and AMD GPUs does not involve access from dom0 at all though, so I don't think there should be any complication there.

Yes, in that case they would be treated as regular PCI devices, no involvement
from Dom0 would be needed. I'm more worried about this mixed cases, where some
Dom0 interaction is needed in order to perform the passthrough.

> Does that all make sense?

I guess, could you please keep an eye on further design documents? Just to
make sure that what's described here would work for the more complex
passthrough scenarios that XenServer supports.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

  reply	other threads:[~2017-02-10 12:57 UTC|newest]

Thread overview: 82+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-12-29 14:04 [early RFC] ARM PCI Passthrough design document Julien Grall
2016-12-29 14:16 ` Jaggi, Manish
2016-12-29 17:03   ` Julien Grall
2016-12-29 18:41     ` Jaggi, Manish
2016-12-29 19:38       ` Julien Grall
2017-01-04  0:24 ` Stefano Stabellini
2017-01-24 14:28   ` Julien Grall
2017-01-24 20:07     ` Stefano Stabellini
2017-01-25 11:21       ` Roger Pau Monné
2017-01-25 18:53       ` Julien Grall
2017-01-31 16:53         ` Edgar E. Iglesias
2017-01-31 17:09           ` Julien Grall
2017-01-31 19:06             ` Edgar E. Iglesias
2017-01-31 22:08               ` Stefano Stabellini
2017-02-01 19:04               ` Julien Grall
2017-02-01 19:31                 ` Stefano Stabellini
2017-02-01 20:24                   ` Julien Grall
2017-02-02 15:33                 ` Edgar E. Iglesias
2017-02-02 23:12                   ` Stefano Stabellini
2017-02-02 23:44                     ` Edgar E. Iglesias
2017-02-10  1:01                       ` Stefano Stabellini
2017-02-13 15:39                         ` Julien Grall
2017-02-13 19:59                           ` Stefano Stabellini
2017-02-14 17:21                             ` Julien Grall
2017-02-14 18:20                               ` Stefano Stabellini
2017-02-14 20:18                                 ` Julien Grall
2017-02-13 15:35                   ` Julien Grall
2017-02-22  4:03                     ` Edgar E. Iglesias
2017-02-23 16:47                       ` Julien Grall
2017-03-02 21:13                         ` Edgar E. Iglesias
2017-02-02 15:40                 ` Roger Pau Monné
2017-02-13 16:22                   ` Julien Grall
2017-01-31 21:58         ` Stefano Stabellini
2017-02-01 20:12           ` Julien Grall
2017-02-01 10:55         ` Roger Pau Monné
2017-02-01 18:50           ` Stefano Stabellini
2017-02-10  9:48             ` Roger Pau Monné
2017-02-10 10:11               ` Paul Durrant
2017-02-10 12:57                 ` Roger Pau Monne [this message]
2017-02-10 13:02                   ` Paul Durrant
2017-02-10 21:04                     ` Stefano Stabellini
2017-02-02 12:38           ` Julien Grall
2017-02-02 23:06             ` Stefano Stabellini
2017-03-08 19:06               ` Julien Grall
2017-03-08 19:12                 ` Konrad Rzeszutek Wilk
2017-03-08 19:55                   ` Stefano Stabellini
2017-03-08 21:51                     ` Julien Grall
2017-03-09  2:59                   ` Roger Pau Monné
2017-03-09 11:17                     ` Konrad Rzeszutek Wilk
2017-03-09 13:26                       ` Julien Grall
2017-03-10  0:29                         ` Konrad Rzeszutek Wilk
2017-03-10  3:23                           ` Roger Pau Monné
2017-03-10 15:28                             ` Konrad Rzeszutek Wilk
2017-03-15 12:07                               ` Roger Pau Monné
2017-03-15 12:42                                 ` Konrad Rzeszutek Wilk
2017-03-15 12:56                                   ` Roger Pau Monné
2017-03-15 15:11                                     ` Venu Busireddy
2017-03-15 16:38                                       ` Roger Pau Monn?
2017-03-15 16:54                                         ` Venu Busireddy
2017-03-15 17:00                                           ` Roger Pau Monn?
2017-05-03 12:38                                             ` Julien Grall
2017-05-03 12:53                                         ` Julien Grall
2017-01-25  4:23     ` Manish Jaggi
2017-01-06 15:12 ` Roger Pau Monné
2017-01-06 21:16   ` Stefano Stabellini
2017-01-24 17:17   ` Julien Grall
2017-01-25 11:42     ` Roger Pau Monné
2017-01-31 15:59       ` Julien Grall
2017-01-31 22:03         ` Stefano Stabellini
2017-02-01 10:28           ` Roger Pau Monné
2017-02-01 18:45             ` Stefano Stabellini
2017-01-06 16:27 ` Edgar E. Iglesias
2017-01-06 21:12   ` Stefano Stabellini
2017-01-09 17:50     ` Edgar E. Iglesias
2017-01-19  5:09 ` Manish Jaggi
2017-01-24 17:43   ` Julien Grall
2017-01-25  4:37     ` Manish Jaggi
2017-01-25 15:25       ` Julien Grall
2017-01-30  7:41         ` Manish Jaggi
2017-01-31 13:33           ` Julien Grall
2017-05-19  6:38 ` Goel, Sameer
2017-05-19 16:48   ` Julien Grall

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170210125748.2636p25agqld5kg4@dhcp-3-221.uk.xensource.com \
    --to=roger.pau@citrix.com \
    --cc=Andrew.Cooper3@citrix.com \
    --cc=Paul.Durrant@citrix.com \
    --cc=Steve.Capper@arm.com \
    --cc=Wei.Chen@arm.com \
    --cc=alistair.francis@xilinx.com \
    --cc=anjiandi@codeaurora.org \
    --cc=anshul.makkar@citrix.com \
    --cc=edgar.iglesias@xilinx.com \
    --cc=julien.grall@linaro.org \
    --cc=manish.jaggi@caviumnetworks.com \
    --cc=punit.agrawal@arm.com \
    --cc=scampbel@codeaurora.org \
    --cc=shankerd@codeaurora.org \
    --cc=sstabellini@kernel.org \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.