linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jacob Pan <jacob.jun.pan@linux.intel.com>
To: Jean-Philippe Brucker <jean-philippe.brucker@arm.com>
Cc: "iommu@lists.linux-foundation.org"
	<iommu@lists.linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Joerg Roedel <joro@8bytes.org>,
	David Woodhouse <dwmw2@infradead.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Rafael Wysocki <rafael.j.wysocki@intel.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	Lan Tianyu <tianyu.lan@intel.com>,
	Jean Delvare <khali@linux-fr.org>,
	Will Deacon <Will.Deacon@arm.com>,
	jacob.jun.pan@linux.intel.com, "Kumar,
	Sanjay K" <sanjay.k.kumar@intel.com>
Subject: Re: [PATCH v3 15/16] iommu: introduce page response function
Date: Wed, 6 Dec 2017 11:25:21 -0800	[thread overview]
Message-ID: <20171206112521.1edf8e9b@jacob-builder> (raw)
In-Reply-To: <a6cfc27a-6121-1e67-6e0d-f94a383bcd6f@arm.com>

On Tue, 5 Dec 2017 17:21:15 +0000
Jean-Philippe Brucker <jean-philippe.brucker@arm.com> wrote:

> Hi Jacob,
> 
> On 04/12/17 21:37, Jacob Pan wrote:
> > On Fri, 24 Nov 2017 12:03:50 +0000
> > Jean-Philippe Brucker <jean-philippe.brucker@arm.com> wrote:
> >   
> >> On 17/11/17 18:55, Jacob Pan wrote:  
> >>> When nested translation is turned on and guest owns the
> >>> first level page tables, device page request can be forwared
> >>> to the guest for handling faults. As the page response returns
> >>> by the guest, IOMMU driver on the host need to process the
> >>> response which informs the device and completes the page request
> >>> transaction.
> >>>
> >>> This patch introduces generic API function for page response
> >>> passing from the guest or other in-kernel users. The definitions
> >>> of the generic data is based on PCI ATS specification not limited
> >>> to any vendor.>
> >>> Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>  
> [...]
> > I think the simpler interface works for in-kernel driver use case
> > very well. But in case of VFIO, the callback function does not turn
> > around send back page response. The page response comes from guest
> > and qemu, where they don;t keep track of the the prq event data.  
> 
> Is it safe to trust whatever response the guest or userspace gives
> us? The answer seems fairly vendor- and device-specific so I wonder
> if VFIO or IOMMU shouldn't do a bit of sanity checking somewhere, and
> keep track of all injected page requests.
> 
> From SMMUv3 POV, it seems safe (haven't looked at SMMUv2 but I'm not
> so confident).
> 
> * The guest can only send page responses to devices assigned to it,
> that's a given.
> 
Agree, IOMMU driver cannot enforce it. I think VFIO layer can make sure
page response come from the assigned device and its guest/container.
> * If, after we injected a page request, the guest doesn't reply at
> all, then the device leaks page request credits and at some point it
> will stop sending requests.
>   -> So the PRI capability needs to be reset whenever we change the  
>      device's domain, to clear the credit counter and pending states.
> 
>   For SMMUv3, the stall buffer may be shared between devices on some
>   implementations, in which case the guest could prevent other
> devices to stall by letting the buffer fill up.
>   -> We might have to keep track of stalls in the host driver and set
> a credit or timeout to each stall, if it comes to that.
>   -> In addition, send a terminate-all-stalls command when changing
> the device's domain.
> 
We have the same situation in VT-d with shared queue which in turn may
affect other guests. Letting host driver maintain record of pending page
request seems the best way to go. VT-d has a way to drain PRQ per PASID
and RID combination. I guess this is the same as your
"terminate-all-stalls" but with finer control? Or
"terminate-all-stalls" only applies to a given device.
Seems we can implement a generic timeout/credit mechanism in IOMMU
driver with model specific action to drain/terminate. The timeout value
can also be model specific.

> * If the guest sends spurious or duplicate page responses (where the
> PRGI or PASID doesn't exist in any outstanding page request of the
> device)
> 
If we keep track of pending PRQ in host IOMMU driver, then it can
detect duplicated case.
>   For PRI if we send an invalid PRG Response, the endpoint sets UPRGI
> in the PRI cap, and issues an Unexpected Completion. Then I suppose
> the worst that happens is we get an AER report that we can't handle?
> I'm not too familiar with that part of PCIe.
> 
I don;t see this mentioned in the PCI ATS spec., but in general this
sounds like a case HW has to handle, perhaps ignoring them is
reasonable as you said below.
>   Stall is designed to tolerate this and will just ignore the
> response.
> 
> * If PRI/stall isn't even enabled, the IOMMU driver can check that in
> the device configuration and not send the reply.
> 
> 
> 
> 
> Regardless, I have a few comments on the page_response_msg:
> 
Thanks, all points are taken unless commented.
> > +/**
> > + * Generic page response information based on PCI ATS and PASID
> > spec.
> > + * @paddr: servicing page address  
> 
> Maybe call it @addr, so we don't read this field as "phys addr"
> 
> > + * @pasid: contains process address space ID, used in shared
> > virtual memory(SVM)  
> 
> The "used in shared virtual memory(SVM)" part isn't necessary and
> we're changing the API name.
> 
> > + * @rid: requestor ID
> > + * @did: destination device ID  
> 
> I guess you can remove @rid and @did
> 
> > + * @last_req: last request in a page request group  
> 
> Is @last_req needed at all, since only the last request requires a
> response?
> 
right, i was thinking we had single page response in vt-d, but there is
not need either.
> > + * @resp_code: response code  
> 
> The comment is missing a description for @pasid_present here
> 
> > + * @page_req_group_id: page request group index
> > + * @prot: page access protection flag, e.g. IOMMU_FAULT_READ,
> > IOMMU_FAULT_WRITE  
> 
> Is @prot really needed in the response?
> 
no, you are right.
> > + * @type: group or stream response  
> 
> The page request doesn't provide this information
> 
this is vt-d specific. it is in the vt-d page request descriptor and
response descriptors are different depending on the type.
Since we intend the generic data to be super set of models, I add this
field.
> > + * @private_data: uniquely identify device-specific private data
> > for an
> > + *                individual page response
> > +
> > + */
> > +struct page_response_msg {
> > +	u64 paddr;
> > +	u32 pasid;
> > +	u32 rid:16;
> > +	u32 did:16;
> > +	u32 resp_code:4;
> > +	u32 last_req:1;
> > +	u32 pasid_present:1;
> > +#define IOMMU_PAGE_RESP_SUCCESS	0
> > +#define IOMMU_PAGE_RESP_INVALID	1
> > +#define IOMMU_PAGE_RESP_FAILURE	0xF  
> 
> Maybe move these defines closer to resp_code.
> For someone not familiar with PRI, we should add some comments about
> those values:
> 
> * SUCCESS: the request was paged-in successfully
> * INVALID: could not page-in one or more pages in the group
> * FAILURE: permanent PRI error, may disable faults in the device
> 
> > +	u32 page_req_group_id : 9;
> > +	u32 prot;
> > +	enum page_response_type type;
> > +	u32 private_data;
> > +};
> > +  
> 
> Thanks,
> Jean

[Jacob Pan]

  reply	other threads:[~2017-12-06 19:24 UTC|newest]

Thread overview: 56+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-11-17 18:54 [PATCH v3 00/16] [PATCH v3 00/16] IOMMU driver support for SVM virtualization Jacob Pan
2017-11-17 18:54 ` [PATCH v3 01/16] iommu: introduce bind_pasid_table API function Jacob Pan
2017-11-24 12:04   ` Jean-Philippe Brucker
2017-11-29 22:01     ` Jacob Pan
2017-11-17 18:55 ` [PATCH v3 02/16] iommu/vt-d: add bind_pasid_table function Jacob Pan
2017-11-17 18:55 ` [PATCH v3 03/16] iommu: introduce iommu invalidate API function Jacob Pan
2017-11-24 12:04   ` Jean-Philippe Brucker
2017-12-15 19:02     ` Jean-Philippe Brucker
2017-12-28 19:25     ` Jacob Pan
2018-01-10 12:00       ` Jean-Philippe Brucker
2017-11-17 18:55 ` [PATCH v3 04/16] iommu/vt-d: move device_domain_info to header Jacob Pan
2017-11-17 18:55 ` [PATCH v3 05/16] iommu/vt-d: support flushing more TLB types Jacob Pan
2017-11-20 14:20   ` Lukoshkov, Maksim
2017-11-20 18:40     ` Jacob Pan
2017-11-17 18:55 ` [PATCH v3 06/16] iommu/vt-d: add svm/sva invalidate function Jacob Pan
2017-12-05  5:43   ` Lu Baolu
2017-11-17 18:55 ` [PATCH v3 07/16] iommu/vt-d: assign PFSID in device TLB invalidation Jacob Pan
2017-12-05  5:45   ` Lu Baolu
2017-11-17 18:55 ` [PATCH v3 08/16] iommu: introduce device fault data Jacob Pan
2017-11-24 12:03   ` Jean-Philippe Brucker
2017-11-29 21:55     ` Jacob Pan
2018-01-10 11:41   ` Jean-Philippe Brucker
2018-01-11 21:10     ` Jacob Pan
2017-11-17 18:55 ` [PATCH v3 09/16] driver core: add iommu device fault reporting data Jacob Pan
2017-12-18 14:37   ` Greg Kroah-Hartman
2017-11-17 18:55 ` [PATCH v3 10/16] iommu: introduce device fault report API Jacob Pan
2017-12-05  6:22   ` Lu Baolu
2017-12-08 21:22     ` Jacob Pan
2017-12-07 21:27   ` Alex Williamson
2017-12-08 20:23     ` Jacob Pan
2017-12-08 20:59       ` Alex Williamson
2017-12-08 21:22         ` Jacob Pan
2018-01-10 12:39   ` Jean-Philippe Brucker
2018-01-18 19:24   ` Jean-Philippe Brucker
2018-01-23 20:01     ` Jacob Pan
2017-11-17 18:55 ` [PATCH v3 11/16] iommu/vt-d: use threaded irq for dmar_fault Jacob Pan
2017-11-17 18:55 ` [PATCH v3 12/16] iommu/vt-d: report unrecoverable device faults Jacob Pan
2017-12-05  6:34   ` Lu Baolu
2017-11-17 18:55 ` [PATCH v3 13/16] iommu/intel-svm: notify page request to guest Jacob Pan
2017-12-05  7:37   ` Lu Baolu
2017-11-17 18:55 ` [PATCH v3 14/16] iommu/intel-svm: replace dev ops with fault report API Jacob Pan
2017-11-17 18:55 ` [PATCH v3 15/16] iommu: introduce page response function Jacob Pan
2017-11-24 12:03   ` Jean-Philippe Brucker
2017-12-04 21:37     ` Jacob Pan
2017-12-05 17:21       ` Jean-Philippe Brucker
2017-12-06 19:25         ` Jacob Pan [this message]
2017-12-07 12:56           ` Jean-Philippe Brucker
2017-12-07 21:56             ` Alex Williamson
2017-12-08 13:51               ` Jean-Philippe Brucker
2017-12-08  1:17             ` Jacob Pan
2017-12-08 13:51               ` Jean-Philippe Brucker
2017-12-07 21:51           ` Alex Williamson
2017-12-08 13:52             ` Jean-Philippe Brucker
2017-12-08 20:40               ` Jacob Pan
2017-12-08 23:01                 ` Alex Williamson
2017-11-17 18:55 ` [PATCH v3 16/16] iommu/vt-d: add intel iommu " Jacob Pan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20171206112521.1edf8e9b@jacob-builder \
    --to=jacob.jun.pan@linux.intel.com \
    --cc=Will.Deacon@arm.com \
    --cc=alex.williamson@redhat.com \
    --cc=dwmw2@infradead.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=iommu@lists.linux-foundation.org \
    --cc=jean-philippe.brucker@arm.com \
    --cc=joro@8bytes.org \
    --cc=khali@linux-fr.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=rafael.j.wysocki@intel.com \
    --cc=sanjay.k.kumar@intel.com \
    --cc=tianyu.lan@intel.com \
    --subject='Re: [PATCH v3 15/16] iommu: introduce page response function' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).