From: Leonardo Bras <leobras.c@gmail.com>
To: Alexey Kardashevskiy <aik@ozlabs.ru>,
Michael Ellerman <mpe@ellerman.id.au>,
Benjamin Herrenschmidt <benh@kernel.crashing.org>,
Paul Mackerras <paulus@samba.org>,
Thiago Jung Bauermann <bauerman@linux.ibm.com>,
Ram Pai <linuxram@us.ibm.com>
Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 4/4] powerpc/pseries/iommu: Remove default DMA window before creating DDW
Date: Mon, 22 Jun 2020 23:31:18 -0300 [thread overview]
Message-ID: <205edd45b7bbf39d2fc1d2d75fd7e35336f109ac.camel@gmail.com> (raw)
In-Reply-To: <afd1c5ac-d291-5281-1592-a345ee3c0c8c@ozlabs.ru>
On Tue, 2020-06-23 at 11:11 +1000, Alexey Kardashevskiy wrote:
>
> On 23/06/2020 04:59, Leonardo Bras wrote:
> > Hello Alexey, thanks for the feedback!
> >
> > On Mon, 2020-06-22 at 20:02 +1000, Alexey Kardashevskiy wrote:
> > > On 19/06/2020 15:06, Leonardo Bras wrote:
> > > > On LoPAR "DMA Window Manipulation Calls", it's recommended to remove the
> > > > default DMA window for the device, before attempting to configure a DDW,
> > > > in order to make the maximum resources available for the next DDW to be
> > > > created.
> > > >
> > > > This is a requirement for some devices to use DDW, given they only
> > > > allow one DMA window.
> > > >
> > > > If setting up a new DDW fails anywhere after the removal of this
> > > > default DMA window, restore it using reset_dma_window.
> > >
> > > Nah... If we do it like this, then under pHyp we lose 32bit DMA for good
> > > as pHyp can only create a single window and it has to map at
> > > 0x800.0000.0000.0000. They probably do not care though.
> > >
> > > Under KVM, this will fail as VFIO allows creating 2 windows and it
> > > starts from 0 but the existing iommu_bypass_supported_pSeriesLP() treats
> > > the window address == 0 as a failure. And we want to keep both DMA
> > > windows for PCI adapters with both 64bit and 32bit PCI functions (I
> > > heard AMD GPU video + audio are like this) or someone could hotplug
> > > 32bit DMA device on a vphb with already present 64bit DMA window so we
> > > do not remove the default window.
> >
> > Well, then I would suggest doing something like this:
> > query_ddw(...);
> > if (query.windows_available == 0){
> > remove_dma_window(...,default_win);
> > query_ddw(...);
> > }
> >
> > This would make sure to cover cases of windows available == 1
> > and windows available > 1;
>
> Is "1" what pHyp returns on query? And was it always like that? Then it
> is probably ok. I just never really explored the idea of removing the
> default window as we did not have to.
I am not sure if this is true in general, but in this device (SR-IOV
VF) I am testing it will return 0 windows if the default DMA window is
not deleted, and 1 after it's deleted.
>
>
> > > The last discussed thing I remember was that there was supposed to be a
> > > new bit in "ibm,architecture-vec-5" (forgot the details), we could use
> > > that to decide whether to keep the default window or not, like this.
> >
> > I checked on the latest LoPAR draft (soon to be published), for the
> > ibm,architecture-vec 'option array 5' and this entry was the only
> > recently added one that is related to this patchset:
> >
> > Byte 8 - Bit 0:
> > SRIOV Virtual Functions Support Dynamic DMA Windows (DDW):
> > 0: SRIOV Virtual Functions do not support DDW
> > 1: SRIOV Virtual Functions do support DDW
> >
> > Isn't this equivalent to having a "ibm,ddw-applicable" property?
>
> I am not sure, is there anything else to this new bit?
I copied everything from the LoPAR, and IIRC the ACR for this change
only described this change in the document.
> I'd think if the
> client supports it, then pHyp will create one 64bit window per a PE and
> DDW API won't be needed. Thanks,
That would make sense, and be great.
I will dig some more.
Thank you!
> > > > Signed-off-by: Leonardo Bras <leobras.c@gmail.com>
> > > > ---
> > > > arch/powerpc/platforms/pseries/iommu.c | 20 +++++++++++++++++---
> > > > 1 file changed, 17 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
> > > > index de633f6ae093..68d1ea957ac7 100644
> > > > --- a/arch/powerpc/platforms/pseries/iommu.c
> > > > +++ b/arch/powerpc/platforms/pseries/iommu.c
> > > > @@ -1074,8 +1074,9 @@ static u64 enable_ddw(struct pci_dev *dev, struct device_node *pdn)
> > > > u64 dma_addr, max_addr;
> > > > struct device_node *dn;
> > > > u32 ddw_avail[3];
> > > > +
> > > > struct direct_window *window;
> > > > - struct property *win64;
> > > > + struct property *win64, *dfl_win;
> > >
> > > Make it "default_win" or "def_win", "dfl" hurts to read :)
> >
> > Sure, no problem :)
> >
> > > > struct dynamic_dma_window_prop *ddwprop;
> > > > struct failed_ddw_pdn *fpdn;
> > > >
> > > > @@ -1110,8 +1111,19 @@ static u64 enable_ddw(struct pci_dev *dev, struct device_node *pdn)
> > > > if (ret)
> > > > goto out_failed;
> > > >
> > > > - /*
> > > > - * Query if there is a second window of size to map the
> > > > + /*
> > > > + * First step of setting up DDW is removing the default DMA window,
> > > > + * if it's present. It will make all the resources available to the
> > > > + * new DDW window.
> > > > + * If anything fails after this, we need to restore it.
> > > > + */
> > > > +
> > > > + dfl_win = of_find_property(pdn, "ibm,dma-window", NULL);
> > > > + if (dfl_win)
> > > > + remove_dma_window(pdn, ddw_avail, dfl_win);
> > >
> > > Before doing so, you want to make sure that the "reset" is actually
> > > supported. Thanks,
> >
> > Good catch, I will improve that.
> >
> > >
> > > > +
> > > > + /*
> > > > + * Query if there is a window of size to map the
> > > > * whole partition. Query returns number of windows, largest
> > > > * block assigned to PE (partition endpoint), and two bitmasks
> > > > * of page sizes: supported and supported for migrate-dma.
> > > > @@ -1219,6 +1231,8 @@ static u64 enable_ddw(struct pci_dev *dev, struct device_node *pdn)
> > > > kfree(win64);
> > > >
> > > > out_failed:
> > > > + if (dfl_win)
> > > > + reset_dma_window(dev, pdn);
> > > >
> > > > fpdn = kzalloc(sizeof(*fpdn), GFP_KERNEL);
> > > > if (!fpdn)
> > > >
> >
> > Best regards,
> > Leonardo
> >
next prev parent reply other threads:[~2020-06-23 2:31 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-19 5:06 [PATCH 0/4] Remove default DMA window before creating DDW Leonardo Bras
2020-06-19 5:06 ` [PATCH 1/4] powerpc/pseries/iommu: Update call to ibm,query-pe-dma-windows Leonardo Bras
2020-06-22 10:02 ` Alexey Kardashevskiy
2020-06-22 18:58 ` Leonardo Bras
2020-06-23 1:12 ` Alexey Kardashevskiy
2020-06-23 2:14 ` Leonardo Bras
2020-06-23 2:29 ` Alexey Kardashevskiy
2020-06-19 5:06 ` [PATCH 2/4] powerpc/pseries/iommu: Implement ibm,reset-pe-dma-windows rtas call Leonardo Bras
2020-06-22 10:02 ` Alexey Kardashevskiy
2020-06-22 18:58 ` Leonardo Bras
2020-06-23 1:11 ` Alexey Kardashevskiy
2020-06-23 2:20 ` Leonardo Bras
2020-06-19 5:06 ` [PATCH 3/4] powerpc/pseries/iommu: Move window-removing part of remove_ddw into remove_dma_window Leonardo Bras
2020-06-22 10:02 ` Alexey Kardashevskiy
2020-06-22 18:59 ` Leonardo Bras
2020-06-23 1:12 ` Alexey Kardashevskiy
2020-06-23 1:33 ` Oliver O'Halloran
2020-06-23 2:26 ` Leonardo Bras
2020-06-23 2:22 ` Leonardo Bras
2020-06-19 5:06 ` [PATCH 4/4] powerpc/pseries/iommu: Remove default DMA window before creating DDW Leonardo Bras
2020-06-20 6:13 ` kernel test robot
2020-06-22 10:02 ` Alexey Kardashevskiy
2020-06-22 18:59 ` Leonardo Bras
2020-06-23 1:11 ` Alexey Kardashevskiy
2020-06-23 2:31 ` Leonardo Bras [this message]
2020-06-23 2:35 ` Alexey Kardashevskiy
2020-06-23 2:43 ` Leonardo Bras
2020-06-23 3:52 ` Alexey Kardashevskiy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=205edd45b7bbf39d2fc1d2d75fd7e35336f109ac.camel@gmail.com \
--to=leobras.c@gmail.com \
--cc=aik@ozlabs.ru \
--cc=bauerman@linux.ibm.com \
--cc=benh@kernel.crashing.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=linuxram@us.ibm.com \
--cc=mpe@ellerman.id.au \
--cc=paulus@samba.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).