RE: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent

["Radovanovic, Aleksandar" <aleksandar.radovanovic@amd.com>]

[AMD Official Use Only - General]



> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: 12 October 2022 15:38
> To: Radovanovic, Aleksandar <aleksandar.radovanovic@amd.com>
> Cc: Gupta, Nipun <Nipun.Gupta@amd.com>; Marc Zyngier
> <maz@kernel.org>; Robin Murphy <robin.murphy@arm.com>;
> robh+dt@kernel.org; krzysztof.kozlowski+dt@linaro.org;
> gregkh@linuxfoundation.org; rafael@kernel.org; eric.auger@redhat.com;
> alex.williamson@redhat.com; cohuck@redhat.com; Gupta, Puneet (DCG-
> ENG) <puneet.gupta@amd.com>; song.bao.hua@hisilicon.com;
> mchehab+huawei@kernel.org; f.fainelli@gmail.com;
> jeffrey.l.hugo@gmail.com; saravanak@google.com;
> Michael.Srba@seznam.cz; mani@kernel.org; yishaih@nvidia.com;
> will@kernel.org; joro@8bytes.org; masahiroy@kernel.org;
> ndesaulniers@google.com; linux-arm-kernel@lists.infradead.org; linux-
> kbuild@vger.kernel.org; linux-kernel@vger.kernel.org;
> devicetree@vger.kernel.org; kvm@vger.kernel.org; okaya@kernel.org;
> Anand, Harpreet <harpreet.anand@amd.com>; Agarwal, Nikhil
> <nikhil.agarwal@amd.com>; Simek, Michal <michal.simek@amd.com>; git
> (AMD-Xilinx) <git@amd.com>
> Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its
> domain as parent
> 
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
> 
> 
> On Wed, Oct 12, 2022 at 01:37:54PM +0000, Radovanovic, Aleksandar wrote:
> > > On Wed, Oct 12, 2022 at 10:34:23AM +0000, Radovanovic, Aleksandar
> wrote:
> > >
> > >
> > > > As for GITS_TRANSLATER, we can take up to 4 different IOVAs, which
> > > > limits us to 4 CDX devices (should be sufficient for current HW
> > > > use-cases). Also, it means that the address part must be the same
> > > > for all vectors within a single CDX device. I'm assuming this is
> > > > OK as it is going to be a single interrupt and IOMMU domain anyway.
> > >
> > > This is not at all how MSI is supposed to work.
> >
> > In the general case, no, they're not.
> 
> I don't mean that you can hack this to work - I mean that in MSI the
> addr/data is supposed to come from the end point itself, not from some kind
> of shared structure. This is important because the actual act of generating
> the write has to be coherent with the DMA the device is doing, as the MSI
> write must push any DMA data to visibility to meet the "producer /
> consumer" model.
> 

I'm not sure I follow your argument, the limitation here is that the MSI
address value is shared between vectors of the same device (requester id
or endpoint, whichever way you prefer to call it), not between devices. 
This in no way implies that it is unordered with respect to device DMA - 
it is ordered and takes the same AXI path into the CPU cluster, so the
producer/consumer semantics are preserved.

Thanks,
Aleksandar