Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent

From: Jason Gunthorpe <jgg@ziepe.ca>
To: "Radovanovic, Aleksandar" <aleksandar.radovanovic@amd.com>
Cc: "Gupta, Nipun" <Nipun.Gupta@amd.com>,
	Marc Zyngier <maz@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	"robh+dt@kernel.org" <robh+dt@kernel.org>,
	"krzysztof.kozlowski+dt@linaro.org"
	<krzysztof.kozlowski+dt@linaro.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	"rafael@kernel.org" <rafael@kernel.org>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"Gupta, Puneet (DCG-ENG)" <puneet.gupta@amd.com>,
	"song.bao.hua@hisilicon.com" <song.bao.hua@hisilicon.com>,
	"mchehab+huawei@kernel.org" <mchehab+huawei@kernel.org>,
	"f.fainelli@gmail.com" <f.fainelli@gmail.com>,
	"jeffrey.l.hugo@gmail.com" <jeffrey.l.hugo@gmail.com>,
	"saravanak@google.com" <saravanak@google.com>,
	"Michael.Srba@seznam.cz" <Michael.Srba@seznam.cz>,
	"mani@kernel.org" <mani@kernel.org>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"will@kernel.org" <will@kernel.org>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"masahiroy@kernel.org" <masahiroy@kernel.org>,
	"ndesaulniers@google.com" <ndesaulniers@google.com>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	"linux-kbuild@vger.kernel.org" <linux-kbuild@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"devicetree@vger.kernel.org" <devicetree@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"okaya@kernel.org" <okaya@kernel.org>,
	"Anand, Harpreet" <harpreet.anand@amd.com>,
	"Agarwal, Nikhil" <nikhil.agarwal@amd.com>,
	"Simek, Michal" <michal.simek@amd.com>,
	"git (AMD-Xilinx)" <git@amd.com>
Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent
Date: Thu, 13 Oct 2022 09:43:29 -0300	[thread overview]
Message-ID: <Y0gH8R8tEqn6sqZ5@ziepe.ca> (raw)
In-Reply-To: <MN2PR12MB4358277977C1B7E0BC214AC789229@MN2PR12MB4358.namprd12.prod.outlook.com>

On Wed, Oct 12, 2022 at 03:09:26PM +0000, Radovanovic, Aleksandar wrote:

> > On Wed, Oct 12, 2022 at 01:37:54PM +0000, Radovanovic, Aleksandar wrote:
> > > > On Wed, Oct 12, 2022 at 10:34:23AM +0000, Radovanovic, Aleksandar
> > wrote:
> > > >
> > > >
> > > > > As for GITS_TRANSLATER, we can take up to 4 different IOVAs, which
> > > > > limits us to 4 CDX devices (should be sufficient for current HW
> > > > > use-cases). Also, it means that the address part must be the same
> > > > > for all vectors within a single CDX device. I'm assuming this is
> > > > > OK as it is going to be a single interrupt and IOMMU domain anyway.
> > > >
> > > > This is not at all how MSI is supposed to work.
> > >
> > > In the general case, no, they're not.
> > 
> > I don't mean that you can hack this to work - I mean that in MSI the
> > addr/data is supposed to come from the end point itself, not from some kind
> > of shared structure. This is important because the actual act of generating
> > the write has to be coherent with the DMA the device is doing, as the MSI
> > write must push any DMA data to visibility to meet the "producer /
> > consumer" model.
> > 
> 
> I'm not sure I follow your argument, the limitation here is that the MSI
> address value is shared between vectors of the same device (requester id
> or endpoint, whichever way you prefer to call it), not between
> devices.

That isn't what you said, you said "we can take up to 4 different
IOVAs, which limits us to 4 CDX devices" - which sounds like HW being
shared across devices??

Jason

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel