Re: [PATCH RFC 00/15] Add VFIO mediated device support and IMS support for the idxd driver.

From: Jason Gunthorpe <jgg@mellanox.com>
To: "Dey, Megha" <megha.dey@linux.intel.com>
Cc: Dave Jiang <dave.jiang@intel.com>,
	vkoul@kernel.org, maz@kernel.org, bhelgaas@google.com,
	rafael@kernel.org, gregkh@linuxfoundation.org,
	tglx@linutronix.de, hpa@zytor.com, alex.williamson@redhat.com,
	jacob.jun.pan@intel.com, ashok.raj@intel.com, yi.l.liu@intel.com,
	baolu.lu@intel.com, kevin.tian@intel.com,
	sanjay.k.kumar@intel.com, tony.luck@intel.com,
	jing.lin@intel.com, dan.j.williams@intel.com,
	kwankhede@nvidia.com, eric.auger@redhat.com, parav@mellanox.com,
	dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org,
	x86@kernel.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org
Subject: Re: [PATCH RFC 00/15] Add VFIO mediated device support and IMS support for the idxd driver.
Date: Thu, 23 Apr 2020 16:44:47 -0300	[thread overview]
Message-ID: <20200423194447.GF13640@mellanox.com> (raw)
In-Reply-To: <d6b3c133-ac19-21af-b7a7-b9e7166b8166@linux.intel.com>

> > > The mdev utilizes Interrupt Message Store or IMS[3] instead of MSIX for
> > > interrupts for the guest. This preserves MSIX for host usages and also allows a
> > > significantly larger number of interrupt vectors for guest usage.
> > 
> > I never did get a reply to my earlier remarks on the IMS patches.
> > 
> > The concept of a device specific addr/data table format for MSI is not
> > Intel specific. This should be general code. We have a device that can
> > use this kind of kernel capability today.
> 
> I am sorry if I did not address your comments earlier.

It appears noboy from Intel bothered to answer anyone else on that RFC
thread:

https://lore.kernel.org/lkml/1568338328-22458-1-git-send-email-megha.dey@linux.intel.com/

However, it seems kind of moot as I see now that this verion of IMS
bears almost no resemblance to the original RFC.

That said, the similiarity to platform-msi was striking, does this new
version harmonize with that?

> The present IMS code is quite generic, most of the code is in the drivers/
> folder. We basically introduce 2 APIS: allocate and free IMS interrupts and
> a IMS IRQ domain to allocate these interrupts from. These APIs are
> architecture agnostic.
>
> We also introduce a new IMS IRQ domain which is architecture specific. This
> is because IMS generates interrupts only in the remappable format, hence
> interrupt remapping should be enabled for IMS. Currently, the interrupt
> remapping code is only available for Intel and AMD and I don’t see anything
> for ARM.

I don't understand these remarks though - IMS is simply the mapping of
a MemWr addr/data pair to a Linux IRQ number? Why does this intersect
with remapping?

AFAIK, any platform that supports MSI today should have the inherent
HW capability to support IMS.

> Also, could you give more details on the device that could use IMS? Do you
> have some driver code already? We could then see if and how the current IMS
> code could be made more generic.

We have several devices of interest, our NICs have very flexible PCI,
so it is no problem to take the MemWR addr/data from someplace other
than the MSI tables.

For this we want to have some way to allocate Linux IRQs dynamically
and get a addr/data pair to trigger them.

Our NIC devices are also linked to our ARM SOC family, so I'd expect
our ARM's to also be able to provide these APIs as the platform.

Jason