Re: [patch RFC 38/38] irqchip: Add IMS array driver - NOT FOR MERGING

From: Thomas Gleixner <tglx@linutronix.de>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Dimitri Sivanich <sivanich@hpe.com>,
	linux-hyperv@vger.kernel.org, Steve Wahl <steve.wahl@hpe.com>,
	linux-pci@vger.kernel.org, "K. Y. Srinivasan" <kys@microsoft.com>,
	Boris Ostrovsky <boris.ostrovsky@oracle.com>,
	Wei Liu <wei.liu@kernel.org>, Dave Jiang <dave.jiang@intel.com>,
	Baolu Lu <baolu.lu@intel.com>, Marc Zyngier <maz@kernel.org>,
	x86@kernel.org, Megha Dey <megha.dey@intel.com>,
	xen-devel@lists.xenproject.org, Kevin Tian <kevin.tian@intel.com>,
	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Stephen Hemminger <sthemmin@microsoft.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Jon Derrick <jonathan.derrick@intel.com>,
	Juergen Gross <jgross@suse.com>, Russ Anderson <rja@hpe.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	iommu@lists.linux-foundation.org,
	Jacob Pan <jacob.jun.pan@intel.com>,
	"Rafael J. Wysocki" <rafael@kernel.org>
Subject: Re: [patch RFC 38/38] irqchip: Add IMS array driver - NOT FOR MERGING
Date: Fri, 21 Aug 2020 21:47:43 +0200	[thread overview]
Message-ID: <874kovsrvk.fsf@nanos.tec.linutronix.de> (raw)
In-Reply-To: <20200821124547.GY1152540@nvidia.com>

On Fri, Aug 21 2020 at 09:45, Jason Gunthorpe wrote:
> On Fri, Aug 21, 2020 at 02:25:02AM +0200, Thomas Gleixner wrote:
>> +static void ims_mask_irq(struct irq_data *data)
>> +{
>> +	struct msi_desc *desc = irq_data_get_msi_desc(data);
>> +	struct ims_array_slot __iomem *slot = desc->device_msi.priv_iomem;
>> +	u32 __iomem *ctrl = &slot->ctrl;
>> +
>> +	iowrite32(ioread32(ctrl) & ~IMS_VECTOR_CTRL_UNMASK, ctrl);
>
> Just to be clear, this is exactly the sort of operation we can't do
> with non-MSI interrupts. For a real PCI device to execute this it
> would have to keep the data on die.

We means NVIDIA and your new device, right?

So if I understand correctly then the queue memory where the MSI
descriptor sits is in RAM.

How is that supposed to work if interrupt remapping is disabled?

That means irq migration and proper disabling of an interrupt become an
interesting exercise. I'm so not looking forward to that.

If interrupt remapping is enabled then both are trivial because then the
irq chip can delegate everything to the parent chip, i.e. the remapping
unit.

Can you please explain that a bit more precise?

> I saw the idxd driver was doing something like this, I assume it
> avoids trouble because it is a fake PCI device integrated with the
> CPU, not on a real PCI bus?

That's how it is implemented as far as I understood the patches. It's
device memory therefore iowrite32().

> It is really nice to see irq_domain used properly in x86!

If you ignore the abuse in XEN :)

And to be fair proper and usable (hierarchical) irq domains originate
from x86 and happened to solve quite a few horrorshows on the ARM side.

Just back then when we converted the original maze, nobody had a good
idea and the stomach to touch XEN.

Thanks,

        tglx
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu