From: Logan Gunthorpe <logang@deltatee.com>
To: Dave Jiang <dave.jiang@intel.com>,
linux-kernel@vger.kernel.org, linux-ntb@googlegroups.com,
linux-pci@vger.kernel.org, iommu@lists.linux-foundation.org,
linux-kselftest@vger.kernel.org, Jon Mason <jdmason@kudzu.us>,
Bjorn Helgaas <bhelgaas@google.com>,
Joerg Roedel <joro@8bytes.org>
Cc: Allen Hubbe <allenbh@gmail.com>,
Serge Semin <fancer.lancer@gmail.com>,
Eric Pilmore <epilmore@gigaio.com>
Subject: Re: [PATCH 0/9] Support using MSI interrupts in ntb_transport
Date: Thu, 31 Jan 2019 16:41:32 -0700 [thread overview]
Message-ID: <5f8b1616-a005-7658-978f-16466fcc3886@deltatee.com> (raw)
In-Reply-To: <7e3eed52-8e3a-c472-392a-5dc5689290ae@intel.com>
On 2019-01-31 3:46 p.m., Dave Jiang wrote:
> I believe irqbalance writes to the file /proc/irq/N/smp_affinity. So
> maybe take a look at the code that starts from there and see if it would
> have any impact on your stuff.
Ok, well on my system I can write to the smp_affinity all day and the
MSI interrupts still work fine.
The MSI code is a bit difficult to trace and audit with all the
different chips and the parent chips which I don't have a good
understanding of. But I can definitely see that it could be possible for
some chips to change the address as smp_affinitiy will eventually
sometimes call msi_domain_set_affinity() which does seem to recompose
the message and write it back to the chip.
So, I could relatively easily add a callback to msi_desc to catch this
and resend the MSI address/data. However, I'm not sure how this is ever
done atomically. It seems like there would be a race while the device
updates its address where old interrupts could be triggered. This race
would be much longer for us when sending this information over the NTB
link. Though, I guess if the only change is that it encodes CPU
information in the address then that would not be an issue. However, I'm
not sure I can say that for certain without a comprehensive
understanding of all the IRQ chips.
Any thoughts on this?
Logan
WARNING: multiple messages have this Message-ID (diff)
From: logang at deltatee.com (Logan Gunthorpe)
Subject: [PATCH 0/9] Support using MSI interrupts in ntb_transport
Date: Thu, 31 Jan 2019 16:41:32 -0700 [thread overview]
Message-ID: <5f8b1616-a005-7658-978f-16466fcc3886@deltatee.com> (raw)
In-Reply-To: <7e3eed52-8e3a-c472-392a-5dc5689290ae@intel.com>
On 2019-01-31 3:46 p.m., Dave Jiang wrote:
> I believe irqbalance writes to the file /proc/irq/N/smp_affinity. So
> maybe take a look at the code that starts from there and see if it would
> have any impact on your stuff.
Ok, well on my system I can write to the smp_affinity all day and the
MSI interrupts still work fine.
The MSI code is a bit difficult to trace and audit with all the
different chips and the parent chips which I don't have a good
understanding of. But I can definitely see that it could be possible for
some chips to change the address as smp_affinitiy will eventually
sometimes call msi_domain_set_affinity() which does seem to recompose
the message and write it back to the chip.
So, I could relatively easily add a callback to msi_desc to catch this
and resend the MSI address/data. However, I'm not sure how this is ever
done atomically. It seems like there would be a race while the device
updates its address where old interrupts could be triggered. This race
would be much longer for us when sending this information over the NTB
link. Though, I guess if the only change is that it encodes CPU
information in the address then that would not be an issue. However, I'm
not sure I can say that for certain without a comprehensive
understanding of all the IRQ chips.
Any thoughts on this?
Logan
WARNING: multiple messages have this Message-ID (diff)
From: logang@deltatee.com (Logan Gunthorpe)
Subject: [PATCH 0/9] Support using MSI interrupts in ntb_transport
Date: Thu, 31 Jan 2019 16:41:32 -0700 [thread overview]
Message-ID: <5f8b1616-a005-7658-978f-16466fcc3886@deltatee.com> (raw)
Message-ID: <20190131234132.Y3cpgkUwpLKDVFG_VtUGf9khP6JsVsJd9XebUDzrEyo@z> (raw)
In-Reply-To: <7e3eed52-8e3a-c472-392a-5dc5689290ae@intel.com>
On 2019-01-31 3:46 p.m., Dave Jiang wrote:
> I believe irqbalance writes to the file /proc/irq/N/smp_affinity. So
> maybe take a look at the code that starts from there and see if it would
> have any impact on your stuff.
Ok, well on my system I can write to the smp_affinity all day and the
MSI interrupts still work fine.
The MSI code is a bit difficult to trace and audit with all the
different chips and the parent chips which I don't have a good
understanding of. But I can definitely see that it could be possible for
some chips to change the address as smp_affinitiy will eventually
sometimes call msi_domain_set_affinity() which does seem to recompose
the message and write it back to the chip.
So, I could relatively easily add a callback to msi_desc to catch this
and resend the MSI address/data. However, I'm not sure how this is ever
done atomically. It seems like there would be a race while the device
updates its address where old interrupts could be triggered. This race
would be much longer for us when sending this information over the NTB
link. Though, I guess if the only change is that it encodes CPU
information in the address then that would not be an issue. However, I'm
not sure I can say that for certain without a comprehensive
understanding of all the IRQ chips.
Any thoughts on this?
Logan
next prev parent reply other threads:[~2019-01-31 23:41 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-31 18:56 [PATCH 0/9] Support using MSI interrupts in ntb_transport Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 18:56 ` [PATCH 1/9] iommu/vt-d: Allow interrupts from the entire bus for aliased devices Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-02-01 16:44 ` Joerg Roedel
2019-02-01 16:44 ` Joerg Roedel
2019-02-01 16:44 ` joro
2019-02-01 17:27 ` Logan Gunthorpe
2019-02-01 17:27 ` Logan Gunthorpe
2019-02-01 17:27 ` logang
2019-02-05 19:19 ` Jacob Pan
2019-02-05 19:19 ` Jacob Pan
2019-02-05 19:19 ` jacob.jun.pan
2019-02-05 20:40 ` Logan Gunthorpe
2019-02-05 20:40 ` Logan Gunthorpe
2019-02-05 20:40 ` logang
2019-02-05 23:58 ` Jacob Pan
2019-02-05 23:58 ` Jacob Pan
2019-02-05 23:58 ` jacob.jun.pan
2019-01-31 18:56 ` [PATCH 2/9] PCI/MSI: Support allocating virtual MSI interrupts Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 22:39 ` Bjorn Helgaas
2019-01-31 22:39 ` Bjorn Helgaas
2019-01-31 22:39 ` helgaas
2019-01-31 22:52 ` Logan Gunthorpe
2019-01-31 22:52 ` Logan Gunthorpe
2019-01-31 22:52 ` logang
2019-02-01 19:23 ` Bjorn Helgaas
2019-02-01 19:23 ` Bjorn Helgaas
2019-02-01 19:23 ` helgaas
2019-01-31 18:56 ` [PATCH 3/9] PCI/switchtec: Add module parameter to request more interrupts Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 18:56 ` [PATCH 4/9] NTB: Introduce functions to calculate multi-port resource index Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 18:56 ` [PATCH 5/9] NTB: Rename ntb.c to support multiple source files in the module Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 18:56 ` [PATCH 6/9] NTB: Introduce MSI library Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 18:56 ` [PATCH 7/9] NTB: Introduce NTB MSI Test Client Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 18:56 ` [PATCH 8/9] NTB: Add ntb_msi_test support to ntb_test Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 18:56 ` [PATCH 9/9] NTB: Add MSI interrupt support to ntb_transport Logan Gunthorpe
2019-01-31 18:56 ` Logan Gunthorpe
2019-01-31 18:56 ` logang
2019-01-31 20:20 ` [PATCH 0/9] Support using MSI interrupts in ntb_transport Dave Jiang
2019-01-31 20:20 ` Dave Jiang
2019-01-31 20:20 ` dave.jiang
2019-01-31 20:48 ` Logan Gunthorpe
2019-01-31 20:48 ` Logan Gunthorpe
2019-01-31 20:48 ` logang
2019-01-31 20:58 ` Dave Jiang
2019-01-31 20:58 ` Dave Jiang
2019-01-31 20:58 ` Dave Jiang
2019-01-31 20:58 ` dave.jiang
2019-01-31 22:39 ` Logan Gunthorpe
2019-01-31 22:39 ` Logan Gunthorpe
2019-01-31 22:39 ` Logan Gunthorpe
2019-01-31 22:39 ` logang
2019-01-31 22:46 ` Dave Jiang
2019-01-31 22:46 ` Dave Jiang
2019-01-31 22:46 ` Dave Jiang
2019-01-31 22:46 ` dave.jiang
2019-01-31 23:41 ` Logan Gunthorpe [this message]
2019-01-31 23:41 ` Logan Gunthorpe
2019-01-31 23:41 ` logang
2019-01-31 23:48 ` Dave Jiang
2019-01-31 23:48 ` Dave Jiang
2019-01-31 23:48 ` dave.jiang
2019-01-31 23:52 ` Logan Gunthorpe
2019-01-31 23:52 ` Logan Gunthorpe
2019-01-31 23:52 ` logang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5f8b1616-a005-7658-978f-16466fcc3886@deltatee.com \
--to=logang@deltatee.com \
--cc=allenbh@gmail.com \
--cc=bhelgaas@google.com \
--cc=dave.jiang@intel.com \
--cc=epilmore@gigaio.com \
--cc=fancer.lancer@gmail.com \
--cc=iommu@lists.linux-foundation.org \
--cc=jdmason@kudzu.us \
--cc=joro@8bytes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-ntb@googlegroups.com \
--cc=linux-pci@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.