From: Logan Gunthorpe <logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
To: Dave Jiang <dave.jiang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-ntb-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org,
linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org,
linux-kselftest-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Jon Mason <jdmason-zqedIHPRSYI@public.gmane.org>,
Bjorn Helgaas <bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>,
Joerg Roedel <joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org>
Cc: Allen Hubbe <allenbh-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
Serge Semin
<fancer.lancer-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
Eric Pilmore <epilmore-Op3I1peydIbQT0dZR+AlfA@public.gmane.org>
Subject: Re: [PATCH 0/9] Support using MSI interrupts in ntb_transport
Date: Thu, 31 Jan 2019 15:39:51 -0700 [thread overview]
Message-ID: <29f7e3fe-5354-6156-1243-7248ffb2249f@deltatee.com> (raw)
In-Reply-To: <ea7b7c3c-8ed5-51ec-022a-7d28a5afce82-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
On 2019-01-31 1:58 p.m., Dave Jiang wrote:
>
> On 1/31/2019 1:48 PM, Logan Gunthorpe wrote:
>>
>> On 2019-01-31 1:20 p.m., Dave Jiang wrote:
>>> Does this work when the system moves the MSI vector either via software
>>> (irqbalance) or BIOS APIC programming (some modes cause round robin
>>> behavior)?
>>
>> I don't know how irqbalance works, and I'm not sure what you are
>> referring to by BIOS APIC programming, however I would expect these
>> things would not be a problem.
>>
>> The MSI code I'm presenting here doesn't do anything crazy with the
>> interrupts, it allocates and uses them just as any PCI driver would. The
>> only real difference here is that instead of a piece of hardware sending
>> the IRQ TLP, it will be sent through the memory window (which, from the
>> OS's perspective, is just coming from an NTB hardware proxy alias).
>>
>> Logan
> Right. I did that as a hack a while back for some silicon errata
> workaround. When the vector moves, the address for the LAPIC changes. So
> unless it gets updated, you end up writing to the old location and lose
> all the new interrupts. irqbalance is a user daemon that rotates the
> system interrupts around to ensure that not all interrupts are pinned on
> a single core.
Yes, that would be a problem if something changes the MSI vectors out
from under us. Seems like that would be a bit difficult to do even with
regular hardware. So far I haven't seen anything that would do that. If
you know of where in the kernel this happens I'd be interested in
getting a pointer to the flow in the code. If that is the case this MSI
stuff will need to get much more complicated...
> I think it's enabled by default on several distros.
> Although MSIX has nothing to do with the IOAPIC, the mode that the APIC
> is programmed can have an influence on how the interrupts are delivered.
> There are certain Intel platforms (I don't know if AMD does anything
> like that) puts the IOAPIC in a certain configuration that causes the
> interrupts to be moved in a round robin fashion. I think it's physical
> flat mode? I don't quite recall. Normally on the low end Xeons. It's
> probably worth doing a test run with the irqbalance daemon running and
> make sure you traffic stream doesn't all of sudden stop.
I've tested with irqbalance running and haven't found any noticeable
difference.
Logan
next prev parent reply other threads:[~2019-01-31 22:39 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-31 18:56 [PATCH 0/9] Support using MSI interrupts in ntb_transport Logan Gunthorpe
2019-01-31 18:56 ` [PATCH 1/9] iommu/vt-d: Allow interrupts from the entire bus for aliased devices Logan Gunthorpe
2019-02-01 16:44 ` Joerg Roedel
2019-02-01 17:27 ` Logan Gunthorpe
2019-02-05 19:19 ` Jacob Pan
2019-02-05 20:40 ` Logan Gunthorpe
2019-02-05 23:58 ` Jacob Pan
2019-01-31 18:56 ` [PATCH 2/9] PCI/MSI: Support allocating virtual MSI interrupts Logan Gunthorpe
2019-01-31 22:39 ` Bjorn Helgaas
2019-01-31 22:52 ` Logan Gunthorpe
2019-02-01 19:23 ` Bjorn Helgaas
2019-01-31 18:56 ` [PATCH 3/9] PCI/switchtec: Add module parameter to request more interrupts Logan Gunthorpe
2019-01-31 18:56 ` [PATCH 4/9] NTB: Introduce functions to calculate multi-port resource index Logan Gunthorpe
2019-01-31 18:56 ` [PATCH 5/9] NTB: Rename ntb.c to support multiple source files in the module Logan Gunthorpe
2019-01-31 18:56 ` [PATCH 6/9] NTB: Introduce MSI library Logan Gunthorpe
2019-01-31 18:56 ` [PATCH 7/9] NTB: Introduce NTB MSI Test Client Logan Gunthorpe
2019-01-31 18:56 ` [PATCH 8/9] NTB: Add ntb_msi_test support to ntb_test Logan Gunthorpe
2019-01-31 18:56 ` [PATCH 9/9] NTB: Add MSI interrupt support to ntb_transport Logan Gunthorpe
2019-01-31 20:20 ` [PATCH 0/9] Support using MSI interrupts in ntb_transport Dave Jiang
2019-01-31 20:48 ` Logan Gunthorpe
[not found] ` <bafe9db4-6d5b-0453-d7e7-e49e9293dd01-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
2019-01-31 20:58 ` Dave Jiang
[not found] ` <ea7b7c3c-8ed5-51ec-022a-7d28a5afce82-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
2019-01-31 22:39 ` Logan Gunthorpe [this message]
[not found] ` <29f7e3fe-5354-6156-1243-7248ffb2249f-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
2019-01-31 22:46 ` Dave Jiang
2019-01-31 23:41 ` Logan Gunthorpe
2019-01-31 23:48 ` Dave Jiang
2019-01-31 23:52 ` Logan Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=29f7e3fe-5354-6156-1243-7248ffb2249f@deltatee.com \
--to=logang-otvngxwrz7hwk0htik3j/w@public.gmane.org \
--cc=allenbh-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
--cc=bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org \
--cc=dave.jiang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
--cc=epilmore-Op3I1peydIbQT0dZR+AlfA@public.gmane.org \
--cc=fancer.lancer-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
--cc=iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org \
--cc=jdmason-zqedIHPRSYI@public.gmane.org \
--cc=joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org \
--cc=linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=linux-kselftest-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=linux-ntb-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org \
--cc=linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).