x86 multi-msi w/o iommu

x86 multi-msi w/o iommu

[Jeffrey Hugo]

Hi Thomas,

I'd like to get multi-MSI without IOMMU working on x86.  I'm hoping you 
could help me understand the current state of things, and if there is a 
path toward enabling this.

This is an overly simplistic assessment, but this reportedly works in 
other OSes, so it looks like Linux is broken in comparison.

In my investigation so far, the failure stems from 
x86_vector_alloc_irqs() in arch/x86/kernel/apic/vector.c [1].  If we 
need a contiguous allocation of more than one irq, the allocation 
immediately fails:

	/* Currently vector allocator can't guarantee contiguous allocations */
	if ((info->flags & X86_IRQ_ALLOC_CONTIGUOUS_VECTORS) && nr_irqs > 1)
		return -ENOSYS;


As I'm sure you are aware, this only impacts MSI without IOMMU as both 
MSI-X and MSI with IOMMU can handle a discontinuous allocation (the flag 
is not set for either of those cases).

That check was added back in 2015 with [2].  In 2017, it looks like you 
refactored the allocator to the irq_matrix component [3].  However, the 
limitation remains to today.

Digging a bit further, it looks like the internal function 
matrix_alloc_area() [4] is capable of doing a contiguous allocation, but 
all the callers of that via the public api hardcode num to 1.  I 
wouldn't say it would be trivial to refactor the irq_matrix public api 
to do a contigious range allocation, and refactor 
x86_vector_alloc_irqs() to do that based on 
X86_IRQ_ALLOC_CONTIGUOUS_VECTORS, but since it seems like that hasn't 
been tackled in 5-7 years (depending on how you look at the history), I 
suspect I'm missing something.

Since I'm poking my nose in an area of the kernel that I haven't gone 
poking into before, I'm guessing there is a lot I don't know.  Would you 
kindly educate me on what the concerns might be?

I look forward to your expertise and opinions.

Thanks

-Jeff



[1]: 
https://elixir.bootlin.com/linux/v5.18-rc1/source/arch/x86/kernel/apic/vector.c#L542
[2]: 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/kernel/apic/vector.c?h=v5.18-rc1&id=b5dc8e6c21e7ffba0246bf39cea97805c142bf85
[3]: 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/x86/kernel/apic/vector.c?h=v5.18-rc1&id=69cde0004a4b5cfc7d1cec4ef9ce4cf4e26142f0
[4]: 
https://elixir.bootlin.com/linux/v5.18-rc1/source/kernel/irq/matrix.c#L110

Re: x86 multi-msi w/o iommu

[Thomas Gleixner]

Jeffrey!

On Fri, Apr 08 2022 at 11:34, Jeffrey Hugo wrote:
> I'd like to get multi-MSI without IOMMU working on x86. I'm hoping you
> could help me understand the current state of things, and if there is
> a path toward enabling this.

Hope is definitely required here, but you did not explain yet WHY you
want that to work. Making it work just because is not really a good
starting point.

> This is an overly simplistic assessment, but this reportedly works in 
> other OSes, so it looks like Linux is broken in comparison.
>
> In my investigation so far, the failure stems from
> x86_vector_alloc_irqs() in arch/x86/kernel/apic/vector.c [1].

The failure? That's not a failure, that's a deliberate decision.

Side note: Please spare us the links to random code sites and git web
interfaces. A function name and a commit id is good enough.

> If we need a contiguous allocation of more than one irq, the
> allocation immediately fails:
>
> 	/* Currently vector allocator can't guarantee contiguous allocations */
> 	if ((info->flags & X86_IRQ_ALLOC_CONTIGUOUS_VECTORS) && nr_irqs > 1)
> 		return -ENOSYS;
>
> As I'm sure you are aware, this only impacts MSI without IOMMU as both 
> MSI-X and MSI with IOMMU can handle a discontinuous allocation (the flag 
> is not set for either of those cases).
>
> That check was added back in 2015 with [2].

X86 never supported multi-MSI in Linux. See:

    commit 1c8d7b0a562d ("PCI MSI: Add support for multiple MSI")

> In 2017, it looks like you refactored the allocator to the irq_matrix
> component [3].  However, the limitation remains to today.

For very good reasons.

> Digging a bit further, it looks like the internal function 
> matrix_alloc_area() [4] is capable of doing a contiguous allocation, but 
> all the callers of that via the public api hardcode num to 1.  I 
> wouldn't say it would be trivial to refactor the irq_matrix public api 
> to do a contigious range allocation, and refactor 
> x86_vector_alloc_irqs() to do that based on 
> X86_IRQ_ALLOC_CONTIGUOUS_VECTORS, but since it seems like that hasn't 
> been tackled in 5-7 years (depending on how you look at the history), I 
> suspect I'm missing something.

There are two fundamental issues with multi-MSI:

   1) General

      a) Multi-MSI is a single MSI message. The device sets the vector
         offset in the lower bits of message data.

      b) As a consequence the Multi-MSI interrupts of a given device are
         all affine to a single target and you cannot set affinity for
         them separately, which limits the usefulness very much.

   2) x86

      a) Due to #1a this requires N consecutive vectors on one CPU and
         these vectors have to be aligned so that the device can set the
         vector offset into the lower bits of message data. The
         alignment requirement depends on the number of vectors and is
         always power of 2 in the range (2, 4, 8, 16, 32).

         Easy to do in theory. But in practice there is the limited
         vector space on x86 (~200) for device interrupts per CPU.

         That creates a problem for allocations in general and for CPU
         hotplug. On hotplug all active interrupts are moved away from
         the outgoing CPU. Due to the alignment requirement this is
         pretty much a guarantee for vector exhaustion.

      b) Changing interrupt affinity for MSI w/o IOMMU is an interesting
         exercise on X86 when the interrupt is not maskable, which is
         unfortunately the case in the majority of MSI hardware
         implementations.

         In that case while the new vectors are installed interrupts can
         be issued by the device. So you need to be very careful _not_
         to lose an interrupt in the case that both the message address
         (the target APIC) and the message data (the vector) are
         changed. You can find the gory details in:

                  arch/x86/kerne/apic/msi.c::msi_set_affinity()

         and the related change logs espescially:
         6f1a4891a592 ("x86/apic/msi: Plug non-maskable MSI affinity race")

         Due to #1b changing interrupt affinity has to move _all_
         vectors associated to the device at once, which makes this
         excercise even more interesting.

         This code is already horrible as hell and the thought alone to
         expand it for multi MSI makes me shudder.

Now with interrupt remapping all of the above goes away:

    - The alignment problem and consecutive space issue moves into
      the remap tables which have plenty of space.

    - Each interrupts affinity can be individually controlled because
      the affinity setting happens in the remap table and does not
      require the horrors of the non remapped case.

That means in Multi-MSI can be implemented on x86 w/o remapping, but is
it worth the trouble? From looking at the problem space and under
consideration that the advantage of multi-MSI is very limited the
decision was made to not support it. That's not broken as you claim,
that's a very reasonable technical decision.

From experimentation I know that multi-MSI w/o remapping has a very
limited benefit because it cannot provide scalability through
parallelism obviously. The only benefit is a clear separation of
interrupt functionality which spares a MMIO read to get the pending bits
and a few conditionals. Here is a trivial experiment to demonstrate the
benefit or the lack of it:

    Use a system with interrupt remapping and add a module parameter to
    your driver which allows you to select ONE or MANY MSIs. Then run
    performance tests with both setups.
    
The result might not be what you expect depending on the device and the
resulting interrupt patterns.

The problems of MSI are known since two decades, but hardware folks
still insist on it. MSI-X exists for a reason, but sure it makes the
chips $0.01 more expensive and lots of hardware people still work under
the assumption that whatever they get wrong can be fixed in software,
which is unfortunately wishful thinking.

I hope this clarifies it for you.

Thanks,

        tglx

Re: x86 multi-msi w/o iommu

[Jeffrey Hugo]

On 4/9/2022 2:08 PM, Thomas Gleixner wrote:
> Jeffrey!
> 
> On Fri, Apr 08 2022 at 11:34, Jeffrey Hugo wrote:
>> I'd like to get multi-MSI without IOMMU working on x86. I'm hoping you
>> could help me understand the current state of things, and if there is
>> a path toward enabling this.
> 
> Hope is definitely required here, but you did not explain yet WHY you
> want that to work. Making it work just because is not really a good
> starting point.

Arguably Linux exists "just because", but its a fair question.

Someone made a request for this.  Trying to figure out the logistics of 
the request before responding to the requestor.

There is a particular device that due to hardware limitations (arguably 
mistakes were made) it only supports MSI and it requires more than 1 MSI 
to work.

While the device is documented as requiring an IOMMU on x86, the 
requesting entity wants to use the device without an IOMMU in a unusual 
setup and pointed it that it works (for some definition of working) on 
other Operating Systems.  So from their viewpoint is not a system issue, 
but a Linux issue.

Granted, I suspect you disagree with this being a Linux issue per the 
below.  I'm trying to understand all of the aspects.

I suspect this is not the level of detail you would like, but from my 
perspective parts of this situation are sensitive so I'm trying to 
proceed as best I can currently (arguably with one hand tied behind my 
back).  To be clear, this level of secrecy is unacceptable when working 
with the community and if I were to move forward to propose changes 
based on the information you've provided, I would be more transparent.

I greatly appreciate your response given the limitations in what I've 
provided.

>> This is an overly simplistic assessment, but this reportedly works in
>> other OSes, so it looks like Linux is broken in comparison.
>>
>> In my investigation so far, the failure stems from
>> x86_vector_alloc_irqs() in arch/x86/kernel/apic/vector.c [1].
> 
> The failure? That's not a failure, that's a deliberate decision.

I'm sorry, I choose my words poorly.  I was viewing things from the 
perspective of reproducing the reported "issue" and seeing why that is 
the case.  The device requests multiple MSIs and doesn't get them, which 
is a "failure" from the perspective that the expectation is things work.

I do however see that this is deliberate per the code, and thus would 
like to understand why.  Fundamentally, that is why I asked you for help.

 From one perspective, its a "failure", from another, its "working as 
intended".  I did not intend any offense, and I'm more interested with 
understanding the "why" than applying a specific label.

> Side note: Please spare us the links to random code sites and git web
> interfaces. A function name and a commit id is good enough.
> 
>> If we need a contiguous allocation of more than one irq, the
>> allocation immediately fails:
>>
>> 	/* Currently vector allocator can't guarantee contiguous allocations */
>> 	if ((info->flags & X86_IRQ_ALLOC_CONTIGUOUS_VECTORS) && nr_irqs > 1)
>> 		return -ENOSYS;
>>
>> As I'm sure you are aware, this only impacts MSI without IOMMU as both
>> MSI-X and MSI with IOMMU can handle a discontinuous allocation (the flag
>> is not set for either of those cases).
>>
>> That check was added back in 2015 with [2].
> 
> X86 never supported multi-MSI in Linux. See:
> 
>      commit 1c8d7b0a562d ("PCI MSI: Add support for multiple MSI")
> 
>> In 2017, it looks like you refactored the allocator to the irq_matrix
>> component [3].  However, the limitation remains to today.
> 
> For very good reasons.
> 
>> Digging a bit further, it looks like the internal function
>> matrix_alloc_area() [4] is capable of doing a contiguous allocation, but
>> all the callers of that via the public api hardcode num to 1.  I
>> wouldn't say it would be trivial to refactor the irq_matrix public api
>> to do a contigious range allocation, and refactor
>> x86_vector_alloc_irqs() to do that based on
>> X86_IRQ_ALLOC_CONTIGUOUS_VECTORS, but since it seems like that hasn't
>> been tackled in 5-7 years (depending on how you look at the history), I
>> suspect I'm missing something.
> 
> There are two fundamental issues with multi-MSI:
> 
>     1) General
> 
>        a) Multi-MSI is a single MSI message. The device sets the vector
>           offset in the lower bits of message data.
> 
>        b) As a consequence the Multi-MSI interrupts of a given device are
>           all affine to a single target and you cannot set affinity for
>           them separately, which limits the usefulness very much.
> 
>     2) x86
> 
>        a) Due to #1a this requires N consecutive vectors on one CPU and
>           these vectors have to be aligned so that the device can set the
>           vector offset into the lower bits of message data. The
>           alignment requirement depends on the number of vectors and is
>           always power of 2 in the range (2, 4, 8, 16, 32).
> 
>           Easy to do in theory. But in practice there is the limited
>           vector space on x86 (~200) for device interrupts per CPU.
> 
>           That creates a problem for allocations in general and for CPU
>           hotplug. On hotplug all active interrupts are moved away from
>           the outgoing CPU. Due to the alignment requirement this is
>           pretty much a guarantee for vector exhaustion.
> 
>        b) Changing interrupt affinity for MSI w/o IOMMU is an interesting
>           exercise on X86 when the interrupt is not maskable, which is
>           unfortunately the case in the majority of MSI hardware
>           implementations.
> 
>           In that case while the new vectors are installed interrupts can
>           be issued by the device. So you need to be very careful _not_
>           to lose an interrupt in the case that both the message address
>           (the target APIC) and the message data (the vector) are
>           changed. You can find the gory details in:
> 
>                    arch/x86/kerne/apic/msi.c::msi_set_affinity()
> 
>           and the related change logs espescially:
>           6f1a4891a592 ("x86/apic/msi: Plug non-maskable MSI affinity race")
> 
>           Due to #1b changing interrupt affinity has to move _all_
>           vectors associated to the device at once, which makes this
>           excercise even more interesting.
> 
>           This code is already horrible as hell and the thought alone to
>           expand it for multi MSI makes me shudder.
> 
> Now with interrupt remapping all of the above goes away:
> 
>      - The alignment problem and consecutive space issue moves into
>        the remap tables which have plenty of space.
> 
>      - Each interrupts affinity can be individually controlled because
>        the affinity setting happens in the remap table and does not
>        require the horrors of the non remapped case.
> 
> That means in Multi-MSI can be implemented on x86 w/o remapping, but is
> it worth the trouble? From looking at the problem space and under
> consideration that the advantage of multi-MSI is very limited the
> decision was made to not support it. That's not broken as you claim,
> that's a very reasonable technical decision.

Again, sorry I choose my words poorly.  I understand the summary of the 
issues which lead to the decision, which I did not previously understand 
before.  I think I need to dig into the references you provided to fully 
comprehend the situation and come to the same conclusion.

I think the above is what I was looking for.  I thank you for 
elaborating on the issues and pointing me toward where I can learn more.

>  From experimentation I know that multi-MSI w/o remapping has a very
> limited benefit because it cannot provide scalability through
> parallelism obviously. The only benefit is a clear separation of
> interrupt functionality which spares a MMIO read to get the pending bits
> and a few conditionals. Here is a trivial experiment to demonstrate the
> benefit or the lack of it:
> 
>      Use a system with interrupt remapping and add a module parameter to
>      your driver which allows you to select ONE or MANY MSIs. Then run
>      performance tests with both setups.
>      
> The result might not be what you expect depending on the device and the
> resulting interrupt patterns.

This is a fine suggestion and if it were possible, I would go forward 
with it rather than investigate understanding the issues and developing 
a solution to address them (under the assumption its possible, theory 
and reality are often different).

Sadly, as mentioned, device limitations prevent operation with just one 
MSI which is why I'm seeking to understand if multi-MSI with no IOMMU is 
possible in Linux.  As you point out below sometimes software is put in 
an unfortunate position ("between a rock and a hard place").  All I can 
do is evaluate the various options and pick the least bad one.  Right 
now, I don't know that $SUBJECT is the best option, but I think I can 
evaluate it and compare based on your assistance.

> The problems of MSI are known since two decades, but hardware folks
> still insist on it. MSI-X exists for a reason, but sure it makes the
> chips $0.01 more expensive and lots of hardware people still work under
> the assumption that whatever they get wrong can be fixed in software,
> which is unfortunately wishful thinking >
> I hope this clarifies it for you.

Yes, very much so.  Thank you!

-Jeff