Re: [PATCH 15/15] powerpc/powernv/sriov: Make single PE mode a per-BAR setting

[Alexey Kardashevskiy <aik@ozlabs.ru>]

On 22/07/2020 15:39, Oliver O'Halloran wrote:
> On Wed, Jul 15, 2020 at 6:00 PM Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
>>
>>>>>                *
>>>>> -              * Generally, one M64 BAR maps one IOV BAR. To avoid conflict
>>>>> -              * with other devices, IOV BAR size is expanded to be
>>>>> -              * (total_pe * VF_BAR_size).  When VF_BAR_size is half of M64
>>>>> -              * segment size , the expanded size would equal to half of the
>>>>> -              * whole M64 space size, which will exhaust the M64 Space and
>>>>> -              * limit the system flexibility.  This is a design decision to
>>>>> -              * set the boundary to quarter of the M64 segment size.
>>>>> +              * The 1/4 limit is arbitrary and can be tweaked.
>>>>>                */
>>>>> -             if (total_vf_bar_sz > gate) {
>>>>> -                     mul = roundup_pow_of_two(total_vfs);
>>>>> -                     dev_info(&pdev->dev,
>>>>> -                             "VF BAR Total IOV size %llx > %llx, roundup to %d VFs\n",
>>>>> -                             total_vf_bar_sz, gate, mul);
>>>>> -                     iov->m64_single_mode = true;
>>>>> -                     break;
>>>>> -             }
>>>>> -     }
>>>>> +             if (vf_bar_sz > (phb->ioda.m64_segsize >> 2)) {
>>>>> +                     /*
>>>>> +                      * On PHB3, the minimum size alignment of M64 BAR in
>>>>> +                      * single mode is 32MB. If this VF BAR is smaller than
>>>>> +                      * 32MB, but still too large for a segmented window
>>>>> +                      * then we can't map it and need to disable SR-IOV for
>>>>> +                      * this device.
>>>>
>>>>
>>>> Why not use single PE mode for such BAR? Better than nothing.
>>>
>>> Suppose you could, but I figured VFs were mainly interesting since you
>>> could give each VF to a separate guest. If there's multiple VFs under
>>> the same single PE BAR then they'd have to be assigned to the same
>>
>> True. But with one PE per VF we can still have 15 (or 14?) isolated VFs
>> which is not hundreds but better than 0.
> 
> We can only use single PE BARs if the per-VF size is >= 32MB due to
> the alignment requirements on P8. If the per-VF size is smaller then
> we're stuck with multiple VFs inside the same BAR which is bad due to
> the PAPR requirements mentioned below. Sure we could look at doing
> something else, but considering this matches the current behaviour
> it's a bit hard to care...
>
>>> guest in order to retain the freeze/unfreeze behaviour that PAPR
>>> requires. I guess that's how it used to work, but it seems better just
>>> to disable them rather than having VFs which sort of work.
>>
>> Well, realistically the segment size should be 8MB to make this matter
>> (or the whole window 2GB) which does not seem to happen so it does not
>> matter.
> 
> I'm not sure what you mean.

I mean how can we possibly hit this case, what m64_segsize would the
platform have to trigger this. The whole check seems useless but whatever.



-- 
Alexey