vgic initialization with maximum number of interrupt lines fails

vgic initialization with maximum number of interrupt lines fails

[Lukas Jünger]

[-- Attachment #1.1: Type: text/plain, Size: 1587 bytes --]

Hi all,

I use Xen in an ARMv8 simulator for research purposes.
I ran into the following problem during dom0 creation in vgic 
initialization.
Bringup fails because -EINVAL is returned by the code below.

xen/arch/arm/vgic.c in domain_vgic_init():
132     nr_spis = ROUNDUP(nr_spis, 32);
133
134     /* Limit the number of virtual SPIs supported to (1020 - 32) = 
988  */
135     if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
136         return -EINVAL;

nr_spis comes from config->arch.nr_spis in xen/arch/arm/domain.c in 
arch_domain_create():
702     if ( (rc = domain_vgic_init(d, config->arch.nr_spis)) != 0 )
703         goto fail;

arch.nr_spis comes from xen/arch/arm/setup.c in start_xen():
891     dom0_cfg.arch.nr_spis = gic_number_lines() - 32;

gic_number_lines() in xen/arch/arm/gic.c:
62 unsigned int gic_number_lines(void)
63 {
64     return gic_hw_ops->info->nr_lines;
65 }

populated in xen/arch/arm/gic-v2.c in gicv2_dist_init:
354     type = readl_gicd(GICD_TYPER);
355     nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
356     /* Only 1020 interrupts are supported */
357     nr_lines = min(1020U, nr_lines);
358     gicv2_info.nr_lines = nr_lines;

GICv2 supports up to 1020 interrupts.
In this case type will be (type & GICD_TYPE_LINES) will be 31, as the 
bits [4:0] of GICD_TYPER will be 1.
nr_lines will be 1024 and finally reduced to 1020, because of special 
purpose interrupts.
Later it will be rounded up to 1024 again and then the check will fail.
Is this expected behavior and if yes, why?

Best regards,
Lukas


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: vgic initialization with maximum number of interrupt lines fails

[Julien Grall]

On 22/03/2019 13:30, Lukas Jünger wrote:
> Hi all,

Hi,

> I use Xen in an ARMv8 simulator for research purposes.
> I ran into the following problem during dom0 creation in vgic initialization.
> Bringup fails because -EINVAL is returned by the code below.
> 
> xen/arch/arm/vgic.c in domain_vgic_init():
> 132     nr_spis = ROUNDUP(nr_spis, 32);
> 133
> 134     /* Limit the number of virtual SPIs supported to (1020 - 32) = 988  */
> 135     if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
> 136         return -EINVAL;
> 
> nr_spis comes from config->arch.nr_spis in xen/arch/arm/domain.c in 
> arch_domain_create():
> 702     if ( (rc = domain_vgic_init(d, config->arch.nr_spis)) != 0 )
> 703         goto fail;
> 
> arch.nr_spis comes from xen/arch/arm/setup.c in start_xen():
> 891     dom0_cfg.arch.nr_spis = gic_number_lines() - 32;
> 
> gic_number_lines() in xen/arch/arm/gic.c:
> 62 unsigned int gic_number_lines(void)
> 63 {
> 64     return gic_hw_ops->info->nr_lines;
> 65 }
> 
> populated in xen/arch/arm/gic-v2.c in gicv2_dist_init:
> 354     type = readl_gicd(GICD_TYPER);
> 355     nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
> 356     /* Only 1020 interrupts are supported */
> 357     nr_lines = min(1020U, nr_lines);
> 358     gicv2_info.nr_lines = nr_lines;
> 
> GICv2 supports up to 1020 interrupts.
> In this case type will be (type & GICD_TYPE_LINES) will be 31, as the bits [4:0] 
> of GICD_TYPER will be 1.
> nr_lines will be 1024 and finally reduced to 1020, because of special purpose 
> interrupts.
> Later it will be rounded up to 1024 again and then the check will fail.
> Is this expected behavior and if yes, why?

This is a known issues I haven't had time to properly fix yet. The GIC requires 
a multiple of 32 interrupts. In addition, the 2 vGIC implementation rely that 
all "existing" interrupts are fully implemented.

Some of the last lot of 32 interrupts (IRQ 992 - IRQ 1023) contains special 
purpose interrupts. The special purpose interrupts are not fully implemented. So 
it would expose a potential vulnerability issue if we were supporting up to 1020
interrupts.

This is why we limit to 992 interrupts. As Dom0 exposes the same number of 
interrupts as the host, this is an issue on platform where the GIC report 1024 
interrupts. AFAIK, those platforms are only virtual, so my idea was to limit the 
number interrupts exposed to Dom0. Something like:

dom0_cfg.arch.nr_spis = min(gic_number_lines(), 992) - 32;

Does you platform have interrupts wired above 992?

Cheers,

> 
> Best regards,
> Lukas
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: vgic initialization with maximum number of interrupt lines fails

[Lukas Jünger]

[-- Attachment #1.1: Type: text/plain, Size: 3398 bytes --]

On 3/22/19 4:20 PM, Julien Grall wrote:
> 
> 
> On 22/03/2019 13:30, Lukas Jünger wrote:
>> Hi all,
> 
> Hi,
> 

Hi Julien,

Thank you for your quick reply.

>> I use Xen in an ARMv8 simulator for research purposes.
>> I ran into the following problem during dom0 creation in vgic 
>> initialization.
>> Bringup fails because -EINVAL is returned by the code below.
>>
>> xen/arch/arm/vgic.c in domain_vgic_init():
>> 132     nr_spis = ROUNDUP(nr_spis, 32);
>> 133
>> 134     /* Limit the number of virtual SPIs supported to (1020 - 32) = 
>> 988  */
>> 135     if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>> 136         return -EINVAL;
>>
>> nr_spis comes from config->arch.nr_spis in xen/arch/arm/domain.c in 
>> arch_domain_create():
>> 702     if ( (rc = domain_vgic_init(d, config->arch.nr_spis)) != 0 )
>> 703         goto fail;
>>
>> arch.nr_spis comes from xen/arch/arm/setup.c in start_xen():
>> 891     dom0_cfg.arch.nr_spis = gic_number_lines() - 32;
>>
>> gic_number_lines() in xen/arch/arm/gic.c:
>> 62 unsigned int gic_number_lines(void)
>> 63 {
>> 64     return gic_hw_ops->info->nr_lines;
>> 65 }
>>
>> populated in xen/arch/arm/gic-v2.c in gicv2_dist_init:
>> 354     type = readl_gicd(GICD_TYPER);
>> 355     nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
>> 356     /* Only 1020 interrupts are supported */
>> 357     nr_lines = min(1020U, nr_lines);
>> 358     gicv2_info.nr_lines = nr_lines;
>>
>> GICv2 supports up to 1020 interrupts.
>> In this case type will be (type & GICD_TYPE_LINES) will be 31, as the 
>> bits [4:0] of GICD_TYPER will be 1.
>> nr_lines will be 1024 and finally reduced to 1020, because of special 
>> purpose interrupts.
>> Later it will be rounded up to 1024 again and then the check will fail.
>> Is this expected behavior and if yes, why?
> 
> This is a known issues I haven't had time to properly fix yet. The GIC 
> requires a multiple of 32 interrupts. In addition, the 2 vGIC 
> implementation rely that all "existing" interrupts are fully implemented.
> 
> Some of the last lot of 32 interrupts (IRQ 992 - IRQ 1023) contains 
> special purpose interrupts. The special purpose interrupts are not fully 
> implemented. So it would expose a potential vulnerability issue if we 
> were supporting up to 1020
> interrupts.
> 

I was not aware that there were special interrupts starting from IRQ 
992. The GICv2 architecture spec mentions only IRQ1020-1023 to be 
special (Section 3.2.5).

> This is why we limit to 992 interrupts. As Dom0 exposes the same number 
> of interrupts as the host, this is an issue on platform where the GIC 
> report 1024 interrupts. AFAIK, those platforms are only virtual, so my 
> idea was to limit the number interrupts exposed to Dom0. Something like:
> 
> dom0_cfg.arch.nr_spis = min(gic_number_lines(), 992) - 32;
> 
> Does you platform have interrupts wired above 992?

In my simulator I just reduced the number of interrupts reported to be 
available by the GICv2 model. This solves the issue for my use case.

Thanks,
Lukas

> Cheers,
> 
>>
>> Best regards,
>> Lukas
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xenproject.org
>> https://lists.xenproject.org/mailman/listinfo/xen-devel
>>
> 



[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: vgic initialization with maximum number of interrupt lines fails

[Julien Grall]

Hi,

On 22/03/2019 15:39, Lukas Jünger wrote:
> On 3/22/19 4:20 PM, Julien Grall wrote:
>> On 22/03/2019 13:30, Lukas Jünger wrote:
>>> I use Xen in an ARMv8 simulator for research purposes.
>>> I ran into the following problem during dom0 creation in vgic initialization.
>>> Bringup fails because -EINVAL is returned by the code below.
>>>
>>> xen/arch/arm/vgic.c in domain_vgic_init():
>>> 132     nr_spis = ROUNDUP(nr_spis, 32);
>>> 133
>>> 134     /* Limit the number of virtual SPIs supported to (1020 - 32) = 988  */
>>> 135     if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>>> 136         return -EINVAL;
>>>
>>> nr_spis comes from config->arch.nr_spis in xen/arch/arm/domain.c in 
>>> arch_domain_create():
>>> 702     if ( (rc = domain_vgic_init(d, config->arch.nr_spis)) != 0 )
>>> 703         goto fail;
>>>
>>> arch.nr_spis comes from xen/arch/arm/setup.c in start_xen():
>>> 891     dom0_cfg.arch.nr_spis = gic_number_lines() - 32;
>>>
>>> gic_number_lines() in xen/arch/arm/gic.c:
>>> 62 unsigned int gic_number_lines(void)
>>> 63 {
>>> 64     return gic_hw_ops->info->nr_lines;
>>> 65 }
>>>
>>> populated in xen/arch/arm/gic-v2.c in gicv2_dist_init:
>>> 354     type = readl_gicd(GICD_TYPER);
>>> 355     nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
>>> 356     /* Only 1020 interrupts are supported */
>>> 357     nr_lines = min(1020U, nr_lines);
>>> 358     gicv2_info.nr_lines = nr_lines;
>>>
>>> GICv2 supports up to 1020 interrupts.
>>> In this case type will be (type & GICD_TYPE_LINES) will be 31, as the bits 
>>> [4:0] of GICD_TYPER will be 1.
>>> nr_lines will be 1024 and finally reduced to 1020, because of special purpose 
>>> interrupts.
>>> Later it will be rounded up to 1024 again and then the check will fail.
>>> Is this expected behavior and if yes, why?
>>
>> This is a known issues I haven't had time to properly fix yet. The GIC 
>> requires a multiple of 32 interrupts. In addition, the 2 vGIC implementation 
>> rely that all "existing" interrupts are fully implemented.
>>
>> Some of the last lot of 32 interrupts (IRQ 992 - IRQ 1023) contains special 
>> purpose interrupts. The special purpose interrupts are not fully implemented. 
>> So it would expose a potential vulnerability issue if we were supporting up to 
>> 1020
>> interrupts.
>>
> 
> I was not aware that there were special interrupts starting from IRQ 992. The 
> GICv2 architecture spec mentions only IRQ1020-1023 to be special (Section 3.2.5).

Sorry, I wasn't clear enough. You are right, Only interrupts 1020-1023 are 
special, the interrupts 992-1019 are still valid. However, because of the design 
of the vGIC, we require all interrupts in a range of 32 interrupts to be valid. 
That's why we limit to 992 interrupts.

> 
>> This is why we limit to 992 interrupts. As Dom0 exposes the same number of 
>> interrupts as the host, this is an issue on platform where the GIC report 1024 
>> interrupts. AFAIK, those platforms are only virtual, so my idea was to limit 
>> the number interrupts exposed to Dom0. Something like:
>>
>> dom0_cfg.arch.nr_spis = min(gic_number_lines(), 992) - 32;
>>
>> Does you platform have interrupts wired above 992?
> 
> In my simulator I just reduced the number of interrupts reported to be available 
> by the GICv2 model. This solves the issue for my use case.

Thank you for the confirmation. So a patch like above in Xen should work for us.

OOI, which simulator are you using?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: vgic initialization with maximum number of interrupt lines fails

[Lukas Jünger]

[-- Attachment #1.1: Type: text/plain, Size: 4054 bytes --]

Hi,

On 3/22/19 5:12 PM, Julien Grall wrote:
> Hi,
> 
> On 22/03/2019 15:39, Lukas Jünger wrote:
>> On 3/22/19 4:20 PM, Julien Grall wrote:
>>> On 22/03/2019 13:30, Lukas Jünger wrote:
>>>> I use Xen in an ARMv8 simulator for research purposes.
>>>> I ran into the following problem during dom0 creation in vgic 
>>>> initialization.
>>>> Bringup fails because -EINVAL is returned by the code below.
>>>>
>>>> xen/arch/arm/vgic.c in domain_vgic_init():
>>>> 132     nr_spis = ROUNDUP(nr_spis, 32);
>>>> 133
>>>> 134     /* Limit the number of virtual SPIs supported to (1020 - 32) 
>>>> = 988  */
>>>> 135     if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>>>> 136         return -EINVAL;
>>>>
>>>> nr_spis comes from config->arch.nr_spis in xen/arch/arm/domain.c in 
>>>> arch_domain_create():
>>>> 702     if ( (rc = domain_vgic_init(d, config->arch.nr_spis)) != 0 )
>>>> 703         goto fail;
>>>>
>>>> arch.nr_spis comes from xen/arch/arm/setup.c in start_xen():
>>>> 891     dom0_cfg.arch.nr_spis = gic_number_lines() - 32;
>>>>
>>>> gic_number_lines() in xen/arch/arm/gic.c:
>>>> 62 unsigned int gic_number_lines(void)
>>>> 63 {
>>>> 64     return gic_hw_ops->info->nr_lines;
>>>> 65 }
>>>>
>>>> populated in xen/arch/arm/gic-v2.c in gicv2_dist_init:
>>>> 354     type = readl_gicd(GICD_TYPER);
>>>> 355     nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
>>>> 356     /* Only 1020 interrupts are supported */
>>>> 357     nr_lines = min(1020U, nr_lines);
>>>> 358     gicv2_info.nr_lines = nr_lines;
>>>>
>>>> GICv2 supports up to 1020 interrupts.
>>>> In this case type will be (type & GICD_TYPE_LINES) will be 31, as 
>>>> the bits [4:0] of GICD_TYPER will be 1.
>>>> nr_lines will be 1024 and finally reduced to 1020, because of 
>>>> special purpose interrupts.
>>>> Later it will be rounded up to 1024 again and then the check will fail.
>>>> Is this expected behavior and if yes, why?
>>>
>>> This is a known issues I haven't had time to properly fix yet. The 
>>> GIC requires a multiple of 32 interrupts. In addition, the 2 vGIC 
>>> implementation rely that all "existing" interrupts are fully 
>>> implemented.
>>>
>>> Some of the last lot of 32 interrupts (IRQ 992 - IRQ 1023) contains 
>>> special purpose interrupts. The special purpose interrupts are not 
>>> fully implemented. So it would expose a potential vulnerability issue 
>>> if we were supporting up to 1020
>>> interrupts.
>>>
>>
>> I was not aware that there were special interrupts starting from IRQ 
>> 992. The GICv2 architecture spec mentions only IRQ1020-1023 to be 
>> special (Section 3.2.5).
> 
> Sorry, I wasn't clear enough. You are right, Only interrupts 1020-1023 
> are special, the interrupts 992-1019 are still valid. However, because 
> of the design of the vGIC, we require all interrupts in a range of 32 
> interrupts to be valid. That's why we limit to 992 interrupts.
> 

Ah, I see. Thanks for the input.

>>
>>> This is why we limit to 992 interrupts. As Dom0 exposes the same 
>>> number of interrupts as the host, this is an issue on platform where 
>>> the GIC report 1024 interrupts. AFAIK, those platforms are only 
>>> virtual, so my idea was to limit the number interrupts exposed to 
>>> Dom0. Something like:
>>>
>>> dom0_cfg.arch.nr_spis = min(gic_number_lines(), 992) - 32;
>>>
>>> Does you platform have interrupts wired above 992?
>>
>> In my simulator I just reduced the number of interrupts reported to be 
>> available by the GICv2 model. This solves the issue for my use case.
> 
> Thank you for the confirmation. So a patch like above in Xen should work 
> for us.

Should I submit a patch, or do you just patch this yourself? Do you want 
to limit to 992 or 980 like the original code does?

> OOI, which simulator are you using?

It's something based on QEMU, but the GICv2 model is from VCML 
(https://github.com/janweinstock/vcml).


> Cheers,
> 

Thanks,
Lukas


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: vgic initialization with maximum number of interrupt lines fails

[Julien Grall]

Hi,

On 22/03/2019 16:19, Lukas Jünger wrote:
> On 3/22/19 5:12 PM, Julien Grall wrote:
>>>
>>>> This is why we limit to 992 interrupts. As Dom0 exposes the same number of 
>>>> interrupts as the host, this is an issue on platform where the GIC report 
>>>> 1024 interrupts. AFAIK, those platforms are only virtual, so my idea was to 
>>>> limit the number interrupts exposed to Dom0. Something like:
>>>>
>>>> dom0_cfg.arch.nr_spis = min(gic_number_lines(), 992) - 32;
>>>>
>>>> Does you platform have interrupts wired above 992?
>>>
>>> In my simulator I just reduced the number of interrupts reported to be 
>>> available by the GICv2 model. This solves the issue for my use case.
>>
>> Thank you for the confirmation. So a patch like above in Xen should work for us.
> 
> Should I submit a patch, or do you just patch this yourself? 

If you don't mind, please send a patch.

> Do you want to limit to 992 or 980 like the original code does?

I guess you mean 992 or 988. 988 is the maximum number of SPIs. In our case, we 
want maximum 1024 - 32 - 32 = 960 SPIs. The first 32 is to remove the last 32 
interrupts, the second is to subtract the local IRQs.

So the number can either be 992 or 960 depending on where you want to place the 
min(). I am happy with either way.

In both case, you would want to write a comment on top explaining the reason for 
the limitation.

I am happy to guide you through the change if you need to.

> 
>> OOI, which simulator are you using?
> 
> It's something based on QEMU, but the GICv2 model is from VCML 
> (https://github.com/janweinstock/vcml).

Thank you!

Cheers,


-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel