ACPI fixmap overflow

ACPI fixmap overflow

[Boris Ostrovsky]

In addition to running out of e820 entries on that large machine that
Alex was referring to in [0] he is also running out of ACPI fixmap space
while parsing MADT (the box has *lots* of processors). The
quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
wonder whether we should move to dynamic memory allocation.

-boris

[0]
https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg01077.html


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: ACPI fixmap overflow

[Andrew Cooper]

On 15/11/16 19:34, Boris Ostrovsky wrote:
> In addition to running out of e820 entries on that large machine that
> Alex was referring to in [0] he is also running out of ACPI fixmap space
> while parsing MADT (the box has *lots* of processors). The
> quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
> wonder whether we should move to dynamic memory allocation.

Why do we use fixmap anyway?  It doesn't look too hard to reorder
vm_init() slightly higher, and be able to use ioremap() for all APCI tables.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: ACPI fixmap overflow

[Boris Ostrovsky]

On 11/15/2016 02:45 PM, Andrew Cooper wrote:
> On 15/11/16 19:34, Boris Ostrovsky wrote:
>> In addition to running out of e820 entries on that large machine that
>> Alex was referring to in [0] he is also running out of ACPI fixmap space
>> while parsing MADT (the box has *lots* of processors). The
>> quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
>> wonder whether we should move to dynamic memory allocation.
> Why do we use fixmap anyway?  It doesn't look too hard to reorder
> vm_init() slightly higher, and be able to use ioremap() for all APCI tables.

Hmm... Let me see how possible this is. Just moving it up won't work
since heap allocator is initialized after ACPI tables.

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: ACPI fixmap overflow

[Andrew Cooper]

On 15/11/2016 20:39, Boris Ostrovsky wrote:
> On 11/15/2016 02:45 PM, Andrew Cooper wrote:
>> On 15/11/16 19:34, Boris Ostrovsky wrote:
>>> In addition to running out of e820 entries on that large machine that
>>> Alex was referring to in [0] he is also running out of ACPI fixmap space
>>> while parsing MADT (the box has *lots* of processors). The
>>> quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
>>> wonder whether we should move to dynamic memory allocation.
>> Why do we use fixmap anyway?  It doesn't look too hard to reorder
>> vm_init() slightly higher, and be able to use ioremap() for all APCI tables.
> Hmm... Let me see how possible this is. Just moving it up won't work
> since heap allocator is initialized after ACPI tables.

We have plenty of usable PTEs already allocated at boot, mainly from the
init pagetables.  Given a static __init vm_bitmap, a new boot-time-only
vm range should be usable without any heap allocations at all.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: ACPI fixmap overflow

[Boris Ostrovsky]

On 11/15/2016 03:44 PM, Andrew Cooper wrote:
> On 15/11/2016 20:39, Boris Ostrovsky wrote:
>> On 11/15/2016 02:45 PM, Andrew Cooper wrote:
>>> On 15/11/16 19:34, Boris Ostrovsky wrote:
>>>> In addition to running out of e820 entries on that large machine that
>>>> Alex was referring to in [0] he is also running out of ACPI fixmap space
>>>> while parsing MADT (the box has *lots* of processors). The
>>>> quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
>>>> wonder whether we should move to dynamic memory allocation.
>>> Why do we use fixmap anyway?  It doesn't look too hard to reorder
>>> vm_init() slightly higher, and be able to use ioremap() for all APCI tables.
>> Hmm... Let me see how possible this is. Just moving it up won't work
>> since heap allocator is initialized after ACPI tables.
> We have plenty of usable PTEs already allocated at boot, mainly from the
> init pagetables.  Given a static __init vm_bitmap, a new boot-time-only
> vm range should be usable without any heap allocations at all.

Wouldn't that (using pre-allocated PTEs), in a way, be equivalent to
increasing fixmap size?

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: ACPI fixmap overflow

[Jan Beulich]

>>> On 15.11.16 at 21:51, <boris.ostrovsky@oracle.com> wrote:
> On 11/15/2016 03:44 PM, Andrew Cooper wrote:
>> On 15/11/2016 20:39, Boris Ostrovsky wrote:
>>> On 11/15/2016 02:45 PM, Andrew Cooper wrote:
>>>> On 15/11/16 19:34, Boris Ostrovsky wrote:
>>>>> In addition to running out of e820 entries on that large machine that
>>>>> Alex was referring to in [0] he is also running out of ACPI fixmap space
>>>>> while parsing MADT (the box has *lots* of processors). The
>>>>> quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
>>>>> wonder whether we should move to dynamic memory allocation.
>>>> Why do we use fixmap anyway?  It doesn't look too hard to reorder
>>>> vm_init() slightly higher, and be able to use ioremap() for all APCI tables.
>>> Hmm... Let me see how possible this is. Just moving it up won't work
>>> since heap allocator is initialized after ACPI tables.
>> We have plenty of usable PTEs already allocated at boot, mainly from the
>> init pagetables.  Given a static __init vm_bitmap, a new boot-time-only
>> vm range should be usable without any heap allocations at all.
> 
> Wouldn't that (using pre-allocated PTEs), in a way, be equivalent to
> increasing fixmap size?

Indeed. For the time being I think growing the fixmap should be
fine. Clearly it being a fixed 4 pages has been wrong for a long
time - there's no point in it being smaller than MAX_LOCAL_APIC
x2apic entries, plus min(MAX_LOCAL_APIC, 256) lapic ones, plus
MAX_IO_APICS ioapic ones etc.

Otoh the actual parsing of MADT happens after the heap allocator
has been initialized. The only earlier use is via acpi_table_init() ->
check_multiple_madt(); acpi_initialize_tables() doesn't appear to
map full tables. And check_multiple_madt() not being able to map
the full table would not prevent boot from continuing afaics. So
Andrew's suggestion to switch to dynamic mapping earlier would
still seem to be possible (and then preferred). In fact
acpi_boot_init() already gets called after vm_init(), so switching
acpi_os_map_memory() to use dynamic mappings when
>= SYS_STATE_boot should already work today (on x86 at least,
not sure about ARM).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: ACPI fixmap overflow

[Boris Ostrovsky]

On 11/16/2016 05:08 AM, Jan Beulich wrote:
>>>> On 15.11.16 at 21:51, <boris.ostrovsky@oracle.com> wrote:
>> On 11/15/2016 03:44 PM, Andrew Cooper wrote:
>>> On 15/11/2016 20:39, Boris Ostrovsky wrote:
>>>> On 11/15/2016 02:45 PM, Andrew Cooper wrote:
>>>>> On 15/11/16 19:34, Boris Ostrovsky wrote:
>>>>>> In addition to running out of e820 entries on that large machine that
>>>>>> Alex was referring to in [0] he is also running out of ACPI fixmap space
>>>>>> while parsing MADT (the box has *lots* of processors). The
>>>>>> quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
>>>>>> wonder whether we should move to dynamic memory allocation.
>>>>> Why do we use fixmap anyway?  It doesn't look too hard to reorder
>>>>> vm_init() slightly higher, and be able to use ioremap() for all APCI tables.
>>>> Hmm... Let me see how possible this is. Just moving it up won't work
>>>> since heap allocator is initialized after ACPI tables.
>>> We have plenty of usable PTEs already allocated at boot, mainly from the
>>> init pagetables.  Given a static __init vm_bitmap, a new boot-time-only
>>> vm range should be usable without any heap allocations at all.
>> Wouldn't that (using pre-allocated PTEs), in a way, be equivalent to
>> increasing fixmap size?
> Indeed. For the time being I think growing the fixmap should be
> fine. Clearly it being a fixed 4 pages has been wrong for a long
> time - there's no point in it being smaller than MAX_LOCAL_APIC
> x2apic entries, plus min(MAX_LOCAL_APIC, 256) lapic ones, plus
> MAX_IO_APICS ioapic ones etc.
>
> Otoh the actual parsing of MADT happens after the heap allocator
> has been initialized. The only earlier use is via acpi_table_init() ->
> check_multiple_madt(); acpi_initialize_tables() doesn't appear to
> map full tables. And check_multiple_madt() not being able to map
> the full table would not prevent boot from continuing afaics. So
> Andrew's suggestion to switch to dynamic mapping earlier would
> still seem to be possible (and then preferred). In fact
> acpi_boot_init() already gets called after vm_init(), so switching
> acpi_os_map_memory() to use dynamic mappings when
>> = SYS_STATE_boot should already work today (on x86 at least,
> not sure about ARM).

Yes, switching to ioremap on SYS_STATE_boot seems to work. I asked Alex
to test this on OVM (which is 4.4-based).

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: ACPI fixmap overflow

[Alex Thorlton]

On Thu, Nov 17, 2016 at 10:01:15AM -0500, Boris Ostrovsky wrote:
> On 11/16/2016 05:08 AM, Jan Beulich wrote:
> >>>> On 15.11.16 at 21:51, <boris.ostrovsky@oracle.com> wrote:
> >> On 11/15/2016 03:44 PM, Andrew Cooper wrote:
> >>> On 15/11/2016 20:39, Boris Ostrovsky wrote:
> >>>> On 11/15/2016 02:45 PM, Andrew Cooper wrote:
> >>>>> On 15/11/16 19:34, Boris Ostrovsky wrote:
> >>>>>> In addition to running out of e820 entries on that large machine that
> >>>>>> Alex was referring to in [0] he is also running out of ACPI fixmap space
> >>>>>> while parsing MADT (the box has *lots* of processors). The
> >>>>>> quick-and-dirty solution is to increase NUM_FIXMAP_ACPI_PAGES but I
> >>>>>> wonder whether we should move to dynamic memory allocation.
> >>>>> Why do we use fixmap anyway?  It doesn't look too hard to reorder
> >>>>> vm_init() slightly higher, and be able to use ioremap() for all APCI tables.
> >>>> Hmm... Let me see how possible this is. Just moving it up won't work
> >>>> since heap allocator is initialized after ACPI tables.
> >>> We have plenty of usable PTEs already allocated at boot, mainly from the
> >>> init pagetables.  Given a static __init vm_bitmap, a new boot-time-only
> >>> vm range should be usable without any heap allocations at all.
> >> Wouldn't that (using pre-allocated PTEs), in a way, be equivalent to
> >> increasing fixmap size?
> > Indeed. For the time being I think growing the fixmap should be
> > fine. Clearly it being a fixed 4 pages has been wrong for a long
> > time - there's no point in it being smaller than MAX_LOCAL_APIC
> > x2apic entries, plus min(MAX_LOCAL_APIC, 256) lapic ones, plus
> > MAX_IO_APICS ioapic ones etc.
> >
> > Otoh the actual parsing of MADT happens after the heap allocator
> > has been initialized. The only earlier use is via acpi_table_init() ->
> > check_multiple_madt(); acpi_initialize_tables() doesn't appear to
> > map full tables. And check_multiple_madt() not being able to map
> > the full table would not prevent boot from continuing afaics. So
> > Andrew's suggestion to switch to dynamic mapping earlier would
> > still seem to be possible (and then preferred). In fact
> > acpi_boot_init() already gets called after vm_init(), so switching
> > acpi_os_map_memory() to use dynamic mappings when
> >> = SYS_STATE_boot should already work today (on x86 at least,
> > not sure about ARM).
> 
> Yes, switching to ioremap on SYS_STATE_boot seems to work. I asked Alex
> to test this on OVM (which is 4.4-based).

FYI, I'm going to get this done a little later today.  I'll let you know
how it worked out this evening!

- Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel