DMA restriction and NUMA node number

DMA restriction and NUMA node number

[Wei Chen]

Hi,

I am doing some NUMA testing on Xen. And I find the DMA restriction is
based on NUMA node number [1].
    if ( !dma_bitsize && (num_online_nodes() > 1) )
        dma_bitsize = arch_get_dma_bitsize();

On Arm64, we will set dma_bitsize [2] to 0, that means we don't need to
reserve DMA memory. But when num_online_nodes > 1, the dma_bitsize
will override to 32. This may be caused by the Arm64 version
arch_get_dma_bitsize, it may be a simple implementation and not NUMA
aware.

But I still quite curious about why DMA restriction depends on NUMA
node number. In Arm64, dma_bitsize does not change when the NUMA node
changes. So we didn't expect arch_get_dma_bitsize to be called here.

I copied Keir's commit message from 2008. It seems this code was considered
only for x86, when he was working on it. But I'm not an x86 expert, so I
hope Xen x86 folks can give some help. Understanding this will help us to
do some adaptations to Arm in subsequent modifications : )

commit accacb43cb7f16e9d1d8c0e58ea72c9d0c32cec2
Author: Keir Fraser <keir.fraser@citrix.com>
Date:   Mon Jul 28 16:40:30 2008 +0100

    Simplify 'dma heap' logic.

    1. Only useful for NUMA systems, so turn it off on non-NUMA systems by
       default.
    2. On NUMA systems, by default relate the DMA heap size to NUMA node 0
       memory size (so that not all of node 0's memory ends up being 'DMA
       heap').
    3. Remove the 'dma emergency pool'. It's less useful now that running
       out of low memory isn;t as fatal as it used to be (e.g., when we
       needed to be able to allocate low-memory PAE page directories).

[1] https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/common/page_alloc.c;h=958ba0cd9256c8270e38585d272be2bf5cc0679e;hb=refs/heads/master#l1876
[2] https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/common/page_alloc.c;h=958ba0cd9256c8270e38585d272be2bf5cc0679e;hb=refs/heads/master#l226


--
Cheers,
Wei Chen

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: DMA restriction and NUMA node number

[Julien Grall]

(+Andrew, Jan, Roger)

On 13/07/2021 04:19, Wei Chen wrote:
> Hi,
> 
> I am doing some NUMA testing on Xen. And I find the DMA restriction is
> based on NUMA node number [1].
>      if ( !dma_bitsize && (num_online_nodes() > 1) )
>          dma_bitsize = arch_get_dma_bitsize();
> 
> On Arm64, we will set dma_bitsize [2] to 0, that means we don't need to
> reserve DMA memory. But when num_online_nodes > 1, the dma_bitsize
> will override to 32. This may be caused by the Arm64 version
> arch_get_dma_bitsize, it may be a simple implementation and not NUMA
> aware.
> 
> But I still quite curious about why DMA restriction depends on NUMA
> node number. In Arm64, dma_bitsize does not change when the NUMA node
> changes. So we didn't expect arch_get_dma_bitsize to be called here.
> 
> I copied Keir's commit message from 2008. It seems this code was considered
> only for x86, when he was working on it. But I'm not an x86 expert, so I
> hope Xen x86 folks can give some help. Understanding this will help us to

It is best to CCed the relevant person so they know you have requested 
there input. I have added the x86 maintainers in the thread.

> do some adaptations to Arm in subsequent modifications : )
> 
> commit accacb43cb7f16e9d1d8c0e58ea72c9d0c32cec2
> Author: Keir Fraser <keir.fraser@citrix.com>
> Date:   Mon Jul 28 16:40:30 2008 +0100
> 
>      Simplify 'dma heap' logic.
> 
>      1. Only useful for NUMA systems, so turn it off on non-NUMA systems by
>         default.
>      2. On NUMA systems, by default relate the DMA heap size to NUMA node 0
>         memory size (so that not all of node 0's memory ends up being 'DMA
>         heap').
>      3. Remove the 'dma emergency pool'. It's less useful now that running
>         out of low memory isn;t as fatal as it used to be (e.g., when we
>         needed to be able to allocate low-memory PAE page directories).
> 
> [1] https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/common/page_alloc.c;h=958ba0cd9256c8270e38585d272be2bf5cc0679e;hb=refs/heads/master#l1876
> [2] https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/common/page_alloc.c;h=958ba0cd9256c8270e38585d272be2bf5cc0679e;hb=refs/heads/master#l226

Cheers,

-- 
Julien Grall

Re: DMA restriction and NUMA node number

[Jan Beulich]

On 13.07.2021 11:26, Julien Grall wrote:
> On 13/07/2021 04:19, Wei Chen wrote:
>> I am doing some NUMA testing on Xen. And I find the DMA restriction is
>> based on NUMA node number [1].
>>      if ( !dma_bitsize && (num_online_nodes() > 1) )
>>          dma_bitsize = arch_get_dma_bitsize();
>>
>> On Arm64, we will set dma_bitsize [2] to 0, that means we don't need to
>> reserve DMA memory. But when num_online_nodes > 1, the dma_bitsize
>> will override to 32. This may be caused by the Arm64 version
>> arch_get_dma_bitsize, it may be a simple implementation and not NUMA
>> aware.
>>
>> But I still quite curious about why DMA restriction depends on NUMA
>> node number.

So really do you mean "node count", not "node number"?

>> In Arm64, dma_bitsize does not change when the NUMA node
>> changes. So we didn't expect arch_get_dma_bitsize to be called here.
>>
>> I copied Keir's commit message from 2008. It seems this code was considered
>> only for x86, when he was working on it. But I'm not an x86 expert, so I
>> hope Xen x86 folks can give some help. Understanding this will help us to
> 
> It is best to CCed the relevant person so they know you have requested 
> there input. I have added the x86 maintainers in the thread.
> 
>> do some adaptations to Arm in subsequent modifications : )
>>
>> commit accacb43cb7f16e9d1d8c0e58ea72c9d0c32cec2
>> Author: Keir Fraser <keir.fraser@citrix.com>
>> Date:   Mon Jul 28 16:40:30 2008 +0100
>>
>>      Simplify 'dma heap' logic.
>>
>>      1. Only useful for NUMA systems, so turn it off on non-NUMA systems by
>>         default.
>>      2. On NUMA systems, by default relate the DMA heap size to NUMA node 0
>>         memory size (so that not all of node 0's memory ends up being 'DMA
>>         heap').
>>      3. Remove the 'dma emergency pool'. It's less useful now that running
>>         out of low memory isn;t as fatal as it used to be (e.g., when we
>>         needed to be able to allocate low-memory PAE page directories).

So on x86 memory starts from 0, and we want to be cautious with giving
out memory that may be needed for special purposes (first and foremost
DMA). With the buddy allocator working from high addresses to lower ones,
low addresses will be used last (unless specifically requested) without
any further precautions when not taking NUMA into account. This in
particular covers the case of just a single NUMA node.

When taking NUMA into account we need to be more careful: If a single
node contains the majority (or all) of the more precious memory, we
want to prefer non-local allocations over exhausting the more precious
memory ranges. Hence we need to set aside some largely arbitrary amount
allocation of which would happen only after also exhausting all other
nodes' memory.

I hope I have suitably reconstructed the thinking back then. And yes,
there are x86 implications in here.

Jan

RE: DMA restriction and NUMA node number

[Wei Chen]

Hi Jan,

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: 2021年7月13日 18:21
> To: Julien Grall <julien@xen.org>; Wei Chen <Wei.Chen@arm.com>
> Cc: Penny Zheng <Penny.Zheng@arm.com>; Bertrand Marquis
> <Bertrand.Marquis@arm.com>; xen-devel@lists.xen.org; Roger Pau Monné
> <roger.pau@citrix.com>; Stefano Stabellini <sstabellini@kernel.org>;
> Andrew Cooper <andrew.cooper3@citrix.com>
> Subject: Re: DMA restriction and NUMA node number
>
> On 13.07.2021 11:26, Julien Grall wrote:
> > On 13/07/2021 04:19, Wei Chen wrote:
> >> I am doing some NUMA testing on Xen. And I find the DMA restriction is
> >> based on NUMA node number [1].
> >>      if ( !dma_bitsize && (num_online_nodes() > 1) )
> >>          dma_bitsize = arch_get_dma_bitsize();
> >>
> >> On Arm64, we will set dma_bitsize [2] to 0, that means we don't need to
> >> reserve DMA memory. But when num_online_nodes > 1, the dma_bitsize
> >> will override to 32. This may be caused by the Arm64 version
> >> arch_get_dma_bitsize, it may be a simple implementation and not NUMA
> >> aware.
> >>
> >> But I still quite curious about why DMA restriction depends on NUMA
> >> node number.
>
> So really do you mean "node count", not "node number"?
>

I mean NUMA online nodes. If "node count" here means "count how many NUMA
nodes are online", my answer is yes for "node count".

> >> In Arm64, dma_bitsize does not change when the NUMA node
> >> changes. So we didn't expect arch_get_dma_bitsize to be called here.
> >>
> >> I copied Keir's commit message from 2008. It seems this code was
> considered
> >> only for x86, when he was working on it. But I'm not an x86 expert, so
> I
> >> hope Xen x86 folks can give some help. Understanding this will help us
> to
> >
> > It is best to CCed the relevant person so they know you have requested
> > there input. I have added the x86 maintainers in the thread.
> >
> >> do some adaptations to Arm in subsequent modifications : )
> >>
> >> commit accacb43cb7f16e9d1d8c0e58ea72c9d0c32cec2
> >> Author: Keir Fraser <keir.fraser@citrix.com>
> >> Date:   Mon Jul 28 16:40:30 2008 +0100
> >>
> >>      Simplify 'dma heap' logic.
> >>
> >>      1. Only useful for NUMA systems, so turn it off on non-NUMA
> systems by
> >>         default.
> >>      2. On NUMA systems, by default relate the DMA heap size to NUMA
> node 0
> >>         memory size (so that not all of node 0's memory ends up being
> 'DMA
> >>         heap').
> >>      3. Remove the 'dma emergency pool'. It's less useful now that
> running
> >>         out of low memory isn;t as fatal as it used to be (e.g., when
> we
> >>         needed to be able to allocate low-memory PAE page directories).
>
> So on x86 memory starts from 0, and we want to be cautious with giving
> out memory that may be needed for special purposes (first and foremost
> DMA). With the buddy allocator working from high addresses to lower ones,
> low addresses will be used last (unless specifically requested) without
> any further precautions when not taking NUMA into account. This in
> particular covers the case of just a single NUMA node.
>
> When taking NUMA into account we need to be more careful: If a single
> node contains the majority (or all) of the more precious memory, we
> want to prefer non-local allocations over exhausting the more precious
> memory ranges. Hence we need to set aside some largely arbitrary amount
> allocation of which would happen only after also exhausting all other
> nodes' memory.
>
> I hope I have suitably reconstructed the thinking back then. And yes,
> there are x86 implications in here.
>

Thanks for your explanation, we have a deeper understanding of the
thoughts of this DMA restriction. Although the memory layout of Arm
and x86 is not the same, we can try to keep this idea still in common,
and try resolve this difference in Arm's ARCH.

Thanks,
Wei Chen

> Jan

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.