Re: RFC: Memory Tiering Kernel Interfaces

[Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>]

On 5/11/22 12:42 PM, Alistair Popple wrote:
> 
> Wei Xu <weixugc@google.com> writes:
> 
>> On Tue, May 10, 2022 at 5:10 AM Aneesh Kumar K V
>> <aneesh.kumar@linux.ibm.com> wrote:
>>>
>>> On 5/10/22 3:29 PM, Hesham Almatary wrote:
>>>> Hello Yang,
>>>>
>>>> On 5/10/2022 4:24 AM, Yang Shi wrote:
>>>>> On Mon, May 9, 2022 at 7:32 AM Hesham Almatary
>>>>> <hesham.almatary@huawei.com> wrote:
>>>
>>>
>>> ...
>>>
>>>>>>
>>>>>> node 0 has a CPU and DDR memory in tier 0, node 1 has GPU and DDR memory
>>>>>> in tier 0,
>>>>>> node 2 has NVMM memory in tier 1, node 3 has some sort of bigger memory
>>>>>> (could be a bigger DDR or something) in tier 2. The distances are as
>>>>>> follows:
>>>>>>
>>>>>> --------------          --------------
>>>>>> |   Node 0   |          |   Node 1   |
>>>>>> |  -------   |          |  -------   |
>>>>>> | |  DDR  |  |          | |  DDR  |  |
>>>>>> |  -------   |          |  -------   |
>>>>>> |            |          |            |
>>>>>> --------------          --------------
>>>>>>           | 20               | 120    |
>>>>>>           v                  v        |
>>>>>> ----------------------------       |
>>>>>> | Node 2     PMEM          |       | 100
>>>>>> ----------------------------       |
>>>>>>           | 100                       |
>>>>>>           v                           v
>>>>>> --------------------------------------
>>>>>> | Node 3    Large mem                |
>>>>>> --------------------------------------
>>>>>>
>>>>>> node distances:
>>>>>> node   0    1    2    3
>>>>>>       0  10   20   20  120
>>>>>>       1  20   10  120  100
>>>>>>       2  20  120   10  100
>>>>>>       3  120 100  100   10
>>>>>>
>>>>>> /sys/devices/system/node/memory_tiers
>>>>>> 0-1
>>>>>> 2
>>>>>> 3
>>>>>>
>>>>>> N_TOPTIER_MEMORY: 0-1
>>>>>>
>>>>>>
>>>>>> In this case, we want to be able to "skip" the demotion path from Node 1
>>>>>> to Node 2,
>>>>>>
>>>>>> and make demotion go directely to Node 3 as it is closer, distance wise.
>>>>>> How can
>>>>>>
>>>>>> we accommodate this scenario (or at least not rule it out as future
>>>>>> work) with the
>>>>>>
>>>>>> current RFC?
>>>>> If I remember correctly NUMA distance is hardcoded in SLIT by the
>>>>> firmware, it is supposed to reflect the latency. So I suppose it is
>>>>> the firmware's responsibility to have correct information. And the RFC
>>>>> assumes higher tier memory has better performance than lower tier
>>>>> memory (latency, bandwidth, throughput, etc), so it sounds like a
>>>>> buggy firmware to have lower tier memory with shorter distance than
>>>>> higher tier memory IMHO.
>>>>
>>>> You are correct if you're assuming the topology is all hierarchically
>>>>
>>>> symmetric, but unfortuantely, in real hardware (e.g., my example above)
>>>>
>>>> it is not. The distance/latency between two nodes in the same tier
>>>>
>>>> and a third node, is different. The firmware still provides the correct
>>>>
>>>> latency, but putting a node in a tier is up to the kernel/user, and
>>>>
>>>> is relative: e.g., Node 3 could belong to tier 1 from Node 1's
>>>>
>>>> perspective, but to tier 2 from Node 0's.
>>>>
>>>>
>>>> A more detailed example (building on my previous one) is when having
>>>>
>>>> the GPU connected to a switch:
>>>>
>>>> ----------------------------
>>>> | Node 2     PMEM          |
>>>> ----------------------------
>>>>         ^
>>>>         |
>>>> --------------          --------------
>>>> |   Node 0   |          |   Node 1   |
>>>> |  -------   |          |  -------   |
>>>> | |  DDR  |  |          | |  DDR  |  |
>>>> |  -------   |          |  -------   |
>>>> |    CPU     |          |    GPU     |
>>>> --------------          --------------
>>>>          |                  |
>>>>          v                  v
>>>> ----------------------------
>>>> |         Switch           |
>>>> ----------------------------
>>>>          |
>>>>          v
>>>> --------------------------------------
>>>> | Node 3    Large mem                |
>>>> --------------------------------------
>>>>
>>>> Here, demoting from Node 1 to Node 3 directly would be faster as
>>>>
>>>> it only has to go through one hub, compared to demoting from Node 1
>>>>
>>>> to Node 2, where it goes through two hubs. I hope that example
>>>>
>>>> clarifies things a little bit.
>>>>
>>>
>>> Alistair mentioned that we want to consider GPU memory to be expensive
>>> and want to demote from GPU to regular DRAM. In that case for the above
>>> case we should end up with
>>>
>>>
>>> tier 0 - > Node3
>>> tier 1 ->  Node0, Node1
>>> tier 2 ->  Node2
> 
> I'm a little bit confused by the tiering here as I don't think it's
> quite what we want. As pointed out GPU memory is expensive and therefore
> we don't want anything demoting to it. That implies it should be in the
> top tier:
> 


I didn't look closely at the topology and assumed that Node3 is the GPU 
connected to the switch. Hence all the confusion.


> tier 0 -> Node1
> tier 1 -> Node0, Node3
> tier 2 -> Node2
> 
> Hence:
> 
> node 0: allowed=2
> node 1: allowed=0,3,2
> node 2: allowed=empty
> node 3: allowed=2

looks good to be default and simple.

> 
> Alternatively Node3 could be put in tier 2 which would prevent demotion
> to PMEM via the switch/CPU:
> 
> tier 0 -> Node1
> tier 1 -> Node0
> tier 2 -> Node2, Node3
> 
> node 0: allowed=2,3
> node 1: allowed=0,3,2
> node 2: allowed=empty
> node 3: allowed=empty
> 

and this can be configured via userspace?

> Both of these would be an improvement over the current situation
> upstream, which demotes everything to GPU memory and doesn't support
> demoting from the GPU (meaning reclaim on GPU memory pages everything to
> disk).
> 
>>>
>>> Hence
>>>
>>>    node 0: allowed=2
>>>    node 1: allowed=2
>>>    node 2: allowed = empty
>>>    node 3: allowed = 0-1 , based on fallback order 1, 0
>>
>> If we have 3 tiers as defined above, then we'd better to have:
>>
>> node 0: allowed = 2
>> node 1: allowed = 2
>> node 2: allowed = empty
>> node 3: allowed = 0-2, based on fallback order: 1,0,2
>>
>> The firmware should provide the node distance values to reflect that
>> PMEM is slowest and should have the largest distance away from node 3.
> 
> Right. In my above example firmware would have to provide reasonable
> distance values to ensure optimal fallback order.
> 
>>> -aneesh
>>>
>>>