Re: [PATCH v4 2/2] RISC-V: Implement sparsemem

[Logan Gunthorpe <logang@deltatee.com>]

On 2019-08-11 10:01 p.m., Greentime Hu wrote:
> Hi Logan,
> 
> Logan Gunthorpe <logang@deltatee.com> 於 2019年8月10日 週六 上午3:03寫道：
>>
>>
>>
>> On 2019-08-09 11:01 a.m., Greentime Hu wrote:
>>> Hi Logan,
>>>
>>> Logan Gunthorpe <logang@deltatee.com> 於 2019年8月9日 週五 下午11:47寫道：
>>>>
>>>>
>>>>
>>>> On 2019-08-08 10:23 p.m., Greentime Hu wrote:
>>>>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>>>>> index 3f12b069af1d..208b3e14ccd8 100644
>>>>> --- a/arch/riscv/Kconfig
>>>>> +++ b/arch/riscv/Kconfig
>>>>> @@ -116,7 +116,8 @@ config PGTABLE_LEVELS
>>>>>         default 2
>>>>>
>>>>>  config HAVE_ARCH_PFN_VALID
>>>>> -       def_bool y
>>>>> +       bool
>>>>> +       default !SPARSEMEM_VMEMMAP
>>>>>
>>>>>  menu "Platform type"
>>>>>
>>>>> diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h
>>>>> index 8ddb6c7fedac..6991f7a5a4a7 100644
>>>>> --- a/arch/riscv/include/asm/page.h
>>>>> +++ b/arch/riscv/include/asm/page.h
>>>>> @@ -93,16 +93,20 @@ extern unsigned long min_low_pfn;
>>>>>  #define virt_to_pfn(vaddr)     (phys_to_pfn(__pa(vaddr)))
>>>>>  #define pfn_to_virt(pfn)       (__va(pfn_to_phys(pfn)))
>>>>>
>>>>> +#if !defined(CONFIG_SPARSEMEM_VMEMMAP)
>>>>> +#define pfn_valid(pfn) \
>>>>> +       (((pfn) >= pfn_base) && (((pfn)-pfn_base) < max_mapnr))
>>>>>  #define virt_to_page(vaddr)    (pfn_to_page(virt_to_pfn(vaddr)))
>>>>>  #define page_to_virt(page)     (pfn_to_virt(page_to_pfn(page)))
>>>>> +#else
>>>>> +#define virt_to_page(vaddr)    ((struct page *)((((u64)vaddr -
>>>>> va_pa_offset) / PAGE_SIZE) * sizeof(struct page) + VMEMMAP_START))
>>>>> +#define page_to_virt(pg)       ((void *)(((((u64)pg - VMEMMAP_START) /
>>>>> sizeof(struct page)) * PAGE_SIZE) + va_pa_offset))
>>>>> +#endif
>>>>
>>>> This doesn't make sense to me at all. It should always use pfn_to_page()
>>>> for virt_to_page() and the generic pfn_to_page()/page_to_pfn()
>>>> implementations essentially already do what you are doing in a cleaner
>>>> way. So I'd be really surprised if this does anything at all.
>>>>
>>>
>>> Thank you for point me out that. I just checked the generic
>>> implementation and I should use that one.
>>> Sorry I didn't check the generic one and just implement it again.
>>> I think the only patch we need is the first part to use generic
>>> pfn_valid(). I just tested it and yes it can boot successfully in dts
>>> with hole.
>>>
>>> It will fail in this check ((pfn)-pfn_base) < max_mapnr.
>>
>> Sounds to me like max_mapnr is not set correctly. See the code in
>> setup_bootmem(). Seems like 'mem_size' should be set to the largest
>> memory block, not just the one that contains the kernel...
>>
>>
>>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>>> index 3f12b069af1d..208b3e14ccd8 100644
>>> --- a/arch/riscv/Kconfig
>>> +++ b/arch/riscv/Kconfig
>>> @@ -116,7 +116,8 @@ config PGTABLE_LEVELS
>>>         default 2
>>>
>>>  config HAVE_ARCH_PFN_VALID
>>> -       def_bool y
>>> +       bool
>>> +       default !SPARSEMEM_VMEMMAP
>>>
>>>  menu "Platform type"
>>>
>>> diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h
>>> index 8ddb6c7fedac..80d28fa1e2eb 100644
>>> --- a/arch/riscv/include/asm/page.h
>>> +++ b/arch/riscv/include/asm/page.h
>>> @@ -100,8 +100,10 @@ extern unsigned long min_low_pfn;
>>>  #define page_to_bus(page)      (page_to_phys(page))
>>>  #define phys_to_page(paddr)    (pfn_to_page(phys_to_pfn(paddr)))
>>>
>>> +#if !defined(CONFIG_SPARSEMEM_VMEMMAP)
>>>  #define pfn_valid(pfn) \
>>>         (((pfn) >= pfn_base) && (((pfn)-pfn_base) < max_mapnr))
>>> +#endif
>>>
>>>  #define ARCH_PFN_OFFSET                (pfn_base)
>>
>>
>> This patch still makes no sense. I'm not sure why we have an arch
>> specific pfn_valid() because it's very similar to the generic one. But
>> my guess is there's a reason for it and it's not doing what it is
>> supposed when you remove it for the sparsemem case.
> 
> It will use another pfn_valid() implementation in
> include/linux/mmzone.h if CONFIG_SPARSEMEM and
> !CONFIG_HAVE_ARCH_PFN_VALID
> It will be this one.
> 
> static inline int pfn_valid(unsigned long pfn)
> {
>         if (pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS)
>                 return 0;
>         return valid_section(__nr_to_section(pfn_to_section_nr(pfn)));
> }

Ah, ok I see. "page.h" is only included in no-mmu arches. Which explains
why riscv re-implements that macro. Couple follow up questions then:

* Did you test the memory-with-hole scenario without the sparsemem
patches? It seems pfn_valid() will be wrong regardless of sparse/flat mem.

* Any chance we can just use the generic pfn_valid() function in all
cases not just sparsemem? Can you test that?

Thanks,

Logan