[PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[Miles Chen]

To build this driver as a kernel module, we cannot use
the unexported symbol "max_pfn" to setup enable_4GB.

Use totalram_pages() instead to setup enable_4GB.

Suggested-by: Mike Rapoport <rppt@linux.ibm.com>
Signed-off-by: Miles Chen <miles.chen@mediatek.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Yong Wu <yong.wu@mediatek.com>
Cc: Chao Hao <chao.hao@mediatek.com>
---
 drivers/iommu/mtk_iommu.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
index 5f4d6df59cf6..c2798a6e0e38 100644
--- a/drivers/iommu/mtk_iommu.c
+++ b/drivers/iommu/mtk_iommu.c
@@ -3,7 +3,6 @@
  * Copyright (c) 2015-2016 MediaTek Inc.
  * Author: Yong Wu <yong.wu@mediatek.com>
  */
-#include <linux/memblock.h>
 #include <linux/bug.h>
 #include <linux/clk.h>
 #include <linux/component.h>
@@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
 		return -ENOMEM;
 	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
 
-	/* Whether the current dram is over 4GB */
-	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
+	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
+	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
 	if (!data->plat_data->has_4gb_mode)
 		data->enable_4GB = false;
 
-- 
2.18.0

Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[David Hildenbrand]

On 04.06.20 10:01, Miles Chen wrote:
> To build this driver as a kernel module, we cannot use
> the unexported symbol "max_pfn" to setup enable_4GB.
> 
> Use totalram_pages() instead to setup enable_4GB.
> 
> Suggested-by: Mike Rapoport <rppt@linux.ibm.com>
> Signed-off-by: Miles Chen <miles.chen@mediatek.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Yong Wu <yong.wu@mediatek.com>
> Cc: Chao Hao <chao.hao@mediatek.com>
> ---
>  drivers/iommu/mtk_iommu.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> index 5f4d6df59cf6..c2798a6e0e38 100644
> --- a/drivers/iommu/mtk_iommu.c
> +++ b/drivers/iommu/mtk_iommu.c
> @@ -3,7 +3,6 @@
>   * Copyright (c) 2015-2016 MediaTek Inc.
>   * Author: Yong Wu <yong.wu@mediatek.com>
>   */
> -#include <linux/memblock.h>
>  #include <linux/bug.h>
>  #include <linux/clk.h>
>  #include <linux/component.h>
> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
>  		return -ENOMEM;
>  	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
>  
> -	/* Whether the current dram is over 4GB */
> -	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> +	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
> +	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));

A similar thing seems to be done by
drivers/media/platform/mtk-vpu/mtk_vpu.c:
	vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));

I do wonder if some weird memory hotplug setups might give you false
negatives.

E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
return something < 2GB.

Same can happen when you have a VM and use ballooning to fake-unplug
memory, making totalram_pages() return something < 4GB, but leaving
usable pfns >= 4GB.

but
... I don't know if I understood what "enable_4GB" needs/implies
... I don't know if this is applicable to VMs at all (on real HW such
    memory hotplug setups should not exist)
... I don't know how this code would react to memory hotplug, so if the
    condition changes after the driver loaded and enable_4GB would
    suddenly apply. Again, most probably not relevant on real HW, only
    for VMs.

-- 
Thanks,

David / dhildenb

Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[Miles Chen]

On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote:
> On 04.06.20 10:01, Miles Chen wrote:
> > To build this driver as a kernel module, we cannot use
> > the unexported symbol "max_pfn" to setup enable_4GB.
> > 
> > Use totalram_pages() instead to setup enable_4GB.
> > 
> > Suggested-by: Mike Rapoport <rppt@linux.ibm.com>
> > Signed-off-by: Miles Chen <miles.chen@mediatek.com>
> > Cc: David Hildenbrand <david@redhat.com>
> > Cc: Yong Wu <yong.wu@mediatek.com>
> > Cc: Chao Hao <chao.hao@mediatek.com>
> > ---
> >  drivers/iommu/mtk_iommu.c | 5 ++---
> >  1 file changed, 2 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> > index 5f4d6df59cf6..c2798a6e0e38 100644
> > --- a/drivers/iommu/mtk_iommu.c
> > +++ b/drivers/iommu/mtk_iommu.c
> > @@ -3,7 +3,6 @@
> >   * Copyright (c) 2015-2016 MediaTek Inc.
> >   * Author: Yong Wu <yong.wu@mediatek.com>
> >   */
> > -#include <linux/memblock.h>
> >  #include <linux/bug.h>
> >  #include <linux/clk.h>
> >  #include <linux/component.h>
> > @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
> >  		return -ENOMEM;
> >  	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
> >  
> > -	/* Whether the current dram is over 4GB */
> > -	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> > +	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
> > +	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
> 
> A similar thing seems to be done by
> drivers/media/platform/mtk-vpu/mtk_vpu.c:
> 	vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));
> 
> I do wonder if some weird memory hotplug setups might give you false
> negatives.
> 
> E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
> x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
> return something < 2GB.
> 
> Same can happen when you have a VM and use ballooning to fake-unplug
> memory, making totalram_pages() return something < 4GB, but leaving
> usable pfns >= 4GB

Yes. Yingjoe also told me that this patch is not correct.

Thanks for pointing this out. totalram_pages() does not work 
for some cases:

e.g., DRAM start @0x4000_0000 and DRAM size is 0x1_0000_0000 but we
reserve large amount of memory, which makes totalram_pages() < 3GB but
it is possible to allocate a pfn >= 4GB.

I will discuss this internally.

Miles
> .
> 
> but
> ... I don't know if I understood what "enable_4GB" needs/implies
> ... I don't know if this is applicable to VMs
>  at all (on real HW such
>     memory hotplug setups should not exist)
> ... I don't know how this code would react to memory hotplug, so if the
>     condition changes after the driver loaded and enable_4GB would
>     suddenly apply. Again, most probably not relevant on real HW, only
>     for VMs.
> 

Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[David Hildenbrand]

On 04.06.20 11:49, Miles Chen wrote:
> On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote:
>> On 04.06.20 10:01, Miles Chen wrote:
>>> To build this driver as a kernel module, we cannot use
>>> the unexported symbol "max_pfn" to setup enable_4GB.
>>>
>>> Use totalram_pages() instead to setup enable_4GB.
>>>
>>> Suggested-by: Mike Rapoport <rppt@linux.ibm.com>
>>> Signed-off-by: Miles Chen <miles.chen@mediatek.com>
>>> Cc: David Hildenbrand <david@redhat.com>
>>> Cc: Yong Wu <yong.wu@mediatek.com>
>>> Cc: Chao Hao <chao.hao@mediatek.com>
>>> ---
>>>  drivers/iommu/mtk_iommu.c | 5 ++---
>>>  1 file changed, 2 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
>>> index 5f4d6df59cf6..c2798a6e0e38 100644
>>> --- a/drivers/iommu/mtk_iommu.c
>>> +++ b/drivers/iommu/mtk_iommu.c
>>> @@ -3,7 +3,6 @@
>>>   * Copyright (c) 2015-2016 MediaTek Inc.
>>>   * Author: Yong Wu <yong.wu@mediatek.com>
>>>   */
>>> -#include <linux/memblock.h>
>>>  #include <linux/bug.h>
>>>  #include <linux/clk.h>
>>>  #include <linux/component.h>
>>> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
>>>  		return -ENOMEM;
>>>  	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
>>>  
>>> -	/* Whether the current dram is over 4GB */
>>> -	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
>>> +	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
>>> +	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
>>
>> A similar thing seems to be done by
>> drivers/media/platform/mtk-vpu/mtk_vpu.c:
>> 	vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));
>>
>> I do wonder if some weird memory hotplug setups might give you false
>> negatives.
>>
>> E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
>> x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
>> return something < 2GB.
>>
>> Same can happen when you have a VM and use ballooning to fake-unplug
>> memory, making totalram_pages() return something < 4GB, but leaving
>> usable pfns >= 4GB
> 
> Yes. Yingjoe also told me that this patch is not correct.
> 
> Thanks for pointing this out. totalram_pages() does not work 
> for some cases:
> 

Just a thought: If memory hotplug is applicable as well, you might
either want to always assume data->enable_4GB, or handle memory hotplug
events from the memory notifier, when new memory gets onlined (not sure
how tricky that is).


-- 
Thanks,

David / dhildenb

Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[Christoph Hellwig]

On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
> Just a thought: If memory hotplug is applicable as well, you might
> either want to always assume data->enable_4GB, or handle memory hotplug
> events from the memory notifier, when new memory gets onlined (not sure
> how tricky that is).

We probably want a highest_pfn_possible() or similar API instead of
having drivers poking into random VM internals.

Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[David Hildenbrand]

On 04.06.20 17:06, Christoph Hellwig wrote:
> On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
>> Just a thought: If memory hotplug is applicable as well, you might
>> either want to always assume data->enable_4GB, or handle memory hotplug
>> events from the memory notifier, when new memory gets onlined (not sure
>> how tricky that is).
> 
> We probably want a highest_pfn_possible() or similar API instead of
> having drivers poking into random VM internals.

Well, memory notifiers are a reasonable api used accross the kernel to
get notified when new memory is onlined to the buddy that could be used
for allocations.

highest_pfn_possible() would have to default to something linked to
MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure
how helpful that is (IOW, you can just default to enable_4GB=true in
that case instead in most cases).

-- 
Thanks,

David / dhildenb

Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

[David Hildenbrand]

On 04.06.20 17:27, David Hildenbrand wrote:
> On 04.06.20 17:06, Christoph Hellwig wrote:
>> On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
>>> Just a thought: If memory hotplug is applicable as well, you might
>>> either want to always assume data->enable_4GB, or handle memory hotplug
>>> events from the memory notifier, when new memory gets onlined (not sure
>>> how tricky that is).
>>
>> We probably want a highest_pfn_possible() or similar API instead of
>> having drivers poking into random VM internals.
> 
> Well, memory notifiers are a reasonable api used accross the kernel to
> get notified when new memory is onlined to the buddy that could be used
> for allocations.
> 
> highest_pfn_possible() would have to default to something linked to
> MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure
> how helpful that is (IOW, you can just default to enable_4GB=true in
> that case instead in most cases).

Correction: At least on x86-64 we have max_possible_pfn, which will
consult the ACPI SRAT table to figure out the maximum possible PFN.
(Without SRAT, max_possible_pfn will point at the end of initial boot
memory and not consider hotplug memory - something that e.g., newer QEMU
versions work around by creating SRAT tables if memory hotplug might be
possible, even if there is no actual NUMA configuration).

pci-swiotlb.c similarly relies on that to figure out if there are any
!DMA addresses to handle.

-- 
Thanks,

David / dhildenb