[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

On boards with more than 2GB of RAM booting goes wrong with things not working
and we're getting lots of l3 warnings:

WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
...
[<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
[<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
[<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
[<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
[<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
[<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
[<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)

Let's fix the issue by enabling ZONE_DMA for LPAE.

Signed-off-by: Tony Lindgren <tony@atomide.com>
---
 arch/arm/mach-omap2/Kconfig | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm/mach-omap2/Kconfig b/arch/arm/mach-omap2/Kconfig
index b3a0dff..33d1460 100644
--- a/arch/arm/mach-omap2/Kconfig
+++ b/arch/arm/mach-omap2/Kconfig
@@ -49,6 +49,7 @@ config SOC_OMAP5
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config SOC_AM33XX
 	bool "TI AM33XX"
@@ -78,6 +79,7 @@ config SOC_DRA7XX
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config ARCH_OMAP2PLUS
 	bool
-- 
2.1.4

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

On boards with more than 2GB of RAM booting goes wrong with things not working
and we're getting lots of l3 warnings:

WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
...
[<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
[<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
[<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
[<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
[<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
[<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
[<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)

Let's fix the issue by enabling ZONE_DMA for LPAE.

Signed-off-by: Tony Lindgren <tony@atomide.com>
---
 arch/arm/mach-omap2/Kconfig | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm/mach-omap2/Kconfig b/arch/arm/mach-omap2/Kconfig
index b3a0dff..33d1460 100644
--- a/arch/arm/mach-omap2/Kconfig
+++ b/arch/arm/mach-omap2/Kconfig
@@ -49,6 +49,7 @@ config SOC_OMAP5
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config SOC_AM33XX
 	bool "TI AM33XX"
@@ -78,6 +79,7 @@ config SOC_DRA7XX
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config ARCH_OMAP2PLUS
 	bool
-- 
2.1.4

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

Hi Tony,

On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> On boards with more than 2GB of RAM booting goes wrong with things not working
> and we're getting lots of l3 warnings:
> 
> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> ...
> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> 
> Let's fix the issue by enabling ZONE_DMA for LPAE.

May I know on which platform you have reproduced this?
Just wondering what other changes you made for booting a OMAP5+ based
board with more than 2GB.

Thanks and regards,
Lokesh
> 
> Signed-off-by: Tony Lindgren <tony@atomide.com>
> ---
>  arch/arm/mach-omap2/Kconfig | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/arm/mach-omap2/Kconfig b/arch/arm/mach-omap2/Kconfig
> index b3a0dff..33d1460 100644
> --- a/arch/arm/mach-omap2/Kconfig
> +++ b/arch/arm/mach-omap2/Kconfig
> @@ -49,6 +49,7 @@ config SOC_OMAP5
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config SOC_AM33XX
>  	bool "TI AM33XX"
> @@ -78,6 +79,7 @@ config SOC_DRA7XX
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config ARCH_OMAP2PLUS
>  	bool
> 

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

Hi Tony,

On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> On boards with more than 2GB of RAM booting goes wrong with things not working
> and we're getting lots of l3 warnings:
> 
> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> ...
> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> 
> Let's fix the issue by enabling ZONE_DMA for LPAE.

May I know on which platform you have reproduced this?
Just wondering what other changes you made for booting a OMAP5+ based
board with more than 2GB.

Thanks and regards,
Lokesh
> 
> Signed-off-by: Tony Lindgren <tony@atomide.com>
> ---
>  arch/arm/mach-omap2/Kconfig | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/arm/mach-omap2/Kconfig b/arch/arm/mach-omap2/Kconfig
> index b3a0dff..33d1460 100644
> --- a/arch/arm/mach-omap2/Kconfig
> +++ b/arch/arm/mach-omap2/Kconfig
> @@ -49,6 +49,7 @@ config SOC_OMAP5
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config SOC_AM33XX
>  	bool "TI AM33XX"
> @@ -78,6 +79,7 @@ config SOC_DRA7XX
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config ARCH_OMAP2PLUS
>  	bool
> 

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
> Hi Tony,
> 
> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> > On boards with more than 2GB of RAM booting goes wrong with things not working
> > and we're getting lots of l3 warnings:
> > 
> > WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> > 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> > ...
> > [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> > [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> > [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> > [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> > [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> > [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> > [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> > 
> > Let's fix the issue by enabling ZONE_DMA for LPAE.
> 
> May I know on which platform you have reproduced this?

This is on the 4GB version of isee igepv5 GEP0050-RB10

https://isee.biz/products/igep-processor-boards/igepv5-omap5432

> Just wondering what other changes you made for booting a OMAP5+ based
> board with more than 2GB.

Just the minimal dts changes I posted yesterday to use shared dtsi file
for omap5-uevm variants. Then boot with something like this in the kernel
cmdline:

mem=2032M@0x80000000 mem=2048M@0x300000000

Regards,

Tony

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
> Hi Tony,
> 
> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> > On boards with more than 2GB of RAM booting goes wrong with things not working
> > and we're getting lots of l3 warnings:
> > 
> > WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> > 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> > ...
> > [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> > [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> > [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> > [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> > [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> > [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> > [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> > 
> > Let's fix the issue by enabling ZONE_DMA for LPAE.
> 
> May I know on which platform you have reproduced this?

This is on the 4GB version of isee igepv5 GEP0050-RB10

https://isee.biz/products/igep-processor-boards/igepv5-omap5432

> Just wondering what other changes you made for booting a OMAP5+ based
> board with more than 2GB.

Just the minimal dts changes I posted yesterday to use shared dtsi file
for omap5-uevm variants. Then boot with something like this in the kernel
cmdline:

mem=2032M at 0x80000000 mem=2048M at 0x300000000

Regards,

Tony

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

Hi Tony,

On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
> * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
>> Hi Tony,
>>
>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
>>> On boards with more than 2GB of RAM booting goes wrong with things not working
>>> and we're getting lots of l3 warnings:
>>>
>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
>>> ...
>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
>>>
>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
>>
>> May I know on which platform you have reproduced this?
> 
> This is on the 4GB version of isee igepv5 GEP0050-RB10
> 
> https://isee.biz/products/igep-processor-boards/igepv5-omap5432
> 
>> Just wondering what other changes you made for booting a OMAP5+ based
>> board with more than 2GB.
> 
> Just the minimal dts changes I posted yesterday to use shared dtsi file
> for omap5-uevm variants. Then boot with something like this in the kernel
> cmdline:
> 
> mem=2032M@0x80000000 mem=2048M@0x300000000
Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
0x2_0000_0000.(or am I missing something?)

Any chance you tried doing memtest for all this memory?

IIUC, It is not allowed to use the mix of L3 address mapping and MA
physical address mapping. TRM specifies that
"It is expected that the OS uses either the lower 2-GiB space and the
lower aliased address of emif(a) and (b), or the continuous 8-GiB space
and the upper aliasing of the emif(a) and (b) for all EMIF accesses."

I tried a similar 4GB setup and observed few stuff:
- Need to have ARM_PV_FIXUP enabled something similar that is done in
Keystone inorder to shift to higher address space.[1] [2]
- All the peripherals other than MPU, can access only 2GB. So we need to
restrict the DMA and need an addresses translation. Along with this
patch [3] [4] are needed.

[1]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
[2]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
[3]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
[4]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8

Please correct me if I am wrong.

Thanks and regards,
Lokesh

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

Hi Tony,

On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
> * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
>> Hi Tony,
>>
>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
>>> On boards with more than 2GB of RAM booting goes wrong with things not working
>>> and we're getting lots of l3 warnings:
>>>
>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
>>> ...
>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
>>>
>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
>>
>> May I know on which platform you have reproduced this?
> 
> This is on the 4GB version of isee igepv5 GEP0050-RB10
> 
> https://isee.biz/products/igep-processor-boards/igepv5-omap5432
> 
>> Just wondering what other changes you made for booting a OMAP5+ based
>> board with more than 2GB.
> 
> Just the minimal dts changes I posted yesterday to use shared dtsi file
> for omap5-uevm variants. Then boot with something like this in the kernel
> cmdline:
> 
> mem=2032M at 0x80000000 mem=2048M at 0x300000000
Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
0x2_0000_0000.(or am I missing something?)

Any chance you tried doing memtest for all this memory?

IIUC, It is not allowed to use the mix of L3 address mapping and MA
physical address mapping. TRM specifies that
"It is expected that the OS uses either the lower 2-GiB space and the
lower aliased address of emif(a) and (b), or the continuous 8-GiB space
and the upper aliasing of the emif(a) and (b) for all EMIF accesses."

I tried a similar 4GB setup and observed few stuff:
- Need to have ARM_PV_FIXUP enabled something similar that is done in
Keystone inorder to shift to higher address space.[1] [2]
- All the peripherals other than MPU, can access only 2GB. So we need to
restrict the DMA and need an addresses translation. Along with this
patch [3] [4] are needed.

[1]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
[2]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
[3]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
[4]
https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8

Please correct me if I am wrong.

Thanks and regards,
Lokesh

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151015 01:02]:
> Hi Tony,
> 
> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
> > * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
> >> Hi Tony,
> >>
> >> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> >>> On boards with more than 2GB of RAM booting goes wrong with things not working
> >>> and we're getting lots of l3 warnings:
> >>>
> >>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> >>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> >>> ...
> >>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> >>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> >>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> >>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> >>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> >>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> >>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> >>>
> >>> Let's fix the issue by enabling ZONE_DMA for LPAE.
> >>
> >> May I know on which platform you have reproduced this?
> > 
> > This is on the 4GB version of isee igepv5 GEP0050-RB10
> > 
> > https://isee.biz/products/igep-processor-boards/igepv5-omap5432
> > 
> >> Just wondering what other changes you made for booting a OMAP5+ based
> >> board with more than 2GB.
> > 
> > Just the minimal dts changes I posted yesterday to use shared dtsi file
> > for omap5-uevm variants. Then boot with something like this in the kernel
> > cmdline:
> > 
> > mem=2032M@0x80000000 mem=2048M@0x300000000
> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
> 0x2_0000_0000.(or am I missing something?)
> 
> Any chance you tried doing memtest for all this memory?

Seems to work just fine, it's been up since I posted $subject patch., I tried
a kernel compile on it already earlier and just started memtester 3900M which
seems to only capable of testing 2924MB though.

> IIUC, It is not allowed to use the mix of L3 address mapping and MA
> physical address mapping. TRM specifies that
> "It is expected that the OS uses either the lower 2-GiB space and the
> lower aliased address of emif(a) and (b), or the continuous 8-GiB space
> and the upper aliasing of the emif(a) and (b) for all EMIF accesses."

I think that depends on the chip selects that can be used. The above
is probably for the case where all 8GB was to be used.

> I tried a similar 4GB setup and observed few stuff:
> - Need to have ARM_PV_FIXUP enabled something similar that is done in
> Keystone inorder to shift to higher address space.[1] [2]

Seems that's optional for 4GB.. But probably a must for 8GB :)

> - All the peripherals other than MPU, can access only 2GB. So we need to
> restrict the DMA and need an addresses translation. Along with this
> patch [3] [4] are needed.

Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll
post an updated patch with the 2GB limit for dma_zone_size after some
more tests.

For the dma-ranges, looks like I have not hit that one yet. Which drivers
did you run into problems without that?

Regards,

Tony

> [1]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
> [2]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
> [3]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
> [4]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151015 01:02]:
> Hi Tony,
> 
> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
> > * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
> >> Hi Tony,
> >>
> >> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> >>> On boards with more than 2GB of RAM booting goes wrong with things not working
> >>> and we're getting lots of l3 warnings:
> >>>
> >>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> >>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> >>> ...
> >>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> >>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> >>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> >>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> >>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> >>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> >>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> >>>
> >>> Let's fix the issue by enabling ZONE_DMA for LPAE.
> >>
> >> May I know on which platform you have reproduced this?
> > 
> > This is on the 4GB version of isee igepv5 GEP0050-RB10
> > 
> > https://isee.biz/products/igep-processor-boards/igepv5-omap5432
> > 
> >> Just wondering what other changes you made for booting a OMAP5+ based
> >> board with more than 2GB.
> > 
> > Just the minimal dts changes I posted yesterday to use shared dtsi file
> > for omap5-uevm variants. Then boot with something like this in the kernel
> > cmdline:
> > 
> > mem=2032M at 0x80000000 mem=2048M at 0x300000000
> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
> 0x2_0000_0000.(or am I missing something?)
> 
> Any chance you tried doing memtest for all this memory?

Seems to work just fine, it's been up since I posted $subject patch., I tried
a kernel compile on it already earlier and just started memtester 3900M which
seems to only capable of testing 2924MB though.

> IIUC, It is not allowed to use the mix of L3 address mapping and MA
> physical address mapping. TRM specifies that
> "It is expected that the OS uses either the lower 2-GiB space and the
> lower aliased address of emif(a) and (b), or the continuous 8-GiB space
> and the upper aliasing of the emif(a) and (b) for all EMIF accesses."

I think that depends on the chip selects that can be used. The above
is probably for the case where all 8GB was to be used.

> I tried a similar 4GB setup and observed few stuff:
> - Need to have ARM_PV_FIXUP enabled something similar that is done in
> Keystone inorder to shift to higher address space.[1] [2]

Seems that's optional for 4GB.. But probably a must for 8GB :)

> - All the peripherals other than MPU, can access only 2GB. So we need to
> restrict the DMA and need an addresses translation. Along with this
> patch [3] [4] are needed.

Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll
post an updated patch with the 2GB limit for dma_zone_size after some
more tests.

For the dma-ranges, looks like I have not hit that one yet. Which drivers
did you run into problems without that?

Regards,

Tony

> [1]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
> [2]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
> [3]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
> [4]
> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

+Richard

On Thursday 15 October 2015 07:35 PM, Tony Lindgren wrote:
> * Lokesh Vutla <a0131933@ti.com> [151015 01:02]:
>> Hi Tony,
>>
>> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
>>> * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
>>>> Hi Tony,
>>>>
>>>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
>>>>> On boards with more than 2GB of RAM booting goes wrong with things not working
>>>>> and we're getting lots of l3 warnings:
>>>>>
>>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
>>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
>>>>> ...
>>>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
>>>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
>>>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
>>>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
>>>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
>>>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
>>>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
>>>>>
>>>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
>>>>
>>>> May I know on which platform you have reproduced this?
>>>
>>> This is on the 4GB version of isee igepv5 GEP0050-RB10
>>>
>>> https://isee.biz/products/igep-processor-boards/igepv5-omap5432
>>>
>>>> Just wondering what other changes you made for booting a OMAP5+ based
>>>> board with more than 2GB.
>>>
>>> Just the minimal dts changes I posted yesterday to use shared dtsi file
>>> for omap5-uevm variants. Then boot with something like this in the kernel
>>> cmdline:
>>>
>>> mem=2032M@0x80000000 mem=2048M@0x300000000
>> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
>> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
>> 0x2_0000_0000.(or am I missing something?)
>>
>> Any chance you tried doing memtest for all this memory?
> 
> Seems to work just fine, it's been up since I posted $subject patch., I tried
> a kernel compile on it already earlier and just started memtester 3900M which
> seems to only capable of testing 2924MB though.
> 
>> IIUC, It is not allowed to use the mix of L3 address mapping and MA
>> physical address mapping. TRM specifies that
>> "It is expected that the OS uses either the lower 2-GiB space and the
>> lower aliased address of emif(a) and (b), or the continuous 8-GiB space
>> and the upper aliasing of the emif(a) and (b) for all EMIF accesses."
> 
> I think that depends on the chip selects that can be used. The above
> is probably for the case where all 8GB was to be used.

Richard, Do you agree with this?

> 
>> I tried a similar 4GB setup and observed few stuff:
>> - Need to have ARM_PV_FIXUP enabled something similar that is done in
>> Keystone inorder to shift to higher address space.[1] [2]
> 
> Seems that's optional for 4GB.. But probably a must for 8GB :)
> 
>> - All the peripherals other than MPU, can access only 2GB. So we need to
>> restrict the DMA and need an addresses translation. Along with this
>> patch [3] [4] are needed.
> 
> Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll
> post an updated patch with the 2GB limit for dma_zone_size after some
> more tests.
> 
> For the dma-ranges, looks like I have not hit that one yet. Which drivers
> did you run into problems without that?

This is needed only when you completely shift to higher address space.

> 
> Regards,
> 
> Tony
> 
>> [1]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
>> [2]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
>> [3]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
>> [4]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8
> 

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

+Richard

On Thursday 15 October 2015 07:35 PM, Tony Lindgren wrote:
> * Lokesh Vutla <a0131933@ti.com> [151015 01:02]:
>> Hi Tony,
>>
>> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
>>> * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
>>>> Hi Tony,
>>>>
>>>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
>>>>> On boards with more than 2GB of RAM booting goes wrong with things not working
>>>>> and we're getting lots of l3 warnings:
>>>>>
>>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
>>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
>>>>> ...
>>>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
>>>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
>>>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
>>>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
>>>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
>>>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
>>>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
>>>>>
>>>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
>>>>
>>>> May I know on which platform you have reproduced this?
>>>
>>> This is on the 4GB version of isee igepv5 GEP0050-RB10
>>>
>>> https://isee.biz/products/igep-processor-boards/igepv5-omap5432
>>>
>>>> Just wondering what other changes you made for booting a OMAP5+ based
>>>> board with more than 2GB.
>>>
>>> Just the minimal dts changes I posted yesterday to use shared dtsi file
>>> for omap5-uevm variants. Then boot with something like this in the kernel
>>> cmdline:
>>>
>>> mem=2032M at 0x80000000 mem=2048M at 0x300000000
>> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
>> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
>> 0x2_0000_0000.(or am I missing something?)
>>
>> Any chance you tried doing memtest for all this memory?
> 
> Seems to work just fine, it's been up since I posted $subject patch., I tried
> a kernel compile on it already earlier and just started memtester 3900M which
> seems to only capable of testing 2924MB though.
> 
>> IIUC, It is not allowed to use the mix of L3 address mapping and MA
>> physical address mapping. TRM specifies that
>> "It is expected that the OS uses either the lower 2-GiB space and the
>> lower aliased address of emif(a) and (b), or the continuous 8-GiB space
>> and the upper aliasing of the emif(a) and (b) for all EMIF accesses."
> 
> I think that depends on the chip selects that can be used. The above
> is probably for the case where all 8GB was to be used.

Richard, Do you agree with this?

> 
>> I tried a similar 4GB setup and observed few stuff:
>> - Need to have ARM_PV_FIXUP enabled something similar that is done in
>> Keystone inorder to shift to higher address space.[1] [2]
> 
> Seems that's optional for 4GB.. But probably a must for 8GB :)
> 
>> - All the peripherals other than MPU, can access only 2GB. So we need to
>> restrict the DMA and need an addresses translation. Along with this
>> patch [3] [4] are needed.
> 
> Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll
> post an updated patch with the 2GB limit for dma_zone_size after some
> more tests.
> 
> For the dma-ranges, looks like I have not hit that one yet. Which drivers
> did you run into problems without that?

This is needed only when you completely shift to higher address space.

> 
> Regards,
> 
> Tony
> 
>> [1]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
>> [2]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
>> [3]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
>> [4]
>> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8
> 

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151015 08:21]:
> +Richard
> 
> On Thursday 15 October 2015 07:35 PM, Tony Lindgren wrote:
> > * Lokesh Vutla <a0131933@ti.com> [151015 01:02]:
> >> Hi Tony,
> >>
> >> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
> >>> * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
> >>>> Hi Tony,
> >>>>
> >>>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> >>>>> On boards with more than 2GB of RAM booting goes wrong with things not working
> >>>>> and we're getting lots of l3 warnings:
> >>>>>
> >>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> >>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> >>>>> ...
> >>>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> >>>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> >>>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> >>>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> >>>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> >>>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> >>>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> >>>>>
> >>>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
> >>>>
> >>>> May I know on which platform you have reproduced this?
> >>>
> >>> This is on the 4GB version of isee igepv5 GEP0050-RB10
> >>>
> >>> https://isee.biz/products/igep-processor-boards/igepv5-omap5432
> >>>
> >>>> Just wondering what other changes you made for booting a OMAP5+ based
> >>>> board with more than 2GB.
> >>>
> >>> Just the minimal dts changes I posted yesterday to use shared dtsi file
> >>> for omap5-uevm variants. Then boot with something like this in the kernel
> >>> cmdline:
> >>>
> >>> mem=2032M@0x80000000 mem=2048M@0x300000000
> >> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
> >> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
> >> 0x2_0000_0000.(or am I missing something?)
> >>
> >> Any chance you tried doing memtest for all this memory?
> > 
> > Seems to work just fine, it's been up since I posted $subject patch., I tried
> > a kernel compile on it already earlier and just started memtester 3900M which
> > seems to only capable of testing 2924MB though.
> > 
> >> IIUC, It is not allowed to use the mix of L3 address mapping and MA
> >> physical address mapping. TRM specifies that
> >> "It is expected that the OS uses either the lower 2-GiB space and the
> >> lower aliased address of emif(a) and (b), or the continuous 8-GiB space
> >> and the upper aliasing of the emif(a) and (b) for all EMIF accesses."
> > 
> > I think that depends on the chip selects that can be used. The above
> > is probably for the case where all 8GB was to be used.
> 
> Richard, Do you agree with this?
> 
> > 
> >> I tried a similar 4GB setup and observed few stuff:
> >> - Need to have ARM_PV_FIXUP enabled something similar that is done in
> >> Keystone inorder to shift to higher address space.[1] [2]
> > 
> > Seems that's optional for 4GB.. But probably a must for 8GB :)
> > 
> >> - All the peripherals other than MPU, can access only 2GB. So we need to
> >> restrict the DMA and need an addresses translation. Along with this
> >> patch [3] [4] are needed.
> > 
> > Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll
> > post an updated patch with the 2GB limit for dma_zone_size after some
> > more tests.
> > 
> > For the dma-ranges, looks like I have not hit that one yet. Which drivers
> > did you run into problems without that?
> 
> This is needed only when you completely shift to higher address space.

OK. Below is an updated version of the $subject patch.

Regards,

Tony

> >> [1]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
> >> [2]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
> >> [3]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
> >> [4]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8
> > 

8< ----------------------From c37732f6cea23d15faaaaa357cf698833c03ce05 Mon Sep 17 00:00:00 2001
Date: Fri, 16 Oct 2015 12:16:21 -0700
Subject: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

On boards with more than 2GB of RAM booting goes wrong with things not
working and we're getting lots of l3 warnings:

WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147
l3_interrupt_handler+0x260/0x384()
44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle):
Data Access in User mode during Functional access
...
[<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>]
(ata_scsi_add_hosts+0x5c/0x18c)
[<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>]
(ata_host_register+0x150/0x2cc)
[<c046b13c>] (ata_host_register) from [<c046b38c>]
(ata_host_activate+0xd4/0x124)
[<c046b38c>] (ata_host_activate) from [<c047f42c>]
(ahci_host_activate+0x5c/0x194)
[<c047f42c>] (ahci_host_activate) from [<c0480854>]
(ahci_platform_init_host+0x1f0/0x3f0)
[<c0480854>] (ahci_platform_init_host) from [<c047c9dc>]
(ahci_probe+0x70/0x98)
[<c047c9dc>] (ahci_probe) from [<c04220cc>]
(platform_drv_probe+0x54/0xb4)

Let's fix the issue by enabling ZONE_DMA for LPAE. Note that we need to
limit dma_zone_size to 2GB as the rest of the RAM is beyond the 4GB limit.

Let's also fix things for dra7 as done in similar patches in the TI tree
by Lokesh Vutla <lokeshvutla@ti.com>.

Signed-off-by: Tony Lindgren <tony@atomide.com>

--- a/arch/arm/mach-omap2/Kconfig
+++ b/arch/arm/mach-omap2/Kconfig
@@ -49,6 +49,7 @@ config SOC_OMAP5
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config SOC_AM33XX
 	bool "TI AM33XX"
@@ -78,6 +79,7 @@ config SOC_DRA7XX
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config ARCH_OMAP2PLUS
 	bool
--- a/arch/arm/mach-omap2/board-generic.c
+++ b/arch/arm/mach-omap2/board-generic.c
@@ -243,6 +243,9 @@ static const char *const omap5_boards_compat[] __initconst = {
 };
 
 DT_MACHINE_START(OMAP5_DT, "Generic OMAP5 (Flattened Device Tree)")
+#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
+	.dma_zone_size	= SZ_2G,
+#endif
 	.reserve	= omap_reserve,
 	.smp		= smp_ops(omap4_smp_ops),
 	.map_io		= omap5_map_io,
@@ -288,6 +291,9 @@ static const char *const dra74x_boards_compat[] __initconst = {
 };
 
 DT_MACHINE_START(DRA74X_DT, "Generic DRA74X (Flattened Device Tree)")
+#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
+	.dma_zone_size	= SZ_2G,
+#endif
 	.reserve	= omap_reserve,
 	.smp		= smp_ops(omap4_smp_ops),
 	.map_io		= dra7xx_map_io,
@@ -308,6 +314,9 @@ static const char *const dra72x_boards_compat[] __initconst = {
 };
 
 DT_MACHINE_START(DRA72X_DT, "Generic DRA72X (Flattened Device Tree)")
+#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
+	.dma_zone_size	= SZ_2G,
+#endif
 	.reserve	= omap_reserve,
 	.map_io		= dra7xx_map_io,
 	.init_early	= dra7xx_init_early,

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151015 08:21]:
> +Richard
> 
> On Thursday 15 October 2015 07:35 PM, Tony Lindgren wrote:
> > * Lokesh Vutla <a0131933@ti.com> [151015 01:02]:
> >> Hi Tony,
> >>
> >> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote:
> >>> * Lokesh Vutla <a0131933@ti.com> [151013 20:53]:
> >>>> Hi Tony,
> >>>>
> >>>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote:
> >>>>> On boards with more than 2GB of RAM booting goes wrong with things not working
> >>>>> and we're getting lots of l3 warnings:
> >>>>>
> >>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> >>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> >>>>> ...
> >>>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> >>>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> >>>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> >>>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> >>>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> >>>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> >>>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> >>>>>
> >>>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
> >>>>
> >>>> May I know on which platform you have reproduced this?
> >>>
> >>> This is on the 4GB version of isee igepv5 GEP0050-RB10
> >>>
> >>> https://isee.biz/products/igep-processor-boards/igepv5-omap5432
> >>>
> >>>> Just wondering what other changes you made for booting a OMAP5+ based
> >>>> board with more than 2GB.
> >>>
> >>> Just the minimal dts changes I posted yesterday to use shared dtsi file
> >>> for omap5-uevm variants. Then boot with something like this in the kernel
> >>> cmdline:
> >>>
> >>> mem=2032M at 0x80000000 mem=2048M at 0x300000000
> >> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the
> >> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from
> >> 0x2_0000_0000.(or am I missing something?)
> >>
> >> Any chance you tried doing memtest for all this memory?
> > 
> > Seems to work just fine, it's been up since I posted $subject patch., I tried
> > a kernel compile on it already earlier and just started memtester 3900M which
> > seems to only capable of testing 2924MB though.
> > 
> >> IIUC, It is not allowed to use the mix of L3 address mapping and MA
> >> physical address mapping. TRM specifies that
> >> "It is expected that the OS uses either the lower 2-GiB space and the
> >> lower aliased address of emif(a) and (b), or the continuous 8-GiB space
> >> and the upper aliasing of the emif(a) and (b) for all EMIF accesses."
> > 
> > I think that depends on the chip selects that can be used. The above
> > is probably for the case where all 8GB was to be used.
> 
> Richard, Do you agree with this?
> 
> > 
> >> I tried a similar 4GB setup and observed few stuff:
> >> - Need to have ARM_PV_FIXUP enabled something similar that is done in
> >> Keystone inorder to shift to higher address space.[1] [2]
> > 
> > Seems that's optional for 4GB.. But probably a must for 8GB :)
> > 
> >> - All the peripherals other than MPU, can access only 2GB. So we need to
> >> restrict the DMA and need an addresses translation. Along with this
> >> patch [3] [4] are needed.
> > 
> > Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll
> > post an updated patch with the 2GB limit for dma_zone_size after some
> > more tests.
> > 
> > For the dma-ranges, looks like I have not hit that one yet. Which drivers
> > did you run into problems without that?
> 
> This is needed only when you completely shift to higher address space.

OK. Below is an updated version of the $subject patch.

Regards,

Tony

> >> [1]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b
> >> [2]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac
> >> [3]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d
> >> [4]
> >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8
> > 

8< ----------------------From c37732f6cea23d15faaaaa357cf698833c03ce05 Mon Sep 17 00:00:00 2001
Date: Fri, 16 Oct 2015 12:16:21 -0700
Subject: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

On boards with more than 2GB of RAM booting goes wrong with things not
working and we're getting lots of l3 warnings:

WARNING: CPU: 0 PID: 1@drivers/bus/omap_l3_noc.c:147
l3_interrupt_handler+0x260/0x384()
44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle):
Data Access in User mode during Functional access
...
[<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>]
(ata_scsi_add_hosts+0x5c/0x18c)
[<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>]
(ata_host_register+0x150/0x2cc)
[<c046b13c>] (ata_host_register) from [<c046b38c>]
(ata_host_activate+0xd4/0x124)
[<c046b38c>] (ata_host_activate) from [<c047f42c>]
(ahci_host_activate+0x5c/0x194)
[<c047f42c>] (ahci_host_activate) from [<c0480854>]
(ahci_platform_init_host+0x1f0/0x3f0)
[<c0480854>] (ahci_platform_init_host) from [<c047c9dc>]
(ahci_probe+0x70/0x98)
[<c047c9dc>] (ahci_probe) from [<c04220cc>]
(platform_drv_probe+0x54/0xb4)

Let's fix the issue by enabling ZONE_DMA for LPAE. Note that we need to
limit dma_zone_size to 2GB as the rest of the RAM is beyond the 4GB limit.

Let's also fix things for dra7 as done in similar patches in the TI tree
by Lokesh Vutla <lokeshvutla@ti.com>.

Signed-off-by: Tony Lindgren <tony@atomide.com>

--- a/arch/arm/mach-omap2/Kconfig
+++ b/arch/arm/mach-omap2/Kconfig
@@ -49,6 +49,7 @@ config SOC_OMAP5
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config SOC_AM33XX
 	bool "TI AM33XX"
@@ -78,6 +79,7 @@ config SOC_DRA7XX
 	select OMAP_INTERCONNECT
 	select OMAP_INTERCONNECT_BARRIER
 	select PM_OPP if PM
+	select ZONE_DMA if ARM_LPAE
 
 config ARCH_OMAP2PLUS
 	bool
--- a/arch/arm/mach-omap2/board-generic.c
+++ b/arch/arm/mach-omap2/board-generic.c
@@ -243,6 +243,9 @@ static const char *const omap5_boards_compat[] __initconst = {
 };
 
 DT_MACHINE_START(OMAP5_DT, "Generic OMAP5 (Flattened Device Tree)")
+#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
+	.dma_zone_size	= SZ_2G,
+#endif
 	.reserve	= omap_reserve,
 	.smp		= smp_ops(omap4_smp_ops),
 	.map_io		= omap5_map_io,
@@ -288,6 +291,9 @@ static const char *const dra74x_boards_compat[] __initconst = {
 };
 
 DT_MACHINE_START(DRA74X_DT, "Generic DRA74X (Flattened Device Tree)")
+#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
+	.dma_zone_size	= SZ_2G,
+#endif
 	.reserve	= omap_reserve,
 	.smp		= smp_ops(omap4_smp_ops),
 	.map_io		= dra7xx_map_io,
@@ -308,6 +314,9 @@ static const char *const dra72x_boards_compat[] __initconst = {
 };
 
 DT_MACHINE_START(DRA72X_DT, "Generic DRA72X (Flattened Device Tree)")
+#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
+	.dma_zone_size	= SZ_2G,
+#endif
 	.reserve	= omap_reserve,
 	.map_io		= dra7xx_map_io,
 	.init_early	= dra7xx_init_early,

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

On Saturday 17 October 2015 12:53 AM, Tony Lindgren wrote:
> * Lokesh Vutla <a0131933@ti.com> [151015 08:21]:

[..snip..]

>>>
> 
> 8< ----------------------From c37732f6cea23d15faaaaa357cf698833c03ce05 Mon Sep 17 00:00:00 2001
> Date: Fri, 16 Oct 2015 12:16:21 -0700
> Subject: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory
> 
> On boards with more than 2GB of RAM booting goes wrong with things not
> working and we're getting lots of l3 warnings:
> 
> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147
> l3_interrupt_handler+0x260/0x384()
> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle):
> Data Access in User mode during Functional access
> ...
> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>]
> (ata_scsi_add_hosts+0x5c/0x18c)
> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>]
> (ata_host_register+0x150/0x2cc)
> [<c046b13c>] (ata_host_register) from [<c046b38c>]
> (ata_host_activate+0xd4/0x124)
> [<c046b38c>] (ata_host_activate) from [<c047f42c>]
> (ahci_host_activate+0x5c/0x194)
> [<c047f42c>] (ahci_host_activate) from [<c0480854>]
> (ahci_platform_init_host+0x1f0/0x3f0)
> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>]
> (ahci_probe+0x70/0x98)
> [<c047c9dc>] (ahci_probe) from [<c04220cc>]
> (platform_drv_probe+0x54/0xb4)
> 
> Let's fix the issue by enabling ZONE_DMA for LPAE. Note that we need to
> limit dma_zone_size to 2GB as the rest of the RAM is beyond the 4GB limit.
> 
> Let's also fix things for dra7 as done in similar patches in the TI tree
> by Lokesh Vutla <lokeshvutla@ti.com>.

Looks good to me.
Reviewed-by: Lokesh Vutla <lokeshvutla@ti.com>

Since there is a big hole with in the memory space, we might also need
to enable SPARSEMEM ?

Thanks and regards,
Lokesh

> 
> Signed-off-by: Tony Lindgren <tony@atomide.com>
> 
> --- a/arch/arm/mach-omap2/Kconfig
> +++ b/arch/arm/mach-omap2/Kconfig
> @@ -49,6 +49,7 @@ config SOC_OMAP5
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config SOC_AM33XX
>  	bool "TI AM33XX"
> @@ -78,6 +79,7 @@ config SOC_DRA7XX
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config ARCH_OMAP2PLUS
>  	bool
> --- a/arch/arm/mach-omap2/board-generic.c
> +++ b/arch/arm/mach-omap2/board-generic.c
> @@ -243,6 +243,9 @@ static const char *const omap5_boards_compat[] __initconst = {
>  };
>  
>  DT_MACHINE_START(OMAP5_DT, "Generic OMAP5 (Flattened Device Tree)")
> +#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
> +	.dma_zone_size	= SZ_2G,
> +#endif
>  	.reserve	= omap_reserve,
>  	.smp		= smp_ops(omap4_smp_ops),
>  	.map_io		= omap5_map_io,
> @@ -288,6 +291,9 @@ static const char *const dra74x_boards_compat[] __initconst = {
>  };
>  
>  DT_MACHINE_START(DRA74X_DT, "Generic DRA74X (Flattened Device Tree)")
> +#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
> +	.dma_zone_size	= SZ_2G,
> +#endif
>  	.reserve	= omap_reserve,
>  	.smp		= smp_ops(omap4_smp_ops),
>  	.map_io		= dra7xx_map_io,
> @@ -308,6 +314,9 @@ static const char *const dra72x_boards_compat[] __initconst = {
>  };
>  
>  DT_MACHINE_START(DRA72X_DT, "Generic DRA72X (Flattened Device Tree)")
> +#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
> +	.dma_zone_size	= SZ_2G,
> +#endif
>  	.reserve	= omap_reserve,
>  	.map_io		= dra7xx_map_io,
>  	.init_early	= dra7xx_init_early,
> 

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Lokesh Vutla]

On Saturday 17 October 2015 12:53 AM, Tony Lindgren wrote:
> * Lokesh Vutla <a0131933@ti.com> [151015 08:21]:

[..snip..]

>>>
> 
> 8< ----------------------From c37732f6cea23d15faaaaa357cf698833c03ce05 Mon Sep 17 00:00:00 2001
> Date: Fri, 16 Oct 2015 12:16:21 -0700
> Subject: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory
> 
> On boards with more than 2GB of RAM booting goes wrong with things not
> working and we're getting lots of l3 warnings:
> 
> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147
> l3_interrupt_handler+0x260/0x384()
> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle):
> Data Access in User mode during Functional access
> ...
> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>]
> (ata_scsi_add_hosts+0x5c/0x18c)
> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>]
> (ata_host_register+0x150/0x2cc)
> [<c046b13c>] (ata_host_register) from [<c046b38c>]
> (ata_host_activate+0xd4/0x124)
> [<c046b38c>] (ata_host_activate) from [<c047f42c>]
> (ahci_host_activate+0x5c/0x194)
> [<c047f42c>] (ahci_host_activate) from [<c0480854>]
> (ahci_platform_init_host+0x1f0/0x3f0)
> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>]
> (ahci_probe+0x70/0x98)
> [<c047c9dc>] (ahci_probe) from [<c04220cc>]
> (platform_drv_probe+0x54/0xb4)
> 
> Let's fix the issue by enabling ZONE_DMA for LPAE. Note that we need to
> limit dma_zone_size to 2GB as the rest of the RAM is beyond the 4GB limit.
> 
> Let's also fix things for dra7 as done in similar patches in the TI tree
> by Lokesh Vutla <lokeshvutla@ti.com>.

Looks good to me.
Reviewed-by: Lokesh Vutla <lokeshvutla@ti.com>

Since there is a big hole with in the memory space, we might also need
to enable SPARSEMEM ?

Thanks and regards,
Lokesh

> 
> Signed-off-by: Tony Lindgren <tony@atomide.com>
> 
> --- a/arch/arm/mach-omap2/Kconfig
> +++ b/arch/arm/mach-omap2/Kconfig
> @@ -49,6 +49,7 @@ config SOC_OMAP5
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config SOC_AM33XX
>  	bool "TI AM33XX"
> @@ -78,6 +79,7 @@ config SOC_DRA7XX
>  	select OMAP_INTERCONNECT
>  	select OMAP_INTERCONNECT_BARRIER
>  	select PM_OPP if PM
> +	select ZONE_DMA if ARM_LPAE
>  
>  config ARCH_OMAP2PLUS
>  	bool
> --- a/arch/arm/mach-omap2/board-generic.c
> +++ b/arch/arm/mach-omap2/board-generic.c
> @@ -243,6 +243,9 @@ static const char *const omap5_boards_compat[] __initconst = {
>  };
>  
>  DT_MACHINE_START(OMAP5_DT, "Generic OMAP5 (Flattened Device Tree)")
> +#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
> +	.dma_zone_size	= SZ_2G,
> +#endif
>  	.reserve	= omap_reserve,
>  	.smp		= smp_ops(omap4_smp_ops),
>  	.map_io		= omap5_map_io,
> @@ -288,6 +291,9 @@ static const char *const dra74x_boards_compat[] __initconst = {
>  };
>  
>  DT_MACHINE_START(DRA74X_DT, "Generic DRA74X (Flattened Device Tree)")
> +#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
> +	.dma_zone_size	= SZ_2G,
> +#endif
>  	.reserve	= omap_reserve,
>  	.smp		= smp_ops(omap4_smp_ops),
>  	.map_io		= dra7xx_map_io,
> @@ -308,6 +314,9 @@ static const char *const dra72x_boards_compat[] __initconst = {
>  };
>  
>  DT_MACHINE_START(DRA72X_DT, "Generic DRA72X (Flattened Device Tree)")
> +#if defined(CONFIG_ZONE_DMA) && defined(CONFIG_ARM_LPAE)
> +	.dma_zone_size	= SZ_2G,
> +#endif
>  	.reserve	= omap_reserve,
>  	.map_io		= dra7xx_map_io,
>  	.init_early	= dra7xx_init_early,
> 

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151019 07:03]:
> 
> Since there is a big hole with in the memory space, we might also need
> to enable SPARSEMEM ?

No from what I remember that depends on section sizes and smaller
chunks of physical memory. For example, on the old Psions we had to
use DISCONTIGMEM/SPARSEMEM because of four discontiguous 8MB memory
chips.

Regards,

Tony

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Lokesh Vutla <a0131933@ti.com> [151019 07:03]:
> 
> Since there is a big hole with in the memory space, we might also need
> to enable SPARSEMEM ?

No from what I remember that depends on section sizes and smaller
chunks of physical memory. For example, on the old Psions we had to
use DISCONTIGMEM/SPARSEMEM because of four discontiguous 8MB memory
chips.

Regards,

Tony

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Arnd Bergmann]

On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> On boards with more than 2GB of RAM booting goes wrong with things not working
> and we're getting lots of l3 warnings:
> 
> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> ...
> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> 
> Let's fix the issue by enabling ZONE_DMA for LPAE.
> 
> Signed-off-by: Tony Lindgren <tony@atomide.com>
> 

I suspect this is not the correct fix, even if it works around the problem.

Am I right that the AHCI device can access the first 4GB of address space
using 32-bit DMA, and that any RAM beyond 2GB is above that limit?

Does the ZONE_DMA have the same size? If not, you get more bounce buffers
than you want, which is bad for performance/

Another problem here is that it only works with the SCSI and net subsystems
that have a hack in there to create manual bounce buffers, but other drivers
that can do DMA to high addresses will not know about this.

The right solution would be to force the use of an IOMMU, and if that not
works, add support for SWIOTLB on ARM.

	Arnd

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Arnd Bergmann]

On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> On boards with more than 2GB of RAM booting goes wrong with things not working
> and we're getting lots of l3 warnings:
> 
> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> ...
> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> 
> Let's fix the issue by enabling ZONE_DMA for LPAE.
> 
> Signed-off-by: Tony Lindgren <tony@atomide.com>
> 

I suspect this is not the correct fix, even if it works around the problem.

Am I right that the AHCI device can access the first 4GB of address space
using 32-bit DMA, and that any RAM beyond 2GB is above that limit?

Does the ZONE_DMA have the same size? If not, you get more bounce buffers
than you want, which is bad for performance/

Another problem here is that it only works with the SCSI and net subsystems
that have a hack in there to create manual bounce buffers, but other drivers
that can do DMA to high addresses will not know about this.

The right solution would be to force the use of an IOMMU, and if that not
works, add support for SWIOTLB on ARM.

	Arnd

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
> On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> > On boards with more than 2GB of RAM booting goes wrong with things not working
> > and we're getting lots of l3 warnings:
> > 
> > WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> > 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> > ...
> > [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> > [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> > [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> > [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> > [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> > [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> > [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> > 
> > Let's fix the issue by enabling ZONE_DMA for LPAE.
> > 
> > Signed-off-by: Tony Lindgren <tony@atomide.com>
> > 
> 
> I suspect this is not the correct fix, even if it works around the problem.
> 
> Am I right that the AHCI device can access the first 4GB of address space
> using 32-bit DMA, and that any RAM beyond 2GB is above that limit?

Yes the memory above 2GB is at 0x300000000. The same problem is there at least
with MMC too.

> Does the ZONE_DMA have the same size? If not, you get more bounce buffers
> than you want, which is bad for performance/

Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
case then as only 2GB of RAM can directly be accessed.

> Another problem here is that it only works with the SCSI and net subsystems
> that have a hack in there to create manual bounce buffers, but other drivers
> that can do DMA to high addresses will not know about this.

OK good to know.

> The right solution would be to force the use of an IOMMU, and if that not
> works, add support for SWIOTLB on ARM.

Suman might have some accelerators in mind we could use for DMA.

For SWIOTLB, It seems we have it for arm64 but not for arm?

Regards,

Tony

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
> On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> > On boards with more than 2GB of RAM booting goes wrong with things not working
> > and we're getting lots of l3 warnings:
> > 
> > WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> > 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> > ...
> > [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> > [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> > [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> > [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> > [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> > [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> > [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> > 
> > Let's fix the issue by enabling ZONE_DMA for LPAE.
> > 
> > Signed-off-by: Tony Lindgren <tony@atomide.com>
> > 
> 
> I suspect this is not the correct fix, even if it works around the problem.
> 
> Am I right that the AHCI device can access the first 4GB of address space
> using 32-bit DMA, and that any RAM beyond 2GB is above that limit?

Yes the memory above 2GB is at 0x300000000. The same problem is there at least
with MMC too.

> Does the ZONE_DMA have the same size? If not, you get more bounce buffers
> than you want, which is bad for performance/

Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
case then as only 2GB of RAM can directly be accessed.

> Another problem here is that it only works with the SCSI and net subsystems
> that have a hack in there to create manual bounce buffers, but other drivers
> that can do DMA to high addresses will not know about this.

OK good to know.

> The right solution would be to force the use of an IOMMU, and if that not
> works, add support for SWIOTLB on ARM.

Suman might have some accelerators in mind we could use for DMA.

For SWIOTLB, It seems we have it for arm64 but not for arm?

Regards,

Tony

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Arnd Bergmann]

On Wednesday 14 October 2015 09:17:56 Tony Lindgren wrote:
> * Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
> > On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> > > On boards with more than 2GB of RAM booting goes wrong with things not working
> > > and we're getting lots of l3 warnings:
> > > 
> > > WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> > > 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> > > ...
> > > [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> > > [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> > > [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> > > [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> > > [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> > > [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> > > [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> > > 
> > > Let's fix the issue by enabling ZONE_DMA for LPAE.
> > > 
> > > Signed-off-by: Tony Lindgren <tony@atomide.com>
> > > 
> > 
> > I suspect this is not the correct fix, even if it works around the problem.
> > 
> > Am I right that the AHCI device can access the first 4GB of address space
> > using 32-bit DMA, and that any RAM beyond 2GB is above that limit?
> 
> Yes the memory above 2GB is at 0x300000000. The same problem is there at least
> with MMC too.
> 
> > Does the ZONE_DMA have the same size? If not, you get more bounce buffers
> > than you want, which is bad for performance/
> 
> Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
> case then as only 2GB of RAM can directly be accessed.

I think you need ".dma_zone_size  = SZ_4G", but I'm not completely sure
whether this takes PHYS_OFFSET into account or not.

> > Another problem here is that it only works with the SCSI and net subsystems
> > that have a hack in there to create manual bounce buffers, but other drivers
> > that can do DMA to high addresses will not know about this.
> 
> OK good to know.
> 
> > The right solution would be to force the use of an IOMMU, and if that not
> > works, add support for SWIOTLB on ARM.
> 
> Suman might have some accelerators in mind we could use for DMA.
> 
> For SWIOTLB, It seems we have it for arm64 but not for arm?

Correct. A number of people have run into problems with this before, but
it has never been added to the kernel. At some point, I tried to help
Peter Senna Tschudin get it implemented, but I think he gave up when it
turned out that the platform he was using to test this did not need it
after all.

The reason we have it on ARM64 is that you are completely screwed there
if you have no IOMMU but use a 32-bit DMA master, while on 32-bit platforms
most of the time you don't have memory above 4GB, and if you do then most
of the time you don't have any DMA masters other than network and scsi
(including sata) on them, or you have an IOMMU for each one.

	Arnd

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Arnd Bergmann]

On Wednesday 14 October 2015 09:17:56 Tony Lindgren wrote:
> * Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
> > On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> > > On boards with more than 2GB of RAM booting goes wrong with things not working
> > > and we're getting lots of l3 warnings:
> > > 
> > > WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> > > 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> > > ...
> > > [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> > > [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> > > [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> > > [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> > > [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> > > [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> > > [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> > > 
> > > Let's fix the issue by enabling ZONE_DMA for LPAE.
> > > 
> > > Signed-off-by: Tony Lindgren <tony@atomide.com>
> > > 
> > 
> > I suspect this is not the correct fix, even if it works around the problem.
> > 
> > Am I right that the AHCI device can access the first 4GB of address space
> > using 32-bit DMA, and that any RAM beyond 2GB is above that limit?
> 
> Yes the memory above 2GB is at 0x300000000. The same problem is there at least
> with MMC too.
> 
> > Does the ZONE_DMA have the same size? If not, you get more bounce buffers
> > than you want, which is bad for performance/
> 
> Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
> case then as only 2GB of RAM can directly be accessed.

I think you need ".dma_zone_size  = SZ_4G", but I'm not completely sure
whether this takes PHYS_OFFSET into account or not.

> > Another problem here is that it only works with the SCSI and net subsystems
> > that have a hack in there to create manual bounce buffers, but other drivers
> > that can do DMA to high addresses will not know about this.
> 
> OK good to know.
> 
> > The right solution would be to force the use of an IOMMU, and if that not
> > works, add support for SWIOTLB on ARM.
> 
> Suman might have some accelerators in mind we could use for DMA.
> 
> For SWIOTLB, It seems we have it for arm64 but not for arm?

Correct. A number of people have run into problems with this before, but
it has never been added to the kernel. At some point, I tried to help
Peter Senna Tschudin get it implemented, but I think he gave up when it
turned out that the platform he was using to test this did not need it
after all.

The reason we have it on ARM64 is that you are completely screwed there
if you have no IOMMU but use a 32-bit DMA master, while on 32-bit platforms
most of the time you don't have memory above 4GB, and if you do then most
of the time you don't have any DMA masters other than network and scsi
(including sata) on them, or you have an IOMMU for each one.

	Arnd

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Suman Anna]

On 10/14/2015 03:12 PM, Arnd Bergmann wrote:
> On Wednesday 14 October 2015 09:17:56 Tony Lindgren wrote:
>> * Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
>>> On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
>>>> On boards with more than 2GB of RAM booting goes wrong with things not working
>>>> and we're getting lots of l3 warnings:
>>>>
>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
>>>> ...
>>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
>>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
>>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
>>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
>>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
>>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
>>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
>>>>
>>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
>>>>
>>>> Signed-off-by: Tony Lindgren <tony@atomide.com>
>>>>
>>>
>>> I suspect this is not the correct fix, even if it works around the problem.
>>>
>>> Am I right that the AHCI device can access the first 4GB of address space
>>> using 32-bit DMA, and that any RAM beyond 2GB is above that limit?
>>
>> Yes the memory above 2GB is at 0x300000000. The same problem is there at least
>> with MMC too.
>>
>>> Does the ZONE_DMA have the same size? If not, you get more bounce buffers
>>> than you want, which is bad for performance/
>>
>> Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
>> case then as only 2GB of RAM can directly be accessed.
> 
> I think you need ".dma_zone_size  = SZ_4G", but I'm not completely sure
> whether this takes PHYS_OFFSET into account or not.
> 
>>> Another problem here is that it only works with the SCSI and net subsystems
>>> that have a hack in there to create manual bounce buffers, but other drivers
>>> that can do DMA to high addresses will not know about this.
>>
>> OK good to know.
>>
>>> The right solution would be to force the use of an IOMMU, and if that not
>>> works, add support for SWIOTLB on ARM.
>>
>> Suman might have some accelerators in mind we could use for DMA.

Only MMUs that we have on OMAP5 are the ones for the processors on the
SoC - MPU, DSP or the IPU (Dual-Cortex M4). There is one eDMA engine
within the DSP subsystem that goes through the DSP MMU, but this is
typically used by software running on DSP.

regards
Suman

>>
>> For SWIOTLB, It seems we have it for arm64 but not for arm?
> 
> Correct. A number of people have run into problems with this before, but
> it has never been added to the kernel. At some point, I tried to help
> Peter Senna Tschudin get it implemented, but I think he gave up when it
> turned out that the platform he was using to test this did not need it
> after all.
> 
> The reason we have it on ARM64 is that you are completely screwed there
> if you have no IOMMU but use a 32-bit DMA master, while on 32-bit platforms
> most of the time you don't have memory above 4GB, and if you do then most
> of the time you don't have any DMA masters other than network and scsi
> (including sata) on them, or you have an IOMMU for each one.
> 
> 	Arnd
> 

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Suman Anna]

On 10/14/2015 03:12 PM, Arnd Bergmann wrote:
> On Wednesday 14 October 2015 09:17:56 Tony Lindgren wrote:
>> * Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
>>> On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
>>>> On boards with more than 2GB of RAM booting goes wrong with things not working
>>>> and we're getting lots of l3 warnings:
>>>>
>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
>>>> ...
>>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
>>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
>>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
>>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
>>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
>>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
>>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
>>>>
>>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
>>>>
>>>> Signed-off-by: Tony Lindgren <tony@atomide.com>
>>>>
>>>
>>> I suspect this is not the correct fix, even if it works around the problem.
>>>
>>> Am I right that the AHCI device can access the first 4GB of address space
>>> using 32-bit DMA, and that any RAM beyond 2GB is above that limit?
>>
>> Yes the memory above 2GB is at 0x300000000. The same problem is there at least
>> with MMC too.
>>
>>> Does the ZONE_DMA have the same size? If not, you get more bounce buffers
>>> than you want, which is bad for performance/
>>
>> Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
>> case then as only 2GB of RAM can directly be accessed.
> 
> I think you need ".dma_zone_size  = SZ_4G", but I'm not completely sure
> whether this takes PHYS_OFFSET into account or not.
> 
>>> Another problem here is that it only works with the SCSI and net subsystems
>>> that have a hack in there to create manual bounce buffers, but other drivers
>>> that can do DMA to high addresses will not know about this.
>>
>> OK good to know.
>>
>>> The right solution would be to force the use of an IOMMU, and if that not
>>> works, add support for SWIOTLB on ARM.
>>
>> Suman might have some accelerators in mind we could use for DMA.

Only MMUs that we have on OMAP5 are the ones for the processors on the
SoC - MPU, DSP or the IPU (Dual-Cortex M4). There is one eDMA engine
within the DSP subsystem that goes through the DSP MMU, but this is
typically used by software running on DSP.

regards
Suman

>>
>> For SWIOTLB, It seems we have it for arm64 but not for arm?
> 
> Correct. A number of people have run into problems with this before, but
> it has never been added to the kernel. At some point, I tried to help
> Peter Senna Tschudin get it implemented, but I think he gave up when it
> turned out that the platform he was using to test this did not need it
> after all.
> 
> The reason we have it on ARM64 is that you are completely screwed there
> if you have no IOMMU but use a 32-bit DMA master, while on 32-bit platforms
> most of the time you don't have memory above 4GB, and if you do then most
> of the time you don't have any DMA masters other than network and scsi
> (including sata) on them, or you have an IOMMU for each one.
> 
> 	Arnd
> 

Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Suman Anna <s-anna@ti.com> [151014 13:32]:
> On 10/14/2015 03:12 PM, Arnd Bergmann wrote:
> > On Wednesday 14 October 2015 09:17:56 Tony Lindgren wrote:
> >> * Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
> >>> On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> >>>> On boards with more than 2GB of RAM booting goes wrong with things not working
> >>>> and we're getting lots of l3 warnings:
> >>>>
> >>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> >>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> >>>> ...
> >>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> >>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> >>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> >>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> >>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> >>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> >>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> >>>>
> >>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
> >>>>
> >>>> Signed-off-by: Tony Lindgren <tony@atomide.com>
> >>>>
> >>>
> >>> I suspect this is not the correct fix, even if it works around the problem.
> >>>
> >>> Am I right that the AHCI device can access the first 4GB of address space
> >>> using 32-bit DMA, and that any RAM beyond 2GB is above that limit?
> >>
> >> Yes the memory above 2GB is at 0x300000000. The same problem is there at least
> >> with MMC too.
> >>
> >>> Does the ZONE_DMA have the same size? If not, you get more bounce buffers
> >>> than you want, which is bad for performance/
> >>
> >> Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
> >> case then as only 2GB of RAM can directly be accessed.
> > 
> > I think you need ".dma_zone_size  = SZ_4G", but I'm not completely sure
> > whether this takes PHYS_OFFSET into account or not.

OK I'll check and do some tests.

> >>> Another problem here is that it only works with the SCSI and net subsystems
> >>> that have a hack in there to create manual bounce buffers, but other drivers
> >>> that can do DMA to high addresses will not know about this.
> >>
> >> OK good to know.
> >>
> >>> The right solution would be to force the use of an IOMMU, and if that not
> >>> works, add support for SWIOTLB on ARM.
> >>
> >> Suman might have some accelerators in mind we could use for DMA.
> 
> Only MMUs that we have on OMAP5 are the ones for the processors on the
> SoC - MPU, DSP or the IPU (Dual-Cortex M4). There is one eDMA engine
> within the DSP subsystem that goes through the DSP MMU, but this is
> typically used by software running on DSP.

OK. Felipe mentioned maybe using tiler :)

> >> For SWIOTLB, It seems we have it for arm64 but not for arm?
> > 
> > Correct. A number of people have run into problems with this before, but
> > it has never been added to the kernel. At some point, I tried to help
> > Peter Senna Tschudin get it implemented, but I think he gave up when it
> > turned out that the platform he was using to test this did not need it
> > after all.
> > 
> > The reason we have it on ARM64 is that you are completely screwed there
> > if you have no IOMMU but use a 32-bit DMA master, while on 32-bit platforms
> > most of the time you don't have memory above 4GB, and if you do then most
> > of the time you don't have any DMA masters other than network and scsi
> > (including sata) on them, or you have an IOMMU for each one.

OK. Sounds like the way to go use use the ZONE_DMA for now, then use
SWIOTLB when available.

Regards,

Tony

[PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory

[Tony Lindgren]

* Suman Anna <s-anna@ti.com> [151014 13:32]:
> On 10/14/2015 03:12 PM, Arnd Bergmann wrote:
> > On Wednesday 14 October 2015 09:17:56 Tony Lindgren wrote:
> >> * Arnd Bergmann <arnd@arndb.de> [151014 02:20]:
> >>> On Tuesday 13 October 2015 16:13:20 Tony Lindgren wrote:
> >>>> On boards with more than 2GB of RAM booting goes wrong with things not working
> >>>> and we're getting lots of l3 warnings:
> >>>>
> >>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384()
> >>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access
> >>>> ...
> >>>> [<c044e158>] (scsi_add_host_with_dma) from [<c04705c8>] (ata_scsi_add_hosts+0x5c/0x18c)
> >>>> [<c04705c8>] (ata_scsi_add_hosts) from [<c046b13c>] (ata_host_register+0x150/0x2cc)
> >>>> [<c046b13c>] (ata_host_register) from [<c046b38c>] (ata_host_activate+0xd4/0x124)
> >>>> [<c046b38c>] (ata_host_activate) from [<c047f42c>] (ahci_host_activate+0x5c/0x194)
> >>>> [<c047f42c>] (ahci_host_activate) from [<c0480854>] (ahci_platform_init_host+0x1f0/0x3f0)
> >>>> [<c0480854>] (ahci_platform_init_host) from [<c047c9dc>] (ahci_probe+0x70/0x98)
> >>>> [<c047c9dc>] (ahci_probe) from [<c04220cc>] (platform_drv_probe+0x54/0xb4)
> >>>>
> >>>> Let's fix the issue by enabling ZONE_DMA for LPAE.
> >>>>
> >>>> Signed-off-by: Tony Lindgren <tony@atomide.com>
> >>>>
> >>>
> >>> I suspect this is not the correct fix, even if it works around the problem.
> >>>
> >>> Am I right that the AHCI device can access the first 4GB of address space
> >>> using 32-bit DMA, and that any RAM beyond 2GB is above that limit?
> >>
> >> Yes the memory above 2GB is at 0x300000000. The same problem is there at least
> >> with MMC too.
> >>
> >>> Does the ZONE_DMA have the same size? If not, you get more bounce buffers
> >>> than you want, which is bad for performance/
> >>
> >> Hmm good question. I guess we should set .dma_zone_size  = SZ_2G in this
> >> case then as only 2GB of RAM can directly be accessed.
> > 
> > I think you need ".dma_zone_size  = SZ_4G", but I'm not completely sure
> > whether this takes PHYS_OFFSET into account or not.

OK I'll check and do some tests.

> >>> Another problem here is that it only works with the SCSI and net subsystems
> >>> that have a hack in there to create manual bounce buffers, but other drivers
> >>> that can do DMA to high addresses will not know about this.
> >>
> >> OK good to know.
> >>
> >>> The right solution would be to force the use of an IOMMU, and if that not
> >>> works, add support for SWIOTLB on ARM.
> >>
> >> Suman might have some accelerators in mind we could use for DMA.
> 
> Only MMUs that we have on OMAP5 are the ones for the processors on the
> SoC - MPU, DSP or the IPU (Dual-Cortex M4). There is one eDMA engine
> within the DSP subsystem that goes through the DSP MMU, but this is
> typically used by software running on DSP.

OK. Felipe mentioned maybe using tiler :)

> >> For SWIOTLB, It seems we have it for arm64 but not for arm?
> > 
> > Correct. A number of people have run into problems with this before, but
> > it has never been added to the kernel. At some point, I tried to help
> > Peter Senna Tschudin get it implemented, but I think he gave up when it
> > turned out that the platform he was using to test this did not need it
> > after all.
> > 
> > The reason we have it on ARM64 is that you are completely screwed there
> > if you have no IOMMU but use a 32-bit DMA master, while on 32-bit platforms
> > most of the time you don't have memory above 4GB, and if you do then most
> > of the time you don't have any DMA masters other than network and scsi
> > (including sata) on them, or you have an IOMMU for each one.

OK. Sounds like the way to go use use the ZONE_DMA for now, then use
SWIOTLB when available.

Regards,

Tony