[P5600 && EVA memory caching question] PCI region

[P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

Hi,

I'm trying to expand PCI mapped region using EVA addressing scheme for 
'Baikal' P5600-based family of boards.

Currently, we've got only one PCIe mapped region, 0x08000000 - 
0x1BDBFFFF (~330Mb).

This window is unsufficient for some PCIe cards, so I'm trying to expand 
(or add one more) PCIe mapped region by moving 'Hi Memory DRAM block' 
from 0x20000000 to 0x40000000 (dts changed, BAIKAL_HIGHMEM_START and 
HIGHMEM_START are changed in 
arch/mips/include/asm/mach-baikal/hardware.h 
/arch/mips/include/asm/mach-baikal/spaces.h respectively) and and 
allocating this region for PCIe.

So far, PCI subsystem is initialized during the boot successfully (as 
far as I can see):

dw_pcie_init: DEV_ID_VEND_ID=0x80601d39 CLASS_CODE_REV_ID=0x1
dw_pcie_init: PCIe error code = 0x0
dw_pcie_init: PCIe link speed GEN1
device: 'pci0000:01': device_add
PM: Adding info for No Bus:pci0000:01
device: '0000:01': device_add
PM: Adding info for No Bus:0000:01
PCI host bridge to bus 0000:01
pci_bus 0000:01: root bus resource [mem 0x20000000-0x37ffffff]
pci_bus 0000:01: root bus resource [io  0x18020000-0x1bdaffff]
pci_bus 0000:01: root bus resource [bus 01-ff]
pci_bus 0000:01: scanning bus
pci 0000:01:00.0: [8086:10d3] type 00 class 0x020000
pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x98
pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0007ffff]
pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
pci 0000:01:00.0: PME# supported from D0 D3hot D3cold

pci 0000:01:00.0: PME# disabled
device: '0000:01:00.0': device_add
bus: 'pci': add device 0000:01:00.0
PM: Adding info for pci:0000:01:00.0
pci_bus 0000:01: fixups for bus
pci_bus 0000:01: bus scan returning with max=01
pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2007ffff]
pci 0000:01:00.0: BAR 6: assigned [mem 0x20080000-0x200bffff pref]
pci 0000:01:00.0: BAR 0: assigned [mem 0x200c0000-0x200dffff]
pci 0000:01:00.0: BAR 3: assigned [mem 0x200e0000-0x200e3fff]
pci 0000:01:00.0: BAR 2: assigned [io  0x18020000-0x1802001f]
pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x20c
pci 0000:01:00.0: calling baikal_t1_pcie_link_speed_fixup+0x0/0x228
pci 0000:01:00.0: Link Capability is GEN1, x1

but PCIe networking card (e1000e) driver fails to load:

bus: 'pci': driver_probe_device: matched device 0000:01:00.0 with driver 
e1000e
bus: 'pci': really_probe: probing driver e1000e with device 0000:01:00.0
devices_kset: Moving 0000:01:00.0 to end of list
PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
e1000e 0000:01:00.0: enabling bus mastering
e1000_probe:7054 mmio_start 0x200c0000 mmio_len 0x20000

e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic 
conservative mode

PCI MSI: setup hwirq:0  virq:51
PCI MSI: setup hwirq:1  virq:52
PCI MSI: setup hwirq:2  virq:53

e1000_get_phy_id_82571:444 ret_val = e1e_rphy(hw, MII_PHYSID1, &phy_id); 
ret_val = -2

It turns out, that PHY ID can't be read from the memory, mapped on 
driver initialization function by ioremap.

  __ioremap (arch/mips/mm/ioremap.c) considers all the memory above the 
low 512Mb as DRAM and maps the physical addresses 0x20000000 - 
0x3FFFFFFF to logical addresses starting from c0000000 and higher.

arch/mips/include/asm/mach-baikal/kernel-entry-init.h defines EVA memory 
map with virtual addresses range 0xc0000000 - 0xdfffffff as 'MK (kseg2)' 
i.e. "mapped kernel" with cashing segment attributes defined in SegCtl0, 
Cfg1: "cacheable, coherent, write-back, write-allocate, read misses 
request shared attribute".

I've tried to change the caching attribute (bits 16:18 in SegCtl0) to 
0x2 ("uncached, not coherent"), but

this hasn't yield any visible effect on PCIe e1000e initialization.

Could the experts suggest something?..

TIA,

Yuri

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 4939 bytes --]

Hi Yuri,

On Wed, Dec 06, 2017 at 01:36:48PM +0300, Yuri Frolov wrote:
> Hi,
> 
> I'm trying to expand PCI mapped region using EVA addressing scheme for 
> 'Baikal' P5600-based family of boards.
> 
> Currently, we've got only one PCIe mapped region, 0x08000000 - 
> 0x1BDBFFFF (~330Mb).
> 
> This window is unsufficient for some PCIe cards, so I'm trying to expand 
> (or add one more) PCIe mapped region by moving 'Hi Memory DRAM block' 
> from 0x20000000 to 0x40000000 (dts changed, BAIKAL_HIGHMEM_START and 
> HIGHMEM_START are changed in 
> arch/mips/include/asm/mach-baikal/hardware.h 
> /arch/mips/include/asm/mach-baikal/spaces.h respectively) and and 
> allocating this region for PCIe.
> 
> So far, PCI subsystem is initialized during the boot successfully (as 
> far as I can see):
> 
> dw_pcie_init: DEV_ID_VEND_ID=0x80601d39 CLASS_CODE_REV_ID=0x1
> dw_pcie_init: PCIe error code = 0x0
> dw_pcie_init: PCIe link speed GEN1
> device: 'pci0000:01': device_add
> PM: Adding info for No Bus:pci0000:01
> device: '0000:01': device_add
> PM: Adding info for No Bus:0000:01
> PCI host bridge to bus 0000:01
> pci_bus 0000:01: root bus resource [mem 0x20000000-0x37ffffff]
> pci_bus 0000:01: root bus resource [io  0x18020000-0x1bdaffff]
> pci_bus 0000:01: root bus resource [bus 01-ff]
> pci_bus 0000:01: scanning bus
> pci 0000:01:00.0: [8086:10d3] type 00 class 0x020000
> pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x98
> pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
> pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0007ffff]
> pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
> pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
> pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
> pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> 
> pci 0000:01:00.0: PME# disabled
> device: '0000:01:00.0': device_add
> bus: 'pci': add device 0000:01:00.0
> PM: Adding info for pci:0000:01:00.0
> pci_bus 0000:01: fixups for bus
> pci_bus 0000:01: bus scan returning with max=01
> pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2007ffff]
> pci 0000:01:00.0: BAR 6: assigned [mem 0x20080000-0x200bffff pref]
> pci 0000:01:00.0: BAR 0: assigned [mem 0x200c0000-0x200dffff]
> pci 0000:01:00.0: BAR 3: assigned [mem 0x200e0000-0x200e3fff]
> pci 0000:01:00.0: BAR 2: assigned [io  0x18020000-0x1802001f]
> pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x20c
> pci 0000:01:00.0: calling baikal_t1_pcie_link_speed_fixup+0x0/0x228
> pci 0000:01:00.0: Link Capability is GEN1, x1
> 
> but PCIe networking card (e1000e) driver fails to load:
> 
> bus: 'pci': driver_probe_device: matched device 0000:01:00.0 with driver 
> e1000e
> bus: 'pci': really_probe: probing driver e1000e with device 0000:01:00.0
> devices_kset: Moving 0000:01:00.0 to end of list
> PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> e1000e 0000:01:00.0: enabling bus mastering
> e1000_probe:7054 mmio_start 0x200c0000 mmio_len 0x20000
> 
> e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic 
> conservative mode
> 
> PCI MSI: setup hwirq:0  virq:51
> PCI MSI: setup hwirq:1  virq:52
> PCI MSI: setup hwirq:2  virq:53
> 
> e1000_get_phy_id_82571:444 ret_val = e1e_rphy(hw, MII_PHYSID1, &phy_id); 
> ret_val = -2
> 
> It turns out, that PHY ID can't be read from the memory, mapped on 
> driver initialization function by ioremap.
> 
>   __ioremap (arch/mips/mm/ioremap.c) considers all the memory above the 
> low 512Mb as DRAM and maps the physical addresses 0x20000000 - 
> 0x3FFFFFFF to logical addresses starting from c0000000 and higher.
> 
> arch/mips/include/asm/mach-baikal/kernel-entry-init.h defines EVA memory 
> map with virtual addresses range 0xc0000000 - 0xdfffffff as 'MK (kseg2)' 
> i.e. "mapped kernel" with cashing segment attributes defined in SegCtl0, 
> Cfg1: "cacheable, coherent, write-back, write-allocate, read misses 
> request shared attribute".
> 
> I've tried to change the caching attribute (bits 16:18 in SegCtl0) to 
> 0x2 ("uncached, not coherent"), but
> 
> this hasn't yield any visible effect on PCIe e1000e initialization.

Mapped kernel segments don't use the CCA in SegCtl. Each page mapping
has its own CCA specified in the TLB, which would have been loaded from
page tables set up by e.g. remap_area_pte(), and the ioremap() macro
does pass _CACHE_UNCACHED as the flags argument. I would recommend
checking the phys_addr and flags coming in to remap_area_pte().

If that all looks correct you could also try:
- Dump the TLB contents straight after accessing that address to check
  they are loaded correctly, i.e. call dump_tlb_all() from
  <asm/tlbdebug.h>.
- If the TLB contents look wrong, try disabling hardware page table
  walker to determine whether that is misconfigured (add nohtw to kernel
  command line).

Hope that helps

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 4939 bytes --]

Hi Yuri,

On Wed, Dec 06, 2017 at 01:36:48PM +0300, Yuri Frolov wrote:
> Hi,
> 
> I'm trying to expand PCI mapped region using EVA addressing scheme for 
> 'Baikal' P5600-based family of boards.
> 
> Currently, we've got only one PCIe mapped region, 0x08000000 - 
> 0x1BDBFFFF (~330Mb).
> 
> This window is unsufficient for some PCIe cards, so I'm trying to expand 
> (or add one more) PCIe mapped region by moving 'Hi Memory DRAM block' 
> from 0x20000000 to 0x40000000 (dts changed, BAIKAL_HIGHMEM_START and 
> HIGHMEM_START are changed in 
> arch/mips/include/asm/mach-baikal/hardware.h 
> /arch/mips/include/asm/mach-baikal/spaces.h respectively) and and 
> allocating this region for PCIe.
> 
> So far, PCI subsystem is initialized during the boot successfully (as 
> far as I can see):
> 
> dw_pcie_init: DEV_ID_VEND_ID=0x80601d39 CLASS_CODE_REV_ID=0x1
> dw_pcie_init: PCIe error code = 0x0
> dw_pcie_init: PCIe link speed GEN1
> device: 'pci0000:01': device_add
> PM: Adding info for No Bus:pci0000:01
> device: '0000:01': device_add
> PM: Adding info for No Bus:0000:01
> PCI host bridge to bus 0000:01
> pci_bus 0000:01: root bus resource [mem 0x20000000-0x37ffffff]
> pci_bus 0000:01: root bus resource [io  0x18020000-0x1bdaffff]
> pci_bus 0000:01: root bus resource [bus 01-ff]
> pci_bus 0000:01: scanning bus
> pci 0000:01:00.0: [8086:10d3] type 00 class 0x020000
> pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x98
> pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
> pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0007ffff]
> pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
> pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
> pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
> pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> 
> pci 0000:01:00.0: PME# disabled
> device: '0000:01:00.0': device_add
> bus: 'pci': add device 0000:01:00.0
> PM: Adding info for pci:0000:01:00.0
> pci_bus 0000:01: fixups for bus
> pci_bus 0000:01: bus scan returning with max=01
> pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2007ffff]
> pci 0000:01:00.0: BAR 6: assigned [mem 0x20080000-0x200bffff pref]
> pci 0000:01:00.0: BAR 0: assigned [mem 0x200c0000-0x200dffff]
> pci 0000:01:00.0: BAR 3: assigned [mem 0x200e0000-0x200e3fff]
> pci 0000:01:00.0: BAR 2: assigned [io  0x18020000-0x1802001f]
> pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x20c
> pci 0000:01:00.0: calling baikal_t1_pcie_link_speed_fixup+0x0/0x228
> pci 0000:01:00.0: Link Capability is GEN1, x1
> 
> but PCIe networking card (e1000e) driver fails to load:
> 
> bus: 'pci': driver_probe_device: matched device 0000:01:00.0 with driver 
> e1000e
> bus: 'pci': really_probe: probing driver e1000e with device 0000:01:00.0
> devices_kset: Moving 0000:01:00.0 to end of list
> PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> e1000e 0000:01:00.0: enabling bus mastering
> e1000_probe:7054 mmio_start 0x200c0000 mmio_len 0x20000
> 
> e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic 
> conservative mode
> 
> PCI MSI: setup hwirq:0  virq:51
> PCI MSI: setup hwirq:1  virq:52
> PCI MSI: setup hwirq:2  virq:53
> 
> e1000_get_phy_id_82571:444 ret_val = e1e_rphy(hw, MII_PHYSID1, &phy_id); 
> ret_val = -2
> 
> It turns out, that PHY ID can't be read from the memory, mapped on 
> driver initialization function by ioremap.
> 
>   __ioremap (arch/mips/mm/ioremap.c) considers all the memory above the 
> low 512Mb as DRAM and maps the physical addresses 0x20000000 - 
> 0x3FFFFFFF to logical addresses starting from c0000000 and higher.
> 
> arch/mips/include/asm/mach-baikal/kernel-entry-init.h defines EVA memory 
> map with virtual addresses range 0xc0000000 - 0xdfffffff as 'MK (kseg2)' 
> i.e. "mapped kernel" with cashing segment attributes defined in SegCtl0, 
> Cfg1: "cacheable, coherent, write-back, write-allocate, read misses 
> request shared attribute".
> 
> I've tried to change the caching attribute (bits 16:18 in SegCtl0) to 
> 0x2 ("uncached, not coherent"), but
> 
> this hasn't yield any visible effect on PCIe e1000e initialization.

Mapped kernel segments don't use the CCA in SegCtl. Each page mapping
has its own CCA specified in the TLB, which would have been loaded from
page tables set up by e.g. remap_area_pte(), and the ioremap() macro
does pass _CACHE_UNCACHED as the flags argument. I would recommend
checking the phys_addr and flags coming in to remap_area_pte().

If that all looks correct you could also try:
- Dump the TLB contents straight after accessing that address to check
  they are loaded correctly, i.e. call dump_tlb_all() from
  <asm/tlbdebug.h>.
- If the TLB contents look wrong, try disabling hardware page table
  walker to determine whether that is misconfigured (add nohtw to kernel
  command line).

Hope that helps

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

Hi, James.

Do I understand correctly, that in case of CONFIG_EVA=y, any logical 
address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel 
space is:
1) Unmapped (no TLB translations)
2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is, 
in fact a read from physical address 0x20000000? I mean, in legacy 
memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in 
kernel space were being translated to the physical addresses from the 
low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or 
something for EVA, the mapping is 1:1?

As for userspace addresses, the addresses from the 0x00000000 - 
0xBFFFFFFF range are:
1) Overlayed with the range which is directly accessible from the kernel 
space
2) These addresses are mapped in userland, so, read from logical address 
0x20000000 in userland may result in read from any physical address 
located in range 0x00000000 - 0xBFFFFFFF as defined by TLB for that 
particular userland thread?

TIA,
Yuri

On 12/06/2017 02:46 PM, James Hogan wrote:
> Hi Yuri,
>
> On Wed, Dec 06, 2017 at 01:36:48PM +0300, Yuri Frolov wrote:
>> Hi,
>>
>> I'm trying to expand PCI mapped region using EVA addressing scheme for
>> 'Baikal' P5600-based family of boards.
>>
>> Currently, we've got only one PCIe mapped region, 0x08000000 -
>> 0x1BDBFFFF (~330Mb).
>>
>> This window is unsufficient for some PCIe cards, so I'm trying to expand
>> (or add one more) PCIe mapped region by moving 'Hi Memory DRAM block'
>> from 0x20000000 to 0x40000000 (dts changed, BAIKAL_HIGHMEM_START and
>> HIGHMEM_START are changed in
>> arch/mips/include/asm/mach-baikal/hardware.h
>> /arch/mips/include/asm/mach-baikal/spaces.h respectively) and and
>> allocating this region for PCIe.
>>
>> So far, PCI subsystem is initialized during the boot successfully (as
>> far as I can see):
>>
>> dw_pcie_init: DEV_ID_VEND_ID=0x80601d39 CLASS_CODE_REV_ID=0x1
>> dw_pcie_init: PCIe error code = 0x0
>> dw_pcie_init: PCIe link speed GEN1
>> device: 'pci0000:01': device_add
>> PM: Adding info for No Bus:pci0000:01
>> device: '0000:01': device_add
>> PM: Adding info for No Bus:0000:01
>> PCI host bridge to bus 0000:01
>> pci_bus 0000:01: root bus resource [mem 0x20000000-0x37ffffff]
>> pci_bus 0000:01: root bus resource [io  0x18020000-0x1bdaffff]
>> pci_bus 0000:01: root bus resource [bus 01-ff]
>> pci_bus 0000:01: scanning bus
>> pci 0000:01:00.0: [8086:10d3] type 00 class 0x020000
>> pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x98
>> pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
>> pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0007ffff]
>> pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
>> pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
>> pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
>> pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
>>
>> pci 0000:01:00.0: PME# disabled
>> device: '0000:01:00.0': device_add
>> bus: 'pci': add device 0000:01:00.0
>> PM: Adding info for pci:0000:01:00.0
>> pci_bus 0000:01: fixups for bus
>> pci_bus 0000:01: bus scan returning with max=01
>> pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2007ffff]
>> pci 0000:01:00.0: BAR 6: assigned [mem 0x20080000-0x200bffff pref]
>> pci 0000:01:00.0: BAR 0: assigned [mem 0x200c0000-0x200dffff]
>> pci 0000:01:00.0: BAR 3: assigned [mem 0x200e0000-0x200e3fff]
>> pci 0000:01:00.0: BAR 2: assigned [io  0x18020000-0x1802001f]
>> pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x20c
>> pci 0000:01:00.0: calling baikal_t1_pcie_link_speed_fixup+0x0/0x228
>> pci 0000:01:00.0: Link Capability is GEN1, x1
>>
>> but PCIe networking card (e1000e) driver fails to load:
>>
>> bus: 'pci': driver_probe_device: matched device 0000:01:00.0 with driver
>> e1000e
>> bus: 'pci': really_probe: probing driver e1000e with device 0000:01:00.0
>> devices_kset: Moving 0000:01:00.0 to end of list
>> PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
>> e1000e 0000:01:00.0: enabling bus mastering
>> e1000_probe:7054 mmio_start 0x200c0000 mmio_len 0x20000
>>
>> e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic
>> conservative mode
>>
>> PCI MSI: setup hwirq:0  virq:51
>> PCI MSI: setup hwirq:1  virq:52
>> PCI MSI: setup hwirq:2  virq:53
>>
>> e1000_get_phy_id_82571:444 ret_val = e1e_rphy(hw, MII_PHYSID1, &phy_id);
>> ret_val = -2
>>
>> It turns out, that PHY ID can't be read from the memory, mapped on
>> driver initialization function by ioremap.
>>
>>    __ioremap (arch/mips/mm/ioremap.c) considers all the memory above the
>> low 512Mb as DRAM and maps the physical addresses 0x20000000 -
>> 0x3FFFFFFF to logical addresses starting from c0000000 and higher.
>>
>> arch/mips/include/asm/mach-baikal/kernel-entry-init.h defines EVA memory
>> map with virtual addresses range 0xc0000000 - 0xdfffffff as 'MK (kseg2)'
>> i.e. "mapped kernel" with cashing segment attributes defined in SegCtl0,
>> Cfg1: "cacheable, coherent, write-back, write-allocate, read misses
>> request shared attribute".
>>
>> I've tried to change the caching attribute (bits 16:18 in SegCtl0) to
>> 0x2 ("uncached, not coherent"), but
>>
>> this hasn't yield any visible effect on PCIe e1000e initialization.
> Mapped kernel segments don't use the CCA in SegCtl. Each page mapping
> has its own CCA specified in the TLB, which would have been loaded from
> page tables set up by e.g. remap_area_pte(), and the ioremap() macro
> does pass _CACHE_UNCACHED as the flags argument. I would recommend
> checking the phys_addr and flags coming in to remap_area_pte().
>
> If that all looks correct you could also try:
> - Dump the TLB contents straight after accessing that address to check
>    they are loaded correctly, i.e. call dump_tlb_all() from
>    <asm/tlbdebug.h>.
> - If the TLB contents look wrong, try disabling hardware page table
>    walker to determine whether that is misconfigured (add nohtw to kernel
>    command line).
>
> Hope that helps
>
> Cheers
> James

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 7207 bytes --]

Hi Yuri,

On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
> Hi, James.
> 
> Do I understand correctly, that in case of CONFIG_EVA=y, any logical 
> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel 
> space is:
> 1) Unmapped (no TLB translations)
> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is, 
> in fact a read from physical address 0x20000000? I mean, in legacy 
> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in 
> kernel space were being translated to the physical addresses from the 
> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or 
> something for EVA, the mapping is 1:1?

That depends on the EVA configuration. The hardware is fairly flexible
(which is both useful and painful - making supporting EVA from
multiplatform kernels particularly awkward), but that is one possible
configuration.

E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
MUSK (TLB mapped for user & kernel) so that null dereferences from
kernel code are safely caught, but that costs you 1GB of directly
accessible physical address space from kernel mode.

> 
> As for userspace addresses, the addresses from the 0x00000000 - 
> 0xBFFFFFFF range are:
> 1) Overlayed with the range which is directly accessible from the kernel 
> space

Yes. Userland accesses access through tlb, kernel direct, and kernel eva
accesses (LWE/SWE etc) access through tlb from kernel

> 2) These addresses are mapped in userland, so, read from logical address 
> 0x20000000 in userland may result in read from any physical address 
> located in range 0x00000000 - 0xBFFFFFFF as defined by TLB for that 
> particular userland thread?

Yes, though it could in theory be any physical address mapped by the
TLB, not just that range.

Cheers
James

> 
> TIA,
> Yuri
> 
> On 12/06/2017 02:46 PM, James Hogan wrote:
> > Hi Yuri,
> >
> > On Wed, Dec 06, 2017 at 01:36:48PM +0300, Yuri Frolov wrote:
> >> Hi,
> >>
> >> I'm trying to expand PCI mapped region using EVA addressing scheme for
> >> 'Baikal' P5600-based family of boards.
> >>
> >> Currently, we've got only one PCIe mapped region, 0x08000000 -
> >> 0x1BDBFFFF (~330Mb).
> >>
> >> This window is unsufficient for some PCIe cards, so I'm trying to expand
> >> (or add one more) PCIe mapped region by moving 'Hi Memory DRAM block'
> >> from 0x20000000 to 0x40000000 (dts changed, BAIKAL_HIGHMEM_START and
> >> HIGHMEM_START are changed in
> >> arch/mips/include/asm/mach-baikal/hardware.h
> >> /arch/mips/include/asm/mach-baikal/spaces.h respectively) and and
> >> allocating this region for PCIe.
> >>
> >> So far, PCI subsystem is initialized during the boot successfully (as
> >> far as I can see):
> >>
> >> dw_pcie_init: DEV_ID_VEND_ID=0x80601d39 CLASS_CODE_REV_ID=0x1
> >> dw_pcie_init: PCIe error code = 0x0
> >> dw_pcie_init: PCIe link speed GEN1
> >> device: 'pci0000:01': device_add
> >> PM: Adding info for No Bus:pci0000:01
> >> device: '0000:01': device_add
> >> PM: Adding info for No Bus:0000:01
> >> PCI host bridge to bus 0000:01
> >> pci_bus 0000:01: root bus resource [mem 0x20000000-0x37ffffff]
> >> pci_bus 0000:01: root bus resource [io  0x18020000-0x1bdaffff]
> >> pci_bus 0000:01: root bus resource [bus 01-ff]
> >> pci_bus 0000:01: scanning bus
> >> pci 0000:01:00.0: [8086:10d3] type 00 class 0x020000
> >> pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x98
> >> pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
> >> pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0007ffff]
> >> pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
> >> pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
> >> pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
> >> pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> >>
> >> pci 0000:01:00.0: PME# disabled
> >> device: '0000:01:00.0': device_add
> >> bus: 'pci': add device 0000:01:00.0
> >> PM: Adding info for pci:0000:01:00.0
> >> pci_bus 0000:01: fixups for bus
> >> pci_bus 0000:01: bus scan returning with max=01
> >> pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2007ffff]
> >> pci 0000:01:00.0: BAR 6: assigned [mem 0x20080000-0x200bffff pref]
> >> pci 0000:01:00.0: BAR 0: assigned [mem 0x200c0000-0x200dffff]
> >> pci 0000:01:00.0: BAR 3: assigned [mem 0x200e0000-0x200e3fff]
> >> pci 0000:01:00.0: BAR 2: assigned [io  0x18020000-0x1802001f]
> >> pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x20c
> >> pci 0000:01:00.0: calling baikal_t1_pcie_link_speed_fixup+0x0/0x228
> >> pci 0000:01:00.0: Link Capability is GEN1, x1
> >>
> >> but PCIe networking card (e1000e) driver fails to load:
> >>
> >> bus: 'pci': driver_probe_device: matched device 0000:01:00.0 with driver
> >> e1000e
> >> bus: 'pci': really_probe: probing driver e1000e with device 0000:01:00.0
> >> devices_kset: Moving 0000:01:00.0 to end of list
> >> PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> >> e1000e 0000:01:00.0: enabling bus mastering
> >> e1000_probe:7054 mmio_start 0x200c0000 mmio_len 0x20000
> >>
> >> e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic
> >> conservative mode
> >>
> >> PCI MSI: setup hwirq:0  virq:51
> >> PCI MSI: setup hwirq:1  virq:52
> >> PCI MSI: setup hwirq:2  virq:53
> >>
> >> e1000_get_phy_id_82571:444 ret_val = e1e_rphy(hw, MII_PHYSID1, &phy_id);
> >> ret_val = -2
> >>
> >> It turns out, that PHY ID can't be read from the memory, mapped on
> >> driver initialization function by ioremap.
> >>
> >>    __ioremap (arch/mips/mm/ioremap.c) considers all the memory above the
> >> low 512Mb as DRAM and maps the physical addresses 0x20000000 -
> >> 0x3FFFFFFF to logical addresses starting from c0000000 and higher.
> >>
> >> arch/mips/include/asm/mach-baikal/kernel-entry-init.h defines EVA memory
> >> map with virtual addresses range 0xc0000000 - 0xdfffffff as 'MK (kseg2)'
> >> i.e. "mapped kernel" with cashing segment attributes defined in SegCtl0,
> >> Cfg1: "cacheable, coherent, write-back, write-allocate, read misses
> >> request shared attribute".
> >>
> >> I've tried to change the caching attribute (bits 16:18 in SegCtl0) to
> >> 0x2 ("uncached, not coherent"), but
> >>
> >> this hasn't yield any visible effect on PCIe e1000e initialization.
> > Mapped kernel segments don't use the CCA in SegCtl. Each page mapping
> > has its own CCA specified in the TLB, which would have been loaded from
> > page tables set up by e.g. remap_area_pte(), and the ioremap() macro
> > does pass _CACHE_UNCACHED as the flags argument. I would recommend
> > checking the phys_addr and flags coming in to remap_area_pte().
> >
> > If that all looks correct you could also try:
> > - Dump the TLB contents straight after accessing that address to check
> >    they are loaded correctly, i.e. call dump_tlb_all() from
> >    <asm/tlbdebug.h>.
> > - If the TLB contents look wrong, try disabling hardware page table
> >    walker to determine whether that is misconfigured (add nohtw to kernel
> >    command line).
> >
> > Hope that helps
> >
> > Cheers
> > James
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 7207 bytes --]

Hi Yuri,

On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
> Hi, James.
> 
> Do I understand correctly, that in case of CONFIG_EVA=y, any logical 
> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel 
> space is:
> 1) Unmapped (no TLB translations)
> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is, 
> in fact a read from physical address 0x20000000? I mean, in legacy 
> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in 
> kernel space were being translated to the physical addresses from the 
> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or 
> something for EVA, the mapping is 1:1?

That depends on the EVA configuration. The hardware is fairly flexible
(which is both useful and painful - making supporting EVA from
multiplatform kernels particularly awkward), but that is one possible
configuration.

E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
MUSK (TLB mapped for user & kernel) so that null dereferences from
kernel code are safely caught, but that costs you 1GB of directly
accessible physical address space from kernel mode.

> 
> As for userspace addresses, the addresses from the 0x00000000 - 
> 0xBFFFFFFF range are:
> 1) Overlayed with the range which is directly accessible from the kernel 
> space

Yes. Userland accesses access through tlb, kernel direct, and kernel eva
accesses (LWE/SWE etc) access through tlb from kernel

> 2) These addresses are mapped in userland, so, read from logical address 
> 0x20000000 in userland may result in read from any physical address 
> located in range 0x00000000 - 0xBFFFFFFF as defined by TLB for that 
> particular userland thread?

Yes, though it could in theory be any physical address mapped by the
TLB, not just that range.

Cheers
James

> 
> TIA,
> Yuri
> 
> On 12/06/2017 02:46 PM, James Hogan wrote:
> > Hi Yuri,
> >
> > On Wed, Dec 06, 2017 at 01:36:48PM +0300, Yuri Frolov wrote:
> >> Hi,
> >>
> >> I'm trying to expand PCI mapped region using EVA addressing scheme for
> >> 'Baikal' P5600-based family of boards.
> >>
> >> Currently, we've got only one PCIe mapped region, 0x08000000 -
> >> 0x1BDBFFFF (~330Mb).
> >>
> >> This window is unsufficient for some PCIe cards, so I'm trying to expand
> >> (or add one more) PCIe mapped region by moving 'Hi Memory DRAM block'
> >> from 0x20000000 to 0x40000000 (dts changed, BAIKAL_HIGHMEM_START and
> >> HIGHMEM_START are changed in
> >> arch/mips/include/asm/mach-baikal/hardware.h
> >> /arch/mips/include/asm/mach-baikal/spaces.h respectively) and and
> >> allocating this region for PCIe.
> >>
> >> So far, PCI subsystem is initialized during the boot successfully (as
> >> far as I can see):
> >>
> >> dw_pcie_init: DEV_ID_VEND_ID=0x80601d39 CLASS_CODE_REV_ID=0x1
> >> dw_pcie_init: PCIe error code = 0x0
> >> dw_pcie_init: PCIe link speed GEN1
> >> device: 'pci0000:01': device_add
> >> PM: Adding info for No Bus:pci0000:01
> >> device: '0000:01': device_add
> >> PM: Adding info for No Bus:0000:01
> >> PCI host bridge to bus 0000:01
> >> pci_bus 0000:01: root bus resource [mem 0x20000000-0x37ffffff]
> >> pci_bus 0000:01: root bus resource [io  0x18020000-0x1bdaffff]
> >> pci_bus 0000:01: root bus resource [bus 01-ff]
> >> pci_bus 0000:01: scanning bus
> >> pci 0000:01:00.0: [8086:10d3] type 00 class 0x020000
> >> pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x98
> >> pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
> >> pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0007ffff]
> >> pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
> >> pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
> >> pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
> >> pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> >>
> >> pci 0000:01:00.0: PME# disabled
> >> device: '0000:01:00.0': device_add
> >> bus: 'pci': add device 0000:01:00.0
> >> PM: Adding info for pci:0000:01:00.0
> >> pci_bus 0000:01: fixups for bus
> >> pci_bus 0000:01: bus scan returning with max=01
> >> pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2007ffff]
> >> pci 0000:01:00.0: BAR 6: assigned [mem 0x20080000-0x200bffff pref]
> >> pci 0000:01:00.0: BAR 0: assigned [mem 0x200c0000-0x200dffff]
> >> pci 0000:01:00.0: BAR 3: assigned [mem 0x200e0000-0x200e3fff]
> >> pci 0000:01:00.0: BAR 2: assigned [io  0x18020000-0x1802001f]
> >> pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x20c
> >> pci 0000:01:00.0: calling baikal_t1_pcie_link_speed_fixup+0x0/0x228
> >> pci 0000:01:00.0: Link Capability is GEN1, x1
> >>
> >> but PCIe networking card (e1000e) driver fails to load:
> >>
> >> bus: 'pci': driver_probe_device: matched device 0000:01:00.0 with driver
> >> e1000e
> >> bus: 'pci': really_probe: probing driver e1000e with device 0000:01:00.0
> >> devices_kset: Moving 0000:01:00.0 to end of list
> >> PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> >> e1000e 0000:01:00.0: enabling bus mastering
> >> e1000_probe:7054 mmio_start 0x200c0000 mmio_len 0x20000
> >>
> >> e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic
> >> conservative mode
> >>
> >> PCI MSI: setup hwirq:0  virq:51
> >> PCI MSI: setup hwirq:1  virq:52
> >> PCI MSI: setup hwirq:2  virq:53
> >>
> >> e1000_get_phy_id_82571:444 ret_val = e1e_rphy(hw, MII_PHYSID1, &phy_id);
> >> ret_val = -2
> >>
> >> It turns out, that PHY ID can't be read from the memory, mapped on
> >> driver initialization function by ioremap.
> >>
> >>    __ioremap (arch/mips/mm/ioremap.c) considers all the memory above the
> >> low 512Mb as DRAM and maps the physical addresses 0x20000000 -
> >> 0x3FFFFFFF to logical addresses starting from c0000000 and higher.
> >>
> >> arch/mips/include/asm/mach-baikal/kernel-entry-init.h defines EVA memory
> >> map with virtual addresses range 0xc0000000 - 0xdfffffff as 'MK (kseg2)'
> >> i.e. "mapped kernel" with cashing segment attributes defined in SegCtl0,
> >> Cfg1: "cacheable, coherent, write-back, write-allocate, read misses
> >> request shared attribute".
> >>
> >> I've tried to change the caching attribute (bits 16:18 in SegCtl0) to
> >> 0x2 ("uncached, not coherent"), but
> >>
> >> this hasn't yield any visible effect on PCIe e1000e initialization.
> > Mapped kernel segments don't use the CCA in SegCtl. Each page mapping
> > has its own CCA specified in the TLB, which would have been loaded from
> > page tables set up by e.g. remap_area_pte(), and the ioremap() macro
> > does pass _CACHE_UNCACHED as the flags argument. I would recommend
> > checking the phys_addr and flags coming in to remap_area_pte().
> >
> > If that all looks correct you could also try:
> > - Dump the TLB contents straight after accessing that address to check
> >    they are loaded correctly, i.e. call dump_tlb_all() from
> >    <asm/tlbdebug.h>.
> > - If the TLB contents look wrong, try disabling hardware page table
> >    walker to determine whether that is misconfigured (add nohtw to kernel
> >    command line).
> >
> > Hope that helps
> >
> > Cheers
> > James
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

Hi, James

On 12/14/2017 06:21 PM, James Hogan wrote:
> Hi Yuri,
>
> On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
>> Hi, James.
>>
>> Do I understand correctly, that in case of CONFIG_EVA=y, any logical
>> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel
>> space is:
>> 1) Unmapped (no TLB translations)
>> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is,
>> in fact a read from physical address 0x20000000? I mean, in legacy
>> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in
>> kernel space were being translated to the physical addresses from the
>> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or
>> something for EVA, the mapping is 1:1?
> That depends on the EVA configuration. The hardware is fairly flexible
> (which is both useful and painful - making supporting EVA from
> multiplatform kernels particularly awkward), but that is one possible
> configuration.
>
> E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
> MUSK (TLB mapped for user & kernel) so that null dereferences from
> kernel code are safely caught, but that costs you 1GB of directly
> accessible physical address space from kernel mode.
So, do I understand correctly, that provided we have these TLB entries 
in kernel,

Index: 71 pgmask=16kb va=c0038000 asid=00
         [ri=0 xi=0 pa=200e0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=00000000 
c=0 d=0 v=0 g=1]
Index: 72 pgmask=16kb va=c0040000 asid=00
         [ri=0 xi=0 pa=200c0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=200c4000 
c=2 d=1 v=1 g=1]

u32 l;

l = readl((const volatile void *)(0x200c0000 + 0x20))
and
l = *(u32 *)(0xc0040000+ 0x20)

should return the same value?
>
>> As for userspace addresses, the addresses from the 0x00000000 -
>> 0xBFFFFFFF range are:
>> 1) Overlayed with the range which is directly accessible from the kernel
>> space
> Yes. Userland accesses access through tlb, kernel direct, and kernel eva
> accesses (LWE/SWE etc) access through tlb from kernel
>
>> 2) These addresses are mapped in userland, so, read from logical address
>> 0x20000000 in userland may result in read from any physical address
>> located in range 0x00000000 - 0xBFFFFFFF as defined by TLB for that
>> particular userland thread?
> Yes, though it could in theory be any physical address mapped by the
> TLB, not just that range.
>
> Cheers
> James

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

On Fri, Dec 15, 2017 at 05:05:12PM +0300, Yuri Frolov wrote:
> Hi, James
> 
> On 12/14/2017 06:21 PM, James Hogan wrote:
> > Hi Yuri,
> >
> > On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
> >> Hi, James.
> >>
> >> Do I understand correctly, that in case of CONFIG_EVA=y, any logical
> >> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel
> >> space is:
> >> 1) Unmapped (no TLB translations)
> >> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is,
> >> in fact a read from physical address 0x20000000? I mean, in legacy
> >> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in
> >> kernel space were being translated to the physical addresses from the
> >> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or
> >> something for EVA, the mapping is 1:1?
> > That depends on the EVA configuration. The hardware is fairly flexible
> > (which is both useful and painful - making supporting EVA from
> > multiplatform kernels particularly awkward), but that is one possible
> > configuration.
> >
> > E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
> > MUSK (TLB mapped for user & kernel) so that null dereferences from
> > kernel code are safely caught, but that costs you 1GB of directly
> > accessible physical address space from kernel mode.
> So, do I understand correctly, that provided we have these TLB entries 
> in kernel,
> 
> Index: 71 pgmask=16kb va=c0038000 asid=00
>          [ri=0 xi=0 pa=200e0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=00000000 
> c=0 d=0 v=0 g=1]
> Index: 72 pgmask=16kb va=c0040000 asid=00
>          [ri=0 xi=0 pa=200c0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=200c4000 
> c=2 d=1 v=1 g=1]
> 
> u32 l;
> 
> l = readl((const volatile void *)(0x200c0000 + 0x20))

assuming you have segment5 (0x00000000..0x3FFFFFFF) set to MUSUK, with
PA 0 (i.e. direct 1:1 mapping), it'd access PA 0x200c0020, but
presumably the segment will be configured to be cached (CCA 3) or cached
coherent (CCA 5).

> and
> l = *(u32 *)(0xc0040000+ 0x20)

this will access physical addres 0x200c0020 uncached (CCA 2).

> 
> should return the same value?

So that would depend I think on whether there is a dirty value in the
cache. Possibly not. The cached mapping would see the dirty value. The
uncached mapping may see a stale value straight from RAM.

Cheers
James

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

On Fri, Dec 15, 2017 at 05:05:12PM +0300, Yuri Frolov wrote:
> Hi, James
> 
> On 12/14/2017 06:21 PM, James Hogan wrote:
> > Hi Yuri,
> >
> > On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
> >> Hi, James.
> >>
> >> Do I understand correctly, that in case of CONFIG_EVA=y, any logical
> >> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel
> >> space is:
> >> 1) Unmapped (no TLB translations)
> >> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is,
> >> in fact a read from physical address 0x20000000? I mean, in legacy
> >> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in
> >> kernel space were being translated to the physical addresses from the
> >> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or
> >> something for EVA, the mapping is 1:1?
> > That depends on the EVA configuration. The hardware is fairly flexible
> > (which is both useful and painful - making supporting EVA from
> > multiplatform kernels particularly awkward), but that is one possible
> > configuration.
> >
> > E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
> > MUSK (TLB mapped for user & kernel) so that null dereferences from
> > kernel code are safely caught, but that costs you 1GB of directly
> > accessible physical address space from kernel mode.
> So, do I understand correctly, that provided we have these TLB entries 
> in kernel,
> 
> Index: 71 pgmask=16kb va=c0038000 asid=00
>          [ri=0 xi=0 pa=200e0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=00000000 
> c=0 d=0 v=0 g=1]
> Index: 72 pgmask=16kb va=c0040000 asid=00
>          [ri=0 xi=0 pa=200c0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=200c4000 
> c=2 d=1 v=1 g=1]
> 
> u32 l;
> 
> l = readl((const volatile void *)(0x200c0000 + 0x20))

assuming you have segment5 (0x00000000..0x3FFFFFFF) set to MUSUK, with
PA 0 (i.e. direct 1:1 mapping), it'd access PA 0x200c0020, but
presumably the segment will be configured to be cached (CCA 3) or cached
coherent (CCA 5).

> and
> l = *(u32 *)(0xc0040000+ 0x20)

this will access physical addres 0x200c0020 uncached (CCA 2).

> 
> should return the same value?

So that would depend I think on whether there is a dirty value in the
cache. Possibly not. The cached mapping would see the dirty value. The
uncached mapping may see a stale value straight from RAM.

Cheers
James

Re: [P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

Hi, James

On 12/16/2017 02:28 AM, James Hogan wrote:
> On Fri, Dec 15, 2017 at 05:05:12PM +0300, Yuri Frolov wrote:
>> Hi, James
>>
>> On 12/14/2017 06:21 PM, James Hogan wrote:
>>> Hi Yuri,
>>>
>>> On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
>>>> Hi, James.
>>>>
>>>> Do I understand correctly, that in case of CONFIG_EVA=y, any logical
>>>> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel
>>>> space is:
>>>> 1) Unmapped (no TLB translations)
>>>> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is,
>>>> in fact a read from physical address 0x20000000? I mean, in legacy
>>>> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in
>>>> kernel space were being translated to the physical addresses from the
>>>> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or
>>>> something for EVA, the mapping is 1:1?
>>> That depends on the EVA configuration. The hardware is fairly flexible
>>> (which is both useful and painful - making supporting EVA from
>>> multiplatform kernels particularly awkward), but that is one possible
>>> configuration.
>>>
>>> E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
>>> MUSK (TLB mapped for user & kernel) so that null dereferences from
>>> kernel code are safely caught, but that costs you 1GB of directly
>>> accessible physical address space from kernel mode.
>> So, do I understand correctly, that provided we have these TLB entries
>> in kernel,
>>
>> Index: 71 pgmask=16kb va=c0038000 asid=00
>>           [ri=0 xi=0 pa=200e0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=00000000
>> c=0 d=0 v=0 g=1]
>> Index: 72 pgmask=16kb va=c0040000 asid=00
>>           [ri=0 xi=0 pa=200c0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=200c4000
>> c=2 d=1 v=1 g=1]
>>
>> u32 l;
>>
>> l = readl((const volatile void *)(0x200c0000 + 0x20))
> assuming you have segment5 (0x00000000..0x3FFFFFFF) set to MUSUK, with
> PA 0 (i.e. direct 1:1 mapping), it'd access PA 0x200c0020, but
> presumably the segment will be configured to be cached (CCA 3) or cached
> coherent (CCA 5).
>
>> and
>> l = *(u32 *)(0xc0040000+ 0x20)
> this will access physical addres 0x200c0020 uncached (CCA 2).
>
>> should return the same value?
> So that would depend I think on whether there is a dirty value in the
> cache. Possibly not. The cached mapping would see the dirty value. The
> uncached mapping may see a stale value straight from RAM.
>
> Cheers
> James
I'm looking at arch/mips/include/asm//mach-malta/kernel-entry-init.h and 
there is a definition for SegCtl2:

         /* SegCtl2 */
         li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |      \
                 (6 << MIPS_SEGCFG_PA_SHIFT) |                           \
                 (1 << MIPS_SEGCFG_EU_SHIFT)) |                          \
                 (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |         \
                 (4 << MIPS_SEGCFG_PA_SHIFT) |                           \
                 (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)

it defines, that kernel logical addresses from the range 0x00000000 - 
0x7fffffff are unmapped (no tlbs) and dictates, that in order to get a 
physical address for any logical addresses from 0x00000000 - 0x3fffffff 
range in kernel space, bits [31:29] of the logical address must be 
changed to 100,
and (again in kernel space) for any logical addresses from 0x40000000 - 
0x7fffffff range, bits [31:29] of the logical address must be changed to 
110, right?

What physical addresses will logical addresses 0x00000000 and 0x20000000 
be translated in kernel space?.. logical 0x00000000 --> physical 
0x80000000, and logical 0x20000000 --> .... 0x80000000 too?
Since we must to change bits [31:29], we have to change bit 29 ('1') in 
logical address 0x200000000 to '0' (since PA for this range is 100).

So, what physical addresses will all logical addresses which have '1' at 
29 bit be translated, if we define PA as 100 and 110 in SegCtl2? It 
looks like there's no flat translation of logical addresses to physical 
addresses in kernel space, and this is obviously just not correct, there 
is something simple I've been overlooking.

Thank you,
Yuri

Re: [P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

 > I'm looking at arch/mips/include/asm//mach-malta/kernel-entry-init.h 
and there is a definition for
 > SegCtl2:

>
>         /* SegCtl2 */
>         li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |      \
>                 (6 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                 (1 << MIPS_SEGCFG_EU_SHIFT)) |                          \
>                 (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |         \
>                 (4 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                 (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
>
> it defines, that kernel logical addresses from the range 0x00000000 - 
> 0x7fffffff are unmapped (no tlbs) and dictates, that in order to get a 
> physical address for any logical addresses from 0x00000000 - 
> 0x3fffffff range in kernel space, bits [31:29] of the logical address 
> must be changed to 100,
> and (again in kernel space) for any logical addresses from 0x40000000 
> - 0x7fffffff range, bits [31:29] of the logical address must be 
> changed to 110, right?
>
> What physical addresses will logical addresses 0x00000000 and 
> 0x20000000 be translated in kernel space?.. logical 0x00000000 --> 
> physical 0x80000000, and logical 0x20000000 --> .... 0x80000000 too?
> Since we must to change bits [31:29], we have to change bit 29 ('1') 
> in logical address 0x200000000 to '0' (since PA for this range is 100).
>
> So, what physical addresses will all logical addresses which have '1' 
> at 29 bit be translated, if we define PA as 100 and 110 in SegCtl2? It 
> looks like there's no flat translation of logical addresses to 
> physical addresses in kernel space, and this is obviously just not 
> correct, there is something simple I've been overlooking.
>
> Thank you,
> Yuri

The only way this scheme can work, is that 'these bits are used' phrase 
from Table 3.5 CFG (Segment Configuration) Field Description means not 
'these bits substitute bits [31:29] in virtual address', but 'these bits 
are 'OR'-ed with bits [31:29] in virtual address.

Re: [P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

 > I'm looking at arch/mips/include/asm//mach-malta/kernel-entry-init.h 
and there is a definition for
 > SegCtl2:

>
>         /* SegCtl2 */
>         li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |      \
>                 (6 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                 (1 << MIPS_SEGCFG_EU_SHIFT)) |                          \
>                 (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |         \
>                 (4 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                 (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
>
> it defines, that kernel logical addresses from the range 0x00000000 - 
> 0x7fffffff are unmapped (no tlbs) and dictates, that in order to get a 
> physical address for any logical addresses from 0x00000000 - 
> 0x3fffffff range in kernel space, bits [31:29] of the logical address 
> must be changed to 100,
> and (again in kernel space) for any logical addresses from 0x40000000 
> - 0x7fffffff range, bits [31:29] of the logical address must be 
> changed to 110, right?
>
> What physical addresses will logical addresses 0x00000000 and 
> 0x20000000 be translated in kernel space?.. logical 0x00000000 --> 
> physical 0x80000000, and logical 0x20000000 --> .... 0x80000000 too?
> Since we must to change bits [31:29], we have to change bit 29 ('1') 
> in logical address 0x200000000 to '0' (since PA for this range is 100).
>
> So, what physical addresses will all logical addresses which have '1' 
> at 29 bit be translated, if we define PA as 100 and 110 in SegCtl2? It 
> looks like there's no flat translation of logical addresses to 
> physical addresses in kernel space, and this is obviously just not 
> correct, there is something simple I've been overlooking.
>
> Thank you,
> Yuri

The only way this scheme can work, is that 'these bits are used' phrase 
from Table 3.5 CFG (Segment Configuration) Field Description means not 
'these bits substitute bits [31:29] in virtual address', but 'these bits 
are 'OR'-ed with bits [31:29] in virtual address.

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

On Wed, Dec 20, 2017 at 06:11:04PM +0300, Yuri Frolov wrote:
> Hi, James
> 
> On 12/16/2017 02:28 AM, James Hogan wrote:
> > On Fri, Dec 15, 2017 at 05:05:12PM +0300, Yuri Frolov wrote:
> >> Hi, James
> >>
> >> On 12/14/2017 06:21 PM, James Hogan wrote:
> >>> Hi Yuri,
> >>>
> >>> On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
> >>>> Hi, James.
> >>>>
> >>>> Do I understand correctly, that in case of CONFIG_EVA=y, any logical
> >>>> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel
> >>>> space is:
> >>>> 1) Unmapped (no TLB translations)
> >>>> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is,
> >>>> in fact a read from physical address 0x20000000? I mean, in legacy
> >>>> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in
> >>>> kernel space were being translated to the physical addresses from the
> >>>> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or
> >>>> something for EVA, the mapping is 1:1?
> >>> That depends on the EVA configuration. The hardware is fairly flexible
> >>> (which is both useful and painful - making supporting EVA from
> >>> multiplatform kernels particularly awkward), but that is one possible
> >>> configuration.
> >>>
> >>> E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
> >>> MUSK (TLB mapped for user & kernel) so that null dereferences from
> >>> kernel code are safely caught, but that costs you 1GB of directly
> >>> accessible physical address space from kernel mode.
> >> So, do I understand correctly, that provided we have these TLB entries
> >> in kernel,
> >>
> >> Index: 71 pgmask=16kb va=c0038000 asid=00
> >>           [ri=0 xi=0 pa=200e0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=00000000
> >> c=0 d=0 v=0 g=1]
> >> Index: 72 pgmask=16kb va=c0040000 asid=00
> >>           [ri=0 xi=0 pa=200c0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=200c4000
> >> c=2 d=1 v=1 g=1]
> >>
> >> u32 l;
> >>
> >> l = readl((const volatile void *)(0x200c0000 + 0x20))
> > assuming you have segment5 (0x00000000..0x3FFFFFFF) set to MUSUK, with
> > PA 0 (i.e. direct 1:1 mapping), it'd access PA 0x200c0020, but
> > presumably the segment will be configured to be cached (CCA 3) or cached
> > coherent (CCA 5).
> >
> >> and
> >> l = *(u32 *)(0xc0040000+ 0x20)
> > this will access physical addres 0x200c0020 uncached (CCA 2).
> >
> >> should return the same value?
> > So that would depend I think on whether there is a dirty value in the
> > cache. Possibly not. The cached mapping would see the dirty value. The
> > uncached mapping may see a stale value straight from RAM.
> >
> > Cheers
> > James
> I'm looking at arch/mips/include/asm//mach-malta/kernel-entry-init.h and 
> there is a definition for SegCtl2:
> 
>          /* SegCtl2 */
>          li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |      \
>                  (6 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                  (1 << MIPS_SEGCFG_EU_SHIFT)) |                          \
>                  (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |         \
>                  (4 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                  (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
> 
> it defines, that kernel logical addresses from the range 0x00000000 - 
> 0x7fffffff are unmapped (no tlbs) and dictates, that in order to get a 
> physical address for any logical addresses from 0x00000000 - 0x3fffffff 
> range in kernel space, bits [31:29] of the logical address must be 
> changed to 100,
> and (again in kernel space) for any logical addresses from 0x40000000 - 
> 0x7fffffff range, bits [31:29] of the logical address must be changed to 
> 110, right?

yes, the Malta implementation is slightly ugly as it relies on a
hardware physical memory alias of RAM starting at PA 0x80000000.

> 
> What physical addresses will logical addresses 0x00000000 and 0x20000000 
> be translated in kernel space?.. logical 0x00000000 --> physical 
> 0x80000000, and logical 0x20000000 --> .... 0x80000000 too?

VA 0x20000000 -> PA 0xa0000000, since seg4 and seg5 are 1GB segments, so
its only bits 30 and up that can be changed. I seem to remember the bit
corresponding to bit 29 isn't even writable in the SegCtl2 register.

Does that clarify things?

Cheers
James

> Since we must to change bits [31:29], we have to change bit 29 ('1') in 
> logical address 0x200000000 to '0' (since PA for this range is 100).
> 
> So, what physical addresses will all logical addresses which have '1' at 
> 29 bit be translated, if we define PA as 100 and 110 in SegCtl2? It 
> looks like there's no flat translation of logical addresses to physical 
> addresses in kernel space, and this is obviously just not correct, there 
> is something simple I've been overlooking.
> 
> Thank you,
> Yuri

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

On Wed, Dec 20, 2017 at 06:11:04PM +0300, Yuri Frolov wrote:
> Hi, James
> 
> On 12/16/2017 02:28 AM, James Hogan wrote:
> > On Fri, Dec 15, 2017 at 05:05:12PM +0300, Yuri Frolov wrote:
> >> Hi, James
> >>
> >> On 12/14/2017 06:21 PM, James Hogan wrote:
> >>> Hi Yuri,
> >>>
> >>> On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote:
> >>>> Hi, James.
> >>>>
> >>>> Do I understand correctly, that in case of CONFIG_EVA=y, any logical
> >>>> address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel
> >>>> space is:
> >>>> 1) Unmapped (no TLB translations)
> >>>> 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is,
> >>>> in fact a read from physical address 0x20000000? I mean, in legacy
> >>>> memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in
> >>>> kernel space were being translated to the physical addresses from the
> >>>> low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or
> >>>> something for EVA, the mapping is 1:1?
> >>> That depends on the EVA configuration. The hardware is fairly flexible
> >>> (which is both useful and painful - making supporting EVA from
> >>> multiplatform kernels particularly awkward), but that is one possible
> >>> configuration.
> >>>
> >>> E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF)
> >>> MUSK (TLB mapped for user & kernel) so that null dereferences from
> >>> kernel code are safely caught, but that costs you 1GB of directly
> >>> accessible physical address space from kernel mode.
> >> So, do I understand correctly, that provided we have these TLB entries
> >> in kernel,
> >>
> >> Index: 71 pgmask=16kb va=c0038000 asid=00
> >>           [ri=0 xi=0 pa=200e0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=00000000
> >> c=0 d=0 v=0 g=1]
> >> Index: 72 pgmask=16kb va=c0040000 asid=00
> >>           [ri=0 xi=0 pa=200c0000 c=2 d=1 v=1 g=1] [ri=0 xi=0 pa=200c4000
> >> c=2 d=1 v=1 g=1]
> >>
> >> u32 l;
> >>
> >> l = readl((const volatile void *)(0x200c0000 + 0x20))
> > assuming you have segment5 (0x00000000..0x3FFFFFFF) set to MUSUK, with
> > PA 0 (i.e. direct 1:1 mapping), it'd access PA 0x200c0020, but
> > presumably the segment will be configured to be cached (CCA 3) or cached
> > coherent (CCA 5).
> >
> >> and
> >> l = *(u32 *)(0xc0040000+ 0x20)
> > this will access physical addres 0x200c0020 uncached (CCA 2).
> >
> >> should return the same value?
> > So that would depend I think on whether there is a dirty value in the
> > cache. Possibly not. The cached mapping would see the dirty value. The
> > uncached mapping may see a stale value straight from RAM.
> >
> > Cheers
> > James
> I'm looking at arch/mips/include/asm//mach-malta/kernel-entry-init.h and 
> there is a definition for SegCtl2:
> 
>          /* SegCtl2 */
>          li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |      \
>                  (6 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                  (1 << MIPS_SEGCFG_EU_SHIFT)) |                          \
>                  (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |         \
>                  (4 << MIPS_SEGCFG_PA_SHIFT) |                           \
>                  (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
> 
> it defines, that kernel logical addresses from the range 0x00000000 - 
> 0x7fffffff are unmapped (no tlbs) and dictates, that in order to get a 
> physical address for any logical addresses from 0x00000000 - 0x3fffffff 
> range in kernel space, bits [31:29] of the logical address must be 
> changed to 100,
> and (again in kernel space) for any logical addresses from 0x40000000 - 
> 0x7fffffff range, bits [31:29] of the logical address must be changed to 
> 110, right?

yes, the Malta implementation is slightly ugly as it relies on a
hardware physical memory alias of RAM starting at PA 0x80000000.

> 
> What physical addresses will logical addresses 0x00000000 and 0x20000000 
> be translated in kernel space?.. logical 0x00000000 --> physical 
> 0x80000000, and logical 0x20000000 --> .... 0x80000000 too?

VA 0x20000000 -> PA 0xa0000000, since seg4 and seg5 are 1GB segments, so
its only bits 30 and up that can be changed. I seem to remember the bit
corresponding to bit 29 isn't even writable in the SegCtl2 register.

Does that clarify things?

Cheers
James

> Since we must to change bits [31:29], we have to change bit 29 ('1') in 
> logical address 0x200000000 to '0' (since PA for this range is 100).
> 
> So, what physical addresses will all logical addresses which have '1' at 
> 29 bit be translated, if we define PA as 100 and 110 in SegCtl2? It 
> looks like there's no flat translation of logical addresses to physical 
> addresses in kernel space, and this is obviously just not correct, there 
> is something simple I've been overlooking.
> 
> Thank you,
> Yuri

Re: [P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

>> I'm looking at arch/mips/include/asm//mach-malta/kernel-entry-init.h and
>> there is a definition for SegCtl2:
>>
>>           /* SegCtl2 */
>>           li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |      \
>>                   (6 << MIPS_SEGCFG_PA_SHIFT) |                           \
>>                   (1 << MIPS_SEGCFG_EU_SHIFT)) |                          \
>>                   (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |         \
>>                   (4 << MIPS_SEGCFG_PA_SHIFT) |                           \
>>                   (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
>>
>> it defines, that kernel logical addresses from the range 0x00000000 -
>> 0x7fffffff are unmapped (no tlbs) and dictates, that in order to get a
>> physical address for any logical addresses from 0x00000000 - 0x3fffffff
>> range in kernel space, bits [31:29] of the logical address must be
>> changed to 100,
>> and (again in kernel space) for any logical addresses from 0x40000000 -
>> 0x7fffffff range, bits [31:29] of the logical address must be changed to
>> 110, right?
> yes, the Malta implementation is slightly ugly as it relies on a
> hardware physical memory alias of RAM starting at PA 0x80000000.
>
>> What physical addresses will logical addresses 0x00000000 and 0x20000000
>> be translated in kernel space?.. logical 0x00000000 --> physical
>> 0x80000000, and logical 0x20000000 --> .... 0x80000000 too?
> VA 0x20000000 -> PA 0xa0000000, since seg4 and seg5 are 1GB segments, so
> its only bits 30 and up that can be changed.
Ah, I've got it now.
>   I seem to remember the bit
> corresponding to bit 29 isn't even writable in the SegCtl2 register.
I shoulda read the actual definition of SegCtl2, my bad. I presumed the 
definitions are the same for
SegCtl0 - SegCtl2.
> Does that clarify things?
Yes, absolutely.  Thank you very much!

-- yuri

Re: [P5600 && EVA memory caching question] PCI region

[Yuri Frolov]

Sorry, I forgot to ask,
> yes, the Malta implementation is slightly ugly as it relies on a
> hardware physical memory alias of RAM starting at PA 0x80000000.
any good docs about this hardware aliasing? I'm not sure I understand it 
correctly, just want to understand things right.

-- yuri

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

On Fri, Dec 22, 2017 at 02:18:26PM +0300, Yuri Frolov wrote:
> Sorry, I forgot to ask,
> > yes, the Malta implementation is slightly ugly as it relies on a
> > hardware physical memory alias of RAM starting at PA 0x80000000.
> any good docs about this hardware aliasing? I'm not sure I understand it 
> correctly, just want to understand things right.

Basically it keeps kseg0 (seg3) and kseg1 (seg2) pointing at PA 0, which
it runs kernel code from (i.e. same place it is loaded, without the EVA
layout enabled yet), but then it does data accesses from seg4 and seg5
which point at PA 0x80000000. I think the intention is to allow the VA
to PA offset to be the same for all cached segments.

However because the caches don't know that different physical addresses
refer to the same underlying RAM they aren't coherent with one another
and the kernel has to be careful not to use both, and to flush the
caches during boot.

Its not an approach I would personally recommend, and if we get EVA
support added to the generic platform I'd hope it would be a lot
cleaner, perhaps using the since added kernel self-relocation.

Cheers
James


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [P5600 && EVA memory caching question] PCI region

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

On Fri, Dec 22, 2017 at 02:18:26PM +0300, Yuri Frolov wrote:
> Sorry, I forgot to ask,
> > yes, the Malta implementation is slightly ugly as it relies on a
> > hardware physical memory alias of RAM starting at PA 0x80000000.
> any good docs about this hardware aliasing? I'm not sure I understand it 
> correctly, just want to understand things right.

Basically it keeps kseg0 (seg3) and kseg1 (seg2) pointing at PA 0, which
it runs kernel code from (i.e. same place it is loaded, without the EVA
layout enabled yet), but then it does data accesses from seg4 and seg5
which point at PA 0x80000000. I think the intention is to allow the VA
to PA offset to be the same for all cached segments.

However because the caches don't know that different physical addresses
refer to the same underlying RAM they aren't coherent with one another
and the kernel has to be careful not to use both, and to flush the
caches during boot.

Its not an approach I would personally recommend, and if we get EVA
support added to the generic platform I'd hope it would be a lot
cleaner, perhaps using the since added kernel self-relocation.

Cheers
James


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]