Hi Yuri, On Thu, Dec 14, 2017 at 06:03:11PM +0300, Yuri Frolov wrote: > Hi, James. > > Do I understand correctly, that in case of CONFIG_EVA=y, any logical > address from 0x00000000 - 0xBFFFFFFF (3G) range accessed from the kernel > space is: > 1) Unmapped (no TLB translations) > 2) Is mapped 1:1 to physical addresses? E.g, readl from 0x20000000 is, > in fact a read from physical address 0x20000000? I mean, in legacy > memory mapping scheme, logical addresses 0x80000000 - 0xBFFFFFFF in > kernel space were being translated to the physical addresses from the > low 512Mb (phys 0x00000000 - 0x20000000), no such bits stripping or > something for EVA, the mapping is 1:1? That depends on the EVA configuration. The hardware is fairly flexible (which is both useful and painful - making supporting EVA from multiplatform kernels particularly awkward), but that is one possible configuration. E.g. ideally you probably want to keep seg5 (0x00000000..0x3FFFFFFF) MUSK (TLB mapped for user & kernel) so that null dereferences from kernel code are safely caught, but that costs you 1GB of directly accessible physical address space from kernel mode. > > As for userspace addresses, the addresses from the 0x00000000 - > 0xBFFFFFFF range are: > 1) Overlayed with the range which is directly accessible from the kernel > space Yes. Userland accesses access through tlb, kernel direct, and kernel eva accesses (LWE/SWE etc) access through tlb from kernel > 2) These addresses are mapped in userland, so, read from logical address > 0x20000000 in userland may result in read from any physical address > located in range 0x00000000 - 0xBFFFFFFF as defined by TLB for that > particular userland thread? Yes, though it could in theory be any physical address mapped by the TLB, not just that range. Cheers James > > TIA, > Yuri > > On 12/06/2017 02:46 PM, James Hogan wrote: > > Hi Yuri, > > > > On Wed, Dec 06, 2017 at 01:36:48PM +0300, Yuri Frolov wrote: > >> Hi, > >> > >> I'm trying to expand PCI mapped region using EVA addressing scheme for > >> 'Baikal' P5600-based family of boards. > >> > >> Currently, we've got only one PCIe mapped region, 0x08000000 - > >> 0x1BDBFFFF (~330Mb). > >> > >> This window is unsufficient for some PCIe cards, so I'm trying to expand > >> (or add one more) PCIe mapped region by moving 'Hi Memory DRAM block' > >> from 0x20000000 to 0x40000000 (dts changed, BAIKAL_HIGHMEM_START and > >> HIGHMEM_START are changed in > >> arch/mips/include/asm/mach-baikal/hardware.h > >> /arch/mips/include/asm/mach-baikal/spaces.h respectively) and and > >> allocating this region for PCIe. > >> > >> So far, PCI subsystem is initialized during the boot successfully (as > >> far as I can see): > >> > >> dw_pcie_init: DEV_ID_VEND_ID=0x80601d39 CLASS_CODE_REV_ID=0x1 > >> dw_pcie_init: PCIe error code = 0x0 > >> dw_pcie_init: PCIe link speed GEN1 > >> device: 'pci0000:01': device_add > >> PM: Adding info for No Bus:pci0000:01 > >> device: '0000:01': device_add > >> PM: Adding info for No Bus:0000:01 > >> PCI host bridge to bus 0000:01 > >> pci_bus 0000:01: root bus resource [mem 0x20000000-0x37ffffff] > >> pci_bus 0000:01: root bus resource [io  0x18020000-0x1bdaffff] > >> pci_bus 0000:01: root bus resource [bus 01-ff] > >> pci_bus 0000:01: scanning bus > >> pci 0000:01:00.0: [8086:10d3] type 00 class 0x020000 > >> pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x98 > >> pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff] > >> pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0007ffff] > >> pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f] > >> pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00003fff] > >> pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref] > >> pci 0000:01:00.0: PME# supported from D0 D3hot D3cold > >> > >> pci 0000:01:00.0: PME# disabled > >> device: '0000:01:00.0': device_add > >> bus: 'pci': add device 0000:01:00.0 > >> PM: Adding info for pci:0000:01:00.0 > >> pci_bus 0000:01: fixups for bus > >> pci_bus 0000:01: bus scan returning with max=01 > >> pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2007ffff] > >> pci 0000:01:00.0: BAR 6: assigned [mem 0x20080000-0x200bffff pref] > >> pci 0000:01:00.0: BAR 0: assigned [mem 0x200c0000-0x200dffff] > >> pci 0000:01:00.0: BAR 3: assigned [mem 0x200e0000-0x200e3fff] > >> pci 0000:01:00.0: BAR 2: assigned [io  0x18020000-0x1802001f] > >> pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x20c > >> pci 0000:01:00.0: calling baikal_t1_pcie_link_speed_fixup+0x0/0x228 > >> pci 0000:01:00.0: Link Capability is GEN1, x1 > >> > >> but PCIe networking card (e1000e) driver fails to load: > >> > >> bus: 'pci': driver_probe_device: matched device 0000:01:00.0 with driver > >> e1000e > >> bus: 'pci': really_probe: probing driver e1000e with device 0000:01:00.0 > >> devices_kset: Moving 0000:01:00.0 to end of list > >> PCI: Enabling device 0000:01:00.0 (0000 -> 0002) > >> e1000e 0000:01:00.0: enabling bus mastering > >> e1000_probe:7054 mmio_start 0x200c0000 mmio_len 0x20000 > >> > >> e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic > >> conservative mode > >> > >> PCI MSI: setup hwirq:0  virq:51 > >> PCI MSI: setup hwirq:1  virq:52 > >> PCI MSI: setup hwirq:2  virq:53 > >> > >> e1000_get_phy_id_82571:444 ret_val = e1e_rphy(hw, MII_PHYSID1, &phy_id); > >> ret_val = -2 > >> > >> It turns out, that PHY ID can't be read from the memory, mapped on > >> driver initialization function by ioremap. > >> > >>  __ioremap (arch/mips/mm/ioremap.c) considers all the memory above the > >> low 512Mb as DRAM and maps the physical addresses 0x20000000 - > >> 0x3FFFFFFF to logical addresses starting from c0000000 and higher. > >> > >> arch/mips/include/asm/mach-baikal/kernel-entry-init.h defines EVA memory > >> map with virtual addresses range 0xc0000000 - 0xdfffffff as 'MK (kseg2)' > >> i.e. "mapped kernel" with cashing segment attributes defined in SegCtl0, > >> Cfg1: "cacheable, coherent, write-back, write-allocate, read misses > >> request shared attribute". > >> > >> I've tried to change the caching attribute (bits 16:18 in SegCtl0) to > >> 0x2 ("uncached, not coherent"), but > >> > >> this hasn't yield any visible effect on PCIe e1000e initialization. > > Mapped kernel segments don't use the CCA in SegCtl. Each page mapping > > has its own CCA specified in the TLB, which would have been loaded from > > page tables set up by e.g. remap_area_pte(), and the ioremap() macro > > does pass _CACHE_UNCACHED as the flags argument. I would recommend > > checking the phys_addr and flags coming in to remap_area_pte(). > > > > If that all looks correct you could also try: > > - Dump the TLB contents straight after accessing that address to check > > they are loaded correctly, i.e. call dump_tlb_all() from > > . > > - If the TLB contents look wrong, try disabling hardware page table > > walker to determine whether that is misconfigured (add nohtw to kernel > > command line). > > > > Hope that helps > > > > Cheers > > James >