From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: Subject: Re: Neophyte questions about PCIe To: David Laight , 'Mason' , linux-pci , Linux ARM References: <063D6719AE5E284EB5DD2968C1650D6DCFE7A70A@AcuExch.aculab.com> From: Jeremy Linton Message-ID: Date: Thu, 9 Mar 2017 16:01:36 -0600 MIME-Version: 1.0 In-Reply-To: <063D6719AE5E284EB5DD2968C1650D6DCFE7A70A@AcuExch.aculab.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Herring , Arnd Bergmann , Ard Biesheuvel , Marc Zyngier , Thibaud Cornic , Phuong Nguyen , Shawn Lin Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+bjorn=helgaas.com@lists.infradead.org List-ID: Hi, On 03/08/2017 07:54 AM, David Laight wrote: > From: Mason >> Sent: 07 March 2017 22:45 >> Hello, >> >> I've been working with the Linux PCIe framework for a few weeks, >> and there are still a few things that remain unclear to me. >> I thought I'd group them in a single message. >> >> 1) If I understand correctly, PCI defines 3 types of (address?) "spaces" >> - configuration >> - memory >> - I/O >> >> I think PCI has its roots in x86, where there are separate >> instructions for I/O accesses and memory accesses (with MMIO >> sitting somewhere in the middle). I'm on ARMv7 which doesn't >> have I/O instructions AFAIK. I'm not sure what the I/O address >> space is used for in PCIe, especially since I was told that >> one may map I/O-type registers (in my understanding, registers >> for which accesses cause side effects) within mem space. > > There isn't much difference between a memory BAR and an IO BAR. > Both are used for accesses to device registers. > There are subtle differences in the PCIe TLPs (I think io writes > get a completion TLP). > Memory space (maybe only 64bit address??) can be 'pre-fetchable' > but generally the driver maps everything uncachable. > > >> 2) On my platform, there are two revisions of the PCIe controller. >> Rev1 muxes config and mem inside a 256 MB window, and doesn't support >> I/O space. >> Rev2 muxes all 3 spaces inside a 256 MB window. > > Don't think config space fits. > With the 'obvious' mapping the 'bus number' is in the top > 8 bits of the address. > IIRC x86 uses two 32bit addresses for config space. > One is used to hold the 'address' for the cycle, the other > to perform the cycle. > >> Ard has stated that this model is not supported by Linux. >> AFAIU, the reason is that accesses may occur concurrently >> (especially on SMP systems). Thus tweaking a bit before >> the actual access necessarily creates a race condition. >> >> I wondered if there might be (reasonable) software >> work-arounds, in your experience? > > Remember some drivers let applications mmap PCIe addresses > directly into the user page tables. > So you have to stop absolutely everything if you change > your mux. > >> 3) What happens if a device requires more than 256 MB of >> mem space? (Is that common? What kind of device? GPUs?) >> Our controller supports a remapping "facility" to add an >> offset to the bus address. Is such a feature supported >> by Linux at all? The problem is that this creates >> another race condition, as setting the offset register >> before an access may occur concurrently on two cores. >> Perhaps 256 MB is plenty on a 32-bit embedded device? > > GPUs tend to have their own paging scheme. > So don't need humongous windows. > I'm not sure how much space is really needed. The server class (tesla/etc) GPU's use 64-bit BAR's for a while and in the case of something like the K80 require two 16G mappings per board. (see the comment at the bottom of this link) https://devtalk.nvidia.com/default/topic/865872/driver-installing-problem-for-nvidia-tesla-k80-under-linux/ > 32bit x86 reserve the top 1GB of physical address for PCI(e). > >> 4) The HW dev is considering the following fix. >> Instead of muxing the address spaces, provide smaller >> exclusive spaces. For example >> [0x5000_0000, 0x5400_0000] for config (64MB) >> [0x5400_0000, 0x5800_0000] for I/O (64MB) >> [0x5800_0000, 0x6000_0000] for mem (128MB) > > You almost certainly don't need more than 64k of IO. > >> That way, bits 26:27 implicitly select the address space >> 00 = config >> 01 = I/O >> 1x = mem >> >> This would be more in line with what Linux expects, right? >> Are these sizes acceptable? 64 MB config is probably overkill >> (we'll never have 64 devices on this board). 64 MB for I/O >> is probably plenty. The issue might be mem space? > > Config space isn't dense, you (probably) need 25 bits to get a 2nd bus number. > Even 256MB constrains you to 16 bus numbers. > > Is this an ARM cpu inside an altera (now intel) fpga?? > There is a nasty bug in their PCIe to avalon bridge logic (fixed in quartus 16.1). > > David > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel