* [RFC] PCIe bus address at 0x00000000 @ 2017-09-30 0:47 Ray Jui 2017-10-02 19:35 ` Bjorn Helgaas 0 siblings, 1 reply; 6+ messages in thread From: Ray Jui @ 2017-09-30 0:47 UTC (permalink / raw) To: Bjorn Helgaas Cc: linux-pci, Hari Vyas, Abhishek Shah, Srinath Mannam, Vikram Prakash Hi Bjorn and all, One of our ARM64 based processor (NS2) has our system address reversed for PCIe outbound transaction in the range of 0x0000_0000 and 0x1fff_ffff. If we use 1:1 mapping between the system address and PCIe address, we end up having the PCIe address in the same range. In this case, we noticed that certain USB-PCI adapters will decline access to BARs mapped in this address range with "Unsupported Request". In this case, if we change the PCIe address range to something else like 0x8000_0000, it would work without an issue. This sounds like an issue with the USB PCI adapter that cannot allow BARs to be used with specific PCIe address range. Is my understanding correct? Thanks, Ray ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] PCIe bus address at 0x00000000 2017-09-30 0:47 [RFC] PCIe bus address at 0x00000000 Ray Jui @ 2017-10-02 19:35 ` Bjorn Helgaas 2017-10-02 19:47 ` Ray Jui 0 siblings, 1 reply; 6+ messages in thread From: Bjorn Helgaas @ 2017-10-02 19:35 UTC (permalink / raw) To: Ray Jui Cc: Bjorn Helgaas, linux-pci, Hari Vyas, Abhishek Shah, Srinath Mannam, Vikram Prakash On Fri, Sep 29, 2017 at 05:47:23PM -0700, Ray Jui wrote: > Hi Bjorn and all, > > One of our ARM64 based processor (NS2) has our system address > reversed for PCIe outbound transaction in the range of 0x0000_0000 > and 0x1fff_ffff. If we use 1:1 mapping between the system address > and PCIe address, we end up having the PCIe address in the same > range. > > In this case, we noticed that certain USB-PCI adapters will decline > access to BARs mapped in this address range with "Unsupported > Request". In this case, if we change the PCIe address range to > something else like 0x8000_0000, it would work without an issue. > > This sounds like an issue with the USB PCI adapter that cannot allow > BARs to be used with specific PCIe address range. Is my > understanding correct? I assume you're seeing "Unsupported Request" completions on the PCIe link? Can you tell with an analyzer that the TLP actually appears on the link and it's the device that is responding with UR? I'm just wondering if there's something in the RC that could be short-circuiting this. But I guess you mention "certain USB-PCI adapters", which implies that these BAR addresses work for *some* devices, just not these particular USB-PCI ones. AFAIK, there's no PCIe reason why the 0x0000_0000 - 0x1fff_ffff region shouldn't be used for BARs. It's surprising to me that a device would special-case addresses like that, but maybe it does. Bjorn ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] PCIe bus address at 0x00000000 2017-10-02 19:35 ` Bjorn Helgaas @ 2017-10-02 19:47 ` Ray Jui 2017-10-02 20:08 ` Bjorn Helgaas 0 siblings, 1 reply; 6+ messages in thread From: Ray Jui @ 2017-10-02 19:47 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, linux-pci, Hari Vyas, Abhishek Shah, Srinath Mannam, Vikram Prakash [-- Attachment #1: Type: text/plain, Size: 2293 bytes --] Hi Bjorn, On 10/2/2017 12:35 PM, Bjorn Helgaas wrote: > On Fri, Sep 29, 2017 at 05:47:23PM -0700, Ray Jui wrote: >> Hi Bjorn and all, >> >> One of our ARM64 based processor (NS2) has our system address >> reversed for PCIe outbound transaction in the range of 0x0000_0000 >> and 0x1fff_ffff. If we use 1:1 mapping between the system address >> and PCIe address, we end up having the PCIe address in the same >> range. >> >> In this case, we noticed that certain USB-PCI adapters will decline >> access to BARs mapped in this address range with "Unsupported >> Request". In this case, if we change the PCIe address range to >> something else like 0x8000_0000, it would work without an issue. >> >> This sounds like an issue with the USB PCI adapter that cannot allow >> BARs to be used with specific PCIe address range. Is my >> understanding correct? > > I assume you're seeing "Unsupported Request" completions on the PCIe > link? Can you tell with an analyzer that the TLP actually appears on > the link and it's the device that is responding with UR? I'm just > wondering if there's something in the RC that could be > short-circuiting this. But I guess you mention "certain USB-PCI > adapters", which implies that these BAR addresses work for *some* > devices, just not these particular USB-PCI ones. Yes, UR is observed with a PCIe analyzer. See attached two JPGs for captures from PCIe analyzer (one uses PCIe address of 0x0000_0000 and the other uses 0x8000_0000). And yes, this behavior is only seen with certain PCIe-USB adapters. Most other PCIe endpoint devices (NVMe SSDs, Ethernet dongle, PCIe switches, and etc) work fine with PICe address of 0x0000_0000. > > AFAIK, there's no PCIe reason why the 0x0000_0000 - 0x1fff_ffff > region shouldn't be used for BARs. > Okay, thanks for helping to confirm that. > It's surprising to me that a device would special-case addresses like > that, but maybe it does. If this is proven to be a shortcoming of some specific PCIe-USB adapters (and in fact it seems to be the case so far based on all of the data points). Does it make any sense to you to have a workaround done by the root complex, i.e., to avoid using 0x0000_0000 outbound region; instead, use something else, .e.g, 0x1000_0000 > > Bjorn > Thanks, Ray [-- Attachment #2: pcie-capture-0x00000000.jpg --] [-- Type: image/jpeg, Size: 76300 bytes --] [-- Attachment #3: pcie-capture-0x80000000.jpg --] [-- Type: image/jpeg, Size: 83293 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] PCIe bus address at 0x00000000 2017-10-02 19:47 ` Ray Jui @ 2017-10-02 20:08 ` Bjorn Helgaas 2017-10-03 22:50 ` Ray Jui 0 siblings, 1 reply; 6+ messages in thread From: Bjorn Helgaas @ 2017-10-02 20:08 UTC (permalink / raw) To: Ray Jui Cc: Bjorn Helgaas, linux-pci, Hari Vyas, Abhishek Shah, Srinath Mannam, Vikram Prakash On Mon, Oct 02, 2017 at 12:47:16PM -0700, Ray Jui wrote: > Hi Bjorn, > > On 10/2/2017 12:35 PM, Bjorn Helgaas wrote: > >On Fri, Sep 29, 2017 at 05:47:23PM -0700, Ray Jui wrote: > >>Hi Bjorn and all, > >> > >>One of our ARM64 based processor (NS2) has our system address > >>reversed for PCIe outbound transaction in the range of 0x0000_0000 > >>and 0x1fff_ffff. If we use 1:1 mapping between the system address > >>and PCIe address, we end up having the PCIe address in the same > >>range. > >> > >>In this case, we noticed that certain USB-PCI adapters will decline > >>access to BARs mapped in this address range with "Unsupported > >>Request". In this case, if we change the PCIe address range to > >>something else like 0x8000_0000, it would work without an issue. > >> > >>This sounds like an issue with the USB PCI adapter that cannot allow > >>BARs to be used with specific PCIe address range. Is my > >>understanding correct? > > > >I assume you're seeing "Unsupported Request" completions on the PCIe > >link? Can you tell with an analyzer that the TLP actually appears on > >the link and it's the device that is responding with UR? I'm just > >wondering if there's something in the RC that could be > >short-circuiting this. But I guess you mention "certain USB-PCI > >adapters", which implies that these BAR addresses work for *some* > >devices, just not these particular USB-PCI ones. > > Yes, UR is observed with a PCIe analyzer. See attached two JPGs for > captures from PCIe analyzer (one uses PCIe address of 0x0000_0000 > and the other uses 0x8000_0000). > > And yes, this behavior is only seen with certain PCIe-USB adapters. > Most other PCIe endpoint devices (NVMe SSDs, Ethernet dongle, PCIe > switches, and etc) work fine with PICe address of 0x0000_0000. > > > > >AFAIK, there's no PCIe reason why the 0x0000_0000 - 0x1fff_ffff > >region shouldn't be used for BARs. > > > > Okay, thanks for helping to confirm that. > > >It's surprising to me that a device would special-case addresses like > >that, but maybe it does. > > If this is proven to be a shortcoming of some specific PCIe-USB > adapters (and in fact it seems to be the case so far based on all of > the data points). Does it make any sense to you to have a workaround > done by the root complex, i.e., to avoid using 0x0000_0000 outbound > region; instead, use something else, .e.g, 0x1000_0000 We could have a workaround, as long as it's done in a way that doesn't penalize all the correctly-working hardware. I don't want to give up 1/16 of the 32-bit address space for everybody just to accomodate some broken hardware. I'm not sure what form such a workaround would take. A quirk could mark a zero-valued BAR as invalid and force reallocation, but I'm not sure how we'd force the allocator to avoid this region. There might be some development required there. Bjorn ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] PCIe bus address at 0x00000000 2017-10-02 20:08 ` Bjorn Helgaas @ 2017-10-03 22:50 ` Ray Jui 2017-10-04 6:17 ` Hari Vyas 0 siblings, 1 reply; 6+ messages in thread From: Ray Jui @ 2017-10-03 22:50 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, linux-pci, Hari Vyas, Abhishek Shah, Srinath Mannam, Vikram Prakash Hi Bjorn, On 10/2/2017 1:08 PM, Bjorn Helgaas wrote: > On Mon, Oct 02, 2017 at 12:47:16PM -0700, Ray Jui wrote: >> Hi Bjorn, >> >> On 10/2/2017 12:35 PM, Bjorn Helgaas wrote: >>> On Fri, Sep 29, 2017 at 05:47:23PM -0700, Ray Jui wrote: >>>> Hi Bjorn and all, >>>> >>>> One of our ARM64 based processor (NS2) has our system address >>>> reversed for PCIe outbound transaction in the range of 0x0000_0000 >>>> and 0x1fff_ffff. If we use 1:1 mapping between the system address >>>> and PCIe address, we end up having the PCIe address in the same >>>> range. >>>> >>>> In this case, we noticed that certain USB-PCI adapters will decline >>>> access to BARs mapped in this address range with "Unsupported >>>> Request". In this case, if we change the PCIe address range to >>>> something else like 0x8000_0000, it would work without an issue. >>>> >>>> This sounds like an issue with the USB PCI adapter that cannot allow >>>> BARs to be used with specific PCIe address range. Is my >>>> understanding correct? >>> >>> I assume you're seeing "Unsupported Request" completions on the PCIe >>> link? Can you tell with an analyzer that the TLP actually appears on >>> the link and it's the device that is responding with UR? I'm just >>> wondering if there's something in the RC that could be >>> short-circuiting this. But I guess you mention "certain USB-PCI >>> adapters", which implies that these BAR addresses work for *some* >>> devices, just not these particular USB-PCI ones. >> >> Yes, UR is observed with a PCIe analyzer. See attached two JPGs for >> captures from PCIe analyzer (one uses PCIe address of 0x0000_0000 >> and the other uses 0x8000_0000). >> >> And yes, this behavior is only seen with certain PCIe-USB adapters. >> Most other PCIe endpoint devices (NVMe SSDs, Ethernet dongle, PCIe >> switches, and etc) work fine with PICe address of 0x0000_0000. >> >>> >>> AFAIK, there's no PCIe reason why the 0x0000_0000 - 0x1fff_ffff >>> region shouldn't be used for BARs. >>> >> >> Okay, thanks for helping to confirm that. >> >>> It's surprising to me that a device would special-case addresses like >>> that, but maybe it does. >> >> If this is proven to be a shortcoming of some specific PCIe-USB >> adapters (and in fact it seems to be the case so far based on all of >> the data points). Does it make any sense to you to have a workaround >> done by the root complex, i.e., to avoid using 0x0000_0000 outbound >> region; instead, use something else, .e.g, 0x1000_0000 > > We could have a workaround, as long as it's done in a way that doesn't > penalize all the correctly-working hardware. I don't want to give up > 1/16 of the 32-bit address space for everybody just to accomodate some > broken hardware. > > I'm not sure what form such a workaround would take. A quirk could > mark a zero-valued BAR as invalid and force reallocation, but I'm not > sure how we'd force the allocator to avoid this region. There might > be some development required there. Giving that this is obviously a shortcoming from some specific endpoint devices, I would image enforcing certain region of PCIe address space from the root complex side would not make sense, right? I'd assume this needs to be done through some API that an endpoint device can call and mark specific PCIe address invalid? > > Bjorn > ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [RFC] PCIe bus address at 0x00000000 2017-10-03 22:50 ` Ray Jui @ 2017-10-04 6:17 ` Hari Vyas 0 siblings, 0 replies; 6+ messages in thread From: Hari Vyas @ 2017-10-04 6:17 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, linux-pci, Abhishek Shah, Srinath Mannam, Vikram Prakash, Ray Jui Hi Bjorn, -----Original Message----- From: Ray Jui [mailto:ray.jui@broadcom.com] Sent: 04 October 2017 04:21 To: Bjorn Helgaas Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; Hari Vyas; Abhishek Shah; Srinath Mannam; Vikram Prakash Subject: Re: [RFC] PCIe bus address at 0x00000000 Hi Bjorn, On 10/2/2017 1:08 PM, Bjorn Helgaas wrote: > On Mon, Oct 02, 2017 at 12:47:16PM -0700, Ray Jui wrote: >> Hi Bjorn, >> >> On 10/2/2017 12:35 PM, Bjorn Helgaas wrote: >>> On Fri, Sep 29, 2017 at 05:47:23PM -0700, Ray Jui wrote: >>>> Hi Bjorn and all, >>>> >>>> One of our ARM64 based processor (NS2) has our system address >>>> reversed for PCIe outbound transaction in the range of 0x0000_0000 >>>> and 0x1fff_ffff. If we use 1:1 mapping between the system address >>>> and PCIe address, we end up having the PCIe address in the same >>>> range. >>>> >>>> In this case, we noticed that certain USB-PCI adapters will decline >>>> access to BARs mapped in this address range with "Unsupported >>>> Request". In this case, if we change the PCIe address range to >>>> something else like 0x8000_0000, it would work without an issue. >>>> >>>> This sounds like an issue with the USB PCI adapter that cannot >>>> allow BARs to be used with specific PCIe address range. Is my >>>> understanding correct? >>> >>> I assume you're seeing "Unsupported Request" completions on the PCIe >>> link? Can you tell with an analyzer that the TLP actually appears >>> on the link and it's the device that is responding with UR? I'm >>> just wondering if there's something in the RC that could be >>> short-circuiting this. But I guess you mention "certain USB-PCI >>> adapters", which implies that these BAR addresses work for *some* >>> devices, just not these particular USB-PCI ones. >> >> Yes, UR is observed with a PCIe analyzer. See attached two JPGs for >> captures from PCIe analyzer (one uses PCIe address of 0x0000_0000 and >> the other uses 0x8000_0000). >> >> And yes, this behavior is only seen with certain PCIe-USB adapters. >> Most other PCIe endpoint devices (NVMe SSDs, Ethernet dongle, PCIe >> switches, and etc) work fine with PICe address of 0x0000_0000. >> >>> >>> AFAIK, there's no PCIe reason why the 0x0000_0000 - 0x1fff_ffff >>> region shouldn't be used for BARs. >>> >> >> Okay, thanks for helping to confirm that. >> >>> It's surprising to me that a device would special-case addresses >>> like that, but maybe it does. >> >> If this is proven to be a shortcoming of some specific PCIe-USB >> adapters (and in fact it seems to be the case so far based on all of >> the data points). Does it make any sense to you to have a workaround >> done by the root complex, i.e., to avoid using 0x0000_0000 outbound >> region; instead, use something else, .e.g, 0x1000_0000 > > We could have a workaround, as long as it's done in a way that doesn't > penalize all the correctly-working hardware. I don't want to give up > 1/16 of the 32-bit address space for everybody just to accomodate some > broken hardware. > > I'm not sure what form such a workaround would take. A quirk could > mark a zero-valued BAR as invalid and force reallocation, but I'm not > sure how we'd force the allocator to avoid this region. There might > be some development required there. Giving that this is obviously a shortcoming from some specific endpoint devices, I would image enforcing certain region of PCIe address space from the root complex side would not make sense, right? I'd assume this needs to be done through some API that an endpoint device can call and mark specific PCIe address invalid? ----------------------------------------------------------------- I guess, we may think about adding some special quirk to handle this. Anyway, just to be more clear, I am mentioning that specific PCIE-USB adapter for which we are facing issues. 0000:01:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) 00: 33 10 94 01 06 04 10 00 04 30 03 0c 20 00 00 00 10: 04 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 4d 14 a5 c0 30: 00 00 00 00 50 00 00 00 00 00 00 00 82 01 00 00 > > Bjorn > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-10-04 6:17 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-09-30 0:47 [RFC] PCIe bus address at 0x00000000 Ray Jui 2017-10-02 19:35 ` Bjorn Helgaas 2017-10-02 19:47 ` Ray Jui 2017-10-02 20:08 ` Bjorn Helgaas 2017-10-03 22:50 ` Ray Jui 2017-10-04 6:17 ` Hari Vyas
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.
![[-- Attachment #2: pcie-capture-0x00000000.jpg --] [-- Type: image/jpeg, Size: 76300 bytes --]](../../e0a89ffe-a936-9b41-c9c0-3616452376e2@broadcom.com/2-pcie-capture-0x00000000.jpg){kind=link}
![[-- Attachment #3: pcie-capture-0x80000000.jpg --] [-- Type: image/jpeg, Size: 83293 bytes --]](../../e0a89ffe-a936-9b41-c9c0-3616452376e2@broadcom.com/3-pcie-capture-0x80000000.jpg){kind=link}