* [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses @ 2022-08-25 23:54 Will McVicker 2022-08-25 23:54 ` [PATCH v6 1/2] PCI: dwc: Drop dependency on ZONE_DMA32 Will McVicker ` (3 more replies) 0 siblings, 4 replies; 12+ messages in thread From: Will McVicker @ 2022-08-25 23:54 UTC (permalink / raw) To: Jingoo Han, Gustavo Pimentel, Lorenzo Pieralisi, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, Will McVicker Cc: kernel-team, Vidya Sagar, Christoph Hellwig, Robin Murphy, linux-pci, linux-kernel Hi All, I've update patch 2/2 to address Robin's suggestions. This includes: * Dropping the while-loop for retrying with a 64-bit mask in favor of retrying within the error if-statement. * Using an int for the DMA mask instead of a bool and ternary operation. Thanks again for the reviews and sorry for the extra revision today! Hopefully this is the last one :) If not, I'd be fine to submit patch 1/2 without 2/2 to avoid resending patch 1/2 for future revisions of patch 2/2 (unless I don't need to do that anyway). Thanks, Will Will McVicker (2): PCI: dwc: Drop dependency on ZONE_DMA32 v6: * Retrying DMA allocation with 64-bit mask within the error if-statement. * Use an int for the DMA mask instead of a bool and ternary operation. v5: * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, retry with a 64-bit mask if supported. v4: * Updated commit descriptions. * Renamed msi_64b -> msi_64bit. * Dropped msi_64bit ternary use. * Dropped export of dw_pcie_msi_capabilities. v3: * Switched to a managed DMA allocation. * Simplified the DMA allocation cleanup. * Dropped msi_page from struct dw_pcie_rp. * Allocating a u64 instead of a full page. v2: * Fixed build error caught by kernel test robot * Fixed error handling reported by Isaac Manjarres PCI: dwc: Add support for 64-bit MSI target address .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ drivers/pci/controller/dwc/pcie-designware.h | 2 +- 3 files changed, 30 insertions(+), 23 deletions(-) base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 -- 2.37.2.672.g94769d06f0-goog ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 1/2] PCI: dwc: Drop dependency on ZONE_DMA32 2022-08-25 23:54 [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Will McVicker @ 2022-08-25 23:54 ` Will McVicker 2022-08-25 23:54 ` [PATCH v6 2/2] PCI: dwc: Add support for 64-bit MSI target address Will McVicker ` (2 subsequent siblings) 3 siblings, 0 replies; 12+ messages in thread From: Will McVicker @ 2022-08-25 23:54 UTC (permalink / raw) To: Jingoo Han, Gustavo Pimentel, Lorenzo Pieralisi, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, Will McVicker Cc: kernel-team, Vidya Sagar, Christoph Hellwig, Robin Murphy, linux-pci, linux-kernel, Isaac J . Manjarres Re-work the msi_msg DMA allocation logic to use dmam_alloc_coherent() which uses the coherent DMA mask to try to return an allocation within the DMA mask limits. With that, we now can drop the msi_page parameter in struct dw_pcie_rp. This allows kernel configurations that disable ZONE_DMA32 to continue supporting a 32-bit DMA mask. Without this patch, the PCIe host device will fail to probe when ZONE_DMA32 is disabled. Fixes: 35797e672ff0 ("PCI: dwc: Fix MSI msi_msg DMA mapping") Reported-by: Isaac J. Manjarres <isaacmanjarres@google.com> Signed-off-by: Will McVicker <willmcvicker@google.com> Acked-by: Jingoo Han <jingoohan1@gmail.com> Reviewed-by: Rob Herring <robh@kernel.org> --- .../pci/controller/dwc/pcie-designware-host.c | 28 +++++-------------- drivers/pci/controller/dwc/pcie-designware.h | 1 - 2 files changed, 7 insertions(+), 22 deletions(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index 7746f94a715f..39f3b37d4033 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -267,15 +267,6 @@ static void dw_pcie_free_msi(struct dw_pcie_rp *pp) irq_domain_remove(pp->msi_domain); irq_domain_remove(pp->irq_domain); - - if (pp->msi_data) { - struct dw_pcie *pci = to_dw_pcie_from_pp(pp); - struct device *dev = pci->dev; - - dma_unmap_page(dev, pp->msi_data, PAGE_SIZE, DMA_FROM_DEVICE); - if (pp->msi_page) - __free_page(pp->msi_page); - } } static void dw_pcie_msi_init(struct dw_pcie_rp *pp) @@ -336,6 +327,7 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) struct dw_pcie *pci = to_dw_pcie_from_pp(pp); struct device *dev = pci->dev; struct platform_device *pdev = to_platform_device(dev); + u64 *msi_vaddr; int ret; u32 ctrl, num_ctrls; @@ -375,22 +367,16 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) dw_chained_msi_isr, pp); } - ret = dma_set_mask(dev, DMA_BIT_MASK(32)); + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); if (ret) dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n"); - pp->msi_page = alloc_page(GFP_DMA32); - pp->msi_data = dma_map_page(dev, pp->msi_page, 0, - PAGE_SIZE, DMA_FROM_DEVICE); - ret = dma_mapping_error(dev, pp->msi_data); - if (ret) { - dev_err(pci->dev, "Failed to map MSI data\n"); - __free_page(pp->msi_page); - pp->msi_page = NULL; - pp->msi_data = 0; + msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, + GFP_KERNEL); + if (!msi_vaddr) { + dev_err(dev, "Failed to alloc and map MSI data\n"); dw_pcie_free_msi(pp); - - return ret; + return -ENOMEM; } return 0; diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h index 09b887093a84..a871ae7eb59e 100644 --- a/drivers/pci/controller/dwc/pcie-designware.h +++ b/drivers/pci/controller/dwc/pcie-designware.h @@ -243,7 +243,6 @@ struct dw_pcie_rp { struct irq_domain *irq_domain; struct irq_domain *msi_domain; dma_addr_t msi_data; - struct page *msi_page; struct irq_chip *msi_irq_chip; u32 num_vectors; u32 irq_mask[MAX_MSI_CTRLS]; -- 2.37.2.672.g94769d06f0-goog ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v6 2/2] PCI: dwc: Add support for 64-bit MSI target address 2022-08-25 23:54 [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Will McVicker 2022-08-25 23:54 ` [PATCH v6 1/2] PCI: dwc: Drop dependency on ZONE_DMA32 Will McVicker @ 2022-08-25 23:54 ` Will McVicker 2022-08-29 8:03 ` [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Lorenzo Pieralisi 2023-01-31 12:29 ` Evgenii Shatokhin 3 siblings, 0 replies; 12+ messages in thread From: Will McVicker @ 2022-08-25 23:54 UTC (permalink / raw) To: Jingoo Han, Gustavo Pimentel, Lorenzo Pieralisi, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, Will McVicker Cc: kernel-team, Vidya Sagar, Christoph Hellwig, Robin Murphy, linux-pci, linux-kernel, kernel test robot Since not all devices require a 32-bit MSI address, add support to the PCIe host driver to allow setting the DMA mask to 64-bits if the 32-bit allocation fails. This allows kernels to disable ZONE_DMA32 and bounce buffering (swiotlb) without risking not being able to get a 32-bit address during DMA allocation. Basically, in the slim chance that there are no 32-bit allocations available, the current PCIe host driver will fail to allocate the msi_msg page due to a DMA address overflow (seen in [1]). With this patch, the PCIe host can retry the allocation with a 64-bit DMA mask if the current PCIe device advertises 64-bit support via its MSI capabilities. [1] https://lore.kernel.org/all/Yo0soniFborDl7+C@google.com/ Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Will McVicker <willmcvicker@google.com> Reviewed-by: Rob Herring <robh@kernel.org> Acked-by: Jingoo Han <jingoohan1@gmail.com> --- .../pci/controller/dwc/pcie-designware-host.c | 19 ++++++++++++++++--- drivers/pci/controller/dwc/pcie-designware.c | 8 ++++++++ drivers/pci/controller/dwc/pcie-designware.h | 1 + 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index 39f3b37d4033..7e0352861bcb 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -374,9 +374,22 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), &pp->msi_data, GFP_KERNEL); if (!msi_vaddr) { - dev_err(dev, "Failed to alloc and map MSI data\n"); - dw_pcie_free_msi(pp); - return -ENOMEM; + u16 msi_capabilities; + + /* Retry the allocation with a 64-bit mask if supported. */ + msi_capabilities = dw_pcie_msi_capabilities(pci); + if ((msi_capabilities & PCI_MSI_FLAGS_ENABLE) && + (msi_capabilities & PCI_MSI_FLAGS_64BIT)) { + dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); + msi_vaddr = dmam_alloc_coherent(dev, sizeof(u64), + &pp->msi_data, + GFP_KERNEL); + } + if (!msi_vaddr) { + dev_err(dev, "Failed to alloc and map MSI data\n"); + dw_pcie_free_msi(pp); + return -ENOMEM; + } } return 0; diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c index c6725c519a47..650a7f22f9d0 100644 --- a/drivers/pci/controller/dwc/pcie-designware.c +++ b/drivers/pci/controller/dwc/pcie-designware.c @@ -82,6 +82,14 @@ u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap) } EXPORT_SYMBOL_GPL(dw_pcie_find_capability); +u16 dw_pcie_msi_capabilities(struct dw_pcie *pci) +{ + u8 offset; + + offset = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI); + return dw_pcie_readw_dbi(pci, offset + PCI_MSI_FLAGS); +} + static u16 dw_pcie_find_next_ext_capability(struct dw_pcie *pci, u16 start, u8 cap) { diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h index a871ae7eb59e..45fcdfc8c035 100644 --- a/drivers/pci/controller/dwc/pcie-designware.h +++ b/drivers/pci/controller/dwc/pcie-designware.h @@ -332,6 +332,7 @@ void dw_pcie_version_detect(struct dw_pcie *pci); u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap); u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap); +u16 dw_pcie_msi_capabilities(struct dw_pcie *pci); int dw_pcie_read(void __iomem *addr, int size, u32 *val); int dw_pcie_write(void __iomem *addr, int size, u32 val); -- 2.37.2.672.g94769d06f0-goog ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2022-08-25 23:54 [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Will McVicker 2022-08-25 23:54 ` [PATCH v6 1/2] PCI: dwc: Drop dependency on ZONE_DMA32 Will McVicker 2022-08-25 23:54 ` [PATCH v6 2/2] PCI: dwc: Add support for 64-bit MSI target address Will McVicker @ 2022-08-29 8:03 ` Lorenzo Pieralisi 2023-01-31 12:29 ` Evgenii Shatokhin 3 siblings, 0 replies; 12+ messages in thread From: Lorenzo Pieralisi @ 2022-08-29 8:03 UTC (permalink / raw) To: Krzysztof Wilczyński, Will McVicker, Rob Herring, Jingoo Han, Bjorn Helgaas, Gustavo Pimentel Cc: Lorenzo Pieralisi, Robin Murphy, kernel-team, Vidya Sagar, linux-kernel, Christoph Hellwig, linux-pci On Thu, 25 Aug 2022 23:54:01 +0000, Will McVicker wrote: > I've update patch 2/2 to address Robin's suggestions. This includes: > > * Dropping the while-loop for retrying with a 64-bit mask in favor of > retrying within the error if-statement. > * Using an int for the DMA mask instead of a bool and ternary operation. > > Thanks again for the reviews and sorry for the extra revision today! > Hopefully this is the last one :) If not, I'd be fine to submit patch 1/2 > without 2/2 to avoid resending patch 1/2 for future revisions of patch 2/2 > (unless I don't need to do that anyway). > > [...] Applied to pci/dwc, thanks! [1/2] PCI: dwc: Drop dependency on ZONE_DMA32 https://git.kernel.org/lpieralisi/pci/c/423511ec23e2 [2/2] PCI: dwc: Add support for 64-bit MSI target address https://git.kernel.org/lpieralisi/pci/c/e99d8c5e803b Thanks, Lorenzo ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2022-08-25 23:54 [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Will McVicker ` (2 preceding siblings ...) 2022-08-29 8:03 ` [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Lorenzo Pieralisi @ 2023-01-31 12:29 ` Evgenii Shatokhin 2023-01-31 12:42 ` Robin Murphy 3 siblings, 1 reply; 12+ messages in thread From: Evgenii Shatokhin @ 2023-01-31 12:29 UTC (permalink / raw) To: Will McVicker, Lorenzo Pieralisi Cc: kernel-team, Vidya Sagar, Christoph Hellwig, Robin Murphy, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux Hi, On 26.08.2022 02:54, Will McVicker wrote: > Hi All, > > I've update patch 2/2 to address Robin's suggestions. This includes: > > * Dropping the while-loop for retrying with a 64-bit mask in favor of > retrying within the error if-statement. > * Using an int for the DMA mask instead of a bool and ternary operation. > > Thanks again for the reviews and sorry for the extra revision today! > Hopefully this is the last one :) If not, I'd be fine to submit patch 1/2 > without 2/2 to avoid resending patch 1/2 for future revisions of patch 2/2 > (unless I don't need to do that anyway). The first patch of the series made it into the mainline kernel, but, it seems, the second one ("PCI: dwc: Add support for 64-bit MSI target address") did not. As of 6.2-rc6, it is still missing. Was it intentionally dropped because of some issues or, perhaps, just by accident? If it was by accident, could you please queue it for inclusion into mainline again? Support for 64-bit MSI target addresses is needed for some of our SoCs. I ran into a situation when there was no available RAM in ZONE_DMA32 during initialization of PCIe host. Hence, dmam_alloc_coherent() failed in dw_pcie_msi_host_init() and initialization failed with -ENOMEM: [ 0.374834] dw-pcie 4000000.pcie0: host bridge /soc/pcie0@4000000 ranges: [ 0.375813] dw-pcie 4000000.pcie0: MEM 0x0041000000..0x004fffffff -> 0x0041000000 [ 0.376171] dw-pcie 4000000.pcie0: IB MEM 0x0400000000..0x07ffffffff -> 0x0400000000 [ 0.377914] dw-pcie 4000000.pcie0: Failed to alloc and map MSI data [ 0.378191] dw-pcie 4000000.pcie0: Failed to initialize host [ 0.378255] dw-pcie: probe of 4000000.pcie0 failed with error -12 Mainline kernel 6.2-rc6 was used in that test. The hardware supports 64-bit target addresses, so the patch "PCI: dwc: Add support for 64-bit MSI target address" should help with this particular failure. > > Thanks, > Will > > Will McVicker (2): > PCI: dwc: Drop dependency on ZONE_DMA32 > > v6: > * Retrying DMA allocation with 64-bit mask within the error if-statement. > * Use an int for the DMA mask instead of a bool and ternary operation. > > v5: > * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, > retry with a 64-bit mask if supported. > > v4: > * Updated commit descriptions. > * Renamed msi_64b -> msi_64bit. > * Dropped msi_64bit ternary use. > * Dropped export of dw_pcie_msi_capabilities. > > v3: > * Switched to a managed DMA allocation. > * Simplified the DMA allocation cleanup. > * Dropped msi_page from struct dw_pcie_rp. > * Allocating a u64 instead of a full page. > > v2: > * Fixed build error caught by kernel test robot > * Fixed error handling reported by Isaac Manjarres > PCI: dwc: Add support for 64-bit MSI target address > > .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- > drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ > drivers/pci/controller/dwc/pcie-designware.h | 2 +- > 3 files changed, 30 insertions(+), 23 deletions(-) > > > base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 Thank you in advance. Regards, Evgenii ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2023-01-31 12:29 ` Evgenii Shatokhin @ 2023-01-31 12:42 ` Robin Murphy 2023-02-01 13:54 ` Evgenii Shatokhin 0 siblings, 1 reply; 12+ messages in thread From: Robin Murphy @ 2023-01-31 12:42 UTC (permalink / raw) To: Evgenii Shatokhin, Will McVicker, Lorenzo Pieralisi Cc: kernel-team, Vidya Sagar, Christoph Hellwig, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux On 2023-01-31 12:29, Evgenii Shatokhin wrote: > Hi, > > On 26.08.2022 02:54, Will McVicker wrote: >> Hi All, >> >> I've update patch 2/2 to address Robin's suggestions. This includes: >> >> * Dropping the while-loop for retrying with a 64-bit mask in favor of >> retrying within the error if-statement. >> * Using an int for the DMA mask instead of a bool and ternary >> operation. >> >> Thanks again for the reviews and sorry for the extra revision today! >> Hopefully this is the last one :) If not, I'd be fine to submit patch 1/2 >> without 2/2 to avoid resending patch 1/2 for future revisions of patch >> 2/2 >> (unless I don't need to do that anyway). > > The first patch of the series made it into the mainline kernel, but, it > seems, the second one ("PCI: dwc: Add support for 64-bit MSI target > address") did not. As of 6.2-rc6, it is still missing. > > Was it intentionally dropped because of some issues or, perhaps, just by > accident? If it was by accident, could you please queue it for inclusion > into mainline again? Yes, it was dropped due to the PCI_MSI_FLAGS_64BIT usage apparently being incorrect, and some other open debate (which all happened on the v5 thread): https://lore.kernel.org/linux-pci/YzVTmy9MWh+AjshC@lpieralisi/ The DMA mask issues have now been sorted out, so you, or Will, or anyone else interested should be free to rework this on top of linux-next (although at this point, more realistically on top of 6.3-rc1 in a few weeks). Thanks, Robin. > Support for 64-bit MSI target addresses is needed for some of our SoCs. > I ran into a situation when there was no available RAM in ZONE_DMA32 > during initialization of PCIe host. Hence, dmam_alloc_coherent() failed > in dw_pcie_msi_host_init() and initialization failed with -ENOMEM: > > [ 0.374834] dw-pcie 4000000.pcie0: host bridge /soc/pcie0@4000000 > ranges: > [ 0.375813] dw-pcie 4000000.pcie0: MEM > 0x0041000000..0x004fffffff -> 0x0041000000 > [ 0.376171] dw-pcie 4000000.pcie0: IB MEM > 0x0400000000..0x07ffffffff -> 0x0400000000 > [ 0.377914] dw-pcie 4000000.pcie0: Failed to alloc and map MSI data > [ 0.378191] dw-pcie 4000000.pcie0: Failed to initialize host > [ 0.378255] dw-pcie: probe of 4000000.pcie0 failed with error -12 > > Mainline kernel 6.2-rc6 was used in that test. > > The hardware supports 64-bit target addresses, so the patch "PCI: dwc: > Add support for 64-bit MSI target address" should help with this > particular failure. > > >> >> Thanks, >> Will >> >> Will McVicker (2): >> PCI: dwc: Drop dependency on ZONE_DMA32 >> >> v6: >> * Retrying DMA allocation with 64-bit mask within the error >> if-statement. >> * Use an int for the DMA mask instead of a bool and ternary operation. >> >> v5: >> * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, >> retry with a 64-bit mask if supported. >> >> v4: >> * Updated commit descriptions. >> * Renamed msi_64b -> msi_64bit. >> * Dropped msi_64bit ternary use. >> * Dropped export of dw_pcie_msi_capabilities. >> >> v3: >> * Switched to a managed DMA allocation. >> * Simplified the DMA allocation cleanup. >> * Dropped msi_page from struct dw_pcie_rp. >> * Allocating a u64 instead of a full page. >> >> v2: >> * Fixed build error caught by kernel test robot >> * Fixed error handling reported by Isaac Manjarres >> PCI: dwc: Add support for 64-bit MSI target address >> >> .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- >> drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ >> drivers/pci/controller/dwc/pcie-designware.h | 2 +- >> 3 files changed, 30 insertions(+), 23 deletions(-) >> >> >> base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 > > Thank you in advance. > > Regards, > Evgenii > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2023-01-31 12:42 ` Robin Murphy @ 2023-02-01 13:54 ` Evgenii Shatokhin 2023-02-03 22:12 ` Serge Semin 0 siblings, 1 reply; 12+ messages in thread From: Evgenii Shatokhin @ 2023-02-01 13:54 UTC (permalink / raw) To: Robin Murphy Cc: kernel-team, Vidya Sagar, Christoph Hellwig, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux, Lorenzo Pieralisi, Will McVicker, Serge Semin On 31.01.2023 15:42, Robin Murphy wrote: > > On 2023-01-31 12:29, Evgenii Shatokhin wrote: >> Hi, >> >> On 26.08.2022 02:54, Will McVicker wrote: >>> Hi All, >>> >>> I've update patch 2/2 to address Robin's suggestions. This includes: >>> >>> * Dropping the while-loop for retrying with a 64-bit mask in favor of >>> retrying within the error if-statement. >>> * Using an int for the DMA mask instead of a bool and ternary >>> operation. >>> >>> Thanks again for the reviews and sorry for the extra revision today! >>> Hopefully this is the last one :) If not, I'd be fine to submit patch >>> 1/2 >>> without 2/2 to avoid resending patch 1/2 for future revisions of patch >>> 2/2 >>> (unless I don't need to do that anyway). >> >> The first patch of the series made it into the mainline kernel, but, it >> seems, the second one ("PCI: dwc: Add support for 64-bit MSI target >> address") did not. As of 6.2-rc6, it is still missing. >> >> Was it intentionally dropped because of some issues or, perhaps, just by >> accident? If it was by accident, could you please queue it for inclusion >> into mainline again? > > Yes, it was dropped due to the PCI_MSI_FLAGS_64BIT usage apparently > being incorrect, and some other open debate (which all happened on the > v5 thread): > > https://lore.kernel.org/linux-pci/YzVTmy9MWh+AjshC@lpieralisi/ I see. If I understand it correctly, the problem was that PCI_MSI_FLAGS_64BIT flag did not guarantee that 64-bit mask could be used for that particular allocation. Right? > > The DMA mask issues have now been sorted out, I suppose, you mean https://lore.kernel.org/all/20230113171409.30470-26-Sergey.Semin@baikalelectronics.ru/? It still breaks our particular case when the SoC has no 32-bit-addressable RAM. We'd set DMA masks to DMA_BIT_MASK(36) in the platform-specific driver before calling dw_pcie_host_init(). However, dw_pcie_msi_host_init() resets it to 32-bit, tries dmam_alloc_coherent() and fails. With 36-bit masks, the kernel seems to play well with the devices in our case. I saw your comment in https://lore.kernel.org/linux-pci/4dc31a63-00b1-f379-c5ac-7dc9425937f4@arm.com/ that drivers should always explicitly set their masks. Is it a really bad idea to check the current coherent mask's bits in dw_pcie_msi_host_init() and if it is more than 32 - just issue a warning rather than reset it to 32-bit unconditionally? That would help in our case. Or, perhaps, there is a better workaround. Looking forward to your comments. > so you, or Will, or anyone > else interested should be free to rework this on top of linux-next > (although at this point, more realistically on top of 6.3-rc1 in a few > weeks). > > Thanks, > Robin. > >> Support for 64-bit MSI target addresses is needed for some of our SoCs. >> I ran into a situation when there was no available RAM in ZONE_DMA32 >> during initialization of PCIe host. Hence, dmam_alloc_coherent() failed >> in dw_pcie_msi_host_init() and initialization failed with -ENOMEM: >> >> [ 0.374834] dw-pcie 4000000.pcie0: host bridge /soc/pcie0@4000000 >> ranges: >> [ 0.375813] dw-pcie 4000000.pcie0: MEM >> 0x0041000000..0x004fffffff -> 0x0041000000 >> [ 0.376171] dw-pcie 4000000.pcie0: IB MEM >> 0x0400000000..0x07ffffffff -> 0x0400000000 >> [ 0.377914] dw-pcie 4000000.pcie0: Failed to alloc and map MSI data >> [ 0.378191] dw-pcie 4000000.pcie0: Failed to initialize host >> [ 0.378255] dw-pcie: probe of 4000000.pcie0 failed with error -12 >> >> Mainline kernel 6.2-rc6 was used in that test. >> >> The hardware supports 64-bit target addresses, so the patch "PCI: dwc: >> Add support for 64-bit MSI target address" should help with this >> particular failure. >> >> >>> >>> Thanks, >>> Will >>> >>> Will McVicker (2): >>> PCI: dwc: Drop dependency on ZONE_DMA32 >>> >>> v6: >>> * Retrying DMA allocation with 64-bit mask within the error >>> if-statement. >>> * Use an int for the DMA mask instead of a bool and ternary operation. >>> >>> v5: >>> * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, >>> retry with a 64-bit mask if supported. >>> >>> v4: >>> * Updated commit descriptions. >>> * Renamed msi_64b -> msi_64bit. >>> * Dropped msi_64bit ternary use. >>> * Dropped export of dw_pcie_msi_capabilities. >>> >>> v3: >>> * Switched to a managed DMA allocation. >>> * Simplified the DMA allocation cleanup. >>> * Dropped msi_page from struct dw_pcie_rp. >>> * Allocating a u64 instead of a full page. >>> >>> v2: >>> * Fixed build error caught by kernel test robot >>> * Fixed error handling reported by Isaac Manjarres >>> PCI: dwc: Add support for 64-bit MSI target address >>> >>> .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- >>> drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ >>> drivers/pci/controller/dwc/pcie-designware.h | 2 +- >>> 3 files changed, 30 insertions(+), 23 deletions(-) >>> >>> >>> base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 >> >> Thank you in advance. >> >> Regards, >> Evgenii >> >> >> > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2023-02-01 13:54 ` Evgenii Shatokhin @ 2023-02-03 22:12 ` Serge Semin 2023-02-06 11:27 ` Evgenii Shatokhin 0 siblings, 1 reply; 12+ messages in thread From: Serge Semin @ 2023-02-03 22:12 UTC (permalink / raw) To: Evgenii Shatokhin Cc: Serge Semin, Robin Murphy, kernel-team, Vidya Sagar, Christoph Hellwig, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux, Lorenzo Pieralisi, Will McVicker Hi Evgenii On Wed, Feb 01, 2023 at 04:54:55PM +0300, Evgenii Shatokhin wrote: > On 31.01.2023 15:42, Robin Murphy wrote: > > > > On 2023-01-31 12:29, Evgenii Shatokhin wrote: > > > Hi, > > > > > > On 26.08.2022 02:54, Will McVicker wrote: > > > > Hi All, > > > > > > > > I've update patch 2/2 to address Robin's suggestions. This includes: > > > > > > > > * Dropping the while-loop for retrying with a 64-bit mask in favor of > > > > retrying within the error if-statement. > > > > * Using an int for the DMA mask instead of a bool and ternary > > > > operation. > > > > > > > > Thanks again for the reviews and sorry for the extra revision today! > > > > Hopefully this is the last one :) If not, I'd be fine to submit > > > > patch 1/2 > > > > without 2/2 to avoid resending patch 1/2 for future revisions of patch > > > > 2/2 > > > > (unless I don't need to do that anyway). > > > > > > The first patch of the series made it into the mainline kernel, but, it > > > seems, the second one ("PCI: dwc: Add support for 64-bit MSI target > > > address") did not. As of 6.2-rc6, it is still missing. > > > > > > Was it intentionally dropped because of some issues or, perhaps, just by > > > accident? If it was by accident, could you please queue it for inclusion > > > into mainline again? > > > > Yes, it was dropped due to the PCI_MSI_FLAGS_64BIT usage apparently > > being incorrect, and some other open debate (which all happened on the > > v5 thread): > > > > https://lore.kernel.org/linux-pci/YzVTmy9MWh+AjshC@lpieralisi/ > > I see. If I understand it correctly, the problem was that > PCI_MSI_FLAGS_64BIT flag did not guarantee that 64-bit mask could be used > for that particular allocation. Right? > William was trying to utilize for only software cause. Setting PCI_MSI_FLAGS_64BIT didn't actually change the hardware behavior. He could have as well provided just a driver private capability flag. (see below for a more detailed problem description) > > > > The DMA mask issues have now been sorted out, > > I suppose, you mean https://lore.kernel.org/all/20230113171409.30470-26-Sergey.Semin@baikalelectronics.ru/? Well, the way the DMA-mask issue has been solved was a bit of the hacky. I wouldn't call it a fully proper solution. The problem with pointlessly allocating physical memory for the iMSI-RX engine (it doesn't perform any DMA) and artificially restricting the coherent-DMA mask is still there. The patch in the subject was a compromise in order to at least permit unrestricted streaming DMAs but limiting the coherent DMAs for the MSI setup to work properly for all peripheral devices. > > It still breaks our particular case when the SoC has no 32-bit-addressable > RAM. We'd set DMA masks to DMA_BIT_MASK(36) in the platform-specific driver > before calling dw_pcie_host_init(). However, dw_pcie_msi_host_init() resets > it to 32-bit, tries dmam_alloc_coherent() and fails. Yeah. That's another problem with the implemented approach. But are your sure the driver had worked even before this patch? AFAICS the driver allocated the MSI-targeted page from DMA32 zone before this modification. So the allocation must have failed on your platform too. > > With 36-bit masks, the kernel seems to play well with the devices in our > case. > > I saw your comment in https://lore.kernel.org/linux-pci/4dc31a63-00b1-f379-c5ac-7dc9425937f4@arm.com/ > that drivers should always explicitly set their masks. > > Is it a really bad idea to check the current coherent mask's bits in > dw_pcie_msi_host_init() and if it is more than 32 - just issue a warning > rather than reset it to 32-bit unconditionally? That would help in our case. > Or, perhaps, there is a better workaround. The problem isn't in the value the mask is set to. The problem is two-leveled, but is mainly connected with the PCIe device detected on the PCIe bus. There are some of them which can't send MSI TLPs to the 64-bit addresses. Since we can't predict whether such devices exist on the bus beforehand the LLDD probe is performed together with the MSI-engine initialization, the solution was to just restrict the MSIs base address to be allocated within the lowest 4GB. Moreover as I said above the iMSI-RX engine doesn't actually cause any DMA thus there is no need in any memory allocation. Instead reserving some PCIe-bus space/DWORD for MSIs would be enough. Alas the PCIe-subsystem doesn't provide a way to do so. That's why we have what you see in the driver: DMA mask restriction and coherent DMA memory allocation. If only we had a way to auto-detected the PCIe-bus space with no physical memory behind it and take out a DWORD from it to initialize the iMSI-RX engine we could have immediately got rid from the mask setting operation and the memory allocation. It would have solved your problem too. -Serge(y) > > Looking forward to your comments. > > > > so you, or Will, or anyone > > else interested should be free to rework this on top of linux-next > > (although at this point, more realistically on top of 6.3-rc1 in a few > > weeks). > > > > Thanks, > > Robin. > > > > > Support for 64-bit MSI target addresses is needed for some of our SoCs. > > > I ran into a situation when there was no available RAM in ZONE_DMA32 > > > during initialization of PCIe host. Hence, dmam_alloc_coherent() failed > > > in dw_pcie_msi_host_init() and initialization failed with -ENOMEM: > > > > > > [ 0.374834] dw-pcie 4000000.pcie0: host bridge /soc/pcie0@4000000 > > > ranges: > > > [ 0.375813] dw-pcie 4000000.pcie0: MEM > > > 0x0041000000..0x004fffffff -> 0x0041000000 > > > [ 0.376171] dw-pcie 4000000.pcie0: IB MEM > > > 0x0400000000..0x07ffffffff -> 0x0400000000 > > > [ 0.377914] dw-pcie 4000000.pcie0: Failed to alloc and map MSI data > > > [ 0.378191] dw-pcie 4000000.pcie0: Failed to initialize host > > > [ 0.378255] dw-pcie: probe of 4000000.pcie0 failed with error -12 > > > > > > Mainline kernel 6.2-rc6 was used in that test. > > > > > > The hardware supports 64-bit target addresses, so the patch "PCI: dwc: > > > Add support for 64-bit MSI target address" should help with this > > > particular failure. > > > > > > > > > > > > > > Thanks, > > > > Will > > > > > > > > Will McVicker (2): > > > > PCI: dwc: Drop dependency on ZONE_DMA32 > > > > > > > > v6: > > > > * Retrying DMA allocation with 64-bit mask within the error > > > > if-statement. > > > > * Use an int for the DMA mask instead of a bool and ternary operation. > > > > > > > > v5: > > > > * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, > > > > retry with a 64-bit mask if supported. > > > > > > > > v4: > > > > * Updated commit descriptions. > > > > * Renamed msi_64b -> msi_64bit. > > > > * Dropped msi_64bit ternary use. > > > > * Dropped export of dw_pcie_msi_capabilities. > > > > > > > > v3: > > > > * Switched to a managed DMA allocation. > > > > * Simplified the DMA allocation cleanup. > > > > * Dropped msi_page from struct dw_pcie_rp. > > > > * Allocating a u64 instead of a full page. > > > > > > > > v2: > > > > * Fixed build error caught by kernel test robot > > > > * Fixed error handling reported by Isaac Manjarres > > > > PCI: dwc: Add support for 64-bit MSI target address > > > > > > > > .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- > > > > drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ > > > > drivers/pci/controller/dwc/pcie-designware.h | 2 +- > > > > 3 files changed, 30 insertions(+), 23 deletions(-) > > > > > > > > > > > > base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 > > > > > > Thank you in advance. > > > > > > Regards, > > > Evgenii > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2023-02-03 22:12 ` Serge Semin @ 2023-02-06 11:27 ` Evgenii Shatokhin 2023-02-09 0:48 ` Serge Semin 0 siblings, 1 reply; 12+ messages in thread From: Evgenii Shatokhin @ 2023-02-06 11:27 UTC (permalink / raw) To: Serge Semin Cc: Serge Semin, Robin Murphy, kernel-team, Vidya Sagar, Christoph Hellwig, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux, Lorenzo Pieralisi, Will McVicker Hi Sergey, First of all, thank you for the detailed explanation. It is clearer now what is going on and why it is that way. On 04.02.2023 01:12, Serge Semin wrote: > Hi Evgenii > > On Wed, Feb 01, 2023 at 04:54:55PM +0300, Evgenii Shatokhin wrote: >> On 31.01.2023 15:42, Robin Murphy wrote: >>> >>> On 2023-01-31 12:29, Evgenii Shatokhin wrote: >>>> Hi, >>>> >>>> On 26.08.2022 02:54, Will McVicker wrote: >>>>> Hi All, >>>>> >>>>> I've update patch 2/2 to address Robin's suggestions. This includes: >>>>> >>>>> * Dropping the while-loop for retrying with a 64-bit mask in favor of >>>>> retrying within the error if-statement. >>>>> * Using an int for the DMA mask instead of a bool and ternary >>>>> operation. >>>>> >>>>> Thanks again for the reviews and sorry for the extra revision today! >>>>> Hopefully this is the last one :) If not, I'd be fine to submit >>>>> patch 1/2 >>>>> without 2/2 to avoid resending patch 1/2 for future revisions of patch >>>>> 2/2 >>>>> (unless I don't need to do that anyway). >>>> >>>> The first patch of the series made it into the mainline kernel, but, it >>>> seems, the second one ("PCI: dwc: Add support for 64-bit MSI target >>>> address") did not. As of 6.2-rc6, it is still missing. >>>> >>>> Was it intentionally dropped because of some issues or, perhaps, just by >>>> accident? If it was by accident, could you please queue it for inclusion >>>> into mainline again? >>> >>> Yes, it was dropped due to the PCI_MSI_FLAGS_64BIT usage apparently >>> being incorrect, and some other open debate (which all happened on the >>> v5 thread): >>> >>> https://lore.kernel.org/linux-pci/YzVTmy9MWh+AjshC@lpieralisi/ >> > >> I see. If I understand it correctly, the problem was that >> PCI_MSI_FLAGS_64BIT flag did not guarantee that 64-bit mask could be used >> for that particular allocation. Right? >> > > William was trying to utilize for only software cause. Setting > PCI_MSI_FLAGS_64BIT didn't actually change the hardware behavior. > He could have as well provided just a driver private capability > flag. (see below for a more detailed problem description) > >>> >>> The DMA mask issues have now been sorted out, >> >> I suppose, you mean https://lore.kernel.org/all/20230113171409.30470-26-Sergey.Semin@baikalelectronics.ru/? > > Well, the way the DMA-mask issue has been solved was a bit of the > hacky. I wouldn't call it a fully proper solution. The problem with > pointlessly allocating physical memory for the iMSI-RX engine (it > doesn't perform any DMA) and artificially restricting the coherent-DMA > mask is still there. The patch in the subject was a compromise in > order to at least permit unrestricted streaming DMAs but limiting the > coherent DMAs for the MSI setup to work properly for all peripheral > devices. > >> >> It still breaks our particular case when the SoC has no 32-bit-addressable >> RAM. We'd set DMA masks to DMA_BIT_MASK(36) in the platform-specific driver >> before calling dw_pcie_host_init(). However, dw_pcie_msi_host_init() resets >> it to 32-bit, tries dmam_alloc_coherent() and fails. > > Yeah. That's another problem with the implemented approach. But are > your sure the driver had worked even before this patch? AFAICS the > driver allocated the MSI-targeted page from DMA32 zone before this > modification. So the allocation must have failed on your platform too. You are right. I did not notice earlier that the kernel based on 6.0-stable we used before did actually contain our SoC-specific workaround for this. Without that custom patch, initialization of PCIe host does not work. So, yes, the problem was present earlier too. > >> >> With 36-bit masks, the kernel seems to play well with the devices in our >> case. >> >> I saw your comment in https://lore.kernel.org/linux-pci/4dc31a63-00b1-f379-c5ac-7dc9425937f4@arm.com/ >> that drivers should always explicitly set their masks. >> > >> Is it a really bad idea to check the current coherent mask's bits in >> dw_pcie_msi_host_init() and if it is more than 32 - just issue a warning >> rather than reset it to 32-bit unconditionally? That would help in our case. >> Or, perhaps, there is a better workaround. > > The problem isn't in the value the mask is set to. The problem is > two-leveled, but is mainly connected with the PCIe device detected on > the PCIe bus. There are some of them which can't send MSI TLPs to the > 64-bit addresses. Since we can't predict whether such devices exist on > the bus beforehand the LLDD probe is performed together with the > MSI-engine initialization, the solution was to just restrict the MSIs > base address to be allocated within the lowest 4GB. Moreover as I said > above the iMSI-RX engine doesn't actually cause any DMA thus there is > no need in any memory allocation. Instead reserving some PCIe-bus > space/DWORD for MSIs would be enough. Alas the PCIe-subsystem doesn't > provide a way to do so. That's why we have what you see in the driver: > DMA mask restriction and coherent DMA memory allocation. So, if I understand you correctly, what is needed here is a small area of PCIe address space accessible to any of the connected PCIe devices. As the kernel does not know in advance, which restrictions the devices have, it tries to allocate 32-bit-addressable memory, suitable for DMA. This way, it would be OK for any attached PCIe device. Right? > > If only we had a way to auto-detected the PCIe-bus space with no > physical memory behind it and take out a DWORD from it to initialize > the iMSI-RX engine we could have immediately got rid from the mask > setting operation and the memory allocation. It would have solved your > problem too. Yes, it would solve our issue too. I do not know, however, if a generic solution is possible here, but I am no expert in PCIe. For now, we are probably better off with SoC-specific patches, when we know which PCIe devices can possibly be used and what their restrictions are. > > -Serge(y) > >> >> Looking forward to your comments. > > > >> >> >>> so you, or Will, or anyone >>> else interested should be free to rework this on top of linux-next >>> (although at this point, more realistically on top of 6.3-rc1 in a few >>> weeks). >>> >>> Thanks, >>> Robin. >>> >>>> Support for 64-bit MSI target addresses is needed for some of our SoCs. >>>> I ran into a situation when there was no available RAM in ZONE_DMA32 >>>> during initialization of PCIe host. Hence, dmam_alloc_coherent() failed >>>> in dw_pcie_msi_host_init() and initialization failed with -ENOMEM: >>>> >>>> [ 0.374834] dw-pcie 4000000.pcie0: host bridge /soc/pcie0@4000000 >>>> ranges: >>>> [ 0.375813] dw-pcie 4000000.pcie0: MEM >>>> 0x0041000000..0x004fffffff -> 0x0041000000 >>>> [ 0.376171] dw-pcie 4000000.pcie0: IB MEM >>>> 0x0400000000..0x07ffffffff -> 0x0400000000 >>>> [ 0.377914] dw-pcie 4000000.pcie0: Failed to alloc and map MSI data >>>> [ 0.378191] dw-pcie 4000000.pcie0: Failed to initialize host >>>> [ 0.378255] dw-pcie: probe of 4000000.pcie0 failed with error -12 >>>> >>>> Mainline kernel 6.2-rc6 was used in that test. >>>> >>>> The hardware supports 64-bit target addresses, so the patch "PCI: dwc: >>>> Add support for 64-bit MSI target address" should help with this >>>> particular failure. >>>> >>>> >>>>> >>>>> Thanks, >>>>> Will >>>>> >>>>> Will McVicker (2): >>>>> PCI: dwc: Drop dependency on ZONE_DMA32 >>>>> >>>>> v6: >>>>> * Retrying DMA allocation with 64-bit mask within the error >>>>> if-statement. >>>>> * Use an int for the DMA mask instead of a bool and ternary operation. >>>>> >>>>> v5: >>>>> * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, >>>>> retry with a 64-bit mask if supported. >>>>> >>>>> v4: >>>>> * Updated commit descriptions. >>>>> * Renamed msi_64b -> msi_64bit. >>>>> * Dropped msi_64bit ternary use. >>>>> * Dropped export of dw_pcie_msi_capabilities. >>>>> >>>>> v3: >>>>> * Switched to a managed DMA allocation. >>>>> * Simplified the DMA allocation cleanup. >>>>> * Dropped msi_page from struct dw_pcie_rp. >>>>> * Allocating a u64 instead of a full page. >>>>> >>>>> v2: >>>>> * Fixed build error caught by kernel test robot >>>>> * Fixed error handling reported by Isaac Manjarres >>>>> PCI: dwc: Add support for 64-bit MSI target address >>>>> >>>>> .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- >>>>> drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ >>>>> drivers/pci/controller/dwc/pcie-designware.h | 2 +- >>>>> 3 files changed, 30 insertions(+), 23 deletions(-) >>>>> >>>>> >>>>> base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 >>>> >>>> Thank you in advance. Regards, Evgenii ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2023-02-06 11:27 ` Evgenii Shatokhin @ 2023-02-09 0:48 ` Serge Semin 2023-02-09 8:03 ` Evgenii Shatokhin 0 siblings, 1 reply; 12+ messages in thread From: Serge Semin @ 2023-02-09 0:48 UTC (permalink / raw) To: Evgenii Shatokhin Cc: Serge Semin, Robin Murphy, kernel-team, Vidya Sagar, Christoph Hellwig, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux, Lorenzo Pieralisi, Will McVicker On Mon, Feb 06, 2023 at 02:27:41PM +0300, Evgenii Shatokhin wrote: > Hi Sergey, > > First of all, thank you for the detailed explanation. It is clearer now what > is going on and why it is that way. > > On 04.02.2023 01:12, Serge Semin wrote: > > Hi Evgenii > > > > On Wed, Feb 01, 2023 at 04:54:55PM +0300, Evgenii Shatokhin wrote: > > > On 31.01.2023 15:42, Robin Murphy wrote: > > > > > > > > On 2023-01-31 12:29, Evgenii Shatokhin wrote: > > > > > Hi, > > > > > > > > > > On 26.08.2022 02:54, Will McVicker wrote: > > > > > > Hi All, > > > > > > > > > > > > I've update patch 2/2 to address Robin's suggestions. This includes: > > > > > > > > > > > > * Dropping the while-loop for retrying with a 64-bit mask in favor of > > > > > > retrying within the error if-statement. > > > > > > * Using an int for the DMA mask instead of a bool and ternary > > > > > > operation. > > > > > > > > > > > > Thanks again for the reviews and sorry for the extra revision today! > > > > > > Hopefully this is the last one :) If not, I'd be fine to submit > > > > > > patch 1/2 > > > > > > without 2/2 to avoid resending patch 1/2 for future revisions of patch > > > > > > 2/2 > > > > > > (unless I don't need to do that anyway). > > > > > > > > > > The first patch of the series made it into the mainline kernel, but, it > > > > > seems, the second one ("PCI: dwc: Add support for 64-bit MSI target > > > > > address") did not. As of 6.2-rc6, it is still missing. > > > > > > > > > > Was it intentionally dropped because of some issues or, perhaps, just by > > > > > accident? If it was by accident, could you please queue it for inclusion > > > > > into mainline again? > > > > > > > > Yes, it was dropped due to the PCI_MSI_FLAGS_64BIT usage apparently > > > > being incorrect, and some other open debate (which all happened on the > > > > v5 thread): > > > > > > > > https://lore.kernel.org/linux-pci/YzVTmy9MWh+AjshC@lpieralisi/ > > > > > > > > I see. If I understand it correctly, the problem was that > > > PCI_MSI_FLAGS_64BIT flag did not guarantee that 64-bit mask could be used > > > for that particular allocation. Right? > > > > > > > William was trying to utilize for only software cause. Setting > > PCI_MSI_FLAGS_64BIT didn't actually change the hardware behavior. > > He could have as well provided just a driver private capability > > flag. (see below for a more detailed problem description) > > > > > > > > > > The DMA mask issues have now been sorted out, > > > > > > I suppose, you mean https://lore.kernel.org/all/20230113171409.30470-26-Sergey.Semin@baikalelectronics.ru/? > > > > Well, the way the DMA-mask issue has been solved was a bit of the > > hacky. I wouldn't call it a fully proper solution. The problem with > > pointlessly allocating physical memory for the iMSI-RX engine (it > > doesn't perform any DMA) and artificially restricting the coherent-DMA > > mask is still there. The patch in the subject was a compromise in > > order to at least permit unrestricted streaming DMAs but limiting the > > coherent DMAs for the MSI setup to work properly for all peripheral > > devices. > > > > > > > > It still breaks our particular case when the SoC has no 32-bit-addressable > > > RAM. We'd set DMA masks to DMA_BIT_MASK(36) in the platform-specific driver > > > before calling dw_pcie_host_init(). However, dw_pcie_msi_host_init() resets > > > it to 32-bit, tries dmam_alloc_coherent() and fails. > > > > Yeah. That's another problem with the implemented approach. But are > > your sure the driver had worked even before this patch? AFAICS the > > driver allocated the MSI-targeted page from DMA32 zone before this > > modification. So the allocation must have failed on your platform too. > > You are right. I did not notice earlier that the kernel based on 6.0-stable > we used before did actually contain our SoC-specific workaround for this. > Without that custom patch, initialization of PCIe host does not work. So, > yes, the problem was present earlier too. > > > > > > > > > With 36-bit masks, the kernel seems to play well with the devices in our > > > case. > > > > > > I saw your comment in https://lore.kernel.org/linux-pci/4dc31a63-00b1-f379-c5ac-7dc9425937f4@arm.com/ > > > that drivers should always explicitly set their masks. > > > > > > > > Is it a really bad idea to check the current coherent mask's bits in > > > dw_pcie_msi_host_init() and if it is more than 32 - just issue a warning > > > rather than reset it to 32-bit unconditionally? That would help in our case. > > > Or, perhaps, there is a better workaround. > > > > The problem isn't in the value the mask is set to. The problem is > > two-leveled, but is mainly connected with the PCIe device detected on > > the PCIe bus. There are some of them which can't send MSI TLPs to the > > 64-bit addresses. Since we can't predict whether such devices exist on > > the bus beforehand the LLDD probe is performed together with the > > MSI-engine initialization, the solution was to just restrict the MSIs > > base address to be allocated within the lowest 4GB. Moreover as I said > > above the iMSI-RX engine doesn't actually cause any DMA thus there is > > no need in any memory allocation. Instead reserving some PCIe-bus > > space/DWORD for MSIs would be enough. Alas the PCIe-subsystem doesn't > > provide a way to do so. That's why we have what you see in the driver: > > DMA mask restriction and coherent DMA memory allocation. > > So, if I understand you correctly, what is needed here is a small area of > PCIe address space accessible to any of the connected PCIe devices. As the > kernel does not know in advance, which restrictions the devices have, it > tries to allocate 32-bit-addressable memory, suitable for DMA. This way, it > would be OK for any attached PCIe device. Right? Right. The restriction is the 64-bit MSI capability. If any PCIe peripheral device has no PCI_MSI_64_BIT_ADDR_CAP flag set and the MSI target address is selected from the space above 4GB then such device MSIs won't be handled. Note as I said above no DMA actually performed if at least one MSI vector is enabled. The driver just needs a DWORD within the PCIe bus space for the MSI MWr TLPs target address and EP+vector data. The MSI TLP data is decoded by the iMSI-RX engine in order to set the corresponding flag in the MSI IRQ status register. Such TLPs won't be passed to the master AXI-bus. > > > > > If only we had a way to auto-detected the PCIe-bus space with no > > physical memory behind it and take out a DWORD from it to initialize > > the iMSI-RX engine we could have immediately got rid from the mask > > setting operation and the memory allocation. It would have solved your > > problem too. > > Yes, it would solve our issue too. I do not know, however, if a generic > solution is possible here, but I am no expert in PCIe. Currently the kernel PCIe subsystem doesn't provide a way to reserve a range within the PCIe bus memory with no physical RAM behind and left unused during the BARs resource initialization. Implementing such functionality (perhaps in the framework of the P2P module or based on it) would give the generic solution. > > For now, we are probably better off with SoC-specific patches, when we know > which PCIe devices can possibly be used and what their restrictions are. Since you know that there is no any RAM below 4GB and you have matching CPU and PCIe address spaces, then you can just take any address below 4GB and use it to initialize the MSI-target address (dw_pcie_rp.msi_data). But make sure that the peripheral PCIe-devices don't use it for something application-specific (like accessing CPU MMIO devices mapped to that base address). That seems like the most universal solution for your case. -Serge(y) > > > > > -Serge(y) > > > > > > > > Looking forward to your comments. > > > > > > > > > > > > > > > > so you, or Will, or anyone > > > > else interested should be free to rework this on top of linux-next > > > > (although at this point, more realistically on top of 6.3-rc1 in a few > > > > weeks). > > > > > > > > Thanks, > > > > Robin. > > > > > > > > > Support for 64-bit MSI target addresses is needed for some of our SoCs. > > > > > I ran into a situation when there was no available RAM in ZONE_DMA32 > > > > > during initialization of PCIe host. Hence, dmam_alloc_coherent() failed > > > > > in dw_pcie_msi_host_init() and initialization failed with -ENOMEM: > > > > > > > > > > [ 0.374834] dw-pcie 4000000.pcie0: host bridge /soc/pcie0@4000000 > > > > > ranges: > > > > > [ 0.375813] dw-pcie 4000000.pcie0: MEM > > > > > 0x0041000000..0x004fffffff -> 0x0041000000 > > > > > [ 0.376171] dw-pcie 4000000.pcie0: IB MEM > > > > > 0x0400000000..0x07ffffffff -> 0x0400000000 > > > > > [ 0.377914] dw-pcie 4000000.pcie0: Failed to alloc and map MSI data > > > > > [ 0.378191] dw-pcie 4000000.pcie0: Failed to initialize host > > > > > [ 0.378255] dw-pcie: probe of 4000000.pcie0 failed with error -12 > > > > > > > > > > Mainline kernel 6.2-rc6 was used in that test. > > > > > > > > > > The hardware supports 64-bit target addresses, so the patch "PCI: dwc: > > > > > Add support for 64-bit MSI target address" should help with this > > > > > particular failure. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Will > > > > > > > > > > > > Will McVicker (2): > > > > > > PCI: dwc: Drop dependency on ZONE_DMA32 > > > > > > > > > > > > v6: > > > > > > * Retrying DMA allocation with 64-bit mask within the error > > > > > > if-statement. > > > > > > * Use an int for the DMA mask instead of a bool and ternary operation. > > > > > > > > > > > > v5: > > > > > > * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, > > > > > > retry with a 64-bit mask if supported. > > > > > > > > > > > > v4: > > > > > > * Updated commit descriptions. > > > > > > * Renamed msi_64b -> msi_64bit. > > > > > > * Dropped msi_64bit ternary use. > > > > > > * Dropped export of dw_pcie_msi_capabilities. > > > > > > > > > > > > v3: > > > > > > * Switched to a managed DMA allocation. > > > > > > * Simplified the DMA allocation cleanup. > > > > > > * Dropped msi_page from struct dw_pcie_rp. > > > > > > * Allocating a u64 instead of a full page. > > > > > > > > > > > > v2: > > > > > > * Fixed build error caught by kernel test robot > > > > > > * Fixed error handling reported by Isaac Manjarres > > > > > > PCI: dwc: Add support for 64-bit MSI target address > > > > > > > > > > > > .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- > > > > > > drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ > > > > > > drivers/pci/controller/dwc/pcie-designware.h | 2 +- > > > > > > 3 files changed, 30 insertions(+), 23 deletions(-) > > > > > > > > > > > > > > > > > > base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 > > > > > > > > > > Thank you in advance. > > Regards, > Evgenii > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2023-02-09 0:48 ` Serge Semin @ 2023-02-09 8:03 ` Evgenii Shatokhin 2023-02-09 15:54 ` Robin Murphy 0 siblings, 1 reply; 12+ messages in thread From: Evgenii Shatokhin @ 2023-02-09 8:03 UTC (permalink / raw) To: Serge Semin Cc: Serge Semin, Robin Murphy, kernel-team, Vidya Sagar, Christoph Hellwig, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux, Lorenzo Pieralisi, Will McVicker On 09.02.2023 03:48, Serge Semin wrote: > On Mon, Feb 06, 2023 at 02:27:41PM +0300, Evgenii Shatokhin wrote: >> Hi Sergey, >> >> First of all, thank you for the detailed explanation. It is clearer now what >> is going on and why it is that way. >> >> On 04.02.2023 01:12, Serge Semin wrote: >>> Hi Evgenii >>> >>> On Wed, Feb 01, 2023 at 04:54:55PM +0300, Evgenii Shatokhin wrote: >>>> On 31.01.2023 15:42, Robin Murphy wrote: >>>>> >>>>> On 2023-01-31 12:29, Evgenii Shatokhin wrote: >>>>>> Hi, >>>>>> >>>>>> On 26.08.2022 02:54, Will McVicker wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> I've update patch 2/2 to address Robin's suggestions. This includes: >>>>>>> >>>>>>> * Dropping the while-loop for retrying with a 64-bit mask in favor of >>>>>>> retrying within the error if-statement. >>>>>>> * Using an int for the DMA mask instead of a bool and ternary >>>>>>> operation. >>>>>>> >>>>>>> Thanks again for the reviews and sorry for the extra revision today! >>>>>>> Hopefully this is the last one :) If not, I'd be fine to submit >>>>>>> patch 1/2 >>>>>>> without 2/2 to avoid resending patch 1/2 for future revisions of patch >>>>>>> 2/2 >>>>>>> (unless I don't need to do that anyway). >>>>>> >>>>>> The first patch of the series made it into the mainline kernel, but, it >>>>>> seems, the second one ("PCI: dwc: Add support for 64-bit MSI target >>>>>> address") did not. As of 6.2-rc6, it is still missing. >>>>>> >>>>>> Was it intentionally dropped because of some issues or, perhaps, just by >>>>>> accident? If it was by accident, could you please queue it for inclusion >>>>>> into mainline again? >>>>> >>>>> Yes, it was dropped due to the PCI_MSI_FLAGS_64BIT usage apparently >>>>> being incorrect, and some other open debate (which all happened on the >>>>> v5 thread): >>>>> >>>>> https://lore.kernel.org/linux-pci/YzVTmy9MWh+AjshC@lpieralisi/ >>>> >>> >>>> I see. If I understand it correctly, the problem was that >>>> PCI_MSI_FLAGS_64BIT flag did not guarantee that 64-bit mask could be used >>>> for that particular allocation. Right? >>>> >>> >>> William was trying to utilize for only software cause. Setting >>> PCI_MSI_FLAGS_64BIT didn't actually change the hardware behavior. >>> He could have as well provided just a driver private capability >>> flag. (see below for a more detailed problem description) >>> >>>>> >>>>> The DMA mask issues have now been sorted out, >>>> >>>> I suppose, you mean https://lore.kernel.org/all/20230113171409.30470-26-Sergey.Semin@baikalelectronics.ru/? >>> >>> Well, the way the DMA-mask issue has been solved was a bit of the >>> hacky. I wouldn't call it a fully proper solution. The problem with >>> pointlessly allocating physical memory for the iMSI-RX engine (it >>> doesn't perform any DMA) and artificially restricting the coherent-DMA >>> mask is still there. The patch in the subject was a compromise in >>> order to at least permit unrestricted streaming DMAs but limiting the >>> coherent DMAs for the MSI setup to work properly for all peripheral >>> devices. >>> >>>> >>>> It still breaks our particular case when the SoC has no 32-bit-addressable >>>> RAM. We'd set DMA masks to DMA_BIT_MASK(36) in the platform-specific driver >>>> before calling dw_pcie_host_init(). However, dw_pcie_msi_host_init() resets >>>> it to 32-bit, tries dmam_alloc_coherent() and fails. >>> >>> Yeah. That's another problem with the implemented approach. But are >>> your sure the driver had worked even before this patch? AFAICS the >>> driver allocated the MSI-targeted page from DMA32 zone before this >>> modification. So the allocation must have failed on your platform too. >> >> You are right. I did not notice earlier that the kernel based on 6.0-stable >> we used before did actually contain our SoC-specific workaround for this. >> Without that custom patch, initialization of PCIe host does not work. So, >> yes, the problem was present earlier too. >> >>> >>>> >>>> With 36-bit masks, the kernel seems to play well with the devices in our >>>> case. >>>> >>>> I saw your comment in https://lore.kernel.org/linux-pci/4dc31a63-00b1-f379-c5ac-7dc9425937f4@arm.com/ >>>> that drivers should always explicitly set their masks. >>>> >>> >>>> Is it a really bad idea to check the current coherent mask's bits in >>>> dw_pcie_msi_host_init() and if it is more than 32 - just issue a warning >>>> rather than reset it to 32-bit unconditionally? That would help in our case. >>>> Or, perhaps, there is a better workaround. >>> >>> The problem isn't in the value the mask is set to. The problem is >>> two-leveled, but is mainly connected with the PCIe device detected on >>> the PCIe bus. There are some of them which can't send MSI TLPs to the >>> 64-bit addresses. Since we can't predict whether such devices exist on >>> the bus beforehand the LLDD probe is performed together with the >>> MSI-engine initialization, the solution was to just restrict the MSIs >>> base address to be allocated within the lowest 4GB. Moreover as I said >>> above the iMSI-RX engine doesn't actually cause any DMA thus there is >>> no need in any memory allocation. Instead reserving some PCIe-bus >>> space/DWORD for MSIs would be enough. Alas the PCIe-subsystem doesn't >>> provide a way to do so. That's why we have what you see in the driver: >>> DMA mask restriction and coherent DMA memory allocation. >> > >> So, if I understand you correctly, what is needed here is a small area of >> PCIe address space accessible to any of the connected PCIe devices. As the >> kernel does not know in advance, which restrictions the devices have, it >> tries to allocate 32-bit-addressable memory, suitable for DMA. This way, it >> would be OK for any attached PCIe device. Right? > > Right. The restriction is the 64-bit MSI capability. If any PCIe > peripheral device has no PCI_MSI_64_BIT_ADDR_CAP flag set and the MSI > target address is selected from the space above 4GB then such device > MSIs won't be handled. > > Note as I said above no DMA actually performed if at least one MSI > vector is enabled. The driver just needs a DWORD within the PCIe bus > space for the MSI MWr TLPs target address and EP+vector data. The MSI > TLP data is decoded by the iMSI-RX engine in order to set the > corresponding flag in the MSI IRQ status register. Such TLPs won't be > passed to the master AXI-bus. > >> >>> >>> If only we had a way to auto-detected the PCIe-bus space with no >>> physical memory behind it and take out a DWORD from it to initialize >>> the iMSI-RX engine we could have immediately got rid from the mask >>> setting operation and the memory allocation. It would have solved your >>> problem too. >> > >> Yes, it would solve our issue too. I do not know, however, if a generic >> solution is possible here, but I am no expert in PCIe. > > Currently the kernel PCIe subsystem doesn't provide a way to reserve a > range within the PCIe bus memory with no physical RAM behind and left > unused during the BARs resource initialization. Implementing such > functionality (perhaps in the framework of the P2P module or based on > it) would give the generic solution. > >> >> For now, we are probably better off with SoC-specific patches, when we know >> which PCIe devices can possibly be used and what their restrictions are. > > Since you know that there is no any RAM below 4GB and you have > matching CPU and PCIe address spaces, then you can just take any > address below 4GB and use it to initialize the MSI-target address > (dw_pcie_rp.msi_data). But make sure that the peripheral PCIe-devices > don't use it for something application-specific (like accessing CPU > MMIO devices mapped to that base address). That seems like the most > universal solution for your case. Interesting idea! Thank you, Sergey. > > -Serge(y) > >> >>> >>> -Serge(y) >>> >>>> >>>> Looking forward to your comments. >>> >>> >>> >>>> >>>> >>>>> so you, or Will, or anyone >>>>> else interested should be free to rework this on top of linux-next >>>>> (although at this point, more realistically on top of 6.3-rc1 in a few >>>>> weeks). >>>>> >>>>> Thanks, >>>>> Robin. >>>>> >>>>>> Support for 64-bit MSI target addresses is needed for some of our SoCs. >>>>>> I ran into a situation when there was no available RAM in ZONE_DMA32 >>>>>> during initialization of PCIe host. Hence, dmam_alloc_coherent() failed >>>>>> in dw_pcie_msi_host_init() and initialization failed with -ENOMEM: >>>>>> >>>>>> [ 0.374834] dw-pcie 4000000.pcie0: host bridge /soc/pcie0@4000000 >>>>>> ranges: >>>>>> [ 0.375813] dw-pcie 4000000.pcie0: MEM >>>>>> 0x0041000000..0x004fffffff -> 0x0041000000 >>>>>> [ 0.376171] dw-pcie 4000000.pcie0: IB MEM >>>>>> 0x0400000000..0x07ffffffff -> 0x0400000000 >>>>>> [ 0.377914] dw-pcie 4000000.pcie0: Failed to alloc and map MSI data >>>>>> [ 0.378191] dw-pcie 4000000.pcie0: Failed to initialize host >>>>>> [ 0.378255] dw-pcie: probe of 4000000.pcie0 failed with error -12 >>>>>> >>>>>> Mainline kernel 6.2-rc6 was used in that test. >>>>>> >>>>>> The hardware supports 64-bit target addresses, so the patch "PCI: dwc: >>>>>> Add support for 64-bit MSI target address" should help with this >>>>>> particular failure. >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Will >>>>>>> >>>>>>> Will McVicker (2): >>>>>>> PCI: dwc: Drop dependency on ZONE_DMA32 >>>>>>> >>>>>>> v6: >>>>>>> * Retrying DMA allocation with 64-bit mask within the error >>>>>>> if-statement. >>>>>>> * Use an int for the DMA mask instead of a bool and ternary operation. >>>>>>> >>>>>>> v5: >>>>>>> * Updated patch 2/2 to first try with a 32-bit DMA mask. On failure, >>>>>>> retry with a 64-bit mask if supported. >>>>>>> >>>>>>> v4: >>>>>>> * Updated commit descriptions. >>>>>>> * Renamed msi_64b -> msi_64bit. >>>>>>> * Dropped msi_64bit ternary use. >>>>>>> * Dropped export of dw_pcie_msi_capabilities. >>>>>>> >>>>>>> v3: >>>>>>> * Switched to a managed DMA allocation. >>>>>>> * Simplified the DMA allocation cleanup. >>>>>>> * Dropped msi_page from struct dw_pcie_rp. >>>>>>> * Allocating a u64 instead of a full page. >>>>>>> >>>>>>> v2: >>>>>>> * Fixed build error caught by kernel test robot >>>>>>> * Fixed error handling reported by Isaac Manjarres >>>>>>> PCI: dwc: Add support for 64-bit MSI target address >>>>>>> >>>>>>> .../pci/controller/dwc/pcie-designware-host.c | 43 +++++++++---------- >>>>>>> drivers/pci/controller/dwc/pcie-designware.c | 8 ++++ >>>>>>> drivers/pci/controller/dwc/pcie-designware.h | 2 +- >>>>>>> 3 files changed, 30 insertions(+), 23 deletions(-) >>>>>>> >>>>>>> >>>>>>> base-commit: 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 >>>>>> >>>>>> Thank you in advance. >> >> Regards, >> Evgenii >> >> > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses 2023-02-09 8:03 ` Evgenii Shatokhin @ 2023-02-09 15:54 ` Robin Murphy 0 siblings, 0 replies; 12+ messages in thread From: Robin Murphy @ 2023-02-09 15:54 UTC (permalink / raw) To: Evgenii Shatokhin, Serge Semin Cc: Serge Semin, kernel-team, Vidya Sagar, Christoph Hellwig, linux-pci, linux-kernel, Gustavo Pimentel, Jingoo Han, Rob Herring, Krzysztof Wilczyński, Bjorn Helgaas, linux, Lorenzo Pieralisi, Will McVicker On 2023-02-09 08:03, Evgenii Shatokhin wrote: [...] >> Currently the kernel PCIe subsystem doesn't provide a way to reserve a >> range within the PCIe bus memory with no physical RAM behind and left >> unused during the BARs resource initialization. Implementing such >> functionality (perhaps in the framework of the P2P module or based on >> it) would give the generic solution. >> >>> >>> For now, we are probably better off with SoC-specific patches, when >>> we know >>> which PCIe devices can possibly be used and what their restrictions are. >> >> Since you know that there is no any RAM below 4GB and you have >> matching CPU and PCIe address spaces, then you can just take any >> address below 4GB and use it to initialize the MSI-target address >> (dw_pcie_rp.msi_data). But make sure that the peripheral PCIe-devices >> don't use it for something application-specific (like accessing CPU >> MMIO devices mapped to that base address). That seems like the most >> universal solution for your case. > > Interesting idea! > Thank you, Sergey. Yes, if the platform-specific driver knows enough to be able to pick a suitable "safe" MSI address, then I think allowing it to pre-set pp.msi_data, and only falling back to the dma-mapping workaround otherwise, sounds reasonable. Thanks, Robin. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2023-02-09 15:55 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-08-25 23:54 [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Will McVicker 2022-08-25 23:54 ` [PATCH v6 1/2] PCI: dwc: Drop dependency on ZONE_DMA32 Will McVicker 2022-08-25 23:54 ` [PATCH v6 2/2] PCI: dwc: Add support for 64-bit MSI target address Will McVicker 2022-08-29 8:03 ` [PATCH v6 0/2] PCI: dwc: Add support for 64-bit MSI target addresses Lorenzo Pieralisi 2023-01-31 12:29 ` Evgenii Shatokhin 2023-01-31 12:42 ` Robin Murphy 2023-02-01 13:54 ` Evgenii Shatokhin 2023-02-03 22:12 ` Serge Semin 2023-02-06 11:27 ` Evgenii Shatokhin 2023-02-09 0:48 ` Serge Semin 2023-02-09 8:03 ` Evgenii Shatokhin 2023-02-09 15:54 ` Robin Murphy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).