Re: [PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

Re: [PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

[Koehrer Mathias (ETAS/ESW5)]

Hi Greg,
> > thanks for the response.
> > > > Some uio based PCI drivers (e.g. uio_cif) do not work if the
> > > > assigned PCI memory resources are not page aligned.
> > > > By using the kernel option "pci=resource_alignment" it is possible
> > > > to force single PCI boards to use page alignment for their memory
> resources.
> > > > However, this is fairly cumbersome if multiple of these cards are in use.
> > > > This patch adds another kernel option "pci=resources_page_aligned"
> > > > that allows to set the default alignment of PCI memory resources
> > > > to be page aligned.
> > > > Overwriting of this default alignment via "pci=resource_alignment"
> > > > is still possible.
> > >
> > > This seems like a very large hammer for a very specific problem with
> > > a card that seems to be not-in-spec (I can't find anywhere that
> > > requires resources to be "page" aligned in the PCI spec).  Why not fix the
> firmware in these cards?
> > The issue is not the firmware. The issue is that the Linux uio driver
> > (here: uio_cif) does not work as uio_mmap() -> uio_mmap_physical()
> > does not do a mmap on physical memory that is not page aligned...
> 
> Then why not fix the uio_cif driver?
The fix might be in uio and not in uio_cif. However uio_mmap_physical() in
uio.c has introduced checks to reject non page-aligned addresses.
These checks have been introduced somewhere in between kernel V3.2 and kernel V3.18. 
Removing these checks again does not sound to me like a good idea...
The overall question is now if the UIO system is working at all with non page 
aligned PCI memory resources. For me it looks as if this is not the case.
E.g. with the boards supported with the uio_cif driver, the memory resource 2 of the
PCI device is a 128 byte memory. And exactly this memory has to be mmaped by the
driver and this fails.
As the uio driver do (no longer) work with non page aligned PCI memory resources,
the conclusion for me is to force the relevant PCI memory resources to be page aligned.
And that's why the patch might be used for.

Regards

Mathias

Re: [PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

[gregkh]

On Tue, Jun 07, 2016 at 06:51:47AM +0000, Koehrer Mathias (ETAS/ESW5) wrote:
> Hi Greg,
> > > thanks for the response.
> > > > > Some uio based PCI drivers (e.g. uio_cif) do not work if the
> > > > > assigned PCI memory resources are not page aligned.
> > > > > By using the kernel option "pci=resource_alignment" it is possible
> > > > > to force single PCI boards to use page alignment for their memory
> > resources.
> > > > > However, this is fairly cumbersome if multiple of these cards are in use.
> > > > > This patch adds another kernel option "pci=resources_page_aligned"
> > > > > that allows to set the default alignment of PCI memory resources
> > > > > to be page aligned.
> > > > > Overwriting of this default alignment via "pci=resource_alignment"
> > > > > is still possible.
> > > >
> > > > This seems like a very large hammer for a very specific problem with
> > > > a card that seems to be not-in-spec (I can't find anywhere that
> > > > requires resources to be "page" aligned in the PCI spec).  Why not fix the
> > firmware in these cards?
> > > The issue is not the firmware. The issue is that the Linux uio driver
> > > (here: uio_cif) does not work as uio_mmap() -> uio_mmap_physical()
> > > does not do a mmap on physical memory that is not page aligned...
> > 
> > Then why not fix the uio_cif driver?
> The fix might be in uio and not in uio_cif. However uio_mmap_physical() in
> uio.c has introduced checks to reject non page-aligned addresses.

Yes, because we had real problems with that in the past.

> These checks have been introduced somewhere in between kernel V3.2 and kernel V3.18. 
> Removing these checks again does not sound to me like a good idea...

I agree, you should fix your hardware :)

> The overall question is now if the UIO system is working at all with non page 
> aligned PCI memory resources. For me it looks as if this is not the case.

How is your system ending up with such alignment in the first place?
What is the resource list after booting?

thanks,

greg k-h

Re: [PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

[Koehrer Mathias (ETAS/ESW5)]

Hi Greg,
> > > > The issue is not the firmware. The issue is that the Linux uio
> > > > driver
> > > > (here: uio_cif) does not work as uio_mmap() -> uio_mmap_physical()
> > > > does not do a mmap on physical memory that is not page aligned...
> > >
> > > Then why not fix the uio_cif driver?
> > The fix might be in uio and not in uio_cif. However
> > uio_mmap_physical() in uio.c has introduced checks to reject non page-
> aligned addresses.
> 
> Yes, because we had real problems with that in the past.
> 
> > These checks have been introduced somewhere in between kernel V3.2 and
> kernel V3.18.
> > Removing these checks again does not sound to me like a good idea...
> 
> I agree, you should fix your hardware :)
> 
> > The overall question is now if the UIO system is working at all with
> > non page aligned PCI memory resources. For me it looks as if this is not the
> case.
> 
> How is your system ending up with such alignment in the first place?
> What is the resource list after booting?
Here is the resource list of the relevant PCI board. The first memory resource has a size of 128k and is not page aligned.

0f:05.0 Bridge: PLX Technology, Inc. PCI9030 32-bit 33MHz PCI <-> IOBus Bridge (rev 01)
	Subsystem: PLX Technology, Inc. Hilscher CIF50-PB/DPS Profibus
	Flags: medium devsel, IRQ 23
	Memory at eae4f400 (32-bit, non-prefetchable) [size=128]
	I/O ports at 9800 [size=128]
	Memory at eae4c000 (32-bit, non-prefetchable) [size=8K]
	Capabilities: [40] Power Management version 1
	Capabilities: [48] CompactPCI hot-swap <?>
	Capabilities: [4c] Vital Product Data
	Kernel driver in use: hilscher

Regards

Mathias

Re: [PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

[gregkh]

On Mon, Jun 06, 2016 at 02:59:47PM +0000, Koehrer Mathias (ETAS/ESW5) wrote:
> Hi Greg,
> 
> thanks for the response.
> > > Some uio based PCI drivers (e.g. uio_cif) do not work if the assigned
> > > PCI memory resources are not page aligned.
> > > By using the kernel option "pci=resource_alignment" it is possible to
> > > force single PCI boards to use page alignment for their memory resources.
> > > However, this is fairly cumbersome if multiple of these cards are in use.
> > > This patch adds another kernel option "pci=resources_page_aligned"
> > > that allows to set the default alignment of PCI memory resources to be
> > > page aligned.
> > > Overwriting of this default alignment via "pci=resource_alignment" is
> > > still possible.
> > 
> > This seems like a very large hammer for a very specific problem with a card
> > that seems to be not-in-spec (I can't find anywhere that requires resources to
> > be "page" aligned in the PCI spec).  Why not fix the firmware in these cards?
> The issue is not the firmware. The issue is that the Linux uio driver (here: uio_cif)
> does not work as uio_mmap() -> uio_mmap_physical() does not do a mmap 
> on physical memory that is not page aligned...

Then why not fix the uio_cif driver?

greg k-h

Re: [PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

[Koehrer Mathias (ETAS/ESW5)]

Hi Greg,

thanks for the response.
> > Some uio based PCI drivers (e.g. uio_cif) do not work if the assigned
> > PCI memory resources are not page aligned.
> > By using the kernel option "pci=resource_alignment" it is possible to
> > force single PCI boards to use page alignment for their memory resources.
> > However, this is fairly cumbersome if multiple of these cards are in use.
> > This patch adds another kernel option "pci=resources_page_aligned"
> > that allows to set the default alignment of PCI memory resources to be
> > page aligned.
> > Overwriting of this default alignment via "pci=resource_alignment" is
> > still possible.
> 
> This seems like a very large hammer for a very specific problem with a card
> that seems to be not-in-spec (I can't find anywhere that requires resources to
> be "page" aligned in the PCI spec).  Why not fix the firmware in these cards?
The issue is not the firmware. The issue is that the Linux uio driver (here: uio_cif)
does not work as uio_mmap() -> uio_mmap_physical() does not do a mmap 
on physical memory that is not page aligned...

Regards

Mathias

Re: [PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

[gregkh]

On Mon, Jun 06, 2016 at 06:53:28AM +0000, Koehrer Mathias (ETAS/ESW5) wrote:
> Some uio based PCI drivers (e.g. uio_cif) do not work if the assigned 
> PCI memory resources are not page aligned.
> By using the kernel option "pci=resource_alignment" it is possible to force
> single PCI boards to use page alignment for their memory resources.
> However, this is fairly cumbersome if multiple of these cards are in use.
> This patch adds another kernel option "pci=resources_page_aligned"
> that allows to set the default alignment of PCI memory resources to
> be page aligned.
> Overwriting of this default alignment via "pci=resource_alignment" is
> still possible.

This seems like a very large hammer for a very specific problem with a
card that seems to be not-in-spec (I can't find anywhere that requires
resources to be "page" aligned in the PCI spec).  Why not fix the
firmware in these cards?

thanks,

greg k-h

[PATCH] Introducing kernel option to force all PCI memory resources to be page aligned. Fixes issues with uio based drivers.

[Koehrer Mathias (ETAS/ESW5)]

Some uio based PCI drivers (e.g. uio_cif) do not work if the assigned 
PCI memory resources are not page aligned.
By using the kernel option "pci=resource_alignment" it is possible to force
single PCI boards to use page alignment for their memory resources.
However, this is fairly cumbersome if multiple of these cards are in use.
This patch adds another kernel option "pci=resources_page_aligned"
that allows to set the default alignment of PCI memory resources to
be page aligned.
Overwriting of this default alignment via "pci=resource_alignment" is
still possible.

Signed-off-by: Mathias Koehrer <mathias.koehrer@etas.com>

---
 Documentation/kernel-parameters.txt |    1 +
 drivers/pci/pci.c                   |    6 ++++--
 2 files changed, 5 insertions(+), 2 deletions(-)

Index: linux-4.7-rc1/Documentation/kernel-parameters.txt
===================================================================
--- linux-4.7-rc1.orig/Documentation/kernel-parameters.txt
+++ linux-4.7-rc1/Documentation/kernel-parameters.txt
@@ -3004,6 +3004,7 @@ bytes respectively. Such letter suffixes
 				PAGE_SIZE is used as alignment.
 				PCI-PCI bridge can be specified, if resource
 				windows need to be expanded.
+		resources_page_aligned 	Force all memory resources to be page aligned.
 		ecrc=		Enable/disable PCIe ECRC (transaction layer
 				end-to-end CRC checking).
 				bios: Use BIOS/firmware settings. This is the
Index: linux-4.7-rc1/drivers/pci/pci.c
===================================================================
--- linux-4.7-rc1.orig/drivers/pci/pci.c
+++ linux-4.7-rc1/drivers/pci/pci.c
@@ -4744,7 +4744,7 @@ EXPORT_SYMBOL_GPL(pci_ignore_hotplug);
 #define RESOURCE_ALIGNMENT_PARAM_SIZE COMMAND_LINE_SIZE
 static char resource_alignment_param[RESOURCE_ALIGNMENT_PARAM_SIZE] = {0};
 static DEFINE_SPINLOCK(resource_alignment_lock);
-
+static resource_size_t pci_default_resource_alignment = 0;
 /**
  * pci_specified_resource_alignment - get resource alignment specified by user.
  * @dev: the PCI device to get
@@ -4755,7 +4755,7 @@ static DEFINE_SPINLOCK(resource_alignmen
 static resource_size_t pci_specified_resource_alignment(struct pci_dev *dev)
 {
 	int seg, bus, slot, func, align_order, count;
-	resource_size_t align = 0;
+	resource_size_t align = pci_default_resource_alignment;
 	char *p;
 
 	spin_lock(&resource_alignment_lock);
@@ -5015,6 +5015,8 @@ static int __init pci_setup(char *str)
 			} else if (!strncmp(str, "resource_alignment=", 19)) {
 				pci_set_resource_alignment_param(str + 19,
 							strlen(str + 19));
+			} else if (!strncmp(str, "resources_page_aligned", 22)) {
+				pci_default_resource_alignment = PAGE_SIZE;
 			} else if (!strncmp(str, "ecrc=", 5)) {
 				pcie_ecrc_get_policy(str + 5);
 			} else if (!strncmp(str, "hpiosize=", 9)) {