* [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init @ 2012-01-27 19:10 Vaidyanathan Srinivasan 2012-01-27 21:05 ` Yinghai Lu 2012-01-30 3:18 ` Ram Pai 0 siblings, 2 replies; 9+ messages in thread From: Vaidyanathan Srinivasan @ 2012-01-27 19:10 UTC (permalink / raw) To: Ram Pai, Jesse Barnes, Yinghai Lu; +Cc: linux-pci, linux-kernel Hi Ram and Jesse, I found a trivial issue with page size alignment check on IBM POWER box with 64k base page size. In sriov_init(), changing the check from PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was required to use one of the sriov adapter as PF since the resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for pseries boxes. I think resource_size() could be less than SystemPageSize, but I would like your comments/ack/nack on any consequences of checking for only 4k alignment here in a system with larger base page size. Thanks, Vaidy --- pci: check for 4k resource_size alignment in sriov_init pci sriov_init should check for 4k page size alignment of resource_size even if base page size is larger -- like 64k in powerpc. Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 0321fa3..5816fa0 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -474,7 +474,7 @@ found: pos + PCI_SRIOV_BAR + i * 4); if (!res->flags) continue; - if (resource_size(res) & (PAGE_SIZE - 1)) { + if (resource_size(res) & (HW_PAGE_SIZE - 1)) { rc = -EIO; goto failed; } ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-01-27 19:10 [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init Vaidyanathan Srinivasan @ 2012-01-27 21:05 ` Yinghai Lu 2012-01-29 13:11 ` Vaidyanathan Srinivasan 2012-01-30 3:18 ` Ram Pai 1 sibling, 1 reply; 9+ messages in thread From: Yinghai Lu @ 2012-01-27 21:05 UTC (permalink / raw) To: svaidy; +Cc: Ram Pai, Jesse Barnes, linux-pci, linux-kernel On Fri, Jan 27, 2012 at 11:10 AM, Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> wrote: > Hi Ram and Jesse, > > I found a trivial issue with page size alignment check on IBM POWER > box with 64k base page size. In sriov_init(), changing the check from > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > required to use one of the sriov adapter as PF since the > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > pseries boxes. > > I think resource_size() could be less than SystemPageSize, but I would > like your comments/ack/nack on any consequences of checking for only > 4k alignment here in a system with larger base page size. > > Thanks, > Vaidy > > --- > > pci: check for 4k resource_size alignment in sriov_init > > pci sriov_init should check for 4k page size alignment of resource_size > even if base page size is larger -- like 64k in powerpc. > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > index 0321fa3..5816fa0 100644 > --- a/drivers/pci/iov.c > +++ b/drivers/pci/iov.c > @@ -474,7 +474,7 @@ found: > pos + PCI_SRIOV_BAR + i * 4); > if (!res->flags) > continue; > - if (resource_size(res) & (PAGE_SIZE - 1)) { > + if (resource_size(res) & (HW_PAGE_SIZE - 1)) { > rc = -EIO; > goto failed; > } > but HW_PAGE_SIZE is only defined for powerpc. also there is PAGE_SHIFT around in that function. maybe you can just define another MARCO according to IOV spec? Thanks Yinghai ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-01-27 21:05 ` Yinghai Lu @ 2012-01-29 13:11 ` Vaidyanathan Srinivasan 0 siblings, 0 replies; 9+ messages in thread From: Vaidyanathan Srinivasan @ 2012-01-29 13:11 UTC (permalink / raw) To: Yinghai Lu; +Cc: Ram Pai, Jesse Barnes, linux-pci, linux-kernel * Yinghai Lu <yinghai@kernel.org> [2012-01-27 13:05:41]: > On Fri, Jan 27, 2012 at 11:10 AM, Vaidyanathan Srinivasan > <svaidy@linux.vnet.ibm.com> wrote: > > Hi Ram and Jesse, > > > > I found a trivial issue with page size alignment check on IBM POWER > > box with 64k base page size. In sriov_init(), changing the check from > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > > required to use one of the sriov adapter as PF since the > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > > pseries boxes. > > > > I think resource_size() could be less than SystemPageSize, but I would > > like your comments/ack/nack on any consequences of checking for only > > 4k alignment here in a system with larger base page size. > > > > Thanks, > > Vaidy > > > > --- > > > > pci: check for 4k resource_size alignment in sriov_init > > > > pci sriov_init should check for 4k page size alignment of resource_size > > even if base page size is larger -- like 64k in powerpc. > > > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> > > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > > index 0321fa3..5816fa0 100644 > > --- a/drivers/pci/iov.c > > +++ b/drivers/pci/iov.c > > @@ -474,7 +474,7 @@ found: > > pos + PCI_SRIOV_BAR + i * 4); > > if (!res->flags) > > continue; > > - if (resource_size(res) & (PAGE_SIZE - 1)) { > > + if (resource_size(res) & (HW_PAGE_SIZE - 1)) { > > rc = -EIO; > > goto failed; > > } > > > > but HW_PAGE_SIZE is only defined for powerpc. My bad, I picked the #define used in other powerpc code. > also there is PAGE_SHIFT around in that function. This gets defined correctly if CONFIG_PPC_64K_PAGES=y. But the actual problem is the need for a generic 4K #define for x86 and other archs. > maybe you can just define another MARCO according to IOV spec? This is a good idea. I could not find an IOV specific requirement. The resource size has to be a multiple of 4K so that the overall resource-size() * PCI_SRIOV_TOTAL_VF will be a multiple of 4K or more. Let me share a patch that has a simple #define in drivers/pci/pci.h, which is needed to get SRIOV cards started on powerpc box. --Vaidy --- pci: check for 4k resource_size alignment in sriov_init pci sriov_init should check for 4k page size alignment of resource_size even if base page size is larger like 64k in powerpc. Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 0321fa3..10adede 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -474,7 +474,7 @@ found: pos + PCI_SRIOV_BAR + i * 4); if (!res->flags) continue; - if (resource_size(res) & (PAGE_SIZE - 1)) { + if (resource_size(res) & (HW_PAGE_SIZE_4K - 1)) { rc = -EIO; goto failed; } diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 1009a5e..68703ab 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -6,6 +6,9 @@ #define PCI_CFG_SPACE_SIZE 256 #define PCI_CFG_SPACE_EXP_SIZE 4096 +/* Constants used in the PCI core code */ +#define HW_PAGE_SIZE_4K 0x1000 + /* Functions internal to the PCI core code */ extern int pci_uevent(struct device *dev, struct kobj_uevent_env *env); ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-01-27 19:10 [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init Vaidyanathan Srinivasan 2012-01-27 21:05 ` Yinghai Lu @ 2012-01-30 3:18 ` Ram Pai 2012-01-31 17:44 ` Vaidyanathan Srinivasan 1 sibling, 1 reply; 9+ messages in thread From: Ram Pai @ 2012-01-30 3:18 UTC (permalink / raw) To: Vaidyanathan Srinivasan Cc: Ram Pai, Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote: > Hi Ram and Jesse, > > I found a trivial issue with page size alignment check on IBM POWER > box with 64k base page size. In sriov_init(), changing the check from > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > required to use one of the sriov adapter as PF since the > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > pseries boxes. > > I think resource_size() could be less than SystemPageSize, but I would > like your comments/ack/nack on any consequences of checking for only > 4k alignment here in a system with larger base page size. As per the SRIOV specs, the resource has to be System page size aligned. PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB page sizes. In your case if your adapter's PF is not supporting 64K page size then I think it is not conforming to the PCI SRIOV spec. RP ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-01-30 3:18 ` Ram Pai @ 2012-01-31 17:44 ` Vaidyanathan Srinivasan 2012-02-01 6:21 ` Ram Pai 0 siblings, 1 reply; 9+ messages in thread From: Vaidyanathan Srinivasan @ 2012-01-31 17:44 UTC (permalink / raw) To: Ram Pai; +Cc: Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]: > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote: > > Hi Ram and Jesse, > > > > I found a trivial issue with page size alignment check on IBM POWER > > box with 64k base page size. In sriov_init(), changing the check from > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > > required to use one of the sriov adapter as PF since the > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > > pseries boxes. > > > > I think resource_size() could be less than SystemPageSize, but I would > > like your comments/ack/nack on any consequences of checking for only > > 4k alignment here in a system with larger base page size. > > As per the SRIOV specs, the resource has to be System page size aligned. > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB > page sizes. In your case if your adapter's PF is not supporting 64K page size > then I think it is not conforming to the PCI SRIOV spec. Hi Ram, Thanks for the pointer. I did some more experiments and found that the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was set to default 4k when we do the query and check resource_size(). You were correct, the resource_size() has to come up with 64k on 64k PAGE_SIZE system. We should not change that check. I was able to get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before we do the query. This was the case in the original code before you moved these to sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in sriov_init(), then I have the following fix that works for me. Please review and let me know your comments. Thanks, Vaidy --- pci: set pci sriov page size before reading sriov bar For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k, so this change is required on powerpc box with 64k base page size. This is a regression caused due to moving SRIOV init to sriov_enable(). | commit afd24ece5c76af87f6fc477f2747b83a764f161c | Author: Ram Pai <linuxram@us.ibm.com> | PCI: delay configuration of SRIOV capability | The SRIOV capability, namely page size and total_vfs of a device are | configured during enumeration phase of the device. This can potentially | interfere with the PCI operations of the platform, if the IOV capability | of the device is not enabled. Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 0321fa3..0dab5ec 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) return rc; } - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); - iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE; pci_cfg_access_lock(dev); pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); @@ -466,6 +464,7 @@ found: return -EIO; pgsz &= ~(pgsz - 1); + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz); nres = 0; for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-01-31 17:44 ` Vaidyanathan Srinivasan @ 2012-02-01 6:21 ` Ram Pai 2012-02-01 13:02 ` Vaidyanathan Srinivasan 0 siblings, 1 reply; 9+ messages in thread From: Ram Pai @ 2012-02-01 6:21 UTC (permalink / raw) To: Vaidyanathan Srinivasan Cc: Ram Pai, Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote: > * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]: > > > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote: > > > Hi Ram and Jesse, > > > > > > I found a trivial issue with page size alignment check on IBM POWER > > > box with 64k base page size. In sriov_init(), changing the check from > > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > > > required to use one of the sriov adapter as PF since the > > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > > > pseries boxes. > > > > > > I think resource_size() could be less than SystemPageSize, but I would > > > like your comments/ack/nack on any consequences of checking for only > > > 4k alignment here in a system with larger base page size. > > > > As per the SRIOV specs, the resource has to be System page size aligned. > > > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB > > page sizes. In your case if your adapter's PF is not supporting 64K page size > > then I think it is not conforming to the PCI SRIOV spec. > > Hi Ram, > > Thanks for the pointer. I did some more experiments and found that > the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was > set to default 4k when we do the query and check resource_size(). > > You were correct, the resource_size() has to come up with 64k on 64k > PAGE_SIZE system. We should not change that check. I was able to > get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before > we do the query. > > This was the case in the original code before you moved these to > sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in > sriov_init(), then I have the following fix that works for me. > > Please review and let me know your comments. > > Thanks, > Vaidy > --- > > pci: set pci sriov page size before reading sriov bar > > For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before > the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k, > so this change is required on powerpc box with 64k base page size. > > This is a regression caused due to moving SRIOV init to sriov_enable(). > > | commit afd24ece5c76af87f6fc477f2747b83a764f161c > | Author: Ram Pai <linuxram@us.ibm.com> > > | PCI: delay configuration of SRIOV capability > | The SRIOV capability, namely page size and total_vfs of a device are > | configured during enumeration phase of the device. This can potentially > | interfere with the PCI operations of the platform, if the IOV capability > | of the device is not enabled. > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > index 0321fa3..0dab5ec 100644 > --- a/drivers/pci/iov.c > +++ b/drivers/pci/iov.c > @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) > return rc; > } > > - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); > - > iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE; > pci_cfg_access_lock(dev); > pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); > @@ -466,6 +464,7 @@ found: > return -EIO; > > pgsz &= ~(pgsz - 1); > + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz); > > nres = 0; > for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c. RP ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-02-01 6:21 ` Ram Pai @ 2012-02-01 13:02 ` Vaidyanathan Srinivasan 2012-02-10 19:54 ` Jesse Barnes 0 siblings, 1 reply; 9+ messages in thread From: Vaidyanathan Srinivasan @ 2012-02-01 13:02 UTC (permalink / raw) To: Ram Pai; +Cc: Jesse Barnes, Yinghai Lu, linux-pci, linux-kernel * Ram Pai <linuxram@us.ibm.com> [2012-02-01 14:21:45]: > On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote: > > * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]: > > > > > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote: > > > > Hi Ram and Jesse, > > > > > > > > I found a trivial issue with page size alignment check on IBM POWER > > > > box with 64k base page size. In sriov_init(), changing the check from > > > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > > > > required to use one of the sriov adapter as PF since the > > > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > > > > pseries boxes. > > > > > > > > I think resource_size() could be less than SystemPageSize, but I would > > > > like your comments/ack/nack on any consequences of checking for only > > > > 4k alignment here in a system with larger base page size. > > > > > > As per the SRIOV specs, the resource has to be System page size aligned. > > > > > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB > > > page sizes. In your case if your adapter's PF is not supporting 64K page size > > > then I think it is not conforming to the PCI SRIOV spec. > > > > Hi Ram, > > > > Thanks for the pointer. I did some more experiments and found that > > the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was > > set to default 4k when we do the query and check resource_size(). > > > > You were correct, the resource_size() has to come up with 64k on 64k > > PAGE_SIZE system. We should not change that check. I was able to > > get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before > > we do the query. > > > > This was the case in the original code before you moved these to > > sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in > > sriov_init(), then I have the following fix that works for me. > > > > Please review and let me know your comments. > > > > Thanks, > > Vaidy > > --- > > > > pci: set pci sriov page size before reading sriov bar > > > > For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before > > the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k, > > so this change is required on powerpc box with 64k base page size. > > > > This is a regression caused due to moving SRIOV init to sriov_enable(). > > > > | commit afd24ece5c76af87f6fc477f2747b83a764f161c > > | Author: Ram Pai <linuxram@us.ibm.com> > > > > | PCI: delay configuration of SRIOV capability > > | The SRIOV capability, namely page size and total_vfs of a device are > > | configured during enumeration phase of the device. This can potentially > > | interfere with the PCI operations of the platform, if the IOV capability > > | of the device is not enabled. > > > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> > > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > > index 0321fa3..0dab5ec 100644 > > --- a/drivers/pci/iov.c > > +++ b/drivers/pci/iov.c > > @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) > > return rc; > > } > > > > - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); > > - > > iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE; > > pci_cfg_access_lock(dev); > > pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); > > @@ -466,6 +464,7 @@ found: > > return -EIO; > > > > pgsz &= ~(pgsz - 1); > > + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz); > > > > nres = 0; > > for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { > > > ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c. Hi Ram, Thanks for the ack. But afd24ece5c76af87f6fc477f2747b83a764f161c has one more change of moving pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, total) to sriov_enable(). This change is required so that we set the PCI_SRIOV_NUM_VF only during sriov_enable. So we should not revert the entire commit, we can just add this change. --Vaidy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-02-01 13:02 ` Vaidyanathan Srinivasan @ 2012-02-10 19:54 ` Jesse Barnes 2012-02-13 3:08 ` Ram Pai 0 siblings, 1 reply; 9+ messages in thread From: Jesse Barnes @ 2012-02-10 19:54 UTC (permalink / raw) To: svaidy; +Cc: Ram Pai, Yinghai Lu, linux-pci, linux-kernel [-- Attachment #1: Type: text/plain, Size: 4719 bytes --] On Wed, 1 Feb 2012 18:32:06 +0530 Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> wrote: > * Ram Pai <linuxram@us.ibm.com> [2012-02-01 14:21:45]: > > > On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote: > > > * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]: > > > > > > > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote: > > > > > Hi Ram and Jesse, > > > > > > > > > > I found a trivial issue with page size alignment check on IBM POWER > > > > > box with 64k base page size. In sriov_init(), changing the check from > > > > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > > > > > required to use one of the sriov adapter as PF since the > > > > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > > > > > pseries boxes. > > > > > > > > > > I think resource_size() could be less than SystemPageSize, but I would > > > > > like your comments/ack/nack on any consequences of checking for only > > > > > 4k alignment here in a system with larger base page size. > > > > > > > > As per the SRIOV specs, the resource has to be System page size aligned. > > > > > > > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB > > > > page sizes. In your case if your adapter's PF is not supporting 64K page size > > > > then I think it is not conforming to the PCI SRIOV spec. > > > > > > Hi Ram, > > > > > > Thanks for the pointer. I did some more experiments and found that > > > the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was > > > set to default 4k when we do the query and check resource_size(). > > > > > > You were correct, the resource_size() has to come up with 64k on 64k > > > PAGE_SIZE system. We should not change that check. I was able to > > > get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before > > > we do the query. > > > > > > This was the case in the original code before you moved these to > > > sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in > > > sriov_init(), then I have the following fix that works for me. > > > > > > Please review and let me know your comments. > > > > > > Thanks, > > > Vaidy > > > --- > > > > > > pci: set pci sriov page size before reading sriov bar > > > > > > For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before > > > the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k, > > > so this change is required on powerpc box with 64k base page size. > > > > > > This is a regression caused due to moving SRIOV init to sriov_enable(). > > > > > > | commit afd24ece5c76af87f6fc477f2747b83a764f161c > > > | Author: Ram Pai <linuxram@us.ibm.com> > > > > > > | PCI: delay configuration of SRIOV capability > > > | The SRIOV capability, namely page size and total_vfs of a device are > > > | configured during enumeration phase of the device. This can potentially > > > | interfere with the PCI operations of the platform, if the IOV capability > > > | of the device is not enabled. > > > > > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> > > > > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > > > index 0321fa3..0dab5ec 100644 > > > --- a/drivers/pci/iov.c > > > +++ b/drivers/pci/iov.c > > > @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) > > > return rc; > > > } > > > > > > - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); > > > - > > > iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE; > > > pci_cfg_access_lock(dev); > > > pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); > > > @@ -466,6 +464,7 @@ found: > > > return -EIO; > > > > > > pgsz &= ~(pgsz - 1); > > > + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz); > > > > > > nres = 0; > > > for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { > > > > > > ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c. > > Hi Ram, > > Thanks for the ack. But afd24ece5c76af87f6fc477f2747b83a764f161c has > one more change of moving > pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, total) to sriov_enable(). > > This change is required so that we set the PCI_SRIOV_NUM_VF only > during sriov_enable. > > So we should not revert the entire commit, we can just add this change. So which is it Ram, the ack or the revert? :) Having the right page size early seems like the right solution... -- Jesse Barnes, Intel Open Source Technology Center [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init 2012-02-10 19:54 ` Jesse Barnes @ 2012-02-13 3:08 ` Ram Pai 0 siblings, 0 replies; 9+ messages in thread From: Ram Pai @ 2012-02-13 3:08 UTC (permalink / raw) To: Jesse Barnes; +Cc: svaidy, Ram Pai, Yinghai Lu, linux-pci, linux-kernel On Fri, Feb 10, 2012 at 11:54:52AM -0800, Jesse Barnes wrote: > On Wed, 1 Feb 2012 18:32:06 +0530 > Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> wrote: > > > * Ram Pai <linuxram@us.ibm.com> [2012-02-01 14:21:45]: > > > > > On Tue, Jan 31, 2012 at 11:14:02PM +0530, Vaidyanathan Srinivasan wrote: > > > > * Ram Pai <linuxram@us.ibm.com> [2012-01-30 11:18:45]: > > > > > > > > > On Sat, Jan 28, 2012 at 12:40:32AM +0530, Vaidyanathan Srinivasan wrote: > > > > > > Hi Ram and Jesse, > > > > > > > > > > > > I found a trivial issue with page size alignment check on IBM POWER > > > > > > box with 64k base page size. In sriov_init(), changing the check from > > > > > > PAGE_SIZE (arch and config dependent) to HW_PAGE_SIZE (always 4k) was > > > > > > required to use one of the sriov adapter as PF since the > > > > > > resource_size() comes up as 0x8000 and PAGE_SIZE would be 0x10000 for > > > > > > pseries boxes. > > > > > > > > > > > > I think resource_size() could be less than SystemPageSize, but I would > > > > > > like your comments/ack/nack on any consequences of checking for only > > > > > > 4k alignment here in a system with larger base page size. > > > > > > > > > > As per the SRIOV specs, the resource has to be System page size aligned. > > > > > > > > > > PFs are required to support 4-KB, 8-KB, 64-KB, 256-KB, 1-MB, and 4-MB > > > > > page sizes. In your case if your adapter's PF is not supporting 64K page size > > > > > then I think it is not conforming to the PCI SRIOV spec. > > > > > > > > Hi Ram, > > > > > > > > Thanks for the pointer. I did some more experiments and found that > > > > the card does support 64k page size, but the PCI_SRIOV_SYS_PGSIZE was > > > > set to default 4k when we do the query and check resource_size(). > > > > > > > > You were correct, the resource_size() has to come up with 64k on 64k > > > > PAGE_SIZE system. We should not change that check. I was able to > > > > get a working solution by setting PCI_SRIOV_SYS_PGSIZE to 64k before > > > > we do the query. > > > > > > > > This was the case in the original code before you moved these to > > > > sriov_enable(). If it is ok to leave the SYS_PGSIZE setting in > > > > sriov_init(), then I have the following fix that works for me. > > > > > > > > Please review and let me know your comments. > > > > > > > > Thanks, > > > > Vaidy > > > > --- > > > > > > > > pci: set pci sriov page size before reading sriov bar > > > > > > > > For an SRIOV device, PCI_SRIOV_SYS_PGSIZE should be set before > > > > the PCI_SRIOV_BAR is queried. The sys pagesize defaults to 4k, > > > > so this change is required on powerpc box with 64k base page size. > > > > > > > > This is a regression caused due to moving SRIOV init to sriov_enable(). > > > > > > > > | commit afd24ece5c76af87f6fc477f2747b83a764f161c > > > > | Author: Ram Pai <linuxram@us.ibm.com> > > > > > > > > | PCI: delay configuration of SRIOV capability > > > > | The SRIOV capability, namely page size and total_vfs of a device are > > > > | configured during enumeration phase of the device. This can potentially > > > > | interfere with the PCI operations of the platform, if the IOV capability > > > > | of the device is not enabled. > > > > > > > > Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com> > > > > > > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > > > > index 0321fa3..0dab5ec 100644 > > > > --- a/drivers/pci/iov.c > > > > +++ b/drivers/pci/iov.c > > > > @@ -347,8 +347,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) > > > > return rc; > > > > } > > > > > > > > - pci_write_config_dword(dev, iov->pos + PCI_SRIOV_SYS_PGSIZE, iov->pgsz); > > > > - > > > > iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE; > > > > pci_cfg_access_lock(dev); > > > > pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); > > > > @@ -466,6 +464,7 @@ found: > > > > return -EIO; > > > > > > > > pgsz &= ~(pgsz - 1); > > > > + pci_write_config_dword(dev, pos + PCI_SRIOV_SYS_PGSIZE, pgsz); > > > > > > > > nres = 0; > > > > for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { > > > > > > > > > ACK. I think it is better to revert afd24ece5c76af87f6fc477f2747b83a764f161c. > > > > Hi Ram, > > > > Thanks for the ack. But afd24ece5c76af87f6fc477f2747b83a764f161c has > > one more change of moving > > pci_write_config_word(dev, pos + PCI_SRIOV_NUM_VF, total) to sriov_enable(). > > > > This change is required so that we set the PCI_SRIOV_NUM_VF only > > during sriov_enable. > > > > So we should not revert the entire commit, we can just add this change. > > So which is it Ram, the ack or the revert? :) Jesse, As Vaidy mentioned, revert is not the right solution. So dont revert. But apply Vaidy's patch. > > Having the right page size early seems like the right solution... Yes. RP ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-02-13 3:08 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-01-27 19:10 [BUGFIX][PATCH] pci: check for 4k resource_size alignment in sriov_init Vaidyanathan Srinivasan 2012-01-27 21:05 ` Yinghai Lu 2012-01-29 13:11 ` Vaidyanathan Srinivasan 2012-01-30 3:18 ` Ram Pai 2012-01-31 17:44 ` Vaidyanathan Srinivasan 2012-02-01 6:21 ` Ram Pai 2012-02-01 13:02 ` Vaidyanathan Srinivasan 2012-02-10 19:54 ` Jesse Barnes 2012-02-13 3:08 ` Ram Pai
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).