* [PATCH] PCI: Check/Set ARI capability before setting numVFs
@ 2017-10-04 15:52 Tony Nguyen
2017-10-04 23:01 ` Bjorn Helgaas
0 siblings, 1 reply; 6+ messages in thread
From: Tony Nguyen @ 2017-10-04 15:52 UTC (permalink / raw)
To: linux-pci, intel-wired-lan, linux-kernel, netdev
Cc: bhelgaas, Tony Nguyen, Alexander Duyck, Emil Tantilov
This fixes a bug that can occur if an AER error is encountered while SRIOV
devices are present.
This issue was seen by doing the following. Inject an AER error to a device
that has SRIOV devices. After the device has recovered, remove the driver.
Reload the driver and enable SRIOV which causes the following crash to
occur:
kernel BUG at drivers/pci/iov.c:157!
invalid opcode: 0000 [#1] SMP
CPU: 36 PID: 2295 Comm: bash Not tainted 4.14.0-rc1+ #74
Hardware name: Supermicro X9DAi/X9DAi, BIOS 3.0a 04/29/2014
task: ffff9fa41cd45a00 task.stack: ffffb4b2036e8000
RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
RSP: 0018:ffffb4b2036ebcb8 EFLAGS: 00010286
RAX: 00000000fffffff0 RBX: ffff9fa42c1c8800 RCX: ffff9fa421ce2388
RDX: 00000000df900000 RSI: ffff9fa8214fb388 RDI: 00000000df903fff
RBP: ffffb4b2036ebd18 R08: ffff9fa421ce23b8 R09: ffffb4b2036ebc2c
R10: ffff9fa42c1a5548 R11: 000000000000058e R12: ffff9fa8214fb000
R13: ffff9fa42c1a5000 R14: ffff9fa8214fb388 R15: 0000000000000000
FS: 00007f60724b6700(0000) GS:ffff9fa82f300000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559eca8b0f40 CR3: 0000000864146000 CR4: 00000000001606e0
Call Trace:
pci_enable_sriov+0x353/0x440
ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
sriov_numvfs_store+0xf7/0x170
dev_attr_store+0x18/0x30
sysfs_kf_write+0x37/0x40
kernfs_fop_write+0x120/0x1b0
__vfs_write+0x37/0x170
? __alloc_fd+0x3f/0x170
? set_close_on_exec+0x30/0x70
vfs_write+0xb5/0x1a0
SyS_write+0x55/0xc0
entry_SYSCALL_64_fastpath+0x1a/0xa5
RIP: 0033:0x7f6071bafc20
RSP: 002b:00007ffe7d42ba48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000559eca8b0f30 RCX: 00007f6071bafc20
RDX: 0000000000000002 RSI: 0000559eca961f60 RDI: 0000000000000001
RBP: 00007f6071e78ae0 R08: 00007f6071e7a740 R09: 00007f60724b6700
R10: 0000000000000073 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000559eca892170
RIP: pci_iov_add_virtfn+0x2eb/0x350 RSP: ffffb4b2036ebcb8
The occurs since during AER recovery the ARI Capable Hierarchy bit,
which can affect the values for First VF Offset and VF Stride, is not set
until after pci_iov_set_numvfs() is called. This can cause the iov
structure to be populated with values that are incorrect if the bit is
later set. Check and set this bit, if needed, before calling
pci_iov_set_numvfs() so that the values being populated properly take
the ARI bit into account.
CC: Alexander Duyck <alexander.h.duyck@intel.com>
CC: Emil Tantilov <emil.s.tantilov@intel.com>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/pci/iov.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 7492a65..a8896c7 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -497,6 +497,10 @@ static void sriov_restore_state(struct pci_dev *dev)
if (ctrl & PCI_SRIOV_CTRL_VFE)
return;
+ if ((iov->ctrl & PCI_SRIOV_CTRL_ARI) && !(ctrl & PCI_SRIOV_CTRL_ARI))
+ pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL,
+ ctrl | PCI_SRIOV_CTRL_ARI);
+
for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
pci_update_resource(dev, i);
--
2.9.5
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] PCI: Check/Set ARI capability before setting numVFs
2017-10-04 15:52 [PATCH] PCI: Check/Set ARI capability before setting numVFs Tony Nguyen
@ 2017-10-04 23:01 ` Bjorn Helgaas
2017-10-04 23:29 ` [Intel-wired-lan] " Alexander Duyck
0 siblings, 1 reply; 6+ messages in thread
From: Bjorn Helgaas @ 2017-10-04 23:01 UTC (permalink / raw)
To: Tony Nguyen
Cc: linux-pci, intel-wired-lan, linux-kernel, netdev, bhelgaas,
Alexander Duyck, Emil Tantilov
On Wed, Oct 04, 2017 at 08:52:58AM -0700, Tony Nguyen wrote:
> This fixes a bug that can occur if an AER error is encountered while SRIOV
> devices are present.
>
> This issue was seen by doing the following. Inject an AER error to a device
> that has SRIOV devices. After the device has recovered, remove the driver.
> Reload the driver and enable SRIOV which causes the following crash to
> occur:
>
> kernel BUG at drivers/pci/iov.c:157!
> invalid opcode: 0000 [#1] SMP
> CPU: 36 PID: 2295 Comm: bash Not tainted 4.14.0-rc1+ #74
> Hardware name: Supermicro X9DAi/X9DAi, BIOS 3.0a 04/29/2014
> task: ffff9fa41cd45a00 task.stack: ffffb4b2036e8000
> RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
> RSP: 0018:ffffb4b2036ebcb8 EFLAGS: 00010286
> RAX: 00000000fffffff0 RBX: ffff9fa42c1c8800 RCX: ffff9fa421ce2388
> RDX: 00000000df900000 RSI: ffff9fa8214fb388 RDI: 00000000df903fff
> RBP: ffffb4b2036ebd18 R08: ffff9fa421ce23b8 R09: ffffb4b2036ebc2c
> R10: ffff9fa42c1a5548 R11: 000000000000058e R12: ffff9fa8214fb000
> R13: ffff9fa42c1a5000 R14: ffff9fa8214fb388 R15: 0000000000000000
> FS: 00007f60724b6700(0000) GS:ffff9fa82f300000(0000)
> knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000559eca8b0f40 CR3: 0000000864146000 CR4: 00000000001606e0
> Call Trace:
> pci_enable_sriov+0x353/0x440
> ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
> sriov_numvfs_store+0xf7/0x170
> dev_attr_store+0x18/0x30
> sysfs_kf_write+0x37/0x40
> kernfs_fop_write+0x120/0x1b0
> __vfs_write+0x37/0x170
> ? __alloc_fd+0x3f/0x170
> ? set_close_on_exec+0x30/0x70
> vfs_write+0xb5/0x1a0
> SyS_write+0x55/0xc0
> entry_SYSCALL_64_fastpath+0x1a/0xa5
> RIP: 0033:0x7f6071bafc20
> RSP: 002b:00007ffe7d42ba48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> RAX: ffffffffffffffda RBX: 0000559eca8b0f30 RCX: 00007f6071bafc20
> RDX: 0000000000000002 RSI: 0000559eca961f60 RDI: 0000000000000001
> RBP: 00007f6071e78ae0 R08: 00007f6071e7a740 R09: 00007f60724b6700
> R10: 0000000000000073 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000559eca892170
> RIP: pci_iov_add_virtfn+0x2eb/0x350 RSP: ffffb4b2036ebcb8
>
> The occurs since during AER recovery the ARI Capable Hierarchy bit,
> which can affect the values for First VF Offset and VF Stride, is not set
> until after pci_iov_set_numvfs() is called.
Can you elaborate on where exactly this happens? The only place we
explicitly set PCI_SRIOV_CTRL_ARI is in sriov_init(), which is only
called at enumeration-time. So I'm guessing you're talking about this
path:
ixgbe_io_slot_reset
pci_restore_state
pci_restore_iov_state
sriov_restore_state
pci_iov_set_numvfs
where we don't set PCI_SRIOV_CTRL_ARI at all. The fact that you say
PCI_SRIOV_CTRL_ARI isn't set until *after* pci_iov_set_numvfs() is
called suggests that it is being set *somewhere*, but I don't know
where.
> This can cause the iov
> structure to be populated with values that are incorrect if the bit is
> later set. Check and set this bit, if needed, before calling
> pci_iov_set_numvfs() so that the values being populated properly take
> the ARI bit into account.
>
> CC: Alexander Duyck <alexander.h.duyck@intel.com>
> CC: Emil Tantilov <emil.s.tantilov@intel.com>
> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
> ---
> drivers/pci/iov.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 7492a65..a8896c7 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -497,6 +497,10 @@ static void sriov_restore_state(struct pci_dev *dev)
> if (ctrl & PCI_SRIOV_CTRL_VFE)
> return;
>
> + if ((iov->ctrl & PCI_SRIOV_CTRL_ARI) && !(ctrl & PCI_SRIOV_CTRL_ARI))
> + pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL,
> + ctrl | PCI_SRIOV_CTRL_ARI);
> +
> for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
> pci_update_resource(dev, i);
>
> --
> 2.9.5
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Intel-wired-lan] [PATCH] PCI: Check/Set ARI capability before setting numVFs
2017-10-04 23:01 ` Bjorn Helgaas
@ 2017-10-04 23:29 ` Alexander Duyck
2017-10-05 21:07 ` Bjorn Helgaas
0 siblings, 1 reply; 6+ messages in thread
From: Alexander Duyck @ 2017-10-04 23:29 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Tony Nguyen, linux-pci, linux-kernel, intel-wired-lan, Netdev,
Bjorn Helgaas
On Wed, Oct 4, 2017 at 4:01 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Wed, Oct 04, 2017 at 08:52:58AM -0700, Tony Nguyen wrote:
>> This fixes a bug that can occur if an AER error is encountered while SRIOV
>> devices are present.
>>
>> This issue was seen by doing the following. Inject an AER error to a device
>> that has SRIOV devices. After the device has recovered, remove the driver.
>> Reload the driver and enable SRIOV which causes the following crash to
>> occur:
>>
>> kernel BUG at drivers/pci/iov.c:157!
>> invalid opcode: 0000 [#1] SMP
>> CPU: 36 PID: 2295 Comm: bash Not tainted 4.14.0-rc1+ #74
>> Hardware name: Supermicro X9DAi/X9DAi, BIOS 3.0a 04/29/2014
>> task: ffff9fa41cd45a00 task.stack: ffffb4b2036e8000
>> RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
>> RSP: 0018:ffffb4b2036ebcb8 EFLAGS: 00010286
>> RAX: 00000000fffffff0 RBX: ffff9fa42c1c8800 RCX: ffff9fa421ce2388
>> RDX: 00000000df900000 RSI: ffff9fa8214fb388 RDI: 00000000df903fff
>> RBP: ffffb4b2036ebd18 R08: ffff9fa421ce23b8 R09: ffffb4b2036ebc2c
>> R10: ffff9fa42c1a5548 R11: 000000000000058e R12: ffff9fa8214fb000
>> R13: ffff9fa42c1a5000 R14: ffff9fa8214fb388 R15: 0000000000000000
>> FS: 00007f60724b6700(0000) GS:ffff9fa82f300000(0000)
>> knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000559eca8b0f40 CR3: 0000000864146000 CR4: 00000000001606e0
>> Call Trace:
>> pci_enable_sriov+0x353/0x440
>> ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
>> sriov_numvfs_store+0xf7/0x170
>> dev_attr_store+0x18/0x30
>> sysfs_kf_write+0x37/0x40
>> kernfs_fop_write+0x120/0x1b0
>> __vfs_write+0x37/0x170
>> ? __alloc_fd+0x3f/0x170
>> ? set_close_on_exec+0x30/0x70
>> vfs_write+0xb5/0x1a0
>> SyS_write+0x55/0xc0
>> entry_SYSCALL_64_fastpath+0x1a/0xa5
>> RIP: 0033:0x7f6071bafc20
>> RSP: 002b:00007ffe7d42ba48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>> RAX: ffffffffffffffda RBX: 0000559eca8b0f30 RCX: 00007f6071bafc20
>> RDX: 0000000000000002 RSI: 0000559eca961f60 RDI: 0000000000000001
>> RBP: 00007f6071e78ae0 R08: 00007f6071e7a740 R09: 00007f60724b6700
>> R10: 0000000000000073 R11: 0000000000000246 R12: 0000000000000000
>> R13: 0000000000000000 R14: 0000000000000000 R15: 0000559eca892170
>> RIP: pci_iov_add_virtfn+0x2eb/0x350 RSP: ffffb4b2036ebcb8
>>
>> The occurs since during AER recovery the ARI Capable Hierarchy bit,
>> which can affect the values for First VF Offset and VF Stride, is not set
>> until after pci_iov_set_numvfs() is called.
>
> Can you elaborate on where exactly this happens? The only place we
> explicitly set PCI_SRIOV_CTRL_ARI is in sriov_init(), which is only
> called at enumeration-time. So I'm guessing you're talking about this
> path:
>
> ixgbe_io_slot_reset
> pci_restore_state
> pci_restore_iov_state
> sriov_restore_state
> pci_iov_set_numvfs
>
> where we don't set PCI_SRIOV_CTRL_ARI at all. The fact that you say
> PCI_SRIOV_CTRL_ARI isn't set until *after* pci_iov_set_numvfs() is
> called suggests that it is being set *somewhere*, but I don't know
> where.
The ARI bit is initialized in sriov_init, stored in iov->ctrl, and
restored in sriov_restore_state, but it occurs in the line after the
call to pci_iov_set_numvfs.
The problem is you don't want to write the full iov->ctrl value until
after you have reset the the number of VFs since it will set VFE so
pulling out and configuring the ARI value separately is needed.
>> This can cause the iov
>> structure to be populated with values that are incorrect if the bit is
>> later set. Check and set this bit, if needed, before calling
>> pci_iov_set_numvfs() so that the values being populated properly take
>> the ARI bit into account.
>>
>> CC: Alexander Duyck <alexander.h.duyck@intel.com>
>> CC: Emil Tantilov <emil.s.tantilov@intel.com>
>> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
>> ---
>> drivers/pci/iov.c | 4 ++++
>> 1 file changed, 4 insertions(+)
>>
>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>> index 7492a65..a8896c7 100644
>> --- a/drivers/pci/iov.c
>> +++ b/drivers/pci/iov.c
>> @@ -497,6 +497,10 @@ static void sriov_restore_state(struct pci_dev *dev)
>> if (ctrl & PCI_SRIOV_CTRL_VFE)
>> return;
>>
>> + if ((iov->ctrl & PCI_SRIOV_CTRL_ARI) && !(ctrl & PCI_SRIOV_CTRL_ARI))
>> + pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL,
>> + ctrl | PCI_SRIOV_CTRL_ARI);
>> +
>> for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
>> pci_update_resource(dev, i);
>>
>> --
>> 2.9.5
>>
> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan@osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Intel-wired-lan] [PATCH] PCI: Check/Set ARI capability before setting numVFs
2017-10-04 23:29 ` [Intel-wired-lan] " Alexander Duyck
@ 2017-10-05 21:07 ` Bjorn Helgaas
2017-10-05 21:29 ` Alexander Duyck
2017-10-06 18:19 ` Bjorn Helgaas
0 siblings, 2 replies; 6+ messages in thread
From: Bjorn Helgaas @ 2017-10-05 21:07 UTC (permalink / raw)
To: Alexander Duyck
Cc: Tony Nguyen, linux-pci, linux-kernel, intel-wired-lan, Netdev,
Bjorn Helgaas
On Wed, Oct 04, 2017 at 04:29:14PM -0700, Alexander Duyck wrote:
> On Wed, Oct 4, 2017 at 4:01 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Wed, Oct 04, 2017 at 08:52:58AM -0700, Tony Nguyen wrote:
> >> This fixes a bug that can occur if an AER error is encountered while SRIOV
> >> devices are present.
> >>
> >> This issue was seen by doing the following. Inject an AER error to a device
> >> that has SRIOV devices. After the device has recovered, remove the driver.
> >> Reload the driver and enable SRIOV which causes the following crash to
> >> occur:
> >>
> >> kernel BUG at drivers/pci/iov.c:157!
> >> invalid opcode: 0000 [#1] SMP
> >> CPU: 36 PID: 2295 Comm: bash Not tainted 4.14.0-rc1+ #74
> >> Hardware name: Supermicro X9DAi/X9DAi, BIOS 3.0a 04/29/2014
> >> task: ffff9fa41cd45a00 task.stack: ffffb4b2036e8000
> >> RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
> >> RSP: 0018:ffffb4b2036ebcb8 EFLAGS: 00010286
> >> RAX: 00000000fffffff0 RBX: ffff9fa42c1c8800 RCX: ffff9fa421ce2388
> >> RDX: 00000000df900000 RSI: ffff9fa8214fb388 RDI: 00000000df903fff
> >> RBP: ffffb4b2036ebd18 R08: ffff9fa421ce23b8 R09: ffffb4b2036ebc2c
> >> R10: ffff9fa42c1a5548 R11: 000000000000058e R12: ffff9fa8214fb000
> >> R13: ffff9fa42c1a5000 R14: ffff9fa8214fb388 R15: 0000000000000000
> >> FS: 00007f60724b6700(0000) GS:ffff9fa82f300000(0000)
> >> knlGS:0000000000000000
> >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >> CR2: 0000559eca8b0f40 CR3: 0000000864146000 CR4: 00000000001606e0
> >> Call Trace:
> >> pci_enable_sriov+0x353/0x440
> >> ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
> >> sriov_numvfs_store+0xf7/0x170
> >> dev_attr_store+0x18/0x30
> >> sysfs_kf_write+0x37/0x40
> >> kernfs_fop_write+0x120/0x1b0
> >> __vfs_write+0x37/0x170
> >> ? __alloc_fd+0x3f/0x170
> >> ? set_close_on_exec+0x30/0x70
> >> vfs_write+0xb5/0x1a0
> >> SyS_write+0x55/0xc0
> >> entry_SYSCALL_64_fastpath+0x1a/0xa5
> >> RIP: 0033:0x7f6071bafc20
> >> RSP: 002b:00007ffe7d42ba48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> >> RAX: ffffffffffffffda RBX: 0000559eca8b0f30 RCX: 00007f6071bafc20
> >> RDX: 0000000000000002 RSI: 0000559eca961f60 RDI: 0000000000000001
> >> RBP: 00007f6071e78ae0 R08: 00007f6071e7a740 R09: 00007f60724b6700
> >> R10: 0000000000000073 R11: 0000000000000246 R12: 0000000000000000
> >> R13: 0000000000000000 R14: 0000000000000000 R15: 0000559eca892170
> >> RIP: pci_iov_add_virtfn+0x2eb/0x350 RSP: ffffb4b2036ebcb8
> >>
> >> The occurs since during AER recovery the ARI Capable Hierarchy bit,
> >> which can affect the values for First VF Offset and VF Stride, is not set
> >> until after pci_iov_set_numvfs() is called.
> >
> > Can you elaborate on where exactly this happens? The only place we
> > explicitly set PCI_SRIOV_CTRL_ARI is in sriov_init(), which is only
> > called at enumeration-time. So I'm guessing you're talking about this
> > path:
> >
> > ixgbe_io_slot_reset
> > pci_restore_state
> > pci_restore_iov_state
> > sriov_restore_state
> > pci_iov_set_numvfs
> >
> > where we don't set PCI_SRIOV_CTRL_ARI at all. The fact that you say
> > PCI_SRIOV_CTRL_ARI isn't set until *after* pci_iov_set_numvfs() is
> > called suggests that it is being set *somewhere*, but I don't know
> > where.
>
> The ARI bit is initialized in sriov_init, stored in iov->ctrl, and
> restored in sriov_restore_state, but it occurs in the line after the
> call to pci_iov_set_numvfs.
>
> The problem is you don't want to write the full iov->ctrl value until
> after you have reset the the number of VFs since it will set VFE so
> pulling out and configuring the ARI value separately is needed.
Doh, that should have been obvious to me ;)
> >> This can cause the iov
> >> structure to be populated with values that are incorrect if the bit is
> >> later set. Check and set this bit, if needed, before calling
> >> pci_iov_set_numvfs() so that the values being populated properly take
> >> the ARI bit into account.
> >>
> >> CC: Alexander Duyck <alexander.h.duyck@intel.com>
> >> CC: Emil Tantilov <emil.s.tantilov@intel.com>
> >> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
> >> ---
> >> drivers/pci/iov.c | 4 ++++
> >> 1 file changed, 4 insertions(+)
> >>
> >> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> >> index 7492a65..a8896c7 100644
> >> --- a/drivers/pci/iov.c
> >> +++ b/drivers/pci/iov.c
> >> @@ -497,6 +497,10 @@ static void sriov_restore_state(struct pci_dev *dev)
> >> if (ctrl & PCI_SRIOV_CTRL_VFE)
> >> return;
> >>
> >> + if ((iov->ctrl & PCI_SRIOV_CTRL_ARI) && !(ctrl & PCI_SRIOV_CTRL_ARI))
> >> + pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL,
> >> + ctrl | PCI_SRIOV_CTRL_ARI);
This looks a little fiddly and also assumes that we only ever need to
*set* PCI_SRIOV_CTRL_ARI. That's likely the case because it's
probably cleared after reset and during resume. But I'm not *sure*
that's always the case, so what do you think about the proposal below?
> >> for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
> >> pci_update_resource(dev, i);
> >>
commit 95594dedd443e42ab0c16b9fba0109e955e7be13
Author: Tony Nguyen <anthony.l.nguyen@intel.com>
Date: Wed Oct 4 08:52:58 2017 -0700
PCI: Restore "ARI Capable Hierarchy" before setting numVFs
In the restore path, we previously read PCI_SRIOV_VF_OFFSET and
PCI_SRIOV_VF_STRIDE before restoring PCI_SRIOV_CTRL_ARI, which
affects the offset and stride:
pci_restore_state
pci_restore_iov_state
sriov_restore_state
pci_iov_set_numvfs
pci_read_config_word(... PCI_SRIOV_VF_OFFSET, &iov->offset)
pci_write_config_word(... PCI_SRIOV_CTRL, iov->ctrl)
The effect is that suspend/resume and AER recovery, which use
pci_restore_state(), may corrupt iov->offset and iov->stride. The iov
state is associated with the device, not the driver, so if we reload the
driver, it will use the the corrupted data, which may cause crashes like
this:
kernel BUG at drivers/pci/iov.c:157!
RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
Call Trace:
pci_enable_sriov+0x353/0x440
ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
sriov_numvfs_store+0xf7/0x170
dev_attr_store+0x18/0x30
sysfs_kf_write+0x37/0x40
kernfs_fop_write+0x120/0x1b0
vfs_write+0xb5/0x1a0
SyS_write+0x55/0xc0
The occurs since during AER recovery the ARI Capable Hierarchy bit, which
can affect the values for First VF Offset and VF Stride, is not set until
after pci_iov_set_numvfs() is called. This can cause the iov structure to
be populated with values that are incorrect if the bit is later set.
Check and set this bit, if needed, before calling pci_iov_set_numvfs() so
that the values being populated properly take the ARI bit into account.
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
[bhelgaas: changelog, add comment, also clear ARI if necessary]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
CC: Alexander Duyck <alexander.h.duyck@intel.com>
CC: Emil Tantilov <emil.s.tantilov@intel.com>
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index ce24cf235f01..6bacb8995e96 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -498,6 +498,14 @@ static void sriov_restore_state(struct pci_dev *dev)
if (ctrl & PCI_SRIOV_CTRL_VFE)
return;
+ /*
+ * Restore PCI_SRIOV_CTRL_ARI before pci_iov_set_numvfs() because
+ * it reads offset & stride, which depend on PCI_SRIOV_CTRL_ARI.
+ */
+ ctrl &= ~PCI_SRIOV_CTRL_ARI;
+ ctrl |= iov->ctrl & PCI_SRIOV_CTRL_ARI;
+ pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, ctrl);
+
for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
pci_update_resource(dev, i);
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Intel-wired-lan] [PATCH] PCI: Check/Set ARI capability before setting numVFs
2017-10-05 21:07 ` Bjorn Helgaas
@ 2017-10-05 21:29 ` Alexander Duyck
2017-10-06 18:19 ` Bjorn Helgaas
1 sibling, 0 replies; 6+ messages in thread
From: Alexander Duyck @ 2017-10-05 21:29 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Tony Nguyen, linux-pci, linux-kernel, intel-wired-lan, Netdev,
Bjorn Helgaas
On Thu, Oct 5, 2017 at 2:07 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Wed, Oct 04, 2017 at 04:29:14PM -0700, Alexander Duyck wrote:
>> On Wed, Oct 4, 2017 at 4:01 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>> > On Wed, Oct 04, 2017 at 08:52:58AM -0700, Tony Nguyen wrote:
>> >> This fixes a bug that can occur if an AER error is encountered while SRIOV
>> >> devices are present.
>> >>
>> >> This issue was seen by doing the following. Inject an AER error to a device
>> >> that has SRIOV devices. After the device has recovered, remove the driver.
>> >> Reload the driver and enable SRIOV which causes the following crash to
>> >> occur:
>> >>
>> >> kernel BUG at drivers/pci/iov.c:157!
>> >> invalid opcode: 0000 [#1] SMP
>> >> CPU: 36 PID: 2295 Comm: bash Not tainted 4.14.0-rc1+ #74
>> >> Hardware name: Supermicro X9DAi/X9DAi, BIOS 3.0a 04/29/2014
>> >> task: ffff9fa41cd45a00 task.stack: ffffb4b2036e8000
>> >> RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
>> >> RSP: 0018:ffffb4b2036ebcb8 EFLAGS: 00010286
>> >> RAX: 00000000fffffff0 RBX: ffff9fa42c1c8800 RCX: ffff9fa421ce2388
>> >> RDX: 00000000df900000 RSI: ffff9fa8214fb388 RDI: 00000000df903fff
>> >> RBP: ffffb4b2036ebd18 R08: ffff9fa421ce23b8 R09: ffffb4b2036ebc2c
>> >> R10: ffff9fa42c1a5548 R11: 000000000000058e R12: ffff9fa8214fb000
>> >> R13: ffff9fa42c1a5000 R14: ffff9fa8214fb388 R15: 0000000000000000
>> >> FS: 00007f60724b6700(0000) GS:ffff9fa82f300000(0000)
>> >> knlGS:0000000000000000
>> >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> >> CR2: 0000559eca8b0f40 CR3: 0000000864146000 CR4: 00000000001606e0
>> >> Call Trace:
>> >> pci_enable_sriov+0x353/0x440
>> >> ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
>> >> sriov_numvfs_store+0xf7/0x170
>> >> dev_attr_store+0x18/0x30
>> >> sysfs_kf_write+0x37/0x40
>> >> kernfs_fop_write+0x120/0x1b0
>> >> __vfs_write+0x37/0x170
>> >> ? __alloc_fd+0x3f/0x170
>> >> ? set_close_on_exec+0x30/0x70
>> >> vfs_write+0xb5/0x1a0
>> >> SyS_write+0x55/0xc0
>> >> entry_SYSCALL_64_fastpath+0x1a/0xa5
>> >> RIP: 0033:0x7f6071bafc20
>> >> RSP: 002b:00007ffe7d42ba48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>> >> RAX: ffffffffffffffda RBX: 0000559eca8b0f30 RCX: 00007f6071bafc20
>> >> RDX: 0000000000000002 RSI: 0000559eca961f60 RDI: 0000000000000001
>> >> RBP: 00007f6071e78ae0 R08: 00007f6071e7a740 R09: 00007f60724b6700
>> >> R10: 0000000000000073 R11: 0000000000000246 R12: 0000000000000000
>> >> R13: 0000000000000000 R14: 0000000000000000 R15: 0000559eca892170
>> >> RIP: pci_iov_add_virtfn+0x2eb/0x350 RSP: ffffb4b2036ebcb8
>> >>
>> >> The occurs since during AER recovery the ARI Capable Hierarchy bit,
>> >> which can affect the values for First VF Offset and VF Stride, is not set
>> >> until after pci_iov_set_numvfs() is called.
>> >
>> > Can you elaborate on where exactly this happens? The only place we
>> > explicitly set PCI_SRIOV_CTRL_ARI is in sriov_init(), which is only
>> > called at enumeration-time. So I'm guessing you're talking about this
>> > path:
>> >
>> > ixgbe_io_slot_reset
>> > pci_restore_state
>> > pci_restore_iov_state
>> > sriov_restore_state
>> > pci_iov_set_numvfs
>> >
>> > where we don't set PCI_SRIOV_CTRL_ARI at all. The fact that you say
>> > PCI_SRIOV_CTRL_ARI isn't set until *after* pci_iov_set_numvfs() is
>> > called suggests that it is being set *somewhere*, but I don't know
>> > where.
>>
>> The ARI bit is initialized in sriov_init, stored in iov->ctrl, and
>> restored in sriov_restore_state, but it occurs in the line after the
>> call to pci_iov_set_numvfs.
>>
>> The problem is you don't want to write the full iov->ctrl value until
>> after you have reset the the number of VFs since it will set VFE so
>> pulling out and configuring the ARI value separately is needed.
>
> Doh, that should have been obvious to me ;)
>
>> >> This can cause the iov
>> >> structure to be populated with values that are incorrect if the bit is
>> >> later set. Check and set this bit, if needed, before calling
>> >> pci_iov_set_numvfs() so that the values being populated properly take
>> >> the ARI bit into account.
>> >>
>> >> CC: Alexander Duyck <alexander.h.duyck@intel.com>
>> >> CC: Emil Tantilov <emil.s.tantilov@intel.com>
>> >> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
>> >> ---
>> >> drivers/pci/iov.c | 4 ++++
>> >> 1 file changed, 4 insertions(+)
>> >>
>> >> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>> >> index 7492a65..a8896c7 100644
>> >> --- a/drivers/pci/iov.c
>> >> +++ b/drivers/pci/iov.c
>> >> @@ -497,6 +497,10 @@ static void sriov_restore_state(struct pci_dev *dev)
>> >> if (ctrl & PCI_SRIOV_CTRL_VFE)
>> >> return;
>> >>
>> >> + if ((iov->ctrl & PCI_SRIOV_CTRL_ARI) && !(ctrl & PCI_SRIOV_CTRL_ARI))
>> >> + pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL,
>> >> + ctrl | PCI_SRIOV_CTRL_ARI);
>
> This looks a little fiddly and also assumes that we only ever need to
> *set* PCI_SRIOV_CTRL_ARI. That's likely the case because it's
> probably cleared after reset and during resume. But I'm not *sure*
> that's always the case, so what do you think about the proposal below?
>
>> >> for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
>> >> pci_update_resource(dev, i);
>> >>
>
>
> commit 95594dedd443e42ab0c16b9fba0109e955e7be13
> Author: Tony Nguyen <anthony.l.nguyen@intel.com>
> Date: Wed Oct 4 08:52:58 2017 -0700
>
> PCI: Restore "ARI Capable Hierarchy" before setting numVFs
>
> In the restore path, we previously read PCI_SRIOV_VF_OFFSET and
> PCI_SRIOV_VF_STRIDE before restoring PCI_SRIOV_CTRL_ARI, which
> affects the offset and stride:
>
> pci_restore_state
> pci_restore_iov_state
> sriov_restore_state
> pci_iov_set_numvfs
> pci_read_config_word(... PCI_SRIOV_VF_OFFSET, &iov->offset)
> pci_write_config_word(... PCI_SRIOV_CTRL, iov->ctrl)
>
> The effect is that suspend/resume and AER recovery, which use
> pci_restore_state(), may corrupt iov->offset and iov->stride. The iov
> state is associated with the device, not the driver, so if we reload the
> driver, it will use the the corrupted data, which may cause crashes like
> this:
>
> kernel BUG at drivers/pci/iov.c:157!
> RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
> Call Trace:
> pci_enable_sriov+0x353/0x440
> ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
> sriov_numvfs_store+0xf7/0x170
> dev_attr_store+0x18/0x30
> sysfs_kf_write+0x37/0x40
> kernfs_fop_write+0x120/0x1b0
> vfs_write+0xb5/0x1a0
> SyS_write+0x55/0xc0
>
> The occurs since during AER recovery the ARI Capable Hierarchy bit, which
> can affect the values for First VF Offset and VF Stride, is not set until
> after pci_iov_set_numvfs() is called. This can cause the iov structure to
> be populated with values that are incorrect if the bit is later set.
> Check and set this bit, if needed, before calling pci_iov_set_numvfs() so
> that the values being populated properly take the ARI bit into account.
>
> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
> [bhelgaas: changelog, add comment, also clear ARI if necessary]
> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> CC: Alexander Duyck <alexander.h.duyck@intel.com>
> CC: Emil Tantilov <emil.s.tantilov@intel.com>
>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index ce24cf235f01..6bacb8995e96 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -498,6 +498,14 @@ static void sriov_restore_state(struct pci_dev *dev)
> if (ctrl & PCI_SRIOV_CTRL_VFE)
> return;
>
> + /*
> + * Restore PCI_SRIOV_CTRL_ARI before pci_iov_set_numvfs() because
> + * it reads offset & stride, which depend on PCI_SRIOV_CTRL_ARI.
> + */
> + ctrl &= ~PCI_SRIOV_CTRL_ARI;
> + ctrl |= iov->ctrl & PCI_SRIOV_CTRL_ARI;
> + pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, ctrl);
> +
> for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
> pci_update_resource(dev, i);
>
I would think that should work. I don't see any issues since we have
already verified in the previous line that VFE is cleared.
Acked-by: Alexander Duyck <alexander.h.duyck@intel.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Intel-wired-lan] [PATCH] PCI: Check/Set ARI capability before setting numVFs
2017-10-05 21:07 ` Bjorn Helgaas
2017-10-05 21:29 ` Alexander Duyck
@ 2017-10-06 18:19 ` Bjorn Helgaas
1 sibling, 0 replies; 6+ messages in thread
From: Bjorn Helgaas @ 2017-10-06 18:19 UTC (permalink / raw)
To: Alexander Duyck
Cc: Tony Nguyen, linux-pci, linux-kernel, intel-wired-lan, Netdev,
Bjorn Helgaas
On Thu, Oct 05, 2017 at 04:07:41PM -0500, Bjorn Helgaas wrote:
> On Wed, Oct 04, 2017 at 04:29:14PM -0700, Alexander Duyck wrote:
> > On Wed, Oct 4, 2017 at 4:01 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Wed, Oct 04, 2017 at 08:52:58AM -0700, Tony Nguyen wrote:
> > >> This fixes a bug that can occur if an AER error is encountered while SRIOV
> > >> devices are present.
I applied the patch below with Alex's ack to pci/virtualization for v4.15.
> commit 95594dedd443e42ab0c16b9fba0109e955e7be13
> Author: Tony Nguyen <anthony.l.nguyen@intel.com>
> Date: Wed Oct 4 08:52:58 2017 -0700
>
> PCI: Restore "ARI Capable Hierarchy" before setting numVFs
>
> In the restore path, we previously read PCI_SRIOV_VF_OFFSET and
> PCI_SRIOV_VF_STRIDE before restoring PCI_SRIOV_CTRL_ARI, which
> affects the offset and stride:
>
> pci_restore_state
> pci_restore_iov_state
> sriov_restore_state
> pci_iov_set_numvfs
> pci_read_config_word(... PCI_SRIOV_VF_OFFSET, &iov->offset)
> pci_write_config_word(... PCI_SRIOV_CTRL, iov->ctrl)
>
> The effect is that suspend/resume and AER recovery, which use
> pci_restore_state(), may corrupt iov->offset and iov->stride. The iov
> state is associated with the device, not the driver, so if we reload the
> driver, it will use the the corrupted data, which may cause crashes like
> this:
>
> kernel BUG at drivers/pci/iov.c:157!
> RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
> Call Trace:
> pci_enable_sriov+0x353/0x440
> ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
> sriov_numvfs_store+0xf7/0x170
> dev_attr_store+0x18/0x30
> sysfs_kf_write+0x37/0x40
> kernfs_fop_write+0x120/0x1b0
> vfs_write+0xb5/0x1a0
> SyS_write+0x55/0xc0
>
> The occurs since during AER recovery the ARI Capable Hierarchy bit, which
> can affect the values for First VF Offset and VF Stride, is not set until
> after pci_iov_set_numvfs() is called. This can cause the iov structure to
> be populated with values that are incorrect if the bit is later set.
> Check and set this bit, if needed, before calling pci_iov_set_numvfs() so
> that the values being populated properly take the ARI bit into account.
>
> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
> [bhelgaas: changelog, add comment, also clear ARI if necessary]
> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> CC: Alexander Duyck <alexander.h.duyck@intel.com>
> CC: Emil Tantilov <emil.s.tantilov@intel.com>
>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index ce24cf235f01..6bacb8995e96 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -498,6 +498,14 @@ static void sriov_restore_state(struct pci_dev *dev)
> if (ctrl & PCI_SRIOV_CTRL_VFE)
> return;
>
> + /*
> + * Restore PCI_SRIOV_CTRL_ARI before pci_iov_set_numvfs() because
> + * it reads offset & stride, which depend on PCI_SRIOV_CTRL_ARI.
> + */
> + ctrl &= ~PCI_SRIOV_CTRL_ARI;
> + ctrl |= iov->ctrl & PCI_SRIOV_CTRL_ARI;
> + pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, ctrl);
> +
> for (i = PCI_IOV_RESOURCES; i <= PCI_IOV_RESOURCE_END; i++)
> pci_update_resource(dev, i);
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-10-06 18:19 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-10-04 15:52 [PATCH] PCI: Check/Set ARI capability before setting numVFs Tony Nguyen
2017-10-04 23:01 ` Bjorn Helgaas
2017-10-04 23:29 ` [Intel-wired-lan] " Alexander Duyck
2017-10-05 21:07 ` Bjorn Helgaas
2017-10-05 21:29 ` Alexander Duyck
2017-10-06 18:19 ` Bjorn Helgaas
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).