linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
@ 2021-07-15  8:08 Jia He
  2021-07-15 19:50 ` patchwork-bot+netdevbpf
  2021-07-19 10:35 ` Prabhakar Kushwaha
  0 siblings, 2 replies; 9+ messages in thread
From: Jia He @ 2021-07-15  8:08 UTC (permalink / raw)
  To: Ariel Elior, GR-everest-linux-l2, David S. Miller, Jakub Kicinski
  Cc: netdev, linux-kernel, nd, Jia He

Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
QL41000 ethernet controller:
 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
  pstate: 00400009 (nzcv daif +PAN -UAO)
 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18

In this case, qed_hw_prepare() returns error due to hw/fw error, but in
theory work queue should be in process context instead of interrupt.

The root cause might be the unpaired spin_{un}lock_bh() in
_qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.

Reported-by: Lijian Zhang <Lijian.Zhang@arm.com>
Signed-off-by: Jia He <justin.he@arm.com>
---
 drivers/net/ethernet/qlogic/qed/qed_mcp.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.c b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
index 4387292c37e2..79d879a5d663 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
@@ -474,14 +474,18 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 
 		spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
 
-		if (!qed_mcp_has_pending_cmd(p_hwfn))
+		if (!qed_mcp_has_pending_cmd(p_hwfn)) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
+		}
 
 		rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
-		if (!rc)
+		if (!rc) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
-		else if (rc != -EAGAIN)
+		} else if (rc != -EAGAIN) {
 			goto err;
+		}
 
 		spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 
@@ -498,6 +502,8 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 		return -EAGAIN;
 	}
 
+	spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+
 	/* Send the mailbox command */
 	qed_mcp_reread_offsets(p_hwfn, p_ptt);
 	seq_num = ++p_hwfn->mcp_info->drv_mb_seq;
@@ -524,14 +530,18 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 
 		spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
 
-		if (p_cmd_elem->b_is_completed)
+		if (p_cmd_elem->b_is_completed) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
+		}
 
 		rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
-		if (!rc)
+		if (!rc) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
-		else if (rc != -EAGAIN)
+		} else if (rc != -EAGAIN) {
 			goto err;
+		}
 
 		spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 	} while (++cnt < max_retries);
@@ -554,6 +564,7 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 		return -EAGAIN;
 	}
 
+	spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
 	qed_mcp_cmd_del_elem(p_hwfn, p_cmd_elem);
 	spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-15  8:08 [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union() Jia He
@ 2021-07-15 19:50 ` patchwork-bot+netdevbpf
  2021-07-15 23:42   ` Justin He
  2021-07-19 10:35 ` Prabhakar Kushwaha
  1 sibling, 1 reply; 9+ messages in thread
From: patchwork-bot+netdevbpf @ 2021-07-15 19:50 UTC (permalink / raw)
  To: Jia He; +Cc: aelior, GR-everest-linux-l2, davem, kuba, netdev, linux-kernel, nd

Hello:

This patch was applied to netdev/net.git (refs/heads/master):

On Thu, 15 Jul 2021 16:08:21 +0800 you wrote:
> Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> QL41000 ethernet controller:
>  BUG: scheduling while atomic: kworker/0:4/531/0x00000200
>   [qed_probe:488()]hw prepare failed
>   kernel BUG at mm/vmalloc.c:2355!
>   Internal error: Oops - BUG: 0 [#1] SMP
>   CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
>   pstate: 00400009 (nzcv daif +PAN -UAO)
>  Call trace:
>   vunmap+0x4c/0x50
>   iounmap+0x48/0x58
>   qed_free_pci+0x60/0x80 [qed]
>   qed_probe+0x35c/0x688 [qed]
>   __qede_probe+0x88/0x5c8 [qede]
>   qede_probe+0x60/0xe0 [qede]
>   local_pci_probe+0x48/0xa0
>   work_for_cpu_fn+0x24/0x38
>   process_one_work+0x1d0/0x468
>   worker_thread+0x238/0x4e0
>   kthread+0xf0/0x118
>   ret_from_fork+0x10/0x18
> 
> [...]

Here is the summary with links:
  - qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
    https://git.kernel.org/netdev/net/c/6206b7981a36

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html



^ permalink raw reply	[flat|nested] 9+ messages in thread

* RE: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-15 19:50 ` patchwork-bot+netdevbpf
@ 2021-07-15 23:42   ` Justin He
  0 siblings, 0 replies; 9+ messages in thread
From: Justin He @ 2021-07-15 23:42 UTC (permalink / raw)
  To: patchwork-bot+netdevbpf
  Cc: aelior, GR-everest-linux-l2, davem, kuba, netdev, linux-kernel, nd



> -----Original Message-----
> From: patchwork-bot+netdevbpf@kernel.org <patchwork-
> bot+netdevbpf@kernel.org>
> Sent: Friday, July 16, 2021 3:50 AM
> To: Justin He <Justin.He@arm.com>
> Cc: aelior@marvell.com; GR-everest-linux-l2@marvell.com;
> davem@davemloft.net; kuba@kernel.org; netdev@vger.kernel.org; linux-
> kernel@vger.kernel.org; nd <nd@arm.com>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
> 
> Hello:
> 
> This patch was applied to netdev/net.git (refs/heads/master):
> 
> On Thu, 15 Jul 2021 16:08:21 +0800 you wrote:
> > Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> > QL41000 ethernet controller:
> >  BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> >   [qed_probe:488()]hw prepare failed
> >   kernel BUG at mm/vmalloc.c:2355!
> >   Internal error: Oops - BUG: 0 [#1] SMP
> >   CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
> Ubuntu
> >   pstate: 00400009 (nzcv daif +PAN -UAO)
> >  Call trace:
> >   vunmap+0x4c/0x50
> >   iounmap+0x48/0x58
> >   qed_free_pci+0x60/0x80 [qed]
> >   qed_probe+0x35c/0x688 [qed]
> >   __qede_probe+0x88/0x5c8 [qede]
> >   qede_probe+0x60/0xe0 [qede]
> >   local_pci_probe+0x48/0xa0
> >   work_for_cpu_fn+0x24/0x38
> >   process_one_work+0x1d0/0x468
> >   worker_thread+0x238/0x4e0
> >   kthread+0xf0/0x118
> >   ret_from_fork+0x10/0x18
> >
> > [...]
> 
> Here is the summary with links:
>   - qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
>     https://git.kernel.org/netdev/net/c/6206b7981a36
> 
> You are awesome, thank you!

Thanks.
If possible, please also Cc: stable@kernel.org because the bug seemed to
be there for a long time.


--
Cheers,
Justin (Jia He)



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-15  8:08 [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union() Jia He
  2021-07-15 19:50 ` patchwork-bot+netdevbpf
@ 2021-07-19 10:35 ` Prabhakar Kushwaha
  2021-07-19 13:16   ` Justin He
  1 sibling, 1 reply; 9+ messages in thread
From: Prabhakar Kushwaha @ 2021-07-19 10:35 UTC (permalink / raw)
  To: Jia He
  Cc: Ariel Elior, GR-everest-linux-l2, David S. Miller,
	Jakub Kicinski, netdev, Linux Kernel Mailing List, nd,
	Shai Malin, Shai Malin, Prabhakar Kushwaha

Hi Jia,

On Thu, Jul 15, 2021 at 2:28 PM Jia He <justin.he@arm.com> wrote:
>
> Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> QL41000 ethernet controller:
>  BUG: scheduling while atomic: kworker/0:4/531/0x00000200
>   [qed_probe:488()]hw prepare failed
>   kernel BUG at mm/vmalloc.c:2355!
>   Internal error: Oops - BUG: 0 [#1] SMP
>   CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
>   pstate: 00400009 (nzcv daif +PAN -UAO)
>  Call trace:
>   vunmap+0x4c/0x50
>   iounmap+0x48/0x58
>   qed_free_pci+0x60/0x80 [qed]
>   qed_probe+0x35c/0x688 [qed]
>   __qede_probe+0x88/0x5c8 [qede]
>   qede_probe+0x60/0xe0 [qede]
>   local_pci_probe+0x48/0xa0
>   work_for_cpu_fn+0x24/0x38
>   process_one_work+0x1d0/0x468
>   worker_thread+0x238/0x4e0
>   kthread+0xf0/0x118
>   ret_from_fork+0x10/0x18
>
> In this case, qed_hw_prepare() returns error due to hw/fw error, but in
> theory work queue should be in process context instead of interrupt.
>
> The root cause might be the unpaired spin_{un}lock_bh() in
> _qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.
>
> Reported-by: Lijian Zhang <Lijian.Zhang@arm.com>
> Signed-off-by: Jia He <justin.he@arm.com>
> ---

This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired
spin_{un}lock_bh.

Also,
as per description, looks like you are not sure actual the root-cause.
does this patch really solved the problem?

--pk

^ permalink raw reply	[flat|nested] 9+ messages in thread

* RE: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-19 10:35 ` Prabhakar Kushwaha
@ 2021-07-19 13:16   ` Justin He
  2021-07-19 14:50     ` Prabhakar Kushwaha
  0 siblings, 1 reply; 9+ messages in thread
From: Justin He @ 2021-07-19 13:16 UTC (permalink / raw)
  To: Prabhakar Kushwaha
  Cc: Ariel Elior, GR-everest-linux-l2, David S. Miller,
	Jakub Kicinski, netdev, Linux Kernel Mailing List, nd,
	Shai Malin, Shai Malin, Prabhakar Kushwaha

Hi Prabhakar

> -----Original Message-----
> From: Prabhakar Kushwaha <prabhakar.pkin@gmail.com>
> Sent: Monday, July 19, 2021 6:36 PM
> To: Justin He <Justin.He@arm.com>
> Cc: Ariel Elior <aelior@marvell.com>; GR-everest-linux-l2@marvell.com;
> David S. Miller <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>;
> netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; nd <nd@arm.com>; Shai Malin <malin1024@gmail.com>;
> Shai Malin <smalin@marvell.com>; Prabhakar Kushwaha <pkushwaha@marvell.com>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
> 
> Hi Jia,
> 
> On Thu, Jul 15, 2021 at 2:28 PM Jia He <justin.he@arm.com> wrote:
> >
> > Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> > QL41000 ethernet controller:
> >  BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> >   [qed_probe:488()]hw prepare failed
> >   kernel BUG at mm/vmalloc.c:2355!
> >   Internal error: Oops - BUG: 0 [#1] SMP
> >   CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
> Ubuntu
> >   pstate: 00400009 (nzcv daif +PAN -UAO)
> >  Call trace:
> >   vunmap+0x4c/0x50
> >   iounmap+0x48/0x58
> >   qed_free_pci+0x60/0x80 [qed]
> >   qed_probe+0x35c/0x688 [qed]
> >   __qede_probe+0x88/0x5c8 [qede]
> >   qede_probe+0x60/0xe0 [qede]
> >   local_pci_probe+0x48/0xa0
> >   work_for_cpu_fn+0x24/0x38
> >   process_one_work+0x1d0/0x468
> >   worker_thread+0x238/0x4e0
> >   kthread+0xf0/0x118
> >   ret_from_fork+0x10/0x18
> >
> > In this case, qed_hw_prepare() returns error due to hw/fw error, but in
> > theory work queue should be in process context instead of interrupt.
> >
> > The root cause might be the unpaired spin_{un}lock_bh() in
> > _qed_mcp_cmd_and_union(), which causes botton half is disabled
> incorrectly.
> >
> > Reported-by: Lijian Zhang <Lijian.Zhang@arm.com>
> > Signed-off-by: Jia He <justin.he@arm.com>
> > ---
> 
> This patch is adding additional spin_{un}lock_bh().
> Can you please enlighten about the exact flow causing this unpaired
> spin_{un}lock_bh.
> 
For instance:
_qed_mcp_cmd_and_union()
  In while loop
    spin_lock_bh()
    qed_mcp_has_pending_cmd() (assume false), will break the loop
  if (cnt >= max_retries) {
...
    return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock 
  }

> Also,
> as per description, looks like you are not sure actual the root-cause.
> does this patch really solved the problem?

I don't have that ThunderX2 to verify the patch.
But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore
under driver/.../qlogic, this is the only problematic point I could figure
out. And this might be possible code path of qed_probe().

--
Cheers,
Justin (Jia He)



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-19 13:16   ` Justin He
@ 2021-07-19 14:50     ` Prabhakar Kushwaha
  2021-07-20  2:02       ` Justin He
  0 siblings, 1 reply; 9+ messages in thread
From: Prabhakar Kushwaha @ 2021-07-19 14:50 UTC (permalink / raw)
  To: Justin He
  Cc: Ariel Elior, GR-everest-linux-l2, David S. Miller,
	Jakub Kicinski, netdev, Linux Kernel Mailing List, nd,
	Shai Malin, Shai Malin, Prabhakar Kushwaha

Hi Justin,

On Mon, Jul 19, 2021 at 6:47 PM Justin He <Justin.He@arm.com> wrote:
>
> Hi Prabhakar
>
> > -----Original Message-----
> > From: Prabhakar Kushwaha <prabhakar.pkin@gmail.com>
> > Sent: Monday, July 19, 2021 6:36 PM
> > To: Justin He <Justin.He@arm.com>
> > Cc: Ariel Elior <aelior@marvell.com>; GR-everest-linux-l2@marvell.com;
> > David S. Miller <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>;
> > netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> > kernel@vger.kernel.org>; nd <nd@arm.com>; Shai Malin <malin1024@gmail.com>;
> > Shai Malin <smalin@marvell.com>; Prabhakar Kushwaha <pkushwaha@marvell.com>
> > Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> > _qed_mcp_cmd_and_union()
> >
> > Hi Jia,
> >
> > On Thu, Jul 15, 2021 at 2:28 PM Jia He <justin.he@arm.com> wrote:
> > >
> > > Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> > > QL41000 ethernet controller:
> > >  BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> > >   [qed_probe:488()]hw prepare failed
> > >   kernel BUG at mm/vmalloc.c:2355!
> > >   Internal error: Oops - BUG: 0 [#1] SMP
> > >   CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
> > Ubuntu
> > >   pstate: 00400009 (nzcv daif +PAN -UAO)
> > >  Call trace:
> > >   vunmap+0x4c/0x50
> > >   iounmap+0x48/0x58
> > >   qed_free_pci+0x60/0x80 [qed]
> > >   qed_probe+0x35c/0x688 [qed]
> > >   __qede_probe+0x88/0x5c8 [qede]
> > >   qede_probe+0x60/0xe0 [qede]
> > >   local_pci_probe+0x48/0xa0
> > >   work_for_cpu_fn+0x24/0x38
> > >   process_one_work+0x1d0/0x468
> > >   worker_thread+0x238/0x4e0
> > >   kthread+0xf0/0x118
> > >   ret_from_fork+0x10/0x18
> > >
> > > In this case, qed_hw_prepare() returns error due to hw/fw error, but in
> > > theory work queue should be in process context instead of interrupt.
> > >
> > > The root cause might be the unpaired spin_{un}lock_bh() in
> > > _qed_mcp_cmd_and_union(), which causes botton half is disabled
> > incorrectly.
> > >
> > > Reported-by: Lijian Zhang <Lijian.Zhang@arm.com>
> > > Signed-off-by: Jia He <justin.he@arm.com>
> > > ---
> >
> > This patch is adding additional spin_{un}lock_bh().
> > Can you please enlighten about the exact flow causing this unpaired
> > spin_{un}lock_bh.
> >
> For instance:
> _qed_mcp_cmd_and_union()
>   In while loop
>     spin_lock_bh()
>     qed_mcp_has_pending_cmd() (assume false), will break the loop

I agree till here.

>   if (cnt >= max_retries) {
> ...
>     return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
>   }
>

Because of break, cnt has not been increased.
   - cnt is still less than max_retries.
  - if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.

I am not seeing any issue here.

> > Also,
> > as per description, looks like you are not sure actual the root-cause.
> > does this patch really solved the problem?
>
> I don't have that ThunderX2 to verify the patch.
> But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore
> under driver/.../qlogic, this is the only problematic point I could figure
> out. And this might be possible code path of qed_probe().
>

Without testing and proper root-cause,  it is tough to accept the suggested fix.

--pk

^ permalink raw reply	[flat|nested] 9+ messages in thread

* RE: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-19 14:50     ` Prabhakar Kushwaha
@ 2021-07-20  2:02       ` Justin He
  2021-07-20  9:05         ` Jakub Kicinski
  0 siblings, 1 reply; 9+ messages in thread
From: Justin He @ 2021-07-20  2:02 UTC (permalink / raw)
  To: Prabhakar Kushwaha, David S. Miller
  Cc: Ariel Elior, GR-everest-linux-l2, Jakub Kicinski, netdev,
	Linux Kernel Mailing List, nd, Shai Malin, Shai Malin,
	Prabhakar Kushwaha



> -----Original Message-----
> From: Prabhakar Kushwaha <prabhakar.pkin@gmail.com>
> Sent: Monday, July 19, 2021 10:51 PM
> To: Justin He <Justin.He@arm.com>
> Cc: Ariel Elior <aelior@marvell.com>; GR-everest-linux-l2@marvell.com;
> David S. Miller <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>;
> netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; nd <nd@arm.com>; Shai Malin <malin1024@gmail.com>;
> Shai Malin <smalin@marvell.com>; Prabhakar Kushwaha <pkushwaha@marvell.com>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
> 
> Hi Justin,
> 
> On Mon, Jul 19, 2021 at 6:47 PM Justin He <Justin.He@arm.com> wrote:
> >
> > Hi Prabhakar
> >
> > > -----Original Message-----
> > > From: Prabhakar Kushwaha <prabhakar.pkin@gmail.com>
> > > Sent: Monday, July 19, 2021 6:36 PM
> > > To: Justin He <Justin.He@arm.com>
> > > Cc: Ariel Elior <aelior@marvell.com>; GR-everest-linux-l2@marvell.com;
> > > David S. Miller <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>;
> > > netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> > > kernel@vger.kernel.org>; nd <nd@arm.com>; Shai Malin
> <malin1024@gmail.com>;
> > > Shai Malin <smalin@marvell.com>; Prabhakar Kushwaha
> <pkushwaha@marvell.com>
> > > Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> > > _qed_mcp_cmd_and_union()
> > >
> > > Hi Jia,
> > >
> > > On Thu, Jul 15, 2021 at 2:28 PM Jia He <justin.he@arm.com> wrote:
> > > >
> > > > Liajian reported a bug_on hit on a ThunderX2 arm64 server with
> FastLinQ
> > > > QL41000 ethernet controller:
> > > >  BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> > > >   [qed_probe:488()]hw prepare failed
> > > >   kernel BUG at mm/vmalloc.c:2355!
> > > >   Internal error: Oops - BUG: 0 [#1] SMP
> > > >   CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic
> #86-
> > > Ubuntu
> > > >   pstate: 00400009 (nzcv daif +PAN -UAO)
> > > >  Call trace:
> > > >   vunmap+0x4c/0x50
> > > >   iounmap+0x48/0x58
> > > >   qed_free_pci+0x60/0x80 [qed]
> > > >   qed_probe+0x35c/0x688 [qed]
> > > >   __qede_probe+0x88/0x5c8 [qede]
> > > >   qede_probe+0x60/0xe0 [qede]
> > > >   local_pci_probe+0x48/0xa0
> > > >   work_for_cpu_fn+0x24/0x38
> > > >   process_one_work+0x1d0/0x468
> > > >   worker_thread+0x238/0x4e0
> > > >   kthread+0xf0/0x118
> > > >   ret_from_fork+0x10/0x18
> > > >
> > > > In this case, qed_hw_prepare() returns error due to hw/fw error, but
> in
> > > > theory work queue should be in process context instead of interrupt.
> > > >
> > > > The root cause might be the unpaired spin_{un}lock_bh() in
> > > > _qed_mcp_cmd_and_union(), which causes botton half is disabled
> > > incorrectly.
> > > >
> > > > Reported-by: Lijian Zhang <Lijian.Zhang@arm.com>
> > > > Signed-off-by: Jia He <justin.he@arm.com>
> > > > ---
> > >
> > > This patch is adding additional spin_{un}lock_bh().
> > > Can you please enlighten about the exact flow causing this unpaired
> > > spin_{un}lock_bh.
> > >
> > For instance:
> > _qed_mcp_cmd_and_union()
> >   In while loop
> >     spin_lock_bh()
> >     qed_mcp_has_pending_cmd() (assume false), will break the loop
> 
> I agree till here.
> 
> >   if (cnt >= max_retries) {
> > ...
> >     return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
> >   }
> >
> 
> Because of break, cnt has not been increased.
>    - cnt is still less than max_retries.
>   - if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
> Hence pairing completed.

Sorry, indeed. Let me check other possibilities.
@David S. Miller Sorry for the inconvenience, could you please revert it
in netdev tree?

Apologies again.

--
Cheers,
Justin (Jia He)



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-20  2:02       ` Justin He
@ 2021-07-20  9:05         ` Jakub Kicinski
  2021-07-20  9:28           ` Justin He
  0 siblings, 1 reply; 9+ messages in thread
From: Jakub Kicinski @ 2021-07-20  9:05 UTC (permalink / raw)
  To: Justin He
  Cc: Prabhakar Kushwaha, David S. Miller, Ariel Elior,
	GR-everest-linux-l2, netdev, Linux Kernel Mailing List, nd,
	Shai Malin, Shai Malin, Prabhakar Kushwaha

On Tue, 20 Jul 2021 02:02:26 +0000, Justin He wrote:
> > > For instance:
> > > _qed_mcp_cmd_and_union()
> > >   In while loop
> > >     spin_lock_bh()
> > >     qed_mcp_has_pending_cmd() (assume false), will break the loop  
> > 
> > I agree till here.
> >   
> > >   if (cnt >= max_retries) {
> > > ...
> > >     return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
> > >   }
> > >  
> > 
> > Because of break, cnt has not been increased.
> >    - cnt is still less than max_retries.
> >   - if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
> > Hence pairing completed.  
> 
> Sorry, indeed. Let me check other possibilities.
> @David S. Miller Sorry for the inconvenience, could you please revert it
> in netdev tree?

Please submit a revert patch with the conclusions from the discussion
included in the commit message.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* RE: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
  2021-07-20  9:05         ` Jakub Kicinski
@ 2021-07-20  9:28           ` Justin He
  0 siblings, 0 replies; 9+ messages in thread
From: Justin He @ 2021-07-20  9:28 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Prabhakar Kushwaha, David S. Miller, Ariel Elior,
	GR-everest-linux-l2, netdev, Linux Kernel Mailing List, nd,
	Shai Malin, Shai Malin, Prabhakar Kushwaha, nd



> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Tuesday, July 20, 2021 5:06 PM
> To: Justin He <Justin.He@arm.com>
> Cc: Prabhakar Kushwaha <prabhakar.pkin@gmail.com>; David S. Miller
> <davem@davemloft.net>; Ariel Elior <aelior@marvell.com>; GR-everest-linux-
> l2@marvell.com; netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; nd <nd@arm.com>; Shai Malin <malin1024@gmail.com>;
> Shai Malin <smalin@marvell.com>; Prabhakar Kushwaha <pkushwaha@marvell.com>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
> 
> On Tue, 20 Jul 2021 02:02:26 +0000, Justin He wrote:
> > > > For instance:
> > > > _qed_mcp_cmd_and_union()
> > > >   In while loop
> > > >     spin_lock_bh()
> > > >     qed_mcp_has_pending_cmd() (assume false), will break the loop
> > >
> > > I agree till here.
> > >
> > > >   if (cnt >= max_retries) {
> > > > ...
> > > >     return -EAGAIN; <-- here returns -EAGAIN without invoking bh
> unlock
> > > >   }
> > > >
> > >
> > > Because of break, cnt has not been increased.
> > >    - cnt is still less than max_retries.
> > >   - if (cnt >= max_retries) will not be *true*, leading to
> spin_unlock_bh().
> > > Hence pairing completed.
> >
> > Sorry, indeed. Let me check other possibilities.
> > @David S. Miller Sorry for the inconvenience, could you please revert it
> > in netdev tree?
> 
> Please submit a revert patch with the conclusions from the discussion
> included in the commit message.
Okay,will do that
Thanks for the reminder

--
Cheers,
Justin (Jia He)



^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-07-20  9:30 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-15  8:08 [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union() Jia He
2021-07-15 19:50 ` patchwork-bot+netdevbpf
2021-07-15 23:42   ` Justin He
2021-07-19 10:35 ` Prabhakar Kushwaha
2021-07-19 13:16   ` Justin He
2021-07-19 14:50     ` Prabhakar Kushwaha
2021-07-20  2:02       ` Justin He
2021-07-20  9:05         ` Jakub Kicinski
2021-07-20  9:28           ` Justin He

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).