linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20
       [not found] <20200220112231.34FB.409509F4@e16-tech.com>
@ 2020-02-20 13:57 ` Chuck Lever
  2020-02-20 14:05   ` Leon Romanovsky
  0 siblings, 1 reply; 5+ messages in thread
From: Chuck Lever @ 2020-02-20 13:57 UTC (permalink / raw)
  To: Wang Yugui; +Cc: linux-rdma

Hello!

Thanks for your bug report.


> On Feb 19, 2020, at 10:22 PM, Wang Yugui <wangyugui@e16-tech.com> wrote:
> 
> Hi, chuck.lever 
> 
> a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20.
> 
> maybe some releationship to xprtrdma-fix-dma-scatter-gather-list-mapping-imbalance.patch

I don't see an obvious connection to fix-dma-scatter-gather-list-mapping-imbalance.
The backtrace below is through IPoIB code paths. Those have nothing to do with
NFS/RDMA, which is the only ULP code that is changed by my commit.


> maybe the info is useful.

I'm copying linux-rdma for a bigger set of eyeballs.

My knee-jerk recommendation is that if you have a reliable reproducer, try "git bisect"
between .20 and .21 to nail down a specific commit where the BUG starts to occur.


> Feb 20 10:05:58 T630 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000010
> ...
> Feb 20 10:05:58 T630 kernel: port_pkey_list_insert+0x30/0x1a0 [ib_core]
> Feb 20 10:05:58 T630 kernel: ? kmem_cache_alloc_trace+0x219/0x230
> Feb 20 10:05:58 T630 kernel: ib_security_modify_qp+0x244/0x3b0 [ib_core]
> Feb 20 10:05:58 T630 kernel: _ib_modify_qp+0x1c0/0x3c0 [ib_core]
> Feb 20 10:05:58 T630 kernel: ? dma_pool_free+0x24/0xc0
> Feb 20 10:05:58 T630 kernel: ipoib_init_qp+0x77/0x190 [ib_ipoib]
> Feb 20 10:05:58 T630 kernel: ? __mlx4_ib_query_pkey+0xe7/0x110 [mlx4_ib]
> Feb 20 10:05:58 T630 kernel: ? ib_find_pkey+0x98/0xe0 [ib_core]
> Feb 20 10:05:58 T630 kernel: ipoib_ib_dev_open_default+0x1a/0x180 [ib_ipoib]
> Feb 20 10:05:58 T630 kernel: ipoib_ib_dev_open+0x66/0xa0 [ib_ipoib]
> Feb 20 10:05:58 T630 kernel: ipoib_open+0x44/0x110 [ib_ipoib]
> Feb 20 10:05:58 T630 kernel: __dev_open+0xcd/0x160
> 
> 
> # ibstat
> CA 'mlx4_0'
>        CA type: MT4099
>        Number of ports: 2
>        Firmware version: 2.42.5000
>        Hardware version: 1
>        Node GUID: 0xe41d2d03007b4080
>        System image GUID: 0xe41d2d03007b4083
>        Port 1:
>                State: Down
>                Physical state: Polling
>                Rate: 10
>                Base lid: 0
>                LMC: 0
>                SM lid: 0
>                Capability mask: 0x02594868
>                Port GUID: 0xe41d2d03007b4081
>                Link layer: InfiniBand
>        Port 2:
>                State: Down
>                Physical state: Disabled
>                Rate: 40
>                Base lid: 0
>                LMC: 0
>                SM lid: 0
>                Capability mask: 0x00010000
>                Port GUID: 0xe61d2dfffe7b4082
>                Link layer: Ethernet
> 
> Best Regards
> 王玉贵
> 2020/02/20
> 
> --------------------------------------
> 北京京垓科技有限公司
> 王玉贵	wangyugui@e16-tech.com
> 电话：+86-136-71123776
> <bug-of-ib-in-5.4.21.message>

--
Chuck Lever




^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20
  2020-02-20 13:57 ` a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20 Chuck Lever
@ 2020-02-20 14:05   ` Leon Romanovsky
  2020-02-20 16:26     ` Wang Yugui
  0 siblings, 1 reply; 5+ messages in thread
From: Leon Romanovsky @ 2020-02-20 14:05 UTC (permalink / raw)
  To: Chuck Lever; +Cc: Wang Yugui, linux-rdma

On Thu, Feb 20, 2020 at 08:57:29AM -0500, Chuck Lever wrote:
> Hello!
>
> Thanks for your bug report.
>
>
> > On Feb 19, 2020, at 10:22 PM, Wang Yugui <wangyugui@e16-tech.com> wrote:
> >
> > Hi, chuck.lever
> >
> > a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20.
> >
> > maybe some releationship to xprtrdma-fix-dma-scatter-gather-list-mapping-imbalance.patch
>
> I don't see an obvious connection to fix-dma-scatter-gather-list-mapping-imbalance.
> The backtrace below is through IPoIB code paths. Those have nothing to do with
> NFS/RDMA, which is the only ULP code that is changed by my commit.
>
>
> > maybe the info is useful.
>
> I'm copying linux-rdma for a bigger set of eyeballs.
>
> My knee-jerk recommendation is that if you have a reliable reproducer, try "git bisect"
> between .20 and .21 to nail down a specific commit where the BUG starts to occur.

No need to bisect, it is me who broke.
The fix is already accepted, but not yet merged.
https://patchwork.kernel.org/patch/11387567/

Thanks

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20
  2020-02-20 14:05   ` Leon Romanovsky
@ 2020-02-20 16:26     ` Wang Yugui
  2020-02-25 13:05       ` Maor Gottlieb
  0 siblings, 1 reply; 5+ messages in thread
From: Wang Yugui @ 2020-02-20 16:26 UTC (permalink / raw)
  To: Leon Romanovsky; +Cc: Chuck Lever, linux-rdma

Hi, Leon, Chuck 

It is still broken even with the hotfix(https://patchwork.kernel.org/patch/11387567/) for 5.4.21.

the call stack is almost the same.

Feb 20 23:49:53 T630 kernel: Call Trace:
Feb 20 23:49:53 T630 kernel: port_pkey_list_insert+0x30/0x1a0 [ib_core]
Feb 20 23:49:53 T630 kernel: ? kmem_cache_alloc_trace+0x219/0x230
Feb 20 23:49:53 T630 kernel: ib_security_modify_qp+0x244/0x3b0 [ib_core]
Feb 20 23:49:53 T630 kernel: _ib_modify_qp+0x1c0/0x3c0 [ib_core]
Feb 20 23:49:53 T630 kernel: ? dma_pool_free+0x24/0xc0
Feb 20 23:49:53 T630 kernel: ipoib_init_qp+0x77/0x190 [ib_ipoib]
Feb 20 23:49:53 T630 kernel: ? __mlx4_ib_query_pkey+0xe7/0x110 [mlx4_ib]
Feb 20 23:49:53 T630 kernel: ? ib_find_pkey+0x98/0xe0 [ib_core]
Feb 20 23:49:53 T630 kernel: ipoib_ib_dev_open_default+0x1a/0x180 [ib_ipoib]
Feb 20 23:49:53 T630 kernel: ipoib_ib_dev_open+0x66/0xa0 [ib_ipoib]
Feb 20 23:49:53 T630 kernel: ipoib_open+0x44/0x110 [ib_ipoib]
Feb 20 23:49:53 T630 kernel: __dev_open+0xcd/0x160
Feb 20 23:49:53 T630 kernel: __dev_change_flags+0x1ad/0x220
Feb 20 23:49:53 T630 kernel: ? __dev_notify_flags+0x95/0xf0
Feb 20 23:49:53 T630 kernel: dev_change_flags+0x21/0x60
Feb 20 23:49:53 T630 kernel: do_setlink+0x320/0xf00
Feb 20 23:49:53 T630 kernel: ? __nla_validate_parse+0x51/0x840
Feb 20 23:49:53 T630 kernel: ? xas_load+0x8/0x80
Feb 20 23:49:53 T630 kernel: ? __update_load_avg_cfs_rq+0x1d5/0x2c0
Feb 20 23:49:53 T630 kernel: ? cpumask_next+0x17/0x20
Feb 20 23:49:53 T630 kernel: ? __snmp6_fill_stats64.isra.56+0x6b/0x110
Feb 20 23:49:53 T630 kernel: ? __nla_validate_parse+0x51/0x840
Feb 20 23:49:53 T630 kernel: __rtnl_newlink+0x53d/0x890
Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
Feb 20 23:49:53 T630 kernel: ? nla_put+0x2f/0x40
Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
Feb 20 23:49:53 T630 kernel: ? nla_put+0x2f/0x40
Feb 20 23:49:53 T630 kernel: ? rt6_fill_node+0x2d4/0x850
Feb 20 23:49:53 T630 kernel: ? _cond_resched+0x15/0x30
Feb 20 23:49:53 T630 kernel: ? kmem_cache_alloc_trace+0x1c9/0x230
Feb 20 23:49:53 T630 kernel: rtnl_newlink+0x43/0x60
Feb 20 23:49:53 T630 kernel: rtnetlink_rcv_msg+0x2b1/0x360
Feb 20 23:49:53 T630 kernel: ? __kmalloc_node_track_caller+0x241/0x300
Feb 20 23:49:53 T630 kernel: ? _cond_resched+0x15/0x30
Feb 20 23:49:53 T630 kernel: ? rtnl_calcit.isra.32+0x110/0x110
Feb 20 23:49:53 T630 kernel: netlink_rcv_skb+0x49/0x110
Feb 20 23:49:53 T630 kernel: netlink_unicast+0x191/0x220
Feb 20 23:49:53 T630 kernel: netlink_sendmsg+0x21d/0x3f0
Feb 20 23:49:53 T630 kernel: sock_sendmsg+0x5b/0x60
Feb 20 23:49:53 T630 kernel: ____sys_sendmsg+0x1eb/0x260
Feb 20 23:49:53 T630 kernel: ? copy_msghdr_from_user+0xdb/0x160
Feb 20 23:49:53 T630 kernel: ___sys_sendmsg+0x7c/0xc0
Feb 20 23:49:53 T630 kernel: ? do_filp_open+0xa7/0x100
Feb 20 23:49:53 T630 kernel: ? netdev_run_todo+0x5e/0x290
Feb 20 23:49:53 T630 kernel: ? list_lru_add+0xb7/0x1d0
Feb 20 23:49:53 T630 kernel: __sys_sendmsg+0x57/0xa0
Feb 20 23:49:53 T630 kernel: do_syscall_64+0x5b/0x180
Feb 20 23:49:53 T630 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9


This card have 2 port, and port 1 is set as InfiniBand, port 2
is set as Ethernet.

# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0xe41d2d03007b4080
        System image GUID: 0xe41d2d03007b4083
        Port 1:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02594868
                Port GUID: 0xe41d2d03007b4081
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xe61d2dfffe7b4082
                Link layer: Ethernet


Best Regards
王玉贵
2020/02/21

> On Thu, Feb 20, 2020 at 08:57:29AM -0500, Chuck Lever wrote:
> > Hello!
> >
> > Thanks for your bug report.
> >
> >
> > > On Feb 19, 2020, at 10:22 PM, Wang Yugui <wangyugui@e16-tech.com> wrote:
> > >
> > > Hi, chuck.lever
> > >
> > > a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20.
> > >
> > > maybe some releationship to xprtrdma-fix-dma-scatter-gather-list-mapping-imbalance.patch
> >
> > I don't see an obvious connection to fix-dma-scatter-gather-list-mapping-imbalance.
> > The backtrace below is through IPoIB code paths. Those have nothing to do with
> > NFS/RDMA, which is the only ULP code that is changed by my commit.
> >
> >
> > > maybe the info is useful.
> >
> > I'm copying linux-rdma for a bigger set of eyeballs.
> >
> > My knee-jerk recommendation is that if you have a reliable reproducer, try "git bisect"
> > between .20 and .21 to nail down a specific commit where the BUG starts to occur.
> 
> No need to bisect, it is me who broke.
> The fix is already accepted, but not yet merged.
> https://patchwork.kernel.org/patch/11387567/
> 
> Thanks

--------------------------------------
北京京垓科技有限公司
王玉贵	wangyugui@e16-tech.com
电话：+86-136-71123776


^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20
  2020-02-20 16:26     ` Wang Yugui
@ 2020-02-25 13:05       ` Maor Gottlieb
  2020-02-26  0:44         ` Wang Yugui
  0 siblings, 1 reply; 5+ messages in thread
From: Maor Gottlieb @ 2020-02-25 13:05 UTC (permalink / raw)
  To: Wang Yugui, Leon Romanovsky; +Cc: Chuck Lever, linux-rdma

On 2/20/2020 6:26 PM, Wang Yugui wrote:
> Hi, Leon, Chuck
>
> It is still broken even with the hotfix(https://patchwork.kernel.org/patch/11387567/) for 5.4.21.

Hi Wang,

How can I reproduce it ?

Can you please try with the below diff?

iff --git a/drivers/infiniband/core/security.c 
b/drivers/infiniband/core/security.c
index b9a36ea244d4..2d5608315dc8 100644
--- a/drivers/infiniband/core/security.c
+++ b/drivers/infiniband/core/security.c
@@ -340,11 +340,15 @@ static struct ib_ports_pkeys *get_new_pps(const 
struct ib_qp *qp,
                 return NULL;

         if (qp_attr_mask & IB_QP_PORT)
-           new_pps->main.port_num =
-                   (qp_pps) ? qp_pps->main.port_num : qp_attr->port_num;
+         new_pps->main.port_num = qp_attr->port_num;
+ else if (qp_pps)
+         new_pps->main.port_num = qp_pps->main.port_num;
+
         if (qp_attr_mask & IB_QP_PKEY_INDEX)
-           new_pps->main.pkey_index = (qp_pps) ? qp_pps->main.pkey_index :
- qp_attr->pkey_index;
+         new_pps->main.pkey_index = qp_attr->pkey_index;
+ else if (qp_pps)
+         new_pps->main.pkey_index = qp_pps->main.pkey_index;
+
         if ((qp_attr_mask & IB_QP_PKEY_INDEX) && (qp_attr_mask & 
IB_QP_PORT))
                 new_pps->main.state = IB_PORT_PKEY_VALID;

>
> the call stack is almost the same.
>
> Feb 20 23:49:53 T630 kernel: Call Trace:
> Feb 20 23:49:53 T630 kernel: port_pkey_list_insert+0x30/0x1a0 [ib_core]
> Feb 20 23:49:53 T630 kernel: ? kmem_cache_alloc_trace+0x219/0x230
> Feb 20 23:49:53 T630 kernel: ib_security_modify_qp+0x244/0x3b0 [ib_core]
> Feb 20 23:49:53 T630 kernel: _ib_modify_qp+0x1c0/0x3c0 [ib_core]
> Feb 20 23:49:53 T630 kernel: ? dma_pool_free+0x24/0xc0
> Feb 20 23:49:53 T630 kernel: ipoib_init_qp+0x77/0x190 [ib_ipoib]
> Feb 20 23:49:53 T630 kernel: ? __mlx4_ib_query_pkey+0xe7/0x110 [mlx4_ib]
> Feb 20 23:49:53 T630 kernel: ? ib_find_pkey+0x98/0xe0 [ib_core]
> Feb 20 23:49:53 T630 kernel: ipoib_ib_dev_open_default+0x1a/0x180 [ib_ipoib]
> Feb 20 23:49:53 T630 kernel: ipoib_ib_dev_open+0x66/0xa0 [ib_ipoib]
> Feb 20 23:49:53 T630 kernel: ipoib_open+0x44/0x110 [ib_ipoib]
> Feb 20 23:49:53 T630 kernel: __dev_open+0xcd/0x160
> Feb 20 23:49:53 T630 kernel: __dev_change_flags+0x1ad/0x220
> Feb 20 23:49:53 T630 kernel: ? __dev_notify_flags+0x95/0xf0
> Feb 20 23:49:53 T630 kernel: dev_change_flags+0x21/0x60
> Feb 20 23:49:53 T630 kernel: do_setlink+0x320/0xf00
> Feb 20 23:49:53 T630 kernel: ? __nla_validate_parse+0x51/0x840
> Feb 20 23:49:53 T630 kernel: ? xas_load+0x8/0x80
> Feb 20 23:49:53 T630 kernel: ? __update_load_avg_cfs_rq+0x1d5/0x2c0
> Feb 20 23:49:53 T630 kernel: ? cpumask_next+0x17/0x20
> Feb 20 23:49:53 T630 kernel: ? __snmp6_fill_stats64.isra.56+0x6b/0x110
> Feb 20 23:49:53 T630 kernel: ? __nla_validate_parse+0x51/0x840
> Feb 20 23:49:53 T630 kernel: __rtnl_newlink+0x53d/0x890
> Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
> Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
> Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
> Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
> Feb 20 23:49:53 T630 kernel: ? nla_put+0x2f/0x40
> Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
> Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
> Feb 20 23:49:53 T630 kernel: ? nla_put+0x2f/0x40
> Feb 20 23:49:53 T630 kernel: ? rt6_fill_node+0x2d4/0x850
> Feb 20 23:49:53 T630 kernel: ? _cond_resched+0x15/0x30
> Feb 20 23:49:53 T630 kernel: ? kmem_cache_alloc_trace+0x1c9/0x230
> Feb 20 23:49:53 T630 kernel: rtnl_newlink+0x43/0x60
> Feb 20 23:49:53 T630 kernel: rtnetlink_rcv_msg+0x2b1/0x360
> Feb 20 23:49:53 T630 kernel: ? __kmalloc_node_track_caller+0x241/0x300
> Feb 20 23:49:53 T630 kernel: ? _cond_resched+0x15/0x30
> Feb 20 23:49:53 T630 kernel: ? rtnl_calcit.isra.32+0x110/0x110
> Feb 20 23:49:53 T630 kernel: netlink_rcv_skb+0x49/0x110
> Feb 20 23:49:53 T630 kernel: netlink_unicast+0x191/0x220
> Feb 20 23:49:53 T630 kernel: netlink_sendmsg+0x21d/0x3f0
> Feb 20 23:49:53 T630 kernel: sock_sendmsg+0x5b/0x60
> Feb 20 23:49:53 T630 kernel: ____sys_sendmsg+0x1eb/0x260
> Feb 20 23:49:53 T630 kernel: ? copy_msghdr_from_user+0xdb/0x160
> Feb 20 23:49:53 T630 kernel: ___sys_sendmsg+0x7c/0xc0
> Feb 20 23:49:53 T630 kernel: ? do_filp_open+0xa7/0x100
> Feb 20 23:49:53 T630 kernel: ? netdev_run_todo+0x5e/0x290
> Feb 20 23:49:53 T630 kernel: ? list_lru_add+0xb7/0x1d0
> Feb 20 23:49:53 T630 kernel: __sys_sendmsg+0x57/0xa0
> Feb 20 23:49:53 T630 kernel: do_syscall_64+0x5b/0x180
> Feb 20 23:49:53 T630 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
>
> This card have 2 port, and port 1 is set as InfiniBand, port 2
> is set as Ethernet.
>
> # ibstat
> CA 'mlx4_0'
>          CA type: MT4099
>          Number of ports: 2
>          Firmware version: 2.42.5000
>          Hardware version: 1
>          Node GUID: 0xe41d2d03007b4080
>          System image GUID: 0xe41d2d03007b4083
>          Port 1:
>                  State: Down
>                  Physical state: Polling
>                  Rate: 10
>                  Base lid: 0
>                  LMC: 0
>                  SM lid: 0
>                  Capability mask: 0x02594868
>                  Port GUID: 0xe41d2d03007b4081
>                  Link layer: InfiniBand
>          Port 2:
>                  State: Down
>                  Physical state: Disabled
>                  Rate: 40
>                  Base lid: 0
>                  LMC: 0
>                  SM lid: 0
>                  Capability mask: 0x00010000
>                  Port GUID: 0xe61d2dfffe7b4082
>                  Link layer: Ethernet
>
>
> Best Regards
> 王玉贵
> 2020/02/21
>
>> On Thu, Feb 20, 2020 at 08:57:29AM -0500, Chuck Lever wrote:
>>> Hello!
>>>
>>> Thanks for your bug report.
>>>
>>>
>>>> On Feb 19, 2020, at 10:22 PM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>>>>
>>>> Hi, chuck.lever
>>>>
>>>> a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20.
>>>>
>>>> maybe some releationship to xprtrdma-fix-dma-scatter-gather-list-mapping-imbalance.patch
>>> I don't see an obvious connection to fix-dma-scatter-gather-list-mapping-imbalance.
>>> The backtrace below is through IPoIB code paths. Those have nothing to do with
>>> NFS/RDMA, which is the only ULP code that is changed by my commit.
>>>
>>>
>>>> maybe the info is useful.
>>> I'm copying linux-rdma for a bigger set of eyeballs.
>>>
>>> My knee-jerk recommendation is that if you have a reliable reproducer, try "git bisect"
>>> between .20 and .21 to nail down a specific commit where the BUG starts to occur.
>> No need to bisect, it is me who broke.
>> The fix is already accepted, but not yet merged.
>> https://patchwork.kernel.org/patch/11387567/
>>
>> Thanks
> --------------------------------------
> 北京京垓科技有限公司
> 王玉贵	wangyugui@e16-tech.com
> 电话：+86-136-71123776
>

^ permalink raw reply related	[flat|nested] 5+ messages in thread
* Re: a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20
  2020-02-25 13:05       ` Maor Gottlieb
@ 2020-02-26  0:44         ` Wang Yugui
  0 siblings, 0 replies; 5+ messages in thread
From: Wang Yugui @ 2020-02-26  0:44 UTC (permalink / raw)
  To: Maor Gottlieb; +Cc: Leon Romanovsky, Chuck Lever, linux-rdma

[-- Attachment #1: Type: text/plain, Size: 9562 bytes --]

Hi, Maor, Leon

The kernel 5.4.21 plus the two patches successfully boot now
without the NULL pointer problem. And nfs4/rdma sucessfully mount too.
#RDMA-core-Fix-use-of-logical-OR-in-get_new_pps.patch
#RDMA-core-fix-null.patch (the patch from Maor saved as git-am format)

My MCX354A have 2 port, and port 1 is set as InfiniBand, port 2 is set
as Ethernet.
# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0xe41d2d03007b4080
        System image GUID: 0xe41d2d03007b4083
        Port 1:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02594868
                Port GUID: 0xe41d2d03007b4081
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xe61d2dfffe7b4082
                Link layer: Ethernet

# mlxup
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      01T7NW
  Description:      ConnectX-3 VPI adapter; dual-port QSFP; FDR IB (56Gb/s) and 40GbE;PCIe3.0 x8 8GT/s; Dell PowerEdge
  PSID:             DEL1090001019
  PCI Device Name:  0000:84:00.0
  Port1 GUID:       e41d2d03007b4081
  Port2 MAC:        e41d2d7b4082
  Versions:         Current        Available
     FW             2.42.5000      N/A
     PXE            3.4.0752       N/A

  Status:           No matching image found


My server is a dell PowerEdge T630 with some other NIC cards.

# rxe_cfg
  Name        Link  Driver   Speed   NMTU  IPv4_addr      RDEV  RMTU
  em1         yes   igb              1500  192.168.2.63
  em2         no    igb              1500
  p1p1        no    bnx2x    10GigE  9000  10.0.0.63
  p1p2        no    bnx2x    10GigE  9000  10.0.1.63
  p6p2        no    mlx4_en          9000  10.40.1.63
  virbr0      no    bridge           1500  192.168.122.1
  virbr0-nic  no    tun              1500

Best Regards
王玉贵
2020/02/26

> On 2/20/2020 6:26 PM, Wang Yugui wrote:
> > Hi, Leon, Chuck
> >
> > It is still broken even with the hotfix(https://patchwork.kernel.org/patch/11387567/) for 5.4.21.
> 
> Hi Wang,
> 
> How can I reproduce it ?
> 
> Can you please try with the below diff?
> 
> iff --git a/drivers/infiniband/core/security.c b/drivers/infiniband/core/security.c
> index b9a36ea244d4..2d5608315dc8 100644
> --- a/drivers/infiniband/core/security.c
> +++ b/drivers/infiniband/core/security.c
> @@ -340,11 +340,15 @@ static struct ib_ports_pkeys *get_new_pps(const struct ib_qp *qp,
>  ??????????????? return NULL;
> 
>  ??????? if (qp_attr_mask & IB_QP_PORT)
> -?????????? new_pps->main.port_num =
> -?????????????????? (qp_pps) ? qp_pps->main.port_num : qp_attr->port_num;
> +???????? new_pps->main.port_num = qp_attr->port_num;
> + else if (qp_pps)
> +???????? new_pps->main.port_num = qp_pps->main.port_num;
> +
>  ??????? if (qp_attr_mask & IB_QP_PKEY_INDEX)
> -?????????? new_pps->main.pkey_index = (qp_pps) ? qp_pps->main.pkey_index :
> - qp_attr->pkey_index;
> +???????? new_pps->main.pkey_index = qp_attr->pkey_index;
> + else if (qp_pps)
> +???????? new_pps->main.pkey_index = qp_pps->main.pkey_index;
> +
>  ??????? if ((qp_attr_mask & IB_QP_PKEY_INDEX) && (qp_attr_mask & IB_QP_PORT))
>  ??????????????? new_pps->main.state = IB_PORT_PKEY_VALID;
> 
> >
> > the call stack is almost the same.
> >
> > Feb 20 23:49:53 T630 kernel: Call Trace:
> > Feb 20 23:49:53 T630 kernel: port_pkey_list_insert+0x30/0x1a0 [ib_core]
> > Feb 20 23:49:53 T630 kernel: ? kmem_cache_alloc_trace+0x219/0x230
> > Feb 20 23:49:53 T630 kernel: ib_security_modify_qp+0x244/0x3b0 [ib_core]
> > Feb 20 23:49:53 T630 kernel: _ib_modify_qp+0x1c0/0x3c0 [ib_core]
> > Feb 20 23:49:53 T630 kernel: ? dma_pool_free+0x24/0xc0
> > Feb 20 23:49:53 T630 kernel: ipoib_init_qp+0x77/0x190 [ib_ipoib]
> > Feb 20 23:49:53 T630 kernel: ? __mlx4_ib_query_pkey+0xe7/0x110 [mlx4_ib]
> > Feb 20 23:49:53 T630 kernel: ? ib_find_pkey+0x98/0xe0 [ib_core]
> > Feb 20 23:49:53 T630 kernel: ipoib_ib_dev_open_default+0x1a/0x180 [ib_ipoib]
> > Feb 20 23:49:53 T630 kernel: ipoib_ib_dev_open+0x66/0xa0 [ib_ipoib]
> > Feb 20 23:49:53 T630 kernel: ipoib_open+0x44/0x110 [ib_ipoib]
> > Feb 20 23:49:53 T630 kernel: __dev_open+0xcd/0x160
> > Feb 20 23:49:53 T630 kernel: __dev_change_flags+0x1ad/0x220
> > Feb 20 23:49:53 T630 kernel: ? __dev_notify_flags+0x95/0xf0
> > Feb 20 23:49:53 T630 kernel: dev_change_flags+0x21/0x60
> > Feb 20 23:49:53 T630 kernel: do_setlink+0x320/0xf00
> > Feb 20 23:49:53 T630 kernel: ? __nla_validate_parse+0x51/0x840
> > Feb 20 23:49:53 T630 kernel: ? xas_load+0x8/0x80
> > Feb 20 23:49:53 T630 kernel: ? __update_load_avg_cfs_rq+0x1d5/0x2c0
> > Feb 20 23:49:53 T630 kernel: ? cpumask_next+0x17/0x20
> > Feb 20 23:49:53 T630 kernel: ? __snmp6_fill_stats64.isra.56+0x6b/0x110
> > Feb 20 23:49:53 T630 kernel: ? __nla_validate_parse+0x51/0x840
> > Feb 20 23:49:53 T630 kernel: __rtnl_newlink+0x53d/0x890
> > Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
> > Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
> > Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
> > Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
> > Feb 20 23:49:53 T630 kernel: ? nla_put+0x2f/0x40
> > Feb 20 23:49:53 T630 kernel: ? __nla_reserve+0x38/0x50
> > Feb 20 23:49:53 T630 kernel: ? __nla_put+0xc/0x20
> > Feb 20 23:49:53 T630 kernel: ? nla_put+0x2f/0x40
> > Feb 20 23:49:53 T630 kernel: ? rt6_fill_node+0x2d4/0x850
> > Feb 20 23:49:53 T630 kernel: ? _cond_resched+0x15/0x30
> > Feb 20 23:49:53 T630 kernel: ? kmem_cache_alloc_trace+0x1c9/0x230
> > Feb 20 23:49:53 T630 kernel: rtnl_newlink+0x43/0x60
> > Feb 20 23:49:53 T630 kernel: rtnetlink_rcv_msg+0x2b1/0x360
> > Feb 20 23:49:53 T630 kernel: ? __kmalloc_node_track_caller+0x241/0x300
> > Feb 20 23:49:53 T630 kernel: ? _cond_resched+0x15/0x30
> > Feb 20 23:49:53 T630 kernel: ? rtnl_calcit.isra.32+0x110/0x110
> > Feb 20 23:49:53 T630 kernel: netlink_rcv_skb+0x49/0x110
> > Feb 20 23:49:53 T630 kernel: netlink_unicast+0x191/0x220
> > Feb 20 23:49:53 T630 kernel: netlink_sendmsg+0x21d/0x3f0
> > Feb 20 23:49:53 T630 kernel: sock_sendmsg+0x5b/0x60
> > Feb 20 23:49:53 T630 kernel: ____sys_sendmsg+0x1eb/0x260
> > Feb 20 23:49:53 T630 kernel: ? copy_msghdr_from_user+0xdb/0x160
> > Feb 20 23:49:53 T630 kernel: ___sys_sendmsg+0x7c/0xc0
> > Feb 20 23:49:53 T630 kernel: ? do_filp_open+0xa7/0x100
> > Feb 20 23:49:53 T630 kernel: ? netdev_run_todo+0x5e/0x290
> > Feb 20 23:49:53 T630 kernel: ? list_lru_add+0xb7/0x1d0
> > Feb 20 23:49:53 T630 kernel: __sys_sendmsg+0x57/0xa0
> > Feb 20 23:49:53 T630 kernel: do_syscall_64+0x5b/0x180
> > Feb 20 23:49:53 T630 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> >
> > This card have 2 port, and port 1 is set as InfiniBand, port 2
> > is set as Ethernet.
> >
> > # ibstat
> > CA 'mlx4_0'
> >          CA type: MT4099
> >          Number of ports: 2
> >          Firmware version: 2.42.5000
> >          Hardware version: 1
> >          Node GUID: 0xe41d2d03007b4080
> >          System image GUID: 0xe41d2d03007b4083
> >          Port 1:
> >                  State: Down
> >                  Physical state: Polling
> >                  Rate: 10
> >                  Base lid: 0
> >                  LMC: 0
> >                  SM lid: 0
> >                  Capability mask: 0x02594868
> >                  Port GUID: 0xe41d2d03007b4081
> >                  Link layer: InfiniBand
> >          Port 2:
> >                  State: Down
> >                  Physical state: Disabled
> >                  Rate: 40
> >                  Base lid: 0
> >                  LMC: 0
> >                  SM lid: 0
> >                  Capability mask: 0x00010000
> >                  Port GUID: 0xe61d2dfffe7b4082
> >                  Link layer: Ethernet
> >
> >
> > Best Regards
> > 王玉贵
> > 2020/02/21
> >
> >> On Thu, Feb 20, 2020 at 08:57:29AM -0500, Chuck Lever wrote:
> >>> Hello!
> >>>
> >>> Thanks for your bug report.
> >>>
> >>>
> >>>> On Feb 19, 2020, at 10:22 PM, Wang Yugui <wangyugui@e16-tech.com> wrote:
> >>>>
> >>>> Hi, chuck.lever
> >>>>
> >>>> a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20.
> >>>>
> >>>> maybe some releationship to xprtrdma-fix-dma-scatter-gather-list-mapping-imbalance.patch
> >>> I don't see an obvious connection to fix-dma-scatter-gather-list-mapping-imbalance.
> >>> The backtrace below is through IPoIB code paths. Those have nothing to do with
> >>> NFS/RDMA, which is the only ULP code that is changed by my commit.
> >>>
> >>>
> >>>> maybe the info is useful.
> >>> I'm copying linux-rdma for a bigger set of eyeballs.
> >>>
> >>> My knee-jerk recommendation is that if you have a reliable reproducer, try "git bisect"
> >>> between .20 and .21 to nail down a specific commit where the BUG starts to occur.
> >> No need to bisect, it is me who broke.
> >> The fix is already accepted, but not yet merged.
> >> https://patchwork.kernel.org/patch/11387567/
> >>
> >> Thanks
> > --------------------------------------
> > 北京京垓科技有限公司
> > 王玉贵	wangyugui@e16-tech.com
> > 电话：+86-136-71123776
> >

--------------------------------------
北京京垓科技有限公司
王玉贵	wangyugui@e16-tech.com
电话：+86-136-71123776

[-- Attachment #2: RDMA-core-fix-null.patch --]
[-- Type: application/octet-stream, Size: 1218 bytes --]

From d4078b7c5e9782b2ca3d6c6035f4abb995c4dab7 Mon Sep 17 00:00:00 2001
From: maorg@mellanox.com
Date: Wed, 26 Feb 2020 07:58:29 +0800
Subject: [PATCH] RDMA-core-fix-NULL

---
 drivers/infiniband/core/security.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/core/security.c b/drivers/infiniband/core/security.c
index 2b4d803..9e27ca1 100644
--- a/drivers/infiniband/core/security.c
+++ b/drivers/infiniband/core/security.c
@@ -340,11 +340,15 @@ static struct ib_ports_pkeys *get_new_pps(const struct ib_qp *qp,
 		return NULL;
 
 	if (qp_attr_mask & IB_QP_PORT)
-		new_pps->main.port_num =
-			(qp_pps) ? qp_pps->main.port_num : qp_attr->port_num;
+		new_pps->main.port_num = qp_attr->port_num;
+	else if (qp_pps)
+		new_pps->main.port_num = qp_pps->main.port_num;
+
 	if (qp_attr_mask & IB_QP_PKEY_INDEX)
-		new_pps->main.pkey_index = (qp_pps) ? qp_pps->main.pkey_index :
-						      qp_attr->pkey_index;
+		new_pps->main.pkey_index = qp_attr->pkey_index;
+	else if (qp_pps)
+		new_pps->main.pkey_index = qp_pps->main.pkey_index;
+
 	if ((qp_attr_mask & IB_QP_PKEY_INDEX) && (qp_attr_mask & IB_QP_PORT))
 		new_pps->main.state = IB_PORT_PKEY_VALID;
 
-- 
2.24.1


[-- Attachment #3: RDMA-core-Fix-use-of-logical-OR-in-get_new_pps.patch --]
[-- Type: application/octet-stream, Size: 5511 bytes --]

From patchwork Mon Feb 17 20:43:18 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Nathan Chancellor <natechancellor@gmail.com>
X-Patchwork-Id: 11387567
X-Patchwork-Delegate: jgg@ziepe.ca
Return-Path: <SRS0=eK3A=4F=vger.kernel.org=linux-rdma-owner@kernel.org>
Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org
 [172.30.200.123])
	by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B2E114E3
	for <patchwork-linux-rdma@patchwork.kernel.org>;
 Mon, 17 Feb 2020 20:43:40 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
	by mail.kernel.org (Postfix) with ESMTP id EF64120801
	for <patchwork-linux-rdma@patchwork.kernel.org>;
 Mon, 17 Feb 2020 20:43:39 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com
 header.b="NSM1P5Sb"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1728676AbgBQUnj (ORCPT
        <rfc822;patchwork-linux-rdma@patchwork.kernel.org>);
        Mon, 17 Feb 2020 15:43:39 -0500
Received: from mail-oi1-f194.google.com ([209.85.167.194]:42426 "EHLO
        mail-oi1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1727300AbgBQUni (ORCPT
        <rfc822;linux-rdma@vger.kernel.org>); Mon, 17 Feb 2020 15:43:38 -0500
Received: by mail-oi1-f194.google.com with SMTP id j132so17938514oih.9;
        Mon, 17 Feb 2020 12:43:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=from:to:cc:subject:date:message-id:mime-version
         :content-transfer-encoding;
        bh=mt8KEiHVFXt+VI7oyaRToYaExGUPicwbfI4j6wwtPQE=;
        b=NSM1P5SbjhpDBQ9V9I+7JKKNZZ8Xsi/Ao/gOUbQ1xd+3FCSvZBiK2f28jPw8GLxAEi
         aPZehpxvudMkidUrcGsB2Bew1M4jb7qwd7CU6KSuteWVELybmQqqn+sWdTuiGjRa2g10
         +XPrCy7IfzxuiYXxJGNn7Ms7wtLppo/NuXOOLQgDXLpcxFU4SBFDoIcJJzIs6MrZpt5v
         OK9Wpq4viCjxUrxAqvRh/W2VHdxlS/M8ZahbKDXH/U2gJQ5iTtyzaTqqisYboEJxjtVl
         hNHbFyIaNBkGy8Y7gWacVVo0+X77h06DaEi0HIZrwH3mG260jhh4PYTM8+cJZWgXikmi
         bYAg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version
         :content-transfer-encoding;
        bh=mt8KEiHVFXt+VI7oyaRToYaExGUPicwbfI4j6wwtPQE=;
        b=nHxJb/N0aEXI1NLuEX6v7goMRKLqSy2/f6Oe/ur4GDnQuJUWv1J3+8Y9hjmvmTffGd
         hDULZoeMDf0ZI8lNjXGj5mGcQjgm4DPCfCU5lHtSGCztmG9J67UAI2blGUPRa99n8X8h
         hAK6FSREY/mooA0V2D2ww1ry/6800CZI5OBBLhE3xSp8nd38YT9Sco6bBKmkqD8RqF1X
         TQ3JmRGtHeBALgLm5Cwlr1KtB6i35NHyMlHNhdwPSKDvZGvjTqw4YFRHiSIX16K5a9Ag
         Axrw5TOTyicoVx7j0AmPBQI1veCKvoVSC7tCjY2QEEN1K4RjKyAVhZ154iDOanXAwne4
         RiEA==
X-Gm-Message-State: APjAAAWx8xNsmDH5SniaSlaS6gKI0cMNDnb6qfbkgcsQom4cDI5RRHIH
        XioNkIq8Dk7YsiSnNin+azk=
X-Google-Smtp-Source: 
 APXvYqwScEu4D7KeCYqO8/1v9KdWk5GSYNtypdkxNUfqBHecf0KjewwAPXsmUl0Uj1AZUoq+J4MAcA==
X-Received: by 2002:aca:530e:: with SMTP id h14mr505712oib.105.1581972218092;
        Mon, 17 Feb 2020 12:43:38 -0800 (PST)
Received: from localhost.localdomain ([2604:1380:4111:8b00::1])
        by smtp.gmail.com with ESMTPSA id
 w20sm545592otj.21.2020.02.17.12.43.37
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Mon, 17 Feb 2020 12:43:37 -0800 (PST)
From: Nathan Chancellor <natechancellor@gmail.com>
To: Doug Ledford <dledford@redhat.com>, Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org, clang-built-linux@googlegroups.com,
        Nathan Chancellor <natechancellor@gmail.com>
Subject: [PATCH] RDMA/core: Fix use of logical OR in get_new_pps
Date: Mon, 17 Feb 2020 13:43:18 -0700
Message-Id: <20200217204318.13609-1-natechancellor@gmail.com>
X-Mailer: git-send-email 2.25.1
MIME-Version: 1.0
X-Patchwork-Bot: notify
Sender: linux-rdma-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-rdma.vger.kernel.org>
X-Mailing-List: linux-rdma@vger.kernel.org

Clang warns:

../drivers/infiniband/core/security.c:351:41: warning: converting the
enum constant to a boolean [-Wint-in-bool-context]
        if (!(qp_attr_mask & (IB_QP_PKEY_INDEX || IB_QP_PORT)) && qp_pps) {
                                               ^
1 warning generated.

A bitwise OR should have been used instead.

Fixes: 1dd017882e01 ("RDMA/core: Fix protection fault in get_pkey_idx_qp_list")
Link: https://github.com/ClangBuiltLinux/linux/issues/889
Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/core/security.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/security.c b/drivers/infiniband/core/security.c
index 2b4d80393bd0..b9a36ea244d4 100644
--- a/drivers/infiniband/core/security.c
+++ b/drivers/infiniband/core/security.c
@@ -348,7 +348,7 @@ static struct ib_ports_pkeys *get_new_pps(const struct ib_qp *qp,
 	if ((qp_attr_mask & IB_QP_PKEY_INDEX) && (qp_attr_mask & IB_QP_PORT))
 		new_pps->main.state = IB_PORT_PKEY_VALID;
 
-	if (!(qp_attr_mask & (IB_QP_PKEY_INDEX || IB_QP_PORT)) && qp_pps) {
+	if (!(qp_attr_mask & (IB_QP_PKEY_INDEX | IB_QP_PORT)) && qp_pps) {
 		new_pps->main.port_num = qp_pps->main.port_num;
 		new_pps->main.pkey_index = qp_pps->main.pkey_index;
 		if (qp_pps->main.state != IB_PORT_PKEY_NOT_VALID)

^ permalink raw reply related	[flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-02-26  0:44 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20200220112231.34FB.409509F4@e16-tech.com>
2020-02-20 13:57 ` a bug(BUG: kernel NULL pointer dereference) of ib or mlx happened in 5.4.21 but not in 5.4.20 Chuck Lever
2020-02-20 14:05   ` Leon Romanovsky
2020-02-20 16:26     ` Wang Yugui
2020-02-25 13:05       ` Maor Gottlieb
2020-02-26  0:44         ` Wang Yugui

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).