Issue with Infiniband / MLX5 IB driver when running opensm

[Joao Pinto <Joao.Pinto-HKixBCOQz3hWk0Htik3J/w@public.gmane.org>]

Hello,

I am trying to bring up a Connect-X 5 Ex and I am getting an issue when
executing opensm when the infiniband cables are connected (connected from one
port to the other). Could you please give me an hint of what might be hapenning?

# ibstat
CA 'mlx5_0'
        CA type: MT4119
        Number of ports: 1
        Firmware version: 16.19.2244
        Hardware version: 0
        Node GUID: 0x248a0703009ad906
        System image GUID: 0x248a0703009ad906
        Port 1:
                State: Initializing
                Physical state: LinkUp
                Rate: 56
                Base lid: 65535
                LMC: 0
                SM lid: 0
                Capability mask: 0x2651e848
                Port GUID: 0x248a0703009ad906
                Link layer: InfiniBand
CA 'mlx5_1'
        CA type: MT4119
        Number of ports: 1
        Firmware version: 16.19.2244
        Hardware version: 0
        Node GUID: 0x248a0703009ad907
        System image GUID: 0x248a0703009ad906
        Port 1:
                State: Initializing
                Physical state: LinkUp
                Rate: 56
                Base lid: 65535
                LMC: 0
                SM lid: 0
                Capability mask: 0x2651e848
                Port GUID: 0x248a0703009ad907
                Link layer: InfiniBand
#
#
# which opensm
/usr/sbin/opensm
# opensm -g 0x248a0703009ad906 &
# -------------------------------------------------
OpenSM 3.3.20
Command Line Arguments:
 Guid <0x248a0703009ad906>
 Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 3.3.20

Entering DISCOVERING state

------------[ cut here ]------------
WARNING: CPU: 0 PID: 128 at drivers/infiniband/hw/mlx5/mad.c:263
mlx5_ib_process_mad+0x1a6/0x64c
Modules linked in:
CPU: 0 PID: 128 Comm: kworker/0:1H Not tainted
4.12.0-MLNX20170524-ge176cc5-dirty #22
Workqueue: ib-comp-wq ib_cq_poll_work

Stack Trace:
  arc_unwind_core.constprop.2+0xb4/0x100
  warn_slowpath_null+0x48/0xe4
  mlx5_ib_process_mad+0x1a6/0x64c
  ib_mad_recv_done+0x352/0xa7c
  ib_cq_poll_work+0x72/0x130
  process_one_work+0x1c8/0x390
  worker_thread+0x120/0x540
  kthread+0x116/0x13c
  ret_from_fork+0x18/0x1c
---[ end trace 942bc9d60690df3b ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 128 at mm/page_alloc.c:3689
__alloc_pages_nodemask+0x18ec/0x24e4
Modules linked in:
CPU: 0 PID: 128 Comm: kworker/0:1H Tainted: G        W
4.12.0-MLNX20170524-ge176cc5-dirty #22
Workqueue: ib-comp-wq ib_cq_poll_work

Stack Trace:
  arc_unwind_core.constprop.2+0xb4/0x100
  warn_slowpath_null+0x48/0xe4
  __alloc_pages_nodemask+0x18ec/0x24e4
  kmalloc_order+0x16/0x28
  alloc_mad_private+0x12/0x20
  ib_mad_recv_done+0x2bc/0xa7c
  ib_cq_poll_work+0x72/0x130
  process_one_work+0x1c8/0x390
  worker_thread+0x120/0x540
  kthread+0x116/0x13c
  ret_from_fork+0x18/0x1c
---[ end trace 942bc9d60690df3c ]---
BUG: Bad rss-counter state mm:9672c000 idx:1 val:11
BUG: Bad rss-counter state mm:9672c000 idx:3 val:84
BUG: non-zero nr_ptes on freeing mm: 3
Path: /bin/busybox
CPU: 0 PID: 82 Comm: klogd Tainted: G        W
4.12.0-MLNX20170524-ge176cc5-dirty #22
task: 8fe0e3c0 task.stack: 8fe02000

[ECR   ]: 0x00220100 => Invalid Read @ 0x00008088 by insn @ 0x8124babc
[EFA   ]: 0x00008088
[BLINK ]: __d_alloc+0x2c/0x1cc
[ERET  ]: kmem_cache_alloc+0x4c/0xe8
------------[ cut here ]------------
WARNING: CPU: 0 PID: 128 at kernel/workqueue.c:1080 worker_thread+0x120/0x540
Modules linked in:
CPU: 0 PID: 128 Comm: kworker/0:1H Tainted: G        W
4.12.0-MLNX20170524-ge176cc5-dirty #22
------------[ cut here ]------------
WARNING: CPU: 0 PID: 128 at kernel/workqueue.c:1436 __queue_work+0x3e2/0x3e8
workqueue: per-cpu pwq for ib-comp-wq on cpu0 has 0 refcnt
Modules linked in:
CPU: 0 PID: 128 Comm: kworker/0:1H Tainted: G        W
4.12.0-MLNX20170524-ge176cc5-dirty #22

Stack Trace:
  arc_unwind_core.constprop.2+0xb4/0x100
  warn_slowpath_fmt+0x6c/0x110
  __queue_work+0x3e2/0x3e8
  queue_work_on+0x40/0x48
  mlx5_cq_completion+0x62/0xd8
  mlx5_eq_int+0x2dc/0x3a8
  __handle_irq_event_percpu+0xb8/0x150
  handle_irq_event+0x44/0x8c
  handle_simple_irq+0x5c/0xa4
  generic_handle_irq+0x1c/0x2c
  dw_handle_msi_irq+0x5a/0xd4
  dw_chained_msi_isr+0x26/0x78
  generic_handle_irq+0x1c/0x2c
  dw_apb_ictl_handler+0x7e/0xf8
  __handle_domain_irq+0x56/0x98
  handle_interrupt_level1+0xcc/0xd8
---[ end trace 942bc9d60690df3d ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 128 at kernel/workqueue.c:1064 __queue_work+0x31c/0x3e8
Modules linked in:
CPU: 0 PID: 128 Comm: kworker/0:1H Tainted: G        W
4.12.0-MLNX20170524-ge176cc5-dirty #22

Stack Trace:
  arc_unwind_core.constprop.2+0xb4/0x100
  warn_slowpath_null+0x48/0xe4
  __queue_work+0x31c/0x3e8
  queue_work_on+0x40/0x48
  mlx5_cq_completion+0x62/0xd8
  mlx5_eq_int+0x2dc/0x3a8
  __handle_irq_event_percpu+0xb8/0x150
  handle_irq_event+0x44/0x8c
  handle_simple_irq+0x5c/0xa4
  generic_handle_irq+0x1c/0x2c
  dw_handle_msi_irq+0x5a/0xd4
  dw_chained_msi_isr+0x26/0x78
  generic_handle_irq+0x1c/0x2c
  dw_apb_ictl_handler+0x7e/0xf8
  __handle_domain_irq+0x56/0x98
  handle_interrupt_level1+0xcc/0xd8
---[ end trace 942bc9d60690df3e ]---

Stack Trace:
  arc_unwind_core.constprop.2+0xb4/0x100
  warn_slowpath_null+0x48/0xe4
  worker_thread+0x120/0x540
  kthread+0x116/0x13c
  ret_from_fork+0x18/0x1c
---[ end trace 942bc9d60690df3f ]---
[STAT32]: 0x00000406 : K         E2 E1
BTA: 0x8124ba86  SP: 0x8fe03dec  FP: 0x00000000
LPS: 0x81274348 LPE: 0x81274354 LPC: 0x00000000
r00: 0x00008088 r01: 0x014000c0 r02: 0x00008088
r03: 0x00001b1a r04: 0x00000000 r05: 0x00000806
r06: 0x9a19cea0 r07: 0x00000005 r08: 0x00000054
r09: 0x00000000 r10: 0x00000000 r11: 0x2000a038
r12: 0x00000000

Stack Trace:
  kmem_cache_alloc+0x4c/0xe8
  __d_alloc+0x2c/0x1cc
  d_alloc_parallel+0x46/0x3f8
  path_openat+0xd48/0x132c
  do_filp_open+0x44/0xc0
  SyS_openat+0x144/0x1d4
  EV_Trap+0x11c/0x120


Thank you and best regards,

Joao Pinto

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html