Re: [PATCH for-next] RDMA/rxe: Fix bug in rxe_net.c

[Zhu Yanjun <zyjzyj2000@gmail.com>]

On Thu, Jul 22, 2021 at 11:37 PM Olga Kornievskaia <aglo@umich.edu> wrote:
>
> I'm RHEL based in terms of my userland software. I work on upstream
> kernels using that.

Simply, one host with the kernel 5.14-rc1, the other host with the
kernel 5.14-rc2.
Then the following errors will appear
"
...
 [13873.255148] rdma_rxe: bad ICRC from 192.168.1.92
 [13877.567475] rdma_rxe: bad ICRC from 192.168.1.92
 [13882.175544] rdma_rxe: bad ICRC from 192.168.1.92
...
"

Correct?

Zhu Yanjun

>
> Client is running kernel version 5.14-rc1 (when I started, now rc2) on
> a  RHEL8.4 (beta, when I started) VM (RHEL8.2 VM with 5.14-rc1 kernel
> for server). RHEL8.4 beta that came with userland versions
> [aglo@localhost linux-nfs]$ rpm -qa | grep rdma
> rdma-core-devel-32.0-1.el8.x86_64
> librdmacm-utils-32.0-1.el8.x86_64
> rdma-core-32.0-1.el8.x86_64
> librdmacm-32.0-1.el8.x86_64
>
> I upgraded to RHEL8.4GA to make sure it's on an official release of
> the userspace. The results are the same (at the end of the mail).
> [root@localhost yum.repos.d]# rpm -qa | grep rdma
> rdma-core-32.0-4.el8.x86_64
> rdma-core-devel-32.0-4.el8.x86_64
> librdmacm-utils-32.0-4.el8.x86_64
> librdmacm-32.0-4.el8.x86_64
>
> Now, let's go back to NFSoRDMA so that we remove the variable of what
> version are the userland libraries (and if there are any
> interoperability issues with kernel changes and userland). Doing an
> NFS mount, leads to client logging continuously logging "bad ICRC"
> until mount fails with connection refused.
>
> Network trace has "ConnectRequest" which gets back ConnectReject
> (reason 0x001c) which I'm assuming is bad ICRC?
>
> nfs oops (that doesn't actually crash the machine which is nice) (this
> is a snippet and doesn't reflect the #of bad ICRC message in total):
> [  342.290895] rdma_rxe: bad ICRC from 192.168.1.92
> [  348.947562] rdma_rxe: bad ICRC from 192.168.1.92
> [  355.602913] rdma_rxe: invalid mask or state for qp
> [  355.606411] rdma_rxe: invalid mask or state for qp
> [  355.608928] ------------[ cut here ]------------
> [  355.610831] failed to drain recv queue: -22
> [  355.612549] WARNING: CPU: 1 PID: 516 at
> drivers/infiniband/core/verbs.c:2738 __ib_drain_rq+0x258/0x290
> [ib_core]
> [  355.616200] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver
> nfs lockd grace rpcrdma rdma_ucm rdma_cm iw_cm ib_cm rdma_rxe
> ip6_udp_tunnel udp_tunnel ib_uverbs ib_core uinput nls_utf8 isofs
> rfcomm xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter
> nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> nf_tables nfnetlink tun bridge stp llc bnep vsock_loopback
> vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock
> intel_rapl_msr snd_seq_midi snd_seq_midi_event intel_rapl_common
> crct10dif_pclmul crc32_pclmul vmw_balloon ghash_clmulni_intel rapl
> snd_ens1371 joydev pcspkr snd_ac97_codec ac97_bus snd_seq uvcvideo
> btusb snd_pcm videobuf2_vmalloc btrtl videobuf2_memops btbcm
> videobuf2_v4l2 btintel videobuf2_common bluetooth videodev snd_timer
> rfkill snd_rawmidi snd_seq_device mc ecdh_generic snd ecc soundcore
> vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod
> cdrom sg ata_generic crc32c_intel vmwgfx ttm drm_kms_helper nvme ahci
> syscopyarea
> [  355.616399]  sysfillrect libahci sysimgblt ata_piix serio_raw
> fb_sys_fops drm nvme_core libata vmxnet3 t10_pi dm_mirror
> dm_region_hash dm_log dm_mod fuse
> [  355.648889] CPU: 1 PID: 516 Comm: kworker/u256:28 Tainted: G
> W         5.14.0-rc2+ #199
> [  355.651852] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
> [  355.655245] Workqueue: xprtiod xprt_autoclose [sunrpc]
> [  355.657033] RIP: 0010:__ib_drain_rq+0x258/0x290 [ib_core]
> [  355.658808] Code: 00 00 00 48 89 ef e8 f7 a9 cc de 48 85 c0 74 e1
> e9 f6 fe ff ff 89 c6 48 c7 c7 40 09 4d c1 c6 05 0a 60 08 00 01 e8 da
> 29 c6 de <0f> 0b e9 da fe ff ff 80 3d f6 5f 08 00 00 0f 85 cd fe ff ff
> 89 c6
> [  355.665601] RSP: 0018:ffff888008cc7b48 EFLAGS: 00010286
> [  355.667435] RAX: 0000000000000000 RBX: 1ffff11001198f69 RCX: ffffffff9f427a3e
> [  355.669758] RDX: 1ffff1100b98cd35 RSI: 0000000000000008 RDI: ffff88805cc669ac
> [  355.672397] RBP: ffff88801c83c058 R08: ffffed100b98df31 R09: ffffed100b98df31
> [  355.675018] R10: ffff88805cc6f987 R11: ffffed100b98df30 R12: ffff88801312f000
> [  355.677844] R13: ffff8880183ef810 R14: ffffffffc18173c0 R15: ffff888001119000
> [  355.680599] FS:  0000000000000000(0000) GS:ffff88805cc40000(0000)
> knlGS:0000000000000000
> [  355.683904] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  355.686447] CR2: 00003bafb8832fd0 CR3: 00000000142e6002 CR4: 00000000001706e0
> [  355.689154] Call Trace:
> [  355.690157]  ? __ib_drain_sq+0x280/0x280 [ib_core]
> [  355.692013]  ? autoremove_wake_function+0x82/0xa0
> [  355.694000]  ? mutex_lock+0x8e/0xe0
> [  355.695683]  ? mutex_unlock+0x1d/0x40
> [  355.697821]  ? cma_modify_qp_err+0xa5/0xf0 [rdma_cm]
> [  355.700409]  ? rdma_unlock_handler+0x20/0x20 [rdma_cm]
> [  355.702602]  ? __update_load_avg_cfs_rq+0x5a/0x550
> [  355.704558]  ib_drain_rq+0x9f/0xb0 [ib_core]
> [  355.706253]  rpcrdma_xprt_disconnect+0xbe/0x4b0 [rpcrdma]
> [  355.708215]  xprt_rdma_close+0xe/0x50 [rpcrdma]
> [  355.709785]  xprt_autoclose+0x8b/0x160 [sunrpc]
> [  355.711810]  process_one_work+0x3ab/0x6b0
> [  355.713303]  worker_thread+0x57/0x5c0
> [  355.714477]  ? process_one_work+0x6b0/0x6b0
> [  355.715806]  kthread+0x1bf/0x1f0
> [  355.716901]  ? set_kthread_struct+0x80/0x80
> [  355.718333]  ret_from_fork+0x22/0x30
> [  355.719577] ---[ end trace dc0181bd9d91f55b ]---
> [  355.721135] rdma_rxe: invalid mask or state for qp
> [  355.723117] ------------[ cut here ]------------
>
> rping oops.
>
> [13873.255148] rdma_rxe: bad ICRC from 192.168.1.92
> [13877.567475] rdma_rxe: bad ICRC from 192.168.1.92
> [13882.175544] rdma_rxe: bad ICRC from 192.168.1.92
> [13886.784329] rdma_rxe: bad ICRC from 192.168.1.92
> [13891.391534] rdma_rxe: bad ICRC from 192.168.1.92
> [13896.000084] rdma_rxe: bad ICRC from 192.168.1.92
> [13900.608291] rdma_rxe: bad ICRC from 192.168.1.92
> [13905.219925] rdma_rxe: bad ICRC from 192.168.1.92
> [13905.222298] rdma_rxe: bad ICRC from 192.168.1.92
> [13907.392305] rdma_rxe: bad ICRC from 192.168.1.92
> [13909.569156] rdma_rxe: bad ICRC from 192.168.1.92
> [13911.744391] rdma_rxe: bad ICRC from 192.168.1.92
> [13913.921244] rdma_rxe: bad ICRC from 192.168.1.92
> [13916.097423] rdma_rxe: bad ICRC from 192.168.1.92
> [13918.272800] rdma_rxe: bad ICRC from 192.168.1.92
> [13920.449837] BUG: unable to handle page fault for address: ffffc90103782194
> [13920.453440] #PF: supervisor read access in kernel mode
> [13920.455627] #PF: error_code(0x0000) - not-present page
> [13920.457585] PGD 1000067 P4D 1000067 PUD 0
> [13920.459103] Oops: 0000 [#1] SMP KASAN PTI
> [13920.460659] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W
>   5.14.0-rc2+ #199
> [13920.463284] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
> [13920.466820] RIP: 0010:copy_data+0x45/0x3a0 [rdma_rxe]
> [13920.468732] Code: 20 48 89 0c 24 44 89 4c 24 10 45 85 c0 0f 84 6c
> 01 00 00 48 8d 42 04 49 89 d4 45 89 c5 48 89 c7 48 89 44 24 30 e8 fb
> c0 6e ec <45> 8b 7c 24 04 44 89 7c 24 14 45 39 ef 0f 8c 08 03 00 00 49
> 8d 44
> [13920.474397] RSP: 0018:ffff88805cc092e0 EFLAGS: 00010246
> [13920.476010] RAX: 0000000000000000 RBX: ffff88800ed36520 RCX: ffffffffc15b4555
> [13920.478234] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: ffffc90103782194
> [13920.480721] RBP: ffff888042fc9a48 R08: 0000000000000010 R09: 0000000000000000
> [13920.483235] R10: ffff888042fc9a55 R11: ffffed1001da6c01 R12: ffffc90103782190
> [13920.485626] R13: 0000000000000010 R14: ffff8880172a536a R15: ffff88800ed36000
> [13920.488373] FS:  0000000000000000(0000) GS:ffff88805cc00000(0000)
> knlGS:0000000000000000
> [13920.491327] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [13920.493344] CR2: ffffc90103782194 CR3: 0000000044dac005 CR4: 00000000001706f0
> [13920.495705] Call Trace:
> [13920.496608]  <IRQ>
> [13920.497554]  send_data_in.isra.30+0x21/0x40 [rdma_rxe]
> [13920.499371]  rxe_responder+0x1a06/0x3e50 [rdma_rxe]
> [13920.500970]  ? fib_info_nh_uses_dev+0x6d/0x320
> [13920.502530]  ? rxe_resp_queue_pkt+0x60/0x60 [rdma_rxe]
> [13920.504183]  ? crc32_pclmul_update+0x36/0x42 [crc32_pclmul]
> [13920.505895]  ? rxe_crc32.isra.14+0x7d/0x100 [rdma_rxe]
> [13920.507485]  ? check_type_state.isra.8+0x150/0x150 [rdma_rxe]
> [13920.509248]  ? find_gid+0x166/0x210 [ib_core]
> [13920.510978]  ? _raw_spin_lock_irqsave+0x80/0xe0
> [13920.512449]  ? _raw_write_lock_irqsave+0xe0/0xe0
> [13920.513877]  ? rxe_resp_queue_pkt+0x60/0x60 [rdma_rxe]
> [13920.515506]  rxe_do_task+0xd2/0x160 [rdma_rxe]
> [13920.516881]  rxe_rcv+0x5a5/0xe30 [rdma_rxe]
> [13920.518510]  ? rxe_crc32.isra.14+0x100/0x100 [rdma_rxe]
> [13920.520297]  ? __udp4_lib_lookup+0x3fa/0x5b0
> [13920.521617]  ? ib_device_get_by_netdev+0x165/0x1b0 [ib_core]
> [13920.523403]  ? ib_unregister_driver+0x170/0x170 [ib_core]
> [13920.525327]  ? stack_access_ok+0x35/0x80
> [13920.526808]  rxe_udp_encap_recv+0xd0/0x120 [rdma_rxe]
> [13920.528541]  ? rxe_enable_task+0x20/0x20 [rdma_rxe]
> [13920.530252]  udp_queue_rcv_one_skb+0x36d/0x8a0
> [13920.531985]  udp_unicast_rcv_skb.isra.65+0x126/0x140
> [13920.533800]  __udp4_lib_rcv+0x924/0x1310
> [13920.535186]  ? udp_err+0x20/0x20
> [13920.536190]  ? is_bpf_text_address+0x13/0x20
> [13920.537554]  ? kernel_text_address+0x100/0x110
> [13920.538944]  ? __unwind_start+0x2e8/0x370
> [13920.540193]  ? raw_rcv+0x1a0/0x1a0
> [13920.541253]  ? nft_do_chain_arp+0xa0/0xa0 [nf_tables]
> [13920.542913]  ? nft_do_chain_ipv4+0xe4/0x110 [nf_tables]
> [13920.544569]  ? nf_nat_ipv4_fn+0x21/0xc0 [nf_nat]
> [13920.546109]  ip_protocol_deliver_rcu+0x170/0x2c0
> [13920.547907]  ip_local_deliver_finish+0xae/0xc0
> [13920.549598]  ip_local_deliver+0x1ae/0x1c0
> [13920.551031]  ? ip_local_deliver_finish+0xc0/0xc0
> [13920.552586]  ? ip_route_input_rcu+0x421/0x4b0
> [13920.554071]  ? ip_protocol_deliver_rcu+0x2c0/0x2c0
> [13920.555662]  ? ip_sublist_rcv+0x3c0/0x3c0
> [13920.556962]  ? ip_sublist_rcv+0x3c0/0x3c0
> [13920.558439]  ip_rcv+0x159/0x160
> [13920.559549]  ? ip_sublist_rcv+0x3c0/0x3c0
> [13920.560782]  ? secondary_startup_64_no_verify+0xc2/0xcb
> [13920.562683]  ? remove_all_stable_nodes+0x40/0x190
> [13920.564674]  ? ip_local_deliver+0x1c0/0x1c0
> [13920.566054]  ? __napi_poll+0x5d/0x1f0
> [13920.567310]  ? net_rx_action+0x21c/0x4a0
> [13920.568616]  ? __do_softirq+0xf9/0x376
> [13920.569809]  __netif_receive_skb_one_core+0x133/0x150
> [13920.571350]  ? __netif_receive_skb_core+0x1760/0x1760
> [13920.572889]  ? ip_finish_output+0xc0/0xc0
> [13920.574123]  ? _raw_spin_lock_irqsave+0x80/0xe0
> [13920.575505]  ? _raw_write_lock_irqsave+0xe0/0xe0
> [13920.576910]  ? kasan_set_track+0x1c/0x30
> [13920.578205]  netif_receive_skb+0x94/0x240
> [13920.579667]  ? __netif_receive_skb+0xa0/0xa0
> [13920.581132]  ? eth_type_trans+0x134/0x270
> [13920.582422]  ? eth_gro_receive+0x310/0x310
> [13920.583679]  ? __build_skb_around+0x10e/0x130
> [13920.585023]  ? dma_unmap_page_attrs+0x1c6/0x2d0
> [13920.586439]  vmxnet3_rq_rx_complete+0xa76/0x17b0 [vmxnet3]
> [13920.588146]  vmxnet3_poll_rx_only+0x47/0xd0 [vmxnet3]
> [13920.589693]  __napi_poll+0x5d/0x1f0
> [13920.590766]  net_rx_action+0x21c/0x4a0
> [13920.591918]  ? napi_threaded_poll+0x1c0/0x1c0
> [13920.593253]  ? vmxnet3_msix_tx+0x100/0x100 [vmxnet3]
> [13920.594792]  ? note_interrupt+0xf0/0x3a0
> [13920.596042]  ? add_interrupt_randomness+0x15f/0x2a0
> [13920.597677]  ? _raw_spin_lock+0x7a/0xd0
> [13920.598853]  ? _raw_write_lock_bh+0xe0/0xe0
> [13920.600144]  __do_softirq+0xf9/0x376
> [13920.601247]  irq_exit_rcu+0x118/0x130
> [13920.602435]  common_interrupt+0x77/0x90
> [13920.603712]  </IRQ>
> [13920.604411]  asm_common_interrupt+0x1e/0x40
> [13920.606075] RIP: 0010:acpi_idle_do_entry+0x61/0x70
> [13920.607750] Code: ef 01 00 be 08 00 00 00 48 89 df e8 89 10 54 ff
> 48 89 df e8 41 06 54 ff 48 8b 03 a8 08 75 0c eb 07 0f 00 2d 01 b1 73
> 00 fb f4 <fa> 5b c3 48 89 df 5b e9 93 f9 ff ff cc cc cc 0f 1f 44 00 00
> 41 57
> [13920.613840] RSP: 0018:ffffffffaf407d98 EFLAGS: 00000246
> [13920.615463] RAX: 0000000000004000 RBX: ffffffffaf41a400 RCX: ffffffffae76014f
> [13920.617651] RDX: dffffc0000000000 RSI: 0000000000000008 RDI: ffffffffaf41a400
> [13920.619820] RBP: 0000000000000001 R08: fffffbfff5e83481 R09: fffffbfff5e83481
> [13920.622005] R10: ffffffffaf41a407 R11: fffffbfff5e83480 R12: ffff88800953a000
> [13920.624162] R13: 0000000000000001 R14: ffff88800953a004 R15: ffff8880046c1800
> [13920.626335]  ? acpi_idle_do_entry+0x4f/0x70
> [13920.627740]  ? acpi_idle_do_entry+0x4f/0x70
> [13920.629020]  acpi_idle_enter+0x14d/0x1c0
> [13920.630295]  cpuidle_enter_state+0xb2/0x590
> [13920.631603]  ? tick_nohz_stop_tick+0x1f0/0x2d0
> [13920.632987]  cpuidle_enter+0x3c/0x60
> [13920.634136]  do_idle+0x399/0x400
> [13920.635192]  ? arch_cpu_idle_exit+0x40/0x40
> [13920.636471]  ? do_idle+0x26d/0x400
> [13920.637517]  cpu_startup_entry+0x19/0x20
> [13920.638716]  start_kernel+0x378/0x396
> [13920.639925]  secondary_startup_64_no_verify+0xc2/0xcb
> [13920.641564] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm
> rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core uinput nls_utf8
> isofs rfcomm xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat
> nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6
> nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc bnep
> vsock_loopback vmw_vsock_virtio_transport_common
> vmw_vsock_vmci_transport vsock intel_rapl_msr snd_seq_midi
> snd_seq_midi_event intel_rapl_common crct10dif_pclmul crc32_pclmul
> ghash_clmulni_intel vmw_balloon rapl pcspkr joydev snd_ens1371
> snd_ac97_codec ac97_bus btusb snd_seq uvcvideo btrtl btbcm btintel
> videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common
> bluetooth snd_pcm videodev mc snd_timer rfkill snd_rawmidi
> snd_seq_device ecdh_generic snd ecc soundcore vmw_vmci i2c_piix4
> auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sg ata_generic
> crc32c_intel nvme vmwgfx ata_piix serio_raw nvme_core ttm ahci libahci
> drm_kms_helper libata syscopyarea
> [13920.641737]  sysfillrect sysimgblt fb_sys_fops vmxnet3 t10_pi drm
> dm_mirror dm_region_hash dm_log dm_mod fuse
> [13920.671747] CR2: ffffc90103782194
> [13920.674506] ---[ end trace 6ae70b2fba32e277 ]---
>
>
> On Wed, Jul 21, 2021 at 9:56 PM Pearson, Robert B
> <robert.pearson2@hpe.com> wrote:
> >
> > OK. For tomorrow. I need to know more about your setup. Which versions of kernel, rdma-core and what application SW you are running so I can try to reproduce your results.
> >
> > Regards,
> >
> > Bob Pearson
> >
> > -----Original Message-----
> > From: Olga Kornievskaia <aglo@umich.edu>
> > Sent: Wednesday, July 21, 2021 7:31 PM
> > To: Bob Pearson <rpearsonhpe@gmail.com>
> > Cc: Jason Gunthorpe <jgg@nvidia.com>; Zhu Yanjun <zyjzyj2000@gmail.com>; linux-rdma <linux-rdma@vger.kernel.org>
> > Subject: Re: [PATCH for-next] RDMA/rxe: Fix bug in rxe_net.c
> >
> > On Wed, Jul 21, 2021 at 5:42 PM Bob Pearson <rpearsonhpe@gmail.com> wrote:
> > >
> > > An earlier patch removed setting of tot_len in IPV4 headers because it
> > > was also set in ip_local_out. However, this change resulted in an
> > > incorrect ICRC being computed because the tot_len field is not masked
> > > out. This patch restores that line. This fixes the bug reported by Zhu Yanjun.
> > > This bug would have also affected anyone using rxe.
> > >
> > > Fixes: 230bb836ee88 ("RDMA/rxe: Fix redundant call to ip_send_check")
> > > Reported_by: Zhu Yanjun <zyjzyj2000@gmail.com>
> > > Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com>
> > > ---
> > >  drivers/infiniband/sw/rxe/rxe_net.c | 1 +
> > >  1 file changed, 1 insertion(+)
> > >
> > > diff --git a/drivers/infiniband/sw/rxe/rxe_net.c
> > > b/drivers/infiniband/sw/rxe/rxe_net.c
> > > index dec92928a1cd..5ac27f28ace1 100644
> > > --- a/drivers/infiniband/sw/rxe/rxe_net.c
> > > +++ b/drivers/infiniband/sw/rxe/rxe_net.c
> > > @@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry
> > > *dst, struct sk_buff *skb,
> > >
> > >         iph->version    =       IPVERSION;
> > >         iph->ihl        =       sizeof(struct iphdr) >> 2;
> > > +       iph->tot_len    =       htons(skb->len);
> > >         iph->frag_off   =       df;
> > >         iph->protocol   =       proto;
> > >         iph->tos        =       tos;
> > > --
> >
> > This patch made the server crash (just like one of the other crashes I've seen and posted to the list).
> >
> > The client logs:
> >
> > [  206.437839] rdma_rxe: bad ICRC from 192.168.1.92 [  211.043978] rdma_rxe: bad ICRC from 192.168.1.92 [  215.652973] rdma_rxe: bad ICRC from 192.168.1.92
> >
> >
> > Server crash:
> >
> > [11568.440098] BUG: unable to handle page fault for address: ffffaddb21f61180 [11568.442923] #PF: supervisor write access in kernel mode [11568.444452] #PF: error_code(0x0002) - not-present page [11568.445996] PGD 1000067 P4D 1000067 PUD 11b9067 PMD 0 [11568.447527] Oops: 0002 [#1] SMP PTI
> > [11568.448606] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W
> >   5.14.0-rc1+ #42
> > [11568.450911] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [11568.454072] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe] [11568.455613] Code: 8b b3 48 01 00 00 4d 8b 48 08 41 8b 48 28 49 8d
> > b9 80 01 00 00 85 f6 0f 84 78 01 00 00 41 8b 50 34 d3 e2 48 01 fa 48 8b 4d 00 <48> 89 0a 48 8b 4d 08 48 89 4a 08 48 8b 4d 10 48 89 4a 10 48 8b 4d [11568.461093] RSP: 0018:ffffaddb004c0988 EFLAGS: 00010082 [11568.462621] RAX: 0000000000000246 RBX: ffff9c9137df1a00 RCX: 0000000000000000 [11568.464695] RDX: ffffaddb21f61180 RSI: 0000000000000001 RDI: ffffaddb05f5f180 [11568.466779] RBP: ffffaddb004c0a30 R08: ffff9c9123186c00 R09: ffffaddb05f5f000 [11568.468902] R10: 80139a1c70550000 R11: 400000005d050000 R12: 0000000000000000 [11568.470977] R13: ffff9c9137df1b40 R14: ffff9c9137d50008 R15: 000000000000000a [11568.473050] FS:  0000000000000000(0000) GS:ffff9c917be40000(0000)
> > knlGS:0000000000000000
> > [11568.475395] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.477090] CR2: ffffaddb21f61180 CR3: 0000000043476005 CR4: 00000000001706e0 [11568.479170] Call Trace:
> > [11568.479966]  <IRQ>
> > [11568.480683]  rxe_responder+0x612/0x2470 [rdma_rxe] [11568.482122]  rxe_do_task+0x89/0x100 [rdma_rxe] [11568.483427]  rxe_rcv+0x2eb/0x900 [rdma_rxe] [11568.484655]  ? __udp4_lib_lookup+0x2c8/0x440 [11568.486159]  rxe_udp_encap_recv+0x68/0xa0 [rdma_rxe] [11568.487721]  ? rxe_enable_task+0x10/0x10 [rdma_rxe] [11568.489223]  udp_queue_rcv_one_skb+0x1df/0x4e0 [11568.490528]  udp_unicast_rcv_skb.isra.67+0x74/0x90
> > [11568.491926]  __udp4_lib_rcv+0x555/0xb90 [11568.493053]  ip_protocol_deliver_rcu+0xe8/0x1b0
> > [11568.494479]  ip_local_deliver_finish+0x44/0x50 [11568.496204]  ip_local_deliver+0xf1/0x100 [11568.497621]  ? ip_protocol_deliver_rcu+0x1b0/0x1b0
> > [11568.499147]  ip_rcv+0xcb/0xe0
> > [11568.500032]  __netif_receive_skb_core+0x3a2/0x1010
> > [11568.501491]  ? packet_rcv+0x40/0x4b0
> > [11568.502661]  ? select_idle_sibling+0x29/0x970 [11568.504019]  __netif_receive_skb_one_core+0x3c/0xa0
> > [11568.505455]  netif_receive_skb+0x3d/0x130 [11568.506650]  vmxnet3_rq_rx_complete+0x5f0/0xdc0 [vmxnet3] [11568.508808]  vmxnet3_poll_rx_only+0x31/0xa0 [vmxnet3] [11568.510526]  __napi_poll+0x2b/0x120 [11568.511596]  net_rx_action+0xe2/0x240 [11568.512678]  ? vmxnet3_msix_rx+0x4a/0x60 [vmxnet3] [11568.514084]  __do_softirq+0xd9/0x2a1 [11568.515218]  irq_exit_rcu+0xba/0xd0 [11568.516272]  common_interrupt+0x77/0x90 [11568.517438]  </IRQ> [11568.518059]  asm_common_interrupt+0x1e/0x40 [11568.519291] RIP: 0010:acpi_idle_do_entry+0x4c/0x50 [11568.520680] Code: 08 48 8b 15 3a e3 94 01 ed c3 e9 5f fc ff ff 65
> > 48 8b 04 25 00 6f 01 00 48 8b 00 a8 08 75 ea eb 07 0f 00 2d 40 41 50
> > 00 fb f4 <fa> c3 cc cc 0f 1f 44 00 00 41 55 41 89 d5 41 54 49 89 f4 55
> > 53 48
> > [11568.526026] RSP: 0018:ffffaddb0009be68 EFLAGS: 00000246 [11568.527569] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff9c917be40000 [11568.529627] RDX: 0000000000000001 RSI: ffffffff9dcc99c0 RDI: ffff9c917c03b464 [11568.531723] RBP: ffff9c9105f63400 R08: ffff9c917c03b400 R09: 000000000000b0e0 [11568.533772] R10: 0000000000001e99 R11: ffff9c917be6a984 R12: ffffffff9dcc9a40 [11568.535918] R13: ffffffff9dcc99c0 R14: 0000000000000001 R15: 0000000000000000 [11568.538558]  ? sched_clock_cpu+0x9/0xa0 [11568.539706]  acpi_idle_enter+0x4d/0xb0 [11568.540912]  cpuidle_enter_state+0x8c/0x350 [11568.542164]  cpuidle_enter+0x29/0x40 [11568.543211]  do_idle+0x257/0x2a0 [11568.544303]  cpu_startup_entry+0x19/0x20 [11568.545455]  start_secondary+0x116/0x150 [11568.546928]  secondary_startup_64_no_verify+0xc2/0xcb
> > [11568.548479] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core fuse rfcomm xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT
> > nf_reject_ipv4 nft_counter nft_compat nf_tables nfnetlink tun bridge stp llc vmw_vsock_vmci_transport vsock bnep snd_seq_midi snd_seq_midi_event intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul vmw_balloon ghash_clmulni_intel joydev pcspkr btusb btrtl btbcm btintel bluetooth uvcvideo rfkill videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_ens1371 snd_ac97_codec ac97_bus snd_seq videobuf2_common snd_pcm videodev mc ecdh_generic ecc snd_timer snd_rawmidi snd_seq_device snd soundcore vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sg crc32c_intel ata_generic vmwgfx ttm serio_raw nvme drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvme_core t10_pi cec ata_piix ahci vmxnet3 libahci drm libata [11568.575542] CR2: ffffaddb21f61180 [11568.577210] ---[ end trace 8afcc89bb91d9b85 ]--- [11568.578573] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe]
> >
> >
> >
> > > 2.30.2
> > >