All of lore.kernel.org
 help / color / mirror / Atom feed
From: Olga Kornievskaia <aglo@umich.edu>
To: Chuck Lever III <chuck.lever@oracle.com>
Cc: "kernel-tls-handshake@lists.linux.dev"
	<kernel-tls-handshake@lists.linux.dev>
Subject: Re: kernel oops
Date: Tue, 4 Apr 2023 15:26:10 -0400	[thread overview]
Message-ID: <CAN-5tyF5fGNzjjZ-3sy3=da+3CTeKu-Lq22hcJiDfy=qqghxTA@mail.gmail.com> (raw)
In-Reply-To: <C1E14C66-45BA-423B-AF57-95A96CED6F0A@oracle.com>

On Tue, Apr 4, 2023 at 3:14 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>
>
> > On Apr 4, 2023, at 3:11 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
> >
> > On Tue, Apr 4, 2023 at 1:28 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
> >>
> >>
> >>
> >>> On Apr 4, 2023, at 12:15 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
> >>>
> >>> On Tue, Apr 4, 2023 at 10:54 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> On Apr 4, 2023, at 10:44 AM, Olga Kornievskaia <aglo@umich.edu> wrote:
> >>>>>
> >>>>> On Tue, Apr 4, 2023 at 10:32 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
> >>>>>>
> >>>>>>> On Apr 4, 2023, at 10:24 AM, Olga Kornievskaia <aglo@umich.edu> wrote:
> >>>>>>>
> >>>>>>> On Fri, Mar 31, 2023 at 10:25 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mar 31, 2023, at 10:23 AM, Olga Kornievskaia <aglo@umich.edu> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Chuck,
> >>>>>>>>>
> >>>>>>>>> Have you seen this oops? This is not my testing so all I know was that
> >>>>>>>>> it was an attempt to mount with xprtsec=tls (against ONTAP).
> >>>>>>>>>
> >>>>>>>>> [  316.939747] BUG: kernel NULL pointer dereference, address: 0000000000000018
> >>>>>>>>> [  316.939848] #PF: supervisor read access in kernel mode
> >>>>>>>>> [  316.939914] #PF: error_code(0x0000) - not-present page
> >>>>>>>>> [  316.939984] PGD 0 P4D 0
> >>>>>>>>> [  316.940041] Oops: 0000 [#1] PREEMPT SMP PTI
> >>>>>>>>> [  316.940109] CPU: 0 PID: 511 Comm: kworker/u2:30 Kdump: loaded Not
> >>>>>>>>> tainted 6.2.1+ #2
> >>>>>>>>> [  316.940181] Hardware name: VMware, Inc. VMware7,1/440BX Desktop
> >>>>>>>>> Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
> >>>>>>>>> [  316.940259] Workqueue: xprtiod xs_tls_connect [sunrpc]
> >>>>>>>>> [  316.940463] RIP: 0010:xs_tls_connect+0x3e1/0x590 [sunrpc]
> >>>>>>>>> [  316.940556] Code: ff ff ff 48 2b 83 f8 fe ff ff 48 89 83 00 ff ff
> >>>>>>>>> ff e8 03 d3 ff ff e9 a8 fd ff ff 49 8b 97 f8 05 00 00 66 83 bb e0 f9
> >>>>>>>>> ff ff 0a <48> 8b 6a 18 0f 84 fd 00 00 00 48 8d 72 18 4c 89 e7 48 89 14
> >>>>>>>>> 24 e8
> >>>>>>>>> [  316.940719] RSP: 0018:ffffa69d40f5fdc8 EFLAGS: 00010293
> >>>>>>>>> [  316.940786] RAX: 0000000000000000 RBX: ffff9b0c46333e40 RCX: 0000000000000000
> >>>>>>>>> [  316.940872] RDX: 0000000000000000 RSI: ffffa69d40f5fd20 RDI: 0000000000000000
> >>>>>>>>> [  316.940940] RBP: ffff9b0c46350200 R08: ffff9b0cf7e31968 R09: 0000000000000159
> >>>>>>>>> [  316.941008] R10: 0000000000000001 R11: 00000049c8887a40 R12: ffff9b0c46333800
> >>>>>>>>> [  316.941075] R13: 0000000004208160 R14: ffff9b0c44e18a00 R15: ffff9b0c42dab800
> >>>>>>>>> [  316.941159] FS:  0000000000000000(0000) GS:ffff9b0cf7e00000(0000)
> >>>>>>>>> knlGS:0000000000000000
> >>>>>>>>> [  316.941243] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>>>>>>>> [  316.941309] CR2: 0000000000000018 CR3: 0000000102e48004 CR4: 00000000007706f0
> >>>>>>>>> [  316.941391] PKRU: 55555554
> >>>>>>>>> [  316.941443] Call Trace:
> >>>>>>>>> [  316.941513]  <TASK>
> >>>>>>>>> [  316.941578]  process_one_work+0x1b0/0x3c0
> >>>>>>>>> [  316.941689]  worker_thread+0x30/0x360
> >>>>>>>>> [  316.941755]  ? __pfx_worker_thread+0x10/0x10
> >>>>>>>>> [  316.941822]  kthread+0xd7/0x100
> >>>>>>>>> [  316.941900]  ? __pfx_kthread+0x10/0x10
> >>>>>>>>> [  316.941968]  ret_from_fork+0x29/0x50
> >>>>>>>>> [  316.942049]  </TASK>
> >>>>>>>>> [  316.942099] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4
> >>>>>>>>> dns_resolver ib_core nfsv3 nfs_acl nfs lockd grace fscache netfs ext4
> >>>>>>>>> rfkill mbcache jbd2 loop vsock_loopback
> >>>>>>>>> vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock
> >>>>>>>>> sunrpc tls vfat fat dm_multipath intel_rapl_msr intel_rapl_common
> >>>>>>>>> isst_if_mbox_msr isst_if_common nfit libnvdimm crct10dif_pclmul
> >>>>>>>>> crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr
> >>>>>>>>> i2c_piix4 vmw_vmci xfs libcrc32c vmwgfx drm_ttm_helper ttm sr_mod
> >>>>>>>>> cdrom drm_kms_helper syscopyarea sd_mod sysfillrect ata_generic
> >>>>>>>>> sysimgblt sg ahci libahci ata_piix crc32c_intel serio_raw drm vmxnet3
> >>>>>>>>> vmw_pvscsi libata dm_mirror dm_region_hash dm_log dm_mod nvme_tcp
> >>>>>>>>> nvme_fabrics nvme_core t10_pi crc64_rocksoft crc64 ipmi_devintf
> >>>>>>>>> ipmi_msghandler fuse
> >>>>>>>>> [  316.942550] CR2: 0000000000000018
> >>>>>>>>>
> >>>>>>>>> Please note this was still on "commit
> >>>>>>>>> caa52379f0046e974fd16585dd7ede624699d8e2 (HEAD -> 03152023-v7,
> >>>>>>>>> origin/upcall-v7)". If you think somethings done to address it I'll
> >>>>>>>>> have them update to the latest.
> >>>>>>>>
> >>>>>>>> Update and retry. It has a familiar ring, but I can't say I've
> >>>>>>>> seen this recently.
> >>>>>>>
> >>>>>>> Hi Chuck,
> >>>>>>>
> >>>>>>> I've updated the kernel code to the latest  commit
> >>>>>>> 46588cade103b446f74a7ee160a83cd2a80399e7 (HEAD ->
> >>>>>>> chuck-rpc-with-tls-04032023, chuck/topic-rpc-with-tls-upcall) but the
> >>>>>>> client is still crashing but looks like in a different place. This
> >>>>>>> happens when the server terminates the connection (immediately) as the
> >>>>>>> tls handshake happens.
> >>>>>>
> >>>>>> Why is the server doing that?
> >>>>>
> >>>>> I'm not sure what kind of a reason you are looking for. I think the
> >>>>> server can send an RST at any point in a TCP connection and the client
> >>>>> shouldn't crash.
> >>>>
> >>>> No doubt.
> >>>>
> >>>>
> >>>>> But the real reason is probably because the server's
> >>>>> code is getting developed and they ran into some error and resetting
> >>>>> the connection.
> >>>>
> >>>> That's what I was looking for. I wanted to know if it was
> >>>> Linux NFSD doing this.
> >>>>
> >>>>
> >>>>>>> Here's the stack trace:
> >>>>>>> [  677.274299] BUG: kernel NULL pointer dereference, address: 0000000000000030
> >>>>>>> [  677.274424] #PF: supervisor read access in kernel mode
> >>>>>>> [  677.274517] #PF: error_code(0x0000) - not-present page
> >>>>>>> [  677.274609] PGD 0 P4D 0
> >>>>>>> [  677.274689] Oops: 0000 [#1] PREEMPT SMP PTI
> >>>>>>> [  677.274785] CPU: 0 PID: 518 Comm: kworker/u2:29 Kdump: loaded Not
> >>>>>>> tainted 6.3.0-rc4+ #13
> >>>>>>> [  677.274881] Hardware name: VMware, Inc. VMware7,1/440BX Desktop
> >>>>>>> Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
> >>>>>>> [  677.274981] Workqueue: xprtiod xs_tls_connect [sunrpc]
> >>>>>>> [  677.275137] RIP: 0010:handshake_req_cancel+0x2b/0x3b0
> >>>>>>
> >>>>>> scripts/faddr2line on this RIP
> >>>>>>
> >>>>>> Which pointer is NULL?
> >>>>>
> >>>>> I apologize I don't have that info and I'm working on trying to figure
> >>>>> out how to get this information to you. Lab kernels are built with
> >>>>> CONFIG_DEBUG_INFO disabled and thus I can't do faddr2line. I will
> >>>>> attempt to build on with debugging enabled (has been problematic
> >>>>> before for the lab machines so not sure if that'll be successful I
> >>>>> might have to hack the linux kernel to simulate the reset).
> >>>>
> >>>> Jakub asked that NULL checking be removed from handshake_req_cancel,
> >>>> or at least that's what I read into his comment. That choice is
> >>>> definitely flushing bugs.
> >>>>
> >>>> So this crasher is familiar. req_cancel is getting called in some
> >>>> scenario where the socket or request is partially filled in. We
> >>>> have to figure out how that's happening.
> >>>>
> >>>> If debug info is a problem, then just add a sarcastic quantity of
> >>>> NULL checks in handshake_req_cancel, rebuild, and it should tell
> >>>> you exactly which pointer is bad.
> >>>
> >>> I added a check for sock-sk null and if so return true.
> >>
> >> If it returns true, xs_tls_handshake_sync() probably completes
> >> successfully.
> >>
> >> Perhaps handshake_req_cancel needs to return false in cases
> >> where it can't find a pending request.
> >
> > I'm not sure if you were suggesting me to do this but I tried:
> > if (!sock->sk) return false;
> > and that led to a mount not crashing but timing out.
>
> That's the behavior I would expect. Was the failure immediate,
> or did it take a few seconds for the mount command to exit?

It did take quite a long time to exit:

[root@scs000128074 kolga]# time mount -o vers=4.2,sec=sys,xprtsec=tls
10.63.20.131:/ /mnt
mount.nfs: Connection timed out

real    3m0.544s
user    0m0.004s
sys     0m0.013s

Network trace shows numerous (16) attempts to do NULL call

   66 2023-04-04 15:18:17.786579 10.235.232.7 → 10.63.20.131 TCP 74
953 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995198270 TSecr=0 WS=128
   67 2023-04-04 15:18:17.786818 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 953 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=2815692929 TSecr=3995198270
   68 2023-04-04 15:18:17.786871 10.235.232.7 → 10.63.20.131 TCP 66
953 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995198270
TSecr=2815692929
   69 2023-04-04 15:18:17.786951 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
   70 2023-04-04 15:18:17.787488 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 69)
   71 2023-04-04 15:18:17.787520 10.235.232.7 → 10.63.20.131 TCP 66
953 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995198271
TSecr=2815692929
   72 2023-04-04 15:18:17.818060 10.235.232.7 → 10.56.8.183  TCP 66
826 → 2049 [ACK] Seq=293 Ack=237 Win=501 Len=0 TSval=1599771772
TSecr=3825493593
   73 2023-04-04 15:18:17.993341 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 953 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2815693129
TSecr=3995198271
   74 2023-04-04 15:18:19.490669 fe80::21c:73ff:fe00:99 → ff02::1
ICMPv6 118 Router Advertisement from 00:1c:73:00:00:99
   75 2023-04-04 15:18:19.494076 fe80::250:56ff:febf:2f55 → ff02::16
  ICMPv6 90 Multicast Listener Report Message v2
   76 2023-04-04 15:18:19.841191 fe80::250:56ff:febf:2f55 → ff02::16
  ICMPv6 90 Multicast Listener Report Message v2
   77 2023-04-04 15:18:28.201642 10.235.232.7 → 10.63.20.131 TCP 74
692 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995208685 TSecr=0 WS=128
   78 2023-04-04 15:18:28.202171 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 692 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=3904561029 TSecr=3995208685
   79 2023-04-04 15:18:28.202236 10.235.232.7 → 10.63.20.131 TCP 66
692 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995208686
TSecr=3904561029
   80 2023-04-04 15:18:28.202506 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
   81 2023-04-04 15:18:28.202882 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 80)
   82 2023-04-04 15:18:28.202911 10.235.232.7 → 10.63.20.131 TCP 66
692 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995208686
TSecr=3904561029
   83 2023-04-04 15:18:28.329632 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 692 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=3904561159
TSecr=3995208686
   84 2023-04-04 15:18:34.857077 10.235.232.7 → 10.60.248.40 TCP 66
845 → 2049 [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=3160247691
TSecr=2389454709
   85 2023-04-04 15:18:34.857146 10.235.232.7 → 10.60.249.150 TCP 66
970 → 2049 [ACK] Seq=1 Ack=1 Win=502 Len=0 TSval=4116465771
TSecr=1272760298
   86 2023-04-04 15:18:34.857421 10.60.249.150 → 10.235.232.7 TCP 66
[TCP ACKed unseen segment] 2049 → 970 [ACK] Seq=1 Ack=2 Win=257 Len=0
TSval=1272820701 TSecr=4116405409
   87 2023-04-04 15:18:34.857422 10.60.248.40 → 10.235.232.7 TCP 66
[TCP ACKed unseen segment] 2049 → 845 [ACK] Seq=1 Ack=2 Win=257 Len=0
TSval=2389515123 TSecr=3160187318
   88 2023-04-04 15:18:38.441851 10.235.232.7 → 10.63.20.131 TCP 74
971 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995218925 TSecr=0 WS=128
   89 2023-04-04 15:18:38.442269 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 971 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=1698639803 TSecr=3995218925
   90 2023-04-04 15:18:38.442320 10.235.232.7 → 10.63.20.131 TCP 66
971 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995218926
TSecr=1698639803
   91 2023-04-04 15:18:38.442577 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
   92 2023-04-04 15:18:38.442894 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 91)
   93 2023-04-04 15:18:38.442921 10.235.232.7 → 10.63.20.131 TCP 66
971 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995218926
TSecr=1698639803
   94 2023-04-04 15:18:38.478175 10.63.20.131 → 10.235.232.7 TCP 66
[TCP Dup ACK 92#1] 2049 → 971 [ACK] Seq=37 Ack=45 Win=65792 Len=0
TSval=1698639843 TSecr=3995218926
   95 2023-04-04 15:18:38.614265 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 971 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1698639973
TSecr=3995218926
   96 2023-04-04 15:18:48.681763 10.235.232.7 → 10.63.20.131 TCP 74
799 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995229165 TSecr=0 WS=128
   97 2023-04-04 15:18:48.682176 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 799 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=1773399631 TSecr=3995229165
   98 2023-04-04 15:18:48.682203 10.235.232.7 → 10.63.20.131 TCP 66
799 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995229166
TSecr=1773399631
   99 2023-04-04 15:18:48.682260 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  100 2023-04-04 15:18:48.682751 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 99)
  101 2023-04-04 15:18:48.682761 10.235.232.7 → 10.63.20.131 TCP 66
799 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995229166
TSecr=1773399631
  102 2023-04-04 15:18:48.826843 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 799 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1773399771
TSecr=3995229166
  103 2023-04-04 15:18:49.514227 10.235.232.170 → 10.235.235.255
BROWSER 243 Host Announcement WINDOWS-G95UIQM, Workstation, Server, NT
Workstation, NT Server
  104 2023-04-04 15:18:58.921556 10.235.232.7 → 10.63.20.131 TCP 74
729 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995239405 TSecr=0 WS=128
  105 2023-04-04 15:18:58.921926 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 729 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=2487545914 TSecr=3995239405
  106 2023-04-04 15:18:58.921953 10.235.232.7 → 10.63.20.131 TCP 66
729 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995239405
TSecr=2487545914
  107 2023-04-04 15:18:58.922027 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  108 2023-04-04 15:18:58.922802 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 107)
  109 2023-04-04 15:18:58.922822 10.235.232.7 → 10.63.20.131 TCP 66
729 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995239406
TSecr=2487545914
  110 2023-04-04 15:18:59.050600 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 729 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2487546044
TSecr=3995239406
  111 2023-04-04 15:19:06.533098 10.235.232.128 → 10.235.235.255
BROWSER 243 Host Announcement SCS000126759-1, Workstation, Server, NT
Workstation, NT Server
  112 2023-04-04 15:19:09.161611 10.235.232.7 → 10.63.20.131 TCP 74
954 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995249645 TSecr=0 WS=128
  113 2023-04-04 15:19:09.161949 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 954 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=3804453828 TSecr=3995249645
  114 2023-04-04 15:19:09.162009 10.235.232.7 → 10.63.20.131 TCP 66
954 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995249645
TSecr=3804453828
  115 2023-04-04 15:19:09.162055 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  116 2023-04-04 15:19:09.162528 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 115)
  117 2023-04-04 15:19:09.162554 10.235.232.7 → 10.63.20.131 TCP 66
954 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995249646
TSecr=3804453828
  118 2023-04-04 15:19:09.278841 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 954 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=3804453948
TSecr=3995249646
  119 2023-04-04 15:19:12.003989 Vmware_bf:32:23 → Broadcast    ARP 60
Who has 10.235.232.1? Tell 10.235.232.42
  120 2023-04-04 15:19:19.401546 10.235.232.7 → 10.63.20.131 TCP 74
728 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995259885 TSecr=0 WS=128
  121 2023-04-04 15:19:19.401902 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 728 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=2468567773 TSecr=3995259885
  122 2023-04-04 15:19:19.401965 10.235.232.7 → 10.63.20.131 TCP 66
728 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995259885
TSecr=2468567773
  123 2023-04-04 15:19:19.402050 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  124 2023-04-04 15:19:19.402542 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 123)
  125 2023-04-04 15:19:19.402554 10.235.232.7 → 10.63.20.131 TCP 66
728 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995259886
TSecr=2468567773
  126 2023-04-04 15:19:19.595961 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 728 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2468567963
TSecr=3995259886
  127 2023-04-04 15:19:21.449077 10.235.232.7 → 10.60.249.125 TCP 66
[TCP Keep-Alive] 732 → 2049 [ACK] Seq=2176 Ack=1857 Win=502 Len=0
TSval=1628408100 TSecr=708634268
  128 2023-04-04 15:19:21.449206 10.235.232.7 → 10.56.8.183  TCP 66
[TCP Keep-Alive] 826 → 2049 [ACK] Seq=292 Ack=237 Win=501 Len=0
TSval=1599835403 TSecr=3825493593
  129 2023-04-04 15:19:21.449284 10.60.249.125 → 10.235.232.7 TCP 66
[TCP Keep-Alive ACK] 2049 → 732 [ACK] Seq=1857 Ack=2177 Win=257 Len=0
TSval=708698066 TSecr=1628344342
  130 2023-04-04 15:19:21.511308  10.56.8.183 → 10.235.232.7 TCP 66
[TCP Keep-Alive ACK] 2049 → 826 [ACK] Seq=237 Ack=293 Win=257 Len=0
TSval=3825557328 TSecr=1599771772
  131 2023-04-04 15:19:29.641659 10.235.232.7 → 10.63.20.131 TCP 74
924 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995270125 TSecr=0 WS=128
  132 2023-04-04 15:19:29.641987 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 924 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=4239335290 TSecr=3995270125
  133 2023-04-04 15:19:29.642062 10.235.232.7 → 10.63.20.131 TCP 66
924 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995270125
TSecr=4239335290
  134 2023-04-04 15:19:29.642188 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  135 2023-04-04 15:19:29.642585 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 134)
  136 2023-04-04 15:19:29.642626 10.235.232.7 → 10.63.20.131 TCP 66
924 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995270126
TSecr=4239335290
  137 2023-04-04 15:19:29.752894 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 924 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=4239335400
TSecr=3995270126
  138 2023-04-04 15:19:37.833079 10.235.232.7 → 10.60.248.40 TCP 66
[TCP Dup ACK 84#1] 845 → 2049 [ACK] Seq=1 Ack=1 Win=501 Len=0
TSval=3160310667 TSecr=2389515123
  139 2023-04-04 15:19:37.833122 10.235.232.7 → 10.60.249.150 TCP 66
[TCP Dup ACK 85#1] 970 → 2049 [ACK] Seq=1 Ack=1 Win=502 Len=0
TSval=4116528747 TSecr=1272820701
  140 2023-04-04 15:19:37.833308 10.60.248.40 → 10.235.232.7 TCP 66
[TCP Dup ACK 87#1] [TCP ACKed unseen segment] 2049 → 845 [ACK] Seq=1
Ack=2 Win=257 Len=0 TSval=2389578099 TSecr=3160187318
  141 2023-04-04 15:19:37.833346 10.60.249.150 → 10.235.232.7 TCP 66
[TCP Dup ACK 86#1] [TCP ACKed unseen segment] 2049 → 970 [ACK] Seq=1
Ack=2 Win=257 Len=0 TSval=1272883677 TSecr=4116405409
  142 2023-04-04 15:19:39.881507 10.235.232.7 → 10.63.20.131 TCP 74
788 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995280365 TSecr=0 WS=128
  143 2023-04-04 15:19:39.881784 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 788 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=2800122551 TSecr=3995280365
  144 2023-04-04 15:19:39.881847 10.235.232.7 → 10.63.20.131 TCP 66
788 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995280365
TSecr=2800122551
  145 2023-04-04 15:19:39.881898 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  146 2023-04-04 15:19:39.882359 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 145)
  147 2023-04-04 15:19:39.882395 10.235.232.7 → 10.63.20.131 TCP 66
788 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995280366
TSecr=2800122551
  148 2023-04-04 15:19:40.017113 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 788 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2800122681
TSecr=3995280366
  149 2023-04-04 15:19:50.122254 10.235.232.7 → 10.63.20.131 TCP 74
868 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995290606 TSecr=0 WS=128
  150 2023-04-04 15:19:50.122643 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 868 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=1996033015 TSecr=3995290606
  151 2023-04-04 15:19:50.122674 10.235.232.7 → 10.63.20.131 TCP 66
868 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995290606
TSecr=1996033015
  152 2023-04-04 15:19:50.122802 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  153 2023-04-04 15:19:50.123263 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 152)
  154 2023-04-04 15:19:50.123279 10.235.232.7 → 10.63.20.131 TCP 66
868 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995290607
TSecr=1996033015
  155 2023-04-04 15:19:50.268177 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 868 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1996033165
TSecr=3995290607
  156 2023-04-04 15:20:00.361440 10.235.232.7 → 10.63.20.131 TCP 74
988 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995300845 TSecr=0 WS=128
  157 2023-04-04 15:20:00.361776 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 988 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=4120839794 TSecr=3995300845
  158 2023-04-04 15:20:00.361791 10.235.232.7 → 10.63.20.131 TCP 66
988 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995300845
TSecr=4120839794
  159 2023-04-04 15:20:00.361841 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  160 2023-04-04 15:20:00.362321 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 159)
  161 2023-04-04 15:20:00.362327 10.235.232.7 → 10.63.20.131 TCP 66
988 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995300846
TSecr=4120839794
  162 2023-04-04 15:20:00.479087 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 988 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=4120839904
TSecr=3995300846
  163 2023-04-04 15:20:01.559017 10.235.232.173 → 10.235.235.255
BROWSER 243 Host Announcement WINDOWS-PTEFTLL, Workstation, Server, NT
Workstation, NT Server
  164 2023-04-04 15:20:10.426553 10.235.232.7 → 10.60.248.183 NTP 90
NTP Version 4, client
  165 2023-04-04 15:20:10.426824 10.60.248.183 → 10.235.232.7 NTP 90
NTP Version 4, server
  166 2023-04-04 15:20:10.601625 10.235.232.7 → 10.63.20.131 TCP 74
888 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995311085 TSecr=0 WS=128
  167 2023-04-04 15:20:10.601919 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 888 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=531138781 TSecr=3995311085
  168 2023-04-04 15:20:10.601978 10.235.232.7 → 10.63.20.131 TCP 66
888 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995311085
TSecr=531138781
  169 2023-04-04 15:20:10.602058 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  170 2023-04-04 15:20:10.602516 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 169)
  171 2023-04-04 15:20:10.602543 10.235.232.7 → 10.63.20.131 TCP 66
888 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995311086
TSecr=531138781
  172 2023-04-04 15:20:10.731005 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 888 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=531138911
TSecr=3995311086
  173 2023-04-04 15:20:11.998469 10.235.232.7 → 10.235.48.111 NTP 90
NTP Version 4, client
  174 2023-04-04 15:20:11.998803 10.235.48.111 → 10.235.232.7 NTP 90
NTP Version 4, server
  175 2023-04-04 15:20:12.648789 10.235.232.7 → 10.227.204.142 NTP 90
NTP Version 4, client
  176 2023-04-04 15:20:12.649163 10.227.204.142 → 10.235.232.7 NTP 90
NTP Version 4, server
  177 2023-04-04 15:20:20.841429 10.235.232.7 → 10.63.20.131 TCP 74
893 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995321325 TSecr=0 WS=128
  178 2023-04-04 15:20:20.841673 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 893 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=1953607493 TSecr=3995321325
  179 2023-04-04 15:20:20.841688 10.235.232.7 → 10.63.20.131 TCP 66
893 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995321325
TSecr=1953607493
  180 2023-04-04 15:20:20.841767 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  181 2023-04-04 15:20:20.842198 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 180)
  182 2023-04-04 15:20:20.842208 10.235.232.7 → 10.63.20.131 TCP 66
893 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995321326
TSecr=1953607493
  183 2023-04-04 15:20:21.051706 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 893 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1953607703
TSecr=3995321326
  184 2023-04-04 15:20:22.889123 10.235.232.7 → 10.56.8.183  TCP 66
[TCP Keep-Alive] 826 → 2049 [ACK] Seq=292 Ack=237 Win=501 Len=0
TSval=1599896843 TSecr=3825557328
  185 2023-04-04 15:20:22.889130 10.235.232.7 → 10.60.249.125 TCP 66
[TCP Keep-Alive] 732 → 2049 [ACK] Seq=2176 Ack=1857 Win=502 Len=0
TSval=1628469540 TSecr=708698066
  186 2023-04-04 15:20:22.889335 10.60.249.125 → 10.235.232.7 TCP 66
[TCP Keep-Alive ACK] 2049 → 732 [ACK] Seq=1857 Ack=2177 Win=257 Len=0
TSval=708759507 TSecr=1628344342
  187 2023-04-04 15:20:22.951234  10.56.8.183 → 10.235.232.7 TCP 66
[TCP Keep-Alive ACK] 2049 → 826 [ACK] Seq=237 Ack=293 Win=257 Len=0
TSval=3825618767 TSecr=1599771772
  188 2023-04-04 15:20:27.226390 10.235.232.131 → 10.235.235.255
BROWSER 243 Host Announcement SCS000126759-3, Workstation, Server, NT
Workstation, NT Server
  189 2023-04-04 15:20:31.081384 10.235.232.7 → 10.63.20.131 TCP 74
1013 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995331565 TSecr=0 WS=128
  190 2023-04-04 15:20:31.081679 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 1013 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=1446866364 TSecr=3995331565
  191 2023-04-04 15:20:31.081698 10.235.232.7 → 10.63.20.131 TCP 66
1013 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995331565
TSecr=1446866364
  192 2023-04-04 15:20:31.081745 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  193 2023-04-04 15:20:31.082347 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 192)
  194 2023-04-04 15:20:31.082353 10.235.232.7 → 10.63.20.131 TCP 66
1013 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995331566
TSecr=1446866364
  195 2023-04-04 15:20:31.216474 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 1013 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1446866494
TSecr=3995331566
  196 2023-04-04 15:20:39.273073 10.235.232.7 → 10.60.249.150 TCP 66
[TCP Dup ACK 85#2] 970 → 2049 [ACK] Seq=1 Ack=1 Win=502 Len=0
TSval=4116590187 TSecr=1272883677
  197 2023-04-04 15:20:39.273146 10.235.232.7 → 10.60.248.40 TCP 66
[TCP Dup ACK 84#2] 845 → 2049 [ACK] Seq=1 Ack=1 Win=501 Len=0
TSval=3160372107 TSecr=2389578099
  198 2023-04-04 15:20:39.273342 10.60.249.150 → 10.235.232.7 TCP 66
[TCP Dup ACK 86#2] [TCP ACKed unseen segment] 2049 → 970 [ACK] Seq=1
Ack=2 Win=257 Len=0 TSval=1272945117 TSecr=4116405409
  199 2023-04-04 15:20:39.274592 10.60.248.40 → 10.235.232.7 TCP 66
[TCP Dup ACK 87#2] [TCP ACKed unseen segment] 2049 → 845 [ACK] Seq=1
Ack=2 Win=257 Len=0 TSval=2389639540 TSecr=3160187318
  200 2023-04-04 15:20:41.321912 10.235.232.7 → 10.63.20.131 TCP 74
676 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995341805 TSecr=0 WS=128
  201 2023-04-04 15:20:41.322226 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 676 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=710203011 TSecr=3995341805
  202 2023-04-04 15:20:41.322258 10.235.232.7 → 10.63.20.131 TCP 66
676 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995341806
TSecr=710203011
  203 2023-04-04 15:20:41.322382 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  204 2023-04-04 15:20:41.322902 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 203)
  205 2023-04-04 15:20:41.322923 10.235.232.7 → 10.63.20.131 TCP 66
676 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995341806
TSecr=710203011
  206 2023-04-04 15:20:41.502713 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 676 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=710203191
TSecr=3995341806
  207 2023-04-04 15:20:41.671253 fe80::21c:73ff:fe00:99 → ff02::1
ICMPv6 118 Router Advertisement from 00:1c:73:00:00:99
  208 2023-04-04 15:20:41.675089 fe80::250:56ff:febf:2f55 → ff02::16
  ICMPv6 90 Multicast Listener Report Message v2
  209 2023-04-04 15:20:41.684071 fe80::250:56ff:febf:2f55 → ff02::16
  ICMPv6 90 Multicast Listener Report Message v2
  210 2023-04-04 15:20:43.709117 10.235.232.130 → 10.235.235.255
BROWSER 243 Host Announcement SCS000126759-2, Workstation, Server, NT
Workstation, NT Server
  211 2023-04-04 15:20:51.561341 10.235.232.7 → 10.63.20.131 TCP 74
864 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995352045 TSecr=0 WS=128
  212 2023-04-04 15:20:51.561677 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 864 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=1105202286 TSecr=3995352045
  213 2023-04-04 15:20:51.561736 10.235.232.7 → 10.63.20.131 TCP 66
864 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995352045
TSecr=1105202286
  214 2023-04-04 15:20:51.561764 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  215 2023-04-04 15:20:51.562296 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 214)
  216 2023-04-04 15:20:51.562324 10.235.232.7 → 10.63.20.131 TCP 66
864 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995352046
TSecr=1105202286
  217 2023-04-04 15:20:51.764952 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 864 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1105202486
TSecr=3995352046
  218 2023-04-04 15:21:01.801569 10.235.232.7 → 10.63.20.131 TCP 74
779 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995362285 TSecr=0 WS=128
  219 2023-04-04 15:21:01.801903 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 779 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=3669197173 TSecr=3995362285
  220 2023-04-04 15:21:01.801938 10.235.232.7 → 10.63.20.131 TCP 66
779 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995362285
TSecr=3669197173
  221 2023-04-04 15:21:01.801991 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  222 2023-04-04 15:21:01.802750 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 221)
  223 2023-04-04 15:21:01.802761 10.235.232.7 → 10.63.20.131 TCP 66
779 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995362286
TSecr=3669197173
  224 2023-04-04 15:21:01.954179 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 779 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=3669197323
TSecr=3995362286
  225 2023-04-04 15:21:10.142734 fe80::21c:73ff:fe00:99 → ff02::1
ICMPv6 118 Router Advertisement from 00:1c:73:00:00:99
  226 2023-04-04 15:21:10.146082 fe80::250:56ff:febf:2f55 → ff02::16
  ICMPv6 90 Multicast Listener Report Message v2
  227 2023-04-04 15:21:10.889124 fe80::250:56ff:febf:2f55 → ff02::16
  ICMPv6 90 Multicast Listener Report Message v2
  228 2023-04-04 15:21:12.041741 10.235.232.7 → 10.63.20.131 TCP 74
946 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
TSval=3995372525 TSecr=0 WS=128
  229 2023-04-04 15:21:12.042058 10.63.20.131 → 10.235.232.7 TCP 74
2049 → 946 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
SACK_PERM=1 TSval=1985286575 TSecr=3995372525
  230 2023-04-04 15:21:12.042121 10.235.232.7 → 10.63.20.131 TCP 66
946 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995372526
TSecr=1985286575
  231 2023-04-04 15:21:12.042404 10.235.232.7 → 10.63.20.131 NFS 110
V4 NULL Call
  232 2023-04-04 15:21:12.042673 10.63.20.131 → 10.235.232.7 NFS 102
V4 NULL Reply (Call In 231)
  233 2023-04-04 15:21:12.042710 10.235.232.7 → 10.63.20.131 TCP 66
946 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995372526
TSecr=1985286575
  234 2023-04-04 15:21:12.079429 10.63.20.131 → 10.235.232.7 TCP 66
[TCP Dup ACK 232#1] 2049 → 946 [ACK] Seq=37 Ack=45 Win=65792 Len=0
TSval=1985286615 TSecr=3995372526
  235 2023-04-04 15:21:12.184895 10.63.20.131 → 10.235.232.7 TCP 66
2049 → 946 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1985286705
TSecr=3995372526
  236 2023-04-04 15:21:18.186337 10.235.232.7 → 10.60.250.78 SSH 150
Server: Encrypted packet (len=84)
  237 2023-04-04 15:21:18.186573 10.60.250.78 → 10.235.232.7 TCP 66
45496 → 22 [ACK] Seq=417 Ack=629 Win=501 Len=0 TSval=2780826399
TSecr=3765675462
  238 2023-04-04 15:21:18.187900 10.235.232.7 → 10.60.250.78 SSH 150
Server: Encrypted packet (len=84)
  239 2023-04-04 15:21:18.188065 10.60.250.78 → 10.235.232.7 TCP 66
45496 → 22 [ACK] Seq=417 Ack=713 Win=501 Len=0 TSval=2780826401
TSecr=3765675463
  240 2023-04-04 15:21:18.188154 10.235.232.7 → 10.60.250.78 SSH 150
Server: Encrypted packet (len=84)
  241 2023-04-04 15:21:18.188283 10.60.250.78 → 10.235.232.7 TCP 66
45496 → 22 [ACK] Seq=417 Ack=797 Win=501 Len=0 TSval=2780826401
TSecr=3765675464
  242 2023-04-04 15:21:18.188455 10.235.232.7 → 10.60.250.78 SSH 134
Server: Encrypted packet (len=68)
  243 2023-04-04 15:21:18.188599 10.60.250.78 → 10.235.232.7 TCP 66
45496 → 22 [ACK] Seq=417 Ack=865 Win=501 Len=0 TSval=2780826401
TSecr=3765675464
  244 2023-04-04 15:21:22.539976 fe80::21c:73ff:fe00:99 →
fe80::250:56ff:febf:5abd ICMPv6 118 Router Advertisement from
00:1c:73:00:00:99
  245 2023-04-04 15:21:22.764914 fe80::21c:73ff:fe00:99 →
fe80::250:56ff:febf:5abd ICMPv6 118 Router Advertisement from
00:1c:73:00:00:99
  246 2023-04-04 15:21:24.329146 10.235.232.7 → 10.56.8.183  TCP 66
[TCP Keep-Alive] 826 → 2049 [ACK] Seq=292 Ack=237 Win=501 Len=0
TSval=1599958283 TSecr=3825618767
  247 2023-04-04 15:21:24.329156 10.235.232.7 → 10.60.249.125 TCP 66
[TCP Keep-Alive] 732 → 2049 [ACK] Seq=2176 Ack=1857 Win=502 Len=0
TSval=1628530980 TSecr=708759507
  248 2023-04-04 15:21:24.329506 10.60.249.125 → 10.235.232.7 TCP 66
[TCP Keep-Alive ACK] 2049 → 732 [ACK] Seq=1857 Ack=2177 Win=257 Len=0
TSval=708820947 TSecr=1628344342
  249 2023-04-04 15:21:24.391262  10.56.8.183 → 10.235.232.7 TCP 66
[TCP Keep-Alive ACK] 2049 → 826 [ACK] Seq=237 Ack=293 Win=257 Len=0
TSval=3825680208 TSecr=1599771772

>
>
> >>> I thought that
> >>> would "prevent" the crash but it then proceeded to crash elsewhere.
> >>> Sorry I'm still on non-debug-symbol kernel (as I thought I'd add the
> >>> checks that that'd be enough :)).
> >>>
> >>> [  284.368211] AGLO: handshake_req_cancel sock->sk is NULL
> >>> [  284.368249] BUG: kernel NULL pointer dereference, address: 0000000000000018
> >>> [  284.368320] #PF: supervisor read access in kernel mode
> >>> [  284.368363] #PF: error_code(0x0000) - not-present page
> >>> [  284.368411] PGD 0 P4D 0
> >>> [  284.368438] Oops: 0000 [#1] PREEMPT SMP PTI
> >>> [  284.368473] CPU: 0 PID: 521 Comm: kworker/u2:30 Kdump: loaded Not
> >>> tainted 6.3.0-rc4+ #14
> >>> [  284.368523] Hardware name: VMware, Inc. VMware7,1/440BX Desktop
> >>> Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
> >>> [  284.368580] Workqueue: xprtiod xs_tls_connect [sunrpc]
> >>> [  284.368720] RIP: 0010:xs_tls_connect+0x3e5/0x590 [sunrpc]
> >>> [  284.368844] Code: ff ff ff 48 2b 83 f8 fe ff ff 48 89 83 00 ff ff
> >>> ff e8 bf d2 ff ff e9 a8 fd ff ff 49 8b 97 f8 05 00 00 66 83 bb e0 f9
> >>> ff ff 0a <48> 8b 6a 18 0f 84 fd 00 00 00 48 8d 72 18 4c 89 e7 48 89 14
> >>> 24 e8
> >>> [  284.368973] RSP: 0018:ffffba17c0f67dc8 EFLAGS: 00010293
> >>> [  284.369010] RAX: 0000000000000000 RBX: ffff963143c1fe40 RCX: 0000000000000000
> >>> [  284.369051] RDX: 0000000000000000 RSI: ffff9631f7e20840 RDI: ffff9631f7e20840
> >>> [  284.369091] RBP: ffff9631463d8a00 R08: 0000000000000000 R09: c0000000ffff7fff
> >>> [  284.369130] R10: 0000000000000001 R11: ffffba17c0f67bb8 R12: ffff963143c1f800
> >>> [  284.369175] R13: 0000000004208160 R14: ffff963143f6f600 R15: ffff963143c1e000
> >>> [  284.369259] FS:  0000000000000000(0000) GS:ffff9631f7e00000(0000)
> >>> knlGS:0000000000000000
> >>> [  284.369319] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>> [  284.369367] CR2: 0000000000000018 CR3: 0000000102d12004 CR4: 00000000007706f0
> >>> [  284.369458] PKRU: 55555554
> >>> [  284.369483] Call Trace:
> >>> [  284.369523]  <TASK>
> >>> [  284.369583]  process_one_work+0x1b0/0x3c0
> >>> [  284.369633]  worker_thread+0x34/0x360
> >>> [  284.369668]  ? __pfx_worker_thread+0x10/0x10
> >>> [  284.369704]  kthread+0xdb/0x110
> >>> [  284.369736]  ? __pfx_kthread+0x10/0x10
> >>> [  284.369767]  ret_from_fork+0x29/0x50
> >>> [  284.369808]  </TASK>
> >>> [  284.369843] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4
> >>> dns_resolver nfsv3 nfs_acl nfs lockd grace fscache netfs ext4 mbcache
> >>> jbd2 loop rfkill vsock_loopback vmw_vsock_virtio_transport_common
> >>> vmw_vsock_vmci_transport vsock sunrpc vfat fat dm_multipath
> >>> intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit
> >>> libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl
> >>> vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 xfs libcrc32c vmwgfx
> >>> drm_ttm_helper ttm sr_mod cdrom drm_kms_helper sd_mod syscopyarea
> >>> sysfillrect ata_generic sysimgblt sg ata_piix ahci libahci
> >>> crc32c_intel serio_raw vmxnet3 libata drm vmw_pvscsi dm_mirror
> >>> dm_region_hash dm_log dm_mod nvme_tcp nvme_fabrics nvme_core t10_pi
> >>> crc64_rocksoft crc64 ipmi_devintf ipmi_msghandler fuse
> >>> [  284.370300] CR2: 0000000000000018
> >>>
> >>>
> >>>>
> >>>> BPF might also be a solution, but I'm not sure if it relies on
> >>>> debug info. Might be only BTF is needed.
> >>>>
> >>>>
> >>>>>>> [  677.275269] Code: 0f 1e fa 0f 1f 44 00 00 41 57 41 56 41 55 41 54
> >>>>>>> 55 53 48 83 ec 28 48 8b 6f 18 65 48 8b 04 25 28 00 00 00 48 89 44 24
> >>>>>>> 20 31 c0 <48> 8b 5d 30 48 89 6c 24 18 e8 97 15 5f ff 4c 8b 25 50 2e 70
> >>>>>>> 01 41
> >>>>>>> [  677.275474] RSP: 0018:ffffaed400f6fcf8 EFLAGS: 00010246
> >>>>>>> [  677.275570] RAX: 0000000000000000 RBX: ffff93f085b33800 RCX: 0000000000000000
> >>>>>>> [  677.275663] RDX: ffff93f085b33f08 RSI: ffffaed400f6fd18 RDI: ffff93f083aa6700
> >>>>>>> [  677.275756] RBP: 0000000000000000 R08: ffff93f137e32968 R09: 0000009d9663fa40
> >>>>>>> [  677.275850] R10: 0000000000000000 R11: 0000009d9663fa40 R12: ffff93f083aa6700
> >>>>>>> [  677.275943] R13: 0000000000000000 R14: ffff93f082999e00 R15: ffff93f085b33800
> >>>>>>> [  677.276049] FS:  0000000000000000(0000) GS:ffff93f137e00000(0000)
> >>>>>>> knlGS:0000000000000000
> >>>>>>> [  677.276154] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>>>>>> [  677.276247] CR2: 0000000000000030 CR3: 000000010797c002 CR4: 00000000007706f0
> >>>>>>> [  677.276355] PKRU: 55555554
> >>>>>>> [  677.276427] Call Trace:
> >>>>>>> [  677.276521]  <TASK>
> >>>>>>> [  677.276605]  ? wait_for_completion_interruptible_timeout+0xfa/0x170
> >>>>>>> [  677.276725]  xs_tls_handshake_sync+0x14f/0x170 [sunrpc]
> >>>>>>> [  677.276850]  ? __pfx_xs_tls_handshake_done+0x10/0x10 [sunrpc]
> >>>>>>> [  677.276967]  xs_tls_connect+0x14c/0x590 [sunrpc]
> >>>>>>> [  677.277086]  process_one_work+0x1b0/0x3c0
> >>>>>>> [  677.277209]  worker_thread+0x34/0x360
> >>>>>>> [  677.277303]  ? __pfx_worker_thread+0x10/0x10
> >>>>>>> [  677.277396]  kthread+0xdb/0x110
> >>>>>>> [  677.277527]  ? __pfx_kthread+0x10/0x10
> >>>>>>> [  677.277619]  ret_from_fork+0x29/0x50
> >>>>>>> [  677.277731]  </TASK>
> >>>>>>> [  677.277803] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4
> >>>>>>> dns_resolver nfsv3 nfs_acl nfs lockd grace fscache netfs ext4 mbcache
> >>>>>>> jbd2 loop rfkill vsock_loopback vmw_vsock_virtio_transport_common
> >>>>>>> vmw_vsock_vmci_transport vsock sunrpc vfat fat dm_multipath
> >>>>>>> intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit
> >>>>>>> libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl
> >>>>>>> vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 xfs libcrc32c vmwgfx
> >>>>>>> drm_ttm_helper ttm sr_mod cdrom drm_kms_helper syscopyarea sysfillrect
> >>>>>>> sd_mod ata_generic sysimgblt sg ata_piix ahci libahci crc32c_intel
> >>>>>>> serio_raw drm vmxnet3 vmw_pvscsi libata dm_mirror dm_region_hash
> >>>>>>> dm_log dm_mod nvme_tcp nvme_fabrics nvme_core t10_pi crc64_rocksoft
> >>>>>>> crc64 ipmi_devintf ipmi_msghandler fuse
> >>>>>>> [  677.278383] CR2: 0000000000000030
> >>>>>>>
> >>>>>>> Network trace from the server:
> >>>>>>>
> >>>>>>> [kolga@scs000085536 ~/pcaps]$ tshark -t ad -r e0d_20230404_084311.trc0
> >>>>>>> 1 2023-04-04 08:43:28.027546 10.235.232.77 → 10.63.20.131 TCP 74
> >>>>>>> 727 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> >>>>>>> TSval=4059503415 TSecr=0 WS=128
> >>>>>>> 2 2023-04-04 08:43:28.027645 10.63.20.131 → 10.235.232.77 TCP 74
> >>>>>>> 2049 → 727 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> >>>>>>> SACK_PERM=1 TSval=2256845073 TSecr=4059503415
> >>>>>>> 3 2023-04-04 08:43:28.027813 10.235.232.77 → 10.63.20.131 TCP 66
> >>>>>>> 727 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=4059503415
> >>>>>>> TSecr=2256845073
> >>>>>>> 4 2023-04-04 08:43:28.027952 10.235.232.77 → 10.63.20.131 NFS 110
> >>>>>>> V4 NULL Call
> >>>>>>> 5 2023-04-04 08:43:28.028301 10.63.20.131 → 10.235.232.77 NFS 102
> >>>>>>> V4 NULL Reply (Call In 4)
> >>>>>>> 6 2023-04-04 08:43:28.028563 10.235.232.77 → 10.63.20.131 TCP 66
> >>>>>>> 727 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=4059503416
> >>>>>>> TSecr=2256845073
> >>>>>>> 7 2023-04-04 08:43:28.331181 10.63.20.131 → 10.235.232.77 TCP 66
> >>>>>>> 2049 → 727 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2256845383
> >>>>>>> TSecr=4059503416
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Chuck Lever
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Chuck Lever
> >>>>
> >>>>
> >>>> --
> >>>> Chuck Lever
> >>
> >>
> >> --
> >> Chuck Lever
>
>
> --
> Chuck Lever
>
>

  reply	other threads:[~2023-04-04 19:26 UTC|newest]

Thread overview: 97+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-03-31 14:23 kernel oops Olga Kornievskaia
2023-03-31 14:25 ` Chuck Lever III
2023-04-04 14:24   ` Olga Kornievskaia
2023-04-04 14:31     ` Chuck Lever III
2023-04-04 14:44       ` Olga Kornievskaia
2023-04-04 14:54         ` Chuck Lever III
2023-04-04 16:15           ` Olga Kornievskaia
2023-04-04 17:28             ` Chuck Lever III
2023-04-04 19:11               ` Olga Kornievskaia
2023-04-04 19:14                 ` Chuck Lever III
2023-04-04 19:26                   ` Olga Kornievskaia [this message]
2023-04-04 19:30                     ` Chuck Lever III
  -- strict thread matches above, loose matches on Subject: below --
2017-07-24 21:16 Kernel oops Jason Gunthorpe
     [not found] ` <20170724211606.GA1705-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-27 11:46   ` Matan Barak
     [not found]     ` <CAAKD3BAdB2aRk3WGdbeDYof6dUfkEwhQf27cG0FWe5DRuQ15NQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-07-27 12:54       ` Matan Barak
     [not found]         ` <CAAKD3BDFrTMMgX0nErD50rp2je=HC9zeaYWHDKf0mqQwc5fM9g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-07-27 20:44           ` Jason Gunthorpe
     [not found]             ` <20170727204437.GA16986-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-30 10:25               ` Leon Romanovsky
     [not found]                 ` <20170730102514.GQ13672-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-07-31  3:52                   ` Jason Gunthorpe
     [not found]                     ` <20170731035208.GA30615-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-31  5:39                       ` Leon Romanovsky
     [not found]                         ` <20170731053901.GR13672-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-07-31  7:12                           ` Leon Romanovsky
2013-08-30  3:13 Barclay Jameson
2012-07-07 12:54 Kernel Oops RuanZhijie
2012-07-04 11:04 RuanZhijie
2008-07-23 12:52 kernel oops Andrei Popa
2008-07-23 17:11 ` Vegard Nossum
2008-08-18 16:33   ` Vegard Nossum
2008-08-18 16:39     ` Greg KH
     [not found] <e8eb01770803120245x7690e6a9te8ad04296aa3fc4d@mail.gmail.com>
2008-03-12  9:49 ` Zbynek Drlik
2008-03-12 10:33   ` Al Viro
2008-03-12 13:12     ` Zbynek Drlik
2008-02-05 12:57 Andrej Hocevar
2008-02-06 17:55 ` Len Brown
2006-10-27  8:59 Kernel Oops Joël Cuissinat
2006-10-27 17:30 ` Russell Cattelan
2006-09-12 10:21 Marcin Prączko
2006-09-13  3:43 ` Andrew Morton
2005-10-15  1:03 Marc Perkel
2005-10-15  1:21 ` Randy.Dunlap
2005-10-15  1:43   ` Marc Perkel
2005-10-15  1:52     ` Randy.Dunlap
2005-04-25 16:31 Chris Penney
2005-04-25 17:47 ` Dan Stromberg
2005-01-08 12:47 kernel oops ierdnah
2005-01-03 21:10 Kernel oops Marat BN
2005-01-05 10:13 ` Andrew Morton
2004-10-17 12:02 kernel oops Thomas Bleher
2004-10-17 12:59 ` Erich Schubert
2004-10-18 21:09   ` Thomas Bleher
2004-06-11  7:27 Kernel oops tmp
2004-05-24 20:19 tmp
2004-05-16 12:08 Kernel OOPS tmp
2004-05-16 23:27 ` Andrew Morton
2004-05-17  0:33   ` tmp
2004-03-09 22:13 Kernel oops Philipp Baer
2004-03-09 23:11 ` Andrew Morton
2004-03-12  7:46   ` Philipp Baer
2004-02-08 11:05 Kernel Oops Mathieu LESNIAK
2004-02-08 16:35 ` Greg KH
2004-02-09  7:06   ` Mathieu LESNIAK
2003-11-28 23:15 Kernel oops Ville Jutvik
2003-11-28  5:45 Anderson Levi
2003-08-09 12:39 kernel oops Jean-Yves LENHOF
2003-08-09 20:37 ` Jean-Yves LENHOF
2003-08-09  9:28 Jean-Yves LENHOF
2003-07-18 19:44 Kernel OOPS Robert Scussel
2003-07-18 21:31 ` Alan Cox
2003-07-07 12:53 kernel oops Anders Karlsson
2003-07-07 13:14 ` Alan Cox
2003-07-07 13:32   ` Anders Karlsson
2003-07-07 13:37     ` Alan Cox
2003-07-07 13:56       ` Anders Karlsson
2003-07-08  9:39         ` Marcelo Tosatti
     [not found] ` <200307072009.50677.bernd-schubert@web.de>
2003-07-08  5:13   ` Anders Karlsson
2003-05-31  1:32 Nadeem Riaz
2003-03-26 15:52 Steve Terrell
2003-02-03  1:18 Kernel Oops Daniel Espinoza
2003-02-03  3:23 ` vishwas
2002-08-09  5:25 sanket rathi
2002-06-23 19:39 Dirk Schmidt
2002-06-10  8:46 kernel oops Robert Litwiniec
2002-02-26 18:26 Suporte RedeBonja
2002-02-27 13:35 ` Erik Mouw
2001-11-26 19:59 Tracy R Reed
2001-11-26 19:27 ` Stephen Smalley
2001-11-13 13:23 Kernel oops Anthony
2001-11-14  6:02 ` Thiago Rondon
2001-10-08 12:59 kernel oops Terry Kendal
2001-09-27  9:49 kewl
2001-06-01 15:13 Kernel oops David Harris
2001-06-01 15:12 David Harris
2001-04-19 18:32 kernel oops Ronald Bultje
2001-04-19 19:04 ` Alan Cox
2001-04-19 19:08   ` Ronald Bultje
2001-02-19 14:44 Kernel Oops Alberto Bertogli
2000-12-10  0:53 Bastien Nocera
2000-12-10  1:13 ` Bastien Nocera
2000-05-17 17:10 Kernel oops Patrick Higgins

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAN-5tyF5fGNzjjZ-3sy3=da+3CTeKu-Lq22hcJiDfy=qqghxTA@mail.gmail.com' \
    --to=aglo@umich.edu \
    --cc=chuck.lever@oracle.com \
    --cc=kernel-tls-handshake@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.