All of lore.kernel.org
 help / color / mirror / Atom feed
From: Chuck Lever III <chuck.lever@oracle.com>
To: Olga Kornievskaia <aglo@umich.edu>
Cc: "kernel-tls-handshake@lists.linux.dev"
	<kernel-tls-handshake@lists.linux.dev>
Subject: Re: kernel oops
Date: Tue, 4 Apr 2023 19:30:10 +0000	[thread overview]
Message-ID: <28728CA5-6883-41B7-8C0C-121281201663@oracle.com> (raw)
In-Reply-To: <CAN-5tyF5fGNzjjZ-3sy3=da+3CTeKu-Lq22hcJiDfy=qqghxTA@mail.gmail.com>



> On Apr 4, 2023, at 3:26 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
> 
> On Tue, Apr 4, 2023 at 3:14 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>> 
>> 
>> 
>>> On Apr 4, 2023, at 3:11 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>> 
>>> On Tue, Apr 4, 2023 at 1:28 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On Apr 4, 2023, at 12:15 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>> 
>>>>> On Tue, Apr 4, 2023 at 10:54 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Apr 4, 2023, at 10:44 AM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>>>> 
>>>>>>> On Tue, Apr 4, 2023 at 10:32 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>>>>>>>> 
>>>>>>>>> On Apr 4, 2023, at 10:24 AM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>>>>>> 
>>>>>>>>> On Fri, Mar 31, 2023 at 10:25 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Mar 31, 2023, at 10:23 AM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Chuck,
>>>>>>>>>>> 
>>>>>>>>>>> Have you seen this oops? This is not my testing so all I know was that
>>>>>>>>>>> it was an attempt to mount with xprtsec=tls (against ONTAP).
>>>>>>>>>>> 
>>>>>>>>>>> [  316.939747] BUG: kernel NULL pointer dereference, address: 0000000000000018
>>>>>>>>>>> [  316.939848] #PF: supervisor read access in kernel mode
>>>>>>>>>>> [  316.939914] #PF: error_code(0x0000) - not-present page
>>>>>>>>>>> [  316.939984] PGD 0 P4D 0
>>>>>>>>>>> [  316.940041] Oops: 0000 [#1] PREEMPT SMP PTI
>>>>>>>>>>> [  316.940109] CPU: 0 PID: 511 Comm: kworker/u2:30 Kdump: loaded Not
>>>>>>>>>>> tainted 6.2.1+ #2
>>>>>>>>>>> [  316.940181] Hardware name: VMware, Inc. VMware7,1/440BX Desktop
>>>>>>>>>>> Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
>>>>>>>>>>> [  316.940259] Workqueue: xprtiod xs_tls_connect [sunrpc]
>>>>>>>>>>> [  316.940463] RIP: 0010:xs_tls_connect+0x3e1/0x590 [sunrpc]
>>>>>>>>>>> [  316.940556] Code: ff ff ff 48 2b 83 f8 fe ff ff 48 89 83 00 ff ff
>>>>>>>>>>> ff e8 03 d3 ff ff e9 a8 fd ff ff 49 8b 97 f8 05 00 00 66 83 bb e0 f9
>>>>>>>>>>> ff ff 0a <48> 8b 6a 18 0f 84 fd 00 00 00 48 8d 72 18 4c 89 e7 48 89 14
>>>>>>>>>>> 24 e8
>>>>>>>>>>> [  316.940719] RSP: 0018:ffffa69d40f5fdc8 EFLAGS: 00010293
>>>>>>>>>>> [  316.940786] RAX: 0000000000000000 RBX: ffff9b0c46333e40 RCX: 0000000000000000
>>>>>>>>>>> [  316.940872] RDX: 0000000000000000 RSI: ffffa69d40f5fd20 RDI: 0000000000000000
>>>>>>>>>>> [  316.940940] RBP: ffff9b0c46350200 R08: ffff9b0cf7e31968 R09: 0000000000000159
>>>>>>>>>>> [  316.941008] R10: 0000000000000001 R11: 00000049c8887a40 R12: ffff9b0c46333800
>>>>>>>>>>> [  316.941075] R13: 0000000004208160 R14: ffff9b0c44e18a00 R15: ffff9b0c42dab800
>>>>>>>>>>> [  316.941159] FS:  0000000000000000(0000) GS:ffff9b0cf7e00000(0000)
>>>>>>>>>>> knlGS:0000000000000000
>>>>>>>>>>> [  316.941243] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>>>> [  316.941309] CR2: 0000000000000018 CR3: 0000000102e48004 CR4: 00000000007706f0
>>>>>>>>>>> [  316.941391] PKRU: 55555554
>>>>>>>>>>> [  316.941443] Call Trace:
>>>>>>>>>>> [  316.941513]  <TASK>
>>>>>>>>>>> [  316.941578]  process_one_work+0x1b0/0x3c0
>>>>>>>>>>> [  316.941689]  worker_thread+0x30/0x360
>>>>>>>>>>> [  316.941755]  ? __pfx_worker_thread+0x10/0x10
>>>>>>>>>>> [  316.941822]  kthread+0xd7/0x100
>>>>>>>>>>> [  316.941900]  ? __pfx_kthread+0x10/0x10
>>>>>>>>>>> [  316.941968]  ret_from_fork+0x29/0x50
>>>>>>>>>>> [  316.942049]  </TASK>
>>>>>>>>>>> [  316.942099] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4
>>>>>>>>>>> dns_resolver ib_core nfsv3 nfs_acl nfs lockd grace fscache netfs ext4
>>>>>>>>>>> rfkill mbcache jbd2 loop vsock_loopback
>>>>>>>>>>> vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock
>>>>>>>>>>> sunrpc tls vfat fat dm_multipath intel_rapl_msr intel_rapl_common
>>>>>>>>>>> isst_if_mbox_msr isst_if_common nfit libnvdimm crct10dif_pclmul
>>>>>>>>>>> crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr
>>>>>>>>>>> i2c_piix4 vmw_vmci xfs libcrc32c vmwgfx drm_ttm_helper ttm sr_mod
>>>>>>>>>>> cdrom drm_kms_helper syscopyarea sd_mod sysfillrect ata_generic
>>>>>>>>>>> sysimgblt sg ahci libahci ata_piix crc32c_intel serio_raw drm vmxnet3
>>>>>>>>>>> vmw_pvscsi libata dm_mirror dm_region_hash dm_log dm_mod nvme_tcp
>>>>>>>>>>> nvme_fabrics nvme_core t10_pi crc64_rocksoft crc64 ipmi_devintf
>>>>>>>>>>> ipmi_msghandler fuse
>>>>>>>>>>> [  316.942550] CR2: 0000000000000018
>>>>>>>>>>> 
>>>>>>>>>>> Please note this was still on "commit
>>>>>>>>>>> caa52379f0046e974fd16585dd7ede624699d8e2 (HEAD -> 03152023-v7,
>>>>>>>>>>> origin/upcall-v7)". If you think somethings done to address it I'll
>>>>>>>>>>> have them update to the latest.
>>>>>>>>>> 
>>>>>>>>>> Update and retry. It has a familiar ring, but I can't say I've
>>>>>>>>>> seen this recently.
>>>>>>>>> 
>>>>>>>>> Hi Chuck,
>>>>>>>>> 
>>>>>>>>> I've updated the kernel code to the latest  commit
>>>>>>>>> 46588cade103b446f74a7ee160a83cd2a80399e7 (HEAD ->
>>>>>>>>> chuck-rpc-with-tls-04032023, chuck/topic-rpc-with-tls-upcall) but the
>>>>>>>>> client is still crashing but looks like in a different place. This
>>>>>>>>> happens when the server terminates the connection (immediately) as the
>>>>>>>>> tls handshake happens.
>>>>>>>> 
>>>>>>>> Why is the server doing that?
>>>>>>> 
>>>>>>> I'm not sure what kind of a reason you are looking for. I think the
>>>>>>> server can send an RST at any point in a TCP connection and the client
>>>>>>> shouldn't crash.
>>>>>> 
>>>>>> No doubt.
>>>>>> 
>>>>>> 
>>>>>>> But the real reason is probably because the server's
>>>>>>> code is getting developed and they ran into some error and resetting
>>>>>>> the connection.
>>>>>> 
>>>>>> That's what I was looking for. I wanted to know if it was
>>>>>> Linux NFSD doing this.
>>>>>> 
>>>>>> 
>>>>>>>>> Here's the stack trace:
>>>>>>>>> [  677.274299] BUG: kernel NULL pointer dereference, address: 0000000000000030
>>>>>>>>> [  677.274424] #PF: supervisor read access in kernel mode
>>>>>>>>> [  677.274517] #PF: error_code(0x0000) - not-present page
>>>>>>>>> [  677.274609] PGD 0 P4D 0
>>>>>>>>> [  677.274689] Oops: 0000 [#1] PREEMPT SMP PTI
>>>>>>>>> [  677.274785] CPU: 0 PID: 518 Comm: kworker/u2:29 Kdump: loaded Not
>>>>>>>>> tainted 6.3.0-rc4+ #13
>>>>>>>>> [  677.274881] Hardware name: VMware, Inc. VMware7,1/440BX Desktop
>>>>>>>>> Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
>>>>>>>>> [  677.274981] Workqueue: xprtiod xs_tls_connect [sunrpc]
>>>>>>>>> [  677.275137] RIP: 0010:handshake_req_cancel+0x2b/0x3b0
>>>>>>>> 
>>>>>>>> scripts/faddr2line on this RIP
>>>>>>>> 
>>>>>>>> Which pointer is NULL?
>>>>>>> 
>>>>>>> I apologize I don't have that info and I'm working on trying to figure
>>>>>>> out how to get this information to you. Lab kernels are built with
>>>>>>> CONFIG_DEBUG_INFO disabled and thus I can't do faddr2line. I will
>>>>>>> attempt to build on with debugging enabled (has been problematic
>>>>>>> before for the lab machines so not sure if that'll be successful I
>>>>>>> might have to hack the linux kernel to simulate the reset).
>>>>>> 
>>>>>> Jakub asked that NULL checking be removed from handshake_req_cancel,
>>>>>> or at least that's what I read into his comment. That choice is
>>>>>> definitely flushing bugs.
>>>>>> 
>>>>>> So this crasher is familiar. req_cancel is getting called in some
>>>>>> scenario where the socket or request is partially filled in. We
>>>>>> have to figure out how that's happening.
>>>>>> 
>>>>>> If debug info is a problem, then just add a sarcastic quantity of
>>>>>> NULL checks in handshake_req_cancel, rebuild, and it should tell
>>>>>> you exactly which pointer is bad.
>>>>> 
>>>>> I added a check for sock-sk null and if so return true.
>>>> 
>>>> If it returns true, xs_tls_handshake_sync() probably completes
>>>> successfully.
>>>> 
>>>> Perhaps handshake_req_cancel needs to return false in cases
>>>> where it can't find a pending request.
>>> 
>>> I'm not sure if you were suggesting me to do this but I tried:
>>> if (!sock->sk) return false;
>>> and that led to a mount not crashing but timing out.
>> 
>> That's the behavior I would expect. Was the failure immediate,
>> or did it take a few seconds for the mount command to exit?
> 
> It did take quite a long time to exit:
> 
> [root@scs000128074 kolga]# time mount -o vers=4.2,sec=sys,xprtsec=tls
> 10.63.20.131:/ /mnt
> mount.nfs: Connection timed out
> 
> real    3m0.544s
> user    0m0.004s
> sys     0m0.013s
> 
> Network trace shows numerous (16) attempts to do NULL call

The long delay is due to retries, probably driven by mount.nfs.

But looks like there's a 10 second delay between each attempt.
Maybe that's OK.


>   66 2023-04-04 15:18:17.786579 10.235.232.7 → 10.63.20.131 TCP 74
> 953 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995198270 TSecr=0 WS=128
>   67 2023-04-04 15:18:17.786818 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 953 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=2815692929 TSecr=3995198270
>   68 2023-04-04 15:18:17.786871 10.235.232.7 → 10.63.20.131 TCP 66
> 953 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995198270
> TSecr=2815692929
>   69 2023-04-04 15:18:17.786951 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>   70 2023-04-04 15:18:17.787488 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 69)
>   71 2023-04-04 15:18:17.787520 10.235.232.7 → 10.63.20.131 TCP 66
> 953 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995198271
> TSecr=2815692929
>   72 2023-04-04 15:18:17.818060 10.235.232.7 → 10.56.8.183  TCP 66
> 826 → 2049 [ACK] Seq=293 Ack=237 Win=501 Len=0 TSval=1599771772
> TSecr=3825493593
>   73 2023-04-04 15:18:17.993341 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 953 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2815693129
> TSecr=3995198271
>   74 2023-04-04 15:18:19.490669 fe80::21c:73ff:fe00:99 → ff02::1
> ICMPv6 118 Router Advertisement from 00:1c:73:00:00:99
>   75 2023-04-04 15:18:19.494076 fe80::250:56ff:febf:2f55 → ff02::16
>  ICMPv6 90 Multicast Listener Report Message v2
>   76 2023-04-04 15:18:19.841191 fe80::250:56ff:febf:2f55 → ff02::16
>  ICMPv6 90 Multicast Listener Report Message v2
>   77 2023-04-04 15:18:28.201642 10.235.232.7 → 10.63.20.131 TCP 74
> 692 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995208685 TSecr=0 WS=128
>   78 2023-04-04 15:18:28.202171 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 692 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=3904561029 TSecr=3995208685
>   79 2023-04-04 15:18:28.202236 10.235.232.7 → 10.63.20.131 TCP 66
> 692 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995208686
> TSecr=3904561029
>   80 2023-04-04 15:18:28.202506 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>   81 2023-04-04 15:18:28.202882 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 80)
>   82 2023-04-04 15:18:28.202911 10.235.232.7 → 10.63.20.131 TCP 66
> 692 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995208686
> TSecr=3904561029
>   83 2023-04-04 15:18:28.329632 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 692 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=3904561159
> TSecr=3995208686
>   84 2023-04-04 15:18:34.857077 10.235.232.7 → 10.60.248.40 TCP 66
> 845 → 2049 [ACK] Seq=1 Ack=1 Win=501 Len=0 TSval=3160247691
> TSecr=2389454709
>   85 2023-04-04 15:18:34.857146 10.235.232.7 → 10.60.249.150 TCP 66
> 970 → 2049 [ACK] Seq=1 Ack=1 Win=502 Len=0 TSval=4116465771
> TSecr=1272760298
>   86 2023-04-04 15:18:34.857421 10.60.249.150 → 10.235.232.7 TCP 66
> [TCP ACKed unseen segment] 2049 → 970 [ACK] Seq=1 Ack=2 Win=257 Len=0
> TSval=1272820701 TSecr=4116405409
>   87 2023-04-04 15:18:34.857422 10.60.248.40 → 10.235.232.7 TCP 66
> [TCP ACKed unseen segment] 2049 → 845 [ACK] Seq=1 Ack=2 Win=257 Len=0
> TSval=2389515123 TSecr=3160187318
>   88 2023-04-04 15:18:38.441851 10.235.232.7 → 10.63.20.131 TCP 74
> 971 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995218925 TSecr=0 WS=128
>   89 2023-04-04 15:18:38.442269 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 971 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=1698639803 TSecr=3995218925
>   90 2023-04-04 15:18:38.442320 10.235.232.7 → 10.63.20.131 TCP 66
> 971 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995218926
> TSecr=1698639803
>   91 2023-04-04 15:18:38.442577 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>   92 2023-04-04 15:18:38.442894 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 91)
>   93 2023-04-04 15:18:38.442921 10.235.232.7 → 10.63.20.131 TCP 66
> 971 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995218926
> TSecr=1698639803
>   94 2023-04-04 15:18:38.478175 10.63.20.131 → 10.235.232.7 TCP 66
> [TCP Dup ACK 92#1] 2049 → 971 [ACK] Seq=37 Ack=45 Win=65792 Len=0
> TSval=1698639843 TSecr=3995218926
>   95 2023-04-04 15:18:38.614265 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 971 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1698639973
> TSecr=3995218926
>   96 2023-04-04 15:18:48.681763 10.235.232.7 → 10.63.20.131 TCP 74
> 799 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995229165 TSecr=0 WS=128
>   97 2023-04-04 15:18:48.682176 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 799 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=1773399631 TSecr=3995229165
>   98 2023-04-04 15:18:48.682203 10.235.232.7 → 10.63.20.131 TCP 66
> 799 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995229166
> TSecr=1773399631
>   99 2023-04-04 15:18:48.682260 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  100 2023-04-04 15:18:48.682751 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 99)
>  101 2023-04-04 15:18:48.682761 10.235.232.7 → 10.63.20.131 TCP 66
> 799 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995229166
> TSecr=1773399631
>  102 2023-04-04 15:18:48.826843 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 799 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1773399771
> TSecr=3995229166
>  103 2023-04-04 15:18:49.514227 10.235.232.170 → 10.235.235.255
> BROWSER 243 Host Announcement WINDOWS-G95UIQM, Workstation, Server, NT
> Workstation, NT Server
>  104 2023-04-04 15:18:58.921556 10.235.232.7 → 10.63.20.131 TCP 74
> 729 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995239405 TSecr=0 WS=128
>  105 2023-04-04 15:18:58.921926 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 729 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=2487545914 TSecr=3995239405
>  106 2023-04-04 15:18:58.921953 10.235.232.7 → 10.63.20.131 TCP 66
> 729 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995239405
> TSecr=2487545914
>  107 2023-04-04 15:18:58.922027 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  108 2023-04-04 15:18:58.922802 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 107)
>  109 2023-04-04 15:18:58.922822 10.235.232.7 → 10.63.20.131 TCP 66
> 729 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995239406
> TSecr=2487545914
>  110 2023-04-04 15:18:59.050600 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 729 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2487546044
> TSecr=3995239406
>  111 2023-04-04 15:19:06.533098 10.235.232.128 → 10.235.235.255
> BROWSER 243 Host Announcement SCS000126759-1, Workstation, Server, NT
> Workstation, NT Server
>  112 2023-04-04 15:19:09.161611 10.235.232.7 → 10.63.20.131 TCP 74
> 954 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995249645 TSecr=0 WS=128
>  113 2023-04-04 15:19:09.161949 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 954 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=3804453828 TSecr=3995249645
>  114 2023-04-04 15:19:09.162009 10.235.232.7 → 10.63.20.131 TCP 66
> 954 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995249645
> TSecr=3804453828
>  115 2023-04-04 15:19:09.162055 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  116 2023-04-04 15:19:09.162528 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 115)
>  117 2023-04-04 15:19:09.162554 10.235.232.7 → 10.63.20.131 TCP 66
> 954 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995249646
> TSecr=3804453828
>  118 2023-04-04 15:19:09.278841 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 954 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=3804453948
> TSecr=3995249646
>  119 2023-04-04 15:19:12.003989 Vmware_bf:32:23 → Broadcast    ARP 60
> Who has 10.235.232.1? Tell 10.235.232.42
>  120 2023-04-04 15:19:19.401546 10.235.232.7 → 10.63.20.131 TCP 74
> 728 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995259885 TSecr=0 WS=128
>  121 2023-04-04 15:19:19.401902 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 728 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=2468567773 TSecr=3995259885
>  122 2023-04-04 15:19:19.401965 10.235.232.7 → 10.63.20.131 TCP 66
> 728 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995259885
> TSecr=2468567773
>  123 2023-04-04 15:19:19.402050 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  124 2023-04-04 15:19:19.402542 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 123)
>  125 2023-04-04 15:19:19.402554 10.235.232.7 → 10.63.20.131 TCP 66
> 728 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995259886
> TSecr=2468567773
>  126 2023-04-04 15:19:19.595961 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 728 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2468567963
> TSecr=3995259886
>  127 2023-04-04 15:19:21.449077 10.235.232.7 → 10.60.249.125 TCP 66
> [TCP Keep-Alive] 732 → 2049 [ACK] Seq=2176 Ack=1857 Win=502 Len=0
> TSval=1628408100 TSecr=708634268
>  128 2023-04-04 15:19:21.449206 10.235.232.7 → 10.56.8.183  TCP 66
> [TCP Keep-Alive] 826 → 2049 [ACK] Seq=292 Ack=237 Win=501 Len=0
> TSval=1599835403 TSecr=3825493593
>  129 2023-04-04 15:19:21.449284 10.60.249.125 → 10.235.232.7 TCP 66
> [TCP Keep-Alive ACK] 2049 → 732 [ACK] Seq=1857 Ack=2177 Win=257 Len=0
> TSval=708698066 TSecr=1628344342
>  130 2023-04-04 15:19:21.511308  10.56.8.183 → 10.235.232.7 TCP 66
> [TCP Keep-Alive ACK] 2049 → 826 [ACK] Seq=237 Ack=293 Win=257 Len=0
> TSval=3825557328 TSecr=1599771772
>  131 2023-04-04 15:19:29.641659 10.235.232.7 → 10.63.20.131 TCP 74
> 924 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995270125 TSecr=0 WS=128
>  132 2023-04-04 15:19:29.641987 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 924 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=4239335290 TSecr=3995270125
>  133 2023-04-04 15:19:29.642062 10.235.232.7 → 10.63.20.131 TCP 66
> 924 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995270125
> TSecr=4239335290
>  134 2023-04-04 15:19:29.642188 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  135 2023-04-04 15:19:29.642585 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 134)
>  136 2023-04-04 15:19:29.642626 10.235.232.7 → 10.63.20.131 TCP 66
> 924 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995270126
> TSecr=4239335290
>  137 2023-04-04 15:19:29.752894 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 924 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=4239335400
> TSecr=3995270126
>  138 2023-04-04 15:19:37.833079 10.235.232.7 → 10.60.248.40 TCP 66
> [TCP Dup ACK 84#1] 845 → 2049 [ACK] Seq=1 Ack=1 Win=501 Len=0
> TSval=3160310667 TSecr=2389515123
>  139 2023-04-04 15:19:37.833122 10.235.232.7 → 10.60.249.150 TCP 66
> [TCP Dup ACK 85#1] 970 → 2049 [ACK] Seq=1 Ack=1 Win=502 Len=0
> TSval=4116528747 TSecr=1272820701
>  140 2023-04-04 15:19:37.833308 10.60.248.40 → 10.235.232.7 TCP 66
> [TCP Dup ACK 87#1] [TCP ACKed unseen segment] 2049 → 845 [ACK] Seq=1
> Ack=2 Win=257 Len=0 TSval=2389578099 TSecr=3160187318
>  141 2023-04-04 15:19:37.833346 10.60.249.150 → 10.235.232.7 TCP 66
> [TCP Dup ACK 86#1] [TCP ACKed unseen segment] 2049 → 970 [ACK] Seq=1
> Ack=2 Win=257 Len=0 TSval=1272883677 TSecr=4116405409
>  142 2023-04-04 15:19:39.881507 10.235.232.7 → 10.63.20.131 TCP 74
> 788 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995280365 TSecr=0 WS=128
>  143 2023-04-04 15:19:39.881784 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 788 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=2800122551 TSecr=3995280365
>  144 2023-04-04 15:19:39.881847 10.235.232.7 → 10.63.20.131 TCP 66
> 788 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995280365
> TSecr=2800122551
>  145 2023-04-04 15:19:39.881898 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  146 2023-04-04 15:19:39.882359 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 145)
>  147 2023-04-04 15:19:39.882395 10.235.232.7 → 10.63.20.131 TCP 66
> 788 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995280366
> TSecr=2800122551
>  148 2023-04-04 15:19:40.017113 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 788 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2800122681
> TSecr=3995280366
>  149 2023-04-04 15:19:50.122254 10.235.232.7 → 10.63.20.131 TCP 74
> 868 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995290606 TSecr=0 WS=128
>  150 2023-04-04 15:19:50.122643 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 868 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=1996033015 TSecr=3995290606
>  151 2023-04-04 15:19:50.122674 10.235.232.7 → 10.63.20.131 TCP 66
> 868 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995290606
> TSecr=1996033015
>  152 2023-04-04 15:19:50.122802 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  153 2023-04-04 15:19:50.123263 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 152)
>  154 2023-04-04 15:19:50.123279 10.235.232.7 → 10.63.20.131 TCP 66
> 868 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995290607
> TSecr=1996033015
>  155 2023-04-04 15:19:50.268177 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 868 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1996033165
> TSecr=3995290607
>  156 2023-04-04 15:20:00.361440 10.235.232.7 → 10.63.20.131 TCP 74
> 988 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995300845 TSecr=0 WS=128
>  157 2023-04-04 15:20:00.361776 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 988 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=4120839794 TSecr=3995300845
>  158 2023-04-04 15:20:00.361791 10.235.232.7 → 10.63.20.131 TCP 66
> 988 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995300845
> TSecr=4120839794
>  159 2023-04-04 15:20:00.361841 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  160 2023-04-04 15:20:00.362321 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 159)
>  161 2023-04-04 15:20:00.362327 10.235.232.7 → 10.63.20.131 TCP 66
> 988 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995300846
> TSecr=4120839794
>  162 2023-04-04 15:20:00.479087 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 988 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=4120839904
> TSecr=3995300846
>  163 2023-04-04 15:20:01.559017 10.235.232.173 → 10.235.235.255
> BROWSER 243 Host Announcement WINDOWS-PTEFTLL, Workstation, Server, NT
> Workstation, NT Server
>  164 2023-04-04 15:20:10.426553 10.235.232.7 → 10.60.248.183 NTP 90
> NTP Version 4, client
>  165 2023-04-04 15:20:10.426824 10.60.248.183 → 10.235.232.7 NTP 90
> NTP Version 4, server
>  166 2023-04-04 15:20:10.601625 10.235.232.7 → 10.63.20.131 TCP 74
> 888 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995311085 TSecr=0 WS=128
>  167 2023-04-04 15:20:10.601919 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 888 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=531138781 TSecr=3995311085
>  168 2023-04-04 15:20:10.601978 10.235.232.7 → 10.63.20.131 TCP 66
> 888 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995311085
> TSecr=531138781
>  169 2023-04-04 15:20:10.602058 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  170 2023-04-04 15:20:10.602516 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 169)
>  171 2023-04-04 15:20:10.602543 10.235.232.7 → 10.63.20.131 TCP 66
> 888 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995311086
> TSecr=531138781
>  172 2023-04-04 15:20:10.731005 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 888 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=531138911
> TSecr=3995311086
>  173 2023-04-04 15:20:11.998469 10.235.232.7 → 10.235.48.111 NTP 90
> NTP Version 4, client
>  174 2023-04-04 15:20:11.998803 10.235.48.111 → 10.235.232.7 NTP 90
> NTP Version 4, server
>  175 2023-04-04 15:20:12.648789 10.235.232.7 → 10.227.204.142 NTP 90
> NTP Version 4, client
>  176 2023-04-04 15:20:12.649163 10.227.204.142 → 10.235.232.7 NTP 90
> NTP Version 4, server
>  177 2023-04-04 15:20:20.841429 10.235.232.7 → 10.63.20.131 TCP 74
> 893 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995321325 TSecr=0 WS=128
>  178 2023-04-04 15:20:20.841673 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 893 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=1953607493 TSecr=3995321325
>  179 2023-04-04 15:20:20.841688 10.235.232.7 → 10.63.20.131 TCP 66
> 893 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995321325
> TSecr=1953607493
>  180 2023-04-04 15:20:20.841767 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  181 2023-04-04 15:20:20.842198 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 180)
>  182 2023-04-04 15:20:20.842208 10.235.232.7 → 10.63.20.131 TCP 66
> 893 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995321326
> TSecr=1953607493
>  183 2023-04-04 15:20:21.051706 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 893 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1953607703
> TSecr=3995321326
>  184 2023-04-04 15:20:22.889123 10.235.232.7 → 10.56.8.183  TCP 66
> [TCP Keep-Alive] 826 → 2049 [ACK] Seq=292 Ack=237 Win=501 Len=0
> TSval=1599896843 TSecr=3825557328
>  185 2023-04-04 15:20:22.889130 10.235.232.7 → 10.60.249.125 TCP 66
> [TCP Keep-Alive] 732 → 2049 [ACK] Seq=2176 Ack=1857 Win=502 Len=0
> TSval=1628469540 TSecr=708698066
>  186 2023-04-04 15:20:22.889335 10.60.249.125 → 10.235.232.7 TCP 66
> [TCP Keep-Alive ACK] 2049 → 732 [ACK] Seq=1857 Ack=2177 Win=257 Len=0
> TSval=708759507 TSecr=1628344342
>  187 2023-04-04 15:20:22.951234  10.56.8.183 → 10.235.232.7 TCP 66
> [TCP Keep-Alive ACK] 2049 → 826 [ACK] Seq=237 Ack=293 Win=257 Len=0
> TSval=3825618767 TSecr=1599771772
>  188 2023-04-04 15:20:27.226390 10.235.232.131 → 10.235.235.255
> BROWSER 243 Host Announcement SCS000126759-3, Workstation, Server, NT
> Workstation, NT Server
>  189 2023-04-04 15:20:31.081384 10.235.232.7 → 10.63.20.131 TCP 74
> 1013 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995331565 TSecr=0 WS=128
>  190 2023-04-04 15:20:31.081679 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 1013 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=1446866364 TSecr=3995331565
>  191 2023-04-04 15:20:31.081698 10.235.232.7 → 10.63.20.131 TCP 66
> 1013 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995331565
> TSecr=1446866364
>  192 2023-04-04 15:20:31.081745 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  193 2023-04-04 15:20:31.082347 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 192)
>  194 2023-04-04 15:20:31.082353 10.235.232.7 → 10.63.20.131 TCP 66
> 1013 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995331566
> TSecr=1446866364
>  195 2023-04-04 15:20:31.216474 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 1013 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1446866494
> TSecr=3995331566
>  196 2023-04-04 15:20:39.273073 10.235.232.7 → 10.60.249.150 TCP 66
> [TCP Dup ACK 85#2] 970 → 2049 [ACK] Seq=1 Ack=1 Win=502 Len=0
> TSval=4116590187 TSecr=1272883677
>  197 2023-04-04 15:20:39.273146 10.235.232.7 → 10.60.248.40 TCP 66
> [TCP Dup ACK 84#2] 845 → 2049 [ACK] Seq=1 Ack=1 Win=501 Len=0
> TSval=3160372107 TSecr=2389578099
>  198 2023-04-04 15:20:39.273342 10.60.249.150 → 10.235.232.7 TCP 66
> [TCP Dup ACK 86#2] [TCP ACKed unseen segment] 2049 → 970 [ACK] Seq=1
> Ack=2 Win=257 Len=0 TSval=1272945117 TSecr=4116405409
>  199 2023-04-04 15:20:39.274592 10.60.248.40 → 10.235.232.7 TCP 66
> [TCP Dup ACK 87#2] [TCP ACKed unseen segment] 2049 → 845 [ACK] Seq=1
> Ack=2 Win=257 Len=0 TSval=2389639540 TSecr=3160187318
>  200 2023-04-04 15:20:41.321912 10.235.232.7 → 10.63.20.131 TCP 74
> 676 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995341805 TSecr=0 WS=128
>  201 2023-04-04 15:20:41.322226 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 676 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=710203011 TSecr=3995341805
>  202 2023-04-04 15:20:41.322258 10.235.232.7 → 10.63.20.131 TCP 66
> 676 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995341806
> TSecr=710203011
>  203 2023-04-04 15:20:41.322382 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  204 2023-04-04 15:20:41.322902 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 203)
>  205 2023-04-04 15:20:41.322923 10.235.232.7 → 10.63.20.131 TCP 66
> 676 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995341806
> TSecr=710203011
>  206 2023-04-04 15:20:41.502713 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 676 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=710203191
> TSecr=3995341806
>  207 2023-04-04 15:20:41.671253 fe80::21c:73ff:fe00:99 → ff02::1
> ICMPv6 118 Router Advertisement from 00:1c:73:00:00:99
>  208 2023-04-04 15:20:41.675089 fe80::250:56ff:febf:2f55 → ff02::16
>  ICMPv6 90 Multicast Listener Report Message v2
>  209 2023-04-04 15:20:41.684071 fe80::250:56ff:febf:2f55 → ff02::16
>  ICMPv6 90 Multicast Listener Report Message v2
>  210 2023-04-04 15:20:43.709117 10.235.232.130 → 10.235.235.255
> BROWSER 243 Host Announcement SCS000126759-2, Workstation, Server, NT
> Workstation, NT Server
>  211 2023-04-04 15:20:51.561341 10.235.232.7 → 10.63.20.131 TCP 74
> 864 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995352045 TSecr=0 WS=128
>  212 2023-04-04 15:20:51.561677 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 864 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=1105202286 TSecr=3995352045
>  213 2023-04-04 15:20:51.561736 10.235.232.7 → 10.63.20.131 TCP 66
> 864 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995352045
> TSecr=1105202286
>  214 2023-04-04 15:20:51.561764 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  215 2023-04-04 15:20:51.562296 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 214)
>  216 2023-04-04 15:20:51.562324 10.235.232.7 → 10.63.20.131 TCP 66
> 864 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995352046
> TSecr=1105202286
>  217 2023-04-04 15:20:51.764952 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 864 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1105202486
> TSecr=3995352046
>  218 2023-04-04 15:21:01.801569 10.235.232.7 → 10.63.20.131 TCP 74
> 779 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995362285 TSecr=0 WS=128
>  219 2023-04-04 15:21:01.801903 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 779 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=3669197173 TSecr=3995362285
>  220 2023-04-04 15:21:01.801938 10.235.232.7 → 10.63.20.131 TCP 66
> 779 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995362285
> TSecr=3669197173
>  221 2023-04-04 15:21:01.801991 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  222 2023-04-04 15:21:01.802750 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 221)
>  223 2023-04-04 15:21:01.802761 10.235.232.7 → 10.63.20.131 TCP 66
> 779 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995362286
> TSecr=3669197173
>  224 2023-04-04 15:21:01.954179 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 779 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=3669197323
> TSecr=3995362286
>  225 2023-04-04 15:21:10.142734 fe80::21c:73ff:fe00:99 → ff02::1
> ICMPv6 118 Router Advertisement from 00:1c:73:00:00:99
>  226 2023-04-04 15:21:10.146082 fe80::250:56ff:febf:2f55 → ff02::16
>  ICMPv6 90 Multicast Listener Report Message v2
>  227 2023-04-04 15:21:10.889124 fe80::250:56ff:febf:2f55 → ff02::16
>  ICMPv6 90 Multicast Listener Report Message v2
>  228 2023-04-04 15:21:12.041741 10.235.232.7 → 10.63.20.131 TCP 74
> 946 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
> TSval=3995372525 TSecr=0 WS=128
>  229 2023-04-04 15:21:12.042058 10.63.20.131 → 10.235.232.7 TCP 74
> 2049 → 946 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
> SACK_PERM=1 TSval=1985286575 TSecr=3995372525
>  230 2023-04-04 15:21:12.042121 10.235.232.7 → 10.63.20.131 TCP 66
> 946 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3995372526
> TSecr=1985286575
>  231 2023-04-04 15:21:12.042404 10.235.232.7 → 10.63.20.131 NFS 110
> V4 NULL Call
>  232 2023-04-04 15:21:12.042673 10.63.20.131 → 10.235.232.7 NFS 102
> V4 NULL Reply (Call In 231)
>  233 2023-04-04 15:21:12.042710 10.235.232.7 → 10.63.20.131 TCP 66
> 946 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=3995372526
> TSecr=1985286575
>  234 2023-04-04 15:21:12.079429 10.63.20.131 → 10.235.232.7 TCP 66
> [TCP Dup ACK 232#1] 2049 → 946 [ACK] Seq=37 Ack=45 Win=65792 Len=0
> TSval=1985286615 TSecr=3995372526
>  235 2023-04-04 15:21:12.184895 10.63.20.131 → 10.235.232.7 TCP 66
> 2049 → 946 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=1985286705
> TSecr=3995372526
>  236 2023-04-04 15:21:18.186337 10.235.232.7 → 10.60.250.78 SSH 150
> Server: Encrypted packet (len=84)
>  237 2023-04-04 15:21:18.186573 10.60.250.78 → 10.235.232.7 TCP 66
> 45496 → 22 [ACK] Seq=417 Ack=629 Win=501 Len=0 TSval=2780826399
> TSecr=3765675462
>  238 2023-04-04 15:21:18.187900 10.235.232.7 → 10.60.250.78 SSH 150
> Server: Encrypted packet (len=84)
>  239 2023-04-04 15:21:18.188065 10.60.250.78 → 10.235.232.7 TCP 66
> 45496 → 22 [ACK] Seq=417 Ack=713 Win=501 Len=0 TSval=2780826401
> TSecr=3765675463
>  240 2023-04-04 15:21:18.188154 10.235.232.7 → 10.60.250.78 SSH 150
> Server: Encrypted packet (len=84)
>  241 2023-04-04 15:21:18.188283 10.60.250.78 → 10.235.232.7 TCP 66
> 45496 → 22 [ACK] Seq=417 Ack=797 Win=501 Len=0 TSval=2780826401
> TSecr=3765675464
>  242 2023-04-04 15:21:18.188455 10.235.232.7 → 10.60.250.78 SSH 134
> Server: Encrypted packet (len=68)
>  243 2023-04-04 15:21:18.188599 10.60.250.78 → 10.235.232.7 TCP 66
> 45496 → 22 [ACK] Seq=417 Ack=865 Win=501 Len=0 TSval=2780826401
> TSecr=3765675464
>  244 2023-04-04 15:21:22.539976 fe80::21c:73ff:fe00:99 →
> fe80::250:56ff:febf:5abd ICMPv6 118 Router Advertisement from
> 00:1c:73:00:00:99
>  245 2023-04-04 15:21:22.764914 fe80::21c:73ff:fe00:99 →
> fe80::250:56ff:febf:5abd ICMPv6 118 Router Advertisement from
> 00:1c:73:00:00:99
>  246 2023-04-04 15:21:24.329146 10.235.232.7 → 10.56.8.183  TCP 66
> [TCP Keep-Alive] 826 → 2049 [ACK] Seq=292 Ack=237 Win=501 Len=0
> TSval=1599958283 TSecr=3825618767
>  247 2023-04-04 15:21:24.329156 10.235.232.7 → 10.60.249.125 TCP 66
> [TCP Keep-Alive] 732 → 2049 [ACK] Seq=2176 Ack=1857 Win=502 Len=0
> TSval=1628530980 TSecr=708759507
>  248 2023-04-04 15:21:24.329506 10.60.249.125 → 10.235.232.7 TCP 66
> [TCP Keep-Alive ACK] 2049 → 732 [ACK] Seq=1857 Ack=2177 Win=257 Len=0
> TSval=708820947 TSecr=1628344342
>  249 2023-04-04 15:21:24.391262  10.56.8.183 → 10.235.232.7 TCP 66
> [TCP Keep-Alive ACK] 2049 → 826 [ACK] Seq=237 Ack=293 Win=257 Len=0
> TSval=3825680208 TSecr=1599771772
> 
>> 
>> 
>>>>> I thought that
>>>>> would "prevent" the crash but it then proceeded to crash elsewhere.
>>>>> Sorry I'm still on non-debug-symbol kernel (as I thought I'd add the
>>>>> checks that that'd be enough :)).
>>>>> 
>>>>> [  284.368211] AGLO: handshake_req_cancel sock->sk is NULL
>>>>> [  284.368249] BUG: kernel NULL pointer dereference, address: 0000000000000018
>>>>> [  284.368320] #PF: supervisor read access in kernel mode
>>>>> [  284.368363] #PF: error_code(0x0000) - not-present page
>>>>> [  284.368411] PGD 0 P4D 0
>>>>> [  284.368438] Oops: 0000 [#1] PREEMPT SMP PTI
>>>>> [  284.368473] CPU: 0 PID: 521 Comm: kworker/u2:30 Kdump: loaded Not
>>>>> tainted 6.3.0-rc4+ #14
>>>>> [  284.368523] Hardware name: VMware, Inc. VMware7,1/440BX Desktop
>>>>> Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
>>>>> [  284.368580] Workqueue: xprtiod xs_tls_connect [sunrpc]
>>>>> [  284.368720] RIP: 0010:xs_tls_connect+0x3e5/0x590 [sunrpc]
>>>>> [  284.368844] Code: ff ff ff 48 2b 83 f8 fe ff ff 48 89 83 00 ff ff
>>>>> ff e8 bf d2 ff ff e9 a8 fd ff ff 49 8b 97 f8 05 00 00 66 83 bb e0 f9
>>>>> ff ff 0a <48> 8b 6a 18 0f 84 fd 00 00 00 48 8d 72 18 4c 89 e7 48 89 14
>>>>> 24 e8
>>>>> [  284.368973] RSP: 0018:ffffba17c0f67dc8 EFLAGS: 00010293
>>>>> [  284.369010] RAX: 0000000000000000 RBX: ffff963143c1fe40 RCX: 0000000000000000
>>>>> [  284.369051] RDX: 0000000000000000 RSI: ffff9631f7e20840 RDI: ffff9631f7e20840
>>>>> [  284.369091] RBP: ffff9631463d8a00 R08: 0000000000000000 R09: c0000000ffff7fff
>>>>> [  284.369130] R10: 0000000000000001 R11: ffffba17c0f67bb8 R12: ffff963143c1f800
>>>>> [  284.369175] R13: 0000000004208160 R14: ffff963143f6f600 R15: ffff963143c1e000
>>>>> [  284.369259] FS:  0000000000000000(0000) GS:ffff9631f7e00000(0000)
>>>>> knlGS:0000000000000000
>>>>> [  284.369319] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> [  284.369367] CR2: 0000000000000018 CR3: 0000000102d12004 CR4: 00000000007706f0
>>>>> [  284.369458] PKRU: 55555554
>>>>> [  284.369483] Call Trace:
>>>>> [  284.369523]  <TASK>
>>>>> [  284.369583]  process_one_work+0x1b0/0x3c0
>>>>> [  284.369633]  worker_thread+0x34/0x360
>>>>> [  284.369668]  ? __pfx_worker_thread+0x10/0x10
>>>>> [  284.369704]  kthread+0xdb/0x110
>>>>> [  284.369736]  ? __pfx_kthread+0x10/0x10
>>>>> [  284.369767]  ret_from_fork+0x29/0x50
>>>>> [  284.369808]  </TASK>
>>>>> [  284.369843] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4
>>>>> dns_resolver nfsv3 nfs_acl nfs lockd grace fscache netfs ext4 mbcache
>>>>> jbd2 loop rfkill vsock_loopback vmw_vsock_virtio_transport_common
>>>>> vmw_vsock_vmci_transport vsock sunrpc vfat fat dm_multipath
>>>>> intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit
>>>>> libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl
>>>>> vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 xfs libcrc32c vmwgfx
>>>>> drm_ttm_helper ttm sr_mod cdrom drm_kms_helper sd_mod syscopyarea
>>>>> sysfillrect ata_generic sysimgblt sg ata_piix ahci libahci
>>>>> crc32c_intel serio_raw vmxnet3 libata drm vmw_pvscsi dm_mirror
>>>>> dm_region_hash dm_log dm_mod nvme_tcp nvme_fabrics nvme_core t10_pi
>>>>> crc64_rocksoft crc64 ipmi_devintf ipmi_msghandler fuse
>>>>> [  284.370300] CR2: 0000000000000018
>>>>> 
>>>>> 
>>>>>> 
>>>>>> BPF might also be a solution, but I'm not sure if it relies on
>>>>>> debug info. Might be only BTF is needed.
>>>>>> 
>>>>>> 
>>>>>>>>> [  677.275269] Code: 0f 1e fa 0f 1f 44 00 00 41 57 41 56 41 55 41 54
>>>>>>>>> 55 53 48 83 ec 28 48 8b 6f 18 65 48 8b 04 25 28 00 00 00 48 89 44 24
>>>>>>>>> 20 31 c0 <48> 8b 5d 30 48 89 6c 24 18 e8 97 15 5f ff 4c 8b 25 50 2e 70
>>>>>>>>> 01 41
>>>>>>>>> [  677.275474] RSP: 0018:ffffaed400f6fcf8 EFLAGS: 00010246
>>>>>>>>> [  677.275570] RAX: 0000000000000000 RBX: ffff93f085b33800 RCX: 0000000000000000
>>>>>>>>> [  677.275663] RDX: ffff93f085b33f08 RSI: ffffaed400f6fd18 RDI: ffff93f083aa6700
>>>>>>>>> [  677.275756] RBP: 0000000000000000 R08: ffff93f137e32968 R09: 0000009d9663fa40
>>>>>>>>> [  677.275850] R10: 0000000000000000 R11: 0000009d9663fa40 R12: ffff93f083aa6700
>>>>>>>>> [  677.275943] R13: 0000000000000000 R14: ffff93f082999e00 R15: ffff93f085b33800
>>>>>>>>> [  677.276049] FS:  0000000000000000(0000) GS:ffff93f137e00000(0000)
>>>>>>>>> knlGS:0000000000000000
>>>>>>>>> [  677.276154] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>>> [  677.276247] CR2: 0000000000000030 CR3: 000000010797c002 CR4: 00000000007706f0
>>>>>>>>> [  677.276355] PKRU: 55555554
>>>>>>>>> [  677.276427] Call Trace:
>>>>>>>>> [  677.276521]  <TASK>
>>>>>>>>> [  677.276605]  ? wait_for_completion_interruptible_timeout+0xfa/0x170
>>>>>>>>> [  677.276725]  xs_tls_handshake_sync+0x14f/0x170 [sunrpc]
>>>>>>>>> [  677.276850]  ? __pfx_xs_tls_handshake_done+0x10/0x10 [sunrpc]
>>>>>>>>> [  677.276967]  xs_tls_connect+0x14c/0x590 [sunrpc]
>>>>>>>>> [  677.277086]  process_one_work+0x1b0/0x3c0
>>>>>>>>> [  677.277209]  worker_thread+0x34/0x360
>>>>>>>>> [  677.277303]  ? __pfx_worker_thread+0x10/0x10
>>>>>>>>> [  677.277396]  kthread+0xdb/0x110
>>>>>>>>> [  677.277527]  ? __pfx_kthread+0x10/0x10
>>>>>>>>> [  677.277619]  ret_from_fork+0x29/0x50
>>>>>>>>> [  677.277731]  </TASK>
>>>>>>>>> [  677.277803] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4
>>>>>>>>> dns_resolver nfsv3 nfs_acl nfs lockd grace fscache netfs ext4 mbcache
>>>>>>>>> jbd2 loop rfkill vsock_loopback vmw_vsock_virtio_transport_common
>>>>>>>>> vmw_vsock_vmci_transport vsock sunrpc vfat fat dm_multipath
>>>>>>>>> intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit
>>>>>>>>> libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl
>>>>>>>>> vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 xfs libcrc32c vmwgfx
>>>>>>>>> drm_ttm_helper ttm sr_mod cdrom drm_kms_helper syscopyarea sysfillrect
>>>>>>>>> sd_mod ata_generic sysimgblt sg ata_piix ahci libahci crc32c_intel
>>>>>>>>> serio_raw drm vmxnet3 vmw_pvscsi libata dm_mirror dm_region_hash
>>>>>>>>> dm_log dm_mod nvme_tcp nvme_fabrics nvme_core t10_pi crc64_rocksoft
>>>>>>>>> crc64 ipmi_devintf ipmi_msghandler fuse
>>>>>>>>> [  677.278383] CR2: 0000000000000030
>>>>>>>>> 
>>>>>>>>> Network trace from the server:
>>>>>>>>> 
>>>>>>>>> [kolga@scs000085536 ~/pcaps]$ tshark -t ad -r e0d_20230404_084311.trc0
>>>>>>>>> 1 2023-04-04 08:43:28.027546 10.235.232.77 → 10.63.20.131 TCP 74
>>>>>>>>> 727 → 2049 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
>>>>>>>>> TSval=4059503415 TSecr=0 WS=128
>>>>>>>>> 2 2023-04-04 08:43:28.027645 10.63.20.131 → 10.235.232.77 TCP 74
>>>>>>>>> 2049 → 727 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256
>>>>>>>>> SACK_PERM=1 TSval=2256845073 TSecr=4059503415
>>>>>>>>> 3 2023-04-04 08:43:28.027813 10.235.232.77 → 10.63.20.131 TCP 66
>>>>>>>>> 727 → 2049 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=4059503415
>>>>>>>>> TSecr=2256845073
>>>>>>>>> 4 2023-04-04 08:43:28.027952 10.235.232.77 → 10.63.20.131 NFS 110
>>>>>>>>> V4 NULL Call
>>>>>>>>> 5 2023-04-04 08:43:28.028301 10.63.20.131 → 10.235.232.77 NFS 102
>>>>>>>>> V4 NULL Reply (Call In 4)
>>>>>>>>> 6 2023-04-04 08:43:28.028563 10.235.232.77 → 10.63.20.131 TCP 66
>>>>>>>>> 727 → 2049 [ACK] Seq=45 Ack=37 Win=64256 Len=0 TSval=4059503416
>>>>>>>>> TSecr=2256845073
>>>>>>>>> 7 2023-04-04 08:43:28.331181 10.63.20.131 → 10.235.232.77 TCP 66
>>>>>>>>> 2049 → 727 [RST, ACK] Seq=37 Ack=45 Win=0 Len=0 TSval=2256845383
>>>>>>>>> TSecr=4059503416
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Chuck Lever
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Chuck Lever
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Chuck Lever
>>>> 
>>>> 
>>>> --
>>>> Chuck Lever
>> 
>> 
>> --
>> Chuck Lever


--
Chuck Lever



  reply	other threads:[~2023-04-04 19:30 UTC|newest]

Thread overview: 97+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-03-31 14:23 kernel oops Olga Kornievskaia
2023-03-31 14:25 ` Chuck Lever III
2023-04-04 14:24   ` Olga Kornievskaia
2023-04-04 14:31     ` Chuck Lever III
2023-04-04 14:44       ` Olga Kornievskaia
2023-04-04 14:54         ` Chuck Lever III
2023-04-04 16:15           ` Olga Kornievskaia
2023-04-04 17:28             ` Chuck Lever III
2023-04-04 19:11               ` Olga Kornievskaia
2023-04-04 19:14                 ` Chuck Lever III
2023-04-04 19:26                   ` Olga Kornievskaia
2023-04-04 19:30                     ` Chuck Lever III [this message]
  -- strict thread matches above, loose matches on Subject: below --
2017-07-24 21:16 Kernel oops Jason Gunthorpe
     [not found] ` <20170724211606.GA1705-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-27 11:46   ` Matan Barak
     [not found]     ` <CAAKD3BAdB2aRk3WGdbeDYof6dUfkEwhQf27cG0FWe5DRuQ15NQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-07-27 12:54       ` Matan Barak
     [not found]         ` <CAAKD3BDFrTMMgX0nErD50rp2je=HC9zeaYWHDKf0mqQwc5fM9g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2017-07-27 20:44           ` Jason Gunthorpe
     [not found]             ` <20170727204437.GA16986-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-30 10:25               ` Leon Romanovsky
     [not found]                 ` <20170730102514.GQ13672-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-07-31  3:52                   ` Jason Gunthorpe
     [not found]                     ` <20170731035208.GA30615-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-31  5:39                       ` Leon Romanovsky
     [not found]                         ` <20170731053901.GR13672-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-07-31  7:12                           ` Leon Romanovsky
2013-08-30  3:13 Barclay Jameson
2012-07-07 12:54 Kernel Oops RuanZhijie
2012-07-04 11:04 RuanZhijie
2008-07-23 12:52 kernel oops Andrei Popa
2008-07-23 17:11 ` Vegard Nossum
2008-08-18 16:33   ` Vegard Nossum
2008-08-18 16:39     ` Greg KH
     [not found] <e8eb01770803120245x7690e6a9te8ad04296aa3fc4d@mail.gmail.com>
2008-03-12  9:49 ` Zbynek Drlik
2008-03-12 10:33   ` Al Viro
2008-03-12 13:12     ` Zbynek Drlik
2008-02-05 12:57 Andrej Hocevar
2008-02-06 17:55 ` Len Brown
2006-10-27  8:59 Kernel Oops Joël Cuissinat
2006-10-27 17:30 ` Russell Cattelan
2006-09-12 10:21 Marcin Prączko
2006-09-13  3:43 ` Andrew Morton
2005-10-15  1:03 Marc Perkel
2005-10-15  1:21 ` Randy.Dunlap
2005-10-15  1:43   ` Marc Perkel
2005-10-15  1:52     ` Randy.Dunlap
2005-04-25 16:31 Chris Penney
2005-04-25 17:47 ` Dan Stromberg
2005-01-08 12:47 kernel oops ierdnah
2005-01-03 21:10 Kernel oops Marat BN
2005-01-05 10:13 ` Andrew Morton
2004-10-17 12:02 kernel oops Thomas Bleher
2004-10-17 12:59 ` Erich Schubert
2004-10-18 21:09   ` Thomas Bleher
2004-06-11  7:27 Kernel oops tmp
2004-05-24 20:19 tmp
2004-05-16 12:08 Kernel OOPS tmp
2004-05-16 23:27 ` Andrew Morton
2004-05-17  0:33   ` tmp
2004-03-09 22:13 Kernel oops Philipp Baer
2004-03-09 23:11 ` Andrew Morton
2004-03-12  7:46   ` Philipp Baer
2004-02-08 11:05 Kernel Oops Mathieu LESNIAK
2004-02-08 16:35 ` Greg KH
2004-02-09  7:06   ` Mathieu LESNIAK
2003-11-28 23:15 Kernel oops Ville Jutvik
2003-11-28  5:45 Anderson Levi
2003-08-09 12:39 kernel oops Jean-Yves LENHOF
2003-08-09 20:37 ` Jean-Yves LENHOF
2003-08-09  9:28 Jean-Yves LENHOF
2003-07-18 19:44 Kernel OOPS Robert Scussel
2003-07-18 21:31 ` Alan Cox
2003-07-07 12:53 kernel oops Anders Karlsson
2003-07-07 13:14 ` Alan Cox
2003-07-07 13:32   ` Anders Karlsson
2003-07-07 13:37     ` Alan Cox
2003-07-07 13:56       ` Anders Karlsson
2003-07-08  9:39         ` Marcelo Tosatti
     [not found] ` <200307072009.50677.bernd-schubert@web.de>
2003-07-08  5:13   ` Anders Karlsson
2003-05-31  1:32 Nadeem Riaz
2003-03-26 15:52 Steve Terrell
2003-02-03  1:18 Kernel Oops Daniel Espinoza
2003-02-03  3:23 ` vishwas
2002-08-09  5:25 sanket rathi
2002-06-23 19:39 Dirk Schmidt
2002-06-10  8:46 kernel oops Robert Litwiniec
2002-02-26 18:26 Suporte RedeBonja
2002-02-27 13:35 ` Erik Mouw
2001-11-26 19:59 Tracy R Reed
2001-11-26 19:27 ` Stephen Smalley
2001-11-13 13:23 Kernel oops Anthony
2001-11-14  6:02 ` Thiago Rondon
2001-10-08 12:59 kernel oops Terry Kendal
2001-09-27  9:49 kewl
2001-06-01 15:13 Kernel oops David Harris
2001-06-01 15:12 David Harris
2001-04-19 18:32 kernel oops Ronald Bultje
2001-04-19 19:04 ` Alan Cox
2001-04-19 19:08   ` Ronald Bultje
2001-02-19 14:44 Kernel Oops Alberto Bertogli
2000-12-10  0:53 Bastien Nocera
2000-12-10  1:13 ` Bastien Nocera
2000-05-17 17:10 Kernel oops Patrick Higgins

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=28728CA5-6883-41B7-8C0C-121281201663@oracle.com \
    --to=chuck.lever@oracle.com \
    --cc=aglo@umich.edu \
    --cc=kernel-tls-handshake@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.