From: Hannes Reinecke <hare@suse.de>
To: Sagi Grimberg <sagi@grimberg.me>,
Chuck Lever III <chuck.lever@oracle.com>
Cc: Christoph Hellwig <hch@lst.de>, Keith Busch <kbusch@kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
Chuck Lever <cel@kernel.org>,
"kernel-tls-handshake@lists.linux.dev"
<kernel-tls-handshake@lists.linux.dev>
Subject: Re: [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall
Date: Fri, 24 Mar 2023 12:29:23 +0100 [thread overview]
Message-ID: <3ad50302-6b3c-6d6a-669e-0b11c2df6a6b@suse.de> (raw)
In-Reply-To: <ea181844-69f5-66eb-e64e-e9871916532b@grimberg.me>
On 3/23/23 08:21, Sagi Grimberg wrote:
>
>>>>>>> The 'data_ready' call might happen at any time after the 'accept'
>>>>>>> call and us calling into userspace.
>>>>>>> In particular we have this flow of control:
>>>>>>>
>>>>>>> 1. Kernel: accept()
>>>>>>> 2. Kernel: handshake request
>>>>>>> 3. Userspace: read data from socket
>>>>>>> 4. Userspace: tls handshake
>>>>>>> 5. Kernel: handshake complete
>>>>>>>
>>>>>>> If the 'data_ready' event occurs between 1. and 3. userspace
>>>>>>> wouldn't know that something has happened, and will be sitting
>>>>>>> there waiting for data which is already present.
>>>>>>
>>>>>> Umm, doesn't userspace read from the socket once we trigger the
>>>>>> upcall?
>>>>>> it should. But I still don't understand what is the difference
>>>>>> between
>>>>>> us waiking up userspace, from the default sock doing the same?
>>>>>>
>>>>> No, it doesn't (or, rather, can't).
>>>>> After processing 'accept()' (from the kernel code) data might
>>>>> already be present (after all, why would we get an 'accept' call
>>>>> otherwise?).
>>>>> But the daemon has not been started up (yet); that's only done in
>>>>> step 3). But 'data_ready' has already been called, so by the time
>>>>> userland is able to do a 'read()' on the socket it won't be seeing
>>>>> anything.
>>>> Not sure I understand. if data exists, userspace will read from the
>>>> socket and get data, whenever that is. >
>>> That's what I thought, too.
>>> But then the userspace daemon just sat there doing nothing.
>>
>> I haven't been following this discussion in detail, but
>> if the kernel disables the normal TCP data_ready callback,
>> then user space won't get any data. That's why SunRPC's
>> data_ready calls the previous sk_data_ready and then shunts
>> its own data_ready callback during handshakes. Without that
>> call to the old sk_data_ready, the user space endpoint won't
>> see any received data.
>
> Yes that is understood. But the solution that Hannes proposed
> was to introduce nvmet_tcp_tls_data_ready which is overriding
> the default sock data_ready and does pretty much the same thing.
>
> The reason is that today nvmet_tcp_listen_data_ready schedules accept
> and then pretty much immediately replaces the socket data_ready to
> nvmet_tcp_data_ready.
>
> I think that a simpler solution was to make nvmet_tcp_listen_data_ready
> call port->data_ready (default socket stored data_ready), schedule
> the accept_work and only after the handshake bounce to userspace is
> completed, override the socket callbacks.
>
> Something like:
> --
> static void nvmet_tcp_listen_data_ready(struct sock *sk)
> {
> struct nvmet_tcp_port *port;
>
> trace_sk_data_ready(sk);
>
> read_lock_bh(&sk->sk_callback_lock);
> port = sk->sk_user_data;
> if (!port)
> goto out;
>
> port->data_ready(sk); // trigger socket old data_ready
>
> if (sk->sk_state == TCP_LISTEN)
> queue_work(nvmet_wq, &port->accept_work);
> out:
> read_unlock_bh(&sk->sk_callback_lock);
> }
>
Nearly there.
The actual patch would be:
@@ -2031,10 +1988,16 @@ static void nvmet_tcp_listen_data_ready(struct
sock *sk)
trace_sk_data_ready(sk);
read_lock_bh(&sk->sk_callback_lock);
+ /* Ignore if the callback has been changed */
+ if (sk->sk_data_ready != nvmet_tcp_listen_data_ready)
+ goto out;
port = sk->sk_user_data;
if (!port)
goto out;
+ if (port->data_ready)
+ port->data_ready(sk);
+
if (sk->sk_state == TCP_LISTEN)
queue_work(nvmet_wq, &port->accept_work);
out:
As the callbacks will be changed once TLS is activated, and we really
should not attempt to run if sk_data_ready() points to another function,
as then the sk_user_data pointer will most likely be changed, too,
causing all sorts of issues.
Cheers,
Hannes
next prev parent reply other threads:[~2023-03-24 11:29 UTC|newest]
Thread overview: 90+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-21 12:43 [RFC PATCH 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
2023-03-21 12:43 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring Hannes Reinecke
2023-03-21 13:50 ` Sagi Grimberg
2023-03-21 14:11 ` Hannes Reinecke
2023-03-21 12:43 ` [PATCH 02/18] nvme-keyring: define a 'psk' keytype Hannes Reinecke
2023-03-22 8:29 ` Sagi Grimberg
2023-03-22 8:38 ` Hannes Reinecke
2023-03-22 8:49 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 03/18] nvme: add TCP TSAS definitions Hannes Reinecke
2023-03-21 13:46 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 04/18] nvme-tcp: add definitions for TLS cipher suites Hannes Reinecke
2023-03-22 8:18 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 05/18] nvme-tcp: implement recvmsg rx flow for TLS Hannes Reinecke
2023-03-21 13:39 ` Sagi Grimberg
2023-03-21 13:59 ` Hannes Reinecke
2023-03-22 8:01 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 06/18] nvme-tcp: call 'queue->data_ready()' in nvme_tcp_data_ready() Hannes Reinecke
2023-03-21 13:44 ` Sagi Grimberg
2023-03-21 14:09 ` Hannes Reinecke
2023-03-22 0:18 ` Chris Leech
2023-03-22 6:59 ` Hannes Reinecke
2023-03-22 8:12 ` Sagi Grimberg
2023-03-22 8:08 ` Sagi Grimberg
2023-03-22 8:26 ` Hannes Reinecke
2023-03-22 10:13 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 07/18] nvme/tcp: allocate socket file Hannes Reinecke
2023-03-21 13:52 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 08/18] nvme-tcp: enable TLS handshake upcall Hannes Reinecke
2023-03-22 8:45 ` Sagi Grimberg
2023-03-22 9:12 ` Hannes Reinecke
2023-03-22 10:56 ` Sagi Grimberg
2023-03-22 12:54 ` Hannes Reinecke
2023-03-22 13:16 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 09/18] nvme-tcp: add connect option 'tls' Hannes Reinecke
2023-03-22 9:24 ` Sagi Grimberg
2023-03-22 9:59 ` Hannes Reinecke
2023-03-22 10:09 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS Hannes Reinecke
2023-03-22 9:31 ` Sagi Grimberg
2023-03-22 10:08 ` Hannes Reinecke
2023-03-22 11:18 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 11/18] nvme-tcp: control message handling for recvmsg() Hannes Reinecke
2023-03-22 11:33 ` Sagi Grimberg
2023-03-22 11:48 ` Hannes Reinecke
2023-03-22 11:50 ` Sagi Grimberg
2023-03-22 12:17 ` Hannes Reinecke
2023-03-22 12:29 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 12/18] nvmet: make TCP sectype settable via configfs Hannes Reinecke
2023-03-22 11:38 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 13/18] nvmet-tcp: allocate socket file Hannes Reinecke
2023-03-22 11:46 ` Sagi Grimberg
2023-03-22 12:07 ` Hannes Reinecke
2023-03-21 12:43 ` [PATCH 14/18] security/keys: export key_lookup() Hannes Reinecke
2023-03-21 12:43 ` [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall Hannes Reinecke
2023-03-22 12:13 ` Sagi Grimberg
2023-03-22 12:34 ` Hannes Reinecke
2023-03-22 12:51 ` Sagi Grimberg
2023-03-22 13:47 ` Hannes Reinecke
2023-03-22 15:42 ` Sagi Grimberg
2023-03-22 16:43 ` Hannes Reinecke
2023-03-22 16:49 ` Chuck Lever III
2023-03-23 7:21 ` Sagi Grimberg
2023-03-24 11:29 ` Hannes Reinecke [this message]
2023-03-26 7:18 ` Sagi Grimberg
2023-03-27 6:20 ` Hannes Reinecke
2023-03-28 8:44 ` Sagi Grimberg
2023-03-28 9:20 ` Hannes Reinecke
2023-03-28 9:43 ` Sagi Grimberg
2023-03-28 10:04 ` Hannes Reinecke
2023-03-28 13:22 ` Chuck Lever III
2023-03-28 15:29 ` Sagi Grimberg
2023-03-28 15:56 ` Chuck Lever III
2023-03-29 6:33 ` Sagi Grimberg
2023-03-23 7:44 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS Hannes Reinecke
2023-03-22 12:16 ` Sagi Grimberg
2023-03-21 12:43 ` [PATCH 17/18] nvmet-tcp: control messages for recvmsg() Hannes Reinecke
2023-03-21 12:43 ` [PATCH 18/18] nvmet-tcp: peek icreq before starting TLS Hannes Reinecke
2023-03-22 12:24 ` Sagi Grimberg
2023-03-22 12:38 ` Hannes Reinecke
2023-03-21 13:12 ` [RFC PATCH 00/18] nvme: In-kernel TLS support for TCP Sagi Grimberg
2023-03-21 13:30 ` Hannes Reinecke
2023-03-22 8:16 ` Sagi Grimberg
2023-03-22 8:28 ` Hannes Reinecke
2023-03-22 12:53 ` Sagi Grimberg
2023-03-22 15:10 ` Hannes Reinecke
2023-03-22 15:43 ` Sagi Grimberg
2023-03-29 13:59 [PATCHv2 " Hannes Reinecke
2023-03-29 13:59 ` [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall Hannes Reinecke
2023-04-03 12:51 ` Sagi Grimberg
2023-04-03 14:05 ` Hannes Reinecke
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3ad50302-6b3c-6d6a-669e-0b11c2df6a6b@suse.de \
--to=hare@suse.de \
--cc=cel@kernel.org \
--cc=chuck.lever@oracle.com \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=kernel-tls-handshake@lists.linux.dev \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).