Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

From: Sagi Grimberg <sagi@grimberg.me>
To: Hannes Reinecke <hare@suse.de>, Jakub Kicinski <kuba@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>,
	Boris Pismenny <borisp@nvidia.com>,
	john.fastabend@gmail.com, Paolo Abeni <pabeni@redhat.com>,
	Keith Busch <kbusch@kernel.org>,
	linux-nvme@lists.infradead.org,
	Chuck Lever <chuck.lever@oracle.com>,
	kernel-tls-handshake@lists.linux.dev,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Date: Mon, 3 Apr 2023 15:20:13 +0300	[thread overview]
Message-ID: <44fe87ba-e873-fa05-d294-d29d5e6dd4b5@grimberg.me> (raw)
In-Reply-To: <7f057726-8777-2fd3-a207-b3cd96076cb9@suse.de>

Hey Jakub, Hannes.

>>>> kTLS does not support MSG_EOR flag for sendmsg(), and in general
>>>> is really picky about invalid MSG_XXX flags.
>>>
>>> CC'ing TLS folks.
>>>
>>> Can't tls simply ignore MSG_EOR instead of consumers having to be
>>> careful over it?
>>
>> I think we can support EOR, I don't see any fundamental problem there.

It would help at least one consumer (nvme-tcp) to not change its
behavior between tls and non-tls. At the very minimum don't fail
the send operation (just do the same as if it wasn't passed).

>>>> So ensure that the MSG_EOR flags is blanked out for TLS, and that
>>>> the MSG_SENDPAGE_LAST is only set if we actually do sendpage().
>>>
>>> You mean MSG_SENDPAGE_NOTLAST.
>>>
>>> It is also a bit annoying that a tls socket dictates different behavior
>>> than a normal socket.
>>>
>>> The current logic is rather simple:
>>> if more data comming:
>>>     flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
>>> else:
>>>     flags = MSG_EOR
>>>
>>> Would like to keep it that way for tls as well. Can someone
>>> explain why this is a problem with tls?
>>
>> Some of the flags are call specific, others may be internal to the
>> networking stack (e.g. the DECRYPTED flag). Old protocols didn't do
>> any validation because people coded more haphazardly in the 90s.
>> This lack of validation is a major source of technical debt :(
> 
> A-ha. So what is the plan?
> Should the stack validate flags?
> And should the rules for validating be the same for all protocols?

MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was
essentially similar semantics to MSG_MORE but for sendpage. It'd
be great if this can be allowed in tls (again, at the very least
don't fail but continue as if it wasn't passed).

If this turns out to be a big project, I would prefer to change
nvme-tcp for now in order not to block nvme tls support (although it is
a hidden capability interface, which is always bad).