[PATCH v2 0/3] Another crack at a handshake upcall mechanism

[PATCH v2 0/3] Another crack at a handshake upcall mechanism

[Chuck Lever]

Hello Jakub-

Second version of a hybrid listen/accept/netlink upcall mechanism.
This one tries to address a few more of your requests from last
year, and it introduces support for basic server-side upcalls.

These patches are the netdev piece only. The full series, which adds
client and server RPC-with-TLS implementations, can be found in the
topic-rpc-with-tls-upcall branch here:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

The third patch below demonstrates how to set up handshake support
for a kernel transport security layer protocol by adding handshake
support to kTLS.

A sample user space handshake daemon is available here:

   https://github.com/oracle/ktls-utils

The "main" branch contains the latest changes that are required to
operate with the kernel patches presented in this email.

---

Changes since RFC:
- Documentation temporarily removed while code churns
- Split the handshake mechanism away from kTLS
- Added a default TLS priorities string
- Added support for ServerHello

Chuck Lever (3):
      net: Add an AF_HANDSHAKE address family
      net/handshake: Add support for PF_HANDSHAKE
      net/tls: Support AF_HANDSHAKE in kTLS


 include/linux/socket.h                        |   4 +-
 include/net/handshake.h                       |  31 +
 include/net/sock.h                            |   2 +
 include/net/tls.h                             |  16 +
 include/trace/events/handshake.h              | 328 +++++++
 include/uapi/linux/handshake.h                |  73 ++
 net/Makefile                                  |   1 +
 net/core/sock.c                               |   2 +-
 net/handshake/Makefile                        |   7 +
 net/handshake/af_handshake.c                  | 838 ++++++++++++++++++
 net/handshake/handshake.h                     |  33 +
 net/handshake/netlink.c                       | 187 ++++
 net/handshake/trace.c                         |  20 +
 net/socket.c                                  |   1 +
 net/tls/Makefile                              |   2 +-
 net/tls/tls_handshake.c                       | 385 ++++++++
 security/selinux/hooks.c                      |   4 +-
 security/selinux/include/classmap.h           |   4 +-
 .../perf/trace/beauty/include/linux/socket.h  |   4 +-
 19 files changed, 1936 insertions(+), 6 deletions(-)
 create mode 100644 include/net/handshake.h
 create mode 100644 include/trace/events/handshake.h
 create mode 100644 include/uapi/linux/handshake.h
 create mode 100644 net/handshake/Makefile
 create mode 100644 net/handshake/af_handshake.c
 create mode 100644 net/handshake/handshake.h
 create mode 100644 net/handshake/netlink.c
 create mode 100644 net/handshake/trace.c
 create mode 100644 net/tls/tls_handshake.c

--
Chuck Lever

[PATCH v2 1/3] net: Add an AF_HANDSHAKE address family

[Chuck Lever]

Add definitions for an AF_HANDSHAKE address family. This address
family is a mechanism that is used to serially share socket
endpoints between the kernel and user space so that a library can
perform a transport layer security handshake on the socket.

The next patch further explains the purpose and operation of sockets
in this family.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 include/linux/socket.h                         |    4 +++-
 net/core/sock.c                                |    2 +-
 net/socket.c                                   |    1 +
 security/selinux/hooks.c                       |    4 +++-
 security/selinux/include/classmap.h            |    4 +++-
 tools/perf/trace/beauty/include/linux/socket.h |    4 +++-
 6 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/include/linux/socket.h b/include/linux/socket.h
index de3701a2a212..80f0ed0c3820 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -235,8 +235,9 @@ struct ucred {
 #define AF_MCTP		45	/* Management component
 				 * transport protocol
 				 */
+#define AF_HANDSHAKE	46	/* pass an FD to user space */
 
-#define AF_MAX		46	/* For now.. */
+#define AF_MAX		47	/* For now.. */
 
 /* Protocol families, same as address families. */
 #define PF_UNSPEC	AF_UNSPEC
@@ -287,6 +288,7 @@ struct ucred {
 #define PF_SMC		AF_SMC
 #define PF_XDP		AF_XDP
 #define PF_MCTP		AF_MCTP
+#define PF_HANDSHAKE	AF_HANDSHAKE
 #define PF_MAX		AF_MAX
 
 /* Maximum queue length specifiable by listen.  */
diff --git a/net/core/sock.c b/net/core/sock.c
index 30407b2dd2ac..f2bc8fd2a586 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -229,7 +229,7 @@ static struct lock_class_key af_family_kern_slock_keys[AF_MAX];
   x "AF_IEEE802154",	x "AF_CAIF"	,	x "AF_ALG"      , \
   x "AF_NFC"   ,	x "AF_VSOCK"    ,	x "AF_KCM"      , \
   x "AF_QIPCRTR",	x "AF_SMC"	,	x "AF_XDP"	, \
-  x "AF_MCTP"  , \
+  x "AF_MCTP"  ,	x "AF_HANDSHAKE", \
   x "AF_MAX"
 
 static const char *const af_family_key_strings[AF_MAX+1] = {
diff --git a/net/socket.c b/net/socket.c
index 00da9ce3dba0..cef752aa6569 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -214,6 +214,7 @@ static const char * const pf_family_names[] = {
 	[PF_SMC]	= "PF_SMC",
 	[PF_XDP]	= "PF_XDP",
 	[PF_MCTP]	= "PF_MCTP",
+	[PF_HANDSHAKE]	= "PF_HANDSHAKE",
 };
 
 /*
diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c
index f553c370397e..7af3b66cbd6d 100644
--- a/security/selinux/hooks.c
+++ b/security/selinux/hooks.c
@@ -1257,7 +1257,9 @@ static inline u16 socket_type_to_security_class(int family, int type, int protoc
 			return SECCLASS_XDP_SOCKET;
 		case PF_MCTP:
 			return SECCLASS_MCTP_SOCKET;
-#if PF_MAX > 46
+		case PF_HANDSHAKE:
+			return SECCLASS_HANDSHAKE_SOCKET;
+#if PF_MAX > 47
 #error New address family defined, please update this function.
 #endif
 		}
diff --git a/security/selinux/include/classmap.h b/security/selinux/include/classmap.h
index a3c380775d41..5f3d4ff8ca5c 100644
--- a/security/selinux/include/classmap.h
+++ b/security/selinux/include/classmap.h
@@ -237,6 +237,8 @@ const struct security_class_mapping secclass_map[] = {
 	  { COMMON_SOCK_PERMS, NULL } },
 	{ "smc_socket",
 	  { COMMON_SOCK_PERMS, NULL } },
+	{ "handshake_socket",
+	  { COMMON_SOCK_PERMS, NULL } },
 	{ "infiniband_pkey",
 	  { "access", NULL } },
 	{ "infiniband_endport",
@@ -259,6 +261,6 @@ const struct security_class_mapping secclass_map[] = {
 	{ NULL }
   };
 
-#if PF_MAX > 46
+#if PF_MAX > 47
 #error New address family defined, please update secclass_map.
 #endif
diff --git a/tools/perf/trace/beauty/include/linux/socket.h b/tools/perf/trace/beauty/include/linux/socket.h
index de3701a2a212..80f0ed0c3820 100644
--- a/tools/perf/trace/beauty/include/linux/socket.h
+++ b/tools/perf/trace/beauty/include/linux/socket.h
@@ -235,8 +235,9 @@ struct ucred {
 #define AF_MCTP		45	/* Management component
 				 * transport protocol
 				 */
+#define AF_HANDSHAKE	46	/* pass an FD to user space */
 
-#define AF_MAX		46	/* For now.. */
+#define AF_MAX		47	/* For now.. */
 
 /* Protocol families, same as address families. */
 #define PF_UNSPEC	AF_UNSPEC
@@ -287,6 +288,7 @@ struct ucred {
 #define PF_SMC		AF_SMC
 #define PF_XDP		AF_XDP
 #define PF_MCTP		AF_MCTP
+#define PF_HANDSHAKE	AF_HANDSHAKE
 #define PF_MAX		AF_MAX
 
 /* Maximum queue length specifiable by listen.  */

[PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Chuck Lever]

In-kernel TLS consumers need a way to perform a TLS handshake. In
the absence of a TLS handshake implementation in the kernel itself,
a mechanism to perform the handshake in user space, using an
existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to
user space using the traditional listen/accept mechanism. accept(2)
gives us a well-worn building block that can materialize a connected
socket endpoint as a file descriptor in a specific user space
process. Like any open socket descriptor, the accepted FD can then
be passed to a library such as GnuTLS to perform a TLS handshake.

The socket sharing mechanism is built into the kernel for now since
it is a small utility to be used by several transport layer security
mechanisms.

This prototype is net-namespace aware.

NB: The kernel has no mechanism to attest that the listening user
space agent is trustworthy.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 include/net/handshake.h          |   31 +
 include/net/sock.h               |    2 
 include/trace/events/handshake.h |  328 +++++++++++++++
 include/uapi/linux/handshake.h   |   49 ++
 net/Makefile                     |    1 
 net/handshake/Makefile           |    7 
 net/handshake/af_handshake.c     |  838 ++++++++++++++++++++++++++++++++++++++
 net/handshake/handshake.h        |   33 +
 net/handshake/netlink.c          |  169 ++++++++
 net/handshake/trace.c            |   20 +
 10 files changed, 1478 insertions(+)
 create mode 100644 include/net/handshake.h
 create mode 100644 include/trace/events/handshake.h
 create mode 100644 include/uapi/linux/handshake.h
 create mode 100644 net/handshake/Makefile
 create mode 100644 net/handshake/af_handshake.c
 create mode 100644 net/handshake/handshake.h
 create mode 100644 net/handshake/netlink.c
 create mode 100644 net/handshake/trace.c

diff --git a/include/net/handshake.h b/include/net/handshake.h
new file mode 100644
index 000000000000..b3fa1d006dcc
--- /dev/null
+++ b/include/net/handshake.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * PF_HANDSHAKE protocol family socket handler.
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2023, Oracle and/or its affiliates.
+ */
+
+/*
+ * Data structures and functions that are visible only within the
+ * kernel are declared here.
+ */
+
+#ifndef _NET_HANDSHAKE_H
+#define _NET_HANDSHAKE_H
+
+struct handshake_info {
+	void			(*hi_done)(struct handshake_info *hsi);
+	int			(*hi_fd_parms_reply)(struct sk_buff *msg,
+						     struct handshake_info *hsi);
+	void			*hi_data;
+	struct socket_wq	*hi_saved_wq;
+	struct socket		*hi_saved_socket;
+	kuid_t			hi_saved_uid;
+};
+
+extern int handshake_enqueue_sock(struct socket *sock,
+				  struct handshake_info *hsi);
+
+#endif /* _NET_HANDSHAKE_H */
diff --git a/include/net/sock.h b/include/net/sock.h
index e0517ecc6531..5ed2d809a149 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -349,6 +349,7 @@ struct sk_filter;
   *	@sk_txtime_unused: unused txtime flags
   *	@ns_tracker: tracker for netns reference
   *	@sk_bind2_node: bind node in the bhash2 table
+  *	@sk_handshake_data: private data for xprt layer security handshake
   */
 struct sock {
 	/*
@@ -515,6 +516,7 @@ struct sock {
 
 	struct socket		*sk_socket;
 	void			*sk_user_data;
+	void			*sk_handshake_data;
 #ifdef CONFIG_SECURITY
 	void			*sk_security;
 #endif
diff --git a/include/trace/events/handshake.h b/include/trace/events/handshake.h
new file mode 100644
index 000000000000..ae3fd3a1ebe9
--- /dev/null
+++ b/include/trace/events/handshake.h
@@ -0,0 +1,328 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022, 2023 Oracle.  All rights reserved.
+ *
+ * Trace point definitions for the "handshake" trace subsystem.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM handshake
+
+#if !defined(_TRACE_HANDSHAKE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_HANDSHAKE_H
+
+#include <asm/unaligned.h>
+#include <linux/types.h>
+#include <net/tcp_states.h>
+
+#include <linux/tracepoint.h>
+
+#define show_address_family(family)				\
+	__print_symbolic(family,				\
+		{ AF_INET,		"AF_INET" },		\
+		{ AF_INET6,		"AF_INET6" },		\
+		{ AF_HANDSHAKE,		"AF_HANDSHAKE" })
+
+TRACE_DEFINE_ENUM(TCP_ESTABLISHED);
+TRACE_DEFINE_ENUM(TCP_SYN_SENT);
+TRACE_DEFINE_ENUM(TCP_SYN_RECV);
+TRACE_DEFINE_ENUM(TCP_FIN_WAIT1);
+TRACE_DEFINE_ENUM(TCP_FIN_WAIT2);
+TRACE_DEFINE_ENUM(TCP_TIME_WAIT);
+TRACE_DEFINE_ENUM(TCP_CLOSE);
+TRACE_DEFINE_ENUM(TCP_CLOSE_WAIT);
+TRACE_DEFINE_ENUM(TCP_LAST_ACK);
+TRACE_DEFINE_ENUM(TCP_LISTEN);
+TRACE_DEFINE_ENUM(TCP_CLOSING);
+TRACE_DEFINE_ENUM(TCP_NEW_SYN_RECV);
+
+#define show_tcp_state(state)					\
+	__print_symbolic(state,					\
+		{ TCP_ESTABLISHED,	"ESTABLISHED" },	\
+		{ TCP_SYN_SENT,		"SYN_SENT" },		\
+		{ TCP_SYN_RECV,		"SYN_RECV" },		\
+		{ TCP_FIN_WAIT1,	"FIN_WAIT1" },		\
+		{ TCP_FIN_WAIT2,	"FIN_WAIT2" },		\
+		{ TCP_TIME_WAIT,	"TIME_WAIT" },		\
+		{ TCP_CLOSE,		"CLOSE" },		\
+		{ TCP_CLOSE_WAIT,	"CLOSE_WAIT" },		\
+		{ TCP_LAST_ACK,		"LAST_ACK" },		\
+		{ TCP_LISTEN,		"LISTEN" },		\
+		{ TCP_CLOSING,		"CLOSING" },		\
+		{ TCP_NEW_SYN_RECV,	"NEW_SYN_RECV" })
+
+#define show_poll_event_mask(mask)				\
+	__print_flags(mask, "|",				\
+		{ EPOLLIN,		"IN" },			\
+		{ EPOLLPRI,		"PRI" },		\
+		{ EPOLLOUT,		"OUT" },		\
+		{ EPOLLERR,		"ERR" },		\
+		{ EPOLLHUP,		"HUP" },		\
+		{ EPOLLNVAL,		"NVAL" },		\
+		{ EPOLLRDNORM,		"RDNORM" },		\
+		{ EPOLLRDBAND,		"RDBAND" },		\
+		{ EPOLLWRNORM,		"WRNORM" },		\
+		{ EPOLLWRBAND,		"WRBAND" },		\
+		{ EPOLLMSG,		"MSG" },		\
+		{ EPOLLRDHUP,		"RDHUP" })
+
+DECLARE_EVENT_CLASS(handshake_listener_class,
+	TP_PROTO(const struct socket *sock),
+	TP_ARGS(sock),
+	TP_STRUCT__entry(
+		__field(const struct socket *, sock)
+		__field(const struct sock *, sk)
+		__field(int, refcount)
+		__field(unsigned long, family)
+	),
+	TP_fast_assign(
+		const struct sock *sk = sock->sk;
+
+		__entry->sock = sock;
+		__entry->sk = sk;
+		__entry->refcount = refcount_read(&sk->sk_refcnt);
+		__entry->family = handshake_sk((struct sock *)sk)->hs_bind_family;
+	),
+	TP_printk("listener=%p sk=%p(%d) family=%s",
+		__entry->sock, __entry->sk,
+		__entry->refcount, show_address_family(__entry->family)
+	)
+);
+
+#define DEFINE_HANDSHAKE_LISTENER_EVENT(name)			\
+	DEFINE_EVENT(handshake_listener_class, name,		\
+		TP_PROTO(const struct socket *sock),		\
+		TP_ARGS(sock))
+
+DEFINE_HANDSHAKE_LISTENER_EVENT(handshake_bind);
+DEFINE_HANDSHAKE_LISTENER_EVENT(handshake_accept);
+DEFINE_HANDSHAKE_LISTENER_EVENT(handshake_listen);
+DEFINE_HANDSHAKE_LISTENER_EVENT(handshake_pf_create);
+
+TRACE_EVENT(handshake_newsock,
+	TP_PROTO(
+		const struct socket *newsock,
+		const struct sock *newsk
+	),
+	TP_ARGS(newsock, newsk),
+	TP_STRUCT__entry(
+		__field(const struct socket *, newsock)
+		__field(const struct sock *, newsk)
+		__field(int, refcount)
+		__field(unsigned long, family)
+	),
+	TP_fast_assign(
+		__entry->newsock = newsock;
+		__entry->newsk = newsk;
+		__entry->refcount = refcount_read(&newsk->sk_refcnt);
+		__entry->family = newsk->sk_family;
+	),
+	TP_printk("newsock=%p newsk=%p(%d) family=%s",
+		__entry->newsock, __entry->newsk,
+		__entry->refcount, show_address_family(__entry->family)
+	)
+);
+
+DECLARE_EVENT_CLASS(handshake_proto_op_class,
+	TP_PROTO(const struct socket *sock),
+	TP_ARGS(sock),
+	TP_STRUCT__entry(
+		__field(const struct socket *, sock)
+		__field(const struct sock *, sk)
+		__field(int, refcount)
+		__field(unsigned long, family)
+		__field(unsigned long, state)
+	),
+	TP_fast_assign(
+		const struct sock *sk = sock->sk;
+
+		__entry->sock = sock;
+		__entry->sk = sk;
+		__entry->refcount = refcount_read(&sk->sk_refcnt);
+		__entry->family = sk->sk_family;
+		__entry->state = sk->sk_state;
+	),
+	TP_printk("sock=%p sk=%p(%d) family=%s state=%s",
+		__entry->sock, __entry->sk, __entry->refcount,
+		show_address_family(__entry->family),
+		show_tcp_state(__entry->state)
+	)
+);
+
+#define DEFINE_HANDSHAKE_PROTO_OP_EVENT(name)			\
+	DEFINE_EVENT(handshake_proto_op_class, name,		\
+		TP_PROTO(const struct socket *sock),		\
+		TP_ARGS(sock))
+
+DEFINE_HANDSHAKE_PROTO_OP_EVENT(handshake_release);
+DEFINE_HANDSHAKE_PROTO_OP_EVENT(handshake_getname);
+DEFINE_HANDSHAKE_PROTO_OP_EVENT(handshake_shutdown);
+DEFINE_HANDSHAKE_PROTO_OP_EVENT(handshake_setsockopt);
+DEFINE_HANDSHAKE_PROTO_OP_EVENT(handshake_getsockopt);
+
+TRACE_EVENT(handshake_sendmsg_start,
+	TP_PROTO(
+		const struct socket *sock,
+		size_t size
+	),
+	TP_ARGS(sock, size),
+	TP_STRUCT__entry(
+		__field(const struct socket *, sock)
+		__field(const struct sock *, sk)
+		__field(int, refcount)
+		__field(unsigned long, family)
+		__field(unsigned long, state)
+		__field(const void *, op)
+		__field(size_t, size)
+	),
+	TP_fast_assign(
+		const struct sock *sk = sock->sk;
+
+		__entry->sock = sock;
+		__entry->sk = sk;
+		__entry->refcount = refcount_read(&sk->sk_refcnt);
+		__entry->family = sk->sk_family;
+		__entry->state = sk->sk_state;
+		__entry->op = sk->sk_prot->sendmsg;
+		__entry->size = size;
+	),
+	TP_printk("sock=%p sk=%p(%d) family=%s state=%s size=%zu op=%pS",
+		__entry->sock, __entry->sk, __entry->refcount,
+		show_address_family(__entry->family),
+		show_tcp_state(__entry->state),
+		__entry->size, __entry->op
+	)
+);
+
+TRACE_EVENT(handshake_recvmsg_start,
+	TP_PROTO(
+		const struct socket *sock,
+		size_t size
+	),
+	TP_ARGS(sock, size),
+	TP_STRUCT__entry(
+		__field(const struct socket *, sock)
+		__field(const struct sock *, sk)
+		__field(int, refcount)
+		__field(unsigned long, family)
+		__field(unsigned long, state)
+		__field(const void *, op)
+		__field(size_t, size)
+	),
+	TP_fast_assign(
+		const struct sock *sk = sock->sk;
+
+		__entry->sock = sock;
+		__entry->sk = sk;
+		__entry->refcount = refcount_read(&sk->sk_refcnt);
+		__entry->family = sk->sk_family;
+		__entry->state = sk->sk_state;
+		__entry->op = sk->sk_prot->recvmsg;
+		__entry->size = size;
+	),
+	TP_printk("sock=%p sk=%p(%d) family=%s state=%s size=%zu op=%pS",
+		__entry->sock, __entry->sk, __entry->refcount,
+		show_address_family(__entry->family),
+		show_tcp_state(__entry->state),
+		__entry->size, __entry->op
+	)
+);
+
+DECLARE_EVENT_CLASS(handshake_opmsg_result_class,
+	TP_PROTO(
+		const struct socket *sock,
+		int result
+	),
+	TP_ARGS(sock, result),
+	TP_STRUCT__entry(
+		__field(const struct socket *, sock)
+		__field(const struct sock *, sk)
+		__field(int, refcount)
+		__field(unsigned long, family)
+		__field(unsigned long, state)
+		__field(int, result)
+	),
+	TP_fast_assign(
+		const struct sock *sk = sock->sk;
+
+		__entry->sock = sock;
+		__entry->sk = sk;
+		__entry->refcount = refcount_read(&sk->sk_refcnt);
+		__entry->family = sk->sk_family;
+		__entry->state = sk->sk_state;
+		__entry->result = result;
+	),
+	TP_printk("sock=%p sk=%p(%d) family=%s state=%s result=%d",
+		__entry->sock, __entry->sk, __entry->refcount,
+		show_address_family(__entry->family),
+		show_tcp_state(__entry->state),
+		__entry->result
+	)
+);
+
+#define DEFINE_HANDSHAKE_OPMSG_RESULT_EVENT(name)		\
+	DEFINE_EVENT(handshake_opmsg_result_class, name,		\
+		TP_PROTO(					\
+			const struct socket *sock,		\
+			int result				\
+		),						\
+		TP_ARGS(sock, result))
+
+DEFINE_HANDSHAKE_OPMSG_RESULT_EVENT(handshake_sendmsg_result);
+DEFINE_HANDSHAKE_OPMSG_RESULT_EVENT(handshake_recvmsg_result);
+
+TRACE_EVENT(handshake_poll,
+	TP_PROTO(
+		const struct socket *sock,
+		__poll_t mask
+	),
+	TP_ARGS(sock, mask),
+	TP_STRUCT__entry(
+		__field(const struct socket *, sock)
+		__field(const struct sock *, sk)
+		__field(int, refcount)
+		__field(unsigned long, mask)
+	),
+	TP_fast_assign(
+		const struct sock *sk = sock->sk;
+
+		__entry->sock = sock;
+		__entry->sk = sk;
+		__entry->refcount = refcount_read(&sk->sk_refcnt);
+		__entry->mask = (__force unsigned long)mask;
+	),
+	TP_printk("sock=%p sk=%p(%d) mask=%s",
+		__entry->sock, __entry->sk, __entry->refcount,
+		show_poll_event_mask(__entry->mask)
+	)
+);
+
+TRACE_EVENT(handshake_poll_listener,
+	TP_PROTO(
+		const struct socket *sock,
+		__poll_t mask
+	),
+	TP_ARGS(sock, mask),
+	TP_STRUCT__entry(
+		__field(const struct socket *, sock)
+		__field(const struct sock *, sk)
+		__field(int, refcount)
+		__field(unsigned long, mask)
+	),
+	TP_fast_assign(
+		const struct sock *sk = sock->sk;
+
+		__entry->sock = sock;
+		__entry->sk = sk;
+		__entry->refcount = refcount_read(&sk->sk_refcnt);
+		__entry->mask = (__force unsigned long)mask;
+	),
+	TP_printk("sock=%p sk=%p(%d) mask=%s",
+		__entry->sock, __entry->sk, __entry->refcount,
+		show_poll_event_mask(__entry->mask)
+	)
+);
+
+#endif /* _TRACE_HANDSHAKE_H */
+
+#include <trace/define_trace.h>
diff --git a/include/uapi/linux/handshake.h b/include/uapi/linux/handshake.h
new file mode 100644
index 000000000000..72facc352c71
--- /dev/null
+++ b/include/uapi/linux/handshake.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Generic netlink service for handshakes
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2023, Oracle and/or its affiliates.
+ */
+
+/*
+ * Data structures and functions that are visible to user space are
+ * declared here. This file constitutes an API contract between the
+ * Linux kernel and user space.
+ */
+
+#ifndef _UAPI_LINUX_HANDSHAKE_H
+#define _UAPI_LINUX_HANDSHAKE_H
+
+enum handshake_protocol {
+	HANDSHAKE_PROTO_UNSPEC = 0,
+};
+
+#define HANDSHAKE_GENL_NAME	"HANDSHAKE_GENL"
+#define HANDSHAKE_GENL_VERSION	0x01
+
+enum handshake_genl_attrs {
+	HANDSHAKE_GENL_ATTR_UNSPEC = 0,
+	HANDSHAKE_GENL_ATTR_SOCKFD,
+	HANDSHAKE_GENL_ATTR_STATUS,
+	HANDSHAKE_GENL_ATTR_PROTOCOL,
+	__HANDSHAKE_GENL_ATTR_MAX
+};
+#define HANDSHAKE_GENL_ATTR_MAX	(__HANDSHAKE_GENL_ATTR_MAX - 1)
+
+enum handshake_genl_cmds {
+	HANDSHAKE_GENL_CMD_UNSPEC = 0,
+	HANDSHAKE_GENL_CMD_GET_FD_PARAMETERS,
+	__HANDSHAKE_GENL_CMD_MAX
+};
+#define HANDSHAKE_GENL_CMD_MAX	(__HANDSHAKE_GENL_CMD_MAX - 1)
+
+enum handshake_genl_status {
+	HANDSHAKE_GENL_STATUS_OK = 0,
+	HANDSHAKE_GENL_STATUS_INVAL,
+	HANDSHAKE_GENL_STATUS_SOCKNOTFOUND,
+	HANDSHAKE_GENL_STATUS_SOCKNOTVALID,
+};
+
+#endif /* _UAPI_LINUX_HANDSHAKE_H */
diff --git a/net/Makefile b/net/Makefile
index 6a62e5b27378..c1bb53f00486 100644
--- a/net/Makefile
+++ b/net/Makefile
@@ -78,3 +78,4 @@ obj-$(CONFIG_NET_NCSI)		+= ncsi/
 obj-$(CONFIG_XDP_SOCKETS)	+= xdp/
 obj-$(CONFIG_MPTCP)		+= mptcp/
 obj-$(CONFIG_MCTP)		+= mctp/
+obj-y				+= handshake/
diff --git a/net/handshake/Makefile b/net/handshake/Makefile
new file mode 100644
index 000000000000..847e0ab2b99e
--- /dev/null
+++ b/net/handshake/Makefile
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Makefile for the HANDSHAKE subsystem.
+#
+
+obj-y += handshake.o
+handshake-y := af_handshake.o netlink.o trace.o
diff --git a/net/handshake/af_handshake.c b/net/handshake/af_handshake.c
new file mode 100644
index 000000000000..3ba3daeb82d3
--- /dev/null
+++ b/net/handshake/af_handshake.c
@@ -0,0 +1,838 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * PF_HANDSHAKE protocol family socket handler.
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2021-2023 Oracle and/or its affiliates.
+ *
+ * When the kernel needs to invoke a user space service on an open
+ * socket descriptor, it can use this mechanism to make the socket
+ * endpoint available to a user space program.
+ *
+ * The user space program listens on an AF_HANDSHAKE socket. When
+ * the listener is made ready, an accept(2) call materializes
+ * the desired socket endpoint in the listening process's file
+ * descriptor table.
+ *
+ * The listener closes that endpoint when it is finished with it
+ * (or when it exits). The kernel knows that at that point it is
+ * safe to use the socket again.
+ */
+
+/*
+ * Socket reference counting
+ *  A: listener socket initial reference
+ *  B: listener socket on the global listener list
+ *  C: listener socket while a ready AF_INET(6) socket is enqueued
+ *  D: listener socket while its accept queue is drained
+ *
+ *  I: ready AF_INET(6) socket waiting on a listener's accept queue
+ *  J: ready AF_INET(6) socket with a consumer waiting for a completion callback
+ */
+
+#include <linux/types.h>
+#include <linux/socket.h>
+#include <linux/in.h>
+#include <linux/kernel.h>
+#include <linux/poll.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/skbuff.h>
+#include <linux/inet.h>
+
+#include <net/ip.h>
+#include <net/ipv6.h>
+#include <net/tcp.h>
+#include <net/protocol.h>
+#include <net/sock.h>
+#include <net/genetlink.h>
+#include <net/inet_common.h>
+#include <net/net_namespace.h>
+#include <net/handshake.h>
+
+#include "handshake.h"
+
+#include <trace/events/handshake.h>
+
+static DEFINE_RWLOCK(handshake_listener_lock);
+static HLIST_HEAD(handshake_listeners);
+
+static void handshake_register_listener(struct sock *sk)
+{
+	write_lock_bh(&handshake_listener_lock);
+	sk_add_node(sk, &handshake_listeners);	/* Ref: B */
+	write_unlock_bh(&handshake_listener_lock);
+}
+
+static void handshake_unregister_listener(struct sock *sk)
+{
+	write_lock_bh(&handshake_listener_lock);
+	sk_del_node_init(sk);			/* Ref: B */
+	write_unlock_bh(&handshake_listener_lock);
+}
+
+/**
+ * handshake_find_listener - find listener that matches an incoming connection
+ * @net: net namespace to match
+ * @family: address family to match
+ *
+ * Return values:
+ *   On success, address of a listening AF_HANDSHAKE socket
+ *   %NULL: No matching listener found
+ */
+static struct sock *handshake_find_listener(struct net *net, unsigned short family)
+{
+	struct sock *listener;
+
+	read_lock(&handshake_listener_lock);
+
+	sk_for_each(listener, &handshake_listeners) {
+		if (sock_net(listener) != net)
+			continue;
+		if (handshake_sk(listener)->hs_bind_family != AF_UNSPEC &&
+		    handshake_sk(listener)->hs_bind_family != family)
+			continue;
+
+		sock_hold(listener);	/* Ref: C */
+		goto out;
+	}
+	listener = NULL;
+
+out:
+	read_unlock(&handshake_listener_lock);
+	return listener;
+}
+
+/**
+ * handshake_accept_enqueue - add a socket to a listener's accept_q
+ * @listener: listening socket
+ * @sk: socket to enqueue on @listener
+ *
+ * Return values:
+ *   On success, returns 0
+ *   %-ENOMEM: Memory for skbs has been exhausted
+ */
+static int handshake_accept_enqueue(struct sock *listener, struct sock *sk)
+{
+	struct sk_buff *skb;
+
+	skb = alloc_skb(0, GFP_KERNEL);
+	if (!skb)
+		return -ENOMEM;
+
+	sock_hold(sk);	/* Ref: I */
+	skb->sk = sk;
+	skb_queue_tail(&listener->sk_receive_queue, skb);
+	sk_acceptq_added(listener);
+	listener->sk_data_ready(listener);
+	return 0;
+}
+
+/**
+ * handshake_accept_dequeue - remove a socket from a listener's accept_q
+ * @listener: listener socket to check
+ *
+ * Caller must guarantee that @listener won't disappear.
+ *
+ * Return values:
+ *   On success, return a TCP socket waiting for TLS service
+ *   %NULL: No sockets on the accept queue
+ */
+static struct sock *handshake_accept_dequeue(struct sock *listener)
+{
+	struct sk_buff *skb;
+	struct sock *sk;
+
+	skb = skb_dequeue(&listener->sk_receive_queue);
+	if (!skb)
+		return NULL;
+	sk_acceptq_removed(listener);
+	sock_put(listener);	/* Ref: C */
+
+	sk = skb->sk;
+	skb->sk = NULL;
+	kfree_skb(skb);
+	sock_put(sk);	/* Ref: I */
+	return sk;
+}
+
+static void handshake_sock_save(struct sock *sk, struct handshake_info *hsi)
+{
+	sock_hold(sk);	/* Ref: J */
+
+	write_lock_bh(&sk->sk_callback_lock);
+	hsi->hi_saved_wq = sk->sk_wq_raw;
+	hsi->hi_saved_socket = sk->sk_socket;
+	hsi->hi_saved_uid = sk->sk_uid;
+	sk->sk_handshake_data = hsi;
+	write_unlock_bh(&sk->sk_callback_lock);
+}
+
+static void handshake_sock_clear(struct sock *sk)
+{
+	write_lock_bh(&sk->sk_callback_lock);
+	sk->sk_handshake_data = NULL;
+	write_unlock_bh(&sk->sk_callback_lock);
+	sock_put(sk);	/* Ref: J (err) */
+}
+
+static void handshake_sock_restore_locked(struct sock *sk)
+{
+	struct handshake_info *hsi = sk->sk_handshake_data;
+
+	sk->sk_wq_raw = hsi->hi_saved_wq;
+	sk->sk_socket = hsi->hi_saved_socket;
+	sk->sk_uid = hsi->hi_saved_uid;
+	sk->sk_handshake_data = NULL;
+}
+
+static const struct proto_ops *handshake_saved_ops(struct sock *sk)
+{
+	const struct proto_ops *ops = NULL;
+	struct handshake_info *hsi;
+
+	read_lock_bh(&sk->sk_callback_lock);
+	hsi = sk->sk_handshake_data;
+	if (hsi)
+		ops = hsi->hi_saved_socket->ops;
+	read_unlock_bh(&sk->sk_callback_lock);
+	return ops;
+}
+
+/**
+ * handshake_done - call the registered "done" callback for @sk.
+ * @sk: socket that was requesting a handshake
+ *
+ * Return values:
+ *   %true:  Handshake callback was called
+ *   %false: No handshake callback was set, no-op
+ */
+static bool handshake_done(struct sock *sk)
+{
+	struct handshake_info *hsi;
+
+	write_lock_bh(&sk->sk_callback_lock);
+	hsi = sk->sk_handshake_data;
+	if (hsi) {
+		handshake_sock_restore_locked(sk);
+		hsi->hi_done(hsi);
+	}
+	write_unlock_bh(&sk->sk_callback_lock);
+
+	if (hsi) {
+		sock_put(sk);	/* Ref: J */
+		return true;
+	}
+	return false;
+}
+
+/**
+ * handshake_accept_drain - clean up children queued for accept
+ * @listener: listener socket to drain
+ *
+ */
+static void handshake_accept_drain(struct sock *listener)
+{
+	struct sock *sk;
+
+	while ((sk = handshake_accept_dequeue(listener)))
+		handshake_done(sk);
+}
+
+/**
+ * handshake_release - free an AF_HANDSHAKE socket
+ * @sock: socket to release
+ *
+ * Return values:
+ *   %0: success
+ */
+static int handshake_release(struct socket *sock)
+{
+	struct sock *sk = sock->sk;
+	struct handshake_sock *ssk = handshake_sk(sk);
+	int ret = 0;
+
+	if (!sk)
+		return ret;
+
+	trace_handshake_release(sock);
+
+	switch (sk->sk_family) {
+	case AF_HANDSHAKE:
+		sock_hold(sk);	/* Ref: D */
+		sock_orphan(sk);
+		lock_sock(sk);
+
+		handshake_unregister_listener(sk);
+		handshake_accept_drain(sk);
+
+		sk->sk_state = TCP_CLOSE;
+		sk->sk_shutdown |= SEND_SHUTDOWN;
+		sk->sk_state_change(sk);
+
+		ssk->hs_bind_family = AF_UNSPEC;
+		sock->sk = NULL;
+		release_sock(sk);
+		sock_put(sk);	/* Ref: D */
+
+		sock_put(sk);	/* Ref: A */
+		break;
+	case AF_INET:
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+#endif
+		if (!handshake_done(sk)) {
+			const struct proto_ops *ops;
+
+			ops = handshake_saved_ops(sk);
+			if (ops)
+				ret = ops->release(sock);
+		}
+		break;
+	}
+
+	return ret;
+}
+
+/**
+ * handshake_bind - bind a name to an AF_HANDSHAKE socket
+ * @sock: socket to be bound
+ * @uaddr: address to bind to
+ * @addrlen: length in bytes of @uaddr
+ *
+ * Binding an AF_HANDSHAKE socket defines the family of addresses that
+ * are able to be accept(2)'d. So, AF_INET for ipv4, AF_INET6 for
+ * ipv6.
+ *
+ * Return values:
+ *   %0: binding was successful.
+ *   %-EPERM: Caller not privileged
+ *   %-EINVAL: Family of @sock or @uaddr not supported
+ */
+static int handshake_bind(struct socket *sock, struct sockaddr *uaddr, int addrlen)
+{
+	struct sock *listener, *sk = sock->sk;
+	struct handshake_sock *ssk = handshake_sk(sk);
+
+	if (!capable(CAP_NET_BIND_SERVICE))
+		return -EPERM;
+
+	switch (uaddr->sa_family) {
+	case AF_INET:
+		if (addrlen != sizeof(struct sockaddr_in))
+			return -EINVAL;
+		break;
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+		if (addrlen != sizeof(struct sockaddr_in6))
+			return -EINVAL;
+		break;
+#endif
+	default:
+		return -EAFNOSUPPORT;
+	}
+
+	listener = handshake_find_listener(sock_net(sk), uaddr->sa_family);
+	if (listener) {
+		sock_put(listener);	/* Ref: C */
+		return -EADDRINUSE;
+	}
+
+	ssk->hs_bind_family = uaddr->sa_family;
+	trace_handshake_bind(sock);
+	return 0;
+}
+
+/**
+ * handshake_accept - return a connection waiting for a TLS handshake
+ * @listener: listener socket which connection requests arrive on
+ * @newsock: socket to move incoming connection to
+ * @flags: SOCK_NONBLOCK and/or SOCK_CLOEXEC
+ * @kern: "boolean": 1 for kernel-internal sockets
+ *
+ * Return values:
+ *   %0: @newsock has been initialized.
+ *   %-EPERM: caller is not privileged
+ */
+static int handshake_accept(struct socket *listener, struct socket *newsock, int flags,
+			  bool kern)
+{
+	struct sock *sk = listener->sk, *newsk;
+	DECLARE_WAITQUEUE(wait, current);
+	long timeo;
+	int rc;
+
+	trace_handshake_accept(listener);
+
+	rc = -EPERM;
+	if (!capable(CAP_NET_BIND_SERVICE))
+		goto out;
+
+	lock_sock(sk);
+
+	if (sk->sk_state != TCP_LISTEN) {
+		rc = -EBADF;
+		goto out_release;
+	}
+
+	timeo = sock_rcvtimeo(sk, flags & O_NONBLOCK);
+
+	rc = 0;
+	add_wait_queue_exclusive(sk_sleep(sk), &wait);
+	while (!(newsk = handshake_accept_dequeue(sk))) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		if (!timeo) {
+			rc = -EAGAIN;
+			break;
+		}
+		release_sock(sk);
+
+		timeo = schedule_timeout(timeo);
+
+		lock_sock(sk);
+		if (sk->sk_state != TCP_LISTEN) {
+			rc = -EBADF;
+			break;
+		}
+		if (signal_pending(current)) {
+			rc = sock_intr_errno(timeo);
+			break;
+		}
+	}
+	set_current_state(TASK_RUNNING);
+	remove_wait_queue(sk_sleep(sk), &wait);
+	if (rc) {
+		handshake_done(sk);
+		goto out_release;
+	}
+
+	sock_graft(newsk, newsock);
+	trace_handshake_newsock(newsock, newsk);
+
+out_release:
+	release_sock(sk);
+out:
+	return rc;
+}
+
+/**
+ * handshake_getname - retrieve src/dst address information from an AF_HANDSHAKE socket
+ * @sock: socket to query
+ * @uaddr: buffer to fill in
+ * @peer: value indicates which address to retrieve
+ *
+ * Return values:
+ *   On success, a positive length of the address in @uaddr
+ *   On error, a negative errno
+ */
+static int handshake_getname(struct socket *sock, struct sockaddr *uaddr, int peer)
+{
+	struct sock *sk = sock->sk;
+	const struct proto_ops *ops;
+
+	trace_handshake_getname(sock);
+
+	switch (sk->sk_family) {
+	case AF_INET:
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+#endif
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
+
+	ops = handshake_saved_ops(sk);
+	if (!ops)
+		return -EBADFD;
+	return ops->getname(sock, uaddr, peer);
+}
+
+/**
+ * handshake_poll - check for data ready on an AF_HANDSHAKE socket
+ * @file: file to check for work
+ * @sock: socket associated with @file
+ * @wait: poll table
+ *
+ * Return values:
+ *    A mask of flags indicating what type of I/O is ready
+ */
+static __poll_t handshake_poll(struct file *file, struct socket *sock,
+			     poll_table *wait)
+{
+	struct sock *sk = sock->sk;
+	__poll_t mask;
+
+	sock_poll_wait(file, sock, wait);
+
+	mask = 0;
+
+	if (sk->sk_state == TCP_LISTEN) {
+		if (!skb_queue_empty_lockless(&sk->sk_receive_queue))
+			mask |= EPOLLIN | EPOLLRDNORM;
+		if (sk_is_readable(sk))
+			mask |= EPOLLIN | EPOLLRDNORM;
+		trace_handshake_poll_listener(sock, mask);
+		return mask;
+	}
+
+	if (sk->sk_shutdown == SHUTDOWN_MASK || sk->sk_state == TCP_CLOSE)
+		mask |= EPOLLHUP;
+	if (sk->sk_shutdown & RCV_SHUTDOWN)
+		mask |= EPOLLIN | EPOLLRDNORM | EPOLLRDHUP;
+
+	if (!skb_queue_empty_lockless(&sk->sk_receive_queue))
+		mask |= EPOLLIN | EPOLLRDNORM;
+	if (sk_is_readable(sk))
+		mask |= EPOLLIN | EPOLLRDNORM;
+
+	/* This barrier is coupled with smp_wmb() in tcp_reset() */
+	smp_rmb();
+	if (sk->sk_err || !skb_queue_empty_lockless(&sk->sk_error_queue))
+		mask |= EPOLLERR;
+
+	trace_handshake_poll(sock, mask);
+	return mask;
+}
+
+/**
+ * handshake_listen - move an AF_HANDSHAKE socket into a listening state
+ * @sock: socket to transition to listening state
+ * @backlog: size of backlog queue
+ *
+ * Return values:
+ *   %0: @sock is now in a listening state
+ *   %-EPERM: caller is not privileged
+ *   %-EOPNOTSUPP: @sock is not of a type that supports the listen() operation
+ */
+static int handshake_listen(struct socket *sock, int backlog)
+{
+	struct sock *sk = sock->sk;
+	unsigned char old_state;
+	int rc;
+
+	if (!capable(CAP_NET_BIND_SERVICE))
+		return -EPERM;
+
+	lock_sock(sk);
+
+	rc = -EOPNOTSUPP;
+	if (sock->state != SS_UNCONNECTED || sock->type != SOCK_STREAM)
+		goto out;
+	old_state = sk->sk_state;
+	if (!((1 << old_state) & (TCPF_CLOSE | TCPF_LISTEN)))
+		goto out;
+
+	sk->sk_max_ack_backlog = backlog;
+	sk->sk_state = TCP_LISTEN;
+	handshake_register_listener(sk);
+
+	trace_handshake_listen(sock);
+	rc = 0;
+
+out:
+	release_sock(sk);
+	return rc;
+}
+
+/**
+ * handshake_shutdown - Shutdown an AF_HANDSHAKE socket
+ * @sock: socket to shut down
+ * @how: mask
+ *
+ * Return values:
+ *   %0: Success
+ *   %-EINVAL: @sock is not of a type that supports a shutdown
+ */
+static int handshake_shutdown(struct socket *sock, int how)
+{
+	struct sock *sk = sock->sk;
+
+	trace_handshake_shutdown(sock);
+
+	switch (sk->sk_family) {
+	case AF_INET:
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+#endif
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return inet_shutdown(sock, how);
+}
+
+/**
+ * handshake_setsockopt - Set a socket option on an AF_HANDSHAKE socket
+ * @sock: socket to act upon
+ * @level: which network layer to act upon
+ * @optname: which option to set
+ * @optval: new value to set
+ * @optlen: the size of the new value, in bytes
+ *
+ * Return values:
+ *   %0: Success
+ *   %-ENOPROTOOPT: The option is unknown at the level indicated.
+ */
+static int handshake_setsockopt(struct socket *sock, int level, int optname,
+			      sockptr_t optval, unsigned int optlen)
+{
+	struct sock *sk = sock->sk;
+
+	trace_handshake_setsockopt(sock);
+
+	switch (sk->sk_family) {
+	case AF_INET:
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+#endif
+		break;
+	default:
+		return -ENOPROTOOPT;
+	}
+
+	return sock_common_setsockopt(sock, level, optname, optval, optlen);
+}
+
+/**
+ * handshake_getsockopt - Retrieve a socket option from an AF_HANDSHAKE socket
+ * @sock: socket to act upon
+ * @level: which network layer to act upon
+ * @optname: which option to retrieve
+ * @optval: a buffer into which to receive the option's value
+ * @optlen: the size of the receive buffer, in bytes
+ *
+ * Return values:
+ *   %0: Success
+ *   %-ENOPROTOOPT: The option is unknown at the level indicated.
+ *   %-EINVAL: Invalid argument
+ *   %-EFAULT: Output memory not write-able
+ *   %-EBUSY: Option value not available
+ */
+static int handshake_getsockopt(struct socket *sock, int level, int optname,
+			      char __user *optval, int __user *optlen)
+{
+	struct sock *sk = sock->sk;
+
+	trace_handshake_getsockopt(sock);
+
+	switch (sk->sk_family) {
+	case AF_INET:
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+#endif
+		break;
+	default:
+		return -ENOPROTOOPT;
+	}
+
+	return sock_common_getsockopt(sock, level, optname, optval, optlen);
+}
+
+/**
+ * handshake_sendmsg - Send a message on an AF_HANDSHAKE socket
+ * @sock: socket to send on
+ * @msg: message to send
+ * @size: size of message, in bytes
+ *
+ * Return values:
+ *   %0: Success
+ *   %-EOPNOTSUPP: Address family does not support this operation
+ */
+static int handshake_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
+{
+	struct sock *sk = sock->sk;
+	int ret;
+
+	trace_handshake_sendmsg_start(sock, size);
+
+	switch (sk->sk_family) {
+	case AF_INET:
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+#endif
+		break;
+	default:
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+
+	if (unlikely(inet_send_prepare(sk))) {
+		ret = -EAGAIN;
+		goto out;
+	}
+	ret = sk->sk_prot->sendmsg(sk, msg, size);
+
+out:
+	trace_handshake_sendmsg_result(sock, ret);
+	return ret;
+}
+
+/**
+ * handshake_recvmsg - Receive a message from an AF_HANDSHAKE socket
+ * @sock: socket to receive from
+ * @msg: buffer into which to receive
+ * @size: size of buffer, in bytes
+ * @flags: control settings
+ *
+ * Return values:
+ *   %0: Success
+ *   %-EOPNOTSUPP: Address family does not support this operation
+ */
+static int handshake_recvmsg(struct socket *sock, struct msghdr *msg,
+			   size_t size, int flags)
+{
+	struct sock *sk = sock->sk;
+	int ret;
+
+	trace_handshake_recvmsg_start(sock, size);
+
+	switch (sk->sk_family) {
+	case AF_INET:
+#if IS_ENABLED(CONFIG_IPV6)
+	case AF_INET6:
+#endif
+		break;
+	default:
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+
+	if (likely(!(flags & MSG_ERRQUEUE)))
+		sock_rps_record_flow(sk);
+	ret = sock_common_recvmsg(sock, msg, size, flags);
+
+out:
+	trace_handshake_recvmsg_result(sock, ret);
+	return ret;
+}
+
+static const struct proto_ops handshake_proto_ops = {
+	.family		= PF_HANDSHAKE,
+	.owner		= THIS_MODULE,
+
+	.release	= handshake_release,
+	.bind		= handshake_bind,
+	.connect	= sock_no_connect,
+	.socketpair	= sock_no_socketpair,
+	.accept		= handshake_accept,
+	.getname	= handshake_getname,
+	.poll		= handshake_poll,
+	.ioctl		= sock_no_ioctl,
+	.gettstamp	= sock_gettstamp,
+	.listen		= handshake_listen,
+	.shutdown	= handshake_shutdown,
+	.setsockopt	= handshake_setsockopt,
+	.getsockopt	= handshake_getsockopt,
+	.sendmsg	= handshake_sendmsg,
+	.recvmsg	= handshake_recvmsg,
+	.mmap		= sock_no_mmap,
+	.sendpage	= sock_no_sendpage,
+};
+
+static struct proto handshake_prot = {
+	.name			= "HANDSHAKE",
+	.owner			= THIS_MODULE,
+	.obj_size		= sizeof(struct handshake_sock),
+};
+
+/**
+ * handshake_pf_create - create an AF_HANDSHAKE socket
+ * @net: network namespace to own the new socket
+ * @sock: socket to initialize
+ * @protocol: IP protocol number (ignored)
+ * @kern: "boolean": 1 for kernel-internal sockets
+ *
+ * Return values:
+ *   %0: @sock was initialized, and module ref count incremented.
+ *   Negative errno values indicate initialization failed.
+ */
+static int handshake_pf_create(struct net *net, struct socket *sock, int protocol,
+			     int kern)
+{
+	struct sock *sk;
+	int rc;
+
+	sock->state = SS_UNCONNECTED;
+	sock->ops = &handshake_proto_ops;
+
+	/* Ref: A */
+	sk = sk_alloc(net, PF_HANDSHAKE, GFP_KERNEL, &handshake_prot, kern);
+	if (!sk)
+		return -ENOMEM;
+
+	sock_init_data(sock, sk);
+	if (sk->sk_prot->init) {
+		rc = sk->sk_prot->init(sk);
+		if (rc)
+			goto err_sk_put;
+	}
+
+	handshake_sk(sk)->hs_bind_family = AF_UNSPEC;
+	trace_handshake_pf_create(sock);
+	return 0;
+
+err_sk_put:
+	sock_orphan(sk);
+	sk_free(sk);	/* Ref: A (err) */
+	return rc;
+}
+
+/**
+ * handshake_enqueue_sock - Queue a socket to be shared with user space
+ * @sock: a connected socket to share with user space
+ * @hsi: info packet tracking this request
+ *
+ * Return values:
+ *   %0: Successfully queued
+ *   %-ENOENT: No listener is available to handle this request
+ *   %-ENOMEM: Memory allocation failed
+ */
+int handshake_enqueue_sock(struct socket *sock, struct handshake_info *hsi)
+{
+	struct sock *listener, *sk = sock->sk;
+	int rc;
+
+	listener = handshake_find_listener(sock_net(sk), sk->sk_family);
+	if (!listener)
+		return -ENOENT;
+
+	handshake_sock_save(sk, hsi);
+	rc = handshake_accept_enqueue(listener, sk);
+	if (rc) {
+		handshake_sock_clear(sk);
+		sock_put(listener);	/* Ref: C (err) */
+	}
+	return rc;
+}
+EXPORT_SYMBOL(handshake_enqueue_sock);
+
+static const struct net_proto_family handshake_pf_ops = {
+	.family = PF_HANDSHAKE,
+	.create = handshake_pf_create,
+	.owner	= THIS_MODULE,
+};
+
+static int __init handshake_register(void)
+{
+	int rc;
+
+	rc = handshake_genetlink_init();
+	if (rc)
+		return rc;
+
+	sock_register(&handshake_pf_ops);
+	return 0;
+}
+
+static void __exit handshake_unregister(void)
+{
+	sock_unregister(PF_HANDSHAKE);
+	handshake_genetlink_exit();
+}
+
+
+module_init(handshake_register);
+module_exit(handshake_unregister);
diff --git a/net/handshake/handshake.h b/net/handshake/handshake.h
new file mode 100644
index 000000000000..62a6c85c5a17
--- /dev/null
+++ b/net/handshake/handshake.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * PF_HANDSHAKE protocol family socket handler.
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2023, Oracle and/or its affiliates.
+ */
+
+/*
+ * Data structures and functions that are internal to handshake/
+ * are declared here.
+ */
+
+#ifndef _HANDSHAKE_H
+#define _HANDSHAKE_H
+
+struct handshake_sock {
+	/* struct sock must remain the first field */
+	struct sock	hs_sk;
+
+	int		hs_bind_family;
+};
+
+static inline struct handshake_sock *handshake_sk(struct sock *sk)
+{
+	return container_of(sk, struct handshake_sock, hs_sk);
+}
+
+extern int __init handshake_genetlink_init(void);
+extern void handshake_genetlink_exit(void);
+
+#endif /* _HANDSHAKE_H */
diff --git a/net/handshake/netlink.c b/net/handshake/netlink.c
new file mode 100644
index 000000000000..1d209473f106
--- /dev/null
+++ b/net/handshake/netlink.c
@@ -0,0 +1,169 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * HANDSHAKE generic netlink service
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2023, Oracle and/or its affiliates.
+ */
+
+#include <linux/types.h>
+#include <linux/socket.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/skbuff.h>
+#include <linux/inet.h>
+
+#include <net/sock.h>
+#include <net/genetlink.h>
+#include <net/handshake.h>
+
+#include <uapi/linux/handshake.h>
+#include "handshake.h"
+
+static struct genl_family __ro_after_init handshake_genl_family;
+
+static int handshake_genl_op_unsupp(struct sk_buff *skb, struct genl_info *gi)
+{
+	pr_err("Unknown netlink command (%d) ignored\n", gi->genlhdr->cmd);
+	return -EINVAL;
+}
+
+static int handshake_genl_error_reply(struct genl_info *gi,
+				      enum handshake_genl_status status)
+{
+	struct genlmsghdr *hdr;
+	struct sk_buff *msg;
+	int ret;
+
+	ret = -ENOMEM;
+	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+	if (!msg)
+		goto out;
+	hdr = genlmsg_put_reply(msg, gi, &handshake_genl_family, 0,
+				gi->genlhdr->cmd);
+	if (!hdr)
+		goto out_free;
+
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_STATUS, status);
+	if (ret < 0)
+		goto out_cancel;
+
+	genlmsg_end(msg, hdr);
+	return genlmsg_reply(msg, gi);
+
+out_cancel:
+	genlmsg_cancel(msg, hdr);
+out_free:
+	nlmsg_free(msg);
+out:
+	return ret;
+}
+
+static int handshake_genl_reply(struct genl_info *gi, struct handshake_info *hsi)
+{
+	struct genlmsghdr *hdr;
+	struct sk_buff *msg;
+	int ret;
+
+	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+	if (!msg)
+		goto out;
+	hdr = genlmsg_put_reply(msg, gi, &handshake_genl_family, 0,
+				gi->genlhdr->cmd);
+	if (!hdr)
+		goto out_free;
+
+	ret = hsi->hi_fd_parms_reply(msg, hsi);
+	if (ret < 0)
+		goto out_cancel;
+
+	genlmsg_end(msg, hdr);
+	return genlmsg_reply(msg, gi);
+
+out_cancel:
+	genlmsg_cancel(msg, hdr);
+out_free:
+	nlmsg_free(msg);
+out:
+	return ret;
+}
+
+static int handshake_genl_op_get_fd_parms(struct sk_buff *skb, struct genl_info *gi)
+{
+	struct handshake_info *hsi;
+	struct socket *sock;
+	struct sock *sk;
+	int ret;
+
+	if (!gi->attrs[HANDSHAKE_GENL_ATTR_SOCKFD])
+		return handshake_genl_error_reply(gi, HANDSHAKE_GENL_STATUS_INVAL);
+
+	ret = 0;
+	sock = sockfd_lookup(nla_get_u32(gi->attrs[HANDSHAKE_GENL_ATTR_SOCKFD]),
+			     &ret);
+	if (ret)
+		return handshake_genl_error_reply(gi, HANDSHAKE_GENL_STATUS_SOCKNOTFOUND);
+
+	sk = sock->sk;
+	write_lock_bh(&sk->sk_callback_lock);
+	hsi = sk->sk_handshake_data;
+	if (!hsi) {
+		write_unlock_bh(&sk->sk_callback_lock);
+		sockfd_put(sock);
+		return handshake_genl_error_reply(gi, HANDSHAKE_GENL_STATUS_SOCKNOTVALID);
+	}
+	write_unlock_bh(&sk->sk_callback_lock);
+
+	ret = handshake_genl_reply(gi, hsi);
+
+	sockfd_put(sock);
+	return ret;
+}
+
+static const struct nla_policy
+handshake_genl_policy[HANDSHAKE_GENL_ATTR_MAX + 1] = {
+	[HANDSHAKE_GENL_ATTR_SOCKFD] = {
+		.type = NLA_U32
+	},
+	[HANDSHAKE_GENL_ATTR_STATUS] = {
+		.type = NLA_U32
+	},
+	[HANDSHAKE_GENL_ATTR_PROTOCOL] = {
+		.type = NLA_U32
+	},
+};
+
+static const struct genl_ops handshake_genl_ops[] = {
+	{
+		.cmd	= HANDSHAKE_GENL_CMD_UNSPEC,
+		.doit	= handshake_genl_op_unsupp,
+	},
+	{
+		.cmd	= HANDSHAKE_GENL_CMD_GET_FD_PARAMETERS,
+		.doit	= handshake_genl_op_get_fd_parms,
+	},
+};
+
+static struct genl_family __ro_after_init handshake_genl_family = {
+	.hdrsize	= 0,
+	.name		= HANDSHAKE_GENL_NAME,
+	.version	= HANDSHAKE_GENL_VERSION,
+	.maxattr	= HANDSHAKE_GENL_ATTR_MAX,
+	.netnsok	= true,
+	.n_ops		= ARRAY_SIZE(handshake_genl_ops),
+	.resv_start_op	= HANDSHAKE_GENL_CMD_MAX,
+	.policy		= handshake_genl_policy,
+	.ops		= handshake_genl_ops,
+	.module		= THIS_MODULE,
+};
+
+int __init handshake_genetlink_init(void)
+{
+	return genl_register_family(&handshake_genl_family);
+}
+
+void handshake_genetlink_exit(void)
+{
+	genl_unregister_family(&handshake_genl_family);
+}
diff --git a/net/handshake/trace.c b/net/handshake/trace.c
new file mode 100644
index 000000000000..5968848da0c1
--- /dev/null
+++ b/net/handshake/trace.c
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * PF_HANDSHAKE protocol family trace points
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2023 Oracle and/or its affiliates.
+ */
+
+#include <linux/net.h>
+#include <net/sock.h>
+
+#include "handshake.h"
+
+#ifndef __CHECKER__
+
+#define CREATE_TRACE_POINTS
+#include <trace/events/handshake.h>
+
+#endif

[PATCH v2 3/3] net/tls: Support AF_HANDSHAKE in kTLS

[Chuck Lever]

To enable kernel consumers of TLS to request a TLS handshake, add
support to net/tls/ to send a handshake upcall.

This patch also acts as a template for adding handshake upcall
support to other transport layer security mechanisms.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 include/net/tls.h              |   16 ++
 include/uapi/linux/handshake.h |   24 ++
 net/handshake/netlink.c        |   18 ++
 net/tls/Makefile               |    2 
 net/tls/tls_handshake.c        |  385 ++++++++++++++++++++++++++++++++++++++++
 5 files changed, 444 insertions(+), 1 deletion(-)
 create mode 100644 net/tls/tls_handshake.c

diff --git a/include/net/tls.h b/include/net/tls.h
index 154949c7b0c8..5156c3a80faa 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -512,4 +512,20 @@ static inline bool tls_is_sk_rx_device_offloaded(struct sock *sk)
 	return tls_get_ctx(sk)->rx_conf == TLS_HW;
 }
 #endif
+
+#define TLS_DEFAULT_PRIORITIES		(NULL)
+
+int tls_client_hello_anon(struct socket *sock,
+			  void (*done)(void *data, int status), void *data,
+			  const char *priorities);
+int tls_client_hello_x509(struct socket *sock,
+			  void (*done)(void *data, int status), void *data,
+			  const char *priorities, key_serial_t cert,
+			  key_serial_t privkey);
+int tls_client_hello_psk(struct socket *sock,
+			 void (*done)(void *data, int status), void *data,
+			 const char *priorities, key_serial_t peerid);
+int tls_server_hello(struct socket *sock, void (*done)(void *data, int status),
+		     void *data, const char *priorities);
+
 #endif /* _TLS_OFFLOAD_H */
diff --git a/include/uapi/linux/handshake.h b/include/uapi/linux/handshake.h
index 72facc352c71..ec3f4a5e465f 100644
--- a/include/uapi/linux/handshake.h
+++ b/include/uapi/linux/handshake.h
@@ -18,8 +18,26 @@
 
 enum handshake_protocol {
 	HANDSHAKE_PROTO_UNSPEC = 0,
+	HANDSHAKE_PROTO_TLS_13,
 };
 
+enum handshake_type {
+	HANDSHAKE_TYPE_UNSPEC = 0,
+	HANDSHAKE_TYPE_CLIENTHELLO,
+	HANDSHAKE_TYPE_SERVERHELLO,
+};
+
+enum handshake_auth_type {
+	HANDSHAKE_AUTHTYPE_UNSPEC = 0,
+	HANDSHAKE_AUTHTYPE_UNAUTH,
+	HANDSHAKE_AUTHTYPE_X509,
+	HANDSHAKE_AUTHTYPE_PSK,
+};
+
+#define HANDSHAKE_NO_PEERID		(0)
+#define HANDSHAKE_NO_CERT		(0)
+#define HANDSHAKE_NO_PRIVKEY		(0)
+
 #define HANDSHAKE_GENL_NAME	"HANDSHAKE_GENL"
 #define HANDSHAKE_GENL_VERSION	0x01
 
@@ -28,6 +46,12 @@ enum handshake_genl_attrs {
 	HANDSHAKE_GENL_ATTR_SOCKFD,
 	HANDSHAKE_GENL_ATTR_STATUS,
 	HANDSHAKE_GENL_ATTR_PROTOCOL,
+	HANDSHAKE_GENL_ATTR_HANDSHAKE_TYPE,
+	HANDSHAKE_GENL_ATTR_AUTH_TYPE,
+	HANDSHAKE_GENL_ATTR_TLS_PRIORITIES,
+	HANDSHAKE_GENL_ATTR_X509_CERT_SERIAL,
+	HANDSHAKE_GENL_ATTR_X509_PRIVKEY_SERIAL,
+	HANDSHAKE_GENL_ATTR_PSK_SERIAL,
 	__HANDSHAKE_GENL_ATTR_MAX
 };
 #define HANDSHAKE_GENL_ATTR_MAX	(__HANDSHAKE_GENL_ATTR_MAX - 1)
diff --git a/net/handshake/netlink.c b/net/handshake/netlink.c
index 1d209473f106..d993f284ae0a 100644
--- a/net/handshake/netlink.c
+++ b/net/handshake/netlink.c
@@ -132,6 +132,24 @@ handshake_genl_policy[HANDSHAKE_GENL_ATTR_MAX + 1] = {
 	[HANDSHAKE_GENL_ATTR_PROTOCOL] = {
 		.type = NLA_U32
 	},
+	[HANDSHAKE_GENL_ATTR_HANDSHAKE_TYPE] = {
+		.type = NLA_U32
+	},
+	[HANDSHAKE_GENL_ATTR_AUTH_TYPE] = {
+		.type = NLA_U32
+	},
+	[HANDSHAKE_GENL_ATTR_TLS_PRIORITIES] = {
+		.type = NLA_STRING
+	},
+	[HANDSHAKE_GENL_ATTR_X509_CERT_SERIAL] = {
+		.type = NLA_U32
+	},
+	[HANDSHAKE_GENL_ATTR_X509_PRIVKEY_SERIAL] = {
+		.type = NLA_U32
+	},
+	[HANDSHAKE_GENL_ATTR_PSK_SERIAL] = {
+		.type = NLA_U32
+	},
 };
 
 static const struct genl_ops handshake_genl_ops[] = {
diff --git a/net/tls/Makefile b/net/tls/Makefile
index e41c800489ac..7e56b57f14f6 100644
--- a/net/tls/Makefile
+++ b/net/tls/Makefile
@@ -7,7 +7,7 @@ CFLAGS_trace.o := -I$(src)
 
 obj-$(CONFIG_TLS) += tls.o
 
-tls-y := tls_main.o tls_sw.o tls_proc.o trace.o tls_strp.o
+tls-y := tls_handshake.o tls_main.o tls_sw.o tls_proc.o trace.o tls_strp.o
 
 tls-$(CONFIG_TLS_TOE) += tls_toe.o
 tls-$(CONFIG_TLS_DEVICE) += tls_device.o tls_device_fallback.o
diff --git a/net/tls/tls_handshake.c b/net/tls/tls_handshake.c
new file mode 100644
index 000000000000..c718cafd8676
--- /dev/null
+++ b/net/tls/tls_handshake.c
@@ -0,0 +1,385 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Establish a TLS session for a kernel socket consumer
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2021-2023, Oracle and/or its affiliates.
+ */
+
+/**
+ * DOC: kTLS handshake overview
+ *
+ * When a kernel TLS consumer wants to establish a TLS session, it
+ * makes an AF_TLSH Listener ready. When user space accepts on that
+ * listener, the kernel fabricates a user space socket endpoint on
+ * which a user space TLS library can perform the TLS handshake.
+ *
+ * Closing the user space descriptor signals to the kernel that the
+ * library handshake process is complete. If the library has managed
+ * to initialize the socket's TLS crypto_info, the kernel marks the
+ * handshake as a success.
+ */
+
+#include <linux/types.h>
+#include <linux/socket.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+
+#include <net/sock.h>
+
+#include <net/tls.h>
+#include <net/handshake.h>
+
+#include <uapi/linux/handshake.h>
+
+static void tlsh_handshake_done(struct handshake_info *hsi);
+
+struct tlsh_sock_info {
+	struct handshake_info	tsi_handshake_info;
+
+	void			(*tsi_handshake_done)(void *data, int status);
+	void			*tsi_handshake_data;
+
+	char			*tsi_tls_priorities;
+	key_serial_t		tsi_peerid;
+	key_serial_t		tsi_certificate;
+	key_serial_t		tsi_privkey;
+
+};
+
+static struct tlsh_sock_info *tlsh_sock_info_alloc(struct socket *sock,
+						   const char *priorities)
+{
+	struct tlsh_sock_info *tsi;
+
+	tsi = kzalloc(sizeof(*tsi), GFP_KERNEL);
+	if (!tsi)
+		return NULL;
+
+	if (priorities != TLS_DEFAULT_PRIORITIES && strlen(priorities)) {
+		tsi->tsi_tls_priorities = kstrdup(priorities, GFP_KERNEL);
+		if (!tsi->tsi_tls_priorities) {
+			kfree(tsi);
+			return NULL;
+		}
+	}
+
+	tsi->tsi_handshake_info.hi_done = tlsh_handshake_done,
+	tsi->tsi_handshake_info.hi_data = sock->sk;
+	tsi->tsi_peerid = HANDSHAKE_NO_PEERID;
+	tsi->tsi_certificate = HANDSHAKE_NO_CERT;
+	tsi->tsi_privkey = HANDSHAKE_NO_PRIVKEY;
+	return tsi;
+}
+
+static void tlsh_sock_info_free(struct tlsh_sock_info *tsi)
+{
+	if (tsi)
+		kfree(tsi->tsi_tls_priorities);
+	kfree(tsi);
+}
+
+static bool tlsh_crypto_info_initialized(struct sock *sk)
+{
+	struct tls_context *ctx = tls_get_ctx(sk);
+
+	return ctx &&
+		TLS_CRYPTO_INFO_READY(&ctx->crypto_send.info) &&
+		TLS_CRYPTO_INFO_READY(&ctx->crypto_recv.info);
+}
+
+/**
+ * tlsh_handshake_done - call the handshake "done" callback for @sk.
+ * @hsi: socket on which the handshake was performed
+ *
+ */
+static void tlsh_handshake_done(struct handshake_info *hsi)
+{
+	struct tlsh_sock_info *tsi = container_of(hsi, struct tlsh_sock_info,
+						  tsi_handshake_info);
+	void (*done)(void *data, int status) = tsi->tsi_handshake_done;
+	struct sock *sk = hsi->hi_data;
+
+	done(tsi->tsi_handshake_data,
+	     tlsh_crypto_info_initialized(sk) ? 0 : -EACCES);
+	tlsh_sock_info_free(tsi);
+}
+
+/*
+ * Specifically for kernel TLS consumers: enable only TLS v1.3 and the
+ * ciphers that are supported by kTLS.
+ *
+ * This list is generated by hand from the supported ciphers found
+ * in include/uapi/linux/tls.h.
+ */
+#define KTLS_PRIORITIES \
+	"SECURE256:+SECURE128:-COMP-ALL" \
+	":-VERS-ALL:+VERS-TLS1.3:%NO_TICKETS" \
+	":-CIPHER-ALL:+CHACHA20-POLY1305:+AES-256-GCM:+AES-128-GCM:+AES-128-CCM"
+
+static int tlsh_genl_tls_priorities(struct sk_buff *msg,
+				    struct tlsh_sock_info *tsi)
+{
+	int ret;
+
+	if (tsi->tsi_tls_priorities)
+		ret = nla_put(msg, HANDSHAKE_GENL_ATTR_TLS_PRIORITIES,
+			      strlen(tsi->tsi_tls_priorities),
+			      tsi->tsi_tls_priorities);
+	else
+		ret = nla_put(msg, HANDSHAKE_GENL_ATTR_TLS_PRIORITIES,
+			      strlen(KTLS_PRIORITIES), KTLS_PRIORITIES);
+	if (ret < 0)
+		return ret;
+	return 0;
+}
+
+static int tlsh_genl_ch_anon_reply(struct sk_buff *msg,
+				   struct handshake_info *hsi)
+{
+	struct tlsh_sock_info *tsi = container_of(hsi, struct tlsh_sock_info,
+						  tsi_handshake_info);
+	int ret;
+
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_PROTOCOL,
+			  HANDSHAKE_PROTO_TLS_13);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_HANDSHAKE_TYPE,
+			  HANDSHAKE_TYPE_CLIENTHELLO);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_AUTH_TYPE,
+			  HANDSHAKE_AUTHTYPE_UNAUTH);
+	if (ret < 0)
+		return ret;
+	return tlsh_genl_tls_priorities(msg, tsi);
+}
+
+/**
+ * tls_client_hello_anon - request an anonymous TLS handshake on a socket
+ * @sock: connected socket on which to perform the handshake
+ * @done: function to call when the handshake has completed
+ * @data: token to pass back to @done
+ * @priorities: GnuTLS TLS priorities string, or NULL
+ *
+ * Return values:
+ *   %0: Handshake request enqueue; ->done will be called when complete
+ *   %-ENOENT: No user agent is available
+ *   %-ENOMEM: Memory allocation failed
+ */
+int tls_client_hello_anon(struct socket *sock,
+			  void (*done)(void *data, int status), void *data,
+			  const char *priorities)
+{
+	struct tlsh_sock_info *tsi;
+	int rc;
+
+	tsi = tlsh_sock_info_alloc(sock, priorities);
+	if (!tsi)
+		return -ENOMEM;
+
+
+	tsi->tsi_handshake_done = done;
+	tsi->tsi_handshake_data = data;
+
+	tsi->tsi_handshake_info.hi_fd_parms_reply = tlsh_genl_ch_anon_reply;
+
+	rc = handshake_enqueue_sock(sock, &tsi->tsi_handshake_info);
+	if (rc)
+		tlsh_sock_info_free(tsi);
+	return rc;
+}
+EXPORT_SYMBOL(tls_client_hello_anon);
+
+static int tlsh_genl_ch_x509_reply(struct sk_buff *msg,
+				   struct handshake_info *hsi)
+{
+	struct tlsh_sock_info *tsi = container_of(hsi, struct tlsh_sock_info,
+						  tsi_handshake_info);
+	int ret;
+
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_PROTOCOL,
+			  HANDSHAKE_PROTO_TLS_13);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_HANDSHAKE_TYPE,
+			  HANDSHAKE_TYPE_CLIENTHELLO);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_AUTH_TYPE,
+			  HANDSHAKE_AUTHTYPE_X509);
+	if (ret < 0)
+		return ret;
+	ret = tlsh_genl_tls_priorities(msg, tsi);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_X509_CERT_SERIAL,
+			  tsi->tsi_certificate);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_X509_PRIVKEY_SERIAL,
+			  tsi->tsi_privkey);
+	if (ret < 0)
+		return ret;
+	return 0;
+}
+
+/**
+ * tls_client_hello_x509 - request an x.509-based TLS handshake on a socket
+ * @sock: connected socket on which to perform the handshake
+ * @done: function to call when the handshake has completed
+ * @data: token to pass back to @done
+ * @priorities: GnuTLS TLS priorities string
+ * @cert: serial number of key containing client's x.509 certificate
+ * @privkey: serial number of key containing client's private key
+ *
+ * Return values:
+ *   %0: Handshake request enqueue; ->done will be called when complete
+ *   %-ENOENT: No user agent is available
+ *   %-ENOMEM: Memory allocation failed
+ */
+int tls_client_hello_x509(struct socket *sock,
+			  void (*done)(void *data, int status), void *data,
+			  const char *priorities, key_serial_t cert,
+			  key_serial_t privkey)
+{
+	struct tlsh_sock_info *tsi;
+	int rc;
+
+	tsi = tlsh_sock_info_alloc(sock, priorities);
+	if (!tsi)
+		return -ENOMEM;
+
+	tsi->tsi_handshake_done = done;
+	tsi->tsi_handshake_data = data;
+
+	tsi->tsi_handshake_info.hi_fd_parms_reply = tlsh_genl_ch_x509_reply;
+	tsi->tsi_certificate = cert;
+	tsi->tsi_privkey = privkey;
+
+	rc = handshake_enqueue_sock(sock, &tsi->tsi_handshake_info);
+	if (rc)
+		tlsh_sock_info_free(tsi);
+	return rc;
+}
+EXPORT_SYMBOL(tls_client_hello_x509);
+
+static int tlsh_genl_ch_psk_reply(struct sk_buff *msg,
+				  struct handshake_info *hsi)
+{
+	struct tlsh_sock_info *tsi = container_of(hsi, struct tlsh_sock_info,
+						  tsi_handshake_info);
+	int ret;
+
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_PROTOCOL,
+			  HANDSHAKE_PROTO_TLS_13);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_HANDSHAKE_TYPE,
+			  HANDSHAKE_TYPE_CLIENTHELLO);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_AUTH_TYPE,
+			  HANDSHAKE_AUTHTYPE_PSK);
+	if (ret < 0)
+		return ret;
+	ret = tlsh_genl_tls_priorities(msg, tsi);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_PSK_SERIAL,
+			  tsi->tsi_peerid);
+	if (ret < 0)
+		return ret;
+	return 0;
+}
+
+/**
+ * tls_client_hello_psk - request a PSK-based TLS handshake on a socket
+ * @sock: connected socket on which to perform the handshake
+ * @done: function to call when the handshake has completed
+ * @data: token to pass back to @done
+ * @priorities: GnuTLS TLS priorities string
+ * @peerid: serial number of key containing TLS identity
+ *
+ * Return values:
+ *   %0: Handshake request enqueue; ->done will be called when complete
+ *   %-ENOENT: No user agent is available
+ *   %-ENOMEM: Memory allocation failed
+ */
+int tls_client_hello_psk(struct socket *sock,
+			 void (*done)(void *data, int status), void *data,
+			 const char *priorities, key_serial_t peerid)
+{
+	struct tlsh_sock_info *tsi;
+	int rc;
+
+	tsi = tlsh_sock_info_alloc(sock, priorities);
+	if (!tsi)
+		return -ENOMEM;
+
+	tsi->tsi_handshake_done = done;
+	tsi->tsi_handshake_data = data;
+
+	tsi->tsi_handshake_info.hi_fd_parms_reply = tlsh_genl_ch_psk_reply;
+	tsi->tsi_peerid = peerid;
+
+	rc = handshake_enqueue_sock(sock, &tsi->tsi_handshake_info);
+	if (rc)
+		tlsh_sock_info_free(tsi);
+	return rc;
+}
+EXPORT_SYMBOL(tls_client_hello_psk);
+
+static int tlsh_genl_sh_reply(struct sk_buff *msg, struct handshake_info *hsi)
+{
+	struct tlsh_sock_info *tsi = container_of(hsi, struct tlsh_sock_info,
+						  tsi_handshake_info);
+	int ret;
+
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_PROTOCOL,
+			  HANDSHAKE_PROTO_TLS_13);
+	if (ret < 0)
+		return ret;
+	ret = nla_put_u32(msg, HANDSHAKE_GENL_ATTR_HANDSHAKE_TYPE,
+			  HANDSHAKE_TYPE_SERVERHELLO);
+	if (ret < 0)
+		return ret;
+	return tlsh_genl_tls_priorities(msg, tsi);
+}
+
+/**
+ * tls_server_hello - request a server TLS handshake on a socket
+ * @sock: connected socket on which to perform the handshake
+ * @done: function to call when the handshake has completed
+ * @data: token to pass back to @done
+ * @priorities: GnuTLS TLS priorities string
+ *
+ * Return values:
+ *   %0: Handshake request enqueue; ->done will be called when complete
+ *   %-ENOENT: No user agent is available
+ *   %-ENOMEM: Memory allocation failed
+ */
+int tls_server_hello(struct socket *sock, void (*done)(void *data, int status),
+		     void *data, const char *priorities)
+{
+	struct tlsh_sock_info *tsi;
+	int rc;
+
+	tsi = tlsh_sock_info_alloc(sock, priorities);
+	if (!tsi)
+		return -ENOMEM;
+
+	tsi->tsi_handshake_done = done;
+	tsi->tsi_handshake_data = data;
+
+	tsi->tsi_handshake_info.hi_fd_parms_reply = tlsh_genl_sh_reply;
+
+	rc = handshake_enqueue_sock(sock, &tsi->tsi_handshake_info);
+	if (rc)
+		tlsh_sock_info_free(tsi);
+	return rc;
+}
+EXPORT_SYMBOL(tls_server_hello);

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Jakub Kicinski]

On Thu, 26 Jan 2023 11:02:22 -0500 Chuck Lever wrote:
> I've designed a way to pass a connected kernel socket endpoint to
> user space using the traditional listen/accept mechanism. accept(2)
> gives us a well-worn building block that can materialize a connected
> socket endpoint as a file descriptor in a specific user space
> process. Like any open socket descriptor, the accepted FD can then
> be passed to a library such as GnuTLS to perform a TLS handshake.

I can't bring myself to like the new socket family layer.
I'd like a second opinion on that, if anyone within netdev
is willing to share..

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Chuck Lever III]

> On Jan 28, 2023, at 3:32 AM, Jakub Kicinski <kuba@kernel.org> wrote:
> 
> On Thu, 26 Jan 2023 11:02:22 -0500 Chuck Lever wrote:
>> I've designed a way to pass a connected kernel socket endpoint to
>> user space using the traditional listen/accept mechanism. accept(2)
>> gives us a well-worn building block that can materialize a connected
>> socket endpoint as a file descriptor in a specific user space
>> process. Like any open socket descriptor, the accepted FD can then
>> be passed to a library such as GnuTLS to perform a TLS handshake.
> 
> I can't bring myself to like the new socket family layer.

poll/listen/accept is the simplest and most natural way of
materializing a socket endpoint in a process that I can think
of. It's a well-understood building block. What specifically
is troubling you about it?


> I'd like a second opinion on that, if anyone within netdev
> is willing to share..

Hopefully that opinion comes with an alternative way of getting
a connected kernel socket endpoint up to user space without
race issues.

We need to make some progress on this. If you don't have a
technical objection, I think we should go with this with the
idea that eventually something more palatable will come along
to replace it.


--
Chuck Lever

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Stephen Hemminger]

On Sat, 28 Jan 2023 00:32:12 -0800
Jakub Kicinski <kuba@kernel.org> wrote:

> On Thu, 26 Jan 2023 11:02:22 -0500 Chuck Lever wrote:
> > I've designed a way to pass a connected kernel socket endpoint to
> > user space using the traditional listen/accept mechanism. accept(2)
> > gives us a well-worn building block that can materialize a connected
> > socket endpoint as a file descriptor in a specific user space
> > process. Like any open socket descriptor, the accepted FD can then
> > be passed to a library such as GnuTLS to perform a TLS handshake.  
> 
> I can't bring myself to like the new socket family layer.
> I'd like a second opinion on that, if anyone within netdev
> is willing to share..

Why not just pass fd's with Unix Domain socket?
The application is going to need to be changed to handle new AF already.

Also, expanding the address families has security impacts as well.
Either all the container and LSM's need to deny your new AF or they need
to be taught to validate whether this a valid operation.

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Hannes Reinecke]

On 1/28/23 09:32, Jakub Kicinski wrote:
> On Thu, 26 Jan 2023 11:02:22 -0500 Chuck Lever wrote:
>> I've designed a way to pass a connected kernel socket endpoint to
>> user space using the traditional listen/accept mechanism. accept(2)
>> gives us a well-worn building block that can materialize a connected
>> socket endpoint as a file descriptor in a specific user space
>> process. Like any open socket descriptor, the accepted FD can then
>> be passed to a library such as GnuTLS to perform a TLS handshake.
> 
> I can't bring myself to like the new socket family layer.
> I'd like a second opinion on that, if anyone within netdev
> is willing to share..

I am not particularly fond of that, either, but the alternative of using 
netlink doesn't make it any better
You can't pass the fd/socket directly via netlink messages, you can only 
pass the (open!) fd number with the message.
The fd itself _needs_ be be part of the process context of the 
application by the time the application processes that message.
Consequently:
- I can't see how an application can _reject_ the message; the fd needs 
to be present in the fd table even before the message is processed, 
rendering any decision by the application pointless (and I would _so_ 
love to be proven wrong on this point)
- It's slightly tricky to handle processes which go away prior to 
handling the message; I _think_ the process cleanup code will close the 
fd, but I guess it also depends on how and when the fd is stored in the 
process context.

If someone can point me to a solution for these points I would vastly 
prefer to move to netlink. But with these issues in place I'm not sure 
if netlink doesn't cause more issues than it solves.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Chuck Lever III]

> On Jan 28, 2023, at 12:40 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> On Sat, 28 Jan 2023 00:32:12 -0800
> Jakub Kicinski <kuba@kernel.org> wrote:
> 
>> On Thu, 26 Jan 2023 11:02:22 -0500 Chuck Lever wrote:
>>> I've designed a way to pass a connected kernel socket endpoint to
>>> user space using the traditional listen/accept mechanism. accept(2)
>>> gives us a well-worn building block that can materialize a connected
>>> socket endpoint as a file descriptor in a specific user space
>>> process. Like any open socket descriptor, the accepted FD can then
>>> be passed to a library such as GnuTLS to perform a TLS handshake.  
>> 
>> I can't bring myself to like the new socket family layer.
>> I'd like a second opinion on that, if anyone within netdev
>> is willing to share..
> 
> Why not just pass fd's with Unix Domain socket?

Or a pipe. We do need a queue for handshake requests, and a
pipe provides a reliable mechanism to ensure that handshake
requests are not dropped.

The question I have is: would the application then just start
using that fd number as if it had been opened or accepted? Is
reading the fd from the pipe actually an implicit accept(2)?


> The application is going to need to be changed to handle new AF already.

The only application that has to deal with PF_HANDSHAKE is the
user space daemon that performs the handshakes, an example of
which we provide here:

  https://github.com/oracle/ktls-utils

This is for handling handshakes on behalf of kernel TLS consumers.
The new family would not be used by any existing application in
user space.


> Also, expanding the address families has security impacts as well.
> Either all the container and LSM's need to deny your new AF or they need
> to be taught to validate whether this a valid operation.

It wasn't clear to me yesterday whether Jakub's objection was to
the listen/poll/accept part of this contraption, or whether he
was uncomfortable specifically with the addition of PF_HANDSHAKE.
I can certainly see that a new socket family is unwieldy from a
security perspective.

However, what if listen/poll/accept was used with an existing
address family, maybe AF_INET, with either a special bind address,
with a socket option set, or perhaps a new netlink operation can
inform the kernel that the listener is specifically for transport
layer handshake requests...?

A socket that comes from an accept(2) on a PF_HANDSHAKE listener
is also a PF_HANDSHAKE socket, though it behaves in every other
aspect like an AF_INET/AF_INET6 socket. So, using an AF_INET/6
listener instead might be nicer overall.


--
Chuck Lever

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Marcel Holtmann]

Hi Hannes,

>>> I've designed a way to pass a connected kernel socket endpoint to
>>> user space using the traditional listen/accept mechanism. accept(2)
>>> gives us a well-worn building block that can materialize a connected
>>> socket endpoint as a file descriptor in a specific user space
>>> process. Like any open socket descriptor, the accepted FD can then
>>> be passed to a library such as GnuTLS to perform a TLS handshake.
>> I can't bring myself to like the new socket family layer.
>> I'd like a second opinion on that, if anyone within netdev
>> is willing to share..
> 
> I am not particularly fond of that, either, but the alternative of using netlink doesn't make it any better
> You can't pass the fd/socket directly via netlink messages, you can only pass the (open!) fd number with the message.
> The fd itself _needs_ be be part of the process context of the application by the time the application processes that message.
> Consequently:
> - I can't see how an application can _reject_ the message; the fd needs to be present in the fd table even before the message is processed, rendering any decision by the application pointless (and I would _so_ love to be proven wrong on this point)
> - It's slightly tricky to handle processes which go away prior to handling the message; I _think_ the process cleanup code will close the fd, but I guess it also depends on how and when the fd is stored in the process context.
> 
> If someone can point me to a solution for these points I would vastly prefer to move to netlink. But with these issues in place I'm not sure if netlink doesn't cause more issues than it solves.

I think we first need to figure out the security model behind this.

For kTLS you have the TLS Handshake messages inline with the TCP
socket and thus credentials are given by the owner of that socket.
This is simple and makes a lot of sense since whoever opened that
connection has to decide to give a client certificate or accept
the server certificate (in case of session resumption also provide
the PSK).

I like to have a generic TLS Handshake interface as well since more
and more protocols will take TLS 1.3 as reference and use its
handshake protocol. What I would not do is insist on using an fd,
because that is what OpenSSL and others are just used to. The TLS
libraries need to go away from the fd as IO model and provide
appropriate APIs into the TLS Handshake (and also TLS Alert
protocol) for a “codec style” operation.

Fundamentally nothing speaks against TLS Handshake in the kernel. All
the core functionality is already present. All KPP, HKDF and even the
certifiacate handling is present. In a simplified view, you just need
To give the kernel a keyctl keyring that has the CA certs to verify
and provide the keyring with either client or server certificate to
use.

On a TCP socket for example you could do this:

	setsockopt(fd, SOL_TCP, TCP_ULP, “tls+hs", ..);

	tls_client.cert_id = key_id_cert;
	tls_client.ca_id = key_id_ca;

	setsockopt(fd, SOL_TLS, TLS_CLIENT, &tls_client, ..);

Failures or errors would be reported out via socket errors or SCM.
And you need some extra options to select cipher ranges or limit to
TLS 1.3 only etc.

But overall it would make using TCP+TLS really simple. The complicated
part is providing the key ring. Then again, the CA key ring could be
inherited from systemd or some basic component setting it up and
sealing it.

For other protocols or usages the input would be similar. It should
be rather straight forward to provide key ring identifiers as mount
option or via an ioctl.

This however needs to overcome the fear of putting the TLS Handshake
into the kernel. I can understand anybody thinking that it is not a
good idea and with TLS 1.2 and before it is a bit convoluted and
error prone. However starting with TLS 1.3 things are a lot simpler
and streamlined. There are few oddities where TLS 1.3 has to look
like TLS 1.2 on the wire, but that mainly only affects the TLS
record protocol and kTLS does that today already anyway.

For reference ELL (git.kernel.org/pub/scm/libs/ell/ell.git) has a
TLS implementation that utilizes AF_ALG and keyctl for all the
basic crypto needs. Certificates and certificate operations are
purely done via keyctl and that works nicely. If KPP would finally
get an usersapce interface, even shared secret derivation would go
via kernel crypto.

The code is currently TLS 1.2 and earlier, but I have code for
TLS 1.3 and also code for utilizing kTLS. It needs a bit more
cleanup, but then I am happy to publish it. The modified code
for TLS 1.3 support has TLS Handshake+Alert separated from TLS
Record protocol and doesn’t even rely on an fd to operate. This
comes from the requirement that TLS for WiFi Enterprise (or in
the future QUIC) doesn’t have a fd either.

Long story short, who is suppose to run the TLS Handshake if
we push it to userspace. There will be never a generic daemon
that handles all handshakes since they are all application
specific. No daemon can run the TLS Handshake on behalf of
Chrome browser for example. This leads me to AF_HANDSHAKE
is not a good idea.

One nice thing we found with using keyctl for WiFi Enterprise
is that we can have certificates that are backed by the TPM.
Doing that via keyctl was a lot simpler than dealing with the
different oddities of SSL engines or different variations of
crypto libraries. The unification by the kernel is really
nice. I have to re-read how much EFI can provide securely
hardware backed keys, but for everybody working in early
userspace or initramfs it is nice to be able to utilize
this without having to drag in megabytes of TLS library.

Regards

Marcel

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Chuck Lever III]

Hello Marcel-


> On Jan 30, 2023, at 8:44 AM, Marcel Holtmann <marcel@holtmann.org> wrote:
> 
> Hi Hannes,
> 
>>>> I've designed a way to pass a connected kernel socket endpoint to
>>>> user space using the traditional listen/accept mechanism. accept(2)
>>>> gives us a well-worn building block that can materialize a connected
>>>> socket endpoint as a file descriptor in a specific user space
>>>> process. Like any open socket descriptor, the accepted FD can then
>>>> be passed to a library such as GnuTLS to perform a TLS handshake.
>>> I can't bring myself to like the new socket family layer.
>>> I'd like a second opinion on that, if anyone within netdev
>>> is willing to share..
>> 
>> I am not particularly fond of that, either, but the alternative of using netlink doesn't make it any better
>> You can't pass the fd/socket directly via netlink messages, you can only pass the (open!) fd number with the message.
>> The fd itself _needs_ be be part of the process context of the application by the time the application processes that message.
>> Consequently:
>> - I can't see how an application can _reject_ the message; the fd needs to be present in the fd table even before the message is processed, rendering any decision by the application pointless (and I would _so_ love to be proven wrong on this point)
>> - It's slightly tricky to handle processes which go away prior to handling the message; I _think_ the process cleanup code will close the fd, but I guess it also depends on how and when the fd is stored in the process context.
>> 
>> If someone can point me to a solution for these points I would vastly prefer to move to netlink. But with these issues in place I'm not sure if netlink doesn't cause more issues than it solves.
> 
> I think we first need to figure out the security model behind this.
> 
> For kTLS you have the TLS Handshake messages inline with the TCP
> socket and thus credentials are given by the owner of that socket.
> This is simple and makes a lot of sense since whoever opened that
> connection has to decide to give a client certificate or accept
> the server certificate (in case of session resumption also provide
> the PSK).
> 
> I like to have a generic TLS Handshake interface as well since more
> and more protocols will take TLS 1.3 as reference and use its
> handshake protocol. What I would not do is insist on using an fd,
> because that is what OpenSSL and others are just used to. The TLS
> libraries need to go away from the fd as IO model and provide
> appropriate APIs into the TLS Handshake (and also TLS Alert
> protocol) for a “codec style” operation.

Note that we are not attempting to replace handshakes for user
applications -- the purpose of AF_HANDSHAKE is for dealing with
the needs of /kernel/ consumers of transport layer security.


> Fundamentally nothing speaks against TLS Handshake in the kernel. All
> the core functionality is already present. All KPP, HKDF and even the
> certifiacate handling is present. In a simplified view, you just need
> To give the kernel a keyctl keyring that has the CA certs to verify
> and provide the keyring with either client or server certificate to
> use.

I agree strongly that the handshake logic can and should reside
in the kernel. The kernel security community and many file
system developers have told us repeatedly that an in-kernel
design is a non-starter, thus we are going with an upcall for
now.


> On a TCP socket for example you could do this:
> 
> 	setsockopt(fd, SOL_TCP, TCP_ULP, “tls+hs", ..);
> 
> 	tls_client.cert_id = key_id_cert;
> 	tls_client.ca_id = key_id_ca;
> 
> 	setsockopt(fd, SOL_TLS, TLS_CLIENT, &tls_client, ..);
> 
> Failures or errors would be reported out via socket errors or SCM.
> And you need some extra options to select cipher ranges or limit to
> TLS 1.3 only etc.
> 
> But overall it would make using TCP+TLS really simple. The complicated
> part is providing the key ring. Then again, the CA key ring could be
> inherited from systemd or some basic component setting it up and
> sealing it.
> 
> For other protocols or usages the input would be similar. It should
> be rather straight forward to provide key ring identifiers as mount
> option or via an ioctl.

That is essentially what I'm attempting to do here: provide a
stable API that kernel TLS consumers can use that hides the
mechanism of the handshake. It can be an upcall, it can be done
in the kernel, it can use a struct socket * endpoint or a file
descriptor. But the API that kernel consumers see does not
change, and we can build something very much like what you
described behind that API, or replace it all with chewing gum,
bailing wire, and band-aids. The consumers should not care, as
long as it works.


> This however needs to overcome the fear of putting the TLS Handshake
> into the kernel. I can understand anybody thinking that it is not a
> good idea and with TLS 1.2 and before it is a bit convoluted and
> error prone. However starting with TLS 1.3 things are a lot simpler
> and streamlined. There are few oddities where TLS 1.3 has to look
> like TLS 1.2 on the wire, but that mainly only affects the TLS
> record protocol and kTLS does that today already anyway.

We've been looking at the mbed-based in-kernel handshake that
Tempesta did, so it's not a stretch to imagine what such a
thing would look like.

Tempesta, however, is indeed looking at a handshake facility
that can be used by user applications, it should be noted.

The point of having a fixed in-kernel handshake API is that,
if something like Tempesta is ever adopted, in-kernel TLS
consumers can be switched over to that with little fuss.

Also note: the current cohort of kernel consumers do not need
or want anything older than TLSv1.3. Anything that has to
deal with user space applications likely needs compatibility
with TLSv1.2 at least.

The kernel consumers we want to support include:

 - RPC-with-TLS (RFC 9289)
 - NVMe over TCP with TLS (part of NVMe/TCP rev 1.0a)
 - SMB on QUIC (no public specification found)

All of these mandate at least TLSv1.3.


> For reference ELL (git.kernel.org/pub/scm/libs/ell/ell.git) has a
> TLS implementation that utilizes AF_ALG and keyctl for all the
> basic crypto needs. Certificates and certificate operations are
> purely done via keyctl and that works nicely. If KPP would finally
> get an usersapce interface, even shared secret derivation would go
> via kernel crypto.
> 
> The code is currently TLS 1.2 and earlier, but I have code for
> TLS 1.3 and also code for utilizing kTLS. It needs a bit more
> cleanup, but then I am happy to publish it. The modified code
> for TLS 1.3 support has TLS Handshake+Alert separated from TLS
> Record protocol and doesn’t even rely on an fd to operate. This
> comes from the requirement that TLS for WiFi Enterprise (or in
> the future QUIC) doesn’t have a fd either.

We have our eye on QUIC as well, for exactly those reasons.
But as I said, our use case is for kernel consumers of
transport layer security, such as SMB on QUIC support in
the kernel's cifs client.


> Long story short, who is suppose to run the TLS Handshake if
> we push it to userspace. There will be never a generic daemon
> that handles all handshakes since they are all application
> specific. No daemon can run the TLS Handshake on behalf of
> Chrome browser for example. This leads me to AF_HANDSHAKE
> is not a good idea.

In our narrow use case, user space applications will continue
to use existing libraries directly for doing the handshaking
that they need. I'm not advocating for using AF_HANDSHAKE to
replace existing user space TLS library users, because I
agree that user space is going to have a much broader set of
use cases that handshake implementations have to deal with.

For kernel consumers, we have provided an example user space
daemon that handles handshake requests via a direct call to
the GnuTLS library. We can add as much decoration on that
mechanism as kernel consumers require...

   https://github.com/oracle/ktls-utils

We expect there to be one of these daemons running in each
network namespace. The daemon reads authentication material
from local files or gets it from kernel keyrings.


> One nice thing we found with using keyctl for WiFi Enterprise
> is that we can have certificates that are backed by the TPM.

TPM support is one reason we use kernel keyrings in our design,
though for us this is currently untested.


> Doing that via keyctl was a lot simpler than dealing with the
> different oddities of SSL engines or different variations of
> crypto libraries. The unification by the kernel is really
> nice. I have to re-read how much EFI can provide securely
> hardware backed keys, but for everybody working in early
> userspace or initramfs it is nice to be able to utilize
> this without having to drag in megabytes of TLS library.

Performing handshakes during early boot is one very important
use case that doing an upcall does not handle without a lot
of additional complexity. This is one reason we'd like to see
an in-kernel handshake implementation.

--
Chuck Lever

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Jakub Kicinski]

On Sat, 28 Jan 2023 14:06:49 +0000 Chuck Lever III wrote:
> > On Jan 28, 2023, at 3:32 AM, Jakub Kicinski <kuba@kernel.org> wrote:
> > On Thu, 26 Jan 2023 11:02:22 -0500 Chuck Lever wrote:  
> >> I've designed a way to pass a connected kernel socket endpoint to
> >> user space using the traditional listen/accept mechanism. accept(2)
> >> gives us a well-worn building block that can materialize a connected
> >> socket endpoint as a file descriptor in a specific user space
> >> process. Like any open socket descriptor, the accepted FD can then
> >> be passed to a library such as GnuTLS to perform a TLS handshake.  
> > 
> > I can't bring myself to like the new socket family layer.  
> 
> poll/listen/accept is the simplest and most natural way of
> materializing a socket endpoint in a process that I can think
> of. It's a well-understood building block. What specifically
> is troubling you about it?

poll/listen/accept yes, but that's not the entire socket interface. 
Our overall experience with the TCP ULPs is rather painful, proxying
all the other callbacks here may add another dimension.

Also I have a fear (perhaps unjustified) of reusing constructs which are
cornerstones of the networking stack and treating them as abstractions.

> > I'd like a second opinion on that, if anyone within netdev
> > is willing to share..  
> 
> Hopefully that opinion comes with an alternative way of getting
> a connected kernel socket endpoint up to user space without
> race issues.

If the user application decides the fd, wouldn't that solve the problem
in netlink?

  kernel                          user space

   notification     ---------->
 (new connection awaits)

                    <----------
                                  request (target fd=100)

                    ---------->
   reply
 (fd 100 is installed;
  extra params)

> We need to make some progress on this. If you don't have a
> technical objection, I think we should go with this with the
> idea that eventually something more palatable will come along
> to replace it.

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Hannes Reinecke]

On 1/30/23 14:44, Marcel Holtmann wrote:
> Hi Hannes,
> 
>>>> I've designed a way to pass a connected kernel socket endpoint to
>>>> user space using the traditional listen/accept mechanism. accept(2)
>>>> gives us a well-worn building block that can materialize a connected
>>>> socket endpoint as a file descriptor in a specific user space
>>>> process. Like any open socket descriptor, the accepted FD can then
>>>> be passed to a library such as GnuTLS to perform a TLS handshake.
>>> I can't bring myself to like the new socket family layer.
>>> I'd like a second opinion on that, if anyone within netdev
>>> is willing to share..
>>
>> I am not particularly fond of that, either, but the alternative of using
>> netlink doesn't make it any better
>> You can't pass the fd/socket directly via netlink messages, you can only
>> pass the (open!) fd number with the message.
>> The fd itself _needs_ be be part of the process context of the application
>> by the time the application processes that message.
>> Consequently:
>> - I can't see how an application can _reject_ the message; the fd needs to
>> be present in the fd table even before the message is processed, rendering
>> any decision by the application pointless (and I would _so_ love to be proven
>> wrong on this point)
>> - It's slightly tricky to handle processes which go away prior to handling
>> the message; I _think_ the process cleanup code will close the fd, but I guess
>> it also depends on how and when the fd is stored in the process context.
>>
>> If someone can point me to a solution for these points I would vastly prefer
>> to move to netlink. But with these issues in place I'm not sure if netlink
>> doesn't cause more issues than it solves.
> 
> I think we first need to figure out the security model behind this.
> 
> For kTLS you have the TLS Handshake messages inline with the TCP
> socket and thus credentials are given by the owner of that socket.
> This is simple and makes a lot of sense since whoever opened that
> connection has to decide to give a client certificate or accept
> the server certificate (in case of session resumption also provide
> the PSK).
> 
> I like to have a generic TLS Handshake interface as well since more
> and more protocols will take TLS 1.3 as reference and use its
> handshake protocol. What I would not do is insist on using an fd,
> because that is what OpenSSL and others are just used to. The TLS
> libraries need to go away from the fd as IO model and provide
> appropriate APIs into the TLS Handshake (and also TLS Alert
> protocol) for a “codec style” operation.
> 
That's something we have discussed, too.
We could forward the TLS handshake frames via netlink, thus saving us 
the headache of passing an entire socket to userspace.
However, that would require a major infrastructure work on the 
libraries, and my experience with fixing/updating things in gnutls have 
not been stellar. So I didn't pursue this route.

> Fundamentally nothing speaks against TLS Handshake in the kernel. All
> the core functionality is already present. All KPP, HKDF and even the
> certifiacate handling is present. In a simplified view, you just need
> To give the kernel a keyctl keyring that has the CA certs to verify
> and provide the keyring with either client or server certificate to
> use.
> 
> On a TCP socket for example you could do this:
> 
> 	setsockopt(fd, SOL_TCP, TCP_ULP, “tls+hs", ..);
> 
> 	tls_client.cert_id = key_id_cert;
> 	tls_client.ca_id = key_id_ca;
> 
> 	setsockopt(fd, SOL_TLS, TLS_CLIENT, &tls_client, ..);
> 
> Failures or errors would be reported out via socket errors or SCM.
> And you need some extra options to select cipher ranges or limit to
> TLS 1.3 only etc.
> 
Fundamentally you are correct. But these are security relevant areas, 
and any implementation we do will have to be vetted by some security 
people. _And_ will have to be maintained by someone well-versed in 
security, too, lest we have a security breach in the kernel.
And that person will certainly not be me, so I haven't attempt that route.

> But overall it would make using TCP+TLS really simple. The complicated
> part is providing the key ring. Then again, the CA key ring could be
> inherited from systemd or some basic component setting it up and
> sealing it.
> 
I don't think that's a major concern. The good thing with the keyring is 
that it can be populated externally, ie one can have a daemon to fetch 
the certificate and stuff it in the keyring. request_key() and all that ...

> For other protocols or usages the input would be similar. It should
> be rather straight forward to provide key ring identifiers as mount
> option or via an ioctl.
> 
> This however needs to overcome the fear of putting the TLS Handshake
> into the kernel. I can understand anybody thinking that it is not a
> good idea and with TLS 1.2 and before it is a bit convoluted and
> error prone. However starting with TLS 1.3 things are a lot simpler
> and streamlined. There are few oddities where TLS 1.3 has to look
> like TLS 1.2 on the wire, but that mainly only affects the TLS
> record protocol and kTLS does that today already anyway.
> 
See above. It's not so much 'fear' as rather the logistics of it. 
Getting hold of a TLS library is reasonably easy (Chuck had another 
example ready), but massaging it for inclusion into the kernel is quite 
some effort.
You might even succeed in convincing the powers that be to include it 
into the kernel.
But then you are stuck with having to find a capable maintainer, who is 
willing _and qualified_ to take the work and answer awkward questions.
And take the heat when that code introduced a security breach in the 
linux kernel.
Which excluded essentially everybody who had been working on this 
project; we are capable enough engineers in the network and storage 
space, but deep security issues ... not so much.

> For reference ELL (git.kernel.org/pub/scm/libs/ell/ell.git) has a
> TLS implementation that utilizes AF_ALG and keyctl for all the
> basic crypto needs. Certificates and certificate operations are
> purely done via keyctl and that works nicely. If KPP would finally
> get an usersapce interface, even shared secret derivation would go
> via kernel crypto.
> 
> The code is currently TLS 1.2 and earlier, but I have code for
> TLS 1.3 and also code for utilizing kTLS. It needs a bit more
> cleanup, but then I am happy to publish it. The modified code
> for TLS 1.3 support has TLS Handshake+Alert separated from TLS
> Record protocol and doesn’t even rely on an fd to operate. This
> comes from the requirement that TLS for WiFi Enterprise (or in
> the future QUIC) doesn’t have a fd either.
> 
If you have code to update it to 1.3 I would be very willing to look at 
it; the main reason why with went with gnutls was that no-one of us was 
eager (not hat the knowledge) to really delve into TLS and do fancy things.

And that was the other thing; we found quite some TLS implementations, 
but nearly all of the said '1.3 support to come' ...

> Long story short, who is suppose to run the TLS Handshake if
> we push it to userspace. There will be never a generic daemon
> that handles all handshakes since they are all application
> specific. No daemon can run the TLS Handshake on behalf of
> Chrome browser for example. This leads me to AF_HANDSHAKE
> is not a good idea.
> 
> One nice thing we found with using keyctl for WiFi Enterprise
> is that we can have certificates that are backed by the TPM.
> Doing that via keyctl was a lot simpler than dealing with the
> different oddities of SSL engines or different variations of
> crypto libraries. The unification by the kernel is really
> nice. I have to re-read how much EFI can provide securely
> hardware backed keys, but for everybody working in early
> userspace or initramfs it is nice to be able to utilize
> this without having to drag in megabytes of TLS library.
> 
We don't deny that having TLS handshake in the kernel would be a good 
thing. It's just the hurdles to _get_ there are quite high, and we 
thought that the userspace daemon would be an easier route.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Marcel Holtmann]

Hi Hannes,

>>>>> I've designed a way to pass a connected kernel socket endpoint to
>>>>> user space using the traditional listen/accept mechanism. accept(2)
>>>>> gives us a well-worn building block that can materialize a connected
>>>>> socket endpoint as a file descriptor in a specific user space
>>>>> process. Like any open socket descriptor, the accepted FD can then
>>>>> be passed to a library such as GnuTLS to perform a TLS handshake.
>>>> I can't bring myself to like the new socket family layer.
>>>> I'd like a second opinion on that, if anyone within netdev
>>>> is willing to share..
>>> 
>>> I am not particularly fond of that, either, but the alternative of using
>>> netlink doesn't make it any better
>>> You can't pass the fd/socket directly via netlink messages, you can only
>>> pass the (open!) fd number with the message.
>>> The fd itself _needs_ be be part of the process context of the application
>>> by the time the application processes that message.
>>> Consequently:
>>> - I can't see how an application can _reject_ the message; the fd needs to
>>> be present in the fd table even before the message is processed, rendering
>>> any decision by the application pointless (and I would _so_ love to be proven
>>> wrong on this point)
>>> - It's slightly tricky to handle processes which go away prior to handling
>>> the message; I _think_ the process cleanup code will close the fd, but I guess
>>> it also depends on how and when the fd is stored in the process context.
>>> 
>>> If someone can point me to a solution for these points I would vastly prefer
>>> to move to netlink. But with these issues in place I'm not sure if netlink
>>> doesn't cause more issues than it solves.
>> I think we first need to figure out the security model behind this.
>> For kTLS you have the TLS Handshake messages inline with the TCP
>> socket and thus credentials are given by the owner of that socket.
>> This is simple and makes a lot of sense since whoever opened that
>> connection has to decide to give a client certificate or accept
>> the server certificate (in case of session resumption also provide
>> the PSK).
>> I like to have a generic TLS Handshake interface as well since more
>> and more protocols will take TLS 1.3 as reference and use its
>> handshake protocol. What I would not do is insist on using an fd,
>> because that is what OpenSSL and others are just used to. The TLS
>> libraries need to go away from the fd as IO model and provide
>> appropriate APIs into the TLS Handshake (and also TLS Alert
>> protocol) for a “codec style” operation.
> That's something we have discussed, too.
> We could forward the TLS handshake frames via netlink, thus saving us the headache of passing an entire socket to userspace.
> However, that would require a major infrastructure work on the libraries, and my experience with fixing/updating things in gnutls have not been stellar. So I didn't pursue this route.

I know, utilizing existing TLS libraries is a pain if you don’t do
exactly what they had in mind. I started looking at QUIC a while
back and quickly realized, I have to start looking at TLS 1.3 first.

My past experience with GnuTLS and OpenSSL have been bad and that is
why iwd (our WiFi daemon) has its own TLS implementation utilizing
AF_ALG and keyctl.

>> Fundamentally nothing speaks against TLS Handshake in the kernel. All
>> the core functionality is already present. All KPP, HKDF and even the
>> certifiacate handling is present. In a simplified view, you just need
>> To give the kernel a keyctl keyring that has the CA certs to verify
>> and provide the keyring with either client or server certificate to
>> use.
>> On a TCP socket for example you could do this:
>> 	setsockopt(fd, SOL_TCP, TCP_ULP, “tls+hs", ..);
>> 	tls_client.cert_id = key_id_cert;
>> 	tls_client.ca_id = key_id_ca;
>> 	setsockopt(fd, SOL_TLS, TLS_CLIENT, &tls_client, ..);
>> Failures or errors would be reported out via socket errors or SCM.
>> And you need some extra options to select cipher ranges or limit to
>> TLS 1.3 only etc.
> Fundamentally you are correct. But these are security relevant areas, and any implementation we do will have to be vetted by some security people. _And_ will have to be maintained by someone well-versed in security, too, lest we have a security breach in the kernel.
> And that person will certainly not be me, so I haven't attempt that route.

While that might have been true in the past and with TLS 1.2 and earlier,
I am not sure that is all true today.

Lets assume we start with TLS 1.3 and don’t have backwards compatibility
with TLS 1.2 and earlier. And for now we don’t worry about Middleboxes
compatibility mode since you don’t have to for all the modern protocols
that utilize just the TLS 1.3 handshake like QUIC.

Now the key derivation is just choosing 1 out of 5 ciphers and using
its associated hash algorithm to derive the keys. This is all present
functionality in the kernel and so well tested that it doesn’t worry
me at all. We also have a separate RFC with just sample data so you
can check your derivation functionality. Especially if you check it
against AEAD encrypted sample data, any mistake is fatal.

The shared key portion is just ECDHE or DHE and you either end up with
x25519 or secp256r1 and both are in the kernel. Bluetooth has been
using secp256r1 inside the kernel for many years now. We all know how
to handle and verify public keys from secp256r1 and neat part is that
it would be also offloaded to hardware if needed. So the private key
doesn’t need to stay even in kernel memory.

So dealing with generating your key material for your cipher is really
simple and similar things have been done for Bluetooth for a long
time now. And it looks like NVMe is also utilizing KPP as of today.

The tricky part is the authentication portion of TLS utilizing
certificates. That part is complicated, but then again, we already
decided the kernel needs to handle certificates for various places
and you have to assume that it is fairly secure.

Now, you need to secure the handshake protocol like any other protocol
and the only difference is that it will lead to key material and
does authentication with certificates. All of it, the kernel already
does in one form or another.

The TLS 1.3 spec is also really nicely written and explicit in
error behavior in case of attempts to attack the protocol. While
implementing my TLS 1.3 only prototype I have been positively
surprised on how clean it is. I personally think they went over
board with the key verification, but so be it.

Once I have cleaned up my TLS 1.3 prototype, I am happy to take
a stab at a kernel version.

>> But overall it would make using TCP+TLS really simple. The complicated
>> part is providing the key ring. Then again, the CA key ring could be
>> inherited from systemd or some basic component setting it up and
>> sealing it.
> I don't think that's a major concern. The good thing with the keyring is that it can be populated externally, ie one can have a daemon to fetch the certificate and stuff it in the keyring. request_key() and all that ...

It is just painful for the simple reason that there is no real
standard around CA certificates and where to place them. Every
distro is kinda doing it their way and you expect your TLS
library to do the magic.

I like to see systemd create a keyring of the CA certs, seal it
and then provide it do every process/service it launches. And
for non systemd distros they need to find a way to actually
provide that one keyring that can be used as master for all the
CA certs.

We have not bothered with that yet since for WiFi, you always
have a client cert derived from the CA of the server. So you
give a CA cert and a client cert when you connect to WiFi
Enterprise systems.

>> For other protocols or usages the input would be similar. It should
>> be rather straight forward to provide key ring identifiers as mount
>> option or via an ioctl.
>> This however needs to overcome the fear of putting the TLS Handshake
>> into the kernel. I can understand anybody thinking that it is not a
>> good idea and with TLS 1.2 and before it is a bit convoluted and
>> error prone. However starting with TLS 1.3 things are a lot simpler
>> and streamlined. There are few oddities where TLS 1.3 has to look
>> like TLS 1.2 on the wire, but that mainly only affects the TLS
>> record protocol and kTLS does that today already anyway.
> See above. It's not so much 'fear' as rather the logistics of it. Getting hold of a TLS library is reasonably easy (Chuck had another example ready), but massaging it for inclusion into the kernel is quite some effort.
> You might even succeed in convincing the powers that be to include it into the kernel.
> But then you are stuck with having to find a capable maintainer, who is willing _and qualified_ to take the work and answer awkward questions.
> And take the heat when that code introduced a security breach in the linux kernel.
> Which excluded essentially everybody who had been working on this project; we are capable enough engineers in the network and storage space, but deep security issues ... not so much.

Having looked at various TLS libraries during the past few months,
I would not even recommend taking any of them. This needs to be
written from scratch. Some of them are just license wise a problem
others are just too much legacy for TLS 1.3 support.

I am happy to give this stab and see how badly I would fail ;)

But as stated above, I am surprised on how good TLS 1.3 spec is
when it comes to ensuring good and secure implementations. The
thing can be really easily unit testes to death. I think people
underestimate the huge effort from the guys at IETF to make
this simple and more secure.

Some of it comes from the fact that Middleboxes have been doing
stupid things over the years, but it resulted in an even more
encrypted and secure Internet along the way.

I am also wondering if there is a use case for offloading the
TLS Handshake to some crypto accelerator. Especially when it
comes to TLS 1.3 and server role, you really don’t have much
of a handshake. You get all required information in the
ClientHello message and if you like the public key, you can
produce almost all key material in one go.

The reality is that even the shared secret is not required
anymore once you derived your handshake secret.

And WiFi FullMac cards have been offloading EAP-TLS for a while
now.

>> For reference ELL (git.kernel.org/pub/scm/libs/ell/ell.git) has a
>> TLS implementation that utilizes AF_ALG and keyctl for all the
>> basic crypto needs. Certificates and certificate operations are
>> purely done via keyctl and that works nicely. If KPP would finally
>> get an usersapce interface, even shared secret derivation would go
>> via kernel crypto.
>> The code is currently TLS 1.2 and earlier, but I have code for
>> TLS 1.3 and also code for utilizing kTLS. It needs a bit more
>> cleanup, but then I am happy to publish it. The modified code
>> for TLS 1.3 support has TLS Handshake+Alert separated from TLS
>> Record protocol and doesn’t even rely on an fd to operate. This
>> comes from the requirement that TLS for WiFi Enterprise (or in
>> the future QUIC) doesn’t have a fd either.
> If you have code to update it to 1.3 I would be very willing to look at it; the main reason why with went with gnutls was that no-one of us was eager (not hat the knowledge) to really delve into TLS and do fancy things.
> 
> And that was the other thing; we found quite some TLS implementations, but nearly all of the said '1.3 support to come' ...

True to that. I think even OpenSSL started an effort to have a
QUIC specific API now.

The problem that I found is that TLS Handshake, TLS Alert and
TLS Record protocol are not cleanly separated. They are mixed
together.

For example if I want to use kTLS, I mostly just have to deal
with TLS Handshake portion. QUIC was specific and just uses
TLS Handshake and TLS Alert are converted to QUIC errors.

>> Long story short, who is suppose to run the TLS Handshake if
>> we push it to userspace. There will be never a generic daemon
>> that handles all handshakes since they are all application
>> specific. No daemon can run the TLS Handshake on behalf of
>> Chrome browser for example. This leads me to AF_HANDSHAKE
>> is not a good idea.
>> One nice thing we found with using keyctl for WiFi Enterprise
>> is that we can have certificates that are backed by the TPM.
>> Doing that via keyctl was a lot simpler than dealing with the
>> different oddities of SSL engines or different variations of
>> crypto libraries. The unification by the kernel is really
>> nice. I have to re-read how much EFI can provide securely
>> hardware backed keys, but for everybody working in early
>> userspace or initramfs it is nice to be able to utilize
>> this without having to drag in megabytes of TLS library.
> We don't deny that having TLS handshake in the kernel would be a good thing. It's just the hurdles to _get_ there are quite high, and we thought that the userspace daemon would be an easier route.

My problem with doing an upcall for TLS Handshake messages
is to ensure that the right process has the correct rights
to receive and send the messages. And nobody else can
interfere with that or intercept messages without proper
right to do so.

And I am certain that Wireshark would love to get hold of
the unencrypted TLS Handshake traffic. Debugging TLS
and also QUIC transfers is hugely painful. The method of
SSLKEYLOGFILE works but it is so cumbersome and defeats
any kind of live traffic analysis. So having some DIAG
here would help a lot of developers.

Regards

Marcel

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Hannes Reinecke]

On 1/31/23 15:17, Marcel Holtmann wrote:
> Hi Hannes,
> 
[ .. ]
>>> I like to have a generic TLS Handshake interface as well since more
>>> and more protocols will take TLS 1.3 as reference and use its
>>> handshake protocol. What I would not do is insist on using an fd,
>>> because that is what OpenSSL and others are just used to. The TLS
>>> libraries need to go away from the fd as IO model and provide
>>> appropriate APIs into the TLS Handshake (and also TLS Alert
>>> protocol) for a “codec style” operation.
>> That's something we have discussed, too.
>> We could forward the TLS handshake frames via netlink, thus saving
>> us the headache of passing an entire socket to userspace.
>> However, that would require a major infrastructure work on the
>> libraries, and my experience with fixing/updating things in gnutls
>> have not been stellar. So I didn't pursue this route.
> 
> I know, utilizing existing TLS libraries is a pain if you don’t do
> exactly what they had in mind. I started looking at QUIC a while
> back and quickly realized, I have to start looking at TLS 1.3 first.
> 
> My past experience with GnuTLS and OpenSSL have been bad and that is
> why iwd (our WiFi daemon) has its own TLS implementation utilizing
> AF_ALG and keyctl.
> 
I know the feeling :-)

>>> Fundamentally nothing speaks against TLS Handshake in the kernel. All
>>> the core functionality is already present. All KPP, HKDF and even the
>>> certifiacate handling is present. In a simplified view, you just need
>>> To give the kernel a keyctl keyring that has the CA certs to verify
>>> and provide the keyring with either client or server certificate to
>>> use.
>>> On a TCP socket for example you could do this:
>>> 	setsockopt(fd, SOL_TCP, TCP_ULP, “tls+hs", ..);
>>> 	tls_client.cert_id = key_id_cert;
>>> 	tls_client.ca_id = key_id_ca;
>>> 	setsockopt(fd, SOL_TLS, TLS_CLIENT, &tls_client, ..);
>>> Failures or errors would be reported out via socket errors or SCM.
>>> And you need some extra options to select cipher ranges or limit to
>>> TLS 1.3 only etc.
>> Fundamentally you are correct. But these are security relevant areas,
>> and any implementation we do will have to be vetted by some security >> people. _And_ will have to be maintained by someone well-versed in
>> security, too, lest we have a security breach in the kernel.
>> And that person will certainly not be me, so I haven't attempt that route.
> 
> While that might have been true in the past and with TLS 1.2 and earlier,
> I am not sure that is all true today.
> 
> Lets assume we start with TLS 1.3 and don’t have backwards compatibility
> with TLS 1.2 and earlier. And for now we don’t worry about Middleboxes
> compatibility mode since you don’t have to for all the modern protocols
> that utilize just the TLS 1.3 handshake like QUIC.
> 
> Now the key derivation is just choosing 1 out of 5 ciphers and using
> its associated hash algorithm to derive the keys. This is all present
> functionality in the kernel and so well tested that it doesn’t worry
> me at all. We also have a separate RFC with just sample data so you
> can check your derivation functionality. Especially if you check it
> against AEAD encrypted sample data, any mistake is fatal.
> 
> The shared key portion is just ECDHE or DHE and you either end up with
> x25519 or secp256r1 and both are in the kernel. Bluetooth has been
> using secp256r1 inside the kernel for many years now. We all know how
> to handle and verify public keys from secp256r1 and neat part is that
> it would be also offloaded to hardware if needed. So the private key
> doesn’t need to stay even in kernel memory.
> 
ECDHE has now been stabilized, too; I needed that for NVMe 
authentication. So all's good there.

> So dealing with generating your key material for your cipher is really
> simple and similar things have been done for Bluetooth for a long
> time now. And it looks like NVMe is also utilizing KPP as of today.
> 
Yes. Guess who did that.

> The tricky part is the authentication portion of TLS utilizing
> certificates. That part is complicated, but then again, we already
> decided the kernel needs to handle certificates for various places
> and you have to assume that it is fairly secure.
> 
> Now, you need to secure the handshake protocol like any other protocol
> and the only difference is that it will lead to key material and
> does authentication with certificates. All of it, the kernel already
> does in one form or another.
> 
> The TLS 1.3 spec is also really nicely written and explicit in
> error behavior in case of attempts to attack the protocol. While
> implementing my TLS 1.3 only prototype I have been positively
> surprised on how clean it is. I personally think they went over
> board with the key verification, but so be it.
> 
> Once I have cleaned up my TLS 1.3 prototype, I am happy to take
> a stab at a kernel version.
> 
Oh, please. I would so love it to get it done properly; the TLS 
handshake has been a major worry for us.
And even if you would just add TLS1.3 support for ELL that'll be 
fantastic, as then I could give it a stab at the netlink frame handling 
interface (which shouldn't be too hard).

>>> But overall it would make using TCP+TLS really simple. The complicated
>>> part is providing the key ring. Then again, the CA key ring could be
>>> inherited from systemd or some basic component setting it up and
>>> sealing it.
>> I don't think that's a major concern. The good thing with the keyring
>> is that it can be populated externally, ie one can have a daemon to
>> fetch the certificate and stuff it in the keyring. request_key() and all that ...
> 
> It is just painful for the simple reason that there is no real
> standard around CA certificates and where to place them. Every
> distro is kinda doing it their way and you expect your TLS
> library to do the magic.
> 
> I like to see systemd create a keyring of the CA certs, seal it
> and then provide it do every process/service it launches. And
> for non systemd distros they need to find a way to actually
> provide that one keyring that can be used as master for all the
> CA certs.
> 
> We have not bothered with that yet since for WiFi, you always
> have a client cert derived from the CA of the server. So you
> give a CA cert and a client cert when you connect to WiFi
> Enterprise systems.
> 
The good thing is that NVMe is currently PSK-only, so the certificate 
bit is easy for me. Others like NFS will have to do proper X.509 cert 
handling, but I'll let them worry about that :-)

>>> For other protocols or usages the input would be similar. It should
>>> be rather straight forward to provide key ring identifiers as mount
>>> option or via an ioctl.
>>> This however needs to overcome the fear of putting the TLS Handshake
>>> into the kernel. I can understand anybody thinking that it is not a
>>> good idea and with TLS 1.2 and before it is a bit convoluted and
>>> error prone. However starting with TLS 1.3 things are a lot simpler
>>> and streamlined. There are few oddities where TLS 1.3 has to look
>>> like TLS 1.2 on the wire, but that mainly only affects the TLS
>>> record protocol and kTLS does that today already anyway.
>> See above. It's not so much 'fear' as rather the logistics of it.
>> Getting hold of a TLS library is reasonably easy (Chuck had another
>> example ready), but massaging it for inclusion into the kernel is
>> quite some effort.
>> You might even succeed in convincing the powers that be to include
>> it into the kernel.
>> But then you are stuck with having to find a capable maintainer, who
>> is willing _and qualified_ to take the work and answer awkward questions.
>> And take the heat when that code introduced a security breach in the linux kernel.
>> Which excluded essentially everybody who had been working on this project;
>> we are capable enough engineers in the network and storage space, but
>> deep security issues ... not so much.
> 
> Having looked at various TLS libraries during the past few months,
> I would not even recommend taking any of them. This needs to be
> written from scratch. Some of them are just license wise a problem
> others are just too much legacy for TLS 1.3 support.
> 
> I am happy to give this stab and see how badly I would fail ;)
> 
Cool. Count me in; I'll gladly give it a spin for NVMe-TLS where
I've all the surrounding infrastructure like keyrings and certificate 
generation ready. It really just need a TLS handshake protocol handling...

> But as stated above, I am surprised on how good TLS 1.3 spec is
> when it comes to ensuring good and secure implementations. The
> thing can be really easily unit testes to death. I think people
> underestimate the huge effort from the guys at IETF to make
> this simple and more secure.
> 

True. TLS 1.3 _is_ simple, and it might be that quite some issues
around TLS is related to older versions.

[ .. ]

>>> The code is currently TLS 1.2 and earlier, but I have code for
>>> TLS 1.3 and also code for utilizing kTLS. It needs a bit more
>>> cleanup, but then I am happy to publish it. The modified code
>>> for TLS 1.3 support has TLS Handshake+Alert separated from TLS
>>> Record protocol and doesn’t even rely on an fd to operate. This
>>> comes from the requirement that TLS for WiFi Enterprise (or in
>>> the future QUIC) doesn’t have a fd either.
>> If you have code to update it to 1.3 I would be very willing to
>> look at it; the main reason why with went with gnutls was that
>> no-one of us was eager (not hat the knowledge) to really delve
>> into TLS and do fancy things.
>>
>> And that was the other thing; we found quite some TLS implementations,
>> but nearly all of the said '1.3 support to come' ...
> 
> True to that. I think even OpenSSL started an effort to have a
> QUIC specific API now.
> 
> The problem that I found is that TLS Handshake, TLS Alert and
> TLS Record protocol are not cleanly separated. They are mixed
> together.
> 
Yep.

> For example if I want to use kTLS, I mostly just have to deal
> with TLS Handshake portion. QUIC was specific and just uses
> TLS Handshake and TLS Alert are converted to QUIC errors.
> 
Some for us. Alerts don't make sense to us as we have long-lived 
connections, so the prime reason for alerts is gone, and we have to 
re-establish the connection whenever the cipher is changed. So we will 
be converting alerts in errors, too.

>>> Long story short, who is suppose to run the TLS Handshake if
>>> we push it to userspace. There will be never a generic daemon
>>> that handles all handshakes since they are all application
>>> specific. No daemon can run the TLS Handshake on behalf of
>>> Chrome browser for example. This leads me to AF_HANDSHAKE
>>> is not a good idea.
>>> One nice thing we found with using keyctl for WiFi Enterprise
>>> is that we can have certificates that are backed by the TPM.
>>> Doing that via keyctl was a lot simpler than dealing with the
>>> different oddities of SSL engines or different variations of
>>> crypto libraries. The unification by the kernel is really
>>> nice. I have to re-read how much EFI can provide securely
>>> hardware backed keys, but for everybody working in early
>>> userspace or initramfs it is nice to be able to utilize
>>> this without having to drag in megabytes of TLS library.
 >>
>> We don't deny that having TLS handshake in the kernel would
>> be a good thing. It's just the hurdles to _get_ there are
>> quite high, and we thought that the userspace daemon would be an easier route.
> 
> My problem with doing an upcall for TLS Handshake messages
> is to ensure that the right process has the correct rights
> to receive and send the messages. And nobody else can
> interfere with that or intercept messages without proper
> right to do so.
> 
Correct. That is a concern.

> And I am certain that Wireshark would love to get hold of
> the unencrypted TLS Handshake traffic. Debugging TLS
> and also QUIC transfers is hugely painful. The method of
> SSLKEYLOGFILE works but it is so cumbersome and defeats
> any kind of live traffic analysis. So having some DIAG
> here would help a lot of developers.
> 
Oh, yes. That would be nice side-effect.

So, when can I expect the patch?
:-)

Cheers,

Hannes

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Chuck Lever III]

> On Jan 30, 2023, at 11:35 PM, Jakub Kicinski <kuba@kernel.org> wrote:
> 
> On Sat, 28 Jan 2023 14:06:49 +0000 Chuck Lever III wrote:
>>> On Jan 28, 2023, at 3:32 AM, Jakub Kicinski <kuba@kernel.org> wrote:
>>> On Thu, 26 Jan 2023 11:02:22 -0500 Chuck Lever wrote:  
>>>> I've designed a way to pass a connected kernel socket endpoint to
>>>> user space using the traditional listen/accept mechanism. accept(2)
>>>> gives us a well-worn building block that can materialize a connected
>>>> socket endpoint as a file descriptor in a specific user space
>>>> process. Like any open socket descriptor, the accepted FD can then
>>>> be passed to a library such as GnuTLS to perform a TLS handshake.  
>>> 
>>> I can't bring myself to like the new socket family layer.  
>> 
>> poll/listen/accept is the simplest and most natural way of
>> materializing a socket endpoint in a process that I can think
>> of. It's a well-understood building block. What specifically
>> is troubling you about it?
> 
> poll/listen/accept yes, but that's not the entire socket interface. 
> Our overall experience with the TCP ULPs is rather painful, proxying
> all the other callbacks here may add another dimension.

> Also I have a fear (perhaps unjustified) of reusing constructs which are
> cornerstones of the networking stack and treating them as abstractions.

OK, then I take this as a NAK for listen/poll/accept in
any form. I need some finality here because we need to
move forward.


>>> I'd like a second opinion on that, if anyone within netdev
>>> is willing to share..  
>> 
>> Hopefully that opinion comes with an alternative way of getting
>> a connected kernel socket endpoint up to user space without
>> race issues.
> 
> If the user application decides the fd, wouldn't that solve the problem
> in netlink?

David or Hannes will have to answer that because they
understand the races better than I do.

However, I will prototype "fd passing" with netlink and
ignore the races for now, just to get something to
continue the conversation.


>  kernel                          user space
> 
>   notification     ---------->
> (new connection awaits)
> 
>                    <----------
>                                  request (target fd=100)
> 
>                    ---------->
>   reply
> (fd 100 is installed;
>  extra params)

What type of notification do you prefer for this? You've
said in the past that RT signals are not appropriate. It
would be easy for user space to simply wait on nlm_recvmsg()
but I worry that netlink is not a reliable message service.

And, do you have a preferred mechanism or code sample for
installing a socket descriptor? 


--
Chuck Lever

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Jakub Kicinski]

On Tue, 31 Jan 2023 15:18:02 +0000 Chuck Lever III wrote:
> > On Jan 30, 2023, at 11:35 PM, Jakub Kicinski <kuba@kernel.org> wrote:
> > On Sat, 28 Jan 2023 14:06:49 +0000 Chuck Lever III wrote:  
> >> poll/listen/accept is the simplest and most natural way of
> >> materializing a socket endpoint in a process that I can think
> >> of. It's a well-understood building block. What specifically
> >> is troubling you about it?  
> > 
> > poll/listen/accept yes, but that's not the entire socket interface. 
> > Our overall experience with the TCP ULPs is rather painful, proxying
> > all the other callbacks here may add another dimension.  
> >
> > Also I have a fear (perhaps unjustified) of reusing constructs which are
> > cornerstones of the networking stack and treating them as abstractions.  
> 
> OK, then I take this as a NAK for listen/poll/accept in
> any form. I need some finality here because we need to
> move forward.

To be clear - if Paolo, Eric or someone else who knows the socket layer
better than I do thinks that your current implementation is good then 
I won't stand in the way. 

> >  kernel                          user space
> > 
> >   notification     ---------->
> > (new connection awaits)
> > 
> >                    <----------
> >                                  request (target fd=100)
> >   
> >                    ---------->  
> >   reply
> > (fd 100 is installed;
> >  extra params)  
> 
> What type of notification do you prefer for this? You've
> said in the past that RT signals are not appropriate. It
> would be easy for user space to simply wait on nlm_recvmsg()
> but I worry that netlink is not a reliable message service.

There are various bits and bobs in netlink which are supposed to help.
A socket which subscribed to notifications should get an error if a
delivery fails (netlink_overrun()). The kernel commonly supports a GET
request which the user space can exercise after missing notifications 
to get back in sync.

> And, do you have a preferred mechanism or code sample for
> installing a socket descriptor? 

I must admit - I don't.

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Chuck Lever III]

> On Jan 31, 2023, at 2:30 PM, Jakub Kicinski <kuba@kernel.org> wrote:
> 
>> On Tue, 31 Jan 2023 15:18:02 +0000 Chuck Lever III wrote:
>> And, do you have a preferred mechanism or code sample for
>> installing a socket descriptor? 
> 
> I must admit - I don't.

As part of responding to the handshake daemon's netlink call,
I'm thinking of doing something like:

get_unused_fd_flags(), then sock_alloc_file(), and then fd_install() 


--
Chuck Lever

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Marcel Holtmann]

Hi Chuck,

>>> And, do you have a preferred mechanism or code sample for
>>> installing a socket descriptor? 
>> 
>> I must admit - I don't.
> 
> As part of responding to the handshake daemon's netlink call,
> I'm thinking of doing something like:
> 
> get_unused_fd_flags(), then sock_alloc_file(), and then fd_install() 

can we be really careful here. fd passing over Unix sockets is already
complicated to get right on the receiver side. We had this with D-Bus
and man, can you screw up things here. The problem is really that your
fd is part of the receiving process as soon as you receive that message
and you are _required_ to take care of it. Simple things like not
setting CLOEXC is already a path to disaster. And Unix sockets have
SCM_RIGHTS and other fun stuff. I don’t remember having that for
Netlink. And don’t forget the SELinux etc. folks that might want to
have some control here.

Regards

Marcel

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Benjamin Coddington]

On 31 Jan 2023, at 14:34, Chuck Lever III wrote:

>> On Jan 31, 2023, at 2:30 PM, Jakub Kicinski <kuba@kernel.org> wrote:
>>
>>> On Tue, 31 Jan 2023 15:18:02 +0000 Chuck Lever III wrote:
>>> And, do you have a preferred mechanism or code sample for
>>> installing a socket descriptor?
>>
>> I must admit - I don't.
>
> As part of responding to the handshake daemon's netlink call,
> I'm thinking of doing something like:
>
> get_unused_fd_flags(), then sock_alloc_file(), and then fd_install()

It seems odd to me that we're not taking advantage of request_key() to do
this work.  It was designed for exactly this problem: the kernel needs
something/work (a tls handhsake) from/done in userspace.

I have a working implementation here:
https://github.com/bcodding/linux/tree/tls_keys

Perhaps there's no interest because no one likes call_usermode_helper(),
which cannot figure out what set of namespace(s) to use, but there's a
solution for that as well: keyagents can represent a running process to
satisfy request_key().
https://lore.kernel.org/linux-nfs/cover.1657624639.git.bcodding@redhat.com/

Keyagents are not required to simply pass socket fds to userspace, however
they do create a flexible way for containers to specify exactly where the
kernel should send various key requests.

I am happy to continue to explain how these approaches work, though I would
also much prefer to see us doing handshakes in-kernel.

Ben

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Marcel Holtmann]

Hi Hannes,

>>>> I like to have a generic TLS Handshake interface as well since more
>>>> and more protocols will take TLS 1.3 as reference and use its
>>>> handshake protocol. What I would not do is insist on using an fd,
>>>> because that is what OpenSSL and others are just used to. The TLS
>>>> libraries need to go away from the fd as IO model and provide
>>>> appropriate APIs into the TLS Handshake (and also TLS Alert
>>>> protocol) for a “codec style” operation.
>>> That's something we have discussed, too.
>>> We could forward the TLS handshake frames via netlink, thus saving
>>> us the headache of passing an entire socket to userspace.
>>> However, that would require a major infrastructure work on the
>>> libraries, and my experience with fixing/updating things in gnutls
>>> have not been stellar. So I didn't pursue this route.
>> I know, utilizing existing TLS libraries is a pain if you don’t do
>> exactly what they had in mind. I started looking at QUIC a while
>> back and quickly realized, I have to start looking at TLS 1.3 first.
>> My past experience with GnuTLS and OpenSSL have been bad and that is
>> why iwd (our WiFi daemon) has its own TLS implementation utilizing
>> AF_ALG and keyctl.
> I know the feeling :-)
> 
>>>> Fundamentally nothing speaks against TLS Handshake in the kernel. All
>>>> the core functionality is already present. All KPP, HKDF and even the
>>>> certifiacate handling is present. In a simplified view, you just need
>>>> To give the kernel a keyctl keyring that has the CA certs to verify
>>>> and provide the keyring with either client or server certificate to
>>>> use.
>>>> On a TCP socket for example you could do this:
>>>> 	setsockopt(fd, SOL_TCP, TCP_ULP, “tls+hs", ..);
>>>> 	tls_client.cert_id = key_id_cert;
>>>> 	tls_client.ca_id = key_id_ca;
>>>> 	setsockopt(fd, SOL_TLS, TLS_CLIENT, &tls_client, ..);
>>>> Failures or errors would be reported out via socket errors or SCM.
>>>> And you need some extra options to select cipher ranges or limit to
>>>> TLS 1.3 only etc.
>>> Fundamentally you are correct. But these are security relevant areas,
>>> and any implementation we do will have to be vetted by some security >> people. _And_ will have to be maintained by someone well-versed in
>>> security, too, lest we have a security breach in the kernel.
>>> And that person will certainly not be me, so I haven't attempt that route.
>> While that might have been true in the past and with TLS 1.2 and earlier,
>> I am not sure that is all true today.
>> Lets assume we start with TLS 1.3 and don’t have backwards compatibility
>> with TLS 1.2 and earlier. And for now we don’t worry about Middleboxes
>> compatibility mode since you don’t have to for all the modern protocols
>> that utilize just the TLS 1.3 handshake like QUIC.
>> Now the key derivation is just choosing 1 out of 5 ciphers and using
>> its associated hash algorithm to derive the keys. This is all present
>> functionality in the kernel and so well tested that it doesn’t worry
>> me at all. We also have a separate RFC with just sample data so you
>> can check your derivation functionality. Especially if you check it
>> against AEAD encrypted sample data, any mistake is fatal.
>> The shared key portion is just ECDHE or DHE and you either end up with
>> x25519 or secp256r1 and both are in the kernel. Bluetooth has been
>> using secp256r1 inside the kernel for many years now. We all know how
>> to handle and verify public keys from secp256r1 and neat part is that
>> it would be also offloaded to hardware if needed. So the private key
>> doesn’t need to stay even in kernel memory.
> ECDHE has now been stabilized, too; I needed that for NVMe authentication. So all's good there.
> 
>> So dealing with generating your key material for your cipher is really
>> simple and similar things have been done for Bluetooth for a long
>> time now. And it looks like NVMe is also utilizing KPP as of today.
> Yes. Guess who did that.
> 
>> The tricky part is the authentication portion of TLS utilizing
>> certificates. That part is complicated, but then again, we already
>> decided the kernel needs to handle certificates for various places
>> and you have to assume that it is fairly secure.
>> Now, you need to secure the handshake protocol like any other protocol
>> and the only difference is that it will lead to key material and
>> does authentication with certificates. All of it, the kernel already
>> does in one form or another.
>> The TLS 1.3 spec is also really nicely written and explicit in
>> error behavior in case of attempts to attack the protocol. While
>> implementing my TLS 1.3 only prototype I have been positively
>> surprised on how clean it is. I personally think they went over
>> board with the key verification, but so be it.
>> Once I have cleaned up my TLS 1.3 prototype, I am happy to take
>> a stab at a kernel version.
> Oh, please. I would so love it to get it done properly; the TLS handshake has been a major worry for us.
> And even if you would just add TLS1.3 support for ELL that'll be fantastic, as then I could give it a stab at the netlink frame handling interface (which shouldn't be too hard).
> 
>>>> But overall it would make using TCP+TLS really simple. The complicated
>>>> part is providing the key ring. Then again, the CA key ring could be
>>>> inherited from systemd or some basic component setting it up and
>>>> sealing it.
>>> I don't think that's a major concern. The good thing with the keyring
>>> is that it can be populated externally, ie one can have a daemon to
>>> fetch the certificate and stuff it in the keyring. request_key() and all that ...
>> It is just painful for the simple reason that there is no real
>> standard around CA certificates and where to place them. Every
>> distro is kinda doing it their way and you expect your TLS
>> library to do the magic.
>> I like to see systemd create a keyring of the CA certs, seal it
>> and then provide it do every process/service it launches. And
>> for non systemd distros they need to find a way to actually
>> provide that one keyring that can be used as master for all the
>> CA certs.
>> We have not bothered with that yet since for WiFi, you always
>> have a client cert derived from the CA of the server. So you
>> give a CA cert and a client cert when you connect to WiFi
>> Enterprise systems.
> The good thing is that NVMe is currently PSK-only, so the certificate bit is easy for me. Others like NFS will have to do proper X.509 cert handling, but I'll let them worry about that :-)

Interesting. I have not looked at PSK part yet. Do you require PSK-only
or ECDHE+PSK. From what I quickly glanced at the spec, the PSK-only is
really simple. That is the most simplest TLS Handshake I have seen and
I should give that a spin.

So NVMe agrees the PSK out-of-band? I might be able to read up on it,
but I can only keep so much specifications in memory ;)

>>>> For other protocols or usages the input would be similar. It should
>>>> be rather straight forward to provide key ring identifiers as mount
>>>> option or via an ioctl.
>>>> This however needs to overcome the fear of putting the TLS Handshake
>>>> into the kernel. I can understand anybody thinking that it is not a
>>>> good idea and with TLS 1.2 and before it is a bit convoluted and
>>>> error prone. However starting with TLS 1.3 things are a lot simpler
>>>> and streamlined. There are few oddities where TLS 1.3 has to look
>>>> like TLS 1.2 on the wire, but that mainly only affects the TLS
>>>> record protocol and kTLS does that today already anyway.
>>> See above. It's not so much 'fear' as rather the logistics of it.
>>> Getting hold of a TLS library is reasonably easy (Chuck had another
>>> example ready), but massaging it for inclusion into the kernel is
>>> quite some effort.
>>> You might even succeed in convincing the powers that be to include
>>> it into the kernel.
>>> But then you are stuck with having to find a capable maintainer, who
>>> is willing _and qualified_ to take the work and answer awkward questions.
>>> And take the heat when that code introduced a security breach in the linux kernel.
>>> Which excluded essentially everybody who had been working on this project;
>>> we are capable enough engineers in the network and storage space, but
>>> deep security issues ... not so much.
>> Having looked at various TLS libraries during the past few months,
>> I would not even recommend taking any of them. This needs to be
>> written from scratch. Some of them are just license wise a problem
>> others are just too much legacy for TLS 1.3 support.
>> I am happy to give this stab and see how badly I would fail ;)
> Cool. Count me in; I'll gladly give it a spin for NVMe-TLS where
> I've all the surrounding infrastructure like keyrings and certificate generation ready. It really just need a TLS handshake protocol handling...
> 
>> But as stated above, I am surprised on how good TLS 1.3 spec is
>> when it comes to ensuring good and secure implementations. The
>> thing can be really easily unit testes to death. I think people
>> underestimate the huge effort from the guys at IETF to make
>> this simple and more secure.
> 
> True. TLS 1.3 _is_ simple, and it might be that quite some issues
> around TLS is related to older versions.
> 
> [ .. ]
> 
>>>> The code is currently TLS 1.2 and earlier, but I have code for
>>>> TLS 1.3 and also code for utilizing kTLS. It needs a bit more
>>>> cleanup, but then I am happy to publish it. The modified code
>>>> for TLS 1.3 support has TLS Handshake+Alert separated from TLS
>>>> Record protocol and doesn’t even rely on an fd to operate. This
>>>> comes from the requirement that TLS for WiFi Enterprise (or in
>>>> the future QUIC) doesn’t have a fd either.
>>> If you have code to update it to 1.3 I would be very willing to
>>> look at it; the main reason why with went with gnutls was that
>>> no-one of us was eager (not hat the knowledge) to really delve
>>> into TLS and do fancy things.
>>> 
>>> And that was the other thing; we found quite some TLS implementations,
>>> but nearly all of the said '1.3 support to come' ...
>> True to that. I think even OpenSSL started an effort to have a
>> QUIC specific API now.
>> The problem that I found is that TLS Handshake, TLS Alert and
>> TLS Record protocol are not cleanly separated. They are mixed
>> together.
> Yep.
> 
>> For example if I want to use kTLS, I mostly just have to deal
>> with TLS Handshake portion. QUIC was specific and just uses
>> TLS Handshake and TLS Alert are converted to QUIC errors.
> Some for us. Alerts don't make sense to us as we have long-lived connections, so the prime reason for alerts is gone, and we have to re-establish the connection whenever the cipher is changed. So we will be converting alerts in errors, too.

What is the solution for sequence number exhaustion. Do you
re-connect or do you re-key via TLS?

>>>> Long story short, who is suppose to run the TLS Handshake if
>>>> we push it to userspace. There will be never a generic daemon
>>>> that handles all handshakes since they are all application
>>>> specific. No daemon can run the TLS Handshake on behalf of
>>>> Chrome browser for example. This leads me to AF_HANDSHAKE
>>>> is not a good idea.
>>>> One nice thing we found with using keyctl for WiFi Enterprise
>>>> is that we can have certificates that are backed by the TPM.
>>>> Doing that via keyctl was a lot simpler than dealing with the
>>>> different oddities of SSL engines or different variations of
>>>> crypto libraries. The unification by the kernel is really
>>>> nice. I have to re-read how much EFI can provide securely
>>>> hardware backed keys, but for everybody working in early
>>>> userspace or initramfs it is nice to be able to utilize
>>>> this without having to drag in megabytes of TLS library.
> >>
>>> We don't deny that having TLS handshake in the kernel would
>>> be a good thing. It's just the hurdles to _get_ there are
>>> quite high, and we thought that the userspace daemon would be an easier route.
>> My problem with doing an upcall for TLS Handshake messages
>> is to ensure that the right process has the correct rights
>> to receive and send the messages. And nobody else can
>> interfere with that or intercept messages without proper
>> right to do so.
> Correct. That is a concern.
> 
>> And I am certain that Wireshark would love to get hold of
>> the unencrypted TLS Handshake traffic. Debugging TLS
>> and also QUIC transfers is hugely painful. The method of
>> SSLKEYLOGFILE works but it is so cumbersome and defeats
>> any kind of live traffic analysis. So having some DIAG
>> here would help a lot of developers.
> Oh, yes. That would be nice side-effect.
> 
> So, when can I expect the patch?
> :-)

Lol. I need to get a few things cleaned up in the userspace
prototype I have. Then I take a stab at a kernel code. I do
need to build myself a test setup for PSK-only since I have
not yet bothered with that.

Regards

Marcel

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Hannes Reinecke]

On 1/31/23 21:32, Marcel Holtmann wrote:
> Hi Hannes,
> 
[ .. ]
>>> It is just painful for the simple reason that there is no real
>>> standard around CA certificates and where to place them. Every
>>> distro is kinda doing it their way and you expect your TLS
>>> library to do the magic.
>>> I like to see systemd create a keyring of the CA certs, seal it
>>> and then provide it do every process/service it launches. And
>>> for non systemd distros they need to find a way to actually
>>> provide that one keyring that can be used as master for all the
>>> CA certs.
>>> We have not bothered with that yet since for WiFi, you always
>>> have a client cert derived from the CA of the server. So you
>>> give a CA cert and a client cert when you connect to WiFi
>>> Enterprise systems.
>> The good thing is that NVMe is currently PSK-only, so the certificate
> bit is easy for me. Others like NFS will have to do proper X.509 cert
> handling, but I'll let them worry about that :-)
> 
> Interesting. I have not looked at PSK part yet. Do you require PSK-only
> or ECDHE+PSK. From what I quickly glanced at the spec, the PSK-only is
> really simple. That is the most simplest TLS Handshake I have seen and
> I should give that a spin.
> 

ECDHE+PSK, sadly; the PSK-only method will be deprecated eventually.
But I've got code for that already, so that's nothing to worry about.

> So NVMe agrees the PSK out-of-band? I might be able to read up on it,
> but I can only keep so much specifications in memory ;)
> 
Yep, that's the plan. Or, one could say, the 'P' in PSK :-)
That's where keyrings come in; idea is that external agent populate the 
keyring, and the kernel code just looks up keys from there.

[ .. ]
>>>> And that was the other thing; we found quite some TLS implementations,
>>>> but nearly all of the said '1.3 support to come' ...
>>> True to that. I think even OpenSSL started an effort to have a
>>> QUIC specific API now.
>>> The problem that I found is that TLS Handshake, TLS Alert and
>>> TLS Record protocol are not cleanly separated. They are mixed
>>> together.
>> Yep.
>>
>>> For example if I want to use kTLS, I mostly just have to deal
>>> with TLS Handshake portion. QUIC was specific and just uses
>>> TLS Handshake and TLS Alert are converted to QUIC errors.
>> Some for us. Alerts don't make sense to us as we have long-lived
>> connections, so the prime reason for alerts is gone, and we have
>> to re-establish the connection whenever the cipher is changed. So
>> we will be converting alerts in errors, too.
> 
> What is the solution for sequence number exhaustion. Do you
> re-connect or do you re-key via TLS?
> 
Reconnect. We don't have a good way of allowing for re-keying, as the
key material directly relates to information retrieved from the protocol 
itself (here: the TLS key might derived from the DH-CHAP authentication 
protocol running in NVMe space).
So for any re-key operationn we'll should to re-run that protocol. At 
which point we might as well kill the connection and start over, as this 
is basically the same operation.

[ .. ]
>>>>> Long story short, who is suppose to 
>>> And I am certain that Wireshark would love to get hold of
>>> the unencrypted TLS Handshake traffic. Debugging TLS
>>> and also QUIC transfers is hugely painful. The method of
>>> SSLKEYLOGFILE works but it is so cumbersome and defeats
>>> any kind of live traffic analysis. So having some DIAG
>>> here would help a lot of developers.
>> Oh, yes. That would be nice side-effect.
>>
>> So, when can I expect the patch?
>> :-)
> 
> Lol. I need to get a few things cleaned up in the userspace
> prototype I have. Then I take a stab at a kernel code. I do
> need to build myself a test setup for PSK-only since I have
> not yet bothered with that.
> 
I'd be happy to have the userspace code; my plan is to work on a frame 
forwarder (such that the TLS handshake and alert frames are forwarded to 
userspace via netlink).
Then I can repurpose the daemon code from the accept solution to handle 
those netlink frames, and use your library for the tls handshake.

That would give us a nice proof of concept, and if designed properly the 
frame forwarder could later be modified to redirect to the in-kernel code.

So having the userspace code already is a bonus.
And the 'patch' really was just for the userspace library :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Xin Long]

On Tue, Jan 31, 2023 at 9:24 AM Marcel Holtmann <marcel@holtmann.org> wrote:
>
> I know, utilizing existing TLS libraries is a pain if you don’t do
> exactly what they had in mind. I started looking at QUIC a while
> back and quickly realized, I have to start looking at TLS 1.3 first.
>
> My past experience with GnuTLS and OpenSSL have been bad and that is
> why iwd (our WiFi daemon) has its own TLS implementation utilizing
> AF_ALG and keyctl.
Hi Marcel,

I'm no expert on TLS, but I'm a supporter of in-kernel TLS 1.3 Handshake
implementation :). When working on implementing in-kernel QUIC protocol,
the code looks a lot simpler with the pure in-kernel TLS 1.3 Handshake APIs
than the upcall method, and I believe the NFS over TLS 1.3 in kernel will
feel the same.

>
> While that might have been true in the past and with TLS 1.2 and earlier,
> I am not sure that is all true today.
>
> Lets assume we start with TLS 1.3 and don’t have backwards compatibility
> with TLS 1.2 and earlier. And for now we don’t worry about Middleboxes
> compatibility mode since you don’t have to for all the modern protocols
> that utilize just the TLS 1.3 handshake like QUIC.
>
> Now the key derivation is just choosing 1 out of 5 ciphers and using
> its associated hash algorithm to derive the keys. This is all present
> functionality in the kernel and so well tested that it doesn’t worry
> me at all. We also have a separate RFC with just sample data so you
> can check your derivation functionality. Especially if you check it
> against AEAD encrypted sample data, any mistake is fatal.
>
> The shared key portion is just ECDHE or DHE and you either end up with
> x25519 or secp256r1 and both are in the kernel. Bluetooth has been
> using secp256r1 inside the kernel for many years now. We all know how
> to handle and verify public keys from secp256r1 and neat part is that
> it would be also offloaded to hardware if needed. So the private key
> doesn’t need to stay even in kernel memory.
>
> So dealing with generating your key material for your cipher is really
> simple and similar things have been done for Bluetooth for a long
> time now. And it looks like NVMe is also utilizing KPP as of today.
>
> The tricky part is the authentication portion of TLS utilizing
> certificates. That part is complicated, but then again, we already
> decided the kernel needs to handle certificates for various places
> and you have to assume that it is fairly secure.
>
> Now, you need to secure the handshake protocol like any other protocol
> and the only difference is that it will lead to key material and
> does authentication with certificates. All of it, the kernel already
> does in one form or another.
>
> The TLS 1.3 spec is also really nicely written and explicit in
> error behavior in case of attempts to attack the protocol. While
> implementing my TLS 1.3 only prototype I have been positively
> surprised on how clean it is. I personally think they went over
> board with the key verification, but so be it.
>
> Once I have cleaned up my TLS 1.3 prototype, I am happy to take
> a stab at a kernel version.
>
I'm glad to hear that you're planning to add this in kernel space, and I
agree that there won't be a lot of things to do in kernel due to the kernel
crypto APIs. There is also a TLS 1.3 Handshake prototype I worked on and
based on the torvalds/linux kernel code. In case of any duplicate work
when you're doing it, I just share my code here:

  https://github.com/lxin/tls_hs/blob/master/crypto/tls_hs.c

and the TLS_HS APIs docs for QUIC and NFS use are here:

  https://github.com/lxin/tls_hs

Hopefully, it will help.

Besides, There were some security concerns from others for which I didn't
continue:

  https://github.com/lxin/tls_hs#the-security-issues

It will be great if we can have your opinions about it.

Thanks.

Re: [PATCH v2 2/3] net/handshake: Add support for PF_HANDSHAKE

[Hannes Reinecke]

On 2/2/23 18:13, Xin Long wrote:
> On Tue, Jan 31, 2023 at 9:24 AM Marcel Holtmann <marcel@holtmann.org> wrote:
>>
>> I know, utilizing existing TLS libraries is a pain if you don’t do
>> exactly what they had in mind. I started looking at QUIC a while
>> back and quickly realized, I have to start looking at TLS 1.3 first.
>>
>> My past experience with GnuTLS and OpenSSL have been bad and that is
>> why iwd (our WiFi daemon) has its own TLS implementation utilizing
>> AF_ALG and keyctl.
> Hi Marcel,
> 
> I'm no expert on TLS, but I'm a supporter of in-kernel TLS 1.3 Handshake
> implementation :). When working on implementing in-kernel QUIC protocol,
> the code looks a lot simpler with the pure in-kernel TLS 1.3 Handshake APIs
> than the upcall method, and I believe the NFS over TLS 1.3 in kernel will
> feel the same.
> 
>>
>> While that might have been true in the past and with TLS 1.2 and earlier,
>> I am not sure that is all true today.
>>
>> Lets assume we start with TLS 1.3 and don’t have backwards compatibility
>> with TLS 1.2 and earlier. And for now we don’t worry about Middleboxes
>> compatibility mode since you don’t have to for all the modern protocols
>> that utilize just the TLS 1.3 handshake like QUIC.
>>
>> Now the key derivation is just choosing 1 out of 5 ciphers and using
>> its associated hash algorithm to derive the keys. This is all present
>> functionality in the kernel and so well tested that it doesn’t worry
>> me at all. We also have a separate RFC with just sample data so you
>> can check your derivation functionality. Especially if you check it
>> against AEAD encrypted sample data, any mistake is fatal.
>>
>> The shared key portion is just ECDHE or DHE and you either end up with
>> x25519 or secp256r1 and both are in the kernel. Bluetooth has been
>> using secp256r1 inside the kernel for many years now. We all know how
>> to handle and verify public keys from secp256r1 and neat part is that
>> it would be also offloaded to hardware if needed. So the private key
>> doesn’t need to stay even in kernel memory.
>>
>> So dealing with generating your key material for your cipher is really
>> simple and similar things have been done for Bluetooth for a long
>> time now. And it looks like NVMe is also utilizing KPP as of today.
>>
>> The tricky part is the authentication portion of TLS utilizing
>> certificates. That part is complicated, but then again, we already
>> decided the kernel needs to handle certificates for various places
>> and you have to assume that it is fairly secure.
>>
>> Now, you need to secure the handshake protocol like any other protocol
>> and the only difference is that it will lead to key material and
>> does authentication with certificates. All of it, the kernel already
>> does in one form or another.
>>
>> The TLS 1.3 spec is also really nicely written and explicit in
>> error behavior in case of attempts to attack the protocol. While
>> implementing my TLS 1.3 only prototype I have been positively
>> surprised on how clean it is. I personally think they went over
>> board with the key verification, but so be it.
>>
>> Once I have cleaned up my TLS 1.3 prototype, I am happy to take
>> a stab at a kernel version.
>>
> I'm glad to hear that you're planning to add this in kernel space, and I
> agree that there won't be a lot of things to do in kernel due to the kernel
> crypto APIs. There is also a TLS 1.3 Handshake prototype I worked on and
> based on the torvalds/linux kernel code. In case of any duplicate work
> when you're doing it, I just share my code here:
> 
>    https://github.com/lxin/tls_hs/blob/master/crypto/tls_hs.c
> 
> and the TLS_HS APIs docs for QUIC and NFS use are here:
> 
>    https://github.com/lxin/tls_hs
> 
> Hopefully, it will help.
> 
> Besides, There were some security concerns from others for which I didn't
> continue:
> 
>    https://github.com/lxin/tls_hs#the-security-issues
> 
> It will be great if we can have your opinions about it.
> 
> Thanks.
Wow.
That certainly looks good; I'll give it a go and try to integrate it 
with my NVMe-TLS stuff.
Thanks a lot for this!

As for your security concerns:
- Certification management _is_ complete, but we have the kernel keyring
   to help us out. The cert can be looked up in the keyring, and one can
   even call request_key() to get userspace to supply us with one.
   With that we can just use the kernel keyring to lookup certificates
   and don't worry about key management itself.
- TLS 1.3 is nailed down, and won't inherit more encryption algorithms.
   So nothing to worry about
- Corner cases ... yes, the code needs to be validated.
- RSA rewrite ... can't we disallow RSA for in-kernel usage?
- mpi code not constant-time safe; might be. But that goes for all
   in-kernel users, and I'm not sure if _we_ have to worry about that.
- DH and ECDH potentially broken: not anymore; got fixed up to
   be FIPS compliant
- Take advantage of decades of work in userspace: that's _precisely_
   why we're sticking with TLS 1.3. Most of the backwards-compat stuff
   got removed, so we _don't_ have to worry about that.

So it's at least worth a shot, to see where we end up.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman