From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C022BC433E0 for ; Sun, 3 Jan 2021 05:59:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F38E2078D for ; Sun, 3 Jan 2021 05:59:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726337AbhACF6b (ORCPT ); Sun, 3 Jan 2021 00:58:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726308AbhACF6a (ORCPT ); Sun, 3 Jan 2021 00:58:30 -0500 Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DD24C061573 for ; Sat, 2 Jan 2021 21:57:50 -0800 (PST) Received: by mail-pg1-x531.google.com with SMTP id 15so16681249pgx.7 for ; Sat, 02 Jan 2021 21:57:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=7sdfP002QB67rsppcRE4l4DYtVyjtbHqW7DNqcTkLOE=; b=LAvdE0DkDLkJx3aqt9wFRKwYpODYUmNVcm1rvouYzhCylvlu81O8cAFJDHQ0ymOIrB oKz11QbmVQuFElrK0TJi6y1Xa+F7VMxSaogTvh+mrs/Up4BnnMqYomNV3tsFgnvdkmE0 czJxfHVCjc+b2RVr1zMzG+DX/XhkLD9Ec6VRWDYrZiaSaQPdzS96G7QztRCaItuEgh/7 O9UI7jLJ9fzxl8+/a3lgAcDzTC+rrfGCAzZXUzpTptGW0m95xj4eEOXU9L7KBPGxzEYe jGnlC2v2N5cLYpGKAHmzcDAklx6XgvV3Ov77Ld+tz5PszApARQ/WWrFw9tX/9mxcKXnX vypg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=7sdfP002QB67rsppcRE4l4DYtVyjtbHqW7DNqcTkLOE=; b=jnVM8myvlRYGaDzMD+0QT1683ZUzU1gjHtYztfmLWIZUbh2cgA9nNgGK0QRXVoZHRC GkU1Wcy5MD9dExiydevdRs+eoI8pEom1r42ra11WXuMb7jI4OojVWmbFR7QMyI2bav6i z14pWKZoVUs5hHqLN+n4M/cj/6sAfJ2Fz8H2nniETxEAy8ENE2i9G9mC8QT5ZaqrSvnq z5rvTGN9CzkC078knprzMZqimaW6eNcXUPU2Kt4Dv1rAR91YH74GMPqTIoQkJBbBZYQb BVNKEOjYefWEr6oAjfMByzTAnrRxJmSnqHowOVI7l4kdPsqg4AKuiQi5MxApSK3yKmyX REbA== X-Gm-Message-State: AOAM531OtFDy1jVj/EjrX3FF3gKnRPuTMujnVBnQZhNP92XCepZVPF4U 9WYvIY5f+XUxbI64QzcT2tbcZy+52hQ= X-Google-Smtp-Source: ABdhPJz/EcFn6ADabamNEhWGaMP1CueZR0hks+UgDKMl9eRx26kED3QjNjFMfjdQXwADUuMU3ERAPw== X-Received: by 2002:a62:6456:0:b029:1a1:e39e:cb46 with SMTP id y83-20020a6264560000b02901a1e39ecb46mr61823799pfb.0.1609653468017; Sat, 02 Jan 2021 21:57:48 -0800 (PST) Received: from hosting.home (S010664777d4a88b3.cg.shawcable.net. [70.77.224.58]) by smtp.gmail.com with ESMTPSA id k15sm52747771pfp.115.2021.01.02.21.57.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 21:57:47 -0800 (PST) Sender: Sheng Mao Date: Sat, 2 Jan 2021 22:57:45 -0700 From: Sheng Mao To: Wang Yugui Cc: linux-btrfs@vger.kernel.org Subject: Re: [PATCH v2 1/3] btrfs-progs: add Kernel TLS to btrfs send/receive Message-ID: <20210103055745.GA4043@hosting.home> References: <20210102184531.9DC0.409509F4@e16-tech.com> <20210102154713.GA2785@hosting.home> <20210103124513.FBC5.409509F4@e16-tech.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210103124513.FBC5.409509F4@e16-tech.com> User-Agent: Mutt/1.10.1 (2018-07-13) Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Hi Yugui, Thank you so much for the testing and benchmarking! The results are encouraging. For the failure you have seen, let me share more background information with you. On sending side, btrfs kernel driver first writes to a pipe and then splice the pipe to socket. Data pile up in the buffer of socket and cannot be sent out. Either following splice(s) push the socket to send the data, or an explicit write() is needed. In the mode of tls, gnutls_bye() makes sure the data are all sent. Right now, the default mode is tls none. If it is caused by this, the error should show up at the end of the stream only. But the failed cases both fail in the middle. Would you mind to use --tls-mode tls_12_128_gcm to test again? Do you need a patch to dump the header of the message on crc failure? Thank you! Best regards, Sheng On Sun, Jan 03, 2021 at 12:45:13PM +0800, Wang Yugui wrote: > Hi, Sheng > > I test it again with this patch, and huge performance improvement as > reported! > > > I have made a patch to fix it: > > https://patchwork.kernel.org/project/linux-btrfs/patch/20201226214606.49241-1-shngmao@gmail.com/ > > > > The benchmark result is recorded in github issue #324: > > https://github.com/kdave/btrfs-progs/issues/324#issuecomment-751813175 > > but sometimes there is 'ERROR: crc32 mismatch in command'. > > This is a 190G subvolume to send/receive, and then sometime slient error > in transfer? > > failed case 1: > [root@T620 btrfs-progs]# ./test.send.listen-addr.sh > + server=10.0.0.76 > + sleep 2 > + ssh 10.0.0.76 /usr/bin/time btrfs-5.9 receive --listen-addr :: --tcp-port 8080 --dump > + /usr/bin/time btrfs-5.9 send --conn-addr 10.0.0.76 --tcp-port 8080 /archive/movie2 > ERROR: crc32 mismatch in command > ERROR: failed to dump the send stream: Invalid argument > Command exited with non-zero status 1 > 28.39user 49.96system 2:02.01elapsed 64%CPU (0avgtext+0avgdata 5920maxresident)k > 96inputs+0outputs (1major+390minor)pagefaults 0swaps > + sleep 1 > + cat time.send > At subvol /archive/movie2 > ERROR: failed to read stream from kernel: Connection reset by peer > Command exited with non-zero status 104 > 0.30user 130.54system 2:00.35elapsed 108%CPU (0avgtext+0avgdata 5564maxresident)k > 192302416inputs+16outputs (1major+457minor)pagefaults 0swaps > > failed case 2: > [root@T620 btrfs-progs]# ./test.send.listen-addr.sh > + server=10.0.0.76 > + sleep 2 > + ssh 10.0.0.76 /usr/bin/time btrfs-5.9 receive --listen-addr :: --tcp-port 8080 --dump > + /usr/bin/time btrfs-5.9 send --conn-addr 10.0.0.76 --tcp-port 8080 /archive/movie2 > ERROR: crc32 mismatch in command > ERROR: failed to dump the send stream: Invalid argument > Command exited with non-zero status 1 > 28.40user 48.62system 1:39.13elapsed 77%CPU (0avgtext+0avgdata 5920maxresident)k > 0inputs+0outputs (0major+320minor)pagefaults 0swaps > + sleep 1 > + cat time.send > At subvol /archive/movie2 > ERROR: failed to read stream from kernel: Connection reset by peer > > > > > OK case 1: > > + server=10.0.0.76 > + ssh 10.0.0.76 btrfs subvolume delete -c /btrfs/movie2 > Delete subvolume (commit): '/btrfs/movie2' > + sleep 2 > + ssh 10.0.0.76 /usr/bin/time btrfs-5.9 receive --listen-addr :: --tcp-port 8080 /btrfs > + /usr/bin/time btrfs-5.9 send --conn-addr 10.0.0.76 --tcp-port 8080 /archive/movie2 > At subvol movie2 > + sleep 1 > 47.65user 267.49system 5:20.98elapsed 98%CPU (0avgtext+0avgdata 5828maxresident)k > 96inputs+396806624outputs (1major+509minor)pagefaults 0swaps > + cat time.send > At subvol /archive/movie2 > 0.52user 249.44system 5:19.27elapsed 78%CPU (0avgtext+0avgdata 5740maxresident)k > 174679448inputs+8outputs (1major+457minor)pagefaults 0swaps > > > OK case 1: > > + server=10.0.0.76 > + sleep 2 > + ssh 10.0.0.76 /usr/bin/time btrfs-5.9 receive --listen-addr :: --tcp-port 8080 --dump > + /usr/bin/time btrfs-5.9 send --conn-addr 10.0.0.76 --tcp-port 8080 /archive/movie2 > 61.53user 104.62system 3:21.60elapsed 82%CPU (0avgtext+0avgdata 5684maxresident)k > 0inputs+0outputs (0major+313minor)pagefaults 0swaps > + sleep 1 > + cat time.send > At subvol /archive/movie2 > 0.60user 231.95system 3:19.91elapsed 116%CPU (0avgtext+0avgdata 5768maxresident)k > 66606224inputs+8outputs (0major+394minor)pagefaults 0swaps > > > Best Regards > Wang Yugui (wangyugui@e16-tech.com) > 2021/01/03 > > > Hi Yugui, > > > > Thank you for the benchmark! > > > > I forgot to mention the CRC. There is a faster version of CRC but > > current receive buffer `btrfs_send_stream::read_buf` is not aligned > > to unsigned long. > > > > I have made a patch to fix it: > > https://patchwork.kernel.org/project/linux-btrfs/patch/20201226214606.49241-1-shngmao@gmail.com/ > > > > The benchmark result is recorded in github issue #324: > > https://github.com/kdave/btrfs-progs/issues/324#issuecomment-751813175 > > > > In addition, on the server side, would you mind to try: > > > > btrfs-5.9 receive --listen-addr :: --tcp-port 8080 --dump &> /dev/null > > > > --dump only writes to stdout thus we can see network-only performance. > > > > Thank you! > > > > On Sat, Jan 02, 2021 at 06:45:32PM +0800, Wang Yugui wrote: > > > To: Sheng > > > > > > some test result on 10Gbps. > > > > > > Server:dell Precision T7610 Xeon(R) CPU E5-2660 v2 > > > Client: dell PowerEdge T620 Xeon(R) CPU E5-2680 v2 > > > OS: centos 8.3(gnutls-3.6.14-6.el8.x86_64) > > > > > > test script: > > > server=10.0.0.76 > > > ssh ${server} btrfs-5.9 subvolume delete /btrfs/movie2 > > > ssh ${server} /usr/bin/time btrfs-5.9 receive --listen-addr :: --tcp-port 8080 /btrfs & > > > sleep 2 > > > /usr/bin/time btrfs-5.9 send --conn-addr ${server} --tcp-port 8080 /archive/movie2 2>time.send > > > sleep 1 > > > cat time.send > > > > > > test result: > > > It works with about 200 MB/s. slow than expacted. > > > top result: > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 7479 root 20 0 45568 5200 4540 R 100.0 0.0 8:16.05 btrfs-5.9 receive --listen-addr > > > > > > #pstack 7479 > > > #0 __crc32c_le (crc=2848417421, data=0x7ffe68ee038b "\277g\276+\022\nZ", length=) at crypto/crc32c.c:213 > > > #1 0x00000000004637ed in crc32c_le (crc=crc@entry=0, data=data@entry=0x7ffe68edb004 "O\300", length=length@entry=49241) at crypto/crc32c.c:223 > > > #2 0x000000000045b71a in read_cmd (sctx=0x7ffe68edb000) at common/send-stream.c:141 > > > #3 read_and_process_cmd (sctx=0x7ffe68edb000) at common/send-stream.c:330 > > > > > > That is to say, for high speed network/disk config, the current bottleneck > > > of btrfs send/receive is the CRC process in btrfs receive side. > > > > > > Best Regards > > > Wang Yugui (wangyugui@e16-tech.com) > > > 2021/01/02 > > > > > > > From: Sheng Mao > > > > > > > > Currently, btrfs send outputs to a pipe or a file; > > > > btrfs receive inputs from a pipe or a file. > > > > The pipe can be a SSH or a stunnel connection. > > > > btrfs send/receive itself doesn't handle any connection. > > > > > > > > Kernel introduces TLS in version 4.13 (referred as ktls). > > > > Ktls provides a transparent TLS 1.2/1.3 connection: > > > > from user space aspect, applications use a normal socket > > > > fd to read/write from/to. This model fits into btrfs send's > > > > design well: btrfs first writes to a pipe and then splices > > > > data from pipe to the final fd (which is a file or pipe). > > > > Ktls simply replaces the final fd with the ktls socket fd. > > > > According to ktls' author, ktls can boost performance for > > > > 2~7%. Ktls helps less on receiving side: btrfs receive > > > > processes data in user space. But btrfs receive still can > > > > use transparent TLS layer. > > > > > > > > I have implemented ktls for btrfs send/receive. Here are > > > > the key features: > > > > > > > > - Use GnuTLS for handshake. OpenSSL is not suitable for this > > > > task, we need to pass IV and key from handshake session to > > > > kernel setting. > > > > - Use PSK (pre-shared key) for handshake. User can use --key > > > > to use a key file or input password on prompt. > > > > - Key file is checked as PEM format first; if it fails, key > > > > file is treated as raw binary file. > > > > - Three TLS modes are supported: none (raw TCP), TLS 1.2 + GCM > > > > 128/256, TLS 1.3 + GCM 128. > > > > - DTLS, certificate, Secure Remote Password (SRP) are not > > > > supported yet. > > > > > > > > Issue: #326 > > > > Signed-off-by: Sheng Mao > > > > --- > > > > common/ktls.c | 702 ++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > common/ktls.h | 57 ++++ > > > > 2 files changed, 759 insertions(+) > > > > create mode 100644 common/ktls.c > > > > create mode 100644 common/ktls.h > > > > > > > > diff --git a/common/ktls.c b/common/ktls.c > > > > new file mode 100644 > > > > index 00000000..a4d670e3 > > > > --- /dev/null > > > > +++ b/common/ktls.c > > > > @@ -0,0 +1,702 @@ > > > > +/* > > > > + * Copyright (C) 2020 Sheng Mao. All rights reserved. > > > > + * > > > > + * This program is free software; you can redistribute it and/or > > > > + * modify it under the terms of the GNU General Public > > > > + * License v2 as published by the Free Software Foundation. > > > > + * > > > > + * This program is distributed in the hope that it will be useful, > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > > + * General Public License for more details. > > > > + */ > > > > + > > > > +#include > > > > +#include > > > > +#include > > > > + > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > +#include > > > > + > > > > +#include > > > > + > > > > +#include > > > > + > > > > +#include > > > > + > > > > +#include "ktls.h" > > > > + > > > > +enum { KTLS_MAX_PASSWORD_LENGTH = 256, KTLS_MAX_PRIORITY_STRING_LENTH = 256 }; > > > > + > > > > +enum { > > > > + KTLS_STAGE_NOT_HANDSHAKED, > > > > + KTLS_STAGE_HAS_HANDSHAKED, > > > > +}; > > > > + > > > > +enum ktls_tls_mode_t { > > > > + KTLS_TLS_MODE_NONE = 0, > > > > + KTLS_TLS_12_128_GCM, > > > > + KTLS_TLS_13_128_GCM, > > > > + KTLS_TLS_12_256_GCM > > > > +}; > > > > + > > > > +struct ktls_session { > > > > + gnutls_session_t session; > > > > + gnutls_certificate_credentials_t crt_cred; > > > > + > > > > + gnutls_psk_server_credentials_t psk_cred_server; > > > > + gnutls_psk_client_credentials_t psk_cred_client; > > > > + > > > > + uint8_t role; > > > > + uint8_t stage; > > > > + enum ktls_tls_mode_t tls_mode; > > > > +}; > > > > + > > > > +static gnutls_datum_t ktls_psk_username = { 0 }; > > > > +static gnutls_datum_t ktls_psk_key = { 0 }; > > > > + > > > > +static void ktls_print_logs(int level, const char *msg) > > > > +{ > > > > + if (bconf.verbose >= level) > > > > + printf("GnuTLS [%d]: %s", level, msg); > > > > +} > > > > + > > > > +int ktls_set_psk_session_from_password_prompt(struct ktls_session *session, > > > > + const char *username) > > > > +{ > > > > + struct termios orig_term_flags, passwd_term_flags; > > > > + char passwd[KTLS_MAX_PASSWORD_LENGTH]; > > > > + int passwd_sz = 0; > > > > + int stdin_fd = 0; > > > > + > > > > + if (!session) > > > > + return EXIT_FAILURE; > > > > + > > > > + if (session->tls_mode == KTLS_TLS_MODE_NONE) > > > > + return EXIT_SUCCESS; > > > > + > > > > + passwd[0] = '\0'; > > > > + > > > > + stdin_fd = fileno(stdin); > > > > + > > > > + if (!isatty(stdin_fd)) { > > > > + error("tty needed for password input"); > > > > + return EXIT_FAILURE; > > > > + } > > > > + > > > > + tcgetattr(stdin_fd, &orig_term_flags); > > > > + passwd_term_flags = orig_term_flags; > > > > + passwd_term_flags.c_lflag &= ~ECHO; > > > > + passwd_term_flags.c_lflag |= ECHONL; > > > > + > > > > + if (tcsetattr(stdin_fd, TCSANOW, &passwd_term_flags)) { > > > > + error("fail to hide password: %s", strerror(errno)); > > > > + return EXIT_FAILURE; > > > > + } > > > > + > > > > + printf("password: "); > > > > + if (!fgets(passwd, sizeof(passwd), stdin)) { > > > > + error("no password read"); > > > > + return EXIT_FAILURE; > > > > + } > > > > + > > > > + if (tcsetattr(fileno(stdin), TCSANOW, &orig_term_flags)) { > > > > + error("fail to reset tty: %s", strerror(errno)); > > > > + return EXIT_FAILURE; > > > > + } > > > > + > > > > + passwd_sz = strlen(passwd); > > > > + if (passwd_sz <= 0) { > > > > + error("no password read"); > > > > + return EXIT_FAILURE; > > > > + } > > > > + if (passwd[passwd_sz - 1] == '\n') { > > > > + passwd[passwd_sz - 1] = '\0'; > > > > + passwd_sz--; > > > > + } > > > > + > > > > + return ktls_set_psk_session(session, username, > > > > + (const unsigned char *)passwd, passwd_sz); > > > > +} > > > > + > > > > +int ktls_set_psk_session_from_keyfile(struct ktls_session *session, > > > > + const char *username, > > > > + const char *key_file) > > > > +{ > > > > + int rc = GNUTLS_E_SUCCESS; > > > > + struct stat file_stat; > > > > + size_t sz = 0; > > > > + FILE *fp = NULL; > > > > + gnutls_datum_t input = { NULL, 0UL }; > > > > + gnutls_datum_t output = { NULL, 0UL }; > > > > + > > > > + if (!session) > > > > + goto cleanup; > > > > + > > > > + if (session->tls_mode == KTLS_TLS_MODE_NONE) > > > > + return EXIT_SUCCESS; > > > > + > > > > + if (stat(key_file, &file_stat)) { > > > > + error("fail to open keyfile: %s", strerror(errno)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + fp = fopen(key_file, "r"); > > > > + if (!fp) { > > > > + error("fail to open keyfile: %s", strerror(errno)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + input.size = file_stat.st_size; > > > > + > > > > + input.data = gnutls_malloc(input.size); > > > > + > > > > + sz = fread(input.data, 1, input.size, fp); > > > > + > > > > + if (sz != input.size) { > > > > + error("fail to read PEM"); > > > > + goto cleanup; > > > > + } > > > > + > > > > + rc = gnutls_pem_base64_decode2(NULL, &input, &output); > > > > + if (rc != GNUTLS_E_SUCCESS) { > > > > + error("Error! fail to decode PEM: %s", gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + return ktls_set_psk_session(session, username, output.data, > > > > + output.size); > > > > + > > > > +cleanup: > > > > + return EXIT_FAILURE; > > > > +} > > > > + > > > > +struct ktls_session *ktls_create_session(bool is_sender) > > > > +{ > > > > + struct ktls_session *session = NULL; > > > > + > > > > + session = (struct ktls_session *)malloc(sizeof(struct ktls_session)); > > > > + explicit_bzero(session, sizeof(*session)); > > > > + > > > > + gnutls_global_init(); > > > > + > > > > + session->role = is_sender ? GNUTLS_CLIENT : GNUTLS_SERVER; > > > > + session->stage = KTLS_STAGE_NOT_HANDSHAKED; > > > > + > > > > + gnutls_init(&session->session, session->role); > > > > + > > > > + gnutls_global_set_log_level(bconf.verbose); > > > > + gnutls_global_set_log_function(ktls_print_logs); > > > > + > > > > + return session; > > > > +} > > > > + > > > > +void ktls_destroy_session(struct ktls_session *session) > > > > +{ > > > > + if (!session) > > > > + return; > > > > + > > > > + if (session->crt_cred) > > > > + gnutls_certificate_free_credentials(session->crt_cred); > > > > + > > > > + if (session->psk_cred_server) > > > > + gnutls_psk_free_server_credentials(session->psk_cred_server); > > > > + > > > > + if (session->psk_cred_client) > > > > + gnutls_psk_free_client_credentials(session->psk_cred_client); > > > > + > > > > + if (session->session) { > > > > + if (session->stage == KTLS_STAGE_HAS_HANDSHAKED) > > > > + gnutls_bye(session->session, GNUTLS_SHUT_RDWR); > > > > + gnutls_deinit(session->session); > > > > + } > > > > + > > > > + gnutls_global_deinit(); > > > > + > > > > + explicit_bzero(session, sizeof(*session)); > > > > +} > > > > + > > > > +static int ktls_connect_or_bind(int *sock, bool is_sender, int protocol, > > > > + struct sockaddr *serv_addr, size_t serv_addr_sz) > > > > +{ > > > > + *sock = 0; > > > > + > > > > + *sock = socket(protocol, SOCK_STREAM, 0); > > > > + if (*sock == KTLS_INVALID_FD) { > > > > + error("could not create socket: %s", strerror(errno)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + if (is_sender) { > > > > + if (connect(*sock, serv_addr, serv_addr_sz)) { > > > > + error("fail to connect to server: %s", strerror(errno)); > > > > + goto cleanup; > > > > + } > > > > + return EXIT_SUCCESS; > > > > + } > > > > + > > > > + if (setsockopt(*sock, SOL_SOCKET, SO_REUSEADDR, &(int){ 1 }, > > > > + sizeof(int))) { > > > > + error("fail to connect to server: %s", strerror(errno)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + if (bind(*sock, serv_addr, serv_addr_sz) || listen(*sock, 1)) { > > > > + error("fail to serve as server: %s", strerror(errno)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + return EXIT_SUCCESS; > > > > + > > > > +cleanup: > > > > + if (*sock >= 0) > > > > + close(*sock); > > > > + return EXIT_FAILURE; > > > > +} > > > > + > > > > +static int ktls_connect_domain(int *sock, bool is_sender, const char *host, > > > > + const uint16_t port) > > > > +{ > > > > + struct addrinfo hints = { 0 }, *res = NULL; > > > > + int rc = 0; > > > > + struct sockaddr_in addr4; > > > > + struct sockaddr_in6 addr6; > > > > + > > > > + memset(&hints, 0, sizeof(hints)); > > > > + > > > > + hints.ai_family = PF_UNSPEC; > > > > + hints.ai_socktype = SOCK_STREAM; > > > > + hints.ai_flags |= AI_CANONNAME; > > > > + > > > > + if (getaddrinfo(host, NULL, &hints, &res)) { > > > > + error("fail to get address info: %s", strerror(errno)); > > > > + return EXIT_FAILURE; > > > > + } > > > > + > > > > + while (res) { > > > > + switch (res->ai_family) { > > > > + case AF_INET: > > > > + explicit_bzero(&addr4, sizeof(addr4)); > > > > + addr4.sin_addr = > > > > + ((struct sockaddr_in *)res->ai_addr)->sin_addr; > > > > + addr4.sin_port = port; > > > > + addr4.sin_family = res->ai_family; > > > > + if (!ktls_connect_or_bind( > > > > + sock, is_sender, res->ai_family, > > > > + (struct sockaddr *)&addr4, sizeof(addr4))) { > > > > + goto cleanup; > > > > + } > > > > + break; > > > > + case AF_INET6: > > > > + explicit_bzero(&addr6, sizeof(addr6)); > > > > + addr6.sin6_addr = > > > > + ((struct sockaddr_in6 *)res->ai_addr)->sin6_addr; > > > > + addr6.sin6_port = port; > > > > + addr6.sin6_family = res->ai_family; > > > > + if (!ktls_connect_or_bind( > > > > + sock, is_sender, res->ai_family, > > > > + (struct sockaddr *)&addr6, sizeof(addr6))) { > > > > + goto cleanup; > > > > + } > > > > + break; > > > > + } > > > > + res = res->ai_next; > > > > + } > > > > + > > > > + freeaddrinfo(res); > > > > + return EXIT_FAILURE; > > > > + > > > > +cleanup: > > > > + freeaddrinfo(res); > > > > + return rc; > > > > +} > > > > + > > > > +static int ktls_connect_ip(int *sock, bool is_sender, const char *host, > > > > + const uint16_t port) > > > > +{ > > > > + struct sockaddr_in addr4; > > > > + struct sockaddr_in6 addr6; > > > > + struct sockaddr *serv_addr = NULL; > > > > + size_t serv_addr_sz = 0; > > > > + sa_family_t protol = AF_INET; > > > > + > > > > + explicit_bzero(&addr4, sizeof(addr4)); > > > > + explicit_bzero(&addr6, sizeof(addr6)); > > > > + > > > > + if (inet_pton(AF_INET, host, &addr4.sin_addr) == 1) { > > > > + serv_addr = (struct sockaddr *)&addr4; > > > > + serv_addr_sz = sizeof(addr4); > > > > + protol = addr4.sin_family = AF_INET; > > > > + addr4.sin_port = port; > > > > + } > > > > + > > > > + if (!serv_addr && inet_pton(AF_INET6, host, &addr6.sin6_addr) == 1) { > > > > + serv_addr = (struct sockaddr *)&addr6; > > > > + serv_addr_sz = sizeof(addr6); > > > > + protol = addr6.sin6_family = AF_INET6; > > > > + addr6.sin6_port = port; > > > > + } > > > > + > > > > + if (!serv_addr) > > > > + return KTLS_INVALID_FD; > > > > + > > > > + return ktls_connect_or_bind(sock, is_sender, protol, serv_addr, > > > > + serv_addr_sz); > > > > +} > > > > + > > > > +int ktls_set_tls_mode(struct ktls_session *session, const char *mode) > > > > +{ > > > > + if (!session) > > > > + return EXIT_FAILURE; > > > > + > > > > + if (!strcmp("none", mode)) > > > > + session->tls_mode = KTLS_TLS_MODE_NONE; > > > > + else if (!strcmp("tls_12_128_gcm", mode)) > > > > + session->tls_mode = KTLS_TLS_12_128_GCM; > > > > + else if (!strcmp("tls_13_128_gcm", mode)) > > > > + session->tls_mode = KTLS_TLS_13_128_GCM; > > > > + else if (!strcmp("tls_12_256_gcm", mode)) > > > > + session->tls_mode = KTLS_TLS_12_256_GCM; > > > > + else { > > > > + error("unknown tls mode: %s", mode); > > > > + return EXIT_FAILURE; > > > > + } > > > > + return EXIT_SUCCESS; > > > > +} > > > > + > > > > +#define INIT_GCM_WITH_MODE(V, X) \ > > > > + { \ > > > > + struct tls12_crypto_info_aes_gcm_##X crypto_info; \ > > > > +\ > > > > + crypto_info.info.version = TLS_##V##_VERSION; \ > > > > + crypto_info.info.cipher_type = TLS_CIPHER_AES_GCM_##X; \ > > > > + memcpy(crypto_info.iv, seq_number, \ > > > > + TLS_CIPHER_AES_GCM_##X##_IV_SIZE); \ > > > > + memcpy(crypto_info.rec_seq, seq_number, \ > > > > + TLS_CIPHER_AES_GCM_##X##_REC_SEQ_SIZE); \ > > > > + if (cipher_key.size != TLS_CIPHER_AES_GCM_##X##_KEY_SIZE) { \ > > > > + error("mismatch in send key size: %d != %d\n", \ > > > > + cipher_key.size, \ > > > > + TLS_CIPHER_AES_GCM_##X##_KEY_SIZE); \ > > > > + goto cleanup; \ > > > > + } \ > > > > + memcpy(crypto_info.key, cipher_key.data, \ > > > > + TLS_CIPHER_AES_GCM_##X##_KEY_SIZE); \ > > > > + memcpy(crypto_info.salt, iv.data, \ > > > > + TLS_CIPHER_AES_GCM_##X##_SALT_SIZE); \ > > > > + if (setsockopt(sock, SOL_TLS, is_sender ? TLS_TX : TLS_RX, \ > > > > + &crypto_info, sizeof(crypto_info))) { \ > > > > + error("fail to set kernel tls: %s", strerror(errno)); \ > > > > + goto cleanup; \ > > > > + } \ > > > > + } > > > > + > > > > +int ktls_handshake_tls(struct ktls_session *session, int sock) > > > > +{ > > > > + int rc = 0; > > > > + bool is_sender = false; > > > > + int handshake_retry = 3; > > > > + char tls_priority_list[KTLS_MAX_PRIORITY_STRING_LENTH]; > > > > + const char *tls_priority_templ = > > > > + "NONE:+MAC-ALL:+COMP-NULL:+SIGN-ALL:+GROUP-ALL:+ECDHE-PSK:+DHE-PSK:%s:%s"; > > > > + const char *tls_priority_ver_mode = NULL; > > > > + > > > > + gnutls_datum_t mac_key; > > > > + gnutls_datum_t iv; > > > > + gnutls_datum_t cipher_key; > > > > + unsigned char seq_number[8]; > > > > + > > > > + if (!session || !session->session) > > > > + return EXIT_FAILURE; > > > > + > > > > + switch (session->tls_mode) { > > > > + case KTLS_TLS_MODE_NONE: > > > > + return EXIT_SUCCESS; > > > > + case KTLS_TLS_12_128_GCM: > > > > + tls_priority_ver_mode = "+VERS-TLS1.2:+AES-128-GCM"; > > > > + break; > > > > + case KTLS_TLS_13_128_GCM: > > > > + tls_priority_ver_mode = "+VERS-TLS1.3:+AES-128-GCM"; > > > > + break; > > > > + case KTLS_TLS_12_256_GCM: > > > > + tls_priority_ver_mode = "+VERS-TLS1.2:+AES-256-GCM"; > > > > + break; > > > > + default: > > > > + error("unknown tls mode"); > > > > + goto cleanup; > > > > + } > > > > + > > > > + is_sender = session->role == GNUTLS_CLIENT; > > > > + > > > > + if (is_sender && session->psk_cred_client) { > > > > + rc = gnutls_credentials_set(session->session, GNUTLS_CRD_PSK, > > > > + session->psk_cred_client); > > > > + if (rc != GNUTLS_E_SUCCESS) { > > > > + error("fail to set PSK for client: %s", > > > > + gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + } > > > > + > > > > + if (!is_sender && session->psk_cred_server) { > > > > + rc = gnutls_credentials_set(session->session, GNUTLS_CRD_PSK, > > > > + session->psk_cred_server); > > > > + if (rc != GNUTLS_E_SUCCESS) { > > > > + error("fail to set PSK for server: %s", > > > > + gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + } > > > > + > > > > + if (session->crt_cred) { > > > > + rc = gnutls_credentials_set(session->session, > > > > + GNUTLS_CRD_CERTIFICATE, > > > > + session->crt_cred); > > > > + > > > > + if (rc == GNUTLS_E_SUCCESS) { > > > > + error("fail to set certificate: %s", > > > > + gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + } > > > > + > > > > + snprintf(tls_priority_list, KTLS_MAX_PRIORITY_STRING_LENTH, > > > > + tls_priority_templ, > > > > + is_sender ? "+CTYPE-CLI-ALL" : "+CTYPE-SRV-ALL", > > > > + tls_priority_ver_mode); > > > > + > > > > + rc = gnutls_priority_set_direct(session->session, tls_priority_list, > > > > + NULL); > > > > + if (rc != GNUTLS_E_SUCCESS) { > > > > + error("fail to set priority: %s", gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + gnutls_transport_set_int(session->session, sock); > > > > + > > > > + gnutls_handshake_set_timeout(session->session, > > > > + GNUTLS_DEFAULT_HANDSHAKE_TIMEOUT); > > > > + > > > > + do { > > > > + if (handshake_retry < 0) { > > > > + error("exhaust retries on handshake"); > > > > + break; > > > > + } > > > > + rc = gnutls_handshake(session->session); > > > > + handshake_retry--; > > > > + } while (rc < 0 && !gnutls_error_is_fatal(rc)); > > > > + > > > > + if (gnutls_error_is_fatal(rc)) { > > > > + error("fail on handshake: %s", gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + if (bconf.verbose > 0) { > > > > + char *desc = gnutls_session_get_desc(session->session); > > > > + > > > > + printf("TLS session info: %s\n", desc); > > > > + gnutls_free(desc); > > > > + } > > > > + > > > > + session->stage = KTLS_STAGE_HAS_HANDSHAKED; > > > > + > > > > + rc = gnutls_record_get_state(session->session, is_sender ? 0 : 1, > > > > + &mac_key, &iv, &cipher_key, seq_number); > > > > + if (rc != GNUTLS_E_SUCCESS) { > > > > + error("fail on retrieve TLS record: %s", gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + if (setsockopt(sock, SOL_TCP, TCP_ULP, "tls", sizeof("tls"))) { > > > > + error("fail to set kernel TLS on socket: %s", strerror(errno)); > > > > + goto cleanup; > > > > + } > > > > + > > > > + switch (session->tls_mode) { > > > > + case KTLS_TLS_12_128_GCM: > > > > + INIT_GCM_WITH_MODE(1_2, 128); > > > > + break; > > > > + case KTLS_TLS_13_128_GCM: > > > > + INIT_GCM_WITH_MODE(1_3, 128); > > > > + break; > > > > + case KTLS_TLS_12_256_GCM: > > > > + INIT_GCM_WITH_MODE(1_2, 256); > > > > + break; > > > > + default: > > > > + error("unknown tls mode"); > > > > + goto cleanup; > > > > + } > > > > + > > > > + if (bconf.verbose > 0) > > > > + fprintf(stderr, "ktls init done\n"); > > > > + > > > > + return EXIT_SUCCESS; > > > > + > > > > +cleanup: > > > > + return EXIT_FAILURE; > > > > +} > > > > + > > > > +static int ktls_cp_datum(gnutls_datum_t *to, const gnutls_datum_t *from) > > > > +{ > > > > + if (!to || !from) > > > > + return EXIT_FAILURE; > > > > + > > > > + to->size = from->size; > > > > + to->data = (unsigned char *)gnutls_malloc(to->size); > > > > + memmove(to->data, from->data, to->size); > > > > + > > > > + return EXIT_SUCCESS; > > > > +} > > > > + > > > > +static int ktls_cmp_datum(const gnutls_datum_t *lhs, const gnutls_datum_t *rhs) > > > > +{ > > > > + if (!lhs && !rhs) > > > > + return EXIT_SUCCESS; > > > > + > > > > + if (!lhs || !rhs) > > > > + return EXIT_FAILURE; > > > > + > > > > + if (lhs->size != rhs->size) > > > > + return EXIT_FAILURE; > > > > + > > > > + return memcmp(lhs->data, rhs->data, lhs->size); > > > > +} > > > > + > > > > +static int ktls_set_datum(gnutls_datum_t *to, const unsigned char *from, > > > > + int from_size) > > > > +{ > > > > + if (!to || !from || !from_size) > > > > + return EXIT_FAILURE; > > > > + > > > > + if (from_size < 0) > > > > + from_size = strlen((const char *)from); > > > > + > > > > + to->size = from_size; > > > > + to->data = (unsigned char *)gnutls_malloc(to->size); > > > > + memmove(to->data, from, from_size); > > > > + > > > > + return EXIT_SUCCESS; > > > > +} > > > > + > > > > +static int tls_psk_client_callback(gnutls_session_t session, > > > > + gnutls_datum_t *username, > > > > + gnutls_datum_t *key) > > > > +{ > > > > + if (ktls_cp_datum(username, &ktls_psk_username) || > > > > + ktls_cp_datum(key, &ktls_psk_key)) > > > > + return EXIT_FAILURE; > > > > + > > > > + return EXIT_SUCCESS; > > > > +} > > > > + > > > > +static int tls_psk_server_callback(gnutls_session_t session, > > > > + const gnutls_datum_t *username, > > > > + gnutls_datum_t *key) > > > > +{ > > > > + if (ktls_cmp_datum(username, &ktls_psk_username) || > > > > + ktls_cp_datum(key, &ktls_psk_key)) > > > > + return EXIT_FAILURE; > > > > + > > > > + return EXIT_SUCCESS; > > > > +} > > > > + > > > > +int ktls_set_psk_session(struct ktls_session *session, const char *username, > > > > + const unsigned char *passwd, const size_t sz_passwd) > > > > +{ > > > > + bool is_sender = false; > > > > + int rc = 0; > > > > + > > > > + if (!session || !session->session) > > > > + goto cleanup; > > > > + > > > > + is_sender = session->role == GNUTLS_CLIENT; > > > > + > > > > + if (!is_sender && !session->psk_cred_server) { > > > > + rc = gnutls_psk_allocate_server_credentials( > > > > + &session->psk_cred_server); > > > > + if (rc != GNUTLS_E_SUCCESS) { > > > > + error("fail on set psk for server: %s", > > > > + gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + gnutls_psk_set_server_credentials_function2( > > > > + session->psk_cred_server, tls_psk_server_callback); > > > > + } > > > > + > > > > + if (is_sender && !session->psk_cred_client) { > > > > + rc = gnutls_psk_allocate_client_credentials( > > > > + &session->psk_cred_client); > > > > + if (rc != GNUTLS_E_SUCCESS) { > > > > + error("fail on set psk for client: %s", > > > > + gnutls_strerror(rc)); > > > > + goto cleanup; > > > > + } > > > > + gnutls_psk_set_client_credentials_function2( > > > > + session->psk_cred_client, tls_psk_client_callback); > > > > + } > > > > + > > > > + if (!ktls_psk_key.size) > > > > + gnutls_free(ktls_psk_key.data); > > > > + > > > > + if (!ktls_psk_username.size) > > > > + gnutls_free(ktls_psk_username.data); > > > > + > > > > + if (ktls_set_datum(&ktls_psk_username, (const unsigned char *)username, > > > > + -1) || > > > > + ktls_set_datum(&ktls_psk_key, passwd, sz_passwd)) > > > > + goto cleanup; > > > > + > > > > + return EXIT_SUCCESS; > > > > + > > > > +cleanup: > > > > + return EXIT_FAILURE; > > > > +} > > > > + > > > > +int ktls_create_sock_oneshot(struct ktls_session *session, const char *host, > > > > + const char *port) > > > > +{ > > > > + int sock = 0; > > > > + int nport = 0; > > > > + bool is_sender; > > > > + > > > > + if (!session || !session->session) > > > > + return EXIT_FAILURE; > > > > + > > > > + is_sender = session->role == GNUTLS_CLIENT; > > > > + > > > > + nport = atoi(port); > > > > + > > > > + if (nport >= 0 && nport <= 65535) > > > > + nport = htons((uint16_t)nport); > > > > + > > > > + if (ktls_connect_ip(&sock, is_sender, host, (uint16_t)nport)) > > > > + if (ktls_connect_domain(&sock, is_sender, host, nport)) > > > > + goto cleanup; > > > > + > > > > + if (!is_sender) { > > > > + int accepted_sock = KTLS_INVALID_FD; > > > > + > > > > + accepted_sock = accept(sock, (struct sockaddr *)NULL, NULL); > > > > + close(sock); > > > > + sock = accepted_sock; > > > > + } > > > > + > > > > + if (ktls_handshake_tls(session, sock)) > > > > + goto cleanup; > > > > + > > > > + return sock; > > > > + > > > > +cleanup: > > > > + close(sock); > > > > + return KTLS_INVALID_FD; > > > > +} > > > > diff --git a/common/ktls.h b/common/ktls.h > > > > new file mode 100644 > > > > index 00000000..d744e18e > > > > --- /dev/null > > > > +++ b/common/ktls.h > > > > @@ -0,0 +1,57 @@ > > > > +/* > > > > + * Copyright (C) 2020 Sheng Mao. All rights reserved. > > > > + * > > > > + * This program is free software; you can redistribute it and/or > > > > + * modify it under the terms of the GNU General Public > > > > + * License v2 as published by the Free Software Foundation. > > > > + * > > > > + * This program is distributed in the hope that it will be useful, > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > > + * General Public License for more details. > > > > + */ > > > > + > > > > +#ifndef __BTRFS_KTLS_H__ > > > > +#define __BTRFS_KTLS_H__ > > > > + > > > > +#include > > > > +#include > > > > + > > > > +#ifdef __cplusplus > > > > +extern "C" { > > > > +#endif > > > > + > > > > +struct ktls_session; > > > > + > > > > +enum { KTLS_INVALID_FD = -1 }; > > > > + > > > > +struct ktls_session *ktls_create_session(bool is_sender); > > > > +void ktls_destroy_session(struct ktls_session *session); > > > > + > > > > +// ktls_set_psk_session sets PSK (pre-shared key). username is NULL-terminated > > > > +// string; passwd is sized string. Memory of both strings are managed by > > > > +// caller. currently, this API only allows to set PSK before calling > > > > +// ktls_handshake_*() > > > > +int ktls_set_psk_session(struct ktls_session *session, const char *username, > > > > + const unsigned char *passwd, const size_t sz_passwd); > > > > + > > > > +int ktls_set_psk_session_from_password_prompt(struct ktls_session *session, > > > > + const char *username); > > > > + > > > > +int ktls_set_psk_session_from_keyfile(struct ktls_session *session, > > > > + const char *username, > > > > + const char *key_file); > > > > + > > > > +int ktls_set_tls_mode(struct ktls_session *session, const char *mode); > > > > + > > > > +int ktls_handshake_tls(struct ktls_session *session, int sock); > > > > + > > > > +// ktls_create_sock_oneshot returns a sock fd on success. > > > > +int ktls_create_sock_oneshot(struct ktls_session *session, const char *host, > > > > + const char *port); > > > > + > > > > +#ifdef __cplusplus > > > > +} > > > > +#endif > > > > + > > > > +#endif // __BTRFS_KTLS_H__ > > > > -- > > > > 2.29.2 > > > > > > > >