From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4176C433E0 for ; Thu, 4 Feb 2021 14:17:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0215B64F3F for ; Thu, 4 Feb 2021 14:17:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0215B64F3F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:41152 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7fRs-0004o0-5E for qemu-devel@archiver.kernel.org; Thu, 04 Feb 2021 09:17:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55920) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7fQh-0003w0-8s for qemu-devel@nongnu.org; Thu, 04 Feb 2021 09:16:19 -0500 Received: from mail-qv1-xf2b.google.com ([2607:f8b0:4864:20::f2b]:39532) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l7fQb-0006Qf-Jx for qemu-devel@nongnu.org; Thu, 04 Feb 2021 09:16:17 -0500 Received: by mail-qv1-xf2b.google.com with SMTP id j4so1746120qvk.6 for ; Thu, 04 Feb 2021 06:16:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BsdzygtNAIhHqfvL4QoRsZ7Bh3LnAFT5KG+hsIfeTpU=; b=PBBXCGVSK9KuJ38C6iKPoZ+Y2waLq/CQsw43h6S9w6XjGfT3SeNAZeMD/5d6Nvtu1V b5M4rphwVBwrKphrPxTlnLYoZK8w8qVvMKkF8syDOPpAK7RMsIGeCpL345i4U4LiDhVp IRh447SEbrI9MOEVDxFyKAEYhUXZmo3WOTpoH8JSJKbD79B9DrXnKE6PsvHFy/FTYBNu +AN/LoBqI+rSH+FWqtZzFox9+gVuDTnvOnv3V+edCfkau8a7s4r/oQoPkG1tENQ6rVtu uoxUTfEUe6C+VSCjFE/92cA89WzXeX4pSdeSnR92InIfT+iSdKFZF0ml7c1ujcHlolYc qYDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BsdzygtNAIhHqfvL4QoRsZ7Bh3LnAFT5KG+hsIfeTpU=; b=EZk2XiFh5uvRNEVj+SoTUUSO9Gq156sce/c4Tunb/5YwKJyTIbJETZL5twRdlgvuCj A5u+PYq5vApmtuSBldeJccrgu1S90jKIySqRpB4wgUsY38CjvflqRhmGTT+jVJmxtkB8 JVhLzdoJ9ViaQZqcSOmMNmeOikzybhJ7JLm/dgLTbMTH4BBQZWs6DawpwYKSapWoGoI0 qskrodDorj5gMw/1pMT9ANn/KtI2AjDtVfOUrMQRHZ1zlazWf+GHz3u50x0jl1ziokU4 qT1WWhXJYQwhtch4J7VdABS0Ou4prgEX5sGL3PbMJVhcCfZBJwhBprmmrEgMrWfDaSx9 k0KA== X-Gm-Message-State: AOAM5332xxVJ3Cxu2n9qo9NxDv340xiy7nxHHItrkMM432Od1CO6oD99 rHCXluyu6dwtM+Ghr1pWUxjFs85ax4Ge+ETvrwJX3w== X-Google-Smtp-Source: ABdhPJzfcnkyl8sNw4JKBYKZoQYR/jG0yi8D6Sslj7pidtwB6BHAiYrOLAi/6zrxgMEms0cX40ar7eJzGDS9xvSVuUw= X-Received: by 2002:a0c:9e5e:: with SMTP id z30mr7765614qve.56.1612448170810; Thu, 04 Feb 2021 06:16:10 -0800 (PST) MIME-Version: 1.0 References: <20201215162119.27360-1-zhangjiachen.jaycee@bytedance.com> <20201215162119.27360-7-zhangjiachen.jaycee@bytedance.com> <20210204120840.GG3039@work-vm> In-Reply-To: <20210204120840.GG3039@work-vm> From: Jiachen Zhang Date: Thu, 4 Feb 2021 22:16:01 +0800 Message-ID: Subject: Re: [External] Re: [RFC PATCH 6/9] virtiofsd: Add two new options for crash reconnection To: "Dr. David Alan Gilbert" Content-Type: multipart/alternative; boundary="00000000000070c47f05ba83586c" Received-SPF: pass client-ip=2607:f8b0:4864:20::f2b; envelope-from=zhangjiachen.jaycee@bytedance.com; helo=mail-qv1-xf2b.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: virtio-fs@redhat.com, Xie Yongji , QEMU , Stefan Hajnoczi , "Michael S . Tsirkin" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --00000000000070c47f05ba83586c Content-Type: text/plain; charset="UTF-8" On Thu, Feb 4, 2021 at 8:09 PM Dr. David Alan Gilbert wrote: > * Jiachen Zhang (zhangjiachen.jaycee@bytedance.com) wrote: > > We add two options for virtiofsd crash reconnection: reconnect | > > no_reconnect(default) and > > > > User of virtiofsd should set "-o reconnect" for crash reconnection. Note > > that, when "-o reconnect" is set, some options will not be supported and > we > > need to disable them: > > > > - mount namespace: When mount namespace is enabled, reconnected > > virtiofsd will failed to link/rename for EXDEV. The reason is, when > > virtiofsd is sandboxed by mount namespace, attempts to link/rename > > the fds opened before crashing (also recovered from QEMU) will be > > considered as across mount operations between mounts, which is not > > allowed by host kernel. > > > > So we introduce another option "-o mount_ns|no_mount_ns" (mount_ns > > by default, and takes no effect when sandbox=chroot is specified). > > The option "-o no_mount_ns" should be used with "-o reconnect". > > Why not just use -o sandbox=chroot? > > Yes, thanks. I think this is a good idea, and maybe better than add the new options (mount_ns | no_mount_ns). Actually, "-o sandbox=" has not been upstream when we add the new options. > > - remote locking: As it is hard to determine wether a file is locked or > > not when handling inflight locking requests, we should specify "-o > > no_posix_lock" (default) and "-o no_flock" (default). > > > > - extended attributes: When handling inflight removexattr requests > after > > reconnecting, it is hard to determine wether a attr is already > removed > > or not. Therefore, we should also use "-o noxattr" (default) with "-o > > reconnect". > > Can you explain a bit more about why removexattr is hard to restart? > > Consider the following removexattr handling procedure: lo_removexattr() { (1) // ...... (2). fremovexattr / removexattr (3) // ...... (4). fuse_reply_err(req, saverr); } If virtiofsd successfully executed (2) but crashes at (3). When new virtiofsd is reconnected, as (4) is not executed before crashing, this fuse request will be re-executed from (1), and (2) will be executed for the second time. As the xattr is already removed by the first execution of (2), this time an ENODATA will be returned by (2) and mistakenly replied to fuse by (4). Jiachen > Dave > > > > Signed-off-by: Jiachen Zhang > > Signed-off-by: Xie Yongji > > --- > > tools/virtiofsd/helper.c | 9 +++ > > tools/virtiofsd/passthrough_ll.c | 112 ++++++++++++++++++++++--------- > > 2 files changed, 88 insertions(+), 33 deletions(-) > > > > diff --git a/tools/virtiofsd/helper.c b/tools/virtiofsd/helper.c > > index 75ac48dec2..e0d6525b1a 100644 > > --- a/tools/virtiofsd/helper.c > > +++ b/tools/virtiofsd/helper.c > > @@ -174,6 +174,10 @@ void fuse_cmdline_help(void) > > " - chroot: chroot(2) into > shared\n" > > " directory (use in > containers)\n" > > " default: namespace\n" > > + " -o mount_ns|no_mount_ns enable/disable mount > namespace\n" > > + " default: mount_ns\n" > > + " note: if chroot sandbox mode > is used,\n" > > + " mount_ns will not take > effect.\n" > > " -o timeout= I/O timeout (seconds)\n" > > " default: depends on cache= > option.\n" > > " -o writeback|no_writeback enable/disable writeback > cache\n" > > @@ -191,6 +195,11 @@ void fuse_cmdline_help(void) > > " to virtiofsd from guest > applications.\n" > > " default: > no_allow_direct_io\n" > > " -o announce_submounts Announce sub-mount points to > the guest\n" > > + " -o reconnect|no_reconnect enable/disable crash > reconnection\n" > > + " to enable crash > reconnection, the options\n" > > + " no_mount_ns, no_flock, > no_posix_lock, and\n" > > + " no_xattr should also be > set.\n" > > + " default: no_reconnect\n" > > ); > > } > > > > diff --git a/tools/virtiofsd/passthrough_ll.c > b/tools/virtiofsd/passthrough_ll.c > > index 815b001076..73a50bd7a3 100644 > > --- a/tools/virtiofsd/passthrough_ll.c > > +++ b/tools/virtiofsd/passthrough_ll.c > > @@ -170,6 +170,8 @@ struct lo_data { > > pthread_mutex_t mutex; > > int sandbox; > > int debug; > > + int mount_ns; > > + int reconnect; > > int writeback; > > int flock; > > int posix_lock; > > @@ -204,8 +206,12 @@ static const struct fuse_opt lo_opts[] = { > > { "sandbox=chroot", > > offsetof(struct lo_data, sandbox), > > SANDBOX_CHROOT }, > > + { "reconnect", offsetof(struct lo_data, reconnect), 1 }, > > + { "no_reconnect", offsetof(struct lo_data, reconnect), 0 }, > > { "writeback", offsetof(struct lo_data, writeback), 1 }, > > { "no_writeback", offsetof(struct lo_data, writeback), 0 }, > > + { "mount_ns", offsetof(struct lo_data, mount_ns), 1 }, > > + { "no_mount_ns", offsetof(struct lo_data, mount_ns), 0 }, > > { "source=%s", offsetof(struct lo_data, source), 0 }, > > { "flock", offsetof(struct lo_data, flock), 1 }, > > { "no_flock", offsetof(struct lo_data, flock), 0 }, > > @@ -3047,8 +3053,14 @@ static void setup_namespaces(struct lo_data *lo, > struct fuse_session *se) > > * an empty network namespace to prevent TCP/IP and other network > > * activity in case this process is compromised. > > */ > > - if (unshare(CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET) != 0) { > > - fuse_log(FUSE_LOG_ERR, "unshare(CLONE_NEWPID | CLONE_NEWNS): > %m\n"); > > + int unshare_flag; > > + if (lo->mount_ns) { > > + unshare_flag = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET; > > + } else { > > + unshare_flag = CLONE_NEWPID | CLONE_NEWNET; > > + } > > + if (unshare(unshare_flag) != 0) { > > + fuse_log(FUSE_LOG_ERR, "unshare(): %m\n"); > > exit(1); > > } > > > > @@ -3083,38 +3095,47 @@ static void setup_namespaces(struct lo_data *lo, > struct fuse_session *se) > > /* Send us SIGTERM when the parent thread terminates, see prctl(2) > */ > > prctl(PR_SET_PDEATHSIG, SIGTERM); > > > > - /* > > - * If the mounts have shared propagation then we want to opt out so > our > > - * mount changes don't affect the parent mount namespace. > > - */ > > - if (mount(NULL, "/", NULL, MS_REC | MS_SLAVE, NULL) < 0) { > > - fuse_log(FUSE_LOG_ERR, "mount(/, MS_REC|MS_SLAVE): %m\n"); > > - exit(1); > > - } > > + if (lo->mount_ns) { > > + /* > > + * If the mounts have shared propagation then we want to opt > out so our > > + * mount changes don't affect the parent mount namespace. > > + */ > > + if (mount(NULL, "/", NULL, MS_REC | MS_SLAVE, NULL) < 0) { > > + fuse_log(FUSE_LOG_ERR, "mount(/, MS_REC|MS_SLAVE): %m\n"); > > + exit(1); > > + } > > > > - /* The child must remount /proc to use the new pid namespace */ > > - if (mount("proc", "/proc", "proc", > > - MS_NODEV | MS_NOEXEC | MS_NOSUID | MS_RELATIME, NULL) < > 0) { > > - fuse_log(FUSE_LOG_ERR, "mount(/proc): %m\n"); > > - exit(1); > > - } > > + /* The child must remount /proc to use the new pid namespace */ > > + if (mount("proc", "/proc", "proc", > > + MS_NODEV | MS_NOEXEC | MS_NOSUID | MS_RELATIME, NULL) > < 0) { > > + fuse_log(FUSE_LOG_ERR, "mount(/proc): %m\n"); > > + exit(1); > > + } > > > > - /* > > - * We only need /proc/self/fd. Prevent ".." from accessing parent > > - * directories of /proc/self/fd by bind-mounting it over /proc. > Since / was > > - * previously remounted with MS_REC | MS_SLAVE this mount change > only > > - * affects our process. > > - */ > > - if (mount("/proc/self/fd", "/proc", NULL, MS_BIND, NULL) < 0) { > > - fuse_log(FUSE_LOG_ERR, "mount(/proc/self/fd, MS_BIND): %m\n"); > > - exit(1); > > - } > > + /* > > + * We only need /proc/self/fd. Prevent ".." from accessing > parent > > + * directories of /proc/self/fd by bind-mounting it over /proc. > Since > > + * / was previously remounted with MS_REC | MS_SLAVE this mount > change > > + * only affects our process. > > + */ > > + if (mount("/proc/self/fd", "/proc", NULL, MS_BIND, NULL) < 0) { > > + fuse_log(FUSE_LOG_ERR, "mount(/proc/self/fd, MS_BIND): > %m\n"); > > + exit(1); > > + } > > > > - /* Get the /proc (actually /proc/self/fd, see above) file > descriptor */ > > - lo->proc_self_fd = open("/proc", O_PATH); > > - if (lo->proc_self_fd == -1) { > > - fuse_log(FUSE_LOG_ERR, "open(/proc, O_PATH): %m\n"); > > - exit(1); > > + /* Get the /proc (actually /proc/self/fd, see above) file > descriptor */ > > + lo->proc_self_fd = open("/proc", O_PATH); > > + if (lo->proc_self_fd == -1) { > > + fuse_log(FUSE_LOG_ERR, "open(/proc, O_PATH): %m\n"); > > + exit(1); > > + } > > + } else { > > + /* Now we can get our /proc/self/fd directory file descriptor */ > > + lo->proc_self_fd = open("/proc/self/fd", O_PATH); > > + if (lo->proc_self_fd == -1) { > > + fuse_log(FUSE_LOG_ERR, "open(/proc/self/fd, O_PATH): %m\n"); > > + exit(1); > > + } > > } > > } > > > > @@ -3347,7 +3368,9 @@ static void setup_sandbox(struct lo_data *lo, > struct fuse_session *se, > > { > > if (lo->sandbox == SANDBOX_NAMESPACE) { > > setup_namespaces(lo, se); > > - setup_mounts(lo->source); > > + if (lo->mount_ns) { > > + setup_mounts(lo->source); > > + } > > } else { > > setup_chroot(lo); > > } > > @@ -3438,7 +3461,11 @@ static void setup_root(struct lo_data *lo, struct > lo_inode *root) > > struct stat stat; > > uint64_t mnt_id; > > > > - fd = open("/", O_PATH); > > + if (lo->mount_ns) { > > + fd = open("/", O_PATH); > > + } else { > > + fd = open(lo->source, O_PATH); > > + } > > if (fd == -1) { > > fuse_log(FUSE_LOG_ERR, "open(%s, O_PATH): %m\n", lo->source); > > exit(1); > > @@ -3515,6 +3542,9 @@ int main(int argc, char *argv[]) > > lo.allow_direct_io = 0, > > lo.proc_self_fd = -1; > > > > + lo.reconnect = 0; > > + lo.mount_ns = 1; > > + > > /* Don't mask creation mode, kernel already did that */ > > umask(0); > > > > @@ -3577,6 +3607,22 @@ int main(int argc, char *argv[]) > > goto err_out1; > > } > > > > + /* For sandbox mode "chroot", set the mount_ns option to 0. */ > > + if (lo.sandbox == SANDBOX_CHROOT) { > > + lo.mount_ns = 0; > > + } > > + > > + if (lo.reconnect) { > > + if ((lo.sandbox == SANDBOX_NAMESPACE && lo.mount_ns) || lo.flock > > + || lo.posix_lock || > lo.xattr) { > > + printf("Mount namespace, remote lock, and extended > attributes" > > + " are not supported by reconnection (-o reconnect). > Please " > > + "specify -o no_mount_ns, -o no_flock (default), -o" > > + "no_posix_lock (default), and -o no_xattr > (default).\n"); > > + exit(1); > > + } > > + } > > + > > if (opts.log_level != 0) { > > current_log_level = opts.log_level; > > } else { > > -- > > 2.20.1 > > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > --00000000000070c47f05ba83586c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Feb 4, 2021 at 8:09 PM Dr. Da= vid Alan Gilbert <dgilbert@redhat= .com> wrote:
* Jiachen Zhang (zhangjiachen.jaycee@bytedance.com) wrote:
> We add two options for virtiofsd crash reconnection: reconnect |
> no_reconnect(default) and
>
> User of virtiofsd should set "-o reconnect" for crash reconn= ection. Note
> that, when "-o reconnect" is set, some options will not be s= upported and we
> need to disable them:
>
>=C2=A0 =C2=A0- mount namespace: When mount namespace is enabled, reconn= ected
>=C2=A0 =C2=A0 =C2=A0virtiofsd will failed to link/rename for EXDEV. The= reason is, when
>=C2=A0 =C2=A0 =C2=A0virtiofsd is sandboxed by mount namespace, attempts= to link/rename
>=C2=A0 =C2=A0 =C2=A0the fds opened before crashing (also recovered from= QEMU) will be
>=C2=A0 =C2=A0 =C2=A0considered as across mount operations between mount= s, which is not
>=C2=A0 =C2=A0 =C2=A0allowed by host kernel.
>
>=C2=A0 =C2=A0 =C2=A0So we introduce another option "-o mount_ns|no= _mount_ns" (mount_ns
>=C2=A0 =C2=A0 =C2=A0by default, and takes no effect when sandbox=3Dchro= ot is specified).
>=C2=A0 =C2=A0 =C2=A0The option "-o no_mount_ns" should be use= d with "-o reconnect".

Why not just use -o sandbox=3Dchroot?


Yes, thanks. I think this is a good id= ea, and maybe better than add the new options=C2=A0
(mount_ns | n= o_mount_ns). Actually, "-o sandbox=3D" has not been upstream when= we
add the new options.
=C2=A0
>=C2=A0 =C2=A0- remote locking: As it is hard to determine wether a file= is locked or
>=C2=A0 =C2=A0 =C2=A0not when handling inflight locking requests, we sho= uld specify "-o
>=C2=A0 =C2=A0 =C2=A0no_posix_lock" (default) and "-o no_flock= " (default).
>
>=C2=A0 =C2=A0- extended attributes: When handling inflight removexattr = requests after
>=C2=A0 =C2=A0 =C2=A0reconnecting, it is hard to determine wether a attr= is already removed
>=C2=A0 =C2=A0 =C2=A0or not. Therefore, we should also use "-o noxa= ttr" (default) with "-o
>=C2=A0 =C2=A0 =C2=A0reconnect".

Can you explain a bit more about why removexattr is hard to restart?


Consider the following removexattr han= dling procedure:

=C2=A0 =C2=A0 lo_removexattr() {<= /div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (1) // ......
=C2=A0 =C2=A0= =C2=A0 =C2=A0 (2). fremovexattr=C2=A0/ removexattr
=C2=A0 =C2=A0= =C2=A0 =C2=A0 (3) // ......
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (4).=C2= =A0fuse_reply_err(req, saverr);
=C2=A0 =C2=A0 }

If virtiofsd success= fully executed=C2=A0(2) but crashes at (3). When new virtiofsd is
reconnected, as (4) is not executed before= crashing, this fuse request will be
re-executed=C2=A0from= (1), and (2) will be executed for the second time. As the xattr
= is already removed by the first execution of (2), this time=C2=A0an=C2=A0ENODATA will be
returned by (2) and = mistakenly replied to fuse by (4).

Jiachen
=C2=A0
Dave
>
> Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>=
> Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> ---
>=C2=A0 tools/virtiofsd/helper.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A09 +++
>=C2=A0 tools/virtiofsd/passthrough_ll.c | 112 ++++++++++++++++++++++---= ------
>=C2=A0 2 files changed, 88 insertions(+), 33 deletions(-)
>
> diff --git a/tools/virtiofsd/helper.c b/tools/virtiofsd/helper.c
> index 75ac48dec2..e0d6525b1a 100644
> --- a/tools/virtiofsd/helper.c
> +++ b/tools/virtiofsd/helper.c
> @@ -174,6 +174,10 @@ void fuse_cmdline_help(void)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0- chroot: chroot(2) into shared\n"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0directory (use in containers)\n"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0default: namespace\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 -o mount= _ns|no_mount_ns=C2=A0 =C2=A0 enable/disable mount namespace\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0default: mount_ns\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0note: if chroot sandbox mode is used,\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0mount_ns will not take effect.\n"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 -o = timeout=3D<number>=C2=A0 =C2=A0 =C2=A0 =C2=A0 I/O timeout (seconds)\n= "
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0default: depends on cache=3D option.\n"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 -o = writeback|no_writeback=C2=A0 enable/disable writeback cache\n"
> @@ -191,6 +195,11 @@ void fuse_cmdline_help(void)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0to virtiofsd from guest applications.\n"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0default: no_allow_direct_io\n"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 -o = announce_submounts=C2=A0 =C2=A0 =C2=A0 Announce sub-mount points to the gue= st\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 -o recon= nect|no_reconnect=C2=A0 enable/disable crash reconnection\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0to enable crash reconnection, the options\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0no_mount_ns, no_flock, no_posix_lock, and\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0no_xattr should also be set.\n"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0default: no_reconnect\n"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0);
>=C2=A0 }
>=C2=A0
> diff --git a/tools/virtiofsd/passthrough_ll.c b/tools/virtiofsd/passth= rough_ll.c
> index 815b001076..73a50bd7a3 100644
> --- a/tools/virtiofsd/passthrough_ll.c
> +++ b/tools/virtiofsd/passthrough_ll.c
> @@ -170,6 +170,8 @@ struct lo_data {
>=C2=A0 =C2=A0 =C2=A0 pthread_mutex_t mutex;
>=C2=A0 =C2=A0 =C2=A0 int sandbox;
>=C2=A0 =C2=A0 =C2=A0 int debug;
> +=C2=A0 =C2=A0 int mount_ns;
> +=C2=A0 =C2=A0 int reconnect;
>=C2=A0 =C2=A0 =C2=A0 int writeback;
>=C2=A0 =C2=A0 =C2=A0 int flock;
>=C2=A0 =C2=A0 =C2=A0 int posix_lock;
> @@ -204,8 +206,12 @@ static const struct fuse_opt lo_opts[] =3D {
>=C2=A0 =C2=A0 =C2=A0 { "sandbox=3Dchroot",
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 offsetof(struct lo_data, sandbox),
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 SANDBOX_CHROOT },
> +=C2=A0 =C2=A0 { "reconnect", offsetof(struct lo_data, recon= nect), 1 },
> +=C2=A0 =C2=A0 { "no_reconnect", offsetof(struct lo_data, re= connect), 0 },
>=C2=A0 =C2=A0 =C2=A0 { "writeback", offsetof(struct lo_data, = writeback), 1 },
>=C2=A0 =C2=A0 =C2=A0 { "no_writeback", offsetof(struct lo_dat= a, writeback), 0 },
> +=C2=A0 =C2=A0 { "mount_ns", offsetof(struct lo_data, mount_= ns), 1 },
> +=C2=A0 =C2=A0 { "no_mount_ns", offsetof(struct lo_data, mou= nt_ns), 0 },
>=C2=A0 =C2=A0 =C2=A0 { "source=3D%s", offsetof(struct lo_data= , source), 0 },
>=C2=A0 =C2=A0 =C2=A0 { "flock", offsetof(struct lo_data, floc= k), 1 },
>=C2=A0 =C2=A0 =C2=A0 { "no_flock", offsetof(struct lo_data, f= lock), 0 },
> @@ -3047,8 +3053,14 @@ static void setup_namespaces(struct lo_data *lo= , struct fuse_session *se)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0* an empty network namespace to prevent TCP/= IP and other network
>=C2=A0 =C2=A0 =C2=A0 =C2=A0* activity in case this process is compromis= ed.
>=C2=A0 =C2=A0 =C2=A0 =C2=A0*/
> -=C2=A0 =C2=A0 if (unshare(CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET) = !=3D 0) {
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, "unshare(CLON= E_NEWPID | CLONE_NEWNS): %m\n");
> +=C2=A0 =C2=A0 int unshare_flag;
> +=C2=A0 =C2=A0 if (lo->mount_ns) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 unshare_flag =3D CLONE_NEWPID | CLONE_NEW= NS | CLONE_NEWNET;
> +=C2=A0 =C2=A0 } else {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 unshare_flag =3D CLONE_NEWPID | CLONE_NEW= NET;
> +=C2=A0 =C2=A0 }
> +=C2=A0 =C2=A0 if (unshare(unshare_flag) !=3D 0) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, "unshare(): %= m\n");
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
>=C2=A0 =C2=A0 =C2=A0 }
>=C2=A0
> @@ -3083,38 +3095,47 @@ static void setup_namespaces(struct lo_data *l= o, struct fuse_session *se)
>=C2=A0 =C2=A0 =C2=A0 /* Send us SIGTERM when the parent thread terminat= es, see prctl(2) */
>=C2=A0 =C2=A0 =C2=A0 prctl(PR_SET_PDEATHSIG, SIGTERM);
>=C2=A0
> -=C2=A0 =C2=A0 /*
> -=C2=A0 =C2=A0 =C2=A0* If the mounts have shared propagation then we w= ant to opt out so our
> -=C2=A0 =C2=A0 =C2=A0* mount changes don't affect the parent mount= namespace.
> -=C2=A0 =C2=A0 =C2=A0*/
> -=C2=A0 =C2=A0 if (mount(NULL, "/", NULL, MS_REC | MS_SLAVE,= NULL) < 0) {
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, "mount(/, MS_= REC|MS_SLAVE): %m\n");
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> -=C2=A0 =C2=A0 }
> +=C2=A0 =C2=A0 if (lo->mount_ns) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* If the mounts have shared propaga= tion then we want to opt out so our
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* mount changes don't affect th= e parent mount namespace.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (mount(NULL, "/", NULL, MS_R= EC | MS_SLAVE, NULL) < 0) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, &quo= t;mount(/, MS_REC|MS_SLAVE): %m\n");
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
>=C2=A0
> -=C2=A0 =C2=A0 /* The child must remount /proc to use the new pid name= space */
> -=C2=A0 =C2=A0 if (mount("proc", "/proc", "pr= oc",
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 MS_NODEV | MS_NOEXEC= | MS_NOSUID | MS_RELATIME, NULL) < 0) {
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, "mount(/proc)= : %m\n");
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> -=C2=A0 =C2=A0 }
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 /* The child must remount /proc to use th= e new pid namespace */
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (mount("proc", "/proc&q= uot;, "proc",
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 MS_NOD= EV | MS_NOEXEC | MS_NOSUID | MS_RELATIME, NULL) < 0) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, &quo= t;mount(/proc): %m\n");
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
>=C2=A0
> -=C2=A0 =C2=A0 /*
> -=C2=A0 =C2=A0 =C2=A0* We only need /proc/self/fd. Prevent "..&qu= ot; from accessing parent
> -=C2=A0 =C2=A0 =C2=A0* directories of /proc/self/fd by bind-mounting i= t over /proc. Since / was
> -=C2=A0 =C2=A0 =C2=A0* previously remounted with MS_REC | MS_SLAVE thi= s mount change only
> -=C2=A0 =C2=A0 =C2=A0* affects our process.
> -=C2=A0 =C2=A0 =C2=A0*/
> -=C2=A0 =C2=A0 if (mount("/proc/self/fd", "/proc",= NULL, MS_BIND, NULL) < 0) {
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, "mount(/proc/= self/fd, MS_BIND): %m\n");
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> -=C2=A0 =C2=A0 }
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* We only need /proc/self/fd. Preve= nt ".." from accessing parent
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* directories of /proc/self/fd by b= ind-mounting it over /proc. Since
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* / was previously remounted with M= S_REC | MS_SLAVE this mount change
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* only affects our process.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (mount("/proc/self/fd", &quo= t;/proc", NULL, MS_BIND, NULL) < 0) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, &quo= t;mount(/proc/self/fd, MS_BIND): %m\n");
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
>=C2=A0
> -=C2=A0 =C2=A0 /* Get the /proc (actually /proc/self/fd, see above) fi= le descriptor */
> -=C2=A0 =C2=A0 lo->proc_self_fd =3D open("/proc", O_PATH)= ;
> -=C2=A0 =C2=A0 if (lo->proc_self_fd =3D=3D -1) {
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, "open(/proc, = O_PATH): %m\n");
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Get the /proc (actually /proc/self/fd,= see above) file descriptor */
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 lo->proc_self_fd =3D open("/proc&= quot;, O_PATH);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (lo->proc_self_fd =3D=3D -1) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, &quo= t;open(/proc, O_PATH): %m\n");
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
> +=C2=A0 =C2=A0 } else {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Now we can get our /proc/self/fd direc= tory file descriptor */
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 lo->proc_self_fd =3D open("/proc/= self/fd", O_PATH);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (lo->proc_self_fd =3D=3D -1) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, &quo= t;open(/proc/self/fd, O_PATH): %m\n");
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
>=C2=A0 =C2=A0 =C2=A0 }
>=C2=A0 }
>=C2=A0
> @@ -3347,7 +3368,9 @@ static void setup_sandbox(struct lo_data *lo, st= ruct fuse_session *se,
>=C2=A0 {
>=C2=A0 =C2=A0 =C2=A0 if (lo->sandbox =3D=3D SANDBOX_NAMESPACE) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 setup_namespaces(lo, se);
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 setup_mounts(lo->source);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (lo->mount_ns) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 setup_mounts(lo->source)= ;
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
>=C2=A0 =C2=A0 =C2=A0 } else {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 setup_chroot(lo);
>=C2=A0 =C2=A0 =C2=A0 }
> @@ -3438,7 +3461,11 @@ static void setup_root(struct lo_data *lo, stru= ct lo_inode *root)
>=C2=A0 =C2=A0 =C2=A0 struct stat stat;
>=C2=A0 =C2=A0 =C2=A0 uint64_t mnt_id;
>=C2=A0
> -=C2=A0 =C2=A0 fd =3D open("/", O_PATH);
> +=C2=A0 =C2=A0 if (lo->mount_ns) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 fd =3D open("/", O_PATH);
> +=C2=A0 =C2=A0 } else {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 fd =3D open(lo->source, O_PATH);
> +=C2=A0 =C2=A0 }
>=C2=A0 =C2=A0 =C2=A0 if (fd =3D=3D -1) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fuse_log(FUSE_LOG_ERR, "open(%s= , O_PATH): %m\n", lo->source);
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> @@ -3515,6 +3542,9 @@ int main(int argc, char *argv[])
>=C2=A0 =C2=A0 =C2=A0 lo.allow_direct_io =3D 0,
>=C2=A0 =C2=A0 =C2=A0 lo.proc_self_fd =3D -1;
>=C2=A0
> +=C2=A0 =C2=A0 lo.reconnect =3D 0;
> +=C2=A0 =C2=A0 lo.mount_ns =3D 1;
> +
>=C2=A0 =C2=A0 =C2=A0 /* Don't mask creation mode, kernel already di= d that */
>=C2=A0 =C2=A0 =C2=A0 umask(0);
>=C2=A0
> @@ -3577,6 +3607,22 @@ int main(int argc, char *argv[])
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 goto err_out1;
>=C2=A0 =C2=A0 =C2=A0 }
>=C2=A0
> +=C2=A0 =C2=A0 /* For sandbox mode "chroot", set the mount_n= s option to 0. */
> +=C2=A0 =C2=A0 if (lo.sandbox =3D=3D SANDBOX_CHROOT) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 lo.mount_ns =3D 0;
> +=C2=A0 =C2=A0 }
> +
> +=C2=A0 =C2=A0 if (lo.reconnect) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 if ((lo.sandbox =3D=3D SANDBOX_NAMESPACE = && lo.mount_ns) || lo.flock
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0|| lo.posix_lock || lo.xattr) {
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf("Mount namespac= e, remote lock, and extended attributes"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= " are not supported by reconnection (-o reconnect). Please "
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= "specify=C2=A0 -o no_mount_ns, -o no_flock (default), -o"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= "no_posix_lock (default), and -o no_xattr (default).\n");
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 exit(1);
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
> +=C2=A0 =C2=A0 }
> +
>=C2=A0 =C2=A0 =C2=A0 if (opts.log_level !=3D 0) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 current_log_level =3D opts.log_level= ;
>=C2=A0 =C2=A0 =C2=A0 } else {
> --
> 2.20.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--00000000000070c47f05ba83586c--