From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2833EC47082 for ; Mon, 31 May 2021 15:18:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 09E6D6124B for ; Mon, 31 May 2021 15:18:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232622AbhEaPUU (ORCPT ); Mon, 31 May 2021 11:20:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232670AbhEaPSK (ORCPT ); Mon, 31 May 2021 11:18:10 -0400 Received: from mail-ua1-x934.google.com (mail-ua1-x934.google.com [IPv6:2607:f8b0:4864:20::934]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96132C00366F for ; Mon, 31 May 2021 07:11:56 -0700 (PDT) Received: by mail-ua1-x934.google.com with SMTP id w5so6639637uaq.9 for ; Mon, 31 May 2021 07:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=szeredi.hu; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vNIjVp5U1cElMCI70hXwJr7OYzhPLEPInOFf3Mw2iYE=; b=c3SzenalqbMH7AkuKol5QqFTE9dxA+I/YNAdrZqmCT/jg2NUrKdOvNsqrmKwZMnwv/ 0UDObdoyi7Jp3K3PoRmd8MJcE+XRqDmEwRXm2k+saMLoilMYKyJTnBMFPYI2lTtfcKUQ uMtI2YomqxK+1QI9z2ZrDx0L7fXQ9TwUlrs4o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vNIjVp5U1cElMCI70hXwJr7OYzhPLEPInOFf3Mw2iYE=; b=Dl7rEYb2ABP+wc+k/uC8akbsdCR5AZnTWeWBA51TYVKuEbBdmiiELspOIAOLqjCQ+H QmI6HgRIoQGEnxq9/VH0WuTVTf+qX8nep0nnyntvePIkjRlwF7AgRglZq1QX61AecczK B7esp8YJoWePlzY+dFsAWQFP8cNxqj7QrwxHoz4IUtrlZVHhGZ0t6D5ljQPoXt6Em9mh TbF3U2Q9TCKqo/Hv/LIjGZRVXPKQsZM5pcIjXhhm6gfv2tJSI0ZkObuvOv8mrMRl9zUu jnBsGXjt2LjEuJz+maPVk9VzZ7J2fnl9fLKZUC/lWuMosQbi7B4CSB2k1BrWSktLiGGx wSyQ== X-Gm-Message-State: AOAM530WHxZCa6Lv1Lq2Gis10+CqnfCXDfitM62qNt3c/gicZ4MZXJvb pEWapY0fhm0A+V2RDmMjkAk5KMCi7o2sg0z5uVSUHw== X-Google-Smtp-Source: ABdhPJyRNqC7aXSy3nKyxDNn+avdFJ/rdx4p4YQsJJ39xUbRg965T8uztjxLiE6kHR51P7ncUlC5z9PU0kW2PSRvMmE= X-Received: by 2002:a1f:30cd:: with SMTP id w196mr12917140vkw.3.1622470315793; Mon, 31 May 2021 07:11:55 -0700 (PDT) MIME-Version: 1.0 References: <1bb71cbf-0a10-34c7-409d-914058e102f6@virtuozzo.com> <20200922210445.GG57620@redhat.com> In-Reply-To: From: Miklos Szeredi Date: Mon, 31 May 2021 16:11:45 +0200 Message-ID: Subject: Re: virtiofs uuid and file handles To: Amir Goldstein Cc: Vivek Goyal , overlayfs , linux-fsdevel , Max Reitz Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Sat, 29 May 2021 at 18:05, Amir Goldstein wrote: > > On Wed, Sep 23, 2020 at 2:12 PM Miklos Szeredi wrote: > > > > On Wed, Sep 23, 2020 at 11:57 AM Amir Goldstein wrote: > > > > > > On Wed, Sep 23, 2020 at 10:44 AM Miklos Szeredi wrote: > > > > > > > > On Wed, Sep 23, 2020 at 4:49 AM Amir Goldstein wrote: > > > > > > > > > I think that the proper was to implement reliable persistent file > > > > > handles in fuse/virtiofs would be to add ENCODE/DECODE to > > > > > FUSE protocol and allow the server to handle this. > > > > > > > > Max Reitz (Cc-d) is currently looking into this. > > > > > > > > One proposal was to add LOOKUP_HANDLE operation that is similar to > > > > LOOKUP except it takes a {variable length handle, name} as input and > > > > returns a variable length handle *and* a u64 node_id that can be used > > > > normally for all other operations. > > > > > > Miklos, Max, > > Any updates on LOOKUP_HANDLE work? > > > > > The advantage of such a scheme for virtio-fs (and possibly other fuse > > > > based fs) would be that userspace need not keep a refcounted object > > > > around until the kernel sends a FORGET, but can prune its node ID > > > > based cache at any time. If that happens and a request from the > > > > client (kernel) comes in with a stale node ID, the server will return > > > > -ESTALE and the client can ask for a new node ID with a special > > > > lookup_handle(fh, NULL). > > > > > > > > Disadvantages being: > > > > > > > > - cost of generating a file handle on all lookups > > > > > > I never ran into a local fs implementation where this was expensive. > > > > > > > - cost of storing file handle in kernel icache > > > > > > > > I don't think either of those are problematic in the virtiofs case. > > > > The cost of having to keep fds open while the client has them in its > > > > cache is much higher. > > > > > > > > > > Sounds good. > > > I suppose flock() does need to keep the open fd on server. > > > > Open files are a separate issue and do need an active object in the server. > > > > The issue this solves is synchronizing "released" and "evicted" > > states of objects between server and client. I.e. when a file is > > closed (and no more open files exist referencing the same object) the > > dentry refcount goes to zero but it remains in the cache. In this > > state the server could really evict it's own cached object, but can't > > because the client can gain an active reference at any time via > > cached path lookup. > > > > One other solution would be for the server to send a notification > > (NOTIFY_EVICT) that would try to clean out the object from the server > > cache and respond with a FORGET if successful. But I sort of like > > the file handle one better, since it solves multiple problems. > > > > Even with LOOKUP_HANDLE, I am struggling to understand how we > intend to invalidate all fuse dentries referring to ino X in case the server > replies with reused ino X with a different generation that the one stored > in fuse inode cache. > > This is an issue that I encountered when running the passthrough_hp test, > on my filesystem. In tst_readdir_big() for example, underlying files are being > unlinked and new files created reusing the old inode numbers. > > This creates a situation where server gets a lookup request > for file B that uses the reused inode number X, while old file A is > still in fuse dentry cache using the older generation of real inode > number X which is still in fuse inode cache. > > Now the server knows that the real inode has been rused, because > the server caches the old generation value, but it cannot reply to > the lookup request before the old fuse inode has been invalidated. > IIUC, fuse_lowlevel_notify_inval_inode() is not enough(?). > We would also need to change fuse_dentry_revalidate() to > detect the case of reused/invalidated inode. > > The straightforward way I can think of is to store inode generation > in fuse_dentry. It won't even grow the size of the struct. > > Am I over complicating this? In this scheme the generation number is already embedded in the file handle. If LOOKUP_HANDLE returns a nodeid that can be found in the icache, but which doesn't match the new file handle, then the old inode will be marked bad and a new one allocated. Does that answer your worries? Or am I missing something? Thanks, Miklos