archive mirror
 help / color / mirror / Atom feed
From: David Howells <>
	Trond Myklebust <>,
	Anna Schumaker <>,
	Steve French <>
Subject: [LSF/MM/BPF TOPIC] How to make disconnected operation work?
Date: Mon, 09 Dec 2019 14:46:55 +0000	[thread overview]
Message-ID: <> (raw)

I've been rewriting fscache and cachefiles to massively simplify it and make
use of the kiocb interface to do direct-I/O to/from the netfs's pages which
didn't exist when I first did this.

I'm getting towards the point where it's working and able to do basic caching
once again.  So now I've been thinking about what it'd take to support
disconnected operation.  Here's a list of things that I think need to be
considered or dealt with:

 (1) Making sure the working set is present in the cache.

     - Userspace (find/cat/tar)
     - Splice netfs -> cache
     - Metadata storage (e.g. directories)
     - Permissions caching

 (2) Making sure the working set doesn't get culled.

     - Pinning API (cachectl() syscall?)
     - Allow culling to be disabled entirely on a cache
     - Per-fs/per-dir config

 (3) Switching into/out of disconnected mode.

     - Manual, automatic
     - On what granularity?
       - Entirety of fs (eg. all nfs)
       - By logical unit (server, volume, cell, share)

 (4) Local changes in disconnected mode.

     - Journal
     - File identifier allocation
     - statx flag to indicate provisional nature of info
     - New error codes
	- EDISCONNECTED - Op not available in disconnected mode
	- EDISCONDATA - Data not available in disconnected mode
	- EDISCONPERM - Permission cannot be checked in disconnected mode
	- EDISCONFULL - Disconnected mode cache full
     - SIGIO support?

 (5) Reconnection.

     - Proactive or JIT synchronisation
       - Authentication
     - Conflict detection and resolution
	 - ECONFLICTED - Disconnected mode resolution failed
     - Journal replay
     - Directory 'diffing' to find remote deletions
     - Symlink and other non-regular file comparison

 (6) Conflict resolution.

     - Automatic where possible
       - Just create/remove new non-regular files if possible
       - How to handle permission differences?
     - How to let userspace access conflicts?
       - Move local copy to 'lost+found'-like directory
       	 - Might not have been completely downloaded
       - New open() flags?
       - fcntl() to switch variants?

 (7) GUI integration.

     - Entering/exiting disconnected mode notification/switches.
     - Resolution required notification.
     - Cache getting full notification.

Can anyone think of any more considerations?  What do you think of the
proposed error codes and open flags?  Is that the best way to do this?


             reply	other threads:[~2019-12-09 14:47 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-09 14:46 David Howells [this message]
2019-12-09 17:33 ` [Lsf-pc] [LSF/MM/BPF TOPIC] How to make disconnected operation work? Amir Goldstein
2020-01-24 14:13   ` Amir Goldstein
2020-01-27 16:32   ` David Howells
2020-01-27 19:18     ` Amir Goldstein
2019-12-09 23:14 ` Jeff Layton
2020-03-06  7:11   ` Steven French

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).