[LSF/MM/BPF TOPIC] How to make disconnected operation work?

[David Howells <dhowells@redhat.com>]

I've been rewriting fscache and cachefiles to massively simplify it and make
use of the kiocb interface to do direct-I/O to/from the netfs's pages which
didn't exist when I first did this.

	https://lore.kernel.org/lkml/24942.1573667720@warthog.procyon.org.uk/
	https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=fscache-iter

I'm getting towards the point where it's working and able to do basic caching
once again.  So now I've been thinking about what it'd take to support
disconnected operation.  Here's a list of things that I think need to be
considered or dealt with:

 (1) Making sure the working set is present in the cache.

     - Userspace (find/cat/tar)
     - Splice netfs -> cache
     - Metadata storage (e.g. directories)
     - Permissions caching

 (2) Making sure the working set doesn't get culled.

     - Pinning API (cachectl() syscall?)
     - Allow culling to be disabled entirely on a cache
     - Per-fs/per-dir config

 (3) Switching into/out of disconnected mode.

     - Manual, automatic
     - On what granularity?
       - Entirety of fs (eg. all nfs)
       - By logical unit (server, volume, cell, share)

 (4) Local changes in disconnected mode.

     - Journal
     - File identifier allocation
     - statx flag to indicate provisional nature of info
     - New error codes
	- EDISCONNECTED - Op not available in disconnected mode
	- EDISCONDATA - Data not available in disconnected mode
	- EDISCONPERM - Permission cannot be checked in disconnected mode
	- EDISCONFULL - Disconnected mode cache full
     - SIGIO support?

 (5) Reconnection.

     - Proactive or JIT synchronisation
       - Authentication
     - Conflict detection and resolution
	 - ECONFLICTED - Disconnected mode resolution failed
     - Journal replay
     - Directory 'diffing' to find remote deletions
     - Symlink and other non-regular file comparison

 (6) Conflict resolution.

     - Automatic where possible
       - Just create/remove new non-regular files if possible
       - How to handle permission differences?
     - How to let userspace access conflicts?
       - Move local copy to 'lost+found'-like directory
       	 - Might not have been completely downloaded
       - New open() flags?
       	 - O_SERVER_VARIANT, O_CLIENT_VARIANT, O_RESOLVED_VARIANT
       - fcntl() to switch variants?

 (7) GUI integration.

     - Entering/exiting disconnected mode notification/switches.
     - Resolution required notification.
     - Cache getting full notification.

Can anyone think of any more considerations?  What do you think of the
proposed error codes and open flags?  Is that the best way to do this?

David