Re: [LSF/MM/BPF TOPIC] How to make disconnected operation work?

[Steven French <sfrench@samba.org>]

As discussed in hallway discussions at Linux Storage Conference - this 
would make a good topic for LSF/MM

On 12/9/19 5:14 PM, Jeff Layton wrote:
> On Mon, 2019-12-09 at 14:46 +0000, David Howells wrote:
>> I've been rewriting fscache and cachefiles to massively simplify it and make
>> use of the kiocb interface to do direct-I/O to/from the netfs's pages which
>> didn't exist when I first did this.
>>
>> 	https://lore.kernel.org/lkml/24942.1573667720@warthog.procyon.org.uk/
>> 	https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=fscache-iter
>>
>> I'm getting towards the point where it's working and able to do basic caching
>> once again.  So now I've been thinking about what it'd take to support
>> disconnected operation.  Here's a list of things that I think need to be
>> considered or dealt with:
>>
> I'm quite interested in this too. I see that you've already given a lot
> of thought to potential interfaces here. I think we'll end up having to
> add a fair number of new interfaces to make something like this work.
>
>>   (1) Making sure the working set is present in the cache.
>>
>>       - Userspace (find/cat/tar)
>>       - Splice netfs -> cache
>>       - Metadata storage (e.g. directories)
>>       - Permissions caching
>>
>>   (2) Making sure the working set doesn't get culled.
>>
>>       - Pinning API (cachectl() syscall?)
>>       - Allow culling to be disabled entirely on a cache
>>       - Per-fs/per-dir config
>>
>>   (3) Switching into/out of disconnected mode.
>>
>>       - Manual, automatic
>>       - On what granularity?
>>         - Entirety of fs (eg. all nfs)
>>         - By logical unit (server, volume, cell, share)
>>
>>   (4) Local changes in disconnected mode.
>>
>>       - Journal
>>       - File identifier allocation
> Yep, necessary if you want to allow disconnected creates. By coincidence
> I'm working an (experimental) patchset now to add async create support
> to kcephfs, and part of that involves delegating out ranges of inode
> numbers. I may have some experience to report with it by the time LSF
> rolls around.
>
>>       - statx flag to indicate provisional nature of info
>>       - New error codes
>> 	- EDISCONNECTED - Op not available in disconnected mode
>> 	- EDISCONDATA - Data not available in disconnected mode
>> 	- EDISCONPERM - Permission cannot be checked in disconnected mode
>> 	- EDISCONFULL - Disconnected mode cache full
>>       - SIGIO support?
>>
>>   (5) Reconnection.
>>
>>       - Proactive or JIT synchronisation
>>         - Authentication
>>       - Conflict detection and resolution
>> 	 - ECONFLICTED - Disconnected mode resolution failed
> ECONFLICTED sort of implies that reconnection will be manual. If it
> happens automagically in the background you'll have no way to report
> such errors.
>
> Also, you'll need some mechanism to know what inodes are conflicted.
> This is the real difficult part of this problem, IMO.
>
>
>>       - Journal replay
>>       - Directory 'diffing' to find remote deletions
>>       - Symlink and other non-regular file comparison
>>
>>   (6) Conflict resolution.
>>
>>       - Automatic where possible
>>         - Just create/remove new non-regular files if possible
>>         - How to handle permission differences?
>>       - How to let userspace access conflicts?
>>         - Move local copy to 'lost+found'-like directory
>>         	 - Might not have been completely downloaded
>>         - New open() flags?
>>         	 - O_SERVER_VARIANT, O_CLIENT_VARIANT, O_RESOLVED_VARIANT
>>         - fcntl() to switch variants?
>>
> Again, conflict resolution is the difficult part. Maybe the right
> solution is to look at snapshotting-style interfaces -- i.e., handle a
> disconnected mount sort of like you would a writable snapshot. Do any
> (local) fs' currently offer writable snapshots, btw?
>
>>   (7) GUI integration.
>>
>>       - Entering/exiting disconnected mode notification/switches.
>>       - Resolution required notification.
>>       - Cache getting full notification.
>>
>> Can anyone think of any more considerations?  What do you think of the
>> proposed error codes and open flags?  Is that the best way to do this?
>>
>> David
>>