[fuse] getattr() results ignored when writeback cache is active

[fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

Hi,

I'm having another problem with FUSE's writeback cache in SSHFS.

As far as I can tell, the FUSE kernel module issues getattr() requests,
but then silently discards the reported mtime and file size.

For SSHFS, this means that if a file has been accessed, and is then
changed on the server, the changed attributes don't make it to the
client and the file appears truncated or \0-filled.

To me this looks like a bug.. am I missing something?

Steps to reproduce (with SSHFS from Git master at
https://github.com/libfuse/sshfs/ and the debugging patch from
https://github.com/libfuse/sshfs/issues/93):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: issue_93.sh --]
[-- Type: text/x-sh, Size: 734 bytes --]

mkdir /tmp/issue_93
touch /tmp/issue_93/file_1
mkdir /tmp/issue_93_mnt

dd if=/dev/urandom count=1 bs=2048 |base64 > /tmp/issue_93/file_1 2>/dev/null

./sshfs -f -o attr_timeout=0,entry_timeout=0,dir_cache=no,writeback_cache=yes \
        localhost:/tmp/issue_93/ /tmp/issue_93_mnt/ &
sleep 1
stat --format='Kernel: mtime %Y, size %s for %n' /tmp/issue_93_mnt/file_1
stat --format='Local:  mtime %Y, size %s for %n' /tmp/issue_93/file_1

sleep 1
echo "Changing file.."
echo -e "\nrevision 2" >> /tmp/issue_93/file_1

stat --format='Kernel: mtime %Y, size %s for %n' /tmp/issue_93_mnt/file_1
stat --format='Local:  mtime %Y, size %s for %n' /tmp/issue_93/file_1

fusermount -u /tmp/issue_93_mnt/

rm -r /tmp/issue_93 /tmp/issue_93_mnt

[-- Attachment #3: Type: text/plain, Size: 1044 bytes --]


,----
| $ ../issue_93.sh 
| 1+0 records in
| 1+0 records out
| 2048 bytes (2.0 kB, 2.0 KiB) copied, 5.3011e-05 s, 38.6 MB/s
| SSHFS:  mtime 0, size 0 for /.Trash
| SSHFS:  mtime 0, size 0 for /.Trash-1000
| SSHFS:  mtime 1505908107, size 2768 for /file_1
| SSHFS:  mtime 1505908107, size 2768 for /file_1
| Kernel: mtime 1505908107, size 2768 for /tmp/issue_93_mnt/file_1
| Local:  mtime 1505908107, size 2768 for /tmp/issue_93/file_1
| Changing file..
| SSHFS:  mtime 1505908109, size 2780 for /file_1
| SSHFS:  mtime 1505908109, size 2780 for /file_1
| Kernel: mtime 1505908107, size 2768 for /tmp/issue_93_mnt/file_1
| Local:  mtime 1505908109, size 2780 for /tmp/issue_93/file_1
`----

Note how sshfs' getattr handler returns the correct mtime and ctime for
the second stat() call, but the kernel returns the old values. 

This is with kernel 4.9.

Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse] getattr() results ignored when writeback cache is active

[Miklos Szeredi]

On Wed, Sep 20, 2017 at 1:50 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
> Hi,
>
> I'm having another problem with FUSE's writeback cache in SSHFS.
>
> As far as I can tell, the FUSE kernel module issues getattr() requests,
> but then silently discards the reported mtime and file size.
>
> For SSHFS, this means that if a file has been accessed, and is then
> changed on the server, the changed attributes don't make it to the
> client and the file appears truncated or \0-filled.
>
> To me this looks like a bug.. am I missing something?

Writeback cache assumes that the file is never changed outside the
mounted filesystem, so it's not suitable for any network fs currently.

Apparently the above is not documented anywhere :(

Thanks,
Miklos

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
> On Wed, Sep 20, 2017 at 1:50 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
>> Hi,
>>
>> I'm having another problem with FUSE's writeback cache in SSHFS.
>>
>> As far as I can tell, the FUSE kernel module issues getattr() requests,
>> but then silently discards the reported mtime and file size.
>>
>> For SSHFS, this means that if a file has been accessed, and is then
>> changed on the server, the changed attributes don't make it to the
>> client and the file appears truncated or \0-filled.
>>
>> To me this looks like a bug.. am I missing something?
>
> Writeback cache assumes that the file is never changed outside the
> mounted filesystem, so it's not suitable for any network fs currently.
>
> Apparently the above is not documented anywhere :(

Ouch.

I will of course put this into the libfuse documentation, but it would
be much nicer if things like that could be documented somewhere in the
kernel. After all, these are not properties of libfuse but the kernel
module - and some filesystems use the fuse interface without using
libfuse.

(This actually applies to large chunk of information that's currently
only in the libfuse documentation).

Any chance of that happening? I understand that bringing
Documentation/filesystems/fuse.txt up to date would be a major
endeavour, but maybe one could just start by putting at least this
information into e.g. a new Documentation/filesystems/fuse/writeback.txt
file? Together with the requirement that the filesystem has to support
reading from files opened O_WRONLY?


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Miklos Szeredi]

On Wed, Sep 20, 2017 at 5:31 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>> On Wed, Sep 20, 2017 at 1:50 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
>>> Hi,
>>>
>>> I'm having another problem with FUSE's writeback cache in SSHFS.
>>>
>>> As far as I can tell, the FUSE kernel module issues getattr() requests,
>>> but then silently discards the reported mtime and file size.
>>>
>>> For SSHFS, this means that if a file has been accessed, and is then
>>> changed on the server, the changed attributes don't make it to the
>>> client and the file appears truncated or \0-filled.
>>>
>>> To me this looks like a bug.. am I missing something?
>>
>> Writeback cache assumes that the file is never changed outside the
>> mounted filesystem, so it's not suitable for any network fs currently.
>>
>> Apparently the above is not documented anywhere :(
>
> Ouch.
>
> I will of course put this into the libfuse documentation, but it would
> be much nicer if things like that could be documented somewhere in the
> kernel. After all, these are not properties of libfuse but the kernel
> module - and some filesystems use the fuse interface without using
> libfuse.
>
> (This actually applies to large chunk of information that's currently
> only in the libfuse documentation).
>
> Any chance of that happening? I understand that bringing
> Documentation/filesystems/fuse.txt up to date would be a major
> endeavour, but maybe one could just start by putting at least this
> information into e.g. a new Documentation/filesystems/fuse/writeback.txt
> file? Together with the requirement that the filesystem has to support
> reading from files opened O_WRONLY?

Something like this?


Fuse supports the following writeback modes:

- direct-io
- write-through
- writeback-cache

In direct-io mode (enabled by the FOPEN_DIRECT_IO flag) the page cache is
completely bypassed for reads and writes.

In write-through mode (default) each write modifies the page cache as well as
immediately being sent to userspace.

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
the cache only, which means that the write(2) syscall can often complete very
fast.  The dirty pages are later sent to userspace using write requests.  This
mode assumes that the file is never changed outside the mounted filesystem, so
it's not suitable for any network fs.  If a partial page is written, then the
page needs to be first read from userspace.  This means, thet even for files
opened for O_WRONLY it is possible that read requests will be generated by the
kernel.

Thanks,
Miklos

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>> Writeback cache assumes that the file is never changed outside the
>>> mounted filesystem, so it's not suitable for any network fs currently.
>>>
>>> Apparently the above is not documented anywhere :(
>>
>> Ouch.
>>
>> I will of course put this into the libfuse documentation, but it would
>> be much nicer if things like that could be documented somewhere in the
>> kernel. After all, these are not properties of libfuse but the kernel
>> module - and some filesystems use the fuse interface without using
>> libfuse.
>>
>> (This actually applies to large chunk of information that's currently
>> only in the libfuse documentation).
>>
>> Any chance of that happening? I understand that bringing
>> Documentation/filesystems/fuse.txt up to date would be a major
>> endeavour, but maybe one could just start by putting at least this
>> information into e.g. a new Documentation/filesystems/fuse/writeback.txt
>> file? Together with the requirement that the filesystem has to support
>> reading from files opened O_WRONLY?
>
> Something like this?

Anything is better than nothing. Well, almost anything, but that
definitely includes your draft :-).

> Fuse supports the following writeback modes:

"write modes", or maybe "access modes", not "writeback modes", right?

> - direct-io
> - write-through
> - writeback-cache
>
> In direct-io mode (enabled by the FOPEN_DIRECT_IO flag) the page cache is
> completely bypassed for reads and writes.

I'd add "No read-ahead takes place." (if that's true)

>
> In write-through mode (default) each write modifies the page cache as well as
> immediately being sent to userspace.

Reads may be satisfied from the page cache, and data may be read-ahead
by the kernel to fill the page-cache.

> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga)
> writes go to

Typo in "flaga"

> the cache only, which means that the write(2) syscall can often complete very
> fast.  The dirty pages are later sent to userspace using write
> requests.

I think it would be great to be more specific than "later". Maybe "when
the file is no longer opened by any process, when explicitly requested
by an fsync() call, or when the dirty data is evicted from the cache by
the kernel for other reasons" (if that's correct).

> This
> mode assumes that the file is never changed outside the mounted filesystem, so
> it's not suitable for any network fs.  If a partial page is written, then the
> page needs to be first read from userspace.  This means, thet even for
> files

"that", not "thet"

> opened for O_WRONLY it is possible that read requests will be generated by the
> kernel.

Nice!


Thanks!
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
> the cache only, which means that the write(2) syscall can often complete very
> fast.  The dirty pages are later sent to userspace using write requests.  This
> mode assumes that the file is never changed outside the mounted filesystem, so
> it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem
even if no write operations are actually carried out".


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Miklos Szeredi]

On Wed, Sep 20, 2017 at 6:37 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>>> Writeback cache assumes that the file is never changed outside the
>>>> mounted filesystem, so it's not suitable for any network fs currently.
>>>>
>>>> Apparently the above is not documented anywhere :(
>>>
>>> Ouch.
>>>
>>> I will of course put this into the libfuse documentation, but it would
>>> be much nicer if things like that could be documented somewhere in the
>>> kernel. After all, these are not properties of libfuse but the kernel
>>> module - and some filesystems use the fuse interface without using
>>> libfuse.
>>>
>>> (This actually applies to large chunk of information that's currently
>>> only in the libfuse documentation).
>>>
>>> Any chance of that happening? I understand that bringing
>>> Documentation/filesystems/fuse.txt up to date would be a major
>>> endeavour, but maybe one could just start by putting at least this
>>> information into e.g. a new Documentation/filesystems/fuse/writeback.txt
>>> file? Together with the requirement that the filesystem has to support
>>> reading from files opened O_WRONLY?
>>
>> Something like this?
>
> Anything is better than nothing. Well, almost anything, but that
> definitely includes your draft :-).

Thanks for your feedback.  Fixed and expanded version:

Fuse supports the following I/O modes:

- direct-io
- cached
  + write-through
  + writeback-cache

The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the
FUSE_OPEN reply.

In direct-io mode the page cache is completely bypassed for reads and writes.
No read-ahead takes place. Shared mmap is disabled.

In cached mode reads may be satisfied from the page cache, and data may be
read-ahead by the kernel to fill the cache.  The cache is always kept consistent
after any writes to the file.  All mmap modes are supported.

The cached mode has two sub modes controlling how writes are handled.  The
write-through mode is the default and is supported on all kernels.  The
writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the
FUSE_INIT reply.

In write-through mode each write is immediately sent to userspace as one or more
WRITE requests, as well as updating any cached pages (and caching previously
uncached, but fully written pages).  No READ requests are ever sent for writes,
so when an uncached page is partially written, the page is discarded.

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to
the cache only, which means that the write(2) syscall can often complete very
fast.  Dirty pages are written back implicitly (background writeback or page
reclaim on memory pressure) or explicitly (invoked by fsync(2)).  This mode
assumes that the file is never changed outside the mounted filesystem (the size
time attributes are kept uptodate by the kernel), so it's not suitable for any
any network filesystem even if no write operations are actually carried out.  If
a partial page is written, then the page needs to be first read from userspace.
This means, that even for files opened for O_WRONLY it is possible that read
requests will be generated by the kernel.

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Maxim Patlasov]

On 09/21/2017 03:12 AM, Nikolaus Rath wrote:

> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
>> the cache only, which means that the write(2) syscall can often complete very
>> fast.  The dirty pages are later sent to userspace using write requests.  This
>> mode assumes that the file is never changed outside the mounted filesystem, so
>> it's not suitable for any network fs.
> .."this mode of operation is not suitable for any network filesystem
> even if no write operations are actually carried out".

Not true. A network filesystem can guarantee that the file is never 
changed outside by implementing exclusive write lease semantics: when 
someone opens file for writing first time the metadata server grants 
exclusive rights for that mount, then declines all subsequent open 
requests from other mounts; and similarly while a file is being kept 
opened for reading, the metadata server declines all open-for-writing 
requests from other mounts.

>
>
> Best,
> -Nikolaus
>

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
> On 09/21/2017 03:12 AM, Nikolaus Rath wrote:
>
>> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
>>> the cache only, which means that the write(2) syscall can often complete very
>>> fast.  The dirty pages are later sent to userspace using write requests.  This
>>> mode assumes that the file is never changed outside the mounted filesystem, so
>>> it's not suitable for any network fs.
>> .."this mode of operation is not suitable for any network filesystem
>> even if no write operations are actually carried out".
>
> Not true. A network filesystem can guarantee that the file is never
> changed outside by implementing exclusive write lease semantics: when
> someone opens file for writing first time the metadata server grants
> exclusive rights for that mount, then declines all subsequent open
> requests from other mounts; and similarly while a file is being kept
> opened for reading, the metadata server declines all open-for-writing
> requests from other mounts.

In practice that doesn't seem to work, see the example in my first
message. The file is only ever accessed on one mount at a time, yet the
changes do not propagate (and would result in data corruption if another
mount would attempt to read or modify the file afterwards).


Best,
-Nikolaus
-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 21 2017, Miklos Szeredi <miklos@szeredi.hu> wrote:
> Thanks for your feedback.  Fixed and expanded version:
>
> Fuse supports the following I/O modes:
>
> - direct-io
> - cached
>   + write-through
>   + writeback-cache
>
> The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the
> FUSE_OPEN reply.
>
> In direct-io mode the page cache is completely bypassed for reads and writes.
> No read-ahead takes place. Shared mmap is disabled.
>
> In cached mode reads may be satisfied from the page cache, and data may be
> read-ahead by the kernel to fill the cache.  The cache is always kept consistent
> after any writes to the file.  All mmap modes are supported.
>
> The cached mode has two sub modes controlling how writes are handled.  The
> write-through mode is the default and is supported on all kernels.  The
> writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the
> FUSE_INIT reply.
>
> In write-through mode each write is immediately sent to userspace as one or more
> WRITE requests, as well as updating any cached pages (and caching previously
> uncached, but fully written pages).  No READ requests are ever sent for writes,
> so when an uncached page is partially written, the page is discarded.
>
> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to
> the cache only, which means that the write(2) syscall can often complete very
> fast.  Dirty pages are written back implicitly (background writeback or page
> reclaim on memory pressure) or explicitly (invoked by fsync(2)).

I think it would be useful to mention if the cache is flushed when the
file is not opened anymore, or (if not) to explicitly say that that's
not the case.

> This mode assumes that the file is never changed outside the mounted
> filesystem

This sounds a little awkward. How about "assumes that all changes to the
filesystem go through the FUSE kernel module"

> (the size time attributes are kept uptodate by the kernel),

"size and atime/ctime/mtime attributes", and I think "uptodate" should
be up-to-date.

> so it's not suitable for any any network filesystem

duplicated "any"


I think this is great. I may work on similar documentation for some of
the other stuff that currently lives only in include/fuse*.h


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Maxim Patlasov]

On 09/21/2017 10:53 AM, Nikolaus Rath wrote:

> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>> On 09/21/2017 03:12 AM, Nikolaus Rath wrote:
>>
>>> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>>> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
>>>> the cache only, which means that the write(2) syscall can often complete very
>>>> fast.  The dirty pages are later sent to userspace using write requests.  This
>>>> mode assumes that the file is never changed outside the mounted filesystem, so
>>>> it's not suitable for any network fs.
>>> .."this mode of operation is not suitable for any network filesystem
>>> even if no write operations are actually carried out".
>> Not true. A network filesystem can guarantee that the file is never
>> changed outside by implementing exclusive write lease semantics: when
>> someone opens file for writing first time the metadata server grants
>> exclusive rights for that mount, then declines all subsequent open
>> requests from other mounts; and similarly while a file is being kept
>> opened for reading, the metadata server declines all open-for-writing
>> requests from other mounts.
> In practice that doesn't seem to work, see the example in my first
> message. The file is only ever accessed on one mount at a time, yet the
> changes do not propagate (and would result in data corruption if another
> mount would attempt to read or modify the file afterwards).

In your example you modified file bypassing fuse mount:

 > echo -e "\nrevision 2" >> /tmp/issue_93/file_1

That's like mangling block device while ordinary local fs is running on 
top of it.

>
>
> Best,
> -Nikolaus

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
> On 09/21/2017 10:53 AM, Nikolaus Rath wrote:
>
>> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>> On 09/21/2017 03:12 AM, Nikolaus Rath wrote:
>>>
>>>> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>>>> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
>>>>> the cache only, which means that the write(2) syscall can often complete very
>>>>> fast.  The dirty pages are later sent to userspace using write requests.  This
>>>>> mode assumes that the file is never changed outside the mounted filesystem, so
>>>>> it's not suitable for any network fs.
>>>> .."this mode of operation is not suitable for any network filesystem
>>>> even if no write operations are actually carried out".
>>> Not true. A network filesystem can guarantee that the file is never
>>> changed outside by implementing exclusive write lease semantics: when
>>> someone opens file for writing first time the metadata server grants
>>> exclusive rights for that mount, then declines all subsequent open
>>> requests from other mounts; and similarly while a file is being kept
>>> opened for reading, the metadata server declines all open-for-writing
>>> requests from other mounts.
>> In practice that doesn't seem to work, see the example in my first
>> message. The file is only ever accessed on one mount at a time, yet the
>> changes do not propagate (and would result in data corruption if another
>> mount would attempt to read or modify the file afterwards).
>
> In your example you modified file bypassing fuse mount:
>
>> echo -e "\nrevision 2" >> /tmp/issue_93/file_1
>
> That's like mangling block device while ordinary local fs is running
> on top of it.

Huh? That's exactly what a network filesystem would see when another
host writes to the filesystem. The change doesn't come through the local
FUSE layer.


Best,
-Nikolaus
-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Maxim Patlasov]

On 09/21/2017 11:31 AM, Nikolaus Rath wrote:

> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>> On 09/21/2017 10:53 AM, Nikolaus Rath wrote:
>>
>>> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>>> On 09/21/2017 03:12 AM, Nikolaus Rath wrote:
>>>>
>>>>> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>>>>> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
>>>>>> the cache only, which means that the write(2) syscall can often complete very
>>>>>> fast.  The dirty pages are later sent to userspace using write requests.  This
>>>>>> mode assumes that the file is never changed outside the mounted filesystem, so
>>>>>> it's not suitable for any network fs.
>>>>> .."this mode of operation is not suitable for any network filesystem
>>>>> even if no write operations are actually carried out".
>>>> Not true. A network filesystem can guarantee that the file is never
>>>> changed outside by implementing exclusive write lease semantics: when
>>>> someone opens file for writing first time the metadata server grants
>>>> exclusive rights for that mount, then declines all subsequent open
>>>> requests from other mounts; and similarly while a file is being kept
>>>> opened for reading, the metadata server declines all open-for-writing
>>>> requests from other mounts.
>>> In practice that doesn't seem to work, see the example in my first
>>> message. The file is only ever accessed on one mount at a time, yet the
>>> changes do not propagate (and would result in data corruption if another
>>> mount would attempt to read or modify the file afterwards).
>> In your example you modified file bypassing fuse mount:
>>
>>> echo -e "\nrevision 2" >> /tmp/issue_93/file_1
>> That's like mangling block device while ordinary local fs is running
>> on top of it.
> Huh? That's exactly what a network filesystem would see when another
> host writes to the filesystem. The change doesn't come through the local
> FUSE layer.

In your example the change didn't come through any FUSE layer. When 
another host writes to a network filesystem, the filesystem has a choice 
to process the request intelligently. In your example, you didn't give 
FUSE a chance because "/tmp/issue_93" was not fuse mount.

>
>
> Best,
> -Nikolaus

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
> On 09/21/2017 11:31 AM, Nikolaus Rath wrote:
>
>> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>> On 09/21/2017 10:53 AM, Nikolaus Rath wrote:
>>>
>>>> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>>>> On 09/21/2017 03:12 AM, Nikolaus Rath wrote:
>>>>>
>>>>>> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>>>>>> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
>>>>>>> the cache only, which means that the write(2) syscall can often complete very
>>>>>>> fast.  The dirty pages are later sent to userspace using write requests.  This
>>>>>>> mode assumes that the file is never changed outside the mounted filesystem, so
>>>>>>> it's not suitable for any network fs.
>>>>>> .."this mode of operation is not suitable for any network filesystem
>>>>>> even if no write operations are actually carried out".
>>>>> Not true. A network filesystem can guarantee that the file is never
>>>>> changed outside by implementing exclusive write lease semantics: when
>>>>> someone opens file for writing first time the metadata server grants
>>>>> exclusive rights for that mount, then declines all subsequent open
>>>>> requests from other mounts; and similarly while a file is being kept
>>>>> opened for reading, the metadata server declines all open-for-writing
>>>>> requests from other mounts.
>>>> In practice that doesn't seem to work, see the example in my first
>>>> message. The file is only ever accessed on one mount at a time, yet the
>>>> changes do not propagate (and would result in data corruption if another
>>>> mount would attempt to read or modify the file afterwards).
>>> In your example you modified file bypassing fuse mount:
>>>
>>>> echo -e "\nrevision 2" >> /tmp/issue_93/file_1
>>> That's like mangling block device while ordinary local fs is running
>>> on top of it.
>> Huh? That's exactly what a network filesystem would see when another
>> host writes to the filesystem. The change doesn't come through the local
>> FUSE layer.
>
> In your example the change didn't come through any FUSE layer.

It did. The filesystem reported the updated mtime and size in its
getattr() response. The kernel ignored it.

> When another host writes to a network filesystem, the filesystem has a
> choice to process the request intelligently. In your example, you
> didn't give FUSE a chance because "/tmp/issue_93" was not fuse mount.

Right. /tmp/issue_93 substitutes for the network. The filesystem gets
notification through the network, aka by reading this mountpoint. It
then attempts to pass this on to the FUSE layer, but is being ignored.

Best,
-Nikolaus


-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Maxim Patlasov]

On 09/21/2017 12:21 PM, Nikolaus Rath wrote:

> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>> On 09/21/2017 11:31 AM, Nikolaus Rath wrote:
>>
>>> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>>> On 09/21/2017 10:53 AM, Nikolaus Rath wrote:
>>>>
>>>>> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>>>>> On 09/21/2017 03:12 AM, Nikolaus Rath wrote:
>>>>>>
>>>>>>> On Sep 20 2017, Miklos Szeredi <mszeredi@redhat.com> wrote:
>>>>>>>> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to
>>>>>>>> the cache only, which means that the write(2) syscall can often complete very
>>>>>>>> fast.  The dirty pages are later sent to userspace using write requests.  This
>>>>>>>> mode assumes that the file is never changed outside the mounted filesystem, so
>>>>>>>> it's not suitable for any network fs.
>>>>>>> .."this mode of operation is not suitable for any network filesystem
>>>>>>> even if no write operations are actually carried out".
>>>>>> Not true. A network filesystem can guarantee that the file is never
>>>>>> changed outside by implementing exclusive write lease semantics: when
>>>>>> someone opens file for writing first time the metadata server grants
>>>>>> exclusive rights for that mount, then declines all subsequent open
>>>>>> requests from other mounts; and similarly while a file is being kept
>>>>>> opened for reading, the metadata server declines all open-for-writing
>>>>>> requests from other mounts.
>>>>> In practice that doesn't seem to work, see the example in my first
>>>>> message. The file is only ever accessed on one mount at a time, yet the
>>>>> changes do not propagate (and would result in data corruption if another
>>>>> mount would attempt to read or modify the file afterwards).
>>>> In your example you modified file bypassing fuse mount:
>>>>
>>>>> echo -e "\nrevision 2" >> /tmp/issue_93/file_1
>>>> That's like mangling block device while ordinary local fs is running
>>>> on top of it.
>>> Huh? That's exactly what a network filesystem would see when another
>>> host writes to the filesystem. The change doesn't come through the local
>>> FUSE layer.
>> In your example the change didn't come through any FUSE layer.
> It did. The filesystem reported the updated mtime and size in its
> getattr() response. The kernel ignored it.
>
>> When another host writes to a network filesystem, the filesystem has a
>> choice to process the request intelligently. In your example, you
>> didn't give FUSE a chance because "/tmp/issue_93" was not fuse mount.
> Right. /tmp/issue_93 substitutes for the network. The filesystem gets
> notification through the network, aka by reading this mountpoint. It
> then attempts to pass this on to the FUSE layer, but is being ignored.

Oh, I understand now. Your example actually demonstrates that 
FUSE_WRITEBACK_CACHE must not be used if it's possible to modify 
underlying data (/tmp/issue_93/file_1) externally w.r.t fuse filesystem. 
That's correct. But initial statement:

 > this mode of operation is not suitable for any network filesystem 
even if no write operations are actually carried out

is not, because it's not impossible to protect underlying data against 
such an external change somehow. A network filesystem is not obliged to 
keep transparent correspondence like /tmp/issue_93_mnt/file_1 <--> 
/tmp/issue_93/file_1. It can keep all user data (and metadata) in 
internal structures making any external modifications impossible. Then, 
implementing exclusive write semantics would make using 
FUSE_WRITEBACK_CACHE safe. Agree?


>
> Best,
> -Nikolaus
>
>

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
> Oh, I understand now. Your example actually demonstrates that
> FUSE_WRITEBACK_CACHE must not be used if it's possible to modify
> underlying data (/tmp/issue_93/file_1) externally w.r.t fuse
> filesystem. That's correct. But initial statement:
>
>> this mode of operation is not suitable for any network filesystem 
>> even if no write operations are actually carried out
>
> is not, because it's not impossible to protect underlying data against
> such an external change somehow. A network filesystem is not obliged
> to keep transparent correspondence like /tmp/issue_93_mnt/file_1 <--> 
> /tmp/issue_93/file_1. It can keep all user data (and metadata) in
> internal structures making any external modifications
> impossible. Then, implementing exclusive write semantics would make
> using FUSE_WRITEBACK_CACHE safe. Agree?

I agree with what you're saying, but I don't agree that this should go
into the documentation for which the sentence in dispute was intended
:-). I think in 93% of all cases, the simplified statement will be more
helpful to a potential FUSE user than the correct one.

If this bothers you a lot, can we compromise and just be a little more
vague overall? I.e.,

| This mode of operation is therefore generally not suitable for
| filesystems where modifications may not come through the (local) FUSE
| layer (i.e, most network filesystems).


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Miklos Szeredi]

Updated version.

Thanks,
Miklos
---

Fuse supports the following I/O modes:

- direct-io
- cached
  + write-through
  + writeback-cache

The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the
FUSE_OPEN reply.

In direct-io mode the page cache is completely bypassed for reads and writes.
No read-ahead takes place. Shared mmap is disabled.

In cached mode reads may be satisfied from the page cache, and data may be
read-ahead by the kernel to fill the cache.  The cache is always kept consistent
after any writes to the file.  All mmap modes are supported.

The cached mode has two sub modes controlling how writes are handled.  The
write-through mode is the default and is supported on all kernels.  The
writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the
FUSE_INIT reply.

In write-through mode each write is immediately sent to userspace as one or more
WRITE requests, as well as updating any cached pages (and caching previously
uncached, but fully written pages).  No READ requests are ever sent for writes,
so when an uncached page is partially written, the page is discarded.

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to
the cache only, which means that the write(2) syscall can often complete very
fast.  Dirty pages are written back implicitly (background writeback or page
reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and
when the last ref to the file is being released on munmap(2)).  This mode
assumes that all changes to the filesystem go through the FUSE kernel module
(size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so
it's generally not suitable for network filesystems.  If a partial page is
written, then the page needs to be first read from userspace.  This means, that
even for files opened for O_WRONLY it is possible that READ requests will be
generated by the kernel.

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Nikolaus Rath]

On Sep 25 2017, Miklos Szeredi <miklos@szeredi.hu> wrote:
> Fuse supports the following I/O modes:
>
> - direct-io
> - cached
>   + write-through
>   + writeback-cache
>
> The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the
> FUSE_OPEN reply.
>
> In direct-io mode the page cache is completely bypassed for reads and writes.
> No read-ahead takes place. Shared mmap is disabled.
>
> In cached mode reads may be satisfied from the page cache, and data may be
> read-ahead by the kernel to fill the cache.  The cache is always kept consistent
> after any writes to the file.  All mmap modes are supported.
>
> The cached mode has two sub modes controlling how writes are handled.  The
> write-through mode is the default and is supported on all kernels.  The
> writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the
> FUSE_INIT reply.
>
> In write-through mode each write is immediately sent to userspace as one or more
> WRITE requests, as well as updating any cached pages (and caching previously
> uncached, but fully written pages).  No READ requests are ever sent for writes,
> so when an uncached page is partially written, the page is discarded.
>
> In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to
> the cache only, which means that the write(2) syscall can often complete very
> fast.  Dirty pages are written back implicitly (background writeback or page
> reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and
> when the last ref to the file is being released on munmap(2)).  This mode
> assumes that all changes to the filesystem go through the FUSE kernel module
> (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so
> it's generally not suitable for network filesystems.  If a partial page is
> written, then the page needs to be first read from userspace.  This means, that
> even for files opened for O_WRONLY it is possible that READ requests will be
> generated by the kernel.


Looks great to me.


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] [fuse] getattr() results ignored when writeback cache is active

[Maxim Patlasov]

On 09/22/2017 12:34 PM, Nikolaus Rath wrote:

> On Sep 21 2017, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>> Oh, I understand now. Your example actually demonstrates that
>> FUSE_WRITEBACK_CACHE must not be used if it's possible to modify
>> underlying data (/tmp/issue_93/file_1) externally w.r.t fuse
>> filesystem. That's correct. But initial statement:
>>
>>> this mode of operation is not suitable for any network filesystem
>>> even if no write operations are actually carried out
>> is not, because it's not impossible to protect underlying data against
>> such an external change somehow. A network filesystem is not obliged
>> to keep transparent correspondence like /tmp/issue_93_mnt/file_1 <-->
>> /tmp/issue_93/file_1. It can keep all user data (and metadata) in
>> internal structures making any external modifications
>> impossible. Then, implementing exclusive write semantics would make
>> using FUSE_WRITEBACK_CACHE safe. Agree?
> I agree with what you're saying, but I don't agree that this should go
> into the documentation for which the sentence in dispute was intended
> :-). I think in 93% of all cases, the simplified statement will be more
> helpful to a potential FUSE user than the correct one.
>
> If this bothers you a lot, can we compromise and just be a little more
> vague overall? I.e.,
>
> | This mode of operation is therefore generally not suitable for
> | filesystems where modifications may not come through the (local) FUSE
> | layer (i.e, most network filesystems).

The way how Miklos conveyed it:

> This mode
> assumes that all changes to the filesystem go through the FUSE kernel module
> (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so
> it's generally not suitable for network filesystems.

looks perfect. Thank you!

Maxim