[fuse] Effects of opening with O_DIRECT

[fuse] Effects of opening with O_DIRECT

[Nikolaus Rath]

Hi,

What happens if a file (on a FUSE mountpoint) is opened without
O_DIRECT, has some data in the page cache, and is then opened a second
with O_DIRECT?

Will reads with O_DIRECT come from the page cache (if there's a hit), or
be passed through to the fuse daemon?

What happens to writes (with and without O_DIRECT, and assuming that
writeback caching is active)? It seems to me that in order to keep
consistent, either caching has to be disabled for both file descriptors
or enabled for both...


Thanks!
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse] Effects of opening with O_DIRECT

[Miklos Szeredi]

On Sun, Mar 1, 2020 at 2:20 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> Hi,
>
> What happens if a file (on a FUSE mountpoint) is opened without
> O_DIRECT, has some data in the page cache, and is then opened a second
> with O_DIRECT?
>
> Will reads with O_DIRECT come from the page cache (if there's a hit), or
> be passed through to the fuse daemon?

O_DIRECT read will try first directly, and fall back to the cache on
short or zero return count.

>
> What happens to writes (with and without O_DIRECT, and assuming that
> writeback caching is active)? It seems to me that in order to keep
> consistent, either caching has to be disabled for both file descriptors
> or enabled for both...

This is not a fuse specific problem.   The kernel will try to keep
things consistent by flushing dirty data before an O_DIRECT read.
However this mode of operation is not recommended.  See open(2)
manpage:

       Applications should avoid mixing O_DIRECT and normal I/O  to  the  same
       file,  and  especially  to  overlapping  byte regions in the same file.
       Even when the filesystem correctly handles the coherency issues in this
       situation,  overall  I/O  throughput  is likely to be slower than using
       either mode alone.  Likewise, applications should avoid mixing  mmap(2)
       of files with direct I/O to the same files.

[...]
       In summary, O_DIRECT is a potentially powerful tool that should be used
       with  caution.   It  is  recommended  that  applications  treat  use of
       O_DIRECT as a performance option which is disabled by default.

              "The thing that has always disturbed me about O_DIRECT  is  that
              the whole interface is just stupid, and was probably designed by
              a  deranged  monkey  on  some  serious   mind-controlling   sub‐
              stances."—Linus

Thanks,
Miklos

Re: [fuse] Effects of opening with O_DIRECT

[Nikolaus Rath]

On Mar 02 2020, Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Sun, Mar 1, 2020 at 2:20 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
>>
>> Hi,
>>
>> What happens if a file (on a FUSE mountpoint) is opened without
>> O_DIRECT, has some data in the page cache, and is then opened a second
>> with O_DIRECT?
>>
>> Will reads with O_DIRECT come from the page cache (if there's a hit), or
>> be passed through to the fuse daemon?
>
> O_DIRECT read will try first directly, and fall back to the cache on
> short or zero return count.
>
>>
>> What happens to writes (with and without O_DIRECT, and assuming that
>> writeback caching is active)? It seems to me that in order to keep
>> consistent, either caching has to be disabled for both file descriptors
>> or enabled for both...
>
> This is not a fuse specific problem.   The kernel will try to keep
> things consistent by flushing dirty data before an O_DIRECT read.
> However this mode of operation is not recommended.  See open(2)
> manpage:
[...]

Is there currently any other way to execute a read request while making
sure that data does not end-up in the page cache (unless it happens to
be there already)?

I have full control of userspace, so I could do something like read from
a pseudo-extended attribute (assuming there's no size limitations), but
I'm not sure if this is better than O_DIRECT...


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse] Effects of opening with O_DIRECT

[Miklos Szeredi]

On Mon, Mar 2, 2020 at 9:29 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> On Mar 02 2020, Miklos Szeredi <miklos@szeredi.hu> wrote:
> > On Sun, Mar 1, 2020 at 2:20 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
> >>
> >> Hi,
> >>
> >> What happens if a file (on a FUSE mountpoint) is opened without
> >> O_DIRECT, has some data in the page cache, and is then opened a second
> >> with O_DIRECT?
> >>
> >> Will reads with O_DIRECT come from the page cache (if there's a hit), or
> >> be passed through to the fuse daemon?
> >
> > O_DIRECT read will try first directly, and fall back to the cache on
> > short or zero return count.
> >
> >>
> >> What happens to writes (with and without O_DIRECT, and assuming that
> >> writeback caching is active)? It seems to me that in order to keep
> >> consistent, either caching has to be disabled for both file descriptors
> >> or enabled for both...
> >
> > This is not a fuse specific problem.   The kernel will try to keep
> > things consistent by flushing dirty data before an O_DIRECT read.
> > However this mode of operation is not recommended.  See open(2)
> > manpage:
> [...]
>
> Is there currently any other way to execute a read request while making
> sure that data does not end-up in the page cache (unless it happens to
> be there already)?

Hmm, that sounds something like posix_fadvise(..., POSIX_FADV_DONTNEED).

Thanks,
Miklos