Can we change the S_DAX flag immediately on XFS without dropping caches?

Can we change the S_DAX flag immediately on XFS without dropping caches?

[Li, Hao]

Hi,

I have noticed that we have to drop caches to make the changing of S_DAX
flag take effect after using chattr +x to turn on DAX for a existing
regular file. The related function is xfs_diflags_to_iflags, whose
second parameter determines whether we should set S_DAX immediately.

I can't figure out why we do this. Is this because the page caches in
address_space->i_pages are hard to deal with? I also wonder what will
happen if we set S_DAX unconditionally. Thanks!

Regards,
Hao Li

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Dave Chinner]

On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
> Hi,
> 
> I have noticed that we have to drop caches to make the changing of S_DAX
> flag take effect after using chattr +x to turn on DAX for a existing
> regular file. The related function is xfs_diflags_to_iflags, whose
> second parameter determines whether we should set S_DAX immediately.

Yup, as documented in Documentation/filesystems/dax.txt. Specifically:

 6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
    the change in behaviour for existing regular files may not occur
    immediately.  If the change must take effect immediately, the administrator
    needs to:

    a) stop the application so there are no active references to the data set
       the policy change will affect

    b) evict the data set from kernel caches so it will be re-instantiated when
       the application is restarted. This can be achieved by:

       i. drop-caches
       ii. a filesystem unmount and mount cycle
       iii. a system reboot

> I can't figure out why we do this. Is this because the page caches in
> address_space->i_pages are hard to deal with?

Because of unfixable races in the page fault path that prevent
changing the caching behaviour of the inode while concurrent access
is possible. The only way to guarantee races can't happen is to
cycle the inode out of cache.

> I also wonder what will
> happen if we set S_DAX unconditionally. Thanks!

As per the documentation, that's exactly what the dax=always mount
option does.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Yasunori Goto]

Hi,

On 2020/07/28 11:20, Dave Chinner wrote:
> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
>> Hi,
>>
>> I have noticed that we have to drop caches to make the changing of S_DAX
>> flag take effect after using chattr +x to turn on DAX for a existing
>> regular file. The related function is xfs_diflags_to_iflags, whose
>> second parameter determines whether we should set S_DAX immediately.
> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
>
>   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
>      the change in behaviour for existing regular files may not occur
>      immediately.  If the change must take effect immediately, the administrator
>      needs to:
>
>      a) stop the application so there are no active references to the data set
>         the policy change will affect
>
>      b) evict the data set from kernel caches so it will be re-instantiated when
>         the application is restarted. This can be achieved by:
>
>         i. drop-caches
>         ii. a filesystem unmount and mount cycle
>         iii. a system reboot
>
>> I can't figure out why we do this. Is this because the page caches in
>> address_space->i_pages are hard to deal with?
> Because of unfixable races in the page fault path that prevent
> changing the caching behaviour of the inode while concurrent access
> is possible. The only way to guarantee races can't happen is to
> cycle the inode out of cache.

I understand why the drop_cache operation is necessary. Thanks.

BTW, even normal user becomes to able to change DAX flag for an inode,
drop_cache operation still requires root permission, right?

So, if kernel have a feature for normal user can operate drop cache for 
"a inode" with
its permission, I think it improve the above limitation, and
we would like to try to implement it recently.

Do you have any opinion making such feature?
(Agree/opposition, or any other comment?)

Thanks,

-- 
Yasunori Goto

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Ira Weiny]

On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
> Hi,
> 
> On 2020/07/28 11:20, Dave Chinner wrote:
> > On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
> > > Hi,
> > > 
> > > I have noticed that we have to drop caches to make the changing of S_DAX
> > > flag take effect after using chattr +x to turn on DAX for a existing
> > > regular file. The related function is xfs_diflags_to_iflags, whose
> > > second parameter determines whether we should set S_DAX immediately.
> > Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
> > 
> >   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
> >      the change in behaviour for existing regular files may not occur
> >      immediately.  If the change must take effect immediately, the administrator
> >      needs to:
> > 
> >      a) stop the application so there are no active references to the data set
> >         the policy change will affect
> > 
> >      b) evict the data set from kernel caches so it will be re-instantiated when
> >         the application is restarted. This can be achieved by:
> > 
> >         i. drop-caches
> >         ii. a filesystem unmount and mount cycle
> >         iii. a system reboot
> > 
> > > I can't figure out why we do this. Is this because the page caches in
> > > address_space->i_pages are hard to deal with?
> > Because of unfixable races in the page fault path that prevent
> > changing the caching behaviour of the inode while concurrent access
> > is possible. The only way to guarantee races can't happen is to
> > cycle the inode out of cache.
> 
> I understand why the drop_cache operation is necessary. Thanks.
> 
> BTW, even normal user becomes to able to change DAX flag for an inode,
> drop_cache operation still requires root permission, right?
> 
> So, if kernel have a feature for normal user can operate drop cache for "a
> inode" with
> its permission, I think it improve the above limitation, and
> we would like to try to implement it recently.
> 
> Do you have any opinion making such feature?
> (Agree/opposition, or any other comment?)

I would not be opposed but there were many hurdles to that implementation.

What is the use case you are thinking of here?

The compromise of dropping caches was reached because we envisioned that many
users would simply want to chose the file mode when a file was created and
maintain that mode through the lifetime of the file.  To that end one can
simply create directories which have the desired dax mode and any files created
in that directory will inherit the dax mode immediately.  So there is no need
to switch the file mode directly as a normal user.

Would that work for your use case?

Ira

> 
> Thanks,
> 
> -- 
> Yasunori Goto
> 

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Dave Chinner]

On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
> Hi,
> 
> On 2020/07/28 11:20, Dave Chinner wrote:
> > On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
> > > Hi,
> > > 
> > > I have noticed that we have to drop caches to make the changing of S_DAX
> > > flag take effect after using chattr +x to turn on DAX for a existing
> > > regular file. The related function is xfs_diflags_to_iflags, whose
> > > second parameter determines whether we should set S_DAX immediately.
> > Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
> > 
> >   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
> >      the change in behaviour for existing regular files may not occur
> >      immediately.  If the change must take effect immediately, the administrator
> >      needs to:
> > 
> >      a) stop the application so there are no active references to the data set
> >         the policy change will affect
> > 
> >      b) evict the data set from kernel caches so it will be re-instantiated when
> >         the application is restarted. This can be achieved by:
> > 
> >         i. drop-caches
> >         ii. a filesystem unmount and mount cycle
> >         iii. a system reboot
> > 
> > > I can't figure out why we do this. Is this because the page caches in
> > > address_space->i_pages are hard to deal with?
> > Because of unfixable races in the page fault path that prevent
> > changing the caching behaviour of the inode while concurrent access
> > is possible. The only way to guarantee races can't happen is to
> > cycle the inode out of cache.
> 
> I understand why the drop_cache operation is necessary. Thanks.
> 
> BTW, even normal user becomes to able to change DAX flag for an inode,
> drop_cache operation still requires root permission, right?

Step back for a minute and explain why you want to be able to change
the DAX mode of a file -as a user-.

> So, if kernel have a feature for normal user can operate drop cache for "a
> inode" with
> its permission, I think it improve the above limitation, and
> we would like to try to implement it recently.

No, drop_caches is not going to be made available to users. That
makes it s trivial system wide DoS vector.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Li, Hao]

On 2020/7/30 0:10, Ira Weiny wrote:

> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
>> Hi,
>>
>> On 2020/07/28 11:20, Dave Chinner wrote:
>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
>>>> Hi,
>>>>
>>>> I have noticed that we have to drop caches to make the changing of S_DAX
>>>> flag take effect after using chattr +x to turn on DAX for a existing
>>>> regular file. The related function is xfs_diflags_to_iflags, whose
>>>> second parameter determines whether we should set S_DAX immediately.
>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
>>>
>>>   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
>>>      the change in behaviour for existing regular files may not occur
>>>      immediately.  If the change must take effect immediately, the administrator
>>>      needs to:
>>>
>>>      a) stop the application so there are no active references to the data set
>>>         the policy change will affect
>>>
>>>      b) evict the data set from kernel caches so it will be re-instantiated when
>>>         the application is restarted. This can be achieved by:
>>>
>>>         i. drop-caches
>>>         ii. a filesystem unmount and mount cycle
>>>         iii. a system reboot
>>>
>>>> I can't figure out why we do this. Is this because the page caches in
>>>> address_space->i_pages are hard to deal with?
>>> Because of unfixable races in the page fault path that prevent
>>> changing the caching behaviour of the inode while concurrent access
>>> is possible. The only way to guarantee races can't happen is to
>>> cycle the inode out of cache.
>> I understand why the drop_cache operation is necessary. Thanks.
>>
>> BTW, even normal user becomes to able to change DAX flag for an inode,
>> drop_cache operation still requires root permission, right?
>>
>> So, if kernel have a feature for normal user can operate drop cache for "a
>> inode" with
>> its permission, I think it improve the above limitation, and
>> we would like to try to implement it recently.
>>
>> Do you have any opinion making such feature?
>> (Agree/opposition, or any other comment?)
> I would not be opposed but there were many hurdles to that implementation.
>
> What is the use case you are thinking of here?
>
> The compromise of dropping caches was reached because we envisioned that many
> users would simply want to chose the file mode when a file was created and
> maintain that mode through the lifetime of the file.  To that end one can
> simply create directories which have the desired dax mode and any files created
> in that directory will inherit the dax mode immediately.  
Inheriting mechanism for DAX mode is reasonable but chattr&drop_caches
makes things complicated.
> So there is no need
> to switch the file mode directly as a normal user.

The question is, the normal users can indeed use chattr to change the DAX
mode for a regular file as long as they want. However, when they do this,
they have no way to make the change take effect. I think this behavior is
weird. We can say chattr executes successfully because XFS_DIFLAG2_DAX has
been set onto xfs_inode->i_d.di_flags2, but we can also say chattr doesn't
finish things completely because S_DAX is not set onto inode->i_flags.
The user may be confused about why chattr +/-x doesn't work at all. Maybe
we should find a way for the normal user to make chattr take effects
without calling the administrator, or we can make the chattr +/x command
request root permission now that if the user has root permission, he can
make DAX changing take effect through echo 2 > /proc/sys/vm/drop_caches.


Regards,

Hao Li

>
> Would that work for your use case?
>
> Ira

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Li, Hao]

On 2020/7/30 7:21, Dave Chinner wrote:

> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
>> Hi,
>>
>> On 2020/07/28 11:20, Dave Chinner wrote:
>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
>>>> Hi,
>>>>
>>>> I have noticed that we have to drop caches to make the changing of S_DAX
>>>> flag take effect after using chattr +x to turn on DAX for a existing
>>>> regular file. The related function is xfs_diflags_to_iflags, whose
>>>> second parameter determines whether we should set S_DAX immediately.
>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
>>>
>>>   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
>>>      the change in behaviour for existing regular files may not occur
>>>      immediately.  If the change must take effect immediately, the administrator
>>>      needs to:
>>>
>>>      a) stop the application so there are no active references to the data set
>>>         the policy change will affect
>>>
>>>      b) evict the data set from kernel caches so it will be re-instantiated when
>>>         the application is restarted. This can be achieved by:
>>>
>>>         i. drop-caches
>>>         ii. a filesystem unmount and mount cycle
>>>         iii. a system reboot
>>>
>>>> I can't figure out why we do this. Is this because the page caches in
>>>> address_space->i_pages are hard to deal with?
>>> Because of unfixable races in the page fault path that prevent
>>> changing the caching behaviour of the inode while concurrent access
>>> is possible. The only way to guarantee races can't happen is to
>>> cycle the inode out of cache.
>> I understand why the drop_cache operation is necessary. Thanks.
>>
>> BTW, even normal user becomes to able to change DAX flag for an inode,
>> drop_cache operation still requires root permission, right?
> Step back for a minute and explain why you want to be able to change
> the DAX mode of a file -as a user-.
chattr command can be executed by normal users as long as they want.
I think if they do this, they may get confused because the dax mode
doesn't take effects immediately.
>> So, if kernel have a feature for normal user can operate drop cache for "a
>> inode" with
>> its permission, I think it improve the above limitation, and
>> we would like to try to implement it recently.
> No, drop_caches is not going to be made available to users. That
> makes it s trivial system wide DoS vector.
drop_caches have to be limited for root user, but we may need to find
a way for normal users to make dax changing take effect if they have
run chattr.

Regards,
Hao Li

> Cheers,
>
> Dave.

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Yasunori Goto]

On 2020/07/30 8:21, Dave Chinner wrote:
> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
>> Hi,
>>
>> On 2020/07/28 11:20, Dave Chinner wrote:
>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
>>>> Hi,
>>>>
>>>> I have noticed that we have to drop caches to make the changing of S_DAX
>>>> flag take effect after using chattr +x to turn on DAX for a existing
>>>> regular file. The related function is xfs_diflags_to_iflags, whose
>>>> second parameter determines whether we should set S_DAX immediately.
>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
>>>
>>>    6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
>>>       the change in behaviour for existing regular files may not occur
>>>       immediately.  If the change must take effect immediately, the administrator
>>>       needs to:
>>>
>>>       a) stop the application so there are no active references to the data set
>>>          the policy change will affect
>>>
>>>       b) evict the data set from kernel caches so it will be re-instantiated when
>>>          the application is restarted. This can be achieved by:
>>>
>>>          i. drop-caches
>>>          ii. a filesystem unmount and mount cycle
>>>          iii. a system reboot
>>>
>>>> I can't figure out why we do this. Is this because the page caches in
>>>> address_space->i_pages are hard to deal with?
>>> Because of unfixable races in the page fault path that prevent
>>> changing the caching behaviour of the inode while concurrent access
>>> is possible. The only way to guarantee races can't happen is to
>>> cycle the inode out of cache.
>> I understand why the drop_cache operation is necessary. Thanks.
>>
>> BTW, even normal user becomes to able to change DAX flag for an inode,
>> drop_cache operation still requires root permission, right?
> Step back for a minute and explain why you want to be able to change
> the DAX mode of a file -as a user-.


For example, there are 2 containers executed in a system, which is named as
container A and container B, and these host gives FS-DAX files to each 
containers.
If the user of container A would like to change DAX-off for tuning, then 
he will stop his application
and change DAX flag, but the flag may not be changed.

Then he will "need" to ask host operator to execute drop_cache, and the 
operator did it.
As a result, not only container A, but also container B get the impact 
of drop_cache.

Especially, if this is multi tenant container system, then I think this 
is not acceptable.

Probably, there are 2 problems I think.
1) drop_cache requires root permission.
2) drop_cache has too wide effect.

>
>> So, if kernel have a feature for normal user can operate drop cache for "a
>> inode" with
>> its permission, I think it improve the above limitation, and
>> we would like to try to implement it recently.
> No, drop_caches is not going to be made available to users. That
> makes it s trivial system wide DoS vector.

The current drop_cache feature tries to drop ALL of cache (page cache 
and/or slab cache).
Then, I agree that normal user should not drop all of them.

But my intention was that drop cache of ONE file which is changed dax flag,
(and if possible, drop only the inode cache.)
Do you mean it will be still cause of weakness against DoS attack?
If so, I should give up to solve problem 1) at least.


Thanks,

>
> Cheers,
>
> Dave.

-- 
Yasunori Goto

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Yasunori Goto]

On 2020/07/30 1:10, Ira Weiny wrote:
> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
>> Hi,
>>
>> On 2020/07/28 11:20, Dave Chinner wrote:
>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
>>>> Hi,
>>>>
>>>> I have noticed that we have to drop caches to make the changing of S_DAX
>>>> flag take effect after using chattr +x to turn on DAX for a existing
>>>> regular file. The related function is xfs_diflags_to_iflags, whose
>>>> second parameter determines whether we should set S_DAX immediately.
>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
>>>
>>>    6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
>>>       the change in behaviour for existing regular files may not occur
>>>       immediately.  If the change must take effect immediately, the administrator
>>>       needs to:
>>>
>>>       a) stop the application so there are no active references to the data set
>>>          the policy change will affect
>>>
>>>       b) evict the data set from kernel caches so it will be re-instantiated when
>>>          the application is restarted. This can be achieved by:
>>>
>>>          i. drop-caches
>>>          ii. a filesystem unmount and mount cycle
>>>          iii. a system reboot
>>>
>>>> I can't figure out why we do this. Is this because the page caches in
>>>> address_space->i_pages are hard to deal with?
>>> Because of unfixable races in the page fault path that prevent
>>> changing the caching behaviour of the inode while concurrent access
>>> is possible. The only way to guarantee races can't happen is to
>>> cycle the inode out of cache.
>> I understand why the drop_cache operation is necessary. Thanks.
>>
>> BTW, even normal user becomes to able to change DAX flag for an inode,
>> drop_cache operation still requires root permission, right?
>>
>> So, if kernel have a feature for normal user can operate drop cache for "a
>> inode" with
>> its permission, I think it improve the above limitation, and
>> we would like to try to implement it recently.
>>
>> Do you have any opinion making such feature?
>> (Agree/opposition, or any other comment?)
> I would not be opposed but there were many hurdles to that implementation.
>
> What is the use case you are thinking of here?
>
> The compromise of dropping caches was reached because we envisioned that many
> users would simply want to chose the file mode when a file was created and
> maintain that mode through the lifetime of the file.  To that end one can
> simply create directories which have the desired dax mode and any files created
> in that directory will inherit the dax mode immediately.  So there is no need
> to switch the file mode directly as a normal user.
>
> Would that work for your use case?
Though I wrote it on another mail, your information was very helpful for me.
Thank you for your response.

Bye,

>
> Ira
>
>> Thanks,
>>
>> -- 
>> Yasunori Goto
>>

-- 
Yasunori Goto

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Li, Hao]

Hello,

Ping.

Thanks,
Hao Li


On 2020/7/31 17:12, Li, Hao wrote:
> On 2020/7/30 0:10, Ira Weiny wrote:
>
>> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
>>> Hi,
>>>
>>> On 2020/07/28 11:20, Dave Chinner wrote:
>>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
>>>>> Hi,
>>>>>
>>>>> I have noticed that we have to drop caches to make the changing of S_DAX
>>>>> flag take effect after using chattr +x to turn on DAX for a existing
>>>>> regular file. The related function is xfs_diflags_to_iflags, whose
>>>>> second parameter determines whether we should set S_DAX immediately.
>>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
>>>>
>>>>   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
>>>>      the change in behaviour for existing regular files may not occur
>>>>      immediately.  If the change must take effect immediately, the administrator
>>>>      needs to:
>>>>
>>>>      a) stop the application so there are no active references to the data set
>>>>         the policy change will affect
>>>>
>>>>      b) evict the data set from kernel caches so it will be re-instantiated when
>>>>         the application is restarted. This can be achieved by:
>>>>
>>>>         i. drop-caches
>>>>         ii. a filesystem unmount and mount cycle
>>>>         iii. a system reboot
>>>>
>>>>> I can't figure out why we do this. Is this because the page caches in
>>>>> address_space->i_pages are hard to deal with?
>>>> Because of unfixable races in the page fault path that prevent
>>>> changing the caching behaviour of the inode while concurrent access
>>>> is possible. The only way to guarantee races can't happen is to
>>>> cycle the inode out of cache.
>>> I understand why the drop_cache operation is necessary. Thanks.
>>>
>>> BTW, even normal user becomes to able to change DAX flag for an inode,
>>> drop_cache operation still requires root permission, right?
>>>
>>> So, if kernel have a feature for normal user can operate drop cache for "a
>>> inode" with
>>> its permission, I think it improve the above limitation, and
>>> we would like to try to implement it recently.
>>>
>>> Do you have any opinion making such feature?
>>> (Agree/opposition, or any other comment?)
>> I would not be opposed but there were many hurdles to that implementation.
>>
>> What is the use case you are thinking of here?
>>
>> The compromise of dropping caches was reached because we envisioned that many
>> users would simply want to chose the file mode when a file was created and
>> maintain that mode through the lifetime of the file.  To that end one can
>> simply create directories which have the desired dax mode and any files created
>> in that directory will inherit the dax mode immediately.  
> Inheriting mechanism for DAX mode is reasonable but chattr&drop_caches
> makes things complicated.
>> So there is no need
>> to switch the file mode directly as a normal user.
> The question is, the normal users can indeed use chattr to change the DAX
> mode for a regular file as long as they want. However, when they do this,
> they have no way to make the change take effect. I think this behavior is
> weird. We can say chattr executes successfully because XFS_DIFLAG2_DAX has
> been set onto xfs_inode->i_d.di_flags2, but we can also say chattr doesn't
> finish things completely because S_DAX is not set onto inode->i_flags.
> The user may be confused about why chattr +/-x doesn't work at all. Maybe
> we should find a way for the normal user to make chattr take effects
> without calling the administrator, or we can make the chattr +/x command
> request root permission now that if the user has root permission, he can
> make DAX changing take effect through echo 2 > /proc/sys/vm/drop_caches.
>
>
> Regards,
>
> Hao Li
>
>> Would that work for your use case?
>>
>> Ira

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Darrick J. Wong]

On Wed, Aug 05, 2020 at 04:10:05PM +0800, Li, Hao wrote:
> Hello,
> 
> Ping.
> 
> Thanks,
> Hao Li
> 
> 
> On 2020/7/31 17:12, Li, Hao wrote:
> > On 2020/7/30 0:10, Ira Weiny wrote:
> >
> >> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
> >>> Hi,
> >>>
> >>> On 2020/07/28 11:20, Dave Chinner wrote:
> >>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I have noticed that we have to drop caches to make the changing of S_DAX
> >>>>> flag take effect after using chattr +x to turn on DAX for a existing
> >>>>> regular file. The related function is xfs_diflags_to_iflags, whose
> >>>>> second parameter determines whether we should set S_DAX immediately.
> >>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
> >>>>
> >>>>   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
> >>>>      the change in behaviour for existing regular files may not occur
> >>>>      immediately.  If the change must take effect immediately, the administrator
> >>>>      needs to:
> >>>>
> >>>>      a) stop the application so there are no active references to the data set
> >>>>         the policy change will affect
> >>>>
> >>>>      b) evict the data set from kernel caches so it will be re-instantiated when
> >>>>         the application is restarted. This can be achieved by:
> >>>>
> >>>>         i. drop-caches
> >>>>         ii. a filesystem unmount and mount cycle
> >>>>         iii. a system reboot
> >>>>
> >>>>> I can't figure out why we do this. Is this because the page caches in
> >>>>> address_space->i_pages are hard to deal with?
> >>>> Because of unfixable races in the page fault path that prevent
> >>>> changing the caching behaviour of the inode while concurrent access
> >>>> is possible. The only way to guarantee races can't happen is to
> >>>> cycle the inode out of cache.
> >>> I understand why the drop_cache operation is necessary. Thanks.
> >>>
> >>> BTW, even normal user becomes to able to change DAX flag for an inode,
> >>> drop_cache operation still requires root permission, right?
> >>>
> >>> So, if kernel have a feature for normal user can operate drop cache for "a
> >>> inode" with
> >>> its permission, I think it improve the above limitation, and
> >>> we would like to try to implement it recently.
> >>>
> >>> Do you have any opinion making such feature?
> >>> (Agree/opposition, or any other comment?)
> >> I would not be opposed but there were many hurdles to that implementation.
> >>
> >> What is the use case you are thinking of here?
> >>
> >> The compromise of dropping caches was reached because we envisioned that many
> >> users would simply want to chose the file mode when a file was created and
> >> maintain that mode through the lifetime of the file.  To that end one can
> >> simply create directories which have the desired dax mode and any files created
> >> in that directory will inherit the dax mode immediately.  
> > Inheriting mechanism for DAX mode is reasonable but chattr&drop_caches
> > makes things complicated.
> >> So there is no need
> >> to switch the file mode directly as a normal user.
> > The question is, the normal users can indeed use chattr to change the DAX
> > mode for a regular file as long as they want. However, when they do this,
> > they have no way to make the change take effect. I think this behavior is
> > weird. We can say chattr executes successfully because XFS_DIFLAG2_DAX has
> > been set onto xfs_inode->i_d.di_flags2, but we can also say chattr doesn't
> > finish things completely because S_DAX is not set onto inode->i_flags.
> > The user may be confused about why chattr +/-x doesn't work at all. Maybe
> > we should find a way for the normal user to make chattr take effects
> > without calling the administrator, or we can make the chattr +/x command
> > request root permission now that if the user has root permission, he can
> > make DAX changing take effect through echo 2 > /proc/sys/vm/drop_caches.

The kernel can sometimes make S_DAX changes take effect on its own,
provided that there are no other users of the file and the filesystem
agrees to reclaim an inode on file close and the program closes the file
after changing the bit.  None of these behaviors are guaranteed to
exist, so this is not mentioned in the documentation.

(And before anyone asks, yes, we did try to build a way to change the
file ops on the fly, but adding more concurrency control to all io paths
to handle an infrequent state change is not acceptable.)

--D

> >
> >
> > Regards,
> >
> > Hao Li
> >
> >> Would that work for your use case?
> >>
> >> Ira
> 
> 

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Ira Weiny]

On Wed, Aug 05, 2020 at 08:44:43AM -0700, Darrick J. Wong wrote:
> On Wed, Aug 05, 2020 at 04:10:05PM +0800, Li, Hao wrote:
> > Hello,
> > 
> > Ping.
> > 
> > Thanks,
> > Hao Li
> > 
> > 
> > On 2020/7/31 17:12, Li, Hao wrote:
> > > On 2020/7/30 0:10, Ira Weiny wrote:
> > >
> > >> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
> > >>> Hi,
> > >>>
> > >>> On 2020/07/28 11:20, Dave Chinner wrote:
> > >>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> I have noticed that we have to drop caches to make the changing of S_DAX
> > >>>>> flag take effect after using chattr +x to turn on DAX for a existing
> > >>>>> regular file. The related function is xfs_diflags_to_iflags, whose
> > >>>>> second parameter determines whether we should set S_DAX immediately.
> > >>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
> > >>>>
> > >>>>   6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
> > >>>>      the change in behaviour for existing regular files may not occur
> > >>>>      immediately.  If the change must take effect immediately, the administrator
> > >>>>      needs to:
> > >>>>
> > >>>>      a) stop the application so there are no active references to the data set
> > >>>>         the policy change will affect
> > >>>>
> > >>>>      b) evict the data set from kernel caches so it will be re-instantiated when
> > >>>>         the application is restarted. This can be achieved by:
> > >>>>
> > >>>>         i. drop-caches
> > >>>>         ii. a filesystem unmount and mount cycle
> > >>>>         iii. a system reboot
> > >>>>
> > >>>>> I can't figure out why we do this. Is this because the page caches in
> > >>>>> address_space->i_pages are hard to deal with?
> > >>>> Because of unfixable races in the page fault path that prevent
> > >>>> changing the caching behaviour of the inode while concurrent access
> > >>>> is possible. The only way to guarantee races can't happen is to
> > >>>> cycle the inode out of cache.
> > >>> I understand why the drop_cache operation is necessary. Thanks.
> > >>>
> > >>> BTW, even normal user becomes to able to change DAX flag for an inode,
> > >>> drop_cache operation still requires root permission, right?
> > >>>
> > >>> So, if kernel have a feature for normal user can operate drop cache for "a
> > >>> inode" with
> > >>> its permission, I think it improve the above limitation, and
> > >>> we would like to try to implement it recently.
> > >>>
> > >>> Do you have any opinion making such feature?
> > >>> (Agree/opposition, or any other comment?)
> > >> I would not be opposed but there were many hurdles to that implementation.
> > >>
> > >> What is the use case you are thinking of here?
> > >>
> > >> The compromise of dropping caches was reached because we envisioned that many
> > >> users would simply want to chose the file mode when a file was created and
> > >> maintain that mode through the lifetime of the file.  To that end one can
> > >> simply create directories which have the desired dax mode and any files created
> > >> in that directory will inherit the dax mode immediately.  
> > > Inheriting mechanism for DAX mode is reasonable but chattr&drop_caches
> > > makes things complicated.
> > >> So there is no need
> > >> to switch the file mode directly as a normal user.
> > > The question is, the normal users can indeed use chattr to change the DAX
> > > mode for a regular file as long as they want. However, when they do this,
> > > they have no way to make the change take effect. I think this behavior is
> > > weird. We can say chattr executes successfully because XFS_DIFLAG2_DAX has
> > > been set onto xfs_inode->i_d.di_flags2, but we can also say chattr doesn't
> > > finish things completely because S_DAX is not set onto inode->i_flags.
> > > The user may be confused about why chattr +/-x doesn't work at all. Maybe
> > > we should find a way for the normal user to make chattr take effects
> > > without calling the administrator, or we can make the chattr +/x command
> > > request root permission now that if the user has root permission, he can
> > > make DAX changing take effect through echo 2 > /proc/sys/vm/drop_caches.
> 
> The kernel can sometimes make S_DAX changes take effect on its own,
> provided that there are no other users of the file and the filesystem
> agrees to reclaim an inode on file close and the program closes the file
> after changing the bit.  None of these behaviors are guaranteed to
> exist, so this is not mentioned in the documentation.
> 
> (And before anyone asks, yes, we did try to build a way to change the
> file ops on the fly, but adding more concurrency control to all io paths
> to handle an infrequent state change is not acceptable.)

Thanks Darrick.  sorry for the delay...

Darrick is absolutely correct.  But I would like to add that the user can check
the current file status (S_DAX flag) by using statx.  So while they don't have
control they can at least see what is happening.

This was the compromise we had to make considering the complexities of changing
a file on the fly.

This was discussed at length on the mailing list:

https://lore.kernel.org/lkml/20200422212102.3757660-5-ira.weiny@intel.com/

Also, we do try our best to evict the inode as soon as possible 'automatically'

https://lore.kernel.org/lkml/20200409124127.GB18171@lst.de/

But there is just no guarantee that someone out there (a backup utility for
example) is not using the inode.  So the user just can't control this
absolutely.

Ira

> 
> --D
> 
> > >
> > >
> > > Regards,
> > >
> > > Hao Li
> > >
> > >> Would that work for your use case?
> > >>
> > >> Ira
> > 
> > 

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Ira Weiny]

On Fri, Jul 31, 2020 at 06:59:32PM +0900, Yasunori Goto wrote:
> On 2020/07/30 8:21, Dave Chinner wrote:
> > On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
> > > Hi,
> > > 
> > > On 2020/07/28 11:20, Dave Chinner wrote:
> > > > On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
> > > > > Hi,
> > > > > 
> > > > > I have noticed that we have to drop caches to make the changing of S_DAX
> > > > > flag take effect after using chattr +x to turn on DAX for a existing
> > > > > regular file. The related function is xfs_diflags_to_iflags, whose
> > > > > second parameter determines whether we should set S_DAX immediately.
> > > > Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
> > > > 
> > > >    6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
> > > >       the change in behaviour for existing regular files may not occur
> > > >       immediately.  If the change must take effect immediately, the administrator
> > > >       needs to:
> > > > 
> > > >       a) stop the application so there are no active references to the data set
> > > >          the policy change will affect
> > > > 
> > > >       b) evict the data set from kernel caches so it will be re-instantiated when
> > > >          the application is restarted. This can be achieved by:
> > > > 
> > > >          i. drop-caches
> > > >          ii. a filesystem unmount and mount cycle
> > > >          iii. a system reboot
> > > > 
> > > > > I can't figure out why we do this. Is this because the page caches in
> > > > > address_space->i_pages are hard to deal with?
> > > > Because of unfixable races in the page fault path that prevent
> > > > changing the caching behaviour of the inode while concurrent access
> > > > is possible. The only way to guarantee races can't happen is to
> > > > cycle the inode out of cache.
> > > I understand why the drop_cache operation is necessary. Thanks.
> > > 
> > > BTW, even normal user becomes to able to change DAX flag for an inode,
> > > drop_cache operation still requires root permission, right?
> > Step back for a minute and explain why you want to be able to change
> > the DAX mode of a file -as a user-.
> 
> 
> For example, there are 2 containers executed in a system, which is named as
> container A and container B, and these host gives FS-DAX files to each
> containers.
> If the user of container A would like to change DAX-off for tuning, then he
> will stop his application
> and change DAX flag, but the flag may not be changed.
> 
> Then he will "need" to ask host operator to execute drop_cache, and the
> operator did it.
> As a result, not only container A, but also container B get the impact of
> drop_cache.
> 
> Especially, if this is multi tenant container system, then I think this is
> not acceptable.
> 
> Probably, there are 2 problems I think.
> 1) drop_cache requires root permission.
> 2) drop_cache has too wide effect.
> 
> > 
> > > So, if kernel have a feature for normal user can operate drop cache for "a
> > > inode" with
> > > its permission, I think it improve the above limitation, and
> > > we would like to try to implement it recently.
> > No, drop_caches is not going to be made available to users. That
> > makes it s trivial system wide DoS vector.
> 
> The current drop_cache feature tries to drop ALL of cache (page cache and/or
> slab cache).
> Then, I agree that normal user should not drop all of them.
> 
> But my intention was that drop cache of ONE file which is changed dax flag,
> (and if possible, drop only the inode cache.)
> Do you mean it will be still cause of weakness against DoS attack?
> If so, I should give up to solve problem 1) at least.

FWIW changing the on disk flag automatically flags the inode to be dropped as
soon as all references are done.

See:

2c567af418e3 fs: Introduce DCACHE_DONTCACHE
dae2f8ed7992 fs: Lift XFS_IDONTCACHE to the VFS layer

But from a users perspective you just don't know when that will happen.  The
system just can't guarantee it.  The best the user can do is stop taking
references to the file and close all references, and periodically check the
state.  But this will take a reference so...  Kind of a catch-22 here...  :-(

Ira

> 
> 
> Thanks,
> 
> > 
> > Cheers,
> > 
> > Dave.
> 
> -- 
> Yasunori Goto
> 

Re: Can we change the S_DAX flag immediately on XFS without dropping caches?

[Li, Hao]

On 2020/8/8 1:09, Ira Weiny wrote:
> On Fri, Jul 31, 2020 at 06:59:32PM +0900, Yasunori Goto wrote:
>> On 2020/07/30 8:21, Dave Chinner wrote:
>>> On Wed, Jul 29, 2020 at 11:23:21AM +0900, Yasunori Goto wrote:
>>>> Hi,
>>>>
>>>> On 2020/07/28 11:20, Dave Chinner wrote:
>>>>> On Tue, Jul 28, 2020 at 02:00:08AM +0000, Li, Hao wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have noticed that we have to drop caches to make the changing of S_DAX
>>>>>> flag take effect after using chattr +x to turn on DAX for a existing
>>>>>> regular file. The related function is xfs_diflags_to_iflags, whose
>>>>>> second parameter determines whether we should set S_DAX immediately.
>>>>> Yup, as documented in Documentation/filesystems/dax.txt. Specifically:
>>>>>
>>>>>    6. When changing the S_DAX policy via toggling the persistent FS_XFLAG_DAX flag,
>>>>>       the change in behaviour for existing regular files may not occur
>>>>>       immediately.  If the change must take effect immediately, the administrator
>>>>>       needs to:
>>>>>
>>>>>       a) stop the application so there are no active references to the data set
>>>>>          the policy change will affect
>>>>>
>>>>>       b) evict the data set from kernel caches so it will be re-instantiated when
>>>>>          the application is restarted. This can be achieved by:
>>>>>
>>>>>          i. drop-caches
>>>>>          ii. a filesystem unmount and mount cycle
>>>>>          iii. a system reboot
>>>>>
>>>>>> I can't figure out why we do this. Is this because the page caches in
>>>>>> address_space->i_pages are hard to deal with?
>>>>> Because of unfixable races in the page fault path that prevent
>>>>> changing the caching behaviour of the inode while concurrent access
>>>>> is possible. The only way to guarantee races can't happen is to
>>>>> cycle the inode out of cache.
>>>> I understand why the drop_cache operation is necessary. Thanks.
>>>>
>>>> BTW, even normal user becomes to able to change DAX flag for an inode,
>>>> drop_cache operation still requires root permission, right?
>>> Step back for a minute and explain why you want to be able to change
>>> the DAX mode of a file -as a user-.
>>
>> For example, there are 2 containers executed in a system, which is named as
>> container A and container B, and these host gives FS-DAX files to each
>> containers.
>> If the user of container A would like to change DAX-off for tuning, then he
>> will stop his application
>> and change DAX flag, but the flag may not be changed.
>>
>> Then he will "need" to ask host operator to execute drop_cache, and the
>> operator did it.
>> As a result, not only container A, but also container B get the impact of
>> drop_cache.
>>
>> Especially, if this is multi tenant container system, then I think this is
>> not acceptable.
>>
>> Probably, there are 2 problems I think.
>> 1) drop_cache requires root permission.
>> 2) drop_cache has too wide effect.
>>
>>>> So, if kernel have a feature for normal user can operate drop cache for "a
>>>> inode" with
>>>> its permission, I think it improve the above limitation, and
>>>> we would like to try to implement it recently.
>>> No, drop_caches is not going to be made available to users. That
>>> makes it s trivial system wide DoS vector.
>> The current drop_cache feature tries to drop ALL of cache (page cache and/or
>> slab cache).
>> Then, I agree that normal user should not drop all of them.
>>
>> But my intention was that drop cache of ONE file which is changed dax flag,
>> (and if possible, drop only the inode cache.)
>> Do you mean it will be still cause of weakness against DoS attack?
>> If so, I should give up to solve problem 1) at least.
> FWIW changing the on disk flag automatically flags the inode to be dropped as
> soon as all references are done.
>
> See:
>
> 2c567af418e3 fs: Introduce DCACHE_DONTCACHE
> dae2f8ed7992 fs: Lift XFS_IDONTCACHE to the VFS layer
Hi,

I find that DCACHE_DONTCACHE doesn't work well.
If DCACHE_REFERENCED is not set, dput() can drop the inode successfully as
soon as all references are gone. By contrast, if DCACHE_REFERENCED is set,
dput() only decreases the reference count of dentry and don't evict inode.

Example 1:

echo abcdefg > test.txt
echo 3 > /proc/sys/vm/drop_caches
xfs_io -c 'chattr +x' test.txt

In this example, we can say the DAX policy takes effects immediately as we
don't need to drop cache after chattr.
In this circumstance, DCACHE_REFERENCED is not set, and DCACHE_DONTCACHE
can drop the inode as expected.

Example 2:

echo abcdefg > test.txt
xfs_io -c 'chattr +x' test.txt

In this example, we must drop caches after chattr to make DAX policy
take effects. This is because DCACHE_REFERENCED is set, and fast_dput() will
return true, and then retain_dentry() have no chance to check DCACHE_DONTCACHE.

If this is the desired behavior, I can't understand the necessity
of DCACHE_DONTCACHE.

Regards,
Hao Li
>
> But from a users perspective you just don't know when that will happen.  The
> system just can't guarantee it.  The best the user can do is stop taking
> references to the file and close all references, and periodically check the
> state.  But this will take a reference so...  Kind of a catch-22 here...  :-(
>
> Ira
>
>>
>> Thanks,
>>
>>> Cheers,
>>>
>>> Dave.
>> -- 
>> Yasunori Goto
>>
>