linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching
@ 2019-08-29 13:10 Jan Kara
  2019-08-29 13:10 ` [PATCH 1/3] mm: Handle MADV_WILLNEED through vfs_fadvise() Jan Kara
                   ` (3 more replies)
  0 siblings, 4 replies; 24+ messages in thread
From: Jan Kara @ 2019-08-29 13:10 UTC (permalink / raw)
  To: linux-xfs
  Cc: linux-mm, Amir Goldstein, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Jan Kara

Hello,

this is a patch series that addresses a possible race between readahead and
hole punching Amir has discovered [1]. The first patch makes madvise(2) to
handle readahead requests through fadvise infrastructure, the third patch
then adds necessary locking to XFS to protect against the race. Note that
other filesystems need similar protections but e.g. in case of ext4 it isn't
so simple without seriously regressing mixed rw workload performance so
I'm pushing just xfs fix at this moment which is simple.

Changes since v1 (posted at [2]):
* Added reviewed-by tags
* Fixed indentation in xfs_file_fadvise()
* Improved comment and readibility of xfs_file_fadvise()

								Honza

[1] https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
[2] https://lore.kernel.org/linux-fsdevel/20190711140012.1671-1-jack@suse.cz/

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 1/3] mm: Handle MADV_WILLNEED through vfs_fadvise()
  2019-08-29 13:10 [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Jan Kara
@ 2019-08-29 13:10 ` Jan Kara
  2019-08-29 15:49   ` Darrick J. Wong
  2019-08-29 13:10 ` [PATCH 2/3] fs: Export generic_fadvise() Jan Kara
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 24+ messages in thread
From: Jan Kara @ 2019-08-29 13:10 UTC (permalink / raw)
  To: linux-xfs
  Cc: linux-mm, Amir Goldstein, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Jan Kara, stable

Currently handling of MADV_WILLNEED hint calls directly into readahead
code. Handle it by calling vfs_fadvise() instead so that filesystem can
use its ->fadvise() callback to acquire necessary locks or otherwise
prepare for the request.

Suggested-by: Amir Goldstein <amir73il@gmail.com>
Reviewed-by: Boaz Harrosh <boazh@netapp.com>
CC: stable@vger.kernel.org
Signed-off-by: Jan Kara <jack@suse.cz>
---
 mm/madvise.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 968df3aa069f..bac973b9f2cc 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -14,6 +14,7 @@
 #include <linux/userfaultfd_k.h>
 #include <linux/hugetlb.h>
 #include <linux/falloc.h>
+#include <linux/fadvise.h>
 #include <linux/sched.h>
 #include <linux/ksm.h>
 #include <linux/fs.h>
@@ -275,6 +276,7 @@ static long madvise_willneed(struct vm_area_struct *vma,
 			     unsigned long start, unsigned long end)
 {
 	struct file *file = vma->vm_file;
+	loff_t offset;
 
 	*prev = vma;
 #ifdef CONFIG_SWAP
@@ -298,12 +300,20 @@ static long madvise_willneed(struct vm_area_struct *vma,
 		return 0;
 	}
 
-	start = ((start - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
-	if (end > vma->vm_end)
-		end = vma->vm_end;
-	end = ((end - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
-
-	force_page_cache_readahead(file->f_mapping, file, start, end - start);
+	/*
+	 * Filesystem's fadvise may need to take various locks.  We need to
+	 * explicitly grab a reference because the vma (and hence the
+	 * vma's reference to the file) can go away as soon as we drop
+	 * mmap_sem.
+	 */
+	*prev = NULL;	/* tell sys_madvise we drop mmap_sem */
+	get_file(file);
+	up_read(&current->mm->mmap_sem);
+	offset = (loff_t)(start - vma->vm_start)
+			+ ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
+	vfs_fadvise(file, offset, end - start, POSIX_FADV_WILLNEED);
+	fput(file);
+	down_read(&current->mm->mmap_sem);
 	return 0;
 }
 
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 2/3] fs: Export generic_fadvise()
  2019-08-29 13:10 [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Jan Kara
  2019-08-29 13:10 ` [PATCH 1/3] mm: Handle MADV_WILLNEED through vfs_fadvise() Jan Kara
@ 2019-08-29 13:10 ` Jan Kara
  2019-08-29 13:10 ` [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch Jan Kara
  2020-01-17 10:50 ` [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Amir Goldstein
  3 siblings, 0 replies; 24+ messages in thread
From: Jan Kara @ 2019-08-29 13:10 UTC (permalink / raw)
  To: linux-xfs
  Cc: linux-mm, Amir Goldstein, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Jan Kara, stable

Filesystems will need to call this function from their fadvise handlers.

CC: stable@vger.kernel.org
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
 include/linux/fs.h | 2 ++
 mm/fadvise.c       | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/include/linux/fs.h b/include/linux/fs.h
index 997a530ff4e9..bc1b40fb0db7 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -3531,6 +3531,8 @@ extern void inode_nohighmem(struct inode *inode);
 /* mm/fadvise.c */
 extern int vfs_fadvise(struct file *file, loff_t offset, loff_t len,
 		       int advice);
+extern int generic_fadvise(struct file *file, loff_t offset, loff_t len,
+			   int advice);
 
 #if defined(CONFIG_IO_URING)
 extern struct sock *io_uring_get_socket(struct file *file);
diff --git a/mm/fadvise.c b/mm/fadvise.c
index 467bcd032037..4f17c83db575 100644
--- a/mm/fadvise.c
+++ b/mm/fadvise.c
@@ -27,8 +27,7 @@
  * deactivate the pages and clear PG_Referenced.
  */
 
-static int generic_fadvise(struct file *file, loff_t offset, loff_t len,
-			   int advice)
+int generic_fadvise(struct file *file, loff_t offset, loff_t len, int advice)
 {
 	struct inode *inode;
 	struct address_space *mapping;
@@ -178,6 +177,7 @@ static int generic_fadvise(struct file *file, loff_t offset, loff_t len,
 	}
 	return 0;
 }
+EXPORT_SYMBOL(generic_fadvise);
 
 int vfs_fadvise(struct file *file, loff_t offset, loff_t len, int advice)
 {
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-08-29 13:10 [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Jan Kara
  2019-08-29 13:10 ` [PATCH 1/3] mm: Handle MADV_WILLNEED through vfs_fadvise() Jan Kara
  2019-08-29 13:10 ` [PATCH 2/3] fs: Export generic_fadvise() Jan Kara
@ 2019-08-29 13:10 ` Jan Kara
  2019-08-29 15:52   ` Darrick J. Wong
  2020-01-17 10:50 ` [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Amir Goldstein
  3 siblings, 1 reply; 24+ messages in thread
From: Jan Kara @ 2019-08-29 13:10 UTC (permalink / raw)
  To: linux-xfs
  Cc: linux-mm, Amir Goldstein, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Jan Kara, stable

Hole puching currently evicts pages from page cache and then goes on to
remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
racing reads or page faults. However there is currently nothing that
prevents readahead triggered by fadvise() or madvise() from racing with
the hole punch and instantiating page cache page after hole punching has
evicted page cache in xfs_flush_unmap_range() but before it has removed
blocks from the inode. This page cache page will be mapping soon to be
freed block and that can lead to returning stale data to userspace or
even filesystem corruption.

Fix the problem by protecting handling of readahead requests by
XFS_IOLOCK_SHARED similarly as we protect reads.

CC: stable@vger.kernel.org
Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
Reported-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/xfs/xfs_file.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 28101bbc0b78..d952d5962e93 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -28,6 +28,7 @@
 #include <linux/falloc.h>
 #include <linux/backing-dev.h>
 #include <linux/mman.h>
+#include <linux/fadvise.h>
 
 static const struct vm_operations_struct xfs_file_vm_ops;
 
@@ -933,6 +934,30 @@ xfs_file_fallocate(
 	return error;
 }
 
+STATIC int
+xfs_file_fadvise(
+	struct file	*file,
+	loff_t		start,
+	loff_t		end,
+	int		advice)
+{
+	struct xfs_inode *ip = XFS_I(file_inode(file));
+	int ret;
+	int lockflags = 0;
+
+	/*
+	 * Operations creating pages in page cache need protection from hole
+	 * punching and similar ops
+	 */
+	if (advice == POSIX_FADV_WILLNEED) {
+		lockflags = XFS_IOLOCK_SHARED;
+		xfs_ilock(ip, lockflags);
+	}
+	ret = generic_fadvise(file, start, end, advice);
+	if (lockflags)
+		xfs_iunlock(ip, lockflags);
+	return ret;
+}
 
 STATIC loff_t
 xfs_file_remap_range(
@@ -1232,6 +1257,7 @@ const struct file_operations xfs_file_operations = {
 	.fsync		= xfs_file_fsync,
 	.get_unmapped_area = thp_get_unmapped_area,
 	.fallocate	= xfs_file_fallocate,
+	.fadvise	= xfs_file_fadvise,
 	.remap_file_range = xfs_file_remap_range,
 };
 
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* Re: [PATCH 1/3] mm: Handle MADV_WILLNEED through vfs_fadvise()
  2019-08-29 13:10 ` [PATCH 1/3] mm: Handle MADV_WILLNEED through vfs_fadvise() Jan Kara
@ 2019-08-29 15:49   ` Darrick J. Wong
  0 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2019-08-29 15:49 UTC (permalink / raw)
  To: Jan Kara
  Cc: linux-xfs, linux-mm, Amir Goldstein, Boaz Harrosh, linux-fsdevel, stable

On Thu, Aug 29, 2019 at 03:10:32PM +0200, Jan Kara wrote:
> Currently handling of MADV_WILLNEED hint calls directly into readahead
> code. Handle it by calling vfs_fadvise() instead so that filesystem can
> use its ->fadvise() callback to acquire necessary locks or otherwise
> prepare for the request.
> 
> Suggested-by: Amir Goldstein <amir73il@gmail.com>
> Reviewed-by: Boaz Harrosh <boazh@netapp.com>
> CC: stable@vger.kernel.org
> Signed-off-by: Jan Kara <jack@suse.cz>

Looks ok,
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>

--D

> ---
>  mm/madvise.c | 22 ++++++++++++++++------
>  1 file changed, 16 insertions(+), 6 deletions(-)
> 
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 968df3aa069f..bac973b9f2cc 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -14,6 +14,7 @@
>  #include <linux/userfaultfd_k.h>
>  #include <linux/hugetlb.h>
>  #include <linux/falloc.h>
> +#include <linux/fadvise.h>
>  #include <linux/sched.h>
>  #include <linux/ksm.h>
>  #include <linux/fs.h>
> @@ -275,6 +276,7 @@ static long madvise_willneed(struct vm_area_struct *vma,
>  			     unsigned long start, unsigned long end)
>  {
>  	struct file *file = vma->vm_file;
> +	loff_t offset;
>  
>  	*prev = vma;
>  #ifdef CONFIG_SWAP
> @@ -298,12 +300,20 @@ static long madvise_willneed(struct vm_area_struct *vma,
>  		return 0;
>  	}
>  
> -	start = ((start - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
> -	if (end > vma->vm_end)
> -		end = vma->vm_end;
> -	end = ((end - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
> -
> -	force_page_cache_readahead(file->f_mapping, file, start, end - start);
> +	/*
> +	 * Filesystem's fadvise may need to take various locks.  We need to
> +	 * explicitly grab a reference because the vma (and hence the
> +	 * vma's reference to the file) can go away as soon as we drop
> +	 * mmap_sem.
> +	 */
> +	*prev = NULL;	/* tell sys_madvise we drop mmap_sem */
> +	get_file(file);
> +	up_read(&current->mm->mmap_sem);
> +	offset = (loff_t)(start - vma->vm_start)
> +			+ ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
> +	vfs_fadvise(file, offset, end - start, POSIX_FADV_WILLNEED);
> +	fput(file);
> +	down_read(&current->mm->mmap_sem);
>  	return 0;
>  }
>  
> -- 
> 2.16.4
> 

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-08-29 13:10 ` [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch Jan Kara
@ 2019-08-29 15:52   ` Darrick J. Wong
  2019-08-30 15:24     ` Jan Kara
  0 siblings, 1 reply; 24+ messages in thread
From: Darrick J. Wong @ 2019-08-29 15:52 UTC (permalink / raw)
  To: Jan Kara
  Cc: linux-xfs, linux-mm, Amir Goldstein, Boaz Harrosh, linux-fsdevel, stable

On Thu, Aug 29, 2019 at 03:10:34PM +0200, Jan Kara wrote:
> Hole puching currently evicts pages from page cache and then goes on to
> remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
> and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
> racing reads or page faults. However there is currently nothing that
> prevents readahead triggered by fadvise() or madvise() from racing with
> the hole punch and instantiating page cache page after hole punching has
> evicted page cache in xfs_flush_unmap_range() but before it has removed
> blocks from the inode. This page cache page will be mapping soon to be
> freed block and that can lead to returning stale data to userspace or
> even filesystem corruption.
> 
> Fix the problem by protecting handling of readahead requests by
> XFS_IOLOCK_SHARED similarly as we protect reads.
> 
> CC: stable@vger.kernel.org
> Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
> Reported-by: Amir Goldstein <amir73il@gmail.com>
> Signed-off-by: Jan Kara <jack@suse.cz>

Is there a test on xfstests to demonstrate this race?

Will test it out though...

Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>

--D

> ---
>  fs/xfs/xfs_file.c | 26 ++++++++++++++++++++++++++
>  1 file changed, 26 insertions(+)
> 
> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> index 28101bbc0b78..d952d5962e93 100644
> --- a/fs/xfs/xfs_file.c
> +++ b/fs/xfs/xfs_file.c
> @@ -28,6 +28,7 @@
>  #include <linux/falloc.h>
>  #include <linux/backing-dev.h>
>  #include <linux/mman.h>
> +#include <linux/fadvise.h>
>  
>  static const struct vm_operations_struct xfs_file_vm_ops;
>  
> @@ -933,6 +934,30 @@ xfs_file_fallocate(
>  	return error;
>  }
>  
> +STATIC int
> +xfs_file_fadvise(
> +	struct file	*file,
> +	loff_t		start,
> +	loff_t		end,
> +	int		advice)
> +{
> +	struct xfs_inode *ip = XFS_I(file_inode(file));
> +	int ret;
> +	int lockflags = 0;
> +
> +	/*
> +	 * Operations creating pages in page cache need protection from hole
> +	 * punching and similar ops
> +	 */
> +	if (advice == POSIX_FADV_WILLNEED) {
> +		lockflags = XFS_IOLOCK_SHARED;
> +		xfs_ilock(ip, lockflags);
> +	}
> +	ret = generic_fadvise(file, start, end, advice);
> +	if (lockflags)
> +		xfs_iunlock(ip, lockflags);
> +	return ret;
> +}
>  
>  STATIC loff_t
>  xfs_file_remap_range(
> @@ -1232,6 +1257,7 @@ const struct file_operations xfs_file_operations = {
>  	.fsync		= xfs_file_fsync,
>  	.get_unmapped_area = thp_get_unmapped_area,
>  	.fallocate	= xfs_file_fallocate,
> +	.fadvise	= xfs_file_fadvise,
>  	.remap_file_range = xfs_file_remap_range,
>  };
>  
> -- 
> 2.16.4
> 

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-08-29 15:52   ` Darrick J. Wong
@ 2019-08-30 15:24     ` Jan Kara
  2019-08-30 16:02       ` Darrick J. Wong
  2019-09-18 12:31       ` Jan Kara
  0 siblings, 2 replies; 24+ messages in thread
From: Jan Kara @ 2019-08-30 15:24 UTC (permalink / raw)
  To: Darrick J. Wong
  Cc: Jan Kara, linux-xfs, linux-mm, Amir Goldstein, Boaz Harrosh,
	linux-fsdevel, stable

On Thu 29-08-19 08:52:04, Darrick J. Wong wrote:
> On Thu, Aug 29, 2019 at 03:10:34PM +0200, Jan Kara wrote:
> > Hole puching currently evicts pages from page cache and then goes on to
> > remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
> > and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
> > racing reads or page faults. However there is currently nothing that
> > prevents readahead triggered by fadvise() or madvise() from racing with
> > the hole punch and instantiating page cache page after hole punching has
> > evicted page cache in xfs_flush_unmap_range() but before it has removed
> > blocks from the inode. This page cache page will be mapping soon to be
> > freed block and that can lead to returning stale data to userspace or
> > even filesystem corruption.
> > 
> > Fix the problem by protecting handling of readahead requests by
> > XFS_IOLOCK_SHARED similarly as we protect reads.
> > 
> > CC: stable@vger.kernel.org
> > Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
> > Reported-by: Amir Goldstein <amir73il@gmail.com>
> > Signed-off-by: Jan Kara <jack@suse.cz>
> 
> Is there a test on xfstests to demonstrate this race?

No, but I can try to create one.

> Will test it out though...
> 
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks. BTW, will you pick up these patches please?

								Honza

> 
> --D
> 
> > ---
> >  fs/xfs/xfs_file.c | 26 ++++++++++++++++++++++++++
> >  1 file changed, 26 insertions(+)
> > 
> > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> > index 28101bbc0b78..d952d5962e93 100644
> > --- a/fs/xfs/xfs_file.c
> > +++ b/fs/xfs/xfs_file.c
> > @@ -28,6 +28,7 @@
> >  #include <linux/falloc.h>
> >  #include <linux/backing-dev.h>
> >  #include <linux/mman.h>
> > +#include <linux/fadvise.h>
> >  
> >  static const struct vm_operations_struct xfs_file_vm_ops;
> >  
> > @@ -933,6 +934,30 @@ xfs_file_fallocate(
> >  	return error;
> >  }
> >  
> > +STATIC int
> > +xfs_file_fadvise(
> > +	struct file	*file,
> > +	loff_t		start,
> > +	loff_t		end,
> > +	int		advice)
> > +{
> > +	struct xfs_inode *ip = XFS_I(file_inode(file));
> > +	int ret;
> > +	int lockflags = 0;
> > +
> > +	/*
> > +	 * Operations creating pages in page cache need protection from hole
> > +	 * punching and similar ops
> > +	 */
> > +	if (advice == POSIX_FADV_WILLNEED) {
> > +		lockflags = XFS_IOLOCK_SHARED;
> > +		xfs_ilock(ip, lockflags);
> > +	}
> > +	ret = generic_fadvise(file, start, end, advice);
> > +	if (lockflags)
> > +		xfs_iunlock(ip, lockflags);
> > +	return ret;
> > +}
> >  
> >  STATIC loff_t
> >  xfs_file_remap_range(
> > @@ -1232,6 +1257,7 @@ const struct file_operations xfs_file_operations = {
> >  	.fsync		= xfs_file_fsync,
> >  	.get_unmapped_area = thp_get_unmapped_area,
> >  	.fallocate	= xfs_file_fallocate,
> > +	.fadvise	= xfs_file_fadvise,
> >  	.remap_file_range = xfs_file_remap_range,
> >  };
> >  
> > -- 
> > 2.16.4
> > 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-08-30 15:24     ` Jan Kara
@ 2019-08-30 16:02       ` Darrick J. Wong
  2019-09-18 12:31       ` Jan Kara
  1 sibling, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2019-08-30 16:02 UTC (permalink / raw)
  To: Jan Kara
  Cc: linux-xfs, linux-mm, Amir Goldstein, Boaz Harrosh, linux-fsdevel, stable

On Fri, Aug 30, 2019 at 05:24:49PM +0200, Jan Kara wrote:
> On Thu 29-08-19 08:52:04, Darrick J. Wong wrote:
> > On Thu, Aug 29, 2019 at 03:10:34PM +0200, Jan Kara wrote:
> > > Hole puching currently evicts pages from page cache and then goes on to
> > > remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
> > > and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
> > > racing reads or page faults. However there is currently nothing that
> > > prevents readahead triggered by fadvise() or madvise() from racing with
> > > the hole punch and instantiating page cache page after hole punching has
> > > evicted page cache in xfs_flush_unmap_range() but before it has removed
> > > blocks from the inode. This page cache page will be mapping soon to be
> > > freed block and that can lead to returning stale data to userspace or
> > > even filesystem corruption.
> > > 
> > > Fix the problem by protecting handling of readahead requests by
> > > XFS_IOLOCK_SHARED similarly as we protect reads.
> > > 
> > > CC: stable@vger.kernel.org
> > > Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
> > > Reported-by: Amir Goldstein <amir73il@gmail.com>
> > > Signed-off-by: Jan Kara <jack@suse.cz>
> > 
> > Is there a test on xfstests to demonstrate this race?
> 
> No, but I can try to create one.

<nod> I imgaine this race was hard to spot in the first place...

> > Will test it out though...
> > 
> > Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Thanks. BTW, will you pick up these patches please?

Yeah, they looked fine.

--D

> 								Honza
> 
> > 
> > --D
> > 
> > > ---
> > >  fs/xfs/xfs_file.c | 26 ++++++++++++++++++++++++++
> > >  1 file changed, 26 insertions(+)
> > > 
> > > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> > > index 28101bbc0b78..d952d5962e93 100644
> > > --- a/fs/xfs/xfs_file.c
> > > +++ b/fs/xfs/xfs_file.c
> > > @@ -28,6 +28,7 @@
> > >  #include <linux/falloc.h>
> > >  #include <linux/backing-dev.h>
> > >  #include <linux/mman.h>
> > > +#include <linux/fadvise.h>
> > >  
> > >  static const struct vm_operations_struct xfs_file_vm_ops;
> > >  
> > > @@ -933,6 +934,30 @@ xfs_file_fallocate(
> > >  	return error;
> > >  }
> > >  
> > > +STATIC int
> > > +xfs_file_fadvise(
> > > +	struct file	*file,
> > > +	loff_t		start,
> > > +	loff_t		end,
> > > +	int		advice)
> > > +{
> > > +	struct xfs_inode *ip = XFS_I(file_inode(file));
> > > +	int ret;

> > > +	int lockflags = 0;
> > > +
> > > +	/*
> > > +	 * Operations creating pages in page cache need protection from hole
> > > +	 * punching and similar ops
> > > +	 */
> > > +	if (advice == POSIX_FADV_WILLNEED) {
> > > +		lockflags = XFS_IOLOCK_SHARED;
> > > +		xfs_ilock(ip, lockflags);
> > > +	}
> > > +	ret = generic_fadvise(file, start, end, advice);
> > > +	if (lockflags)
> > > +		xfs_iunlock(ip, lockflags);
> > > +	return ret;
> > > +}
> > >  
> > >  STATIC loff_t
> > >  xfs_file_remap_range(
> > > @@ -1232,6 +1257,7 @@ const struct file_operations xfs_file_operations = {
> > >  	.fsync		= xfs_file_fsync,
> > >  	.get_unmapped_area = thp_get_unmapped_area,
> > >  	.fallocate	= xfs_file_fallocate,
> > > +	.fadvise	= xfs_file_fadvise,
> > >  	.remap_file_range = xfs_file_remap_range,
> > >  };
> > >  
> > > -- 
> > > 2.16.4
> > > 
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-08-30 15:24     ` Jan Kara
  2019-08-30 16:02       ` Darrick J. Wong
@ 2019-09-18 12:31       ` Jan Kara
  2019-09-18 16:07         ` Darrick J. Wong
  2019-09-23 12:33         ` Boaz Harrosh
  1 sibling, 2 replies; 24+ messages in thread
From: Jan Kara @ 2019-09-18 12:31 UTC (permalink / raw)
  To: Darrick J. Wong
  Cc: Jan Kara, linux-xfs, linux-mm, Amir Goldstein, Boaz Harrosh,
	linux-fsdevel, stable

On Fri 30-08-19 17:24:49, Jan Kara wrote:
> On Thu 29-08-19 08:52:04, Darrick J. Wong wrote:
> > On Thu, Aug 29, 2019 at 03:10:34PM +0200, Jan Kara wrote:
> > > Hole puching currently evicts pages from page cache and then goes on to
> > > remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
> > > and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
> > > racing reads or page faults. However there is currently nothing that
> > > prevents readahead triggered by fadvise() or madvise() from racing with
> > > the hole punch and instantiating page cache page after hole punching has
> > > evicted page cache in xfs_flush_unmap_range() but before it has removed
> > > blocks from the inode. This page cache page will be mapping soon to be
> > > freed block and that can lead to returning stale data to userspace or
> > > even filesystem corruption.
> > > 
> > > Fix the problem by protecting handling of readahead requests by
> > > XFS_IOLOCK_SHARED similarly as we protect reads.
> > > 
> > > CC: stable@vger.kernel.org
> > > Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
> > > Reported-by: Amir Goldstein <amir73il@gmail.com>
> > > Signed-off-by: Jan Kara <jack@suse.cz>
> > 
> > Is there a test on xfstests to demonstrate this race?
> 
> No, but I can try to create one.

I was experimenting with this but I could not reproduce the issue in my
test VM without inserting artificial delay at appropriate place... So I
don't think there's much point in the fstest for this.

								Honza

-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-09-18 12:31       ` Jan Kara
@ 2019-09-18 16:07         ` Darrick J. Wong
  2019-09-23 12:33         ` Boaz Harrosh
  1 sibling, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2019-09-18 16:07 UTC (permalink / raw)
  To: Jan Kara
  Cc: linux-xfs, linux-mm, Amir Goldstein, Boaz Harrosh, linux-fsdevel, stable

On Wed, Sep 18, 2019 at 02:31:24PM +0200, Jan Kara wrote:
> On Fri 30-08-19 17:24:49, Jan Kara wrote:
> > On Thu 29-08-19 08:52:04, Darrick J. Wong wrote:
> > > On Thu, Aug 29, 2019 at 03:10:34PM +0200, Jan Kara wrote:
> > > > Hole puching currently evicts pages from page cache and then goes on to
> > > > remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
> > > > and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
> > > > racing reads or page faults. However there is currently nothing that
> > > > prevents readahead triggered by fadvise() or madvise() from racing with
> > > > the hole punch and instantiating page cache page after hole punching has
> > > > evicted page cache in xfs_flush_unmap_range() but before it has removed
> > > > blocks from the inode. This page cache page will be mapping soon to be
> > > > freed block and that can lead to returning stale data to userspace or
> > > > even filesystem corruption.
> > > > 
> > > > Fix the problem by protecting handling of readahead requests by
> > > > XFS_IOLOCK_SHARED similarly as we protect reads.
> > > > 
> > > > CC: stable@vger.kernel.org
> > > > Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
> > > > Reported-by: Amir Goldstein <amir73il@gmail.com>
> > > > Signed-off-by: Jan Kara <jack@suse.cz>
> > > 
> > > Is there a test on xfstests to demonstrate this race?
> > 
> > No, but I can try to create one.
> 
> I was experimenting with this but I could not reproduce the issue in my
> test VM without inserting artificial delay at appropriate place... So I
> don't think there's much point in the fstest for this.

<shrug> We've added debugging knobs to XFS that inject delays to
demonstrate race conditions that are hard to reproduce, but OTOH it's
more fun to have a generic/ test that you can use to convince the other
fs maintainers to take your patches. :)

--D

> 								Honza
> 
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-09-18 12:31       ` Jan Kara
  2019-09-18 16:07         ` Darrick J. Wong
@ 2019-09-23 12:33         ` Boaz Harrosh
  2019-09-24 15:23           ` Jan Kara
  1 sibling, 1 reply; 24+ messages in thread
From: Boaz Harrosh @ 2019-09-23 12:33 UTC (permalink / raw)
  To: Jan Kara, Darrick J. Wong
  Cc: linux-xfs, linux-mm, Amir Goldstein, Boaz Harrosh, linux-fsdevel, stable

On 18/09/2019 15:31, Jan Kara wrote:
<>
>>> Is there a test on xfstests to demonstrate this race?
>>
>> No, but I can try to create one.
> 
> I was experimenting with this but I could not reproduce the issue in my
> test VM without inserting artificial delay at appropriate place... So I
> don't think there's much point in the fstest for this.
> 
> 								Honza
> 

If I understand correctly you will need threads that direct-write
files, then fadvise(WILL_NEED) - in parallel to truncate (punch_hole) these
files - In parallel to trash caches.
(Direct-write is so data is not present in cache when you come to WILL_NEED
 it into the cache, otherwise the xfs b-trees are not exercised. Or are you
 more worried about the page_cache races?
)

Also the d-writes might want to exercise multiple size extents + holes as
well.

I have a very different system but its kind of the test we did for this
problem.

The reason it is never hit is because fadvise(WILL_NEED) is never really
used that much, and there are no applications that actually blindly truncate
during IO, this is only us in testing that do this meaningless thing.

Thanks Jan again for working on this
Boaz

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-09-23 12:33         ` Boaz Harrosh
@ 2019-09-24 15:23           ` Jan Kara
  2019-09-24 15:45             ` Boaz Harrosh
  0 siblings, 1 reply; 24+ messages in thread
From: Jan Kara @ 2019-09-24 15:23 UTC (permalink / raw)
  To: Boaz Harrosh
  Cc: Jan Kara, Darrick J. Wong, linux-xfs, linux-mm, Amir Goldstein,
	Boaz Harrosh, linux-fsdevel, stable

On Mon 23-09-19 15:33:05, Boaz Harrosh wrote:
> On 18/09/2019 15:31, Jan Kara wrote:
> <>
> >>> Is there a test on xfstests to demonstrate this race?
> >>
> >> No, but I can try to create one.
> > 
> > I was experimenting with this but I could not reproduce the issue in my
> > test VM without inserting artificial delay at appropriate place... So I
> > don't think there's much point in the fstest for this.
> > 
> > 								Honza
> > 
> 
> If I understand correctly you will need threads that direct-write
> files, then fadvise(WILL_NEED) - in parallel to truncate (punch_hole) these
> files - In parallel to trash caches.
> (Direct-write is so data is not present in cache when you come to WILL_NEED
>  it into the cache, otherwise the xfs b-trees are not exercised. Or are you
>  more worried about the page_cache races?
> )

What I was testing was:
  Fill file with data.
  One process does fadvise(WILLNEED) block by block from end of the file.
  Another process punches hole into the file.

If they race is the right way, following read will show old data instead of
zeros. And as I said I'm able to hit this but only if I add artificial
delay between truncating page cache and actually removing blocks.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-09-24 15:23           ` Jan Kara
@ 2019-09-24 15:45             ` Boaz Harrosh
  0 siblings, 0 replies; 24+ messages in thread
From: Boaz Harrosh @ 2019-09-24 15:45 UTC (permalink / raw)
  To: Jan Kara
  Cc: Darrick J. Wong, linux-xfs, linux-mm, Amir Goldstein,
	Boaz Harrosh, linux-fsdevel, stable

On 24/09/2019 18:23, Jan Kara wrote:
> On Mon 23-09-19 15:33:05, Boaz Harrosh wrote:
>> On 18/09/2019 15:31, Jan Kara wrote:
>> <>
>>>>> Is there a test on xfstests to demonstrate this race?
>>>>
>>>> No, but I can try to create one.
>>>
>>> I was experimenting with this but I could not reproduce the issue in my
>>> test VM without inserting artificial delay at appropriate place... So I
>>> don't think there's much point in the fstest for this.
>>>
>>> 								Honza
>>>
>>
>> If I understand correctly you will need threads that direct-write
>> files, then fadvise(WILL_NEED) - in parallel to truncate (punch_hole) these
>> files - In parallel to trash caches.
>> (Direct-write is so data is not present in cache when you come to WILL_NEED
>>  it into the cache, otherwise the xfs b-trees are not exercised. Or are you
>>  more worried about the page_cache races?
>> )
> 
> What I was testing was:
>   Fill file with data.

But are you sure data is not in page cache after this stage?

Also this stage sould create multiple extents perhaps with gaps in between

>   One process does fadvise(WILLNEED) block by block from end of the file.
>   Another process punches hole into the file.
> 

(Perhaps randome placement that spans multiple extents in one go)

> If they race is the right way, following read will show old data instead of
> zeros. And as I said I'm able to hit this but only if I add artificial
> delay between truncating page cache and actually removing blocks.
> 

I was more afraid of iterating on a btree or xarray in parallel of it being
destroyed / punched.

I think if you iterate backwards in the WILLNEED case the tree/xarry corruption is
less likely

But now that I think about it. Maybe your case is very different to mine, because
read_pages() in xfs does take the ilock. I'm not familiar with this code.

> 								Honza
> 

But I guess the window is very small then

Thanks
Boaz

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching
  2019-08-29 13:10 [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Jan Kara
                   ` (2 preceding siblings ...)
  2019-08-29 13:10 ` [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch Jan Kara
@ 2020-01-17 10:50 ` Amir Goldstein
  2020-01-19  8:35   ` Amir Goldstein
  2020-01-20 12:03   ` Jan Kara
  3 siblings, 2 replies; 24+ messages in thread
From: Amir Goldstein @ 2020-01-17 10:50 UTC (permalink / raw)
  To: Jan Kara
  Cc: linux-xfs, Linux MM, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Matthew Wilcox, Jens Axboe

On Thu, Aug 29, 2019 at 4:10 PM Jan Kara <jack@suse.cz> wrote:
>
> Hello,
>
> this is a patch series that addresses a possible race between readahead and
> hole punching Amir has discovered [1]. The first patch makes madvise(2) to
> handle readahead requests through fadvise infrastructure, the third patch
> then adds necessary locking to XFS to protect against the race. Note that
> other filesystems need similar protections but e.g. in case of ext4 it isn't
> so simple without seriously regressing mixed rw workload performance so
> I'm pushing just xfs fix at this moment which is simple.
>

Jan,

Could you give a quick status update about the state of this issue for
ext4 and other fs. I remember some solutions were discussed.
Perhaps this could be a good topic for a cross track session in LSF/MM?
Aren't the challenges posed by this race also relevant for RWF_UNCACHED?

Thanks,
Amir.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching
  2020-01-17 10:50 ` [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Amir Goldstein
@ 2020-01-19  8:35   ` Amir Goldstein
  2020-01-20 11:47     ` Jan Kara
  2020-01-20 12:03   ` Jan Kara
  1 sibling, 1 reply; 24+ messages in thread
From: Amir Goldstein @ 2020-01-19  8:35 UTC (permalink / raw)
  To: Jan Kara
  Cc: linux-xfs, Linux MM, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Matthew Wilcox, Jens Axboe, Dave Chinner

On Fri, Jan 17, 2020 at 12:50 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Thu, Aug 29, 2019 at 4:10 PM Jan Kara <jack@suse.cz> wrote:
> >
> > Hello,
> >
> > this is a patch series that addresses a possible race between readahead and
> > hole punching Amir has discovered [1]. The first patch makes madvise(2) to
> > handle readahead requests through fadvise infrastructure, the third patch
> > then adds necessary locking to XFS to protect against the race. Note that
> > other filesystems need similar protections but e.g. in case of ext4 it isn't
> > so simple without seriously regressing mixed rw workload performance so
> > I'm pushing just xfs fix at this moment which is simple.
> >
>
> Jan,
>
> Could you give a quick status update about the state of this issue for
> ext4 and other fs. I remember some solutions were discussed.
> Perhaps this could be a good topic for a cross track session in LSF/MM?
> Aren't the challenges posed by this race also relevant for RWF_UNCACHED?
>

Maybe a silly question:

Can someone please explain to me why we even bother truncating pages on
punch hole?
Wouldn't it solve the race if instead we zeroed those pages and marked them
readonly?

The comment above trunacte_pagecache_range() says:
 * This function should typically be called before the filesystem
 * releases resources associated with the freed range (eg. deallocates
 * blocks). This way, pagecache will always stay logically coherent
 * with on-disk format, and the filesystem would not have to deal with
 * situations such as writepage being called for a page that has already
 * had its underlying blocks deallocated.

So in order to prevent writepage from being called on a punched hole,
we need to make sure that page write fault will be called, which is the same
state as if an exiting hole has been read into page cache but not written yet.
Right? Wrong?

What am I missing here?

Thanks,
Amir.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching
  2020-01-19  8:35   ` Amir Goldstein
@ 2020-01-20 11:47     ` Jan Kara
  0 siblings, 0 replies; 24+ messages in thread
From: Jan Kara @ 2020-01-20 11:47 UTC (permalink / raw)
  To: Amir Goldstein
  Cc: Jan Kara, linux-xfs, Linux MM, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Matthew Wilcox, Jens Axboe, Dave Chinner

On Sun 19-01-20 10:35:08, Amir Goldstein wrote:
> On Fri, Jan 17, 2020 at 12:50 PM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > On Thu, Aug 29, 2019 at 4:10 PM Jan Kara <jack@suse.cz> wrote:
> > >
> > > Hello,
> > >
> > > this is a patch series that addresses a possible race between readahead and
> > > hole punching Amir has discovered [1]. The first patch makes madvise(2) to
> > > handle readahead requests through fadvise infrastructure, the third patch
> > > then adds necessary locking to XFS to protect against the race. Note that
> > > other filesystems need similar protections but e.g. in case of ext4 it isn't
> > > so simple without seriously regressing mixed rw workload performance so
> > > I'm pushing just xfs fix at this moment which is simple.
> > >
> >
> > Jan,
> >
> > Could you give a quick status update about the state of this issue for
> > ext4 and other fs. I remember some solutions were discussed.
> > Perhaps this could be a good topic for a cross track session in LSF/MM?
> > Aren't the challenges posed by this race also relevant for RWF_UNCACHED?
> >
> 
> Maybe a silly question:
> 
> Can someone please explain to me why we even bother truncating pages on
> punch hole?
> Wouldn't it solve the race if instead we zeroed those pages and marked them
> readonly?

Not if we also didn't keep them locked. Page reclaim can reclaim clean
unlocked pages any time it wants... Plus the CPU overhead of zeroing
potentially large ranges of pages would be significant.

> The comment above trunacte_pagecache_range() says:
>  * This function should typically be called before the filesystem
>  * releases resources associated with the freed range (eg. deallocates
>  * blocks). This way, pagecache will always stay logically coherent
>  * with on-disk format, and the filesystem would not have to deal with
>  * situations such as writepage being called for a page that has already
>  * had its underlying blocks deallocated.
> 
> So in order to prevent writepage from being called on a punched hole,
> we need to make sure that page write fault will be called, which is the same
> state as if an exiting hole has been read into page cache but not written yet.
> Right? Wrong?

Also the writeback in the comment you mention above is just an example. As
the race you've found shows, there is a problem with reading as well.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching
  2020-01-17 10:50 ` [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Amir Goldstein
  2020-01-19  8:35   ` Amir Goldstein
@ 2020-01-20 12:03   ` Jan Kara
  2020-01-20 13:54     ` Amir Goldstein
  1 sibling, 1 reply; 24+ messages in thread
From: Jan Kara @ 2020-01-20 12:03 UTC (permalink / raw)
  To: Amir Goldstein
  Cc: Jan Kara, linux-xfs, Linux MM, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Matthew Wilcox, Jens Axboe

Hi Amir!

On Fri 17-01-20 12:50:58, Amir Goldstein wrote:
> On Thu, Aug 29, 2019 at 4:10 PM Jan Kara <jack@suse.cz> wrote:
> >
> > Hello,
> >
> > this is a patch series that addresses a possible race between readahead and
> > hole punching Amir has discovered [1]. The first patch makes madvise(2) to
> > handle readahead requests through fadvise infrastructure, the third patch
> > then adds necessary locking to XFS to protect against the race. Note that
> > other filesystems need similar protections but e.g. in case of ext4 it isn't
> > so simple without seriously regressing mixed rw workload performance so
> > I'm pushing just xfs fix at this moment which is simple.
> >
> 
> Could you give a quick status update about the state of this issue for
> ext4 and other fs. I remember some solutions were discussed.

Shortly: I didn't get to this. I'm sorry :-|. I'll bump up a priority but I
can't promise anything at the moment.

> Perhaps this could be a good topic for a cross track session in LSF/MM?

Maybe although this is one of the cases where it's easy to chat about
possible solutions but somewhat tedious to write one so I'm not sure how
productive that would be. BTW my discussion with Kent [1] is in fact very
related to this problem (the interval lock he has is to stop exactly races
like this).

> Aren't the challenges posed by this race also relevant for RWF_UNCACHED?

Do you have anything particular in mind? I don't see how RWF_UNCACHED would
make this any better or worse than DIO / readahead...

								Honza

[1] https://lore.kernel.org/linux-fsdevel/20191216193852.GA8664@kmo-pixel/
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching
  2020-01-20 12:03   ` Jan Kara
@ 2020-01-20 13:54     ` Amir Goldstein
  2020-01-20 16:58       ` Jan Kara
  0 siblings, 1 reply; 24+ messages in thread
From: Amir Goldstein @ 2020-01-20 13:54 UTC (permalink / raw)
  To: Jan Kara
  Cc: linux-xfs, Linux MM, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Matthew Wilcox, Jens Axboe, Kent Overstreet

On Mon, Jan 20, 2020 at 2:03 PM Jan Kara <jack@suse.cz> wrote:
>
> Hi Amir!
>
> On Fri 17-01-20 12:50:58, Amir Goldstein wrote:
> > On Thu, Aug 29, 2019 at 4:10 PM Jan Kara <jack@suse.cz> wrote:
> > >
> > > Hello,
> > >
> > > this is a patch series that addresses a possible race between readahead and
> > > hole punching Amir has discovered [1]. The first patch makes madvise(2) to
> > > handle readahead requests through fadvise infrastructure, the third patch
> > > then adds necessary locking to XFS to protect against the race. Note that
> > > other filesystems need similar protections but e.g. in case of ext4 it isn't
> > > so simple without seriously regressing mixed rw workload performance so
> > > I'm pushing just xfs fix at this moment which is simple.
> > >
> >
> > Could you give a quick status update about the state of this issue for
> > ext4 and other fs. I remember some solutions were discussed.
>
> Shortly: I didn't get to this. I'm sorry :-|. I'll bump up a priority but I
> can't promise anything at the moment.
>
> > Perhaps this could be a good topic for a cross track session in LSF/MM?
>
> Maybe although this is one of the cases where it's easy to chat about
> possible solutions but somewhat tedious to write one so I'm not sure how
> productive that would be. BTW my discussion with Kent [1] is in fact very
> related to this problem (the interval lock he has is to stop exactly races
> like this).
>

Well, I was mostly interested to know if there is an agreement on the way to
solve the problem. If we need to discuss it to reach consensus than it might
be a good topic for LSF/MM. If you already know what needs to be done,
there is no need for a discussion.

> > Aren't the challenges posed by this race also relevant for RWF_UNCACHED?
>
> Do you have anything particular in mind? I don't see how RWF_UNCACHED would
> make this any better or worse than DIO / readahead...
>

Not better nor worse. I meant that RFW_UNCACHED is another case that
would suffer the same races.

Thanks,
Amir.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching
  2020-01-20 13:54     ` Amir Goldstein
@ 2020-01-20 16:58       ` Jan Kara
  0 siblings, 0 replies; 24+ messages in thread
From: Jan Kara @ 2020-01-20 16:58 UTC (permalink / raw)
  To: Amir Goldstein
  Cc: Jan Kara, linux-xfs, Linux MM, Darrick J. Wong, Boaz Harrosh,
	linux-fsdevel, Matthew Wilcox, Jens Axboe, Kent Overstreet

On Mon 20-01-20 15:54:28, Amir Goldstein wrote:
> On Mon, Jan 20, 2020 at 2:03 PM Jan Kara <jack@suse.cz> wrote:
> > On Fri 17-01-20 12:50:58, Amir Goldstein wrote:
> > > On Thu, Aug 29, 2019 at 4:10 PM Jan Kara <jack@suse.cz> wrote:
> > > >
> > > > Hello,
> > > >
> > > > this is a patch series that addresses a possible race between readahead and
> > > > hole punching Amir has discovered [1]. The first patch makes madvise(2) to
> > > > handle readahead requests through fadvise infrastructure, the third patch
> > > > then adds necessary locking to XFS to protect against the race. Note that
> > > > other filesystems need similar protections but e.g. in case of ext4 it isn't
> > > > so simple without seriously regressing mixed rw workload performance so
> > > > I'm pushing just xfs fix at this moment which is simple.
> > > >
> > >
> > > Could you give a quick status update about the state of this issue for
> > > ext4 and other fs. I remember some solutions were discussed.
> >
> > Shortly: I didn't get to this. I'm sorry :-|. I'll bump up a priority but I
> > can't promise anything at the moment.
> >
> > > Perhaps this could be a good topic for a cross track session in LSF/MM?
> >
> > Maybe although this is one of the cases where it's easy to chat about
> > possible solutions but somewhat tedious to write one so I'm not sure how
> > productive that would be. BTW my discussion with Kent [1] is in fact very
> > related to this problem (the interval lock he has is to stop exactly races
> > like this).
> >
> 
> Well, I was mostly interested to know if there is an agreement on the way to
> solve the problem. If we need to discuss it to reach consensus than it might
> be a good topic for LSF/MM. If you already know what needs to be done,
> there is no need for a discussion.

So I have an idea how it could be solved: Change calling convention for
->readpage() so that it gets called without page locked and take
i_mmap_sem there (and in ->readpages()) to protect from the race. But
I wanted to present it in the form of patches as the devil here is in the
details and it may prove to be too ugly to be bearable.

If I won't get to writing the patches, you're right it may be sensible to
present the idea to people at LSF/MM what they think about it.

> > > Aren't the challenges posed by this race also relevant for RWF_UNCACHED?
> >
> > Do you have anything particular in mind? I don't see how RWF_UNCACHED would
> > make this any better or worse than DIO / readahead...
> >
> 
> Not better nor worse. I meant that RFW_UNCACHED is another case that
> would suffer the same races.

Yes, that's right.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-07-12 12:00       ` Jan Kara
@ 2019-07-12 17:56         ` Darrick J. Wong
  0 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2019-07-12 17:56 UTC (permalink / raw)
  To: Jan Kara
  Cc: Amir Goldstein, linux-fsdevel, Linux MM, linux-xfs, Boaz Harrosh, stable

On Fri, Jul 12, 2019 at 02:00:04PM +0200, Jan Kara wrote:
> On Thu 11-07-19 08:49:17, Darrick J. Wong wrote:
> > On Thu, Jul 11, 2019 at 06:28:54PM +0300, Amir Goldstein wrote:
> > > > +{
> > > > +       struct xfs_inode *ip = XFS_I(file_inode(file));
> > > > +       int ret;
> > > > +
> > > > +       /* Readahead needs protection from hole punching and similar ops */
> > > > +       if (advice == POSIX_FADV_WILLNEED)
> > > > +               xfs_ilock(ip, XFS_IOLOCK_SHARED);
> > 
> > It's good to fix this race, but at the same time I wonder what's the
> > impact to processes writing to one part of a file waiting on IOLOCK_EXCL
> > while readahead holds IOLOCK_SHARED?
> > 
> > (bluh bluh range locks ftw bluh bluh)
> 
> Yeah, with range locks this would have less impact. Also note that we hold
> the lock only during page setup and IO submission. IO itself will already
> happen without IOLOCK, only under page lock. But that's enough to stop the
> race.

> > Do we need a lock for DONTNEED?  I think the answer is that you have to
> > lock the page to drop it and that will protect us from <myriad punch and
> > truncate spaghetti> ... ?
> 
> Yeah, DONTNEED is just page writeback + invalidate. So page lock is enough
> to protect from anything bad. Essentially we need IOLOCK only to protect
> the places that creates new pages in page cache.
> 
> > > > +       ret = generic_fadvise(file, start, end, advice);
> > > > +       if (advice == POSIX_FADV_WILLNEED)
> > > > +               xfs_iunlock(ip, XFS_IOLOCK_SHARED);
> > 
> > Maybe it'd be better to do:
> > 
> > 	int	lockflags = 0;
> > 
> > 	if (advice == POSIX_FADV_WILLNEED) {
> > 		lockflags = XFS_IOLOCK_SHARED;
> > 		xfs_ilock(ip, lockflags);
> > 	}
> > 
> > 	ret = generic_fadvise(file, start, end, advice);
> > 
> > 	if (lockflags)
> > 		xfs_iunlock(ip, lockflags);
> > 
> > Just in case we some day want more or different types of inode locks?
> 
> OK, will do. Just I'll get to testing this only after I return from
> vacation.

<nod>

--D
> 
> 								Honza
> 
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-07-11 15:49     ` Darrick J. Wong
@ 2019-07-12 12:00       ` Jan Kara
  2019-07-12 17:56         ` Darrick J. Wong
  0 siblings, 1 reply; 24+ messages in thread
From: Jan Kara @ 2019-07-12 12:00 UTC (permalink / raw)
  To: Darrick J. Wong
  Cc: Amir Goldstein, Jan Kara, linux-fsdevel, Linux MM, linux-xfs,
	Boaz Harrosh, stable

On Thu 11-07-19 08:49:17, Darrick J. Wong wrote:
> On Thu, Jul 11, 2019 at 06:28:54PM +0300, Amir Goldstein wrote:
> > > +{
> > > +       struct xfs_inode *ip = XFS_I(file_inode(file));
> > > +       int ret;
> > > +
> > > +       /* Readahead needs protection from hole punching and similar ops */
> > > +       if (advice == POSIX_FADV_WILLNEED)
> > > +               xfs_ilock(ip, XFS_IOLOCK_SHARED);
> 
> It's good to fix this race, but at the same time I wonder what's the
> impact to processes writing to one part of a file waiting on IOLOCK_EXCL
> while readahead holds IOLOCK_SHARED?
> 
> (bluh bluh range locks ftw bluh bluh)

Yeah, with range locks this would have less impact. Also note that we hold
the lock only during page setup and IO submission. IO itself will already
happen without IOLOCK, only under page lock. But that's enough to stop the
race.

> Do we need a lock for DONTNEED?  I think the answer is that you have to
> lock the page to drop it and that will protect us from <myriad punch and
> truncate spaghetti> ... ?

Yeah, DONTNEED is just page writeback + invalidate. So page lock is enough
to protect from anything bad. Essentially we need IOLOCK only to protect
the places that creates new pages in page cache.

> > > +       ret = generic_fadvise(file, start, end, advice);
> > > +       if (advice == POSIX_FADV_WILLNEED)
> > > +               xfs_iunlock(ip, XFS_IOLOCK_SHARED);
> 
> Maybe it'd be better to do:
> 
> 	int	lockflags = 0;
> 
> 	if (advice == POSIX_FADV_WILLNEED) {
> 		lockflags = XFS_IOLOCK_SHARED;
> 		xfs_ilock(ip, lockflags);
> 	}
> 
> 	ret = generic_fadvise(file, start, end, advice);
> 
> 	if (lockflags)
> 		xfs_iunlock(ip, lockflags);
> 
> Just in case we some day want more or different types of inode locks?

OK, will do. Just I'll get to testing this only after I return from
vacation.

								Honza

-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-07-11 15:28   ` Amir Goldstein
@ 2019-07-11 15:49     ` Darrick J. Wong
  2019-07-12 12:00       ` Jan Kara
  0 siblings, 1 reply; 24+ messages in thread
From: Darrick J. Wong @ 2019-07-11 15:49 UTC (permalink / raw)
  To: Amir Goldstein
  Cc: Jan Kara, linux-fsdevel, Linux MM, linux-xfs, Boaz Harrosh, stable

On Thu, Jul 11, 2019 at 06:28:54PM +0300, Amir Goldstein wrote:
> On Thu, Jul 11, 2019 at 5:00 PM Jan Kara <jack@suse.cz> wrote:
> >
> > Hole puching currently evicts pages from page cache and then goes on to
> > remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
> > and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
> > racing reads or page faults. However there is currently nothing that
> > prevents readahead triggered by fadvise() or madvise() from racing with
> > the hole punch and instantiating page cache page after hole punching has
> > evicted page cache in xfs_flush_unmap_range() but before it has removed
> > blocks from the inode. This page cache page will be mapping soon to be
> > freed block and that can lead to returning stale data to userspace or
> > even filesystem corruption.
> >
> > Fix the problem by protecting handling of readahead requests by
> > XFS_IOLOCK_SHARED similarly as we protect reads.
> >
> > CC: stable@vger.kernel.org
> > Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
> > Reported-by: Amir Goldstein <amir73il@gmail.com>
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
> 
> Looks sane. (I'll let xfs developers offer reviewed-by tags)
> 
> >  fs/xfs/xfs_file.c | 20 ++++++++++++++++++++
> >  1 file changed, 20 insertions(+)
> >
> > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> > index 76748255f843..88fe3dbb3ba2 100644
> > --- a/fs/xfs/xfs_file.c
> > +++ b/fs/xfs/xfs_file.c
> > @@ -33,6 +33,7 @@
> >  #include <linux/pagevec.h>
> >  #include <linux/backing-dev.h>
> >  #include <linux/mman.h>
> > +#include <linux/fadvise.h>
> >
> >  static const struct vm_operations_struct xfs_file_vm_ops;
> >
> > @@ -939,6 +940,24 @@ xfs_file_fallocate(
> >         return error;
> >  }
> >
> > +STATIC int
> > +xfs_file_fadvise(
> > +       struct file *file,
> > +       loff_t start,
> > +       loff_t end,
> > +       int advice)

Indentation needs fixing here.

> > +{
> > +       struct xfs_inode *ip = XFS_I(file_inode(file));
> > +       int ret;
> > +
> > +       /* Readahead needs protection from hole punching and similar ops */
> > +       if (advice == POSIX_FADV_WILLNEED)
> > +               xfs_ilock(ip, XFS_IOLOCK_SHARED);

It's good to fix this race, but at the same time I wonder what's the
impact to processes writing to one part of a file waiting on IOLOCK_EXCL
while readahead holds IOLOCK_SHARED?

(bluh bluh range locks ftw bluh bluh)

Do we need a lock for DONTNEED?  I think the answer is that you have to
lock the page to drop it and that will protect us from <myriad punch and
truncate spaghetti> ... ?

> > +       ret = generic_fadvise(file, start, end, advice);
> > +       if (advice == POSIX_FADV_WILLNEED)
> > +               xfs_iunlock(ip, XFS_IOLOCK_SHARED);

Maybe it'd be better to do:

	int	lockflags = 0;

	if (advice == POSIX_FADV_WILLNEED) {
		lockflags = XFS_IOLOCK_SHARED;
		xfs_ilock(ip, lockflags);
	}

	ret = generic_fadvise(file, start, end, advice);

	if (lockflags)
		xfs_iunlock(ip, lockflags);

Just in case we some day want more or different types of inode locks?

--D

> > +       return ret;
> > +}
> >
> >  STATIC loff_t
> >  xfs_file_remap_range(
> > @@ -1235,6 +1254,7 @@ const struct file_operations xfs_file_operations = {
> >         .fsync          = xfs_file_fsync,
> >         .get_unmapped_area = thp_get_unmapped_area,
> >         .fallocate      = xfs_file_fallocate,
> > +       .fadvise        = xfs_file_fadvise,
> >         .remap_file_range = xfs_file_remap_range,
> >  };
> >
> > --
> > 2.16.4
> >

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-07-11 14:00 ` [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch Jan Kara
@ 2019-07-11 15:28   ` Amir Goldstein
  2019-07-11 15:49     ` Darrick J. Wong
  0 siblings, 1 reply; 24+ messages in thread
From: Amir Goldstein @ 2019-07-11 15:28 UTC (permalink / raw)
  To: Jan Kara; +Cc: linux-fsdevel, Linux MM, linux-xfs, Boaz Harrosh, stable

On Thu, Jul 11, 2019 at 5:00 PM Jan Kara <jack@suse.cz> wrote:
>
> Hole puching currently evicts pages from page cache and then goes on to
> remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
> and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
> racing reads or page faults. However there is currently nothing that
> prevents readahead triggered by fadvise() or madvise() from racing with
> the hole punch and instantiating page cache page after hole punching has
> evicted page cache in xfs_flush_unmap_range() but before it has removed
> blocks from the inode. This page cache page will be mapping soon to be
> freed block and that can lead to returning stale data to userspace or
> even filesystem corruption.
>
> Fix the problem by protecting handling of readahead requests by
> XFS_IOLOCK_SHARED similarly as we protect reads.
>
> CC: stable@vger.kernel.org
> Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
> Reported-by: Amir Goldstein <amir73il@gmail.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---

Looks sane. (I'll let xfs developers offer reviewed-by tags)

>  fs/xfs/xfs_file.c | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>
> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> index 76748255f843..88fe3dbb3ba2 100644
> --- a/fs/xfs/xfs_file.c
> +++ b/fs/xfs/xfs_file.c
> @@ -33,6 +33,7 @@
>  #include <linux/pagevec.h>
>  #include <linux/backing-dev.h>
>  #include <linux/mman.h>
> +#include <linux/fadvise.h>
>
>  static const struct vm_operations_struct xfs_file_vm_ops;
>
> @@ -939,6 +940,24 @@ xfs_file_fallocate(
>         return error;
>  }
>
> +STATIC int
> +xfs_file_fadvise(
> +       struct file *file,
> +       loff_t start,
> +       loff_t end,
> +       int advice)
> +{
> +       struct xfs_inode *ip = XFS_I(file_inode(file));
> +       int ret;
> +
> +       /* Readahead needs protection from hole punching and similar ops */
> +       if (advice == POSIX_FADV_WILLNEED)
> +               xfs_ilock(ip, XFS_IOLOCK_SHARED);
> +       ret = generic_fadvise(file, start, end, advice);
> +       if (advice == POSIX_FADV_WILLNEED)
> +               xfs_iunlock(ip, XFS_IOLOCK_SHARED);
> +       return ret;
> +}
>
>  STATIC loff_t
>  xfs_file_remap_range(
> @@ -1235,6 +1254,7 @@ const struct file_operations xfs_file_operations = {
>         .fsync          = xfs_file_fsync,
>         .get_unmapped_area = thp_get_unmapped_area,
>         .fallocate      = xfs_file_fallocate,
> +       .fadvise        = xfs_file_fadvise,
>         .remap_file_range = xfs_file_remap_range,
>  };
>
> --
> 2.16.4
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch
  2019-07-11 14:00 [PATCH 0/3] " Jan Kara
@ 2019-07-11 14:00 ` Jan Kara
  2019-07-11 15:28   ` Amir Goldstein
  0 siblings, 1 reply; 24+ messages in thread
From: Jan Kara @ 2019-07-11 14:00 UTC (permalink / raw)
  To: linux-fsdevel
  Cc: linux-mm, linux-xfs, Amir Goldstein, Boaz Harrosh, Jan Kara, stable

Hole puching currently evicts pages from page cache and then goes on to
remove blocks from the inode. This happens under both XFS_IOLOCK_EXCL
and XFS_MMAPLOCK_EXCL which provides appropriate serialization with
racing reads or page faults. However there is currently nothing that
prevents readahead triggered by fadvise() or madvise() from racing with
the hole punch and instantiating page cache page after hole punching has
evicted page cache in xfs_flush_unmap_range() but before it has removed
blocks from the inode. This page cache page will be mapping soon to be
freed block and that can lead to returning stale data to userspace or
even filesystem corruption.

Fix the problem by protecting handling of readahead requests by
XFS_IOLOCK_SHARED similarly as we protect reads.

CC: stable@vger.kernel.org
Link: https://lore.kernel.org/linux-fsdevel/CAOQ4uxjQNmxqmtA_VbYW0Su9rKRk2zobJmahcyeaEVOFKVQ5dw@mail.gmail.com/
Reported-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/xfs/xfs_file.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 76748255f843..88fe3dbb3ba2 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -33,6 +33,7 @@
 #include <linux/pagevec.h>
 #include <linux/backing-dev.h>
 #include <linux/mman.h>
+#include <linux/fadvise.h>
 
 static const struct vm_operations_struct xfs_file_vm_ops;
 
@@ -939,6 +940,24 @@ xfs_file_fallocate(
 	return error;
 }
 
+STATIC int
+xfs_file_fadvise(
+	struct file *file,
+	loff_t start,
+	loff_t end,
+	int advice)
+{
+	struct xfs_inode *ip = XFS_I(file_inode(file));
+	int ret;
+
+	/* Readahead needs protection from hole punching and similar ops */
+	if (advice == POSIX_FADV_WILLNEED)
+		xfs_ilock(ip, XFS_IOLOCK_SHARED);
+	ret = generic_fadvise(file, start, end, advice);
+	if (advice == POSIX_FADV_WILLNEED)
+		xfs_iunlock(ip, XFS_IOLOCK_SHARED);
+	return ret;
+}
 
 STATIC loff_t
 xfs_file_remap_range(
@@ -1235,6 +1254,7 @@ const struct file_operations xfs_file_operations = {
 	.fsync		= xfs_file_fsync,
 	.get_unmapped_area = thp_get_unmapped_area,
 	.fallocate	= xfs_file_fallocate,
+	.fadvise	= xfs_file_fadvise,
 	.remap_file_range = xfs_file_remap_range,
 };
 
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2020-01-20 17:04 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-29 13:10 [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Jan Kara
2019-08-29 13:10 ` [PATCH 1/3] mm: Handle MADV_WILLNEED through vfs_fadvise() Jan Kara
2019-08-29 15:49   ` Darrick J. Wong
2019-08-29 13:10 ` [PATCH 2/3] fs: Export generic_fadvise() Jan Kara
2019-08-29 13:10 ` [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch Jan Kara
2019-08-29 15:52   ` Darrick J. Wong
2019-08-30 15:24     ` Jan Kara
2019-08-30 16:02       ` Darrick J. Wong
2019-09-18 12:31       ` Jan Kara
2019-09-18 16:07         ` Darrick J. Wong
2019-09-23 12:33         ` Boaz Harrosh
2019-09-24 15:23           ` Jan Kara
2019-09-24 15:45             ` Boaz Harrosh
2020-01-17 10:50 ` [PATCH 0/3 v2] xfs: Fix races between readahead and hole punching Amir Goldstein
2020-01-19  8:35   ` Amir Goldstein
2020-01-20 11:47     ` Jan Kara
2020-01-20 12:03   ` Jan Kara
2020-01-20 13:54     ` Amir Goldstein
2020-01-20 16:58       ` Jan Kara
  -- strict thread matches above, loose matches on Subject: below --
2019-07-11 14:00 [PATCH 0/3] " Jan Kara
2019-07-11 14:00 ` [PATCH 3/3] xfs: Fix stale data exposure when readahead races with hole punch Jan Kara
2019-07-11 15:28   ` Amir Goldstein
2019-07-11 15:49     ` Darrick J. Wong
2019-07-12 12:00       ` Jan Kara
2019-07-12 17:56         ` Darrick J. Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).