[PATCH] xfs: prevent dropping ioend completions during buftarg wait

[PATCH] xfs: prevent dropping ioend completions during buftarg wait

[Brian Foster]

xfs_wait_buftarg() waits for all pending I/O, drains the ioend
completion workqueue and walks the LRU until all buffers in the cache
have been released. This is traditionally an unmount operation` but the
mechanism is also reused during filesystem freeze.

xfs_wait_buftarg() invokes drain_workqueue() as part of the quiesce,
which is intended more for a shutdown sequence in that it indicates to
the queue that new operations are not expected once the drain has begun.
New work jobs after this point result in a WARN_ON_ONCE() and are
otherwise dropped.

With filesystem freeze, however, read operations are allowed and can
proceed during or after the workqueue drain. If such a read occurs
during the drain sequence, the workqueue infrastructure complains about
the queued ioend completion work item and drops it on the floor. As a
result, the buffer remains on the LRU and the freeze never completes.

Despite the fact that the overall buffer cache cleanup is not necessary
during freeze, fix up this operation such that it is safe to invoke
during non-unmount quiesce operations. Replace the drain_workqueue()
call with flush_workqueue(), which runs a similar serialization on
pending workqueue jobs without causing new jobs to be dropped. This is
safe for unmount as unmount independently locks out new operations by
the time xfs_wait_buftarg() is invoked.

Signed-off-by: Brian Foster <bfoster@redhat.com>
---
 fs/xfs/xfs_buf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 607cc29..b5b9bff 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -1611,7 +1611,7 @@ xfs_wait_buftarg(
 	 */
 	while (percpu_counter_sum(&btp->bt_io_count))
 		delay(100);
-	drain_workqueue(btp->bt_mount->m_buf_workqueue);
+	flush_workqueue(btp->bt_mount->m_buf_workqueue);
 
 	/* loop until there is nothing left on the lru list. */
 	while (list_lru_count(&btp->bt_lru)) {
-- 
2.5.5

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH] xfs: prevent dropping ioend completions during buftarg wait

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH] xfs: prevent dropping ioend completions during buftarg wait

[Brian Foster]

On Tue, Sep 27, 2016 at 04:17:40PM +0200, Greg KH wrote:
> On Tue, Sep 27, 2016 at 10:03:57AM -0400, Brian Foster wrote:
> > commit 800b2694f890cc35a1bda63501fc71c94389d517 upstream.
> > 
> > xfs_wait_buftarg() waits for all pending I/O, drains the ioend
> > completion workqueue and walks the LRU until all buffers in the cache
> > have been released. This is traditionally an unmount operation` but the
> > mechanism is also reused during filesystem freeze.
> > 
> > xfs_wait_buftarg() invokes drain_workqueue() as part of the quiesce,
> > which is intended more for a shutdown sequence in that it indicates to
> > the queue that new operations are not expected once the drain has begun.
> > New work jobs after this point result in a WARN_ON_ONCE() and are
> > otherwise dropped.
> > 
> > With filesystem freeze, however, read operations are allowed and can
> > proceed during or after the workqueue drain. If such a read occurs
> > during the drain sequence, the workqueue infrastructure complains about
> > the queued ioend completion work item and drops it on the floor. As a
> > result, the buffer remains on the LRU and the freeze never completes.
> > 
> > Despite the fact that the overall buffer cache cleanup is not necessary
> > during freeze, fix up this operation such that it is safe to invoke
> > during non-unmount quiesce operations. Replace the drain_workqueue()
> > call with flush_workqueue(), which runs a similar serialization on
> > pending workqueue jobs without causing new jobs to be dropped. This is
> > safe for unmount as unmount independently locks out new operations by
> > the time xfs_wait_buftarg() is invoked.
> > 
> > cc: <stable@vger.kernel.org>
> > Signed-off-by: Brian Foster <bfoster@redhat.com>
> > Reviewed-by: Christoph Hellwig <hch@lst.de>
> > Signed-off-by: Dave Chinner <david@fromorbit.com>
> > ---
> > 
> > Note that this applies to linux-4.7.y.
> 
> It also applied to 4.4-stable, should I put it there as well?
> 

Yeah, it looks like we included "xfs: log mount failures don't wait for
buffers to be released" there so that probably makes sense. Thanks!

Brian

> thanks,
> 
> greg k-h

Re: [PATCH] xfs: prevent dropping ioend completions during buftarg wait

[Greg KH]

On Tue, Sep 27, 2016 at 10:03:57AM -0400, Brian Foster wrote:
> commit 800b2694f890cc35a1bda63501fc71c94389d517 upstream.
> 
> xfs_wait_buftarg() waits for all pending I/O, drains the ioend
> completion workqueue and walks the LRU until all buffers in the cache
> have been released. This is traditionally an unmount operation` but the
> mechanism is also reused during filesystem freeze.
> 
> xfs_wait_buftarg() invokes drain_workqueue() as part of the quiesce,
> which is intended more for a shutdown sequence in that it indicates to
> the queue that new operations are not expected once the drain has begun.
> New work jobs after this point result in a WARN_ON_ONCE() and are
> otherwise dropped.
> 
> With filesystem freeze, however, read operations are allowed and can
> proceed during or after the workqueue drain. If such a read occurs
> during the drain sequence, the workqueue infrastructure complains about
> the queued ioend completion work item and drops it on the floor. As a
> result, the buffer remains on the LRU and the freeze never completes.
> 
> Despite the fact that the overall buffer cache cleanup is not necessary
> during freeze, fix up this operation such that it is safe to invoke
> during non-unmount quiesce operations. Replace the drain_workqueue()
> call with flush_workqueue(), which runs a similar serialization on
> pending workqueue jobs without causing new jobs to be dropped. This is
> safe for unmount as unmount independently locks out new operations by
> the time xfs_wait_buftarg() is invoked.
> 
> cc: <stable@vger.kernel.org>
> Signed-off-by: Brian Foster <bfoster@redhat.com>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Dave Chinner <david@fromorbit.com>
> ---
> 
> Note that this applies to linux-4.7.y.

It also applied to 4.4-stable, should I put it there as well?

thanks,

greg k-h

[PATCH] xfs: prevent dropping ioend completions during buftarg wait

[Brian Foster]

commit 800b2694f890cc35a1bda63501fc71c94389d517 upstream.

xfs_wait_buftarg() waits for all pending I/O, drains the ioend
completion workqueue and walks the LRU until all buffers in the cache
have been released. This is traditionally an unmount operation` but the
mechanism is also reused during filesystem freeze.

xfs_wait_buftarg() invokes drain_workqueue() as part of the quiesce,
which is intended more for a shutdown sequence in that it indicates to
the queue that new operations are not expected once the drain has begun.
New work jobs after this point result in a WARN_ON_ONCE() and are
otherwise dropped.

With filesystem freeze, however, read operations are allowed and can
proceed during or after the workqueue drain. If such a read occurs
during the drain sequence, the workqueue infrastructure complains about
the queued ioend completion work item and drops it on the floor. As a
result, the buffer remains on the LRU and the freeze never completes.

Despite the fact that the overall buffer cache cleanup is not necessary
during freeze, fix up this operation such that it is safe to invoke
during non-unmount quiesce operations. Replace the drain_workqueue()
call with flush_workqueue(), which runs a similar serialization on
pending workqueue jobs without causing new jobs to be dropped. This is
safe for unmount as unmount independently locks out new operations by
the time xfs_wait_buftarg() is invoked.

cc: <stable@vger.kernel.org>
Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
---

Note that this applies to linux-4.7.y.

Brian

 fs/xfs/xfs_buf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index e71cfbd..41c20b6 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -1531,7 +1531,7 @@ xfs_wait_buftarg(
 	 * ensure here that all reference counts have been dropped before we
 	 * start walking the LRU list.
 	 */
-	drain_workqueue(btp->bt_mount->m_buf_workqueue);
+	flush_workqueue(btp->bt_mount->m_buf_workqueue);
 
 	/* loop until there is nothing left on the lru list. */
 	while (list_lru_count(&btp->bt_lru)) {
-- 
2.5.5