* [PATCH 0/2 V4] Resubmit items failed during writeback @ 2017-06-16 10:54 Carlos Maiolino 2017-06-16 10:54 ` [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure Carlos Maiolino ` (2 more replies) 0 siblings, 3 replies; 38+ messages in thread From: Carlos Maiolino @ 2017-06-16 10:54 UTC (permalink / raw) To: linux-xfs Hi, there goes a new version of this patchset based on previous reviews on V3. Changelogs added separated to each patch. -- 2.9.4 ^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure 2017-06-16 10:54 [PATCH 0/2 V4] Resubmit items failed during writeback Carlos Maiolino @ 2017-06-16 10:54 ` Carlos Maiolino 2017-06-19 13:48 ` Brian Foster 2017-06-16 10:54 ` [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback Carlos Maiolino 2017-06-19 13:51 ` [PATCH 0/2 V4] Resubmit items failed during writeback Brian Foster 2 siblings, 1 reply; 38+ messages in thread From: Carlos Maiolino @ 2017-06-16 10:54 UTC (permalink / raw) To: linux-xfs With the current code, XFS never re-submit a failed buffer for IO, because the failed item in the buffer is kept in the flush locked state forever. To be able to resubmit an log item for IO, we need a way to mark an item as failed, if, for any reason the buffer which the item belonged to failed during writeback. Add a new log item callback to be used after an IO completion failure and make the needed clean ups. Signed-off-by: Carlos Maiolino <cmaiolino@redhat.com> --- V2: - Update commit log to include a better description of why this patch is needed and fix spelling mistakes - Move xfs_buf_do_callbacks_fail() call into xfs_buf_iodone_callback_error, so the callbacks can be executed before the buffer is released, and only after it has been retried once V3: - fix some loops according to hch suggestion - whitespace cleanup V4: - Invoke failure callbacks before reset the I/O error - Remove bflags field from iop_error callback - move spin_lock/unlock xa_lock up in the stack, handling all log items in the same buffer into a single lock fs/xfs/xfs_buf_item.c | 23 ++++++++++++++++++++++- fs/xfs/xfs_trans.h | 7 +++++-- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c index 0306168..4fa68c9 100644 --- a/fs/xfs/xfs_buf_item.c +++ b/fs/xfs/xfs_buf_item.c @@ -29,6 +29,7 @@ #include "xfs_error.h" #include "xfs_trace.h" #include "xfs_log.h" +#include "xfs_inode.h" kmem_zone_t *xfs_buf_item_zone; @@ -1051,6 +1052,22 @@ xfs_buf_do_callbacks( } } +STATIC void +xfs_buf_do_callbacks_fail( + struct xfs_buf *bp) +{ + struct xfs_log_item *tli, *lip, *next; + + tli = bp->b_fspriv; + spin_lock(&tli->li_ailp->xa_lock); + for (lip = bp->b_fspriv; lip; lip = next) { + next = lip->li_bio_list; + if (lip->li_ops->iop_error) + lip->li_ops->iop_error(lip, bp); + } + spin_unlock(&tli->li_ailp->xa_lock); +} + static bool xfs_buf_iodone_callback_error( struct xfs_buf *bp) @@ -1120,7 +1137,11 @@ xfs_buf_iodone_callback_error( if ((mp->m_flags & XFS_MOUNT_UNMOUNTING) && mp->m_fail_unmount) goto permanent_error; - /* still a transient error, higher layers will retry */ + /* + * Still a transient error, run IO completion failure callbacks and let + * the higher layers retry the buffer. + */ + xfs_buf_do_callbacks_fail(bp); xfs_buf_ioerror(bp, 0); xfs_buf_relse(bp); return true; diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h index a07acbf..50df5367 100644 --- a/fs/xfs/xfs_trans.h +++ b/fs/xfs/xfs_trans.h @@ -64,11 +64,13 @@ typedef struct xfs_log_item { } xfs_log_item_t; #define XFS_LI_IN_AIL 0x1 -#define XFS_LI_ABORTED 0x2 +#define XFS_LI_ABORTED 0x2 +#define XFS_LI_FAILED 0x4 #define XFS_LI_FLAGS \ { XFS_LI_IN_AIL, "IN_AIL" }, \ - { XFS_LI_ABORTED, "ABORTED" } + { XFS_LI_ABORTED, "ABORTED" }, \ + { XFS_LI_FAILED, "FAILED" } struct xfs_item_ops { void (*iop_size)(xfs_log_item_t *, int *, int *); @@ -79,6 +81,7 @@ struct xfs_item_ops { void (*iop_unlock)(xfs_log_item_t *); xfs_lsn_t (*iop_committed)(xfs_log_item_t *, xfs_lsn_t); void (*iop_committing)(xfs_log_item_t *, xfs_lsn_t); + void (*iop_error)(xfs_log_item_t *, xfs_buf_t *); }; void xfs_log_item_init(struct xfs_mount *mp, struct xfs_log_item *item, -- 2.9.4 ^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure 2017-06-16 10:54 ` [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure Carlos Maiolino @ 2017-06-19 13:48 ` Brian Foster 2017-06-20 7:15 ` Carlos Maiolino 0 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-19 13:48 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs On Fri, Jun 16, 2017 at 12:54:44PM +0200, Carlos Maiolino wrote: > With the current code, XFS never re-submit a failed buffer for IO, > because the failed item in the buffer is kept in the flush locked state > forever. > > To be able to resubmit an log item for IO, we need a way to mark an item > as failed, if, for any reason the buffer which the item belonged to > failed during writeback. > > Add a new log item callback to be used after an IO completion failure > and make the needed clean ups. > > Signed-off-by: Carlos Maiolino <cmaiolino@redhat.com> > --- > V2: > - Update commit log to include a better description of why this > patch is needed and fix spelling mistakes > - Move xfs_buf_do_callbacks_fail() call into > xfs_buf_iodone_callback_error, so the callbacks can be executed > before the buffer is released, and only after it has been > retried once > > V3: > - fix some loops according to hch suggestion > - whitespace cleanup > > V4: > - Invoke failure callbacks before reset the I/O error > - Remove bflags field from iop_error callback > - move spin_lock/unlock xa_lock up in the stack, handling all > log items in the same buffer into a single lock > Looks mostly Ok to me, just a few nits... > fs/xfs/xfs_buf_item.c | 23 ++++++++++++++++++++++- > fs/xfs/xfs_trans.h | 7 +++++-- > 2 files changed, 27 insertions(+), 3 deletions(-) > > diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c > index 0306168..4fa68c9 100644 > --- a/fs/xfs/xfs_buf_item.c > +++ b/fs/xfs/xfs_buf_item.c > @@ -29,6 +29,7 @@ > #include "xfs_error.h" > #include "xfs_trace.h" > #include "xfs_log.h" > +#include "xfs_inode.h" > > > kmem_zone_t *xfs_buf_item_zone; > @@ -1051,6 +1052,22 @@ xfs_buf_do_callbacks( > } > } > > +STATIC void > +xfs_buf_do_callbacks_fail( > + struct xfs_buf *bp) > +{ > + struct xfs_log_item *tli, *lip, *next; > + > + tli = bp->b_fspriv; We can just define a local pointer to the xfs_ail rather than maintain a log item pointer purely for the purpose of the lock. E.g.: struct xfs_log_item *lip = bp->b_fspriv; struct xfs_ail *ailp = lip->li_ailp; spin_lock(&ailp->xa_lock); for (; lip; lip = next) { ... } ... It also couldn't hurt to ASSERT(bp->b_error) somewhere in here. > + spin_lock(&tli->li_ailp->xa_lock); > + for (lip = bp->b_fspriv; lip; lip = next) { > + next = lip->li_bio_list; > + if (lip->li_ops->iop_error) > + lip->li_ops->iop_error(lip, bp); > + } > + spin_unlock(&tli->li_ailp->xa_lock); > +} > + > static bool > xfs_buf_iodone_callback_error( > struct xfs_buf *bp) > @@ -1120,7 +1137,11 @@ xfs_buf_iodone_callback_error( > if ((mp->m_flags & XFS_MOUNT_UNMOUNTING) && mp->m_fail_unmount) > goto permanent_error; > > - /* still a transient error, higher layers will retry */ > + /* > + * Still a transient error, run IO completion failure callbacks and let > + * the higher layers retry the buffer. > + */ > + xfs_buf_do_callbacks_fail(bp); > xfs_buf_ioerror(bp, 0); > xfs_buf_relse(bp); > return true; > diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h > index a07acbf..50df5367 100644 > --- a/fs/xfs/xfs_trans.h > +++ b/fs/xfs/xfs_trans.h > @@ -64,11 +64,13 @@ typedef struct xfs_log_item { > } xfs_log_item_t; > > #define XFS_LI_IN_AIL 0x1 > -#define XFS_LI_ABORTED 0x2 > +#define XFS_LI_ABORTED 0x2 > +#define XFS_LI_FAILED 0x4 Weren't you planning to change these to (1 << N) format? Brian > > #define XFS_LI_FLAGS \ > { XFS_LI_IN_AIL, "IN_AIL" }, \ > - { XFS_LI_ABORTED, "ABORTED" } > + { XFS_LI_ABORTED, "ABORTED" }, \ > + { XFS_LI_FAILED, "FAILED" } > > struct xfs_item_ops { > void (*iop_size)(xfs_log_item_t *, int *, int *); > @@ -79,6 +81,7 @@ struct xfs_item_ops { > void (*iop_unlock)(xfs_log_item_t *); > xfs_lsn_t (*iop_committed)(xfs_log_item_t *, xfs_lsn_t); > void (*iop_committing)(xfs_log_item_t *, xfs_lsn_t); > + void (*iop_error)(xfs_log_item_t *, xfs_buf_t *); > }; > > void xfs_log_item_init(struct xfs_mount *mp, struct xfs_log_item *item, > -- > 2.9.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure 2017-06-19 13:48 ` Brian Foster @ 2017-06-20 7:15 ` Carlos Maiolino 0 siblings, 0 replies; 38+ messages in thread From: Carlos Maiolino @ 2017-06-20 7:15 UTC (permalink / raw) To: Brian Foster; +Cc: linux-xfs > Looks mostly Ok to me, just a few nits... > > > fs/xfs/xfs_buf_item.c | 23 ++++++++++++++++++++++- > > fs/xfs/xfs_trans.h | 7 +++++-- > > 2 files changed, 27 insertions(+), 3 deletions(-) > > > > diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c > > index 0306168..4fa68c9 100644 > > --- a/fs/xfs/xfs_buf_item.c > > +++ b/fs/xfs/xfs_buf_item.c > > @@ -29,6 +29,7 @@ > > #include "xfs_error.h" > > #include "xfs_trace.h" > > #include "xfs_log.h" > > +#include "xfs_inode.h" > > > > > > kmem_zone_t *xfs_buf_item_zone; > > @@ -1051,6 +1052,22 @@ xfs_buf_do_callbacks( > > } > > } > > > > +STATIC void > > +xfs_buf_do_callbacks_fail( > > + struct xfs_buf *bp) > > +{ > > + struct xfs_log_item *tli, *lip, *next; > > + > > + tli = bp->b_fspriv; > > We can just define a local pointer to the xfs_ail rather than maintain a > log item pointer purely for the purpose of the lock. E.g.: > > struct xfs_log_item *lip = bp->b_fspriv; > struct xfs_ail *ailp = lip->li_ailp; > > spin_lock(&ailp->xa_lock); > for (; lip; lip = next) { > ... > } > ... > > It also couldn't hurt to ASSERT(bp->b_error) somewhere in here. Sounds fair, I'll change these things > > > + spin_lock(&tli->li_ailp->xa_lock); > > + for (lip = bp->b_fspriv; lip; lip = next) { > > + next = lip->li_bio_list; > > + if (lip->li_ops->iop_error) > > + lip->li_ops->iop_error(lip, bp); > > + } > > + spin_unlock(&tli->li_ailp->xa_lock); > > +} > > + > > static bool > > xfs_buf_iodone_callback_error( > > struct xfs_buf *bp) > > @@ -1120,7 +1137,11 @@ xfs_buf_iodone_callback_error( > > if ((mp->m_flags & XFS_MOUNT_UNMOUNTING) && mp->m_fail_unmount) > > goto permanent_error; > > > > - /* still a transient error, higher layers will retry */ > > + /* > > + * Still a transient error, run IO completion failure callbacks and let > > + * the higher layers retry the buffer. > > + */ > > + xfs_buf_do_callbacks_fail(bp); > > xfs_buf_ioerror(bp, 0); > > xfs_buf_relse(bp); > > return true; > > diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h > > index a07acbf..50df5367 100644 > > --- a/fs/xfs/xfs_trans.h > > +++ b/fs/xfs/xfs_trans.h > > @@ -64,11 +64,13 @@ typedef struct xfs_log_item { > > } xfs_log_item_t; > > > > #define XFS_LI_IN_AIL 0x1 > > -#define XFS_LI_ABORTED 0x2 > > +#define XFS_LI_ABORTED 0x2 > > +#define XFS_LI_FAILED 0x4 > > Weren't you planning to change these to (1 << N) format? > Yeah, I just forgot to add it to my 'todo' for this patchset and it went to /dev/null during all the other changes :) I'll add it to the next one. > Brian > > > > > #define XFS_LI_FLAGS \ > > { XFS_LI_IN_AIL, "IN_AIL" }, \ > > - { XFS_LI_ABORTED, "ABORTED" } > > + { XFS_LI_ABORTED, "ABORTED" }, \ > > + { XFS_LI_FAILED, "FAILED" } > > > > struct xfs_item_ops { > > void (*iop_size)(xfs_log_item_t *, int *, int *); > > @@ -79,6 +81,7 @@ struct xfs_item_ops { > > void (*iop_unlock)(xfs_log_item_t *); > > xfs_lsn_t (*iop_committed)(xfs_log_item_t *, xfs_lsn_t); > > void (*iop_committing)(xfs_log_item_t *, xfs_lsn_t); > > + void (*iop_error)(xfs_log_item_t *, xfs_buf_t *); > > }; > > > > void xfs_log_item_init(struct xfs_mount *mp, struct xfs_log_item *item, > > -- > > 2.9.4 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 10:54 [PATCH 0/2 V4] Resubmit items failed during writeback Carlos Maiolino 2017-06-16 10:54 ` [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure Carlos Maiolino @ 2017-06-16 10:54 ` Carlos Maiolino 2017-06-16 11:06 ` Carlos Maiolino ` (2 more replies) 2017-06-19 13:51 ` [PATCH 0/2 V4] Resubmit items failed during writeback Brian Foster 2 siblings, 3 replies; 38+ messages in thread From: Carlos Maiolino @ 2017-06-16 10:54 UTC (permalink / raw) To: linux-xfs When a buffer has been failed during writeback, the inode items into it are kept flush locked, and are never resubmitted due the flush lock, so, if any buffer fails to be written, the items in AIL are never written to disk and never unlocked. This causes unmount operation to hang due these items flush locked in AIL, but this also causes the items in AIL to never be written back, even when the IO device comes back to normal. I've been testing this patch with a DM-thin device, creating a filesystem larger than the real device. When writing enough data to fill the DM-thin device, XFS receives ENOSPC errors from the device, and keep spinning on xfsaild (when 'retry forever' configuration is set). At this point, the filesystem can not be unmounted because of the flush locked items in AIL, but worse, the items in AIL are never retried at all (once xfs_inode_item_push() will skip the items that are flush locked), even if the underlying DM-thin device is expanded to the proper size. This patch fixes both cases, retrying any item that has been failed previously, using the infra-structure provided by the previous patch. Signed-off-by: Carlos Maiolino <cmaiolino@redhat.com> --- V2: - Fix XFS_LI_FAILED flag removal - Use atomic operations to set and clear XFS_LI_FAILED flag - Remove check for XBF_WRITE_FAIL in xfs_inode_item_push - Add more comments to the code - Add a helper function to resubmit the failed buffers, so this can be also used in dquot system without duplicating code V3: - kill xfs_imap_to_bp call using a pointer in the log item to hold the buffer address - use xa_lock instead of atomic operations to handle log item flags - Add a hold to the buffer for each log item failed - move buffer resubmission up in xfs_inode_item_push() V4: - Remove bflags argument from iop_error callback - Remove ip argument from xfs_buf_resubmit_failed_buffers - Use helpers to set/clear XFS_LI_FAILED flag - remove ->xa_lock from the iop->error callback and move it up on the stack, so all log items are processed into a single pair of lock/unlock fs/xfs/xfs_buf_item.c | 37 +++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_buf_item.h | 1 + fs/xfs/xfs_inode_item.c | 36 +++++++++++++++++++++++++++++++++--- fs/xfs/xfs_trans.h | 26 ++++++++++++++++++++++++++ fs/xfs/xfs_trans_ail.c | 4 ++-- 5 files changed, 99 insertions(+), 5 deletions(-) diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c index 4fa68c9..c5d21ea 100644 --- a/fs/xfs/xfs_buf_item.c +++ b/fs/xfs/xfs_buf_item.c @@ -1222,3 +1222,40 @@ xfs_buf_iodone( xfs_trans_ail_delete(ailp, lip, SHUTDOWN_CORRUPT_INCORE); xfs_buf_item_free(BUF_ITEM(lip)); } + +/* + * Requeue a failed buffer for writeback + * + * Return true if the buffer has been re-queued properly, false otherwise + */ +bool +xfs_buf_resubmit_failed_buffers( + struct xfs_log_item *lip, + struct list_head *buffer_list) +{ + struct xfs_log_item *next; + struct xfs_buf *bp; + bool ret; + + bp = lip->li_buf; + + /* Clear XFS_LI_FAILED flag from all items before resubmit + * + * XFS_LI_FAILED set/clear is protected by xa_lock, caller this + * function already have it acquired + */ + for (; lip; lip = next) { + next = lip->li_bio_list; + xfs_clear_li_failed(lip); + } + + /* Add this buffer back to the delayed write list */ + xfs_buf_lock(bp); + if (!xfs_buf_delwri_queue(bp, buffer_list)) + ret = false; + else + ret = true; + + xfs_buf_unlock(bp); + return ret; +} diff --git a/fs/xfs/xfs_buf_item.h b/fs/xfs/xfs_buf_item.h index f7eba99..82c3d64 100644 --- a/fs/xfs/xfs_buf_item.h +++ b/fs/xfs/xfs_buf_item.h @@ -70,6 +70,7 @@ void xfs_buf_attach_iodone(struct xfs_buf *, xfs_log_item_t *); void xfs_buf_iodone_callbacks(struct xfs_buf *); void xfs_buf_iodone(struct xfs_buf *, struct xfs_log_item *); +bool xfs_buf_resubmit_failed_buffers(struct xfs_log_item *, struct list_head *); extern kmem_zone_t *xfs_buf_item_zone; diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index 08cb7d1..2719ac6 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -27,6 +27,7 @@ #include "xfs_error.h" #include "xfs_trace.h" #include "xfs_trans_priv.h" +#include "xfs_buf_item.h" #include "xfs_log.h" @@ -475,6 +476,18 @@ xfs_inode_item_unpin( wake_up_bit(&ip->i_flags, __XFS_IPINNED_BIT); } +/* + * Callback used to mark a buffer with XFS_LI_FAILED when items in the buffer + * have been failed during writeback + */ +STATIC void +xfs_inode_item_error( + struct xfs_log_item *lip, + struct xfs_buf *bp) +{ + xfs_set_li_failed(lip, bp); +} + STATIC uint xfs_inode_item_push( struct xfs_log_item *lip, @@ -491,6 +504,17 @@ xfs_inode_item_push( if (xfs_ipincount(ip) > 0) return XFS_ITEM_PINNED; + /* + * The buffer containing this item failed to be written back + * previously. Resubmit the buffer for IO. + */ + if (lip->li_flags & XFS_LI_FAILED) { + if (!xfs_buf_resubmit_failed_buffers(lip, buffer_list)) + rval = XFS_ITEM_FLUSHING; + + return rval; + } + if (!xfs_ilock_nowait(ip, XFS_ILOCK_SHARED)) return XFS_ITEM_LOCKED; @@ -622,7 +646,8 @@ static const struct xfs_item_ops xfs_inode_item_ops = { .iop_unlock = xfs_inode_item_unlock, .iop_committed = xfs_inode_item_committed, .iop_push = xfs_inode_item_push, - .iop_committing = xfs_inode_item_committing + .iop_committing = xfs_inode_item_committing, + .iop_error = xfs_inode_item_error }; @@ -710,7 +735,8 @@ xfs_iflush_done( * the AIL lock. */ iip = INODE_ITEM(blip); - if (iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) + if ((iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) || + lip->li_flags & XFS_LI_FAILED) need_ail++; blip = next; @@ -718,7 +744,8 @@ xfs_iflush_done( /* make sure we capture the state of the initial inode. */ iip = INODE_ITEM(lip); - if (iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) + if ((iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) || + lip->li_flags & XFS_LI_FAILED) need_ail++; /* @@ -739,6 +766,9 @@ xfs_iflush_done( if (INODE_ITEM(blip)->ili_logged && blip->li_lsn == INODE_ITEM(blip)->ili_flush_lsn) mlip_changed |= xfs_ail_delete_one(ailp, blip); + else { + xfs_clear_li_failed(blip); + } } if (mlip_changed) { diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h index 50df5367..2d7cf91 100644 --- a/fs/xfs/xfs_trans.h +++ b/fs/xfs/xfs_trans.h @@ -49,6 +49,7 @@ typedef struct xfs_log_item { struct xfs_ail *li_ailp; /* ptr to AIL */ uint li_type; /* item type */ uint li_flags; /* misc flags */ + struct xfs_buf *li_buf; /* real buffer pointer */ struct xfs_log_item *li_bio_list; /* buffer item list */ void (*li_cb)(struct xfs_buf *, struct xfs_log_item *); @@ -72,6 +73,31 @@ typedef struct xfs_log_item { { XFS_LI_ABORTED, "ABORTED" }, \ { XFS_LI_FAILED, "FAILED" } +static inline void +xfs_clear_li_failed( + struct xfs_log_item *lip) +{ + struct xfs_buf *bp = lip->li_buf; + + if (lip->li_flags & XFS_LI_FAILED) { + lip->li_flags &= ~XFS_LI_FAILED; + lip->li_buf = NULL; + xfs_buf_rele(bp); + } +} + +static inline void +xfs_set_li_failed( + struct xfs_log_item *lip, + struct xfs_buf *bp) +{ + if (lip->li_flags & ~XFS_LI_FAILED) { + xfs_buf_hold(bp); + lip->li_flags |= XFS_LI_FAILED; + lip->li_buf = bp; + } +} + struct xfs_item_ops { void (*iop_size)(xfs_log_item_t *, int *, int *); void (*iop_format)(xfs_log_item_t *, struct xfs_log_vec *); diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c index 9056c0f..c41d640 100644 --- a/fs/xfs/xfs_trans_ail.c +++ b/fs/xfs/xfs_trans_ail.c @@ -687,15 +687,15 @@ xfs_trans_ail_update_bulk( bool xfs_ail_delete_one( struct xfs_ail *ailp, - struct xfs_log_item *lip) + struct xfs_log_item *lip) { struct xfs_log_item *mlip = xfs_ail_min(ailp); trace_xfs_ail_delete(lip, mlip->li_lsn, lip->li_lsn); xfs_ail_delete(ailp, lip); lip->li_flags &= ~XFS_LI_IN_AIL; + xfs_clear_li_failed(lip); lip->li_lsn = 0; - return mlip == lip; } -- 2.9.4 ^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 10:54 ` [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback Carlos Maiolino @ 2017-06-16 11:06 ` Carlos Maiolino 2017-06-16 18:35 ` Luis R. Rodriguez 2017-06-19 13:49 ` Brian Foster 2 siblings, 0 replies; 38+ messages in thread From: Carlos Maiolino @ 2017-06-16 11:06 UTC (permalink / raw) To: linux-xfs I forgot to add "V4" to the subject here, but the patch is still the V4 one -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 10:54 ` [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback Carlos Maiolino 2017-06-16 11:06 ` Carlos Maiolino @ 2017-06-16 18:35 ` Luis R. Rodriguez 2017-06-16 19:24 ` Darrick J. Wong 2017-06-20 7:01 ` Carlos Maiolino 2017-06-19 13:49 ` Brian Foster 2 siblings, 2 replies; 38+ messages in thread From: Luis R. Rodriguez @ 2017-06-16 18:35 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > When a buffer has been failed during writeback, the inode items into it > are kept flush locked, and are never resubmitted due the flush lock, so, > if any buffer fails to be written, the items in AIL are never written to > disk and never unlocked. > > This causes unmount operation to hang due these items flush locked in AIL, What type of hang? If it has occurred in production is there a trace somewhere? what does it look like? You said you would work on an xfstest for this, how's that going? Otherewise a commit log description of how to reproduce would be useful. > but this also causes the items in AIL to never be written back, even when > the IO device comes back to normal. > > I've been testing this patch with a DM-thin device, creating a > filesystem larger than the real device. > > When writing enough data to fill the DM-thin device, XFS receives ENOSPC > errors from the device, and keep spinning on xfsaild (when 'retry > forever' configuration is set). > > At this point, the filesystem can not be unmounted because of the flush locked > items in AIL, but worse, the items in AIL are never retried at all > (once xfs_inode_item_push() will skip the items that are flush locked), > even if the underlying DM-thin device is expanded to the proper size. Jeesh. If the above issue is a real hang, shoudl we not consider a sensible stable fix to start off with ? Luis ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 18:35 ` Luis R. Rodriguez @ 2017-06-16 19:24 ` Darrick J. Wong 2017-06-16 19:37 ` Luis R. Rodriguez 2017-06-20 7:01 ` Carlos Maiolino 1 sibling, 1 reply; 38+ messages in thread From: Darrick J. Wong @ 2017-06-16 19:24 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Carlos Maiolino, linux-xfs On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: > On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > > When a buffer has been failed during writeback, the inode items into it > > are kept flush locked, and are never resubmitted due the flush lock, so, > > if any buffer fails to be written, the items in AIL are never written to > > disk and never unlocked. > > > > This causes unmount operation to hang due these items flush locked in AIL, > > What type of hang? If it has occurred in production is there a trace somewhere? > what does it look like? > > You said you would work on an xfstest for this, how's that going? Otherewise > a commit log description of how to reproduce would be useful. I'm curious for an xfstest too, but I think Carlos /did/ tell us how to reproduce -- create a thinp device, format XFS, fill it up, and try to unmount. I don't think there /is/ much of a trace, just xfsaild doing nothing and a bunch of ail items that are flush locked and stuck that way. > > but this also causes the items in AIL to never be written back, even when > > the IO device comes back to normal. > > > > I've been testing this patch with a DM-thin device, creating a > > filesystem larger than the real device. > > > > When writing enough data to fill the DM-thin device, XFS receives ENOSPC > > errors from the device, and keep spinning on xfsaild (when 'retry > > forever' configuration is set). > > > > At this point, the filesystem can not be unmounted because of the flush locked > > items in AIL, but worse, the items in AIL are never retried at all > > (once xfs_inode_item_push() will skip the items that are flush locked), > > even if the underlying DM-thin device is expanded to the proper size. > > Jeesh. > > If the above issue is a real hang, shoudl we not consider a sensible stable fix > to start off with ? Huh? I thought this series is supposed to be the fix. --D > > Luis > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 19:24 ` Darrick J. Wong @ 2017-06-16 19:37 ` Luis R. Rodriguez 2017-06-16 19:45 ` Eric Sandeen 0 siblings, 1 reply; 38+ messages in thread From: Luis R. Rodriguez @ 2017-06-16 19:37 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Luis R. Rodriguez, Carlos Maiolino, linux-xfs On Fri, Jun 16, 2017 at 12:24:45PM -0700, Darrick J. Wong wrote: > On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: > > On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > > > When a buffer has been failed during writeback, the inode items into it > > > are kept flush locked, and are never resubmitted due the flush lock, so, > > > if any buffer fails to be written, the items in AIL are never written to > > > disk and never unlocked. > > > > > > This causes unmount operation to hang due these items flush locked in AIL, > > > > What type of hang? If it has occurred in production is there a trace somewhere? > > what does it look like? > > > > You said you would work on an xfstest for this, how's that going? Otherewise > > a commit log description of how to reproduce would be useful. > > I'm curious for an xfstest too, but I think Carlos /did/ tell us how to > reproduce -- create a thinp device, format XFS, fill it up, and try to > unmount. Well he did mention to create a Dm-thin device with a fileystem larger than the real device. You seem to have say just filling up a filsystem? Do both cases trigger the issue? > I don't think there /is/ much of a trace, just xfsaild doing > nothing and a bunch of ail items that are flush locked and stuck that way. OK so no hung task seek, no crash, just a system call that never ends? Is the issue recoverable? So unmount just never completes? Can we CTRL-C (SIGINT) out of it? > > > but this also causes the items in AIL to never be written back, even when > > > the IO device comes back to normal. > > > > > > I've been testing this patch with a DM-thin device, creating a > > > filesystem larger than the real device. > > > > > > When writing enough data to fill the DM-thin device, XFS receives ENOSPC > > > errors from the device, and keep spinning on xfsaild (when 'retry > > > forever' configuration is set). > > > > > > At this point, the filesystem can not be unmounted because of the flush locked > > > items in AIL, but worse, the items in AIL are never retried at all > > > (once xfs_inode_item_push() will skip the items that are flush locked), > > > even if the underlying DM-thin device is expanded to the proper size. > > > > Jeesh. > > > > If the above issue is a real hang, shoudl we not consider a sensible stable fix > > to start off with ? > > Huh? I thought this series is supposed to be the fix. It seems like a rather large set of changes, if the issue was sevee I was hoping for a stable candidate fix first. If its not fixing a severe issue then sure. Luis ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 19:37 ` Luis R. Rodriguez @ 2017-06-16 19:45 ` Eric Sandeen 2017-06-19 10:59 ` Brian Foster 0 siblings, 1 reply; 38+ messages in thread From: Eric Sandeen @ 2017-06-16 19:45 UTC (permalink / raw) To: Luis R. Rodriguez, Darrick J. Wong; +Cc: Carlos Maiolino, linux-xfs On 6/16/17 2:37 PM, Luis R. Rodriguez wrote: > On Fri, Jun 16, 2017 at 12:24:45PM -0700, Darrick J. Wong wrote: >> On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: >>> On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: >>>> When a buffer has been failed during writeback, the inode items into it >>>> are kept flush locked, and are never resubmitted due the flush lock, so, >>>> if any buffer fails to be written, the items in AIL are never written to >>>> disk and never unlocked. >>>> >>>> This causes unmount operation to hang due these items flush locked in AIL, >>> >>> What type of hang? If it has occurred in production is there a trace somewhere? >>> what does it look like? >>> >>> You said you would work on an xfstest for this, how's that going? Otherewise >>> a commit log description of how to reproduce would be useful. >> >> I'm curious for an xfstest too, but I think Carlos /did/ tell us how to >> reproduce -- create a thinp device, format XFS, fill it up, and try to >> unmount. > > Well he did mention to create a Dm-thin device with a fileystem larger than > the real device. You seem to have say just filling up a filsystem? No - the case is filling up the /thinp device/, not the filesystem. > > Do both cases trigger the issue? This whole issue has to do with errors from the block device during writeback; that's why the thin device is involved. When the backing store fills, we see the IO errors that cause this problem. >> I don't think there /is/ much of a trace, just xfsaild doing >> nothing and a bunch of ail items that are flush locked and stuck that way. > > OK so no hung task seek, no crash, just a system call that never ends? > > Is the issue recoverable? So unmount just never completes? Can we CTRL-C > (SIGINT) out of it? No, you can't ctrl-c a stuck kernel. >>>> but this also causes the items in AIL to never be written back, even when >>>> the IO device comes back to normal. >>>> >>>> I've been testing this patch with a DM-thin device, creating a >>>> filesystem larger than the real device. >>>> >>>> When writing enough data to fill the DM-thin device, XFS receives ENOSPC >>>> errors from the device, and keep spinning on xfsaild (when 'retry >>>> forever' configuration is set). >>>> >>>> At this point, the filesystem can not be unmounted because of the flush locked >>>> items in AIL, but worse, the items in AIL are never retried at all >>>> (once xfs_inode_item_push() will skip the items that are flush locked), >>>> even if the underlying DM-thin device is expanded to the proper size. >>> >>> Jeesh. >>> >>> If the above issue is a real hang, shoudl we not consider a sensible stable fix >>> to start off with ? >> >> Huh? I thought this series is supposed to be the fix. > > It seems like a rather large set of changes, if the issue was sevee I was hoping > for a stable candidate fix first. If its not fixing a severe issue then sure. Fixes go uptream first, then to stable kernels if appropriate, right? But not every fix is a trivial one-liner. I don't think there is any simpler fix to be had, here. -Eric > Luis > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 19:45 ` Eric Sandeen @ 2017-06-19 10:59 ` Brian Foster 2017-06-20 16:52 ` Luis R. Rodriguez 0 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-19 10:59 UTC (permalink / raw) To: Eric Sandeen Cc: Luis R. Rodriguez, Darrick J. Wong, Carlos Maiolino, linux-xfs On Fri, Jun 16, 2017 at 02:45:13PM -0500, Eric Sandeen wrote: > On 6/16/17 2:37 PM, Luis R. Rodriguez wrote: > > On Fri, Jun 16, 2017 at 12:24:45PM -0700, Darrick J. Wong wrote: > >> On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: > >>> On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > >>>> When a buffer has been failed during writeback, the inode items into it > >>>> are kept flush locked, and are never resubmitted due the flush lock, so, > >>>> if any buffer fails to be written, the items in AIL are never written to > >>>> disk and never unlocked. > >>>> > >>>> This causes unmount operation to hang due these items flush locked in AIL, > >>> > >>> What type of hang? If it has occurred in production is there a trace somewhere? > >>> what does it look like? > >>> > >>> You said you would work on an xfstest for this, how's that going? Otherewise > >>> a commit log description of how to reproduce would be useful. > >> > >> I'm curious for an xfstest too, but I think Carlos /did/ tell us how to > >> reproduce -- create a thinp device, format XFS, fill it up, and try to > >> unmount. > > > > Well he did mention to create a Dm-thin device with a fileystem larger than > > the real device. You seem to have say just filling up a filsystem? > > No - the case is filling up the /thinp device/, not the filesystem. > > > > > Do both cases trigger the issue? > > This whole issue has to do with errors from the block device during writeback; > that's why the thin device is involved. When the backing store fills, we > see the IO errors that cause this problem. > > >> I don't think there /is/ much of a trace, just xfsaild doing > >> nothing and a bunch of ail items that are flush locked and stuck that way. > > > > OK so no hung task seek, no crash, just a system call that never ends? > > > > Is the issue recoverable? So unmount just never completes? Can we CTRL-C > > (SIGINT) out of it? > > No, you can't ctrl-c a stuck kernel. > > >>>> but this also causes the items in AIL to never be written back, even when > >>>> the IO device comes back to normal. > >>>> > >>>> I've been testing this patch with a DM-thin device, creating a > >>>> filesystem larger than the real device. > >>>> > >>>> When writing enough data to fill the DM-thin device, XFS receives ENOSPC > >>>> errors from the device, and keep spinning on xfsaild (when 'retry > >>>> forever' configuration is set). > >>>> > >>>> At this point, the filesystem can not be unmounted because of the flush locked > >>>> items in AIL, but worse, the items in AIL are never retried at all > >>>> (once xfs_inode_item_push() will skip the items that are flush locked), > >>>> even if the underlying DM-thin device is expanded to the proper size. > >>> > >>> Jeesh. > >>> > >>> If the above issue is a real hang, shoudl we not consider a sensible stable fix > >>> to start off with ? > >> > >> Huh? I thought this series is supposed to be the fix. > > > > It seems like a rather large set of changes, if the issue was sevee I was hoping > > for a stable candidate fix first. If its not fixing a severe issue then sure. > > Fixes go uptream first, then to stable kernels if appropriate, right? > > But not every fix is a trivial one-liner. I don't think there is any simpler > fix to be had, here. > Yeah, this is kind of intended to be the simplest fix available as far as I'm aware. TBH, I don't think the fix here is fundamentally that complex (define a state for already flushed/failed log items), but rather the code that needs to be modified to implement it has various states and corner cases to manage that make it tricky to get right. If we truly needed a stable worthy fix in short order, that would probably be to revert ac8809f9a ("xfs: abort metadata writeback on permanent errors"), which caused this regression by making the AIL responsible for failed retries. A couple caveats I can think of with that are that 1.) this would likely short circuit the recently added error configuration api (which is kind of irrelevant if the underlying error management code doesn't work correctly, but suggests that should be reverted as well) and 2.) in doing so, this reverts back to the hardcoded infinite retry behavior in XFS. That means that transient errors may eventually recover, but the thinp enospc use case and whatnot are still going to hang on umount. It hasn't seemed necessary to me to take that approach given the lower prevalence of the issue and the fact that a solution had been worked out for a while now. Though I suppose the longer we go without a fix in place, the stronger the argument for something like that becomes. Brian > -Eric > > > Luis > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-19 10:59 ` Brian Foster @ 2017-06-20 16:52 ` Luis R. Rodriguez 2017-06-20 17:20 ` Brian Foster 0 siblings, 1 reply; 38+ messages in thread From: Luis R. Rodriguez @ 2017-06-20 16:52 UTC (permalink / raw) To: Brian Foster Cc: Eric Sandeen, Luis R. Rodriguez, Darrick J. Wong, Carlos Maiolino, linux-xfs On Mon, Jun 19, 2017 at 06:59:05AM -0400, Brian Foster wrote: > On Fri, Jun 16, 2017 at 02:45:13PM -0500, Eric Sandeen wrote: > > On 6/16/17 2:37 PM, Luis R. Rodriguez wrote: > > > On Fri, Jun 16, 2017 at 12:24:45PM -0700, Darrick J. Wong wrote: > > >> On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: > > > It seems like a rather large set of changes, if the issue was sevee I was hoping > > > for a stable candidate fix first. If its not fixing a severe issue then sure. > > > > Fixes go uptream first, then to stable kernels if appropriate, right? > > > > But not every fix is a trivial one-liner. I don't think there is any simpler > > fix to be had, here. > > > > Yeah, this is kind of intended to be the simplest fix available as far > as I'm aware. TBH, I don't think the fix here is fundamentally that > complex (define a state for already flushed/failed log items), but > rather the code that needs to be modified to implement it has various > states and corner cases to manage that make it tricky to get right. OK this helps, thanks. > If we truly needed a stable worthy fix in short order, that would > probably be to revert ac8809f9a ("xfs: abort metadata writeback on > permanent errors"), which caused this regression by making the AIL > responsible for failed retries. Should the following tag be added then to this commit proposed by Carlos: Fixes: ac8809f9a ("xfs: abort metadata writeback on permanent errors") FWIW this was merged as of v3.13. Even though that seems to have taken the failed buffer and kept it the commit log of that change seems to indicate we already ran into similar situations before, after this commit we seem to just retry IO once, but keep the buffer around on the AIL, to allow further modifications of the buffer. > A couple caveats I can think of with > that are that 1.) this would likely short circuit the recently added > error configuration api (which is kind of irrelevant if the underlying > error management code doesn't work correctly, but suggests that should > be reverted as well) and 2.) in doing so, this reverts back to the > hardcoded infinite retry behavior in XFS. That means that transient > errors may eventually recover, but the thinp enospc use case and whatnot > are still going to hang on umount. Right, and also one of the gains of the patch seems to have been to allow thinp devices to keep on chugging with modifications on the buffer, so that would be lost as well. That seems to be like an optimization though so its worth loosing IMHO if would have resolved the hang. Since that's not the case though... > It hasn't seemed necessary to me to take that approach given the lower > prevalence of the issue Of this issue? I suppose its why I asked about examples of issues, I seem to have found it likely much more possible out in the wild than expected. It would seem folks might be working around it somehow. > and the fact that a solution had been worked out > for a while now. Though I suppose the longer we go without a fix in > place, the stronger the argument for something like that becomes. Luis ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-20 16:52 ` Luis R. Rodriguez @ 2017-06-20 17:20 ` Brian Foster 2017-06-20 18:05 ` Luis R. Rodriguez 2017-06-20 18:38 ` Luis R. Rodriguez 0 siblings, 2 replies; 38+ messages in thread From: Brian Foster @ 2017-06-20 17:20 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Eric Sandeen, Darrick J. Wong, Carlos Maiolino, linux-xfs On Tue, Jun 20, 2017 at 06:52:04PM +0200, Luis R. Rodriguez wrote: > On Mon, Jun 19, 2017 at 06:59:05AM -0400, Brian Foster wrote: > > On Fri, Jun 16, 2017 at 02:45:13PM -0500, Eric Sandeen wrote: > > > On 6/16/17 2:37 PM, Luis R. Rodriguez wrote: > > > > On Fri, Jun 16, 2017 at 12:24:45PM -0700, Darrick J. Wong wrote: > > > >> On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: > > > > It seems like a rather large set of changes, if the issue was sevee I was hoping > > > > for a stable candidate fix first. If its not fixing a severe issue then sure. > > > > > > Fixes go uptream first, then to stable kernels if appropriate, right? > > > > > > But not every fix is a trivial one-liner. I don't think there is any simpler > > > fix to be had, here. > > > > > > > Yeah, this is kind of intended to be the simplest fix available as far > > as I'm aware. TBH, I don't think the fix here is fundamentally that > > complex (define a state for already flushed/failed log items), but > > rather the code that needs to be modified to implement it has various > > states and corner cases to manage that make it tricky to get right. > > OK this helps, thanks. > > > If we truly needed a stable worthy fix in short order, that would > > probably be to revert ac8809f9a ("xfs: abort metadata writeback on > > permanent errors"), which caused this regression by making the AIL > > responsible for failed retries. > > Should the following tag be added then to this commit proposed by Carlos: > > Fixes: ac8809f9a ("xfs: abort metadata writeback on permanent errors") > That seems reasonable to me. > FWIW this was merged as of v3.13. Even though that seems to have taken the > failed buffer and kept it the commit log of that change seems to indicate we > already ran into similar situations before, after this commit we seem to just > retry IO once, but keep the buffer around on the AIL, to allow further > modifications of the buffer. > Before that commit, I believe we would retry the metadata writeback indefinitely so long as it failed. If the underlying I/O failure ceased to occur, then this loop ends and the fs proceeds as normal. If not, then the filesystem is probably going to hang. After that commit, we retry once from I/O completion handling and otherwise rely on the AIL to issue the next (indefinite) retry. If the item that failed has a flush lock, we won't actually ever submit the I/O (which is the bug), however, and thus you're toast regardless of whether the I/O would ultimately succeed or not. > > > A couple caveats I can think of with > > that are that 1.) this would likely short circuit the recently added > > error configuration api (which is kind of irrelevant if the underlying > > error management code doesn't work correctly, but suggests that should > > be reverted as well) and 2.) in doing so, this reverts back to the > > hardcoded infinite retry behavior in XFS. That means that transient > > errors may eventually recover, but the thinp enospc use case and whatnot > > are still going to hang on umount. > > Right, and also one of the gains of the patch seems to have been to allow > thinp devices to keep on chugging with modifications on the buffer, so that > would be lost as well. That seems to be like an optimization though so its > worth loosing IMHO if would have resolved the hang. Since that's not the case > though... > I think that's just a secondary effect of unlocking the buffer. Probably not that important if I/Os are failing. > > It hasn't seemed necessary to me to take that approach given the lower > > prevalence of the issue > > Of this issue? I suppose its why I asked about examples of issues, I seem > to have found it likely much more possible out in the wild than expected. > It would seem folks might be working around it somehow. > If we're talking about the thin provisioning case, I suspect most people work around it by properly configuring their storage. ;) Brian > > and the fact that a solution had been worked out > > for a while now. Though I suppose the longer we go without a fix in > > place, the stronger the argument for something like that becomes. > > Luis > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-20 17:20 ` Brian Foster @ 2017-06-20 18:05 ` Luis R. Rodriguez 2017-06-21 10:10 ` Brian Foster 2017-06-20 18:38 ` Luis R. Rodriguez 1 sibling, 1 reply; 38+ messages in thread From: Luis R. Rodriguez @ 2017-06-20 18:05 UTC (permalink / raw) To: Brian Foster Cc: Luis R. Rodriguez, Eric Sandeen, Darrick J. Wong, Carlos Maiolino, linux-xfs On Tue, Jun 20, 2017 at 01:20:41PM -0400, Brian Foster wrote: > On Tue, Jun 20, 2017 at 06:52:04PM +0200, Luis R. Rodriguez wrote: > > On Mon, Jun 19, 2017 at 06:59:05AM -0400, Brian Foster wrote: > > > It hasn't seemed necessary to me to take that approach given the lower > > > prevalence of the issue > > > > Of this issue? I suppose its why I asked about examples of issues, I seem > > to have found it likely much more possible out in the wild than expected. > > It would seem folks might be working around it somehow. > > > > If we're talking about the thin provisioning case, I suspect most people > work around it by properly configuring their storage. ;) The fact that we *hang* makes it more serious, so even if folks misconfigured storage with less space it should be no reason to consider hangs any less severe. Specially if it seems to be a common issue, and I'm alluding to the fact that this might be more common than the patch describes. Luis ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-20 18:05 ` Luis R. Rodriguez @ 2017-06-21 10:10 ` Brian Foster 2017-06-21 15:25 ` Luis R. Rodriguez 0 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-21 10:10 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Eric Sandeen, Darrick J. Wong, Carlos Maiolino, linux-xfs On Tue, Jun 20, 2017 at 08:05:05PM +0200, Luis R. Rodriguez wrote: > On Tue, Jun 20, 2017 at 01:20:41PM -0400, Brian Foster wrote: > > On Tue, Jun 20, 2017 at 06:52:04PM +0200, Luis R. Rodriguez wrote: > > > On Mon, Jun 19, 2017 at 06:59:05AM -0400, Brian Foster wrote: > > > > It hasn't seemed necessary to me to take that approach given the lower > > > > prevalence of the issue > > > > > > Of this issue? I suppose its why I asked about examples of issues, I seem > > > to have found it likely much more possible out in the wild than expected. > > > It would seem folks might be working around it somehow. > > > > > > > If we're talking about the thin provisioning case, I suspect most people > > work around it by properly configuring their storage. ;) > > The fact that we *hang* makes it more serious, so even if folks misconfigured > storage with less space it should be no reason to consider hangs any less > severe. Specially if it seems to be a common issue, and I'm alluding to the > fact that this might be more common than the patch describes. > My point is simply that a hang was a likely outcome before the patch that introduced the regression as well, so the benefit of doing a proper revert doesn't clearly outweigh the cost. Despite what the side effect is, the fact that this tends to primarily affect users who have thin volumes going inactive also suggests that this can be worked around reasonably well enough via storage configuration. This all suggests to me that Carlos' current approach is the most reasonable one. I'm not following what the line of argument is here. Are you suggesting a different approach? If so, based on what use case/reasoning? Brian > Luis > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-21 10:10 ` Brian Foster @ 2017-06-21 15:25 ` Luis R. Rodriguez 0 siblings, 0 replies; 38+ messages in thread From: Luis R. Rodriguez @ 2017-06-21 15:25 UTC (permalink / raw) To: Brian Foster Cc: Luis R. Rodriguez, Eric Sandeen, Darrick J. Wong, Carlos Maiolino, linux-xfs On Wed, Jun 21, 2017 at 06:10:52AM -0400, Brian Foster wrote: > On Tue, Jun 20, 2017 at 08:05:05PM +0200, Luis R. Rodriguez wrote: > > On Tue, Jun 20, 2017 at 01:20:41PM -0400, Brian Foster wrote: > > > On Tue, Jun 20, 2017 at 06:52:04PM +0200, Luis R. Rodriguez wrote: > > > > On Mon, Jun 19, 2017 at 06:59:05AM -0400, Brian Foster wrote: > > > > > It hasn't seemed necessary to me to take that approach given the lower > > > > > prevalence of the issue > > > > > > > > Of this issue? I suppose its why I asked about examples of issues, I seem > > > > to have found it likely much more possible out in the wild than expected. > > > > It would seem folks might be working around it somehow. > > > > > > > > > > If we're talking about the thin provisioning case, I suspect most people > > > work around it by properly configuring their storage. ;) > > > > The fact that we *hang* makes it more serious, so even if folks misconfigured > > storage with less space it should be no reason to consider hangs any less > > severe. Specially if it seems to be a common issue, and I'm alluding to the > > fact that this might be more common than the patch describes. > > > > My point is simply that a hang was a likely outcome before the patch > that introduced the regression as well, so the benefit of doing a proper > revert doesn't clearly outweigh the cost. Sure agreed. > Despite what the side effect > is, the fact that this tends to primarily affect users who have thin > volumes going inactive also suggests that this can be worked around > reasonably well enough via storage configuration. This all suggests to > me that Carlos' current approach is the most reasonable one. OK thanks. > I'm not following what the line of argument is here. Are you suggesting > a different approach? If so, based on what use case/reasoning? No, it just seemed to me you were indicating that the hang was not that serious of an issue given you could work around it with proper storage configuration. I see now you were using that analogy just to indicate it was also an issue before so the revert is with merit. Luis ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-20 17:20 ` Brian Foster 2017-06-20 18:05 ` Luis R. Rodriguez @ 2017-06-20 18:38 ` Luis R. Rodriguez 1 sibling, 0 replies; 38+ messages in thread From: Luis R. Rodriguez @ 2017-06-20 18:38 UTC (permalink / raw) To: Brian Foster Cc: Luis R. Rodriguez, Eric Sandeen, Darrick J. Wong, Carlos Maiolino, linux-xfs On Tue, Jun 20, 2017 at 01:20:41PM -0400, Brian Foster wrote: > On Tue, Jun 20, 2017 at 06:52:04PM +0200, Luis R. Rodriguez wrote: > > On Mon, Jun 19, 2017 at 06:59:05AM -0400, Brian Foster wrote: > > > If we truly needed a stable worthy fix in short order, that would > > > probably be to revert ac8809f9a ("xfs: abort metadata writeback on > > > permanent errors"), which caused this regression by making the AIL > > > responsible for failed retries. > > > > Should the following tag be added then to this commit proposed by Carlos: > > > > Fixes: ac8809f9a ("xfs: abort metadata writeback on permanent errors") > > > > That seems reasonable to me. Then in this case I'd like the commit log to also explain *why* the described fix did not work. It actually describes the issue as being considered, "thin provisioned devices that have run out of backing space", and this recovering. So did recovery never really work? Does recovery actually work for some cases but not some? If so why not for some? Luis ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 18:35 ` Luis R. Rodriguez 2017-06-16 19:24 ` Darrick J. Wong @ 2017-06-20 7:01 ` Carlos Maiolino 2017-06-20 16:24 ` Luis R. Rodriguez 1 sibling, 1 reply; 38+ messages in thread From: Carlos Maiolino @ 2017-06-20 7:01 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-xfs Hello Luis. On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: > On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > > When a buffer has been failed during writeback, the inode items into it > > are kept flush locked, and are never resubmitted due the flush lock, so, > > if any buffer fails to be written, the items in AIL are never written to > > disk and never unlocked. > > > > This causes unmount operation to hang due these items flush locked in AIL, > > What type of hang? If it has occurred in production is there a trace somewhere? > what does it look like? > No, there isn't any specific trace, the hang can be seen in several different places, when unmounting the filesystem, it will hang in xfs_ail_push_all_sync(), but this will be hit even if no unmount is attempted, with items stuck forever in ail. I think the easier way to track this is to look at the device stats in sysfs, and you will see a forever increase in push_ail statistics even with no work going on in the filesystem. > You said you would work on an xfstest for this, how's that going? Otherewise > a commit log description of how to reproduce would be useful. > The xfstests is not done yet, and I'm actually not focusing on it right now, I already have a reproducer, pointed on the beginning of the discussion from this problem and having this fixed by now is my priority, once the patches are in shape and accepted, I'll work on the xfstests. Not to mention that this problem is still possible to occur not only with inode items, but also with dquot items, which will also be fixed as soon as we reach a consensus of how to best fix this problem by now. Once the dquot items fix will use the same infra-structure as the inode items use in this patchset, and quite the same code, one of the reasons I segmented the buffer resubmission into a different function that can be used for both item types. > > but this also causes the items in AIL to never be written back, even when > > the IO device comes back to normal. > > > > I've been testing this patch with a DM-thin device, creating a > > filesystem larger than the real device. > > > > When writing enough data to fill the DM-thin device, XFS receives ENOSPC > > errors from the device, and keep spinning on xfsaild (when 'retry > > forever' configuration is set). > > > > At this point, the filesystem can not be unmounted because of the flush locked > > items in AIL, but worse, the items in AIL are never retried at all > > (once xfs_inode_item_push() will skip the items that are flush locked), > > even if the underlying DM-thin device is expanded to the proper size. > > Jeesh. > > If the above issue is a real hang, shoudl we not consider a sensible stable fix > to start off with ? > Please take a look at the whole history of this issue, this patchset is supposed to be the stable fix, that's why one of the reqs was to use xa_lock here, to change the log_item flags, instead of using atomic ops, making it easier to backport it to stable kernels, without messing around with atomic ops and field type changes and yes, this is a real hang problem, which we already received several reports on this along the time I'm working on it. Cheers > Luis > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-20 7:01 ` Carlos Maiolino @ 2017-06-20 16:24 ` Luis R. Rodriguez 2017-06-21 11:51 ` Carlos Maiolino 0 siblings, 1 reply; 38+ messages in thread From: Luis R. Rodriguez @ 2017-06-20 16:24 UTC (permalink / raw) To: Luis R. Rodriguez, linux-xfs; +Cc: Gionatan Danti On Tue, Jun 20, 2017 at 09:01:03AM +0200, Carlos Maiolino wrote: > Hello Luis. > > On Fri, Jun 16, 2017 at 08:35:10PM +0200, Luis R. Rodriguez wrote: > > On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > > > When a buffer has been failed during writeback, the inode items into it > > > are kept flush locked, and are never resubmitted due the flush lock, so, > > > if any buffer fails to be written, the items in AIL are never written to > > > disk and never unlocked. > > > > > > This causes unmount operation to hang due these items flush locked in AIL, > > > > What type of hang? If it has occurred in production is there a trace somewhere? > > what does it look like? > > > > No, there isn't any specific trace, the hang can be seen in several different > places, when unmounting the filesystem, it will hang in xfs_ail_push_all_sync(), > but this will be hit even if no unmount is attempted, with items stuck forever > in ail. Curious, since you can reproduce what happens if you do a hard reset on the system when this trigger, once it boots back up? I'd guess it covers but just want to be sure. > I think the easier way to track this is to look at the device stats in sysfs, > and you will see a forever increase in push_ail statistics even with no work > going on in the filesystem. But the above seems to note it can happen for *any* failed buffer on writeback? Has that been really so odd to happen, or was this also just because of a requirement of 'retry forever' option? Or did the 'retry forever' help increase the chances of reproducing? I suppose I was hoping for a real world case which might land us in the worst case. For instance the dm-thin case with a pool that will run out seems to be like a common case for some docker users and from the looks of it folks are as a side consequence seeing this as docker just hanging after trying to unmount [0] and reaching xfs_ail_push_all_sync(). Could this be an example of a real world issue? There was no mention of the requirement of the 'retry forever' option though in these cases. If this is an example alternative real world situation triggering then this might be more common than what the commit log seems to describe. [0] https://github.com/moby/moby/issues/20707 > > You said you would work on an xfstest for this, how's that going? Otherewise > > a commit log description of how to reproduce would be useful. > > > > The xfstests is not done yet, and I'm actually not focusing on it right now, I > already have a reproducer, pointed on the beginning of the discussion from this > problem and having this fixed by now is my priority, once the patches are in > shape and accepted, I'll work on the xfstests. OK thanks. > Not to mention that this problem is still possible to occur not only with > inode items, but also with dquot items, which will also be fixed as soon as we > reach a consensus of how to best fix this problem by now. Once the dquot items > fix will use the same infra-structure as the inode items use in this patchset, > and quite the same code, one of the reasons I segmented the buffer resubmission > into a different function that can be used for both item types. I see... thanks! > > > but this also causes the items in AIL to never be written back, even when > > > the IO device comes back to normal. > > > > > > I've been testing this patch with a DM-thin device, creating a > > > filesystem larger than the real device. > > > > > > When writing enough data to fill the DM-thin device, XFS receives ENOSPC > > > errors from the device, and keep spinning on xfsaild (when 'retry > > > forever' configuration is set). > > > > > > At this point, the filesystem can not be unmounted because of the flush locked > > > items in AIL, but worse, the items in AIL are never retried at all > > > (once xfs_inode_item_push() will skip the items that are flush locked), > > > even if the underlying DM-thin device is expanded to the proper size. > > > > Jeesh. > > > > If the above issue is a real hang, shoudl we not consider a sensible stable fix > > to start off with ? > > > > Please take a look at the whole history of this issue, this patchset is supposed > to be the stable fix, that's why one of the reqs was to use xa_lock here, to > change the log_item flags, instead of using atomic ops, making it easier to > backport it to stable kernels, without messing around with atomic ops and field > type changes and yes, this is a real hang problem, which we already received > several reports on this along the time I'm working on it. I'm thrilled to hear stable is being considered here. Thanks! Luis ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-20 16:24 ` Luis R. Rodriguez @ 2017-06-21 11:51 ` Carlos Maiolino 0 siblings, 0 replies; 38+ messages in thread From: Carlos Maiolino @ 2017-06-21 11:51 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: linux-xfs, Gionatan Danti Hi Luis > > Curious, since you can reproduce what happens if you do a hard reset on the > system when this trigger, once it boots back up? I'd guess it covers but just > want to be sure. > Just for context, the problem with the stuck unmounts happens because the items in AIL can't be written back to their specific locations in the disk due lack of real space. But, instead of shutting down the FS when somebody tries to unmount, or permanently fail the buffer when trying to write it back (if XFS is configured to fail at some point), xfsaild keep spinning around on such buffers because the items are flush locked, and they are not even retried at all. giving this small resumed context, and now answering your question regarding a hard reset. When you hard reset the system in such state, after the system comes back alive, the filesystem in question will be unmountable, because the journal will be dirty, and XFS won't be able to replay it during the mount due lack of space in the physical device: # mount <volume> /mnt [ 91.843429] XFS (dm-5): Mounting V5 Filesystem [ 91.864321] device-mapper: thin: 253:2: reached low water mark for data device: sending event. [ 91.889451] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode [ 91.890821] XFS (dm-5): xfs_do_force_shutdown(0x1) called from line 1201 of file fs/xfs/xfs_buf.c. Return address = 0xffffffff813bb416 [ 91.893590] XFS (dm-5): I/O Error Detected. Shutting down filesystem [ 91.894813] XFS (dm-5): Please umount the filesystem and rectify the problem(s) [ 91.896158] XFS (dm-5): metadata I/O error: block 0x31f80 ("xlog_bwrite") error 28 numblks 4096 [ 91.902234] XFS (dm-5): failed to locate log tail [ 91.902920] XFS (dm-5): log mount/recovery failed: error -28 [ 91.903712] XFS (dm-5): log mount failed mount: mount <volume> on /mnt failed: No space left on device Although, simply my expanding the thin pool, everything comes back to normal again: #lvextend -L +500M <POOL> # mount <volume> /mnt [ 248.935258] XFS (dm-5): Mounting V5 Filesystem [ 248.954288] XFS (dm-5): Starting recovery (logdev: internal) [ 248.985238] XFS (dm-5): Ending recovery (logdev: internal) -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-16 10:54 ` [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback Carlos Maiolino 2017-06-16 11:06 ` Carlos Maiolino 2017-06-16 18:35 ` Luis R. Rodriguez @ 2017-06-19 13:49 ` Brian Foster 2017-06-19 15:09 ` Brian Foster 2 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-19 13:49 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > When a buffer has been failed during writeback, the inode items into it > are kept flush locked, and are never resubmitted due the flush lock, so, > if any buffer fails to be written, the items in AIL are never written to > disk and never unlocked. > > This causes unmount operation to hang due these items flush locked in AIL, > but this also causes the items in AIL to never be written back, even when > the IO device comes back to normal. > > I've been testing this patch with a DM-thin device, creating a > filesystem larger than the real device. > > When writing enough data to fill the DM-thin device, XFS receives ENOSPC > errors from the device, and keep spinning on xfsaild (when 'retry > forever' configuration is set). > > At this point, the filesystem can not be unmounted because of the flush locked > items in AIL, but worse, the items in AIL are never retried at all > (once xfs_inode_item_push() will skip the items that are flush locked), > even if the underlying DM-thin device is expanded to the proper size. > > This patch fixes both cases, retrying any item that has been failed > previously, using the infra-structure provided by the previous patch. > > Signed-off-by: Carlos Maiolino <cmaiolino@redhat.com> > --- > V2: > - Fix XFS_LI_FAILED flag removal > - Use atomic operations to set and clear XFS_LI_FAILED flag > - Remove check for XBF_WRITE_FAIL in xfs_inode_item_push > - Add more comments to the code > - Add a helper function to resubmit the failed buffers, so this > can be also used in dquot system without duplicating code > > V3: > - kill xfs_imap_to_bp call using a pointer in the log item to > hold the buffer address > - use xa_lock instead of atomic operations to handle log item > flags > - Add a hold to the buffer for each log item failed > - move buffer resubmission up in xfs_inode_item_push() > > V4: > - Remove bflags argument from iop_error callback > - Remove ip argument from xfs_buf_resubmit_failed_buffers > - Use helpers to set/clear XFS_LI_FAILED flag > - remove ->xa_lock from the iop->error callback and move it up > on the stack, so all log items are processed into a single > pair of lock/unlock > > fs/xfs/xfs_buf_item.c | 37 +++++++++++++++++++++++++++++++++++++ > fs/xfs/xfs_buf_item.h | 1 + > fs/xfs/xfs_inode_item.c | 36 +++++++++++++++++++++++++++++++++--- > fs/xfs/xfs_trans.h | 26 ++++++++++++++++++++++++++ > fs/xfs/xfs_trans_ail.c | 4 ++-- > 5 files changed, 99 insertions(+), 5 deletions(-) > > diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c > index 4fa68c9..c5d21ea 100644 > --- a/fs/xfs/xfs_buf_item.c > +++ b/fs/xfs/xfs_buf_item.c > @@ -1222,3 +1222,40 @@ xfs_buf_iodone( > xfs_trans_ail_delete(ailp, lip, SHUTDOWN_CORRUPT_INCORE); > xfs_buf_item_free(BUF_ITEM(lip)); > } > + > +/* > + * Requeue a failed buffer for writeback > + * > + * Return true if the buffer has been re-queued properly, false otherwise > + */ > +bool > +xfs_buf_resubmit_failed_buffers( > + struct xfs_log_item *lip, > + struct list_head *buffer_list) > +{ > + struct xfs_log_item *next; > + struct xfs_buf *bp; > + bool ret; > + > + bp = lip->li_buf; > + > + /* Clear XFS_LI_FAILED flag from all items before resubmit Nit: start of the comment (/*) should be on its own line. > + * > + * XFS_LI_FAILED set/clear is protected by xa_lock, caller this > + * function already have it acquired > + */ > + for (; lip; lip = next) { > + next = lip->li_bio_list; > + xfs_clear_li_failed(lip); > + } > + > + /* Add this buffer back to the delayed write list */ > + xfs_buf_lock(bp); This should be a trylock and we should return XFS_ITEM_LOCKED from the caller if it fails so xfsaild can make progress rather than block sitting on a buffer lock (and this is all much cleaner with the trylock/unlock in the caller). > + if (!xfs_buf_delwri_queue(bp, buffer_list)) > + ret = false; > + else > + ret = true; > + > + xfs_buf_unlock(bp); > + return ret; > +} > diff --git a/fs/xfs/xfs_buf_item.h b/fs/xfs/xfs_buf_item.h > index f7eba99..82c3d64 100644 > --- a/fs/xfs/xfs_buf_item.h > +++ b/fs/xfs/xfs_buf_item.h > @@ -70,6 +70,7 @@ void xfs_buf_attach_iodone(struct xfs_buf *, > xfs_log_item_t *); > void xfs_buf_iodone_callbacks(struct xfs_buf *); > void xfs_buf_iodone(struct xfs_buf *, struct xfs_log_item *); > +bool xfs_buf_resubmit_failed_buffers(struct xfs_log_item *, struct list_head *); > > extern kmem_zone_t *xfs_buf_item_zone; > > diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c > index 08cb7d1..2719ac6 100644 > --- a/fs/xfs/xfs_inode_item.c > +++ b/fs/xfs/xfs_inode_item.c > @@ -27,6 +27,7 @@ > #include "xfs_error.h" > #include "xfs_trace.h" > #include "xfs_trans_priv.h" > +#include "xfs_buf_item.h" > #include "xfs_log.h" > > > @@ -475,6 +476,18 @@ xfs_inode_item_unpin( > wake_up_bit(&ip->i_flags, __XFS_IPINNED_BIT); > } > > +/* > + * Callback used to mark a buffer with XFS_LI_FAILED when items in the buffer > + * have been failed during writeback > + */ > +STATIC void > +xfs_inode_item_error( > + struct xfs_log_item *lip, > + struct xfs_buf *bp) > +{ > + xfs_set_li_failed(lip, bp); > +} > + > STATIC uint > xfs_inode_item_push( > struct xfs_log_item *lip, > @@ -491,6 +504,17 @@ xfs_inode_item_push( > if (xfs_ipincount(ip) > 0) > return XFS_ITEM_PINNED; > > + /* > + * The buffer containing this item failed to be written back > + * previously. Resubmit the buffer for IO. > + */ > + if (lip->li_flags & XFS_LI_FAILED) { > + if (!xfs_buf_resubmit_failed_buffers(lip, buffer_list)) > + rval = XFS_ITEM_FLUSHING; > + > + return rval; > + } > + > if (!xfs_ilock_nowait(ip, XFS_ILOCK_SHARED)) > return XFS_ITEM_LOCKED; > > @@ -622,7 +646,8 @@ static const struct xfs_item_ops xfs_inode_item_ops = { > .iop_unlock = xfs_inode_item_unlock, > .iop_committed = xfs_inode_item_committed, > .iop_push = xfs_inode_item_push, > - .iop_committing = xfs_inode_item_committing > + .iop_committing = xfs_inode_item_committing, > + .iop_error = xfs_inode_item_error > }; > > > @@ -710,7 +735,8 @@ xfs_iflush_done( > * the AIL lock. > */ > iip = INODE_ITEM(blip); > - if (iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) > + if ((iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) || > + lip->li_flags & XFS_LI_FAILED) > need_ail++; > > blip = next; > @@ -718,7 +744,8 @@ xfs_iflush_done( > > /* make sure we capture the state of the initial inode. */ > iip = INODE_ITEM(lip); > - if (iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) > + if ((iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) || > + lip->li_flags & XFS_LI_FAILED) > need_ail++; > > /* > @@ -739,6 +766,9 @@ xfs_iflush_done( > if (INODE_ITEM(blip)->ili_logged && > blip->li_lsn == INODE_ITEM(blip)->ili_flush_lsn) > mlip_changed |= xfs_ail_delete_one(ailp, blip); > + else { > + xfs_clear_li_failed(blip); > + } > } > > if (mlip_changed) { > diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h > index 50df5367..2d7cf91 100644 > --- a/fs/xfs/xfs_trans.h > +++ b/fs/xfs/xfs_trans.h > @@ -49,6 +49,7 @@ typedef struct xfs_log_item { > struct xfs_ail *li_ailp; /* ptr to AIL */ > uint li_type; /* item type */ > uint li_flags; /* misc flags */ > + struct xfs_buf *li_buf; /* real buffer pointer */ > struct xfs_log_item *li_bio_list; /* buffer item list */ > void (*li_cb)(struct xfs_buf *, > struct xfs_log_item *); > @@ -72,6 +73,31 @@ typedef struct xfs_log_item { > { XFS_LI_ABORTED, "ABORTED" }, \ > { XFS_LI_FAILED, "FAILED" } > > +static inline void > +xfs_clear_li_failed( > + struct xfs_log_item *lip) > +{ > + struct xfs_buf *bp = lip->li_buf; > + I think we should assert that ->xa_lock is held in both of these helpers. Note that we have to use lockdep_assert_held() for spinlocks. It also couldn't hurt to assert that XFS_LI_IN_AIL is set as well (we'd just have to reorder the _clear_li_failed() call in xfs_ail_delete_one() below). > + if (lip->li_flags & XFS_LI_FAILED) { > + lip->li_flags &= ~XFS_LI_FAILED; > + lip->li_buf = NULL; > + xfs_buf_rele(bp); > + } > +} > + > +static inline void > +xfs_set_li_failed( > + struct xfs_log_item *lip, > + struct xfs_buf *bp) > +{ > + if (lip->li_flags & ~XFS_LI_FAILED) { I think you want !(lip->li_flags & XFS_LI_FAILED). ;) Brian > + xfs_buf_hold(bp); > + lip->li_flags |= XFS_LI_FAILED; > + lip->li_buf = bp; > + } > +} > + > struct xfs_item_ops { > void (*iop_size)(xfs_log_item_t *, int *, int *); > void (*iop_format)(xfs_log_item_t *, struct xfs_log_vec *); > diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c > index 9056c0f..c41d640 100644 > --- a/fs/xfs/xfs_trans_ail.c > +++ b/fs/xfs/xfs_trans_ail.c > @@ -687,15 +687,15 @@ xfs_trans_ail_update_bulk( > bool > xfs_ail_delete_one( > struct xfs_ail *ailp, > - struct xfs_log_item *lip) > + struct xfs_log_item *lip) > { > struct xfs_log_item *mlip = xfs_ail_min(ailp); > > trace_xfs_ail_delete(lip, mlip->li_lsn, lip->li_lsn); > xfs_ail_delete(ailp, lip); > lip->li_flags &= ~XFS_LI_IN_AIL; > + xfs_clear_li_failed(lip); > lip->li_lsn = 0; > - > return mlip == lip; > } > > -- > 2.9.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback 2017-06-19 13:49 ` Brian Foster @ 2017-06-19 15:09 ` Brian Foster 0 siblings, 0 replies; 38+ messages in thread From: Brian Foster @ 2017-06-19 15:09 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs On Mon, Jun 19, 2017 at 09:49:42AM -0400, Brian Foster wrote: > On Fri, Jun 16, 2017 at 12:54:45PM +0200, Carlos Maiolino wrote: > > When a buffer has been failed during writeback, the inode items into it > > are kept flush locked, and are never resubmitted due the flush lock, so, > > if any buffer fails to be written, the items in AIL are never written to > > disk and never unlocked. > > > > This causes unmount operation to hang due these items flush locked in AIL, > > but this also causes the items in AIL to never be written back, even when > > the IO device comes back to normal. > > > > I've been testing this patch with a DM-thin device, creating a > > filesystem larger than the real device. > > > > When writing enough data to fill the DM-thin device, XFS receives ENOSPC > > errors from the device, and keep spinning on xfsaild (when 'retry > > forever' configuration is set). > > > > At this point, the filesystem can not be unmounted because of the flush locked > > items in AIL, but worse, the items in AIL are never retried at all > > (once xfs_inode_item_push() will skip the items that are flush locked), > > even if the underlying DM-thin device is expanded to the proper size. > > > > This patch fixes both cases, retrying any item that has been failed > > previously, using the infra-structure provided by the previous patch. > > > > Signed-off-by: Carlos Maiolino <cmaiolino@redhat.com> > > --- ... > > diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c > > index 08cb7d1..2719ac6 100644 > > --- a/fs/xfs/xfs_inode_item.c > > +++ b/fs/xfs/xfs_inode_item.c ... > > @@ -475,6 +476,18 @@ xfs_inode_item_unpin( > > wake_up_bit(&ip->i_flags, __XFS_IPINNED_BIT); > > } > > > > +/* > > + * Callback used to mark a buffer with XFS_LI_FAILED when items in the buffer > > + * have been failed during writeback > > + */ > > +STATIC void > > +xfs_inode_item_error( > > + struct xfs_log_item *lip, > > + struct xfs_buf *bp) > > +{ Also if we're going to keep the ->iop_error() thing around, could we add an 'ASSERT(xfs_isiflocked(INODE_ITEM(lip)->ili_inode))' here? Brian > > + xfs_set_li_failed(lip, bp); > > +} > > + > > STATIC uint > > xfs_inode_item_push( > > struct xfs_log_item *lip, > > @@ -491,6 +504,17 @@ xfs_inode_item_push( > > if (xfs_ipincount(ip) > 0) > > return XFS_ITEM_PINNED; > > > > + /* > > + * The buffer containing this item failed to be written back > > + * previously. Resubmit the buffer for IO. > > + */ > > + if (lip->li_flags & XFS_LI_FAILED) { > > + if (!xfs_buf_resubmit_failed_buffers(lip, buffer_list)) > > + rval = XFS_ITEM_FLUSHING; > > + > > + return rval; > > + } > > + > > if (!xfs_ilock_nowait(ip, XFS_ILOCK_SHARED)) > > return XFS_ITEM_LOCKED; > > > > @@ -622,7 +646,8 @@ static const struct xfs_item_ops xfs_inode_item_ops = { > > .iop_unlock = xfs_inode_item_unlock, > > .iop_committed = xfs_inode_item_committed, > > .iop_push = xfs_inode_item_push, > > - .iop_committing = xfs_inode_item_committing > > + .iop_committing = xfs_inode_item_committing, > > + .iop_error = xfs_inode_item_error > > }; > > > > > > @@ -710,7 +735,8 @@ xfs_iflush_done( > > * the AIL lock. > > */ > > iip = INODE_ITEM(blip); > > - if (iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) > > + if ((iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) || > > + lip->li_flags & XFS_LI_FAILED) > > need_ail++; > > > > blip = next; > > @@ -718,7 +744,8 @@ xfs_iflush_done( > > > > /* make sure we capture the state of the initial inode. */ > > iip = INODE_ITEM(lip); > > - if (iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) > > + if ((iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) || > > + lip->li_flags & XFS_LI_FAILED) > > need_ail++; > > > > /* > > @@ -739,6 +766,9 @@ xfs_iflush_done( > > if (INODE_ITEM(blip)->ili_logged && > > blip->li_lsn == INODE_ITEM(blip)->ili_flush_lsn) > > mlip_changed |= xfs_ail_delete_one(ailp, blip); > > + else { > > + xfs_clear_li_failed(blip); > > + } > > } > > > > if (mlip_changed) { > > diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h > > index 50df5367..2d7cf91 100644 > > --- a/fs/xfs/xfs_trans.h > > +++ b/fs/xfs/xfs_trans.h > > @@ -49,6 +49,7 @@ typedef struct xfs_log_item { > > struct xfs_ail *li_ailp; /* ptr to AIL */ > > uint li_type; /* item type */ > > uint li_flags; /* misc flags */ > > + struct xfs_buf *li_buf; /* real buffer pointer */ > > struct xfs_log_item *li_bio_list; /* buffer item list */ > > void (*li_cb)(struct xfs_buf *, > > struct xfs_log_item *); > > @@ -72,6 +73,31 @@ typedef struct xfs_log_item { > > { XFS_LI_ABORTED, "ABORTED" }, \ > > { XFS_LI_FAILED, "FAILED" } > > > > +static inline void > > +xfs_clear_li_failed( > > + struct xfs_log_item *lip) > > +{ > > + struct xfs_buf *bp = lip->li_buf; > > + > > I think we should assert that ->xa_lock is held in both of these > helpers. Note that we have to use lockdep_assert_held() for spinlocks. > > It also couldn't hurt to assert that XFS_LI_IN_AIL is set as well (we'd > just have to reorder the _clear_li_failed() call in xfs_ail_delete_one() > below). > > > + if (lip->li_flags & XFS_LI_FAILED) { > > + lip->li_flags &= ~XFS_LI_FAILED; > > + lip->li_buf = NULL; > > + xfs_buf_rele(bp); > > + } > > +} > > + > > +static inline void > > +xfs_set_li_failed( > > + struct xfs_log_item *lip, > > + struct xfs_buf *bp) > > +{ > > + if (lip->li_flags & ~XFS_LI_FAILED) { > > I think you want !(lip->li_flags & XFS_LI_FAILED). ;) > > Brian > > > + xfs_buf_hold(bp); > > + lip->li_flags |= XFS_LI_FAILED; > > + lip->li_buf = bp; > > + } > > +} > > + > > struct xfs_item_ops { > > void (*iop_size)(xfs_log_item_t *, int *, int *); > > void (*iop_format)(xfs_log_item_t *, struct xfs_log_vec *); > > diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c > > index 9056c0f..c41d640 100644 > > --- a/fs/xfs/xfs_trans_ail.c > > +++ b/fs/xfs/xfs_trans_ail.c > > @@ -687,15 +687,15 @@ xfs_trans_ail_update_bulk( > > bool > > xfs_ail_delete_one( > > struct xfs_ail *ailp, > > - struct xfs_log_item *lip) > > + struct xfs_log_item *lip) > > { > > struct xfs_log_item *mlip = xfs_ail_min(ailp); > > > > trace_xfs_ail_delete(lip, mlip->li_lsn, lip->li_lsn); > > xfs_ail_delete(ailp, lip); > > lip->li_flags &= ~XFS_LI_IN_AIL; > > + xfs_clear_li_failed(lip); > > lip->li_lsn = 0; > > - > > return mlip == lip; > > } > > > > -- > > 2.9.4 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-16 10:54 [PATCH 0/2 V4] Resubmit items failed during writeback Carlos Maiolino 2017-06-16 10:54 ` [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure Carlos Maiolino 2017-06-16 10:54 ` [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback Carlos Maiolino @ 2017-06-19 13:51 ` Brian Foster 2017-06-19 17:42 ` Darrick J. Wong 2 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-19 13:51 UTC (permalink / raw) To: Carlos Maiolino; +Cc: linux-xfs On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > Hi, > > there goes a new version of this patchset based on previous reviews on V3. > > Changelogs added separated to each patch. > Hi Carlos, I pointed out the last things that I'm aware of that I think need to be fixed in this series (along with a few nits here and there). That said, it's already been pointed out that we probably want an xfstests test case to go along with this before it would be merged. Is that something you are still planning on? I'd actually like to take this a bit farther than a regression test and see if we can improve our ability to test the error handling code in general. What do you (or anybody else..) think about including a patch in this series that introduces the ability to inject metadata writeback errors on DEBUG kernels? For example, consider something that just sets b_error at the top of xfs_buf_iodone_callbacks() based on a random value and configurable error frequency. This could use XFS_TEST_ERROR() or something like a new DEBUG sysfs attribute in the error configuration (see log_badcrc_factor for a similar example). This would facilitate longer running tests where iodone callback errors occur randomly and transiently and we can thus actually exercise the error handling and recovery code. I'd love to run some fsstress testing, for example, as I'm hoping that would help wring out any further issues that could be lurking (particularly with the tricky xfs_iflush_done() logic and whatnot). If implemented generally enough, that could also be reused for a more simple xfstests regression test for the original problem (e.g., mount fs, set error frequency = 100%, modify an inode, set error frequency = 0, umount), albeit it would require debug mode. Thoughts? Brian > > -- > 2.9.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-19 13:51 ` [PATCH 0/2 V4] Resubmit items failed during writeback Brian Foster @ 2017-06-19 17:42 ` Darrick J. Wong 2017-06-19 18:51 ` Brian Foster 0 siblings, 1 reply; 38+ messages in thread From: Darrick J. Wong @ 2017-06-19 17:42 UTC (permalink / raw) To: Brian Foster; +Cc: Carlos Maiolino, linux-xfs On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > Hi, > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > Changelogs added separated to each patch. > > > > Hi Carlos, > > I pointed out the last things that I'm aware of that I think need to be > fixed in this series (along with a few nits here and there). That said, > it's already been pointed out that we probably want an xfstests test > case to go along with this before it would be merged. Is that something > you are still planning on? > > I'd actually like to take this a bit farther than a regression test and > see if we can improve our ability to test the error handling code in > general. What do you (or anybody else..) think about including a patch > in this series that introduces the ability to inject metadata writeback > errors on DEBUG kernels? For example, consider something that just sets > b_error at the top of xfs_buf_iodone_callbacks() based on a random value > and configurable error frequency. This could use XFS_TEST_ERROR() or > something like a new DEBUG sysfs attribute in the error configuration > (see log_badcrc_factor for a similar example). Sounds reasonable. I wonder if it would be more useful to have individual knobs for each metadata object type so that you could have multiple xfstests, each of which runs the same software scenario but with different failures configured? Then we could test what happens when, say, only inode writes fail, or bmbt writes fail, etc. rather than one big hammer that's harder to control? For a moment I also wondered why not use the error injection mechanism that we already have, rather than inventing more sysfs knobs? But we have limited space in xfs_error_injection (29/32 bits used) so we probably should just switch everything to sysfs knobs. --D > This would facilitate longer running tests where iodone callback errors > occur randomly and transiently and we can thus actually exercise the > error handling and recovery code. I'd love to run some fsstress testing, > for example, as I'm hoping that would help wring out any further issues > that could be lurking (particularly with the tricky xfs_iflush_done() > logic and whatnot). If implemented generally enough, that could also be > reused for a more simple xfstests regression test for the original > problem (e.g., mount fs, set error frequency = 100%, modify an inode, > set error frequency = 0, umount), albeit it would require debug mode. > Thoughts? > > Brian > > > > > -- > > 2.9.4 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-19 17:42 ` Darrick J. Wong @ 2017-06-19 18:51 ` Brian Foster 2017-06-21 0:45 ` Darrick J. Wong 0 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-19 18:51 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Carlos Maiolino, linux-xfs On Mon, Jun 19, 2017 at 10:42:03AM -0700, Darrick J. Wong wrote: > On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > > Hi, > > > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > > > Changelogs added separated to each patch. > > > > > > > Hi Carlos, > > > > I pointed out the last things that I'm aware of that I think need to be > > fixed in this series (along with a few nits here and there). That said, > > it's already been pointed out that we probably want an xfstests test > > case to go along with this before it would be merged. Is that something > > you are still planning on? > > > > I'd actually like to take this a bit farther than a regression test and > > see if we can improve our ability to test the error handling code in > > general. What do you (or anybody else..) think about including a patch > > in this series that introduces the ability to inject metadata writeback > > errors on DEBUG kernels? For example, consider something that just sets > > b_error at the top of xfs_buf_iodone_callbacks() based on a random value > > and configurable error frequency. This could use XFS_TEST_ERROR() or > > something like a new DEBUG sysfs attribute in the error configuration > > (see log_badcrc_factor for a similar example). > > Sounds reasonable. > > I wonder if it would be more useful to have individual knobs for each > metadata object type so that you could have multiple xfstests, each of > which runs the same software scenario but with different failures > configured? Then we could test what happens when, say, only inode > writes fail, or bmbt writes fail, etc. rather than one big hammer that's > harder to control? > I suppose you could do some of that in the test just by making certain modifications in isolation (e.g., test an inode modification, test a dquot, buffer, etc..). It might be harder to do that from a test script at the granularity of things like bmbt buffers, though that may also be the case for error injection. Did you have something in mind to filter the buffer types.. the blf type perhaps? > For a moment I also wondered why not use the error injection mechanism > that we already have, rather than inventing more sysfs knobs? But we > have limited space in xfs_error_injection (29/32 bits used) so we > probably should just switch everything to sysfs knobs. > Yeah, I think we discussed this before as well. IIRC, I had already implemented whatever the sysfs knob was and and corresponding test, and it just wasn't really important enough to reimplement the whole thing. I didn't really have a preference as to how this would be implemented. On a quick look though, it looks like a downside to the existing injection mechanism is that we can't control the error frequency..? If that's the case, I like the idea of stepping back, perhaps audit the granularity of the broader error injection framework and define a new sysfs interface that also allows controlling things like the error frequency dynamically. (I would still like to get something targeted towards testing this series in the meantime.) Brian > --D > > > This would facilitate longer running tests where iodone callback errors > > occur randomly and transiently and we can thus actually exercise the > > error handling and recovery code. I'd love to run some fsstress testing, > > for example, as I'm hoping that would help wring out any further issues > > that could be lurking (particularly with the tricky xfs_iflush_done() > > logic and whatnot). If implemented generally enough, that could also be > > reused for a more simple xfstests regression test for the original > > problem (e.g., mount fs, set error frequency = 100%, modify an inode, > > set error frequency = 0, umount), albeit it would require debug mode. > > Thoughts? > > > > Brian > > > > > > > > -- > > > 2.9.4 > > > > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-19 18:51 ` Brian Foster @ 2017-06-21 0:45 ` Darrick J. Wong 2017-06-21 10:15 ` Brian Foster 0 siblings, 1 reply; 38+ messages in thread From: Darrick J. Wong @ 2017-06-21 0:45 UTC (permalink / raw) To: Brian Foster; +Cc: Carlos Maiolino, linux-xfs On Mon, Jun 19, 2017 at 02:51:11PM -0400, Brian Foster wrote: > On Mon, Jun 19, 2017 at 10:42:03AM -0700, Darrick J. Wong wrote: > > On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > > > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > > > Hi, > > > > > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > > > > > Changelogs added separated to each patch. > > > > > > > > > > Hi Carlos, > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > fixed in this series (along with a few nits here and there). That said, > > > it's already been pointed out that we probably want an xfstests test > > > case to go along with this before it would be merged. Is that something > > > you are still planning on? > > > > > > I'd actually like to take this a bit farther than a regression test and > > > see if we can improve our ability to test the error handling code in > > > general. What do you (or anybody else..) think about including a patch > > > in this series that introduces the ability to inject metadata writeback > > > errors on DEBUG kernels? For example, consider something that just sets > > > b_error at the top of xfs_buf_iodone_callbacks() based on a random value > > > and configurable error frequency. This could use XFS_TEST_ERROR() or > > > something like a new DEBUG sysfs attribute in the error configuration > > > (see log_badcrc_factor for a similar example). > > > > Sounds reasonable. > > > > I wonder if it would be more useful to have individual knobs for each > > metadata object type so that you could have multiple xfstests, each of > > which runs the same software scenario but with different failures > > configured? Then we could test what happens when, say, only inode > > writes fail, or bmbt writes fail, etc. rather than one big hammer that's > > harder to control? > > > > I suppose you could do some of that in the test just by making certain > modifications in isolation (e.g., test an inode modification, test a > dquot, buffer, etc..). It might be harder to do that from a test script > at the granularity of things like bmbt buffers, though that may also be > the case for error injection. Did you have something in mind to filter > the buffer types.. the blf type perhaps? > > > For a moment I also wondered why not use the error injection mechanism > > that we already have, rather than inventing more sysfs knobs? But we > > have limited space in xfs_error_injection (29/32 bits used) so we > > probably should just switch everything to sysfs knobs. > > > > Yeah, I think we discussed this before as well. IIRC, I had already > implemented whatever the sysfs knob was and and corresponding test, and > it just wasn't really important enough to reimplement the whole thing. We probably did, back when the other error injection knobs (drop_writes, log/log_badcrc_factor) got added. Unfortunately now they're all over sysfs, which makes for sort of a mess. > I didn't really have a preference as to how this would be implemented. > On a quick look though, it looks like a downside to the existing > injection mechanism is that we can't control the error frequency..? If > that's the case, I like the idea of stepping back, perhaps audit the > granularity of the broader error injection framework and define a new > sysfs interface that also allows controlling things like the error > frequency dynamically. Yes, I've heard you ask for this twice now. I agree that being able to control the frequency would be a useful feature to enable longer-running tests so that you could, say, have the refcount_finish_one error trigger once every 1,000,000 updates instead of immediately afterwards. So with that in mind I coded up a quick RFC to create sysfs knobs in /sys/fs/xfs/$device/errortag/$tagname ; the units are the same as the XFS_RANDOM_ values (i.e. inverted frequency). Now we're free of the limitation of only being able to inject 10 error types across all mounted fses, and we can individually disable injection too. > (I would still like to get something targeted towards testing this > series in the meantime.) Same here. --D > Brian > > > --D > > > > > This would facilitate longer running tests where iodone callback errors > > > occur randomly and transiently and we can thus actually exercise the > > > error handling and recovery code. I'd love to run some fsstress testing, > > > for example, as I'm hoping that would help wring out any further issues > > > that could be lurking (particularly with the tricky xfs_iflush_done() > > > logic and whatnot). If implemented generally enough, that could also be > > > reused for a more simple xfstests regression test for the original > > > problem (e.g., mount fs, set error frequency = 100%, modify an inode, > > > set error frequency = 0, umount), albeit it would require debug mode. > > > Thoughts? > > > > > > Brian > > > > > > > > > > > -- > > > > 2.9.4 > > > > > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-21 0:45 ` Darrick J. Wong @ 2017-06-21 10:15 ` Brian Foster 2017-06-21 11:03 ` Carlos Maiolino 0 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-21 10:15 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Carlos Maiolino, linux-xfs On Tue, Jun 20, 2017 at 05:45:50PM -0700, Darrick J. Wong wrote: > On Mon, Jun 19, 2017 at 02:51:11PM -0400, Brian Foster wrote: > > On Mon, Jun 19, 2017 at 10:42:03AM -0700, Darrick J. Wong wrote: > > > On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > > > > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > > > > Hi, > > > > > > > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > > > > > > > Changelogs added separated to each patch. > > > > > > > > > > > > > Hi Carlos, > > > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > > fixed in this series (along with a few nits here and there). That said, > > > > it's already been pointed out that we probably want an xfstests test > > > > case to go along with this before it would be merged. Is that something > > > > you are still planning on? > > > > > > > > I'd actually like to take this a bit farther than a regression test and > > > > see if we can improve our ability to test the error handling code in > > > > general. What do you (or anybody else..) think about including a patch > > > > in this series that introduces the ability to inject metadata writeback > > > > errors on DEBUG kernels? For example, consider something that just sets > > > > b_error at the top of xfs_buf_iodone_callbacks() based on a random value > > > > and configurable error frequency. This could use XFS_TEST_ERROR() or > > > > something like a new DEBUG sysfs attribute in the error configuration > > > > (see log_badcrc_factor for a similar example). > > > > > > Sounds reasonable. > > > > > > I wonder if it would be more useful to have individual knobs for each > > > metadata object type so that you could have multiple xfstests, each of > > > which runs the same software scenario but with different failures > > > configured? Then we could test what happens when, say, only inode > > > writes fail, or bmbt writes fail, etc. rather than one big hammer that's > > > harder to control? > > > > > > > I suppose you could do some of that in the test just by making certain > > modifications in isolation (e.g., test an inode modification, test a > > dquot, buffer, etc..). It might be harder to do that from a test script > > at the granularity of things like bmbt buffers, though that may also be > > the case for error injection. Did you have something in mind to filter > > the buffer types.. the blf type perhaps? > > > > > For a moment I also wondered why not use the error injection mechanism > > > that we already have, rather than inventing more sysfs knobs? But we > > > have limited space in xfs_error_injection (29/32 bits used) so we > > > probably should just switch everything to sysfs knobs. > > > > > > > Yeah, I think we discussed this before as well. IIRC, I had already > > implemented whatever the sysfs knob was and and corresponding test, and > > it just wasn't really important enough to reimplement the whole thing. > > We probably did, back when the other error injection knobs (drop_writes, > log/log_badcrc_factor) got added. Unfortunately now they're all over > sysfs, which makes for sort of a mess. > > > I didn't really have a preference as to how this would be implemented. > > On a quick look though, it looks like a downside to the existing > > injection mechanism is that we can't control the error frequency..? If > > that's the case, I like the idea of stepping back, perhaps audit the > > granularity of the broader error injection framework and define a new > > sysfs interface that also allows controlling things like the error > > frequency dynamically. > > Yes, I've heard you ask for this twice now. I agree that being able to > control the frequency would be a useful feature to enable longer-running > tests so that you could, say, have the refcount_finish_one error trigger > once every 1,000,000 updates instead of immediately afterwards. > Yep. It allows modeling for longevity tests as you note above, but I'm also thinking about the ability to turn a particular error on with 100% frequency in order to run fast, deterministic regression tests. > So with that in mind I coded up a quick RFC to create sysfs knobs in > /sys/fs/xfs/$device/errortag/$tagname ; the units are the same as the > XFS_RANDOM_ values (i.e. inverted frequency). Now we're free of the > limitation of only being able to inject 10 error types across all > mounted fses, and we can individually disable injection too. > Nice, that sounds very interesting.. thanks! Brian > > (I would still like to get something targeted towards testing this > > series in the meantime.) > > Same here. > > --D > > > Brian > > > > > --D > > > > > > > This would facilitate longer running tests where iodone callback errors > > > > occur randomly and transiently and we can thus actually exercise the > > > > error handling and recovery code. I'd love to run some fsstress testing, > > > > for example, as I'm hoping that would help wring out any further issues > > > > that could be lurking (particularly with the tricky xfs_iflush_done() > > > > logic and whatnot). If implemented generally enough, that could also be > > > > reused for a more simple xfstests regression test for the original > > > > problem (e.g., mount fs, set error frequency = 100%, modify an inode, > > > > set error frequency = 0, umount), albeit it would require debug mode. > > > > Thoughts? > > > > > > > > Brian > > > > > > > > > > > > > > -- > > > > > 2.9.4 > > > > > > > > > > -- > > > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > > > > the body of a message to majordomo@vger.kernel.org > > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-21 10:15 ` Brian Foster @ 2017-06-21 11:03 ` Carlos Maiolino 2017-06-21 11:51 ` Brian Foster 2017-06-21 16:45 ` Darrick J. Wong 0 siblings, 2 replies; 38+ messages in thread From: Carlos Maiolino @ 2017-06-21 11:03 UTC (permalink / raw) To: Brian Foster; +Cc: Darrick J. Wong, linux-xfs Hey fellows. On Wed, Jun 21, 2017 at 06:15:26AM -0400, Brian Foster wrote: > On Tue, Jun 20, 2017 at 05:45:50PM -0700, Darrick J. Wong wrote: > > On Mon, Jun 19, 2017 at 02:51:11PM -0400, Brian Foster wrote: > > > On Mon, Jun 19, 2017 at 10:42:03AM -0700, Darrick J. Wong wrote: > > > > On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > > > > > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > > > > > Hi, > > > > > > > > > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > > > > > > > > > Changelogs added separated to each patch. > > > > > > > > > > > > > > > > Hi Carlos, > > > > > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > > > fixed in this series (along with a few nits here and there). That said, > > > > > it's already been pointed out that we probably want an xfstests test > > > > > case to go along with this before it would be merged. Is that something > > > > > you are still planning on? > > > > > Well, I am sure planing a xfstests for this case, I just didn't stop to work on it yet, and well, I wasn't expecting to have the test done before merging this patchset, is this a requirement? If so, I'll work on that before finishing this series, otherwise I'll just finish the series and then move to the xfstests. . . . > > > > > something like a new DEBUG sysfs attribute in the error configuration > > > > > (see log_badcrc_factor for a similar example). > > > > > > > > I wonder if it would be more useful to have individual knobs for each > > > > metadata object type so that you could have multiple xfstests, each of > > > > which runs the same software scenario but with different failures > > > > > > > > > > I suppose you could do some of that in the test just by making certain . . . > > XFS_RANDOM_ values (i.e. inverted frequency). Now we're free of the > > limitation of only being able to inject 10 error types across all > > mounted fses, and we can individually disable injection too. > > > > Nice, that sounds very interesting.. thanks! > > Brian > Regarding the error injection knobs (and the bad quoting of previous replies above :), I like the idea, and I can surely work on such implementation, but, I honestly disagree with having the error injection patches in this same patchset. This will recreate a new discussion regarding the implementation, several new reviews, comments, etc and postpone this fix even more. I'd highly appreciate if we could do this in a different patchset, so we can have this fix merged sooner. What you guys think? also, let me know if I should send the xfstest before moving on with this patchset. Cheers -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-21 11:03 ` Carlos Maiolino @ 2017-06-21 11:51 ` Brian Foster 2017-06-21 16:54 ` Darrick J. Wong 2017-06-21 16:45 ` Darrick J. Wong 1 sibling, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-21 11:51 UTC (permalink / raw) To: Darrick J. Wong, linux-xfs On Wed, Jun 21, 2017 at 01:03:49PM +0200, Carlos Maiolino wrote: > Hey fellows. > > On Wed, Jun 21, 2017 at 06:15:26AM -0400, Brian Foster wrote: > > On Tue, Jun 20, 2017 at 05:45:50PM -0700, Darrick J. Wong wrote: > > > On Mon, Jun 19, 2017 at 02:51:11PM -0400, Brian Foster wrote: > > > > On Mon, Jun 19, 2017 at 10:42:03AM -0700, Darrick J. Wong wrote: > > > > > On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > > > > > > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > > > > > > Hi, > > > > > > > > > > > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > > > > > > > > > > > Changelogs added separated to each patch. > > > > > > > > > > > > > > > > > > > Hi Carlos, > > > > > > > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > > > > fixed in this series (along with a few nits here and there). That said, > > > > > > it's already been pointed out that we probably want an xfstests test > > > > > > case to go along with this before it would be merged. Is that something > > > > > > you are still planning on? > > > > > > > > Well, I am sure planing a xfstests for this case, I just didn't stop to work on > it yet, and well, I wasn't expecting to have the test done before merging this > patchset, is this a requirement? If so, I'll work on that before finishing this > series, otherwise I'll just finish the series and then move to the xfstests. > I think it's reasonable to expect to at least have an xfstests test posted and available for review before this is merged, but I'll defer to Darrick on that. I don't think you necessarily have to stop working on these patches to get the xfstests done (i.e., maybe start putting something together after the next version goes out for review?), but that's just my .02. ;) > . > . > . > > > > > > something like a new DEBUG sysfs attribute in the error configuration > > > > > > (see log_badcrc_factor for a similar example). > > > > > > > > > > I wonder if it would be more useful to have individual knobs for each > > > > > metadata object type so that you could have multiple xfstests, each of > > > > > which runs the same software scenario but with different failures > > > > > > > > > > > > > I suppose you could do some of that in the test just by making certain > . > . > . > > > > XFS_RANDOM_ values (i.e. inverted frequency). Now we're free of the > > > limitation of only being able to inject 10 error types across all > > > mounted fses, and we can individually disable injection too. > > > > > > > Nice, that sounds very interesting.. thanks! > > > > Brian > > > > Regarding the error injection knobs (and the bad quoting of previous replies > above :), I like the idea, and I can surely work on such implementation, but, I > honestly disagree with having the error injection patches in this same patchset. > This will recreate a new discussion regarding the implementation, several new > reviews, comments, etc and postpone this fix even more. I'd highly appreciate if > we could do this in a different patchset, so we can have this fix merged sooner. > > What you guys think? > I think we all agree that generic error injection (which Darrick has started playing with, but I haven't looked at yet) doesn't need to be bundled with this series (I hope we didn't scare you there ;). What I was asking for is a single patch that adds error injection in one spot with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: debug mode log record crc error injection") again because it is a simple example of a small DEBUG only hunk of code and boilerplate code to add a sysfs knob. That said, I don't even think such a patch needs to be merged. I'm happy with appending a quick and dirty RFC to the series so long as it works to facilitate testing. I was merely suggesting it could be merged because you could also use it to construct an xfstests test if you so desired. If not, we could leave it as an rfc and perhaps add something later on top of the bits Darrick has started to play with. The reason I'm asking for this is because testing is an important part of code review[1]. We have the ability to test for unrelated regressions already. Presumably, we will have a new regression test for this specific issue based on your current reproducer recipe as well. While I may think the code is ultimately correct, that's not quite enough for me to put a reviewed-by on this set because of the complexity of the area of code it's touching. I'd much prefer to be able to run something that truly stresses out this area of code. Brian [1] In reality, I'm going to do this myself if necessary. I'm just asking to save myself some time (and possibly encourage similar testing from others). ;) > also, let me know if I should send the xfstest before moving on with this > patchset. > > > Cheers > > -- > Carlos > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-21 11:51 ` Brian Foster @ 2017-06-21 16:54 ` Darrick J. Wong 2017-06-22 12:05 ` Carlos Maiolino 0 siblings, 1 reply; 38+ messages in thread From: Darrick J. Wong @ 2017-06-21 16:54 UTC (permalink / raw) To: Brian Foster; +Cc: linux-xfs [Heh, concurrent replies.] On Wed, Jun 21, 2017 at 07:51:32AM -0400, Brian Foster wrote: > On Wed, Jun 21, 2017 at 01:03:49PM +0200, Carlos Maiolino wrote: > > Hey fellows. > > > > On Wed, Jun 21, 2017 at 06:15:26AM -0400, Brian Foster wrote: > > > On Tue, Jun 20, 2017 at 05:45:50PM -0700, Darrick J. Wong wrote: > > > > On Mon, Jun 19, 2017 at 02:51:11PM -0400, Brian Foster wrote: > > > > > On Mon, Jun 19, 2017 at 10:42:03AM -0700, Darrick J. Wong wrote: > > > > > > On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > > > > > > > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > > > > > > > > > > > > > Changelogs added separated to each patch. > > > > > > > > > > > > > > > > > > > > > > Hi Carlos, > > > > > > > > > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > > > > > fixed in this series (along with a few nits here and there). That said, > > > > > > > it's already been pointed out that we probably want an xfstests test > > > > > > > case to go along with this before it would be merged. Is that something > > > > > > > you are still planning on? > > > > > > > > > > > Well, I am sure planing a xfstests for this case, I just didn't stop to work on > > it yet, and well, I wasn't expecting to have the test done before merging this > > patchset, is this a requirement? If so, I'll work on that before finishing this > > series, otherwise I'll just finish the series and then move to the xfstests. > > > > I think it's reasonable to expect to at least have an xfstests test > posted and available for review before this is merged, but I'll defer to > Darrick on that. I don't think you necessarily have to stop working on > these patches to get the xfstests done (i.e., maybe start putting > something together after the next version goes out for review?), but > that's just my .02. ;) Yes, it's fine to have testcases out for review, even if we haven't finalized all the debugging knobs necessary. > > . > > . > > . > > > > > > > something like a new DEBUG sysfs attribute in the error configuration > > > > > > > (see log_badcrc_factor for a similar example). > > > > > > > > > > > > I wonder if it would be more useful to have individual knobs for each > > > > > > metadata object type so that you could have multiple xfstests, each of > > > > > > which runs the same software scenario but with different failures > > > > > > > > > > > > > > > > I suppose you could do some of that in the test just by making certain > > . > > . > > . > > > > > > XFS_RANDOM_ values (i.e. inverted frequency). Now we're free of the > > > > limitation of only being able to inject 10 error types across all > > > > mounted fses, and we can individually disable injection too. > > > > > > > > > > Nice, that sounds very interesting.. thanks! > > > > > > Brian > > > > > > > Regarding the error injection knobs (and the bad quoting of previous replies > > above :), I like the idea, and I can surely work on such implementation, but, I > > honestly disagree with having the error injection patches in this same patchset. > > This will recreate a new discussion regarding the implementation, several new > > reviews, comments, etc and postpone this fix even more. I'd highly appreciate if > > we could do this in a different patchset, so we can have this fix merged sooner. > > > > What you guys think? > > > > I think we all agree that generic error injection (which Darrick has > started playing with, but I haven't looked at yet) doesn't need to be > bundled with this series (I hope we didn't scare you there ;). What I > was asking for is a single patch that adds error injection in one spot > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > debug mode log record crc error injection") again because it is a simple > example of a small DEBUG only hunk of code and boilerplate code to add a > sysfs knob. > > That said, I don't even think such a patch needs to be merged. I'm happy > with appending a quick and dirty RFC to the series so long as it works > to facilitate testing. I was merely suggesting it could be merged > because you could also use it to construct an xfstests test if you so > desired. If not, we could leave it as an rfc and perhaps add something > later on top of the bits Darrick has started to play with. > > The reason I'm asking for this is because testing is an important part > of code review[1]. We have the ability to test for unrelated regressions > already. Presumably, we will have a new regression test for this > specific issue based on your current reproducer recipe as well. While I > may think the code is ultimately correct, that's not quite enough for me > to put a reviewed-by on this set because of the complexity of the area > of code it's touching. I'd much prefer to be able to run something that > truly stresses out this area of code. Agree. I'd love to live in a world where most of my review process is examining the overall approach to make sure that it integrates well with the design and letting the automated tests look for bugs in the implementation details, though obviously we're not there. --D > Brian > > [1] In reality, I'm going to do this myself if necessary. I'm just > asking to save myself some time (and possibly encourage similar testing > from others). ;) > > > also, let me know if I should send the xfstest before moving on with this > > patchset. > > > > > > Cheers > > > > -- > > Carlos > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-21 16:54 ` Darrick J. Wong @ 2017-06-22 12:05 ` Carlos Maiolino 2017-06-22 12:40 ` Brian Foster 0 siblings, 1 reply; 38+ messages in thread From: Carlos Maiolino @ 2017-06-22 12:05 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Brian Foster, linux-xfs Thanks for the reply folks. > > > > > > > > Hi Carlos, > > > > > > > > > > > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > > > > > > fixed in this series (along with a few nits here and there). That said, > > > > > > > > it's already been pointed out that we probably want an xfstests test > > > > > > > > case to go along with this before it would be merged. Is that something > > > > > > > > you are still planning on? > > > > > > > > > > > > > > Well, I am sure planing a xfstests for this case, I just didn't stop to work on > > > it yet, and well, I wasn't expecting to have the test done before merging this > > > patchset, is this a requirement? If so, I'll work on that before finishing this > > > series, otherwise I'll just finish the series and then move to the xfstests. > > > > > > > I think it's reasonable to expect to at least have an xfstests test > > posted and available for review before this is merged, but I'll defer to > > Darrick on that. I don't think you necessarily have to stop working on > > these patches to get the xfstests done (i.e., maybe start putting > > something together after the next version goes out for review?), but > > that's just my .02. ;) > > Yes, it's fine to have testcases out for review, even if we haven't > finalized all the debugging knobs necessary. > I agree here, that it's good to have the test on xfstests, and I apologize for my desire to postpone it, it's not laziness (I promise :). I was just considering the amount of time it's taking to have this problem fixed while I've seen reports of people hitting this problem more and more often. But well, I already have a test case for that, I didn't integrate it into xfstests yet because my test isn't 100% automated yet, once I need to wait for xfsaild to start flushing such buffers, my current test cases asks the user (me :), to hit <enter> when xfsaild runs. Btw, any hint on how could I automate that? For now, I was considering to monitor the stats of the filesystem in question, specifically the "push_ail" stats, and move the test forward when the push_ail statistics starts to show the issue, but it doesn't sound as a decent approach to me. > > > > I think we all agree that generic error injection (which Darrick has > > started playing with, but I haven't looked at yet) doesn't need to be > > bundled with this series (I hope we didn't scare you there ;). What I > > was asking for is a single patch that adds error injection in one spot > > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > > debug mode log record crc error injection") again because it is a simple > > example of a small DEBUG only hunk of code and boilerplate code to add a > > sysfs knob. > > I'll keep this in mind, and try to work on something like that :) > > Agree. I'd love to live in a world where most of my review process is > examining the overall approach to make sure that it integrates well with > the design and letting the automated tests look for bugs in the > implementation details, though obviously we're not there. > > --D > > > Brian > > > > [1] In reality, I'm going to do this myself if necessary. I'm just > > asking to save myself some time (and possibly encourage similar testing > > from others). ;) -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-22 12:05 ` Carlos Maiolino @ 2017-06-22 12:40 ` Brian Foster 2017-06-30 11:09 ` Carlos Maiolino 0 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-22 12:40 UTC (permalink / raw) To: Darrick J. Wong, linux-xfs On Thu, Jun 22, 2017 at 02:05:28PM +0200, Carlos Maiolino wrote: > Thanks for the reply folks. > > > > > > > > > > Hi Carlos, > > > > > > > > > > > > > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > > > > > > > fixed in this series (along with a few nits here and there). That said, > > > > > > > > > it's already been pointed out that we probably want an xfstests test > > > > > > > > > case to go along with this before it would be merged. Is that something > > > > > > > > > you are still planning on? > > > > > > > > > > > > > > > > > Well, I am sure planing a xfstests for this case, I just didn't stop to work on > > > > it yet, and well, I wasn't expecting to have the test done before merging this > > > > patchset, is this a requirement? If so, I'll work on that before finishing this > > > > series, otherwise I'll just finish the series and then move to the xfstests. > > > > > > > > > > I think it's reasonable to expect to at least have an xfstests test > > > posted and available for review before this is merged, but I'll defer to > > > Darrick on that. I don't think you necessarily have to stop working on > > > these patches to get the xfstests done (i.e., maybe start putting > > > something together after the next version goes out for review?), but > > > that's just my .02. ;) > > > > Yes, it's fine to have testcases out for review, even if we haven't > > finalized all the debugging knobs necessary. > > > > I agree here, that it's good to have the test on xfstests, and I apologize for > my desire to postpone it, it's not laziness (I promise :). I was just considering > the amount of time it's taking to have this problem fixed while I've seen reports > of people hitting this problem more and more often. > But well, I already have a test case for that, I didn't integrate it into > xfstests yet because my test isn't 100% automated yet, once I need to wait for > xfsaild to start flushing such buffers, my current test cases asks the user (me :), > to hit <enter> when xfsaild runs. > > Btw, any hint on how could I automate that? For now, I was considering to > monitor the stats of the filesystem in question, specifically the "push_ail" > stats, and move the test forward when the push_ail statistics starts to show the > issue, but it doesn't sound as a decent approach to me. > > Taking a quick look at something I have laying around that you sent previously (I assume the test hasn't changed much), I see we basically create an overprovisioned dm-thin vol, mount it and dio write to it until we start to see async write failures. So is the purpose of the wait that we need the AIL to push and ultimately fail the associated buffers before we attempt an unmount? If so, I'm wondering if you could xfs_freeze the fs rather than wait (it looks like freeze sync pushes the AIL)..? > > > > > > I think we all agree that generic error injection (which Darrick has > > > started playing with, but I haven't looked at yet) doesn't need to be > > > bundled with this series (I hope we didn't scare you there ;). What I > > > was asking for is a single patch that adds error injection in one spot > > > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > > > debug mode log record crc error injection") again because it is a simple > > > example of a small DEBUG only hunk of code and boilerplate code to add a > > > sysfs knob. > > > > > I'll keep this in mind, and try to work on something like that :) > I think error injection would make this test more straightforward because you can explicitly control when I/Os fail and there's no need to play around with dm-thin. That of course doesn't mean there isn't value in having a test for fs' in dm-thin out of space conditions in general. :) Note that it's probably better to refer to Darrick's errortags series at this point rather than adding a custom knob like the bad crc patch does above. See how patch 4 in that series ports over the drop_writes mechanism, for example. It looks like it should only be an XFS_TEST_ERROR() injection point and a couple or so new defines to add the knob, default value, etc. Brian > > > > Agree. I'd love to live in a world where most of my review process is > > examining the overall approach to make sure that it integrates well with > > the design and letting the automated tests look for bugs in the > > implementation details, though obviously we're not there. > > > > --D > > > > > Brian > > > > > > [1] In reality, I'm going to do this myself if necessary. I'm just > > > asking to save myself some time (and possibly encourage similar testing > > > from others). ;) > > -- > Carlos > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-22 12:40 ` Brian Foster @ 2017-06-30 11:09 ` Carlos Maiolino 2017-06-30 11:33 ` Brian Foster 0 siblings, 1 reply; 38+ messages in thread From: Carlos Maiolino @ 2017-06-30 11:09 UTC (permalink / raw) To: Brian Foster; +Cc: Darrick J. Wong, linux-xfs Hey, > Taking a quick look at something I have laying around that you sent > previously (I assume the test hasn't changed much), I see we basically > create an overprovisioned dm-thin vol, mount it and dio write to it > until we start to see async write failures. So is the purpose of the > wait that we need the AIL to push and ultimately fail the associated > buffers before we attempt an unmount? If so, I'm wondering if you could > xfs_freeze the fs rather than wait (it looks like freeze sync pushes the > AIL)..? > As we discussed off-list, yes, I'd done some testing, and I believe we can use freezing to test it. > > > > > > > > I think we all agree that generic error injection (which Darrick has > > > > started playing with, but I haven't looked at yet) doesn't need to be > > > > bundled with this series (I hope we didn't scare you there ;). What I > > > > was asking for is a single patch that adds error injection in one spot > > > > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > > > > debug mode log record crc error injection") again because it is a simple > > > > example of a small DEBUG only hunk of code and boilerplate code to add a > > > > sysfs knob. > > > > > > > > I'll keep this in mind, and try to work on something like that :) > > > > I think error injection would make this test more straightforward > because you can explicitly control when I/Os fail and there's no need to > play around with dm-thin. That of course doesn't mean there isn't value > in having a test for fs' in dm-thin out of space conditions in general. > :) Well, this is doable, although it has a caveat. To do this, we'd need to inject the error in the buffer during IO completion, for example in xfs_buf_ioend(). The problem though, is that we start to have IOs before the 'mp->m_errotag' is actually initialized, so, xfs_buf_ioend() would need to check for m_errortag initialization before calling XFS_TEST_ERROR(). Something like this: bp->b_flags &= ~(XBF_READ | XBF_WRITE | XBF_READ_AHEAD); + if (bp->b_target->bt_mount->m_errortag) { + if (XFS_TEST_ERROR(false, bp->b_target->bt_mount, + XFS_ERRTAG_BUF_WB)) { + bp->b_io_error = -ENOSPC; + } + } This check could also be done in xfs_errortag_test(), although I'm not sure if it's worth, giving that there are very few places where error injection can be useful and m_errortag can be uninitialized. Thoughts? -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-30 11:09 ` Carlos Maiolino @ 2017-06-30 11:33 ` Brian Foster 2017-06-30 12:22 ` Carlos Maiolino 0 siblings, 1 reply; 38+ messages in thread From: Brian Foster @ 2017-06-30 11:33 UTC (permalink / raw) To: Darrick J. Wong, linux-xfs On Fri, Jun 30, 2017 at 01:09:13PM +0200, Carlos Maiolino wrote: > Hey, > > > Taking a quick look at something I have laying around that you sent > > previously (I assume the test hasn't changed much), I see we basically > > create an overprovisioned dm-thin vol, mount it and dio write to it > > until we start to see async write failures. So is the purpose of the > > wait that we need the AIL to push and ultimately fail the associated > > buffers before we attempt an unmount? If so, I'm wondering if you could > > xfs_freeze the fs rather than wait (it looks like freeze sync pushes the > > AIL)..? > > > > As we discussed off-list, yes, I'd done some testing, and I believe we can use > freezing to test it. > > > > > > > > > > > I think we all agree that generic error injection (which Darrick has > > > > > started playing with, but I haven't looked at yet) doesn't need to be > > > > > bundled with this series (I hope we didn't scare you there ;). What I > > > > > was asking for is a single patch that adds error injection in one spot > > > > > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > > > > > debug mode log record crc error injection") again because it is a simple > > > > > example of a small DEBUG only hunk of code and boilerplate code to add a > > > > > sysfs knob. > > > > > > > > > > > I'll keep this in mind, and try to work on something like that :) > > > > > > > I think error injection would make this test more straightforward > > because you can explicitly control when I/Os fail and there's no need to > > play around with dm-thin. That of course doesn't mean there isn't value > > in having a test for fs' in dm-thin out of space conditions in general. > > :) > > Well, this is doable, although it has a caveat. > > To do this, we'd need to inject the error in the buffer during IO completion, > for example in xfs_buf_ioend(). The problem though, is that we start to have IOs > before the 'mp->m_errotag' is actually initialized, so, xfs_buf_ioend() would > need to check for m_errortag initialization before calling XFS_TEST_ERROR(). > > Something like this: > > bp->b_flags &= ~(XBF_READ | XBF_WRITE | XBF_READ_AHEAD); > > + if (bp->b_target->bt_mount->m_errortag) { > + if (XFS_TEST_ERROR(false, bp->b_target->bt_mount, > + XFS_ERRTAG_BUF_WB)) { > + bp->b_io_error = -ENOSPC; > + } > + } > > This check could also be done in xfs_errortag_test(), although I'm not sure if > it's worth, giving that there are very few places where error injection can be > useful and m_errortag can be uninitialized. > > > Thoughts? > This strikes me as a bug in the error injection bits. Are you observing a crash due to the injection point above? We should be able to add injection points anywhere without such issues. IOW, I think your suggestion to check ->m_errortag in xfs_test_error() is probably the appropriate fix. Darrick may want to chime in.. but in the meantime I'd suggest to throw up a patch to fix that. ;) BTW, you may want to consider using somewhere like xfs_buf_iodone_callbacks() as an independent injection point from xfs_buf_ioend(). It seems the latter may potentially cause other I/O errors that interfere with testing AIL writeback error processing. I could be wrong though so it doesn't hurt to try (and we could certainly add two separate injection points). Just something to think about.. Brian > -- > Carlos > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-30 11:33 ` Brian Foster @ 2017-06-30 12:22 ` Carlos Maiolino 2017-06-30 17:01 ` Darrick J. Wong 0 siblings, 1 reply; 38+ messages in thread From: Carlos Maiolino @ 2017-06-30 12:22 UTC (permalink / raw) To: Brian Foster; +Cc: Darrick J. Wong, linux-xfs On Fri, Jun 30, 2017 at 07:33:42AM -0400, Brian Foster wrote: > On Fri, Jun 30, 2017 at 01:09:13PM +0200, Carlos Maiolino wrote: > > Hey, > > > > > Taking a quick look at something I have laying around that you sent > > > previously (I assume the test hasn't changed much), I see we basically > > > create an overprovisioned dm-thin vol, mount it and dio write to it > > > until we start to see async write failures. So is the purpose of the > > > wait that we need the AIL to push and ultimately fail the associated > > > buffers before we attempt an unmount? If so, I'm wondering if you could > > > xfs_freeze the fs rather than wait (it looks like freeze sync pushes the > > > AIL)..? > > > > > > > As we discussed off-list, yes, I'd done some testing, and I believe we can use > > freezing to test it. > > > > > > > > > > > > > > I think we all agree that generic error injection (which Darrick has > > > > > > started playing with, but I haven't looked at yet) doesn't need to be > > > > > > bundled with this series (I hope we didn't scare you there ;). What I > > > > > > was asking for is a single patch that adds error injection in one spot > > > > > > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > > > > > > debug mode log record crc error injection") again because it is a simple > > > > > > example of a small DEBUG only hunk of code and boilerplate code to add a > > > > > > sysfs knob. > > > > > > > > > > > > > > I'll keep this in mind, and try to work on something like that :) > > > > > > > > > > I think error injection would make this test more straightforward > > > because you can explicitly control when I/Os fail and there's no need to > > > play around with dm-thin. That of course doesn't mean there isn't value > > > in having a test for fs' in dm-thin out of space conditions in general. > > > :) > > > > Well, this is doable, although it has a caveat. > > > > To do this, we'd need to inject the error in the buffer during IO completion, > > for example in xfs_buf_ioend(). The problem though, is that we start to have IOs > > before the 'mp->m_errotag' is actually initialized, so, xfs_buf_ioend() would > > need to check for m_errortag initialization before calling XFS_TEST_ERROR(). > > > > Something like this: > > > > bp->b_flags &= ~(XBF_READ | XBF_WRITE | XBF_READ_AHEAD); > > > > + if (bp->b_target->bt_mount->m_errortag) { > > + if (XFS_TEST_ERROR(false, bp->b_target->bt_mount, > > + XFS_ERRTAG_BUF_WB)) { > > + bp->b_io_error = -ENOSPC; > > + } > > + } > > > > This check could also be done in xfs_errortag_test(), although I'm not sure if > > it's worth, giving that there are very few places where error injection can be > > useful and m_errortag can be uninitialized. > > > > > > Thoughts? > > > > This strikes me as a bug in the error injection bits. Are you observing > a crash due to the injection point above? We should be able to add > injection points anywhere without such issues. > Yup, when calling XFS_TEST_ERROR() from xfs_buf_ioend(), this will cause a NULL pointer dereference right when the filesystem is mounted. I didn't really got deeper into it to know from where exactly it comes from, but, I'd guess it comes from the very beginning, in xfs_fs_fill_super(), reading blocks from disk, will end up triggering xfs_buf_ioend(). > IOW, I think your suggestion to check ->m_errortag in xfs_test_error() > is probably the appropriate fix. Darrick may want to chime in.. but in > the meantime I'd suggest to throw up a patch to fix that. ;) > Sounds, reasonable, I'll write the patch and send in the next minutes. > BTW, you may want to consider using somewhere like > xfs_buf_iodone_callbacks() as an independent injection point from > xfs_buf_ioend(). It seems the latter may potentially cause other I/O > errors that interfere with testing AIL writeback error processing. I > could be wrong though so it doesn't hurt to try (and we could certainly > add two separate injection points). Just something to think about.. > Thanks, I'll think about it. Cheers -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-30 12:22 ` Carlos Maiolino @ 2017-06-30 17:01 ` Darrick J. Wong 2017-07-03 8:37 ` Carlos Maiolino 0 siblings, 1 reply; 38+ messages in thread From: Darrick J. Wong @ 2017-06-30 17:01 UTC (permalink / raw) To: Brian Foster, linux-xfs, cmaiolino On Fri, Jun 30, 2017 at 02:22:47PM +0200, Carlos Maiolino wrote: > On Fri, Jun 30, 2017 at 07:33:42AM -0400, Brian Foster wrote: > > On Fri, Jun 30, 2017 at 01:09:13PM +0200, Carlos Maiolino wrote: > > > Hey, > > > > > > > Taking a quick look at something I have laying around that you sent > > > > previously (I assume the test hasn't changed much), I see we basically > > > > create an overprovisioned dm-thin vol, mount it and dio write to it > > > > until we start to see async write failures. So is the purpose of the > > > > wait that we need the AIL to push and ultimately fail the associated > > > > buffers before we attempt an unmount? If so, I'm wondering if you could > > > > xfs_freeze the fs rather than wait (it looks like freeze sync pushes the > > > > AIL)..? > > > > > > > > > > As we discussed off-list, yes, I'd done some testing, and I believe we can use > > > freezing to test it. > > > > > > > > > > > > > > > > > I think we all agree that generic error injection (which Darrick has > > > > > > > started playing with, but I haven't looked at yet) doesn't need to be > > > > > > > bundled with this series (I hope we didn't scare you there ;). What I > > > > > > > was asking for is a single patch that adds error injection in one spot > > > > > > > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > > > > > > > debug mode log record crc error injection") again because it is a simple > > > > > > > example of a small DEBUG only hunk of code and boilerplate code to add a > > > > > > > sysfs knob. > > > > > > > > > > > > > > > > > I'll keep this in mind, and try to work on something like that :) > > > > > > > > > > > > > I think error injection would make this test more straightforward > > > > because you can explicitly control when I/Os fail and there's no need to > > > > play around with dm-thin. That of course doesn't mean there isn't value > > > > in having a test for fs' in dm-thin out of space conditions in general. > > > > :) I like the idea of having two tests -- one where we use error injection to stuff in an ENOSPC so that we can test how XFS reacts to ENOSPC, and a second one that runs a dm-thin out of space to gauge XFS' reaction to that condition. The first test can be thought of as a unit test for our internal nospace handling, whereas the second test is an integration test of nospace handling between dm-thin and XFS. Some day in the future dm-thin could grow a different means to signal ENOSPC to XFS, in which case we'll still need to test what happens when raw storage sends us ENOSPC. Furthermore, dm-thin might not be available on any given tester's machine; a dm-thin test could be generic whereas error injection is an xfs-specific test; etc. > > > > > > Well, this is doable, although it has a caveat. > > > > > > To do this, we'd need to inject the error in the buffer during IO completion, > > > for example in xfs_buf_ioend(). The problem though, is that we start to have IOs > > > before the 'mp->m_errotag' is actually initialized, so, xfs_buf_ioend() would > > > need to check for m_errortag initialization before calling XFS_TEST_ERROR(). > > > > > > Something like this: > > > > > > bp->b_flags &= ~(XBF_READ | XBF_WRITE | XBF_READ_AHEAD); > > > > > > + if (bp->b_target->bt_mount->m_errortag) { > > > + if (XFS_TEST_ERROR(false, bp->b_target->bt_mount, > > > + XFS_ERRTAG_BUF_WB)) { > > > + bp->b_io_error = -ENOSPC; > > > + } > > > + } > > > > > > This check could also be done in xfs_errortag_test(), although I'm not sure if > > > it's worth, giving that there are very few places where error injection can be > > > useful and m_errortag can be uninitialized. > > > > > > > > > Thoughts? > > > > > > > This strikes me as a bug in the error injection bits. Are you observing > > a crash due to the injection point above? We should be able to add > > injection points anywhere without such issues. > > > Yup, when calling XFS_TEST_ERROR() from xfs_buf_ioend(), this will cause a NULL > pointer dereference right when the filesystem is mounted. I didn't really got > deeper into it to know from where exactly it comes from, but, I'd guess it comes > from the very beginning, in xfs_fs_fill_super(), reading blocks from disk, will > end up triggering xfs_buf_ioend(). > > > IOW, I think your suggestion to check ->m_errortag in xfs_test_error() > > is probably the appropriate fix. Darrick may want to chime in.. but in > > the meantime I'd suggest to throw up a patch to fix that. ;) > > > > Sounds, reasonable, I'll write the patch and send in the next minutes. Yes, that was a bug, thanks for the patch. :) --D > > > BTW, you may want to consider using somewhere like > > xfs_buf_iodone_callbacks() as an independent injection point from > > xfs_buf_ioend(). It seems the latter may potentially cause other I/O > > errors that interfere with testing AIL writeback error processing. I > > could be wrong though so it doesn't hurt to try (and we could certainly > > add two separate injection points). Just something to think about.. > > > > Thanks, I'll think about it. > > Cheers > -- > Carlos > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-30 17:01 ` Darrick J. Wong @ 2017-07-03 8:37 ` Carlos Maiolino 0 siblings, 0 replies; 38+ messages in thread From: Carlos Maiolino @ 2017-07-03 8:37 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Brian Foster, linux-xfs On Fri, Jun 30, 2017 at 10:01:32AM -0700, Darrick J. Wong wrote: > On Fri, Jun 30, 2017 at 02:22:47PM +0200, Carlos Maiolino wrote: > > On Fri, Jun 30, 2017 at 07:33:42AM -0400, Brian Foster wrote: > > > On Fri, Jun 30, 2017 at 01:09:13PM +0200, Carlos Maiolino wrote: > > > > Hey, > > > > > > > > > Taking a quick look at something I have laying around that you sent > > > > > previously (I assume the test hasn't changed much), I see we basically > > > > > create an overprovisioned dm-thin vol, mount it and dio write to it > > > > > until we start to see async write failures. So is the purpose of the > > > > > wait that we need the AIL to push and ultimately fail the associated > > > > > buffers before we attempt an unmount? If so, I'm wondering if you could > > > > > xfs_freeze the fs rather than wait (it looks like freeze sync pushes the > > > > > AIL)..? > > > > > > > > > > > > > As we discussed off-list, yes, I'd done some testing, and I believe we can use > > > > freezing to test it. > > > > > > > > > > > > > > > > > > > > I think we all agree that generic error injection (which Darrick has > > > > > > > > started playing with, but I haven't looked at yet) doesn't need to be > > > > > > > > bundled with this series (I hope we didn't scare you there ;). What I > > > > > > > > was asking for is a single patch that adds error injection in one spot > > > > > > > > with a configurable frequency. I'll refer to commit 609adfc2ed ("xfs: > > > > > > > > debug mode log record crc error injection") again because it is a simple > > > > > > > > example of a small DEBUG only hunk of code and boilerplate code to add a > > > > > > > > sysfs knob. > > > > > > > > > > > > > > > > > > > > I'll keep this in mind, and try to work on something like that :) > > > > > > > > > > > > > > > > I think error injection would make this test more straightforward > > > > > because you can explicitly control when I/Os fail and there's no need to > > > > > play around with dm-thin. That of course doesn't mean there isn't value > > > > > in having a test for fs' in dm-thin out of space conditions in general. > > > > > :) > > I like the idea of having two tests -- one where we use error injection > to stuff in an ENOSPC so that we can test how XFS reacts to ENOSPC, and > a second one that runs a dm-thin out of space to gauge XFS' reaction to > that condition. The first test can be thought of as a unit test for our > internal nospace handling, whereas the second test is an integration > test of nospace handling between dm-thin and XFS. Some day in the > future dm-thin could grow a different means to signal ENOSPC to XFS, I don't doubt :) >in > which case we'll still need to test what happens when raw storage sends > us ENOSPC. Furthermore, dm-thin might not be available on any given > tester's machine; a dm-thin test could be generic whereas error makes sense, I can work on both tests, I'll prioritize the dm-thin though, which well, is the one giving me headaches at this specific time :) cheers > injection is an xfs-specific test; etc. > > > > > > > > > Well, this is doable, although it has a caveat. > > > > > > > > To do this, we'd need to inject the error in the buffer during IO completion, > > > > for example in xfs_buf_ioend(). The problem though, is that we start to have IOs > > > > before the 'mp->m_errotag' is actually initialized, so, xfs_buf_ioend() would > > > > need to check for m_errortag initialization before calling XFS_TEST_ERROR(). > > > > > > > > Something like this: > > > > > > > > bp->b_flags &= ~(XBF_READ | XBF_WRITE | XBF_READ_AHEAD); > > > > > > > > + if (bp->b_target->bt_mount->m_errortag) { > > > > + if (XFS_TEST_ERROR(false, bp->b_target->bt_mount, > > > > + XFS_ERRTAG_BUF_WB)) { > > > > + bp->b_io_error = -ENOSPC; > > > > + } > > > > + } > > > > > > > > This check could also be done in xfs_errortag_test(), although I'm not sure if > > > > it's worth, giving that there are very few places where error injection can be > > > > useful and m_errortag can be uninitialized. > > > > > > > > > > > > Thoughts? > > > > > > > > > > This strikes me as a bug in the error injection bits. Are you observing > > > a crash due to the injection point above? We should be able to add > > > injection points anywhere without such issues. > > > > > Yup, when calling XFS_TEST_ERROR() from xfs_buf_ioend(), this will cause a NULL > > pointer dereference right when the filesystem is mounted. I didn't really got > > deeper into it to know from where exactly it comes from, but, I'd guess it comes > > from the very beginning, in xfs_fs_fill_super(), reading blocks from disk, will > > end up triggering xfs_buf_ioend(). > > > > > IOW, I think your suggestion to check ->m_errortag in xfs_test_error() > > > is probably the appropriate fix. Darrick may want to chime in.. but in > > > the meantime I'd suggest to throw up a patch to fix that. ;) > > > > > > > Sounds, reasonable, I'll write the patch and send in the next minutes. > > Yes, that was a bug, thanks for the patch. :) > > --D > > > > > > BTW, you may want to consider using somewhere like > > > xfs_buf_iodone_callbacks() as an independent injection point from > > > xfs_buf_ioend(). It seems the latter may potentially cause other I/O > > > errors that interfere with testing AIL writeback error processing. I > > > could be wrong though so it doesn't hurt to try (and we could certainly > > > add two separate injection points). Just something to think about.. > > > > > > > Thanks, I'll think about it. > > > > Cheers > > -- > > Carlos > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Carlos ^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 0/2 V4] Resubmit items failed during writeback 2017-06-21 11:03 ` Carlos Maiolino 2017-06-21 11:51 ` Brian Foster @ 2017-06-21 16:45 ` Darrick J. Wong 1 sibling, 0 replies; 38+ messages in thread From: Darrick J. Wong @ 2017-06-21 16:45 UTC (permalink / raw) To: Brian Foster, linux-xfs On Wed, Jun 21, 2017 at 01:03:49PM +0200, Carlos Maiolino wrote: > Hey fellows. > > On Wed, Jun 21, 2017 at 06:15:26AM -0400, Brian Foster wrote: > > On Tue, Jun 20, 2017 at 05:45:50PM -0700, Darrick J. Wong wrote: > > > On Mon, Jun 19, 2017 at 02:51:11PM -0400, Brian Foster wrote: > > > > On Mon, Jun 19, 2017 at 10:42:03AM -0700, Darrick J. Wong wrote: > > > > > On Mon, Jun 19, 2017 at 09:51:22AM -0400, Brian Foster wrote: > > > > > > On Fri, Jun 16, 2017 at 12:54:43PM +0200, Carlos Maiolino wrote: > > > > > > > Hi, > > > > > > > > > > > > > > there goes a new version of this patchset based on previous reviews on V3. > > > > > > > > > > > > > > Changelogs added separated to each patch. > > > > > > > > > > > > > > > > > > > Hi Carlos, > > > > > > > > > > > > I pointed out the last things that I'm aware of that I think need to be > > > > > > fixed in this series (along with a few nits here and there). That said, > > > > > > it's already been pointed out that we probably want an xfstests test > > > > > > case to go along with this before it would be merged. Is that something > > > > > > you are still planning on? > > > > > > > > Well, I am sure planing a xfstests for this case, I just didn't stop > to work on it yet, and well, I wasn't expecting to have the test done > before merging this patchset, is this a requirement? Yes, in general I'd like to see testcases being submitted at least as early as the patch. The obvious exceptions to that are obvious one-liner fixes and failures that we've discussed on-list about being difficult to reproduce. If the tests are already upstream then remind me of the number so I can test the patches; if not, I want to be able to apply your tests to my xfstests repo and try it out. To be absolutely clear: the only thing I don't want to see is no test cases at all. > If so, I'll work on that before finishing this series, otherwise I'll > just finish the series and then move to the xfstests. You can do that, but just be aware that my inclination to defer a patch series increases with the complexity of the code changes and (sharply) decreases with the amount of testing that's also supplied with it. :) I understand that the statement is subjective and arbitrary; I seek a balance point between no test coverage at all and the "every LoC must be tested!" mandates that years ago had me writing test cases for binary and logical OR. My end goal is to avoid another copy_file_range/btrfs_ioc_clone* mess where user-visible interfaces go into the kernel with zero validation, and then there's no way to figure out what's supposed to be going on or if the code actually implements that. In the first case there were continual promises about delivery of testcases, but they languished for months; in the second case, I wrote the tests as part of ensuring that XFS reflink behaved in a fashion that didn't break existing users' mental models of reflink... which was difficult to suss out since there were two different reflink models in the kernel and no explicit definition of either. Yuck. > . > . > . > > > > > > something like a new DEBUG sysfs attribute in the error configuration > > > > > > (see log_badcrc_factor for a similar example). > > > > > > > > > > I wonder if it would be more useful to have individual knobs for each > > > > > metadata object type so that you could have multiple xfstests, each of > > > > > which runs the same software scenario but with different failures > > > > > > > > > > > > > I suppose you could do some of that in the test just by making certain > . > . > . > > > > XFS_RANDOM_ values (i.e. inverted frequency). Now we're free of the > > > limitation of only being able to inject 10 error types across all > > > mounted fses, and we can individually disable injection too. > > > > > > > Nice, that sounds very interesting.. thanks! > > > > Brian > > > > Regarding the error injection knobs (and the bad quoting of previous > replies above :), I like the idea, and I can surely work on such > implementation, but, I honestly disagree with having the error > injection patches in this same patchset. This will recreate a new > discussion regarding the implementation, several new reviews, > comments, etc and postpone this fix even more. I'd highly appreciate > if we could do this in a different patchset, so we can have this fix > merged sooner. <shrug> Tack a new patch on the end of the series that implements the injection knob via the existing mechanism (whether that's the stuff in xfs_error.[ch] or (I guess) the sysfs log/ debug knobs); that shouldn't be too controversial. If later we agree to adopt the new scheme, I'll update your code/tests to use the new knobs. > What you guys think? > > also, let me know if I should send the xfstest before moving on with > this patchset. Yes please. --D > > > Cheers > > -- > Carlos > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 38+ messages in thread
end of thread, other threads:[~2017-07-03 8:37 UTC | newest] Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-06-16 10:54 [PATCH 0/2 V4] Resubmit items failed during writeback Carlos Maiolino 2017-06-16 10:54 ` [PATCH 1/2 V4] xfs: Add infrastructure needed for error propagation during buffer IO failure Carlos Maiolino 2017-06-19 13:48 ` Brian Foster 2017-06-20 7:15 ` Carlos Maiolino 2017-06-16 10:54 ` [PATCH 2/2] xfs: Properly retry failed inode items in case of error during buffer writeback Carlos Maiolino 2017-06-16 11:06 ` Carlos Maiolino 2017-06-16 18:35 ` Luis R. Rodriguez 2017-06-16 19:24 ` Darrick J. Wong 2017-06-16 19:37 ` Luis R. Rodriguez 2017-06-16 19:45 ` Eric Sandeen 2017-06-19 10:59 ` Brian Foster 2017-06-20 16:52 ` Luis R. Rodriguez 2017-06-20 17:20 ` Brian Foster 2017-06-20 18:05 ` Luis R. Rodriguez 2017-06-21 10:10 ` Brian Foster 2017-06-21 15:25 ` Luis R. Rodriguez 2017-06-20 18:38 ` Luis R. Rodriguez 2017-06-20 7:01 ` Carlos Maiolino 2017-06-20 16:24 ` Luis R. Rodriguez 2017-06-21 11:51 ` Carlos Maiolino 2017-06-19 13:49 ` Brian Foster 2017-06-19 15:09 ` Brian Foster 2017-06-19 13:51 ` [PATCH 0/2 V4] Resubmit items failed during writeback Brian Foster 2017-06-19 17:42 ` Darrick J. Wong 2017-06-19 18:51 ` Brian Foster 2017-06-21 0:45 ` Darrick J. Wong 2017-06-21 10:15 ` Brian Foster 2017-06-21 11:03 ` Carlos Maiolino 2017-06-21 11:51 ` Brian Foster 2017-06-21 16:54 ` Darrick J. Wong 2017-06-22 12:05 ` Carlos Maiolino 2017-06-22 12:40 ` Brian Foster 2017-06-30 11:09 ` Carlos Maiolino 2017-06-30 11:33 ` Brian Foster 2017-06-30 12:22 ` Carlos Maiolino 2017-06-30 17:01 ` Darrick J. Wong 2017-07-03 8:37 ` Carlos Maiolino 2017-06-21 16:45 ` Darrick J. Wong
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.