* [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace
@ 2021-04-14 12:08 fdmanana
2021-04-14 12:34 ` Johannes Thumshirn
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: fdmanana @ 2021-04-14 12:08 UTC (permalink / raw)
To: linux-btrfs
From: Filipe Manana <fdmanana@suse.com>
When doing a device replace on a zoned filesystem, if we find a block
group with ->to_copy == 0, we jump to the label 'done', which will result
in later calling btrfs_unfreeze_block_group(), even though at this point
we never called btrfs_freeze_block_group().
Since at this point we have neither turned the block group to RO mode nor
made any progress, we don't need to jump to the label 'done'. So fix this
by jumping instead to the label 'skip' and dropping our reference on the
block group before the jump.
Fixes: 78ce9fc269af6e ("btrfs: zoned: mark block groups to copy for device-replace")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
fs/btrfs/scrub.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index 17e49caad1f9..e0d54ed9acee 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -3674,8 +3674,8 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
spin_lock(&cache->lock);
if (!cache->to_copy) {
spin_unlock(&cache->lock);
- ro_set = 0;
- goto done;
+ btrfs_put_block_group(cache);
+ goto skip;
}
spin_unlock(&cache->lock);
}
--
2.28.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace 2021-04-14 12:08 [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace fdmanana @ 2021-04-14 12:34 ` Johannes Thumshirn 2021-04-14 13:06 ` Filipe Manana 2021-04-14 13:05 ` [PATCH v2] " fdmanana 2021-04-14 16:32 ` kernel test robot 2 siblings, 1 reply; 7+ messages in thread From: Johannes Thumshirn @ 2021-04-14 12:34 UTC (permalink / raw) To: fdmanana, linux-btrfs On 14/04/2021 14:09, fdmanana@kernel.org wrote: > From: Filipe Manana <fdmanana@suse.com> > > When doing a device replace on a zoned filesystem, if we find a block > group with ->to_copy == 0, we jump to the label 'done', which will result > in later calling btrfs_unfreeze_block_group(), even though at this point > we never called btrfs_freeze_block_group(). > > Since at this point we have neither turned the block group to RO mode nor > made any progress, we don't need to jump to the label 'done'. So fix this > by jumping instead to the label 'skip' and dropping our reference on the > block group before the jump. > > Fixes: 78ce9fc269af6e ("btrfs: zoned: mark block groups to copy for device-replace") > Signed-off-by: Filipe Manana <fdmanana@suse.com> > --- > fs/btrfs/scrub.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c > index 17e49caad1f9..e0d54ed9acee 100644 > --- a/fs/btrfs/scrub.c > +++ b/fs/btrfs/scrub.c > @@ -3674,8 +3674,8 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx, > spin_lock(&cache->lock); > if (!cache->to_copy) { > spin_unlock(&cache->lock); > - ro_set = 0; > - goto done; > + btrfs_put_block_group(cache); > + goto skip; > } > spin_unlock(&cache->lock); > } > I think we can remove the 'done' label as well, as after this patch it is unused (at least on my clone of misc-next). Otherwise, Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace 2021-04-14 12:34 ` Johannes Thumshirn @ 2021-04-14 13:06 ` Filipe Manana 0 siblings, 0 replies; 7+ messages in thread From: Filipe Manana @ 2021-04-14 13:06 UTC (permalink / raw) To: Johannes Thumshirn; +Cc: linux-btrfs On Wed, Apr 14, 2021 at 1:35 PM Johannes Thumshirn <Johannes.Thumshirn@wdc.com> wrote: > > On 14/04/2021 14:09, fdmanana@kernel.org wrote: > > From: Filipe Manana <fdmanana@suse.com> > > > > When doing a device replace on a zoned filesystem, if we find a block > > group with ->to_copy == 0, we jump to the label 'done', which will result > > in later calling btrfs_unfreeze_block_group(), even though at this point > > we never called btrfs_freeze_block_group(). > > > > Since at this point we have neither turned the block group to RO mode nor > > made any progress, we don't need to jump to the label 'done'. So fix this > > by jumping instead to the label 'skip' and dropping our reference on the > > block group before the jump. > > > > Fixes: 78ce9fc269af6e ("btrfs: zoned: mark block groups to copy for device-replace") > > Signed-off-by: Filipe Manana <fdmanana@suse.com> > > --- > > fs/btrfs/scrub.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c > > index 17e49caad1f9..e0d54ed9acee 100644 > > --- a/fs/btrfs/scrub.c > > +++ b/fs/btrfs/scrub.c > > @@ -3674,8 +3674,8 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx, > > spin_lock(&cache->lock); > > if (!cache->to_copy) { > > spin_unlock(&cache->lock); > > - ro_set = 0; > > - goto done; > > + btrfs_put_block_group(cache); > > + goto skip; > > } > > spin_unlock(&cache->lock); > > } > > > > I think we can remove the 'done' label as well, as after this patch it is > unused (at least on my clone of misc-next). Indeed, I missed that. Removed in v2, thanks. > > Otherwise, > Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2] btrfs: zoned: fix unpaired block group unfreeze during device replace 2021-04-14 12:08 [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace fdmanana 2021-04-14 12:34 ` Johannes Thumshirn @ 2021-04-14 13:05 ` fdmanana 2021-04-20 17:34 ` David Sterba 2021-04-14 16:32 ` kernel test robot 2 siblings, 1 reply; 7+ messages in thread From: fdmanana @ 2021-04-14 13:05 UTC (permalink / raw) To: linux-btrfs From: Filipe Manana <fdmanana@suse.com> When doing a device replace on a zoned filesystem, if we find a block group with ->to_copy == 0, we jump to the label 'done', which will result in later calling btrfs_unfreeze_block_group(), even though at this point we never called btrfs_freeze_block_group(). Since at this point we have neither turned the block group to RO mode nor made any progress, we don't need to jump to the label 'done'. So fix this by jumping instead to the label 'skip' and dropping our reference on the block group before the jump. Fixes: 78ce9fc269af6e ("btrfs: zoned: mark block groups to copy for device-replace") Signed-off-by: Filipe Manana <fdmanana@suse.com> --- V2: Removed label 'done' since it became unused after the fix. fs/btrfs/scrub.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index 17e49caad1f9..485cda3eb8d7 100644 --- a/fs/btrfs/scrub.c +++ b/fs/btrfs/scrub.c @@ -3674,8 +3674,8 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx, spin_lock(&cache->lock); if (!cache->to_copy) { spin_unlock(&cache->lock); - ro_set = 0; - goto done; + btrfs_put_block_group(cache); + goto skip; } spin_unlock(&cache->lock); } @@ -3833,7 +3833,6 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx, cache, found_key.offset)) ro_set = 0; -done: down_write(&dev_replace->rwsem); dev_replace->cursor_left = dev_replace->cursor_right; dev_replace->item_needs_writeback = 1; -- 2.28.0 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v2] btrfs: zoned: fix unpaired block group unfreeze during device replace 2021-04-14 13:05 ` [PATCH v2] " fdmanana @ 2021-04-20 17:34 ` David Sterba 0 siblings, 0 replies; 7+ messages in thread From: David Sterba @ 2021-04-20 17:34 UTC (permalink / raw) To: fdmanana; +Cc: linux-btrfs On Wed, Apr 14, 2021 at 02:05:26PM +0100, fdmanana@kernel.org wrote: > From: Filipe Manana <fdmanana@suse.com> > > When doing a device replace on a zoned filesystem, if we find a block > group with ->to_copy == 0, we jump to the label 'done', which will result > in later calling btrfs_unfreeze_block_group(), even though at this point > we never called btrfs_freeze_block_group(). > > Since at this point we have neither turned the block group to RO mode nor > made any progress, we don't need to jump to the label 'done'. So fix this > by jumping instead to the label 'skip' and dropping our reference on the > block group before the jump. > > Fixes: 78ce9fc269af6e ("btrfs: zoned: mark block groups to copy for device-replace") > Signed-off-by: Filipe Manana <fdmanana@suse.com> Thanks, added to misc-next. It's a regression fix for 5.12 but IMHO not critical enough for a pull request. 5.12 is about to be released in a week, with CC: stable the will be published soon after. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace 2021-04-14 12:08 [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace fdmanana @ 2021-04-14 16:32 ` kernel test robot 2021-04-14 13:05 ` [PATCH v2] " fdmanana 2021-04-14 16:32 ` kernel test robot 2 siblings, 0 replies; 7+ messages in thread From: kernel test robot @ 2021-04-14 16:32 UTC (permalink / raw) To: fdmanana, linux-btrfs; +Cc: kbuild-all, clang-built-linux [-- Attachment #1: Type: text/plain, Size: 24779 bytes --] Hi, Thank you for the patch! Perhaps something to improve: [auto build test WARNING on kdave/for-next] [also build test WARNING on v5.12-rc7 next-20210414] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/fdmanana-kernel-org/btrfs-zoned-fix-unpaired-block-group-unfreeze-during-device-replace/20210414-201003 base: https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-next config: x86_64-randconfig-r033-20210414 (attached as .config) compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 9829f5e6b1bca9b61efc629770d28bb9014dec45) reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # install x86_64 cross compiling tool for clang build # apt-get install binutils-x86-64-linux-gnu # https://github.com/0day-ci/linux/commit/31f44555b4f7341b558a9062f0977e8fe817045d git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review fdmanana-kernel-org/btrfs-zoned-fix-unpaired-block-group-unfreeze-during-device-replace/20210414-201003 git checkout 31f44555b4f7341b558a9062f0977e8fe817045d # save the attached .config to linux build tree COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=x86_64 If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All warnings (new ones prefixed by >>): >> fs/btrfs/scrub.c:3836:1: warning: unused label 'done' [-Wunused-label] done: ^~~~~ 1 warning generated. vim +/done +3836 fs/btrfs/scrub.c de17addce7a20d Naohiro Aota 2021-02-04 3586 a2de733c78fa7a Arne Jansen 2011-03-08 3587 static noinline_for_stack a36cf8b8933e4a Stefan Behrens 2012-11-02 3588 int scrub_enumerate_chunks(struct scrub_ctx *sctx, 32934280967d00 Omar Sandoval 2018-08-14 3589 struct btrfs_device *scrub_dev, u64 start, u64 end) a2de733c78fa7a Arne Jansen 2011-03-08 3590 { a2de733c78fa7a Arne Jansen 2011-03-08 3591 struct btrfs_dev_extent *dev_extent = NULL; a2de733c78fa7a Arne Jansen 2011-03-08 3592 struct btrfs_path *path; 0b246afa62b0cf Jeff Mahoney 2016-06-22 3593 struct btrfs_fs_info *fs_info = sctx->fs_info; 0b246afa62b0cf Jeff Mahoney 2016-06-22 3594 struct btrfs_root *root = fs_info->dev_root; a2de733c78fa7a Arne Jansen 2011-03-08 3595 u64 length; a2de733c78fa7a Arne Jansen 2011-03-08 3596 u64 chunk_offset; 55e3a601c81cdc Zhaolei 2015-08-05 3597 int ret = 0; 76a8efa171bf6c Zhaolei 2015-11-17 3598 int ro_set; a2de733c78fa7a Arne Jansen 2011-03-08 3599 int slot; a2de733c78fa7a Arne Jansen 2011-03-08 3600 struct extent_buffer *l; a2de733c78fa7a Arne Jansen 2011-03-08 3601 struct btrfs_key key; a2de733c78fa7a Arne Jansen 2011-03-08 3602 struct btrfs_key found_key; 32da5386d9a4fd David Sterba 2019-10-29 3603 struct btrfs_block_group *cache; ff023aac31198e Stefan Behrens 2012-11-06 3604 struct btrfs_dev_replace *dev_replace = &fs_info->dev_replace; a2de733c78fa7a Arne Jansen 2011-03-08 3605 a2de733c78fa7a Arne Jansen 2011-03-08 3606 path = btrfs_alloc_path(); a2de733c78fa7a Arne Jansen 2011-03-08 3607 if (!path) a2de733c78fa7a Arne Jansen 2011-03-08 3608 return -ENOMEM; a2de733c78fa7a Arne Jansen 2011-03-08 3609 e4058b54d1e442 David Sterba 2015-11-27 3610 path->reada = READA_FORWARD; a2de733c78fa7a Arne Jansen 2011-03-08 3611 path->search_commit_root = 1; a2de733c78fa7a Arne Jansen 2011-03-08 3612 path->skip_locking = 1; a2de733c78fa7a Arne Jansen 2011-03-08 3613 a36cf8b8933e4a Stefan Behrens 2012-11-02 3614 key.objectid = scrub_dev->devid; a2de733c78fa7a Arne Jansen 2011-03-08 3615 key.offset = 0ull; a2de733c78fa7a Arne Jansen 2011-03-08 3616 key.type = BTRFS_DEV_EXTENT_KEY; a2de733c78fa7a Arne Jansen 2011-03-08 3617 a2de733c78fa7a Arne Jansen 2011-03-08 3618 while (1) { a2de733c78fa7a Arne Jansen 2011-03-08 3619 ret = btrfs_search_slot(NULL, root, &key, path, 0, 0); a2de733c78fa7a Arne Jansen 2011-03-08 3620 if (ret < 0) 8c51032f978bac Arne Jansen 2011-06-03 3621 break; 8c51032f978bac Arne Jansen 2011-06-03 3622 if (ret > 0) { 8c51032f978bac Arne Jansen 2011-06-03 3623 if (path->slots[0] >= 8c51032f978bac Arne Jansen 2011-06-03 3624 btrfs_header_nritems(path->nodes[0])) { 8c51032f978bac Arne Jansen 2011-06-03 3625 ret = btrfs_next_leaf(root, path); 55e3a601c81cdc Zhaolei 2015-08-05 3626 if (ret < 0) 55e3a601c81cdc Zhaolei 2015-08-05 3627 break; 55e3a601c81cdc Zhaolei 2015-08-05 3628 if (ret > 0) { 55e3a601c81cdc Zhaolei 2015-08-05 3629 ret = 0; 8c51032f978bac Arne Jansen 2011-06-03 3630 break; 8c51032f978bac Arne Jansen 2011-06-03 3631 } 55e3a601c81cdc Zhaolei 2015-08-05 3632 } else { 55e3a601c81cdc Zhaolei 2015-08-05 3633 ret = 0; 55e3a601c81cdc Zhaolei 2015-08-05 3634 } 8c51032f978bac Arne Jansen 2011-06-03 3635 } a2de733c78fa7a Arne Jansen 2011-03-08 3636 a2de733c78fa7a Arne Jansen 2011-03-08 3637 l = path->nodes[0]; a2de733c78fa7a Arne Jansen 2011-03-08 3638 slot = path->slots[0]; a2de733c78fa7a Arne Jansen 2011-03-08 3639 a2de733c78fa7a Arne Jansen 2011-03-08 3640 btrfs_item_key_to_cpu(l, &found_key, slot); a2de733c78fa7a Arne Jansen 2011-03-08 3641 a36cf8b8933e4a Stefan Behrens 2012-11-02 3642 if (found_key.objectid != scrub_dev->devid) a2de733c78fa7a Arne Jansen 2011-03-08 3643 break; a2de733c78fa7a Arne Jansen 2011-03-08 3644 962a298f35110e David Sterba 2014-06-04 3645 if (found_key.type != BTRFS_DEV_EXTENT_KEY) a2de733c78fa7a Arne Jansen 2011-03-08 3646 break; a2de733c78fa7a Arne Jansen 2011-03-08 3647 a2de733c78fa7a Arne Jansen 2011-03-08 3648 if (found_key.offset >= end) a2de733c78fa7a Arne Jansen 2011-03-08 3649 break; a2de733c78fa7a Arne Jansen 2011-03-08 3650 a2de733c78fa7a Arne Jansen 2011-03-08 3651 if (found_key.offset < key.offset) a2de733c78fa7a Arne Jansen 2011-03-08 3652 break; a2de733c78fa7a Arne Jansen 2011-03-08 3653 a2de733c78fa7a Arne Jansen 2011-03-08 3654 dev_extent = btrfs_item_ptr(l, slot, struct btrfs_dev_extent); a2de733c78fa7a Arne Jansen 2011-03-08 3655 length = btrfs_dev_extent_length(l, dev_extent); a2de733c78fa7a Arne Jansen 2011-03-08 3656 ced96edc48ba45 Qu Wenruo 2014-06-19 3657 if (found_key.offset + length <= start) ced96edc48ba45 Qu Wenruo 2014-06-19 3658 goto skip; a2de733c78fa7a Arne Jansen 2011-03-08 3659 a2de733c78fa7a Arne Jansen 2011-03-08 3660 chunk_offset = btrfs_dev_extent_chunk_offset(l, dev_extent); a2de733c78fa7a Arne Jansen 2011-03-08 3661 a2de733c78fa7a Arne Jansen 2011-03-08 3662 /* a2de733c78fa7a Arne Jansen 2011-03-08 3663 * get a reference on the corresponding block group to prevent a2de733c78fa7a Arne Jansen 2011-03-08 3664 * the chunk from going away while we scrub it a2de733c78fa7a Arne Jansen 2011-03-08 3665 */ a2de733c78fa7a Arne Jansen 2011-03-08 3666 cache = btrfs_lookup_block_group(fs_info, chunk_offset); ced96edc48ba45 Qu Wenruo 2014-06-19 3667 ced96edc48ba45 Qu Wenruo 2014-06-19 3668 /* some chunks are removed but not committed to disk yet, ced96edc48ba45 Qu Wenruo 2014-06-19 3669 * continue scrubbing */ ced96edc48ba45 Qu Wenruo 2014-06-19 3670 if (!cache) ced96edc48ba45 Qu Wenruo 2014-06-19 3671 goto skip; ced96edc48ba45 Qu Wenruo 2014-06-19 3672 78ce9fc269af6e Naohiro Aota 2021-02-04 3673 if (sctx->is_dev_replace && btrfs_is_zoned(fs_info)) { 78ce9fc269af6e Naohiro Aota 2021-02-04 3674 spin_lock(&cache->lock); 78ce9fc269af6e Naohiro Aota 2021-02-04 3675 if (!cache->to_copy) { 78ce9fc269af6e Naohiro Aota 2021-02-04 3676 spin_unlock(&cache->lock); 31f44555b4f734 Filipe Manana 2021-04-14 3677 btrfs_put_block_group(cache); 31f44555b4f734 Filipe Manana 2021-04-14 3678 goto skip; 78ce9fc269af6e Naohiro Aota 2021-02-04 3679 } 78ce9fc269af6e Naohiro Aota 2021-02-04 3680 spin_unlock(&cache->lock); 78ce9fc269af6e Naohiro Aota 2021-02-04 3681 } 78ce9fc269af6e Naohiro Aota 2021-02-04 3682 2473d24f2b77da Filipe Manana 2020-05-08 3683 /* 2473d24f2b77da Filipe Manana 2020-05-08 3684 * Make sure that while we are scrubbing the corresponding block 2473d24f2b77da Filipe Manana 2020-05-08 3685 * group doesn't get its logical address and its device extents 2473d24f2b77da Filipe Manana 2020-05-08 3686 * reused for another block group, which can possibly be of a 2473d24f2b77da Filipe Manana 2020-05-08 3687 * different type and different profile. We do this to prevent 2473d24f2b77da Filipe Manana 2020-05-08 3688 * false error detections and crashes due to bogus attempts to 2473d24f2b77da Filipe Manana 2020-05-08 3689 * repair extents. 2473d24f2b77da Filipe Manana 2020-05-08 3690 */ 2473d24f2b77da Filipe Manana 2020-05-08 3691 spin_lock(&cache->lock); 2473d24f2b77da Filipe Manana 2020-05-08 3692 if (cache->removed) { 2473d24f2b77da Filipe Manana 2020-05-08 3693 spin_unlock(&cache->lock); 2473d24f2b77da Filipe Manana 2020-05-08 3694 btrfs_put_block_group(cache); 2473d24f2b77da Filipe Manana 2020-05-08 3695 goto skip; 2473d24f2b77da Filipe Manana 2020-05-08 3696 } 6b7304af62d02d Filipe Manana 2020-05-08 3697 btrfs_freeze_block_group(cache); 2473d24f2b77da Filipe Manana 2020-05-08 3698 spin_unlock(&cache->lock); 2473d24f2b77da Filipe Manana 2020-05-08 3699 55e3a601c81cdc Zhaolei 2015-08-05 3700 /* 55e3a601c81cdc Zhaolei 2015-08-05 3701 * we need call btrfs_inc_block_group_ro() with scrubs_paused, 55e3a601c81cdc Zhaolei 2015-08-05 3702 * to avoid deadlock caused by: 55e3a601c81cdc Zhaolei 2015-08-05 3703 * btrfs_inc_block_group_ro() 55e3a601c81cdc Zhaolei 2015-08-05 3704 * -> btrfs_wait_for_commit() 55e3a601c81cdc Zhaolei 2015-08-05 3705 * -> btrfs_commit_transaction() 55e3a601c81cdc Zhaolei 2015-08-05 3706 * -> btrfs_scrub_pause() 55e3a601c81cdc Zhaolei 2015-08-05 3707 */ 55e3a601c81cdc Zhaolei 2015-08-05 3708 scrub_pause_on(fs_info); b12de52896c0e8 Qu Wenruo 2019-11-15 3709 b12de52896c0e8 Qu Wenruo 2019-11-15 3710 /* b12de52896c0e8 Qu Wenruo 2019-11-15 3711 * Don't do chunk preallocation for scrub. b12de52896c0e8 Qu Wenruo 2019-11-15 3712 * b12de52896c0e8 Qu Wenruo 2019-11-15 3713 * This is especially important for SYSTEM bgs, or we can hit b12de52896c0e8 Qu Wenruo 2019-11-15 3714 * -EFBIG from btrfs_finish_chunk_alloc() like: b12de52896c0e8 Qu Wenruo 2019-11-15 3715 * 1. The only SYSTEM bg is marked RO. b12de52896c0e8 Qu Wenruo 2019-11-15 3716 * Since SYSTEM bg is small, that's pretty common. b12de52896c0e8 Qu Wenruo 2019-11-15 3717 * 2. New SYSTEM bg will be allocated b12de52896c0e8 Qu Wenruo 2019-11-15 3718 * Due to regular version will allocate new chunk. b12de52896c0e8 Qu Wenruo 2019-11-15 3719 * 3. New SYSTEM bg is empty and will get cleaned up b12de52896c0e8 Qu Wenruo 2019-11-15 3720 * Before cleanup really happens, it's marked RO again. b12de52896c0e8 Qu Wenruo 2019-11-15 3721 * 4. Empty SYSTEM bg get scrubbed b12de52896c0e8 Qu Wenruo 2019-11-15 3722 * We go back to 2. b12de52896c0e8 Qu Wenruo 2019-11-15 3723 * b12de52896c0e8 Qu Wenruo 2019-11-15 3724 * This can easily boost the amount of SYSTEM chunks if cleaner b12de52896c0e8 Qu Wenruo 2019-11-15 3725 * thread can't be triggered fast enough, and use up all space b12de52896c0e8 Qu Wenruo 2019-11-15 3726 * of btrfs_super_block::sys_chunk_array 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3727 * 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3728 * While for dev replace, we need to try our best to mark block 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3729 * group RO, to prevent race between: 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3730 * - Write duplication 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3731 * Contains latest data 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3732 * - Scrub copy 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3733 * Contains data from commit tree 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3734 * 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3735 * If target block group is not marked RO, nocow writes can 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3736 * be overwritten by scrub copy, causing data corruption. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3737 * So for dev-replace, it's not allowed to continue if a block 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3738 * group is not RO. b12de52896c0e8 Qu Wenruo 2019-11-15 3739 */ 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3740 ret = btrfs_inc_block_group_ro(cache, sctx->is_dev_replace); de17addce7a20d Naohiro Aota 2021-02-04 3741 if (!ret && sctx->is_dev_replace) { de17addce7a20d Naohiro Aota 2021-02-04 3742 ret = finish_extent_writes_for_zoned(root, cache); de17addce7a20d Naohiro Aota 2021-02-04 3743 if (ret) { de17addce7a20d Naohiro Aota 2021-02-04 3744 btrfs_dec_block_group_ro(cache); de17addce7a20d Naohiro Aota 2021-02-04 3745 scrub_pause_off(fs_info); de17addce7a20d Naohiro Aota 2021-02-04 3746 btrfs_put_block_group(cache); de17addce7a20d Naohiro Aota 2021-02-04 3747 break; de17addce7a20d Naohiro Aota 2021-02-04 3748 } de17addce7a20d Naohiro Aota 2021-02-04 3749 } de17addce7a20d Naohiro Aota 2021-02-04 3750 76a8efa171bf6c Zhaolei 2015-11-17 3751 if (ret == 0) { 76a8efa171bf6c Zhaolei 2015-11-17 3752 ro_set = 1; 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3753 } else if (ret == -ENOSPC && !sctx->is_dev_replace) { 76a8efa171bf6c Zhaolei 2015-11-17 3754 /* 76a8efa171bf6c Zhaolei 2015-11-17 3755 * btrfs_inc_block_group_ro return -ENOSPC when it 76a8efa171bf6c Zhaolei 2015-11-17 3756 * failed in creating new chunk for metadata. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3757 * It is not a problem for scrub, because 76a8efa171bf6c Zhaolei 2015-11-17 3758 * metadata are always cowed, and our scrub paused 76a8efa171bf6c Zhaolei 2015-11-17 3759 * commit_transactions. 76a8efa171bf6c Zhaolei 2015-11-17 3760 */ 76a8efa171bf6c Zhaolei 2015-11-17 3761 ro_set = 0; 195a49eaf655eb Filipe Manana 2021-02-05 3762 } else if (ret == -ETXTBSY) { 195a49eaf655eb Filipe Manana 2021-02-05 3763 btrfs_warn(fs_info, 195a49eaf655eb Filipe Manana 2021-02-05 3764 "skipping scrub of block group %llu due to active swapfile", 195a49eaf655eb Filipe Manana 2021-02-05 3765 cache->start); 195a49eaf655eb Filipe Manana 2021-02-05 3766 scrub_pause_off(fs_info); 195a49eaf655eb Filipe Manana 2021-02-05 3767 ret = 0; 195a49eaf655eb Filipe Manana 2021-02-05 3768 goto skip_unfreeze; 76a8efa171bf6c Zhaolei 2015-11-17 3769 } else { 5d163e0e68ce74 Jeff Mahoney 2016-09-20 3770 btrfs_warn(fs_info, 913e153572218c David Sterba 2017-07-13 3771 "failed setting block group ro: %d", ret); 6b7304af62d02d Filipe Manana 2020-05-08 3772 btrfs_unfreeze_block_group(cache); 55e3a601c81cdc Zhaolei 2015-08-05 3773 btrfs_put_block_group(cache); 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3774 scrub_pause_off(fs_info); 55e3a601c81cdc Zhaolei 2015-08-05 3775 break; 55e3a601c81cdc Zhaolei 2015-08-05 3776 } 55e3a601c81cdc Zhaolei 2015-08-05 3777 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3778 /* 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3779 * Now the target block is marked RO, wait for nocow writes to 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3780 * finish before dev-replace. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3781 * COW is fine, as COW never overwrites extents in commit tree. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3782 */ 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3783 if (sctx->is_dev_replace) { 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3784 btrfs_wait_nocow_writers(cache); 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3785 btrfs_wait_ordered_roots(fs_info, U64_MAX, cache->start, 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3786 cache->length); 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3787 } 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3788 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3789 scrub_pause_off(fs_info); 3ec17a67cc340b Dan Carpenter 2019-10-31 3790 down_write(&dev_replace->rwsem); ff023aac31198e Stefan Behrens 2012-11-06 3791 dev_replace->cursor_right = found_key.offset + length; ff023aac31198e Stefan Behrens 2012-11-06 3792 dev_replace->cursor_left = found_key.offset; ff023aac31198e Stefan Behrens 2012-11-06 3793 dev_replace->item_needs_writeback = 1; cb5583dd52fab4 David Sterba 2018-09-07 3794 up_write(&dev_replace->rwsem); cb5583dd52fab4 David Sterba 2018-09-07 3795 8c204c9657c32e Zhao Lei 2015-08-19 3796 ret = scrub_chunk(sctx, scrub_dev, chunk_offset, length, 32934280967d00 Omar Sandoval 2018-08-14 3797 found_key.offset, cache); ff023aac31198e Stefan Behrens 2012-11-06 3798 ff023aac31198e Stefan Behrens 2012-11-06 3799 /* ff023aac31198e Stefan Behrens 2012-11-06 3800 * flush, submit all pending read and write bios, afterwards ff023aac31198e Stefan Behrens 2012-11-06 3801 * wait for them. ff023aac31198e Stefan Behrens 2012-11-06 3802 * Note that in the dev replace case, a read request causes ff023aac31198e Stefan Behrens 2012-11-06 3803 * write requests that are submitted in the read completion ff023aac31198e Stefan Behrens 2012-11-06 3804 * worker. Therefore in the current situation, it is required ff023aac31198e Stefan Behrens 2012-11-06 3805 * that all write requests are flushed, so that all read and ff023aac31198e Stefan Behrens 2012-11-06 3806 * write requests are really completed when bios_in_flight ff023aac31198e Stefan Behrens 2012-11-06 3807 * changes to 0. ff023aac31198e Stefan Behrens 2012-11-06 3808 */ 2073c4c2e51a93 David Sterba 2017-03-31 3809 sctx->flush_all_writes = true; ff023aac31198e Stefan Behrens 2012-11-06 3810 scrub_submit(sctx); 3fb99303c64e31 David Sterba 2017-05-16 3811 mutex_lock(&sctx->wr_lock); ff023aac31198e Stefan Behrens 2012-11-06 3812 scrub_wr_submit(sctx); 3fb99303c64e31 David Sterba 2017-05-16 3813 mutex_unlock(&sctx->wr_lock); ff023aac31198e Stefan Behrens 2012-11-06 3814 ff023aac31198e Stefan Behrens 2012-11-06 3815 wait_event(sctx->list_wait, ff023aac31198e Stefan Behrens 2012-11-06 3816 atomic_read(&sctx->bios_in_flight) == 0); b708ce969af3ce Zhaolei 2015-08-05 3817 b708ce969af3ce Zhaolei 2015-08-05 3818 scrub_pause_on(fs_info); 12cf93728dfba2 Wang Shilong 2014-02-19 3819 12cf93728dfba2 Wang Shilong 2014-02-19 3820 /* 12cf93728dfba2 Wang Shilong 2014-02-19 3821 * must be called before we decrease @scrub_paused. 12cf93728dfba2 Wang Shilong 2014-02-19 3822 * make sure we don't block transaction commit while 12cf93728dfba2 Wang Shilong 2014-02-19 3823 * we are waiting pending workers finished. 12cf93728dfba2 Wang Shilong 2014-02-19 3824 */ ff023aac31198e Stefan Behrens 2012-11-06 3825 wait_event(sctx->list_wait, ff023aac31198e Stefan Behrens 2012-11-06 3826 atomic_read(&sctx->workers_pending) == 0); 2073c4c2e51a93 David Sterba 2017-03-31 3827 sctx->flush_all_writes = false; 12cf93728dfba2 Wang Shilong 2014-02-19 3828 b708ce969af3ce Zhaolei 2015-08-05 3829 scrub_pause_off(fs_info); ff023aac31198e Stefan Behrens 2012-11-06 3830 78ce9fc269af6e Naohiro Aota 2021-02-04 3831 if (sctx->is_dev_replace && 78ce9fc269af6e Naohiro Aota 2021-02-04 3832 !btrfs_finish_block_group_to_copy(dev_replace->srcdev, 78ce9fc269af6e Naohiro Aota 2021-02-04 3833 cache, found_key.offset)) 78ce9fc269af6e Naohiro Aota 2021-02-04 3834 ro_set = 0; 78ce9fc269af6e Naohiro Aota 2021-02-04 3835 78ce9fc269af6e Naohiro Aota 2021-02-04 @3836 done: 3ec17a67cc340b Dan Carpenter 2019-10-31 3837 down_write(&dev_replace->rwsem); 1a1a8b732c7e95 Filipe Manana 2016-05-14 3838 dev_replace->cursor_left = dev_replace->cursor_right; 1a1a8b732c7e95 Filipe Manana 2016-05-14 3839 dev_replace->item_needs_writeback = 1; 3ec17a67cc340b Dan Carpenter 2019-10-31 3840 up_write(&dev_replace->rwsem); 1a1a8b732c7e95 Filipe Manana 2016-05-14 3841 76a8efa171bf6c Zhaolei 2015-11-17 3842 if (ro_set) 2ff7e61e0d30ff Jeff Mahoney 2016-06-22 3843 btrfs_dec_block_group_ro(cache); ff023aac31198e Stefan Behrens 2012-11-06 3844 758f2dfcf8a249 Filipe Manana 2015-11-19 3845 /* 758f2dfcf8a249 Filipe Manana 2015-11-19 3846 * We might have prevented the cleaner kthread from deleting 758f2dfcf8a249 Filipe Manana 2015-11-19 3847 * this block group if it was already unused because we raced 758f2dfcf8a249 Filipe Manana 2015-11-19 3848 * and set it to RO mode first. So add it back to the unused 758f2dfcf8a249 Filipe Manana 2015-11-19 3849 * list, otherwise it might not ever be deleted unless a manual 758f2dfcf8a249 Filipe Manana 2015-11-19 3850 * balance is triggered or it becomes used and unused again. 758f2dfcf8a249 Filipe Manana 2015-11-19 3851 */ 758f2dfcf8a249 Filipe Manana 2015-11-19 3852 spin_lock(&cache->lock); 758f2dfcf8a249 Filipe Manana 2015-11-19 3853 if (!cache->removed && !cache->ro && cache->reserved == 0 && bf38be65f3703d David Sterba 2019-10-23 3854 cache->used == 0) { 758f2dfcf8a249 Filipe Manana 2015-11-19 3855 spin_unlock(&cache->lock); 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3856 if (btrfs_test_opt(fs_info, DISCARD_ASYNC)) 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3857 btrfs_discard_queue_work(&fs_info->discard_ctl, 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3858 cache); 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3859 else 031f24da2c8a7b Qu Wenruo 2018-05-22 3860 btrfs_mark_bg_unused(cache); 758f2dfcf8a249 Filipe Manana 2015-11-19 3861 } else { 758f2dfcf8a249 Filipe Manana 2015-11-19 3862 spin_unlock(&cache->lock); 758f2dfcf8a249 Filipe Manana 2015-11-19 3863 } 195a49eaf655eb Filipe Manana 2021-02-05 3864 skip_unfreeze: 6b7304af62d02d Filipe Manana 2020-05-08 3865 btrfs_unfreeze_block_group(cache); a2de733c78fa7a Arne Jansen 2011-03-08 3866 btrfs_put_block_group(cache); a2de733c78fa7a Arne Jansen 2011-03-08 3867 if (ret) a2de733c78fa7a Arne Jansen 2011-03-08 3868 break; 32934280967d00 Omar Sandoval 2018-08-14 3869 if (sctx->is_dev_replace && af1be4f851db4f Stefan Behrens 2012-11-27 3870 atomic64_read(&dev_replace->num_write_errors) > 0) { ff023aac31198e Stefan Behrens 2012-11-06 3871 ret = -EIO; ff023aac31198e Stefan Behrens 2012-11-06 3872 break; ff023aac31198e Stefan Behrens 2012-11-06 3873 } ff023aac31198e Stefan Behrens 2012-11-06 3874 if (sctx->stat.malloc_errors > 0) { ff023aac31198e Stefan Behrens 2012-11-06 3875 ret = -ENOMEM; ff023aac31198e Stefan Behrens 2012-11-06 3876 break; ff023aac31198e Stefan Behrens 2012-11-06 3877 } ced96edc48ba45 Qu Wenruo 2014-06-19 3878 skip: a2de733c78fa7a Arne Jansen 2011-03-08 3879 key.offset = found_key.offset + length; 712673339a0d08 Chris Mason 2011-05-23 3880 btrfs_release_path(path); a2de733c78fa7a Arne Jansen 2011-03-08 3881 } a2de733c78fa7a Arne Jansen 2011-03-08 3882 a2de733c78fa7a Arne Jansen 2011-03-08 3883 btrfs_free_path(path); 8c51032f978bac Arne Jansen 2011-06-03 3884 55e3a601c81cdc Zhaolei 2015-08-05 3885 return ret; a2de733c78fa7a Arne Jansen 2011-03-08 3886 } a2de733c78fa7a Arne Jansen 2011-03-08 3887 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org [-- Attachment #2: .config.gz --] [-- Type: application/gzip, Size: 41778 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace @ 2021-04-14 16:32 ` kernel test robot 0 siblings, 0 replies; 7+ messages in thread From: kernel test robot @ 2021-04-14 16:32 UTC (permalink / raw) To: kbuild-all [-- Attachment #1: Type: text/plain, Size: 25126 bytes --] Hi, Thank you for the patch! Perhaps something to improve: [auto build test WARNING on kdave/for-next] [also build test WARNING on v5.12-rc7 next-20210414] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/fdmanana-kernel-org/btrfs-zoned-fix-unpaired-block-group-unfreeze-during-device-replace/20210414-201003 base: https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-next config: x86_64-randconfig-r033-20210414 (attached as .config) compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 9829f5e6b1bca9b61efc629770d28bb9014dec45) reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # install x86_64 cross compiling tool for clang build # apt-get install binutils-x86-64-linux-gnu # https://github.com/0day-ci/linux/commit/31f44555b4f7341b558a9062f0977e8fe817045d git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review fdmanana-kernel-org/btrfs-zoned-fix-unpaired-block-group-unfreeze-during-device-replace/20210414-201003 git checkout 31f44555b4f7341b558a9062f0977e8fe817045d # save the attached .config to linux build tree COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=x86_64 If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All warnings (new ones prefixed by >>): >> fs/btrfs/scrub.c:3836:1: warning: unused label 'done' [-Wunused-label] done: ^~~~~ 1 warning generated. vim +/done +3836 fs/btrfs/scrub.c de17addce7a20d Naohiro Aota 2021-02-04 3586 a2de733c78fa7a Arne Jansen 2011-03-08 3587 static noinline_for_stack a36cf8b8933e4a Stefan Behrens 2012-11-02 3588 int scrub_enumerate_chunks(struct scrub_ctx *sctx, 32934280967d00 Omar Sandoval 2018-08-14 3589 struct btrfs_device *scrub_dev, u64 start, u64 end) a2de733c78fa7a Arne Jansen 2011-03-08 3590 { a2de733c78fa7a Arne Jansen 2011-03-08 3591 struct btrfs_dev_extent *dev_extent = NULL; a2de733c78fa7a Arne Jansen 2011-03-08 3592 struct btrfs_path *path; 0b246afa62b0cf Jeff Mahoney 2016-06-22 3593 struct btrfs_fs_info *fs_info = sctx->fs_info; 0b246afa62b0cf Jeff Mahoney 2016-06-22 3594 struct btrfs_root *root = fs_info->dev_root; a2de733c78fa7a Arne Jansen 2011-03-08 3595 u64 length; a2de733c78fa7a Arne Jansen 2011-03-08 3596 u64 chunk_offset; 55e3a601c81cdc Zhaolei 2015-08-05 3597 int ret = 0; 76a8efa171bf6c Zhaolei 2015-11-17 3598 int ro_set; a2de733c78fa7a Arne Jansen 2011-03-08 3599 int slot; a2de733c78fa7a Arne Jansen 2011-03-08 3600 struct extent_buffer *l; a2de733c78fa7a Arne Jansen 2011-03-08 3601 struct btrfs_key key; a2de733c78fa7a Arne Jansen 2011-03-08 3602 struct btrfs_key found_key; 32da5386d9a4fd David Sterba 2019-10-29 3603 struct btrfs_block_group *cache; ff023aac31198e Stefan Behrens 2012-11-06 3604 struct btrfs_dev_replace *dev_replace = &fs_info->dev_replace; a2de733c78fa7a Arne Jansen 2011-03-08 3605 a2de733c78fa7a Arne Jansen 2011-03-08 3606 path = btrfs_alloc_path(); a2de733c78fa7a Arne Jansen 2011-03-08 3607 if (!path) a2de733c78fa7a Arne Jansen 2011-03-08 3608 return -ENOMEM; a2de733c78fa7a Arne Jansen 2011-03-08 3609 e4058b54d1e442 David Sterba 2015-11-27 3610 path->reada = READA_FORWARD; a2de733c78fa7a Arne Jansen 2011-03-08 3611 path->search_commit_root = 1; a2de733c78fa7a Arne Jansen 2011-03-08 3612 path->skip_locking = 1; a2de733c78fa7a Arne Jansen 2011-03-08 3613 a36cf8b8933e4a Stefan Behrens 2012-11-02 3614 key.objectid = scrub_dev->devid; a2de733c78fa7a Arne Jansen 2011-03-08 3615 key.offset = 0ull; a2de733c78fa7a Arne Jansen 2011-03-08 3616 key.type = BTRFS_DEV_EXTENT_KEY; a2de733c78fa7a Arne Jansen 2011-03-08 3617 a2de733c78fa7a Arne Jansen 2011-03-08 3618 while (1) { a2de733c78fa7a Arne Jansen 2011-03-08 3619 ret = btrfs_search_slot(NULL, root, &key, path, 0, 0); a2de733c78fa7a Arne Jansen 2011-03-08 3620 if (ret < 0) 8c51032f978bac Arne Jansen 2011-06-03 3621 break; 8c51032f978bac Arne Jansen 2011-06-03 3622 if (ret > 0) { 8c51032f978bac Arne Jansen 2011-06-03 3623 if (path->slots[0] >= 8c51032f978bac Arne Jansen 2011-06-03 3624 btrfs_header_nritems(path->nodes[0])) { 8c51032f978bac Arne Jansen 2011-06-03 3625 ret = btrfs_next_leaf(root, path); 55e3a601c81cdc Zhaolei 2015-08-05 3626 if (ret < 0) 55e3a601c81cdc Zhaolei 2015-08-05 3627 break; 55e3a601c81cdc Zhaolei 2015-08-05 3628 if (ret > 0) { 55e3a601c81cdc Zhaolei 2015-08-05 3629 ret = 0; 8c51032f978bac Arne Jansen 2011-06-03 3630 break; 8c51032f978bac Arne Jansen 2011-06-03 3631 } 55e3a601c81cdc Zhaolei 2015-08-05 3632 } else { 55e3a601c81cdc Zhaolei 2015-08-05 3633 ret = 0; 55e3a601c81cdc Zhaolei 2015-08-05 3634 } 8c51032f978bac Arne Jansen 2011-06-03 3635 } a2de733c78fa7a Arne Jansen 2011-03-08 3636 a2de733c78fa7a Arne Jansen 2011-03-08 3637 l = path->nodes[0]; a2de733c78fa7a Arne Jansen 2011-03-08 3638 slot = path->slots[0]; a2de733c78fa7a Arne Jansen 2011-03-08 3639 a2de733c78fa7a Arne Jansen 2011-03-08 3640 btrfs_item_key_to_cpu(l, &found_key, slot); a2de733c78fa7a Arne Jansen 2011-03-08 3641 a36cf8b8933e4a Stefan Behrens 2012-11-02 3642 if (found_key.objectid != scrub_dev->devid) a2de733c78fa7a Arne Jansen 2011-03-08 3643 break; a2de733c78fa7a Arne Jansen 2011-03-08 3644 962a298f35110e David Sterba 2014-06-04 3645 if (found_key.type != BTRFS_DEV_EXTENT_KEY) a2de733c78fa7a Arne Jansen 2011-03-08 3646 break; a2de733c78fa7a Arne Jansen 2011-03-08 3647 a2de733c78fa7a Arne Jansen 2011-03-08 3648 if (found_key.offset >= end) a2de733c78fa7a Arne Jansen 2011-03-08 3649 break; a2de733c78fa7a Arne Jansen 2011-03-08 3650 a2de733c78fa7a Arne Jansen 2011-03-08 3651 if (found_key.offset < key.offset) a2de733c78fa7a Arne Jansen 2011-03-08 3652 break; a2de733c78fa7a Arne Jansen 2011-03-08 3653 a2de733c78fa7a Arne Jansen 2011-03-08 3654 dev_extent = btrfs_item_ptr(l, slot, struct btrfs_dev_extent); a2de733c78fa7a Arne Jansen 2011-03-08 3655 length = btrfs_dev_extent_length(l, dev_extent); a2de733c78fa7a Arne Jansen 2011-03-08 3656 ced96edc48ba45 Qu Wenruo 2014-06-19 3657 if (found_key.offset + length <= start) ced96edc48ba45 Qu Wenruo 2014-06-19 3658 goto skip; a2de733c78fa7a Arne Jansen 2011-03-08 3659 a2de733c78fa7a Arne Jansen 2011-03-08 3660 chunk_offset = btrfs_dev_extent_chunk_offset(l, dev_extent); a2de733c78fa7a Arne Jansen 2011-03-08 3661 a2de733c78fa7a Arne Jansen 2011-03-08 3662 /* a2de733c78fa7a Arne Jansen 2011-03-08 3663 * get a reference on the corresponding block group to prevent a2de733c78fa7a Arne Jansen 2011-03-08 3664 * the chunk from going away while we scrub it a2de733c78fa7a Arne Jansen 2011-03-08 3665 */ a2de733c78fa7a Arne Jansen 2011-03-08 3666 cache = btrfs_lookup_block_group(fs_info, chunk_offset); ced96edc48ba45 Qu Wenruo 2014-06-19 3667 ced96edc48ba45 Qu Wenruo 2014-06-19 3668 /* some chunks are removed but not committed to disk yet, ced96edc48ba45 Qu Wenruo 2014-06-19 3669 * continue scrubbing */ ced96edc48ba45 Qu Wenruo 2014-06-19 3670 if (!cache) ced96edc48ba45 Qu Wenruo 2014-06-19 3671 goto skip; ced96edc48ba45 Qu Wenruo 2014-06-19 3672 78ce9fc269af6e Naohiro Aota 2021-02-04 3673 if (sctx->is_dev_replace && btrfs_is_zoned(fs_info)) { 78ce9fc269af6e Naohiro Aota 2021-02-04 3674 spin_lock(&cache->lock); 78ce9fc269af6e Naohiro Aota 2021-02-04 3675 if (!cache->to_copy) { 78ce9fc269af6e Naohiro Aota 2021-02-04 3676 spin_unlock(&cache->lock); 31f44555b4f734 Filipe Manana 2021-04-14 3677 btrfs_put_block_group(cache); 31f44555b4f734 Filipe Manana 2021-04-14 3678 goto skip; 78ce9fc269af6e Naohiro Aota 2021-02-04 3679 } 78ce9fc269af6e Naohiro Aota 2021-02-04 3680 spin_unlock(&cache->lock); 78ce9fc269af6e Naohiro Aota 2021-02-04 3681 } 78ce9fc269af6e Naohiro Aota 2021-02-04 3682 2473d24f2b77da Filipe Manana 2020-05-08 3683 /* 2473d24f2b77da Filipe Manana 2020-05-08 3684 * Make sure that while we are scrubbing the corresponding block 2473d24f2b77da Filipe Manana 2020-05-08 3685 * group doesn't get its logical address and its device extents 2473d24f2b77da Filipe Manana 2020-05-08 3686 * reused for another block group, which can possibly be of a 2473d24f2b77da Filipe Manana 2020-05-08 3687 * different type and different profile. We do this to prevent 2473d24f2b77da Filipe Manana 2020-05-08 3688 * false error detections and crashes due to bogus attempts to 2473d24f2b77da Filipe Manana 2020-05-08 3689 * repair extents. 2473d24f2b77da Filipe Manana 2020-05-08 3690 */ 2473d24f2b77da Filipe Manana 2020-05-08 3691 spin_lock(&cache->lock); 2473d24f2b77da Filipe Manana 2020-05-08 3692 if (cache->removed) { 2473d24f2b77da Filipe Manana 2020-05-08 3693 spin_unlock(&cache->lock); 2473d24f2b77da Filipe Manana 2020-05-08 3694 btrfs_put_block_group(cache); 2473d24f2b77da Filipe Manana 2020-05-08 3695 goto skip; 2473d24f2b77da Filipe Manana 2020-05-08 3696 } 6b7304af62d02d Filipe Manana 2020-05-08 3697 btrfs_freeze_block_group(cache); 2473d24f2b77da Filipe Manana 2020-05-08 3698 spin_unlock(&cache->lock); 2473d24f2b77da Filipe Manana 2020-05-08 3699 55e3a601c81cdc Zhaolei 2015-08-05 3700 /* 55e3a601c81cdc Zhaolei 2015-08-05 3701 * we need call btrfs_inc_block_group_ro() with scrubs_paused, 55e3a601c81cdc Zhaolei 2015-08-05 3702 * to avoid deadlock caused by: 55e3a601c81cdc Zhaolei 2015-08-05 3703 * btrfs_inc_block_group_ro() 55e3a601c81cdc Zhaolei 2015-08-05 3704 * -> btrfs_wait_for_commit() 55e3a601c81cdc Zhaolei 2015-08-05 3705 * -> btrfs_commit_transaction() 55e3a601c81cdc Zhaolei 2015-08-05 3706 * -> btrfs_scrub_pause() 55e3a601c81cdc Zhaolei 2015-08-05 3707 */ 55e3a601c81cdc Zhaolei 2015-08-05 3708 scrub_pause_on(fs_info); b12de52896c0e8 Qu Wenruo 2019-11-15 3709 b12de52896c0e8 Qu Wenruo 2019-11-15 3710 /* b12de52896c0e8 Qu Wenruo 2019-11-15 3711 * Don't do chunk preallocation for scrub. b12de52896c0e8 Qu Wenruo 2019-11-15 3712 * b12de52896c0e8 Qu Wenruo 2019-11-15 3713 * This is especially important for SYSTEM bgs, or we can hit b12de52896c0e8 Qu Wenruo 2019-11-15 3714 * -EFBIG from btrfs_finish_chunk_alloc() like: b12de52896c0e8 Qu Wenruo 2019-11-15 3715 * 1. The only SYSTEM bg is marked RO. b12de52896c0e8 Qu Wenruo 2019-11-15 3716 * Since SYSTEM bg is small, that's pretty common. b12de52896c0e8 Qu Wenruo 2019-11-15 3717 * 2. New SYSTEM bg will be allocated b12de52896c0e8 Qu Wenruo 2019-11-15 3718 * Due to regular version will allocate new chunk. b12de52896c0e8 Qu Wenruo 2019-11-15 3719 * 3. New SYSTEM bg is empty and will get cleaned up b12de52896c0e8 Qu Wenruo 2019-11-15 3720 * Before cleanup really happens, it's marked RO again. b12de52896c0e8 Qu Wenruo 2019-11-15 3721 * 4. Empty SYSTEM bg get scrubbed b12de52896c0e8 Qu Wenruo 2019-11-15 3722 * We go back to 2. b12de52896c0e8 Qu Wenruo 2019-11-15 3723 * b12de52896c0e8 Qu Wenruo 2019-11-15 3724 * This can easily boost the amount of SYSTEM chunks if cleaner b12de52896c0e8 Qu Wenruo 2019-11-15 3725 * thread can't be triggered fast enough, and use up all space b12de52896c0e8 Qu Wenruo 2019-11-15 3726 * of btrfs_super_block::sys_chunk_array 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3727 * 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3728 * While for dev replace, we need to try our best to mark block 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3729 * group RO, to prevent race between: 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3730 * - Write duplication 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3731 * Contains latest data 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3732 * - Scrub copy 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3733 * Contains data from commit tree 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3734 * 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3735 * If target block group is not marked RO, nocow writes can 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3736 * be overwritten by scrub copy, causing data corruption. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3737 * So for dev-replace, it's not allowed to continue if a block 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3738 * group is not RO. b12de52896c0e8 Qu Wenruo 2019-11-15 3739 */ 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3740 ret = btrfs_inc_block_group_ro(cache, sctx->is_dev_replace); de17addce7a20d Naohiro Aota 2021-02-04 3741 if (!ret && sctx->is_dev_replace) { de17addce7a20d Naohiro Aota 2021-02-04 3742 ret = finish_extent_writes_for_zoned(root, cache); de17addce7a20d Naohiro Aota 2021-02-04 3743 if (ret) { de17addce7a20d Naohiro Aota 2021-02-04 3744 btrfs_dec_block_group_ro(cache); de17addce7a20d Naohiro Aota 2021-02-04 3745 scrub_pause_off(fs_info); de17addce7a20d Naohiro Aota 2021-02-04 3746 btrfs_put_block_group(cache); de17addce7a20d Naohiro Aota 2021-02-04 3747 break; de17addce7a20d Naohiro Aota 2021-02-04 3748 } de17addce7a20d Naohiro Aota 2021-02-04 3749 } de17addce7a20d Naohiro Aota 2021-02-04 3750 76a8efa171bf6c Zhaolei 2015-11-17 3751 if (ret == 0) { 76a8efa171bf6c Zhaolei 2015-11-17 3752 ro_set = 1; 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3753 } else if (ret == -ENOSPC && !sctx->is_dev_replace) { 76a8efa171bf6c Zhaolei 2015-11-17 3754 /* 76a8efa171bf6c Zhaolei 2015-11-17 3755 * btrfs_inc_block_group_ro return -ENOSPC when it 76a8efa171bf6c Zhaolei 2015-11-17 3756 * failed in creating new chunk for metadata. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3757 * It is not a problem for scrub, because 76a8efa171bf6c Zhaolei 2015-11-17 3758 * metadata are always cowed, and our scrub paused 76a8efa171bf6c Zhaolei 2015-11-17 3759 * commit_transactions. 76a8efa171bf6c Zhaolei 2015-11-17 3760 */ 76a8efa171bf6c Zhaolei 2015-11-17 3761 ro_set = 0; 195a49eaf655eb Filipe Manana 2021-02-05 3762 } else if (ret == -ETXTBSY) { 195a49eaf655eb Filipe Manana 2021-02-05 3763 btrfs_warn(fs_info, 195a49eaf655eb Filipe Manana 2021-02-05 3764 "skipping scrub of block group %llu due to active swapfile", 195a49eaf655eb Filipe Manana 2021-02-05 3765 cache->start); 195a49eaf655eb Filipe Manana 2021-02-05 3766 scrub_pause_off(fs_info); 195a49eaf655eb Filipe Manana 2021-02-05 3767 ret = 0; 195a49eaf655eb Filipe Manana 2021-02-05 3768 goto skip_unfreeze; 76a8efa171bf6c Zhaolei 2015-11-17 3769 } else { 5d163e0e68ce74 Jeff Mahoney 2016-09-20 3770 btrfs_warn(fs_info, 913e153572218c David Sterba 2017-07-13 3771 "failed setting block group ro: %d", ret); 6b7304af62d02d Filipe Manana 2020-05-08 3772 btrfs_unfreeze_block_group(cache); 55e3a601c81cdc Zhaolei 2015-08-05 3773 btrfs_put_block_group(cache); 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3774 scrub_pause_off(fs_info); 55e3a601c81cdc Zhaolei 2015-08-05 3775 break; 55e3a601c81cdc Zhaolei 2015-08-05 3776 } 55e3a601c81cdc Zhaolei 2015-08-05 3777 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3778 /* 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3779 * Now the target block is marked RO, wait for nocow writes to 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3780 * finish before dev-replace. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3781 * COW is fine, as COW never overwrites extents in commit tree. 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3782 */ 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3783 if (sctx->is_dev_replace) { 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3784 btrfs_wait_nocow_writers(cache); 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3785 btrfs_wait_ordered_roots(fs_info, U64_MAX, cache->start, 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3786 cache->length); 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3787 } 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3788 1bbb97b8ce7ddf Qu Wenruo 2020-01-24 3789 scrub_pause_off(fs_info); 3ec17a67cc340b Dan Carpenter 2019-10-31 3790 down_write(&dev_replace->rwsem); ff023aac31198e Stefan Behrens 2012-11-06 3791 dev_replace->cursor_right = found_key.offset + length; ff023aac31198e Stefan Behrens 2012-11-06 3792 dev_replace->cursor_left = found_key.offset; ff023aac31198e Stefan Behrens 2012-11-06 3793 dev_replace->item_needs_writeback = 1; cb5583dd52fab4 David Sterba 2018-09-07 3794 up_write(&dev_replace->rwsem); cb5583dd52fab4 David Sterba 2018-09-07 3795 8c204c9657c32e Zhao Lei 2015-08-19 3796 ret = scrub_chunk(sctx, scrub_dev, chunk_offset, length, 32934280967d00 Omar Sandoval 2018-08-14 3797 found_key.offset, cache); ff023aac31198e Stefan Behrens 2012-11-06 3798 ff023aac31198e Stefan Behrens 2012-11-06 3799 /* ff023aac31198e Stefan Behrens 2012-11-06 3800 * flush, submit all pending read and write bios, afterwards ff023aac31198e Stefan Behrens 2012-11-06 3801 * wait for them. ff023aac31198e Stefan Behrens 2012-11-06 3802 * Note that in the dev replace case, a read request causes ff023aac31198e Stefan Behrens 2012-11-06 3803 * write requests that are submitted in the read completion ff023aac31198e Stefan Behrens 2012-11-06 3804 * worker. Therefore in the current situation, it is required ff023aac31198e Stefan Behrens 2012-11-06 3805 * that all write requests are flushed, so that all read and ff023aac31198e Stefan Behrens 2012-11-06 3806 * write requests are really completed when bios_in_flight ff023aac31198e Stefan Behrens 2012-11-06 3807 * changes to 0. ff023aac31198e Stefan Behrens 2012-11-06 3808 */ 2073c4c2e51a93 David Sterba 2017-03-31 3809 sctx->flush_all_writes = true; ff023aac31198e Stefan Behrens 2012-11-06 3810 scrub_submit(sctx); 3fb99303c64e31 David Sterba 2017-05-16 3811 mutex_lock(&sctx->wr_lock); ff023aac31198e Stefan Behrens 2012-11-06 3812 scrub_wr_submit(sctx); 3fb99303c64e31 David Sterba 2017-05-16 3813 mutex_unlock(&sctx->wr_lock); ff023aac31198e Stefan Behrens 2012-11-06 3814 ff023aac31198e Stefan Behrens 2012-11-06 3815 wait_event(sctx->list_wait, ff023aac31198e Stefan Behrens 2012-11-06 3816 atomic_read(&sctx->bios_in_flight) == 0); b708ce969af3ce Zhaolei 2015-08-05 3817 b708ce969af3ce Zhaolei 2015-08-05 3818 scrub_pause_on(fs_info); 12cf93728dfba2 Wang Shilong 2014-02-19 3819 12cf93728dfba2 Wang Shilong 2014-02-19 3820 /* 12cf93728dfba2 Wang Shilong 2014-02-19 3821 * must be called before we decrease @scrub_paused. 12cf93728dfba2 Wang Shilong 2014-02-19 3822 * make sure we don't block transaction commit while 12cf93728dfba2 Wang Shilong 2014-02-19 3823 * we are waiting pending workers finished. 12cf93728dfba2 Wang Shilong 2014-02-19 3824 */ ff023aac31198e Stefan Behrens 2012-11-06 3825 wait_event(sctx->list_wait, ff023aac31198e Stefan Behrens 2012-11-06 3826 atomic_read(&sctx->workers_pending) == 0); 2073c4c2e51a93 David Sterba 2017-03-31 3827 sctx->flush_all_writes = false; 12cf93728dfba2 Wang Shilong 2014-02-19 3828 b708ce969af3ce Zhaolei 2015-08-05 3829 scrub_pause_off(fs_info); ff023aac31198e Stefan Behrens 2012-11-06 3830 78ce9fc269af6e Naohiro Aota 2021-02-04 3831 if (sctx->is_dev_replace && 78ce9fc269af6e Naohiro Aota 2021-02-04 3832 !btrfs_finish_block_group_to_copy(dev_replace->srcdev, 78ce9fc269af6e Naohiro Aota 2021-02-04 3833 cache, found_key.offset)) 78ce9fc269af6e Naohiro Aota 2021-02-04 3834 ro_set = 0; 78ce9fc269af6e Naohiro Aota 2021-02-04 3835 78ce9fc269af6e Naohiro Aota 2021-02-04 @3836 done: 3ec17a67cc340b Dan Carpenter 2019-10-31 3837 down_write(&dev_replace->rwsem); 1a1a8b732c7e95 Filipe Manana 2016-05-14 3838 dev_replace->cursor_left = dev_replace->cursor_right; 1a1a8b732c7e95 Filipe Manana 2016-05-14 3839 dev_replace->item_needs_writeback = 1; 3ec17a67cc340b Dan Carpenter 2019-10-31 3840 up_write(&dev_replace->rwsem); 1a1a8b732c7e95 Filipe Manana 2016-05-14 3841 76a8efa171bf6c Zhaolei 2015-11-17 3842 if (ro_set) 2ff7e61e0d30ff Jeff Mahoney 2016-06-22 3843 btrfs_dec_block_group_ro(cache); ff023aac31198e Stefan Behrens 2012-11-06 3844 758f2dfcf8a249 Filipe Manana 2015-11-19 3845 /* 758f2dfcf8a249 Filipe Manana 2015-11-19 3846 * We might have prevented the cleaner kthread from deleting 758f2dfcf8a249 Filipe Manana 2015-11-19 3847 * this block group if it was already unused because we raced 758f2dfcf8a249 Filipe Manana 2015-11-19 3848 * and set it to RO mode first. So add it back to the unused 758f2dfcf8a249 Filipe Manana 2015-11-19 3849 * list, otherwise it might not ever be deleted unless a manual 758f2dfcf8a249 Filipe Manana 2015-11-19 3850 * balance is triggered or it becomes used and unused again. 758f2dfcf8a249 Filipe Manana 2015-11-19 3851 */ 758f2dfcf8a249 Filipe Manana 2015-11-19 3852 spin_lock(&cache->lock); 758f2dfcf8a249 Filipe Manana 2015-11-19 3853 if (!cache->removed && !cache->ro && cache->reserved == 0 && bf38be65f3703d David Sterba 2019-10-23 3854 cache->used == 0) { 758f2dfcf8a249 Filipe Manana 2015-11-19 3855 spin_unlock(&cache->lock); 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3856 if (btrfs_test_opt(fs_info, DISCARD_ASYNC)) 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3857 btrfs_discard_queue_work(&fs_info->discard_ctl, 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3858 cache); 6e80d4f8c422d3 Dennis Zhou 2019-12-13 3859 else 031f24da2c8a7b Qu Wenruo 2018-05-22 3860 btrfs_mark_bg_unused(cache); 758f2dfcf8a249 Filipe Manana 2015-11-19 3861 } else { 758f2dfcf8a249 Filipe Manana 2015-11-19 3862 spin_unlock(&cache->lock); 758f2dfcf8a249 Filipe Manana 2015-11-19 3863 } 195a49eaf655eb Filipe Manana 2021-02-05 3864 skip_unfreeze: 6b7304af62d02d Filipe Manana 2020-05-08 3865 btrfs_unfreeze_block_group(cache); a2de733c78fa7a Arne Jansen 2011-03-08 3866 btrfs_put_block_group(cache); a2de733c78fa7a Arne Jansen 2011-03-08 3867 if (ret) a2de733c78fa7a Arne Jansen 2011-03-08 3868 break; 32934280967d00 Omar Sandoval 2018-08-14 3869 if (sctx->is_dev_replace && af1be4f851db4f Stefan Behrens 2012-11-27 3870 atomic64_read(&dev_replace->num_write_errors) > 0) { ff023aac31198e Stefan Behrens 2012-11-06 3871 ret = -EIO; ff023aac31198e Stefan Behrens 2012-11-06 3872 break; ff023aac31198e Stefan Behrens 2012-11-06 3873 } ff023aac31198e Stefan Behrens 2012-11-06 3874 if (sctx->stat.malloc_errors > 0) { ff023aac31198e Stefan Behrens 2012-11-06 3875 ret = -ENOMEM; ff023aac31198e Stefan Behrens 2012-11-06 3876 break; ff023aac31198e Stefan Behrens 2012-11-06 3877 } ced96edc48ba45 Qu Wenruo 2014-06-19 3878 skip: a2de733c78fa7a Arne Jansen 2011-03-08 3879 key.offset = found_key.offset + length; 712673339a0d08 Chris Mason 2011-05-23 3880 btrfs_release_path(path); a2de733c78fa7a Arne Jansen 2011-03-08 3881 } a2de733c78fa7a Arne Jansen 2011-03-08 3882 a2de733c78fa7a Arne Jansen 2011-03-08 3883 btrfs_free_path(path); 8c51032f978bac Arne Jansen 2011-06-03 3884 55e3a601c81cdc Zhaolei 2015-08-05 3885 return ret; a2de733c78fa7a Arne Jansen 2011-03-08 3886 } a2de733c78fa7a Arne Jansen 2011-03-08 3887 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org [-- Attachment #2: config.gz --] [-- Type: application/gzip, Size: 41778 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-04-20 17:37 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-04-14 12:08 [PATCH] btrfs: zoned: fix unpaired block group unfreeze during device replace fdmanana 2021-04-14 12:34 ` Johannes Thumshirn 2021-04-14 13:06 ` Filipe Manana 2021-04-14 13:05 ` [PATCH v2] " fdmanana 2021-04-20 17:34 ` David Sterba 2021-04-14 16:32 ` [PATCH] " kernel test robot 2021-04-14 16:32 ` kernel test robot
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.