linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 0/5] Fix memory leak on failed cache-writes
@ 2020-02-13 15:57 Johannes Thumshirn
  2020-02-13 15:57 ` [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out Johannes Thumshirn
                   ` (5 more replies)
  0 siblings, 6 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:57 UTC (permalink / raw)
  To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn

Fstests' test case generic/475 reliably leaks the btrfs_io_ctl::pages
allocated in __btrfs_write_out_cache().

The first patch in this series is freeing the pages when we throw away a dirty
block group. The other patches are small things I noticed while hunting down the
problem and are independant of fix.

Changes to v1:
- Move fix to the first position (David)

Johannes Thumshirn (5):
  btrfs: free allocated pages on failed cache write-out
  btrfs: use inode from io_ctl in io_ctl_prepare_pages
  btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
  btrfs: use standard debug config option to enable free-space-cache
    debug prints
  btrfs: simplify error handling in __btrfs_write_out_cache()

 fs/btrfs/disk-io.c          |  6 +++++
 fs/btrfs/free-space-cache.c | 44 ++++++++++++++++++++-----------------
 fs/btrfs/free-space-cache.h |  1 +
 3 files changed, 31 insertions(+), 20 deletions(-)

-- 
2.24.1


^ permalink raw reply	[flat|nested] 11+ messages in thread
* [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out
  2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
@ 2020-02-13 15:57 ` Johannes Thumshirn
  2020-02-13 15:58 ` [PATCH v2 2/5] btrfs: use inode from io_ctl in io_ctl_prepare_pages Johannes Thumshirn
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:57 UTC (permalink / raw)
  To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn

When we fail to write out a dirty block group, we leak the pages allocated
for a block-group's io_ctl. This can be seen with generic/475 and kmemleak
turned on:

unreferenced object 0xffff8882249c9000 (size 128):
  comm "fsstress", pid 1791, jiffies 4294902054 (age 32.100s)
  hex dump (first 32 bytes):
    80 0e 42 08 00 ea ff ff 00 0d 42 08 00 ea ff ff  ..B.......B.....
    00 eb 0e 08 00 ea ff ff 00 e8 0e 08 00 ea ff ff  ................
  backtrace:
    [<00000000cd20c449>] io_ctl_init+0xa2/0x110 [btrfs]
    [<00000000281944cc>] __btrfs_write_out_cache+0x71/0x410 [btrfs]
    [<000000005d518c07>] btrfs_write_out_cache+0x82/0xd0 [btrfs]
    [<000000002bb2675c>] btrfs_start_dirty_block_groups+0x1f6/0x440 [btrfs]
    [<000000004f955ad0>] btrfs_commit_transaction+0xb7/0x970 [btrfs]
    [<00000000a69c8761>] btrfs_sync_file+0x28f/0x390 [btrfs]
    [<00000000fa939e06>] do_fsync+0x33/0x70
    [<000000002ff0388b>] __x64_sys_fdatasync+0xe/0x20
    [<00000000fdbf32d4>] do_syscall_64+0x43/0x120
    [<00000000b782d265>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

When cleaning up a block group release all allocated pages. As the data in
the pages is already lost, we can at least free the memory occupied by
them.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 fs/btrfs/disk-io.c          | 6 ++++++
 fs/btrfs/free-space-cache.c | 6 ++++++
 fs/btrfs/free-space-cache.h | 1 +
 3 files changed, 13 insertions(+)

diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 018681ec159b..b79c194b1126 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -4460,6 +4460,12 @@ static void btrfs_cleanup_bg_io(struct btrfs_block_group *cache)
 {
 	struct inode *inode;
 
+	/*
+	 * If we end up here, we want the pages to be already released
+	 * otherwise we'll leak them.
+	 */
+	btrfs_drop_dirty_io_ctl(&cache->io_ctl);
+
 	inode = cache->io_ctl.inode;
 	if (inode) {
 		invalidate_inode_pages2(inode->i_mapping);
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 0598fd3c6e3f..3c7660b04a81 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -371,6 +371,12 @@ static void io_ctl_drop_pages(struct btrfs_io_ctl *io_ctl)
 	}
 }
 
+void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl)
+{
+	io_ctl_drop_pages(io_ctl);
+	io_ctl_free(io_ctl);
+}
+
 static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, struct inode *inode,
 				int uptodate)
 {
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 2e0a8077aa74..cbe25c31041d 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -147,6 +147,7 @@ int btrfs_trim_block_group_extents(struct btrfs_block_group *block_group,
 int btrfs_trim_block_group_bitmaps(struct btrfs_block_group *block_group,
 				   u64 *trimmed, u64 start, u64 end, u64 minlen,
 				   u64 maxlen, bool async);
+void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl);
 
 /* Support functions for running our sanity tests */
 #ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
-- 
2.24.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread
* [PATCH v2 2/5] btrfs: use inode from io_ctl in io_ctl_prepare_pages
  2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
  2020-02-13 15:57 ` [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
  2020-02-13 15:58 ` [PATCH v2 3/5] btrfs: make the uptodate argument of io_ctl_add_pages() boolean Johannes Thumshirn
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
  To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn

io_ctl_prepare_pages() get's a 'struct btrfs_io_ctl' as well as a 'struct
inode', but btrfs_io_ctl::inode points to the same struct inode as this is
assgined in io_ctl_init().

Use the inode form io_ctl to reduce the arguments of
io_ctl_prepare_pages().

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 fs/btrfs/free-space-cache.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 3c7660b04a81..657a969a93ad 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -377,10 +377,10 @@ void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl)
 	io_ctl_free(io_ctl);
 }
 
-static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, struct inode *inode,
-				int uptodate)
+static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, int uptodate)
 {
 	struct page *page;
+	struct inode *inode = io_ctl->inode;
 	gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
 	int i;
 
@@ -738,7 +738,7 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
 
 	readahead_cache(inode);
 
-	ret = io_ctl_prepare_pages(&io_ctl, inode, 1);
+	ret = io_ctl_prepare_pages(&io_ctl, 1);
 	if (ret)
 		goto out;
 
@@ -1297,7 +1297,7 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
 	}
 
 	/* Lock all pages first so we can lock the extent safely. */
-	ret = io_ctl_prepare_pages(io_ctl, inode, 0);
+	ret = io_ctl_prepare_pages(io_ctl, 0);
 	if (ret)
 		goto out_unlock;
 
-- 
2.24.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread
* [PATCH v2 3/5] btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
  2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
  2020-02-13 15:57 ` [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out Johannes Thumshirn
  2020-02-13 15:58 ` [PATCH v2 2/5] btrfs: use inode from io_ctl in io_ctl_prepare_pages Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
  2020-02-13 15:58 ` [PATCH v2 4/5] btrfs: use standard debug config option to enable free-space-cache debug prints Johannes Thumshirn
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
  To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn

Make the uptodate argument of io_ctl_add_pages() boolean.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 fs/btrfs/free-space-cache.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 657a969a93ad..8da592eaf6f8 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -377,7 +377,7 @@ void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl)
 	io_ctl_free(io_ctl);
 }
 
-static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, int uptodate)
+static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, bool uptodate)
 {
 	struct page *page;
 	struct inode *inode = io_ctl->inode;
@@ -738,7 +738,7 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
 
 	readahead_cache(inode);
 
-	ret = io_ctl_prepare_pages(&io_ctl, 1);
+	ret = io_ctl_prepare_pages(&io_ctl, true);
 	if (ret)
 		goto out;
 
@@ -1297,7 +1297,7 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
 	}
 
 	/* Lock all pages first so we can lock the extent safely. */
-	ret = io_ctl_prepare_pages(io_ctl, 0);
+	ret = io_ctl_prepare_pages(io_ctl, false);
 	if (ret)
 		goto out_unlock;
 
-- 
2.24.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread
* [PATCH v2 4/5] btrfs: use standard debug config option to enable free-space-cache debug prints
  2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
                   ` (2 preceding siblings ...)
  2020-02-13 15:58 ` [PATCH v2 3/5] btrfs: make the uptodate argument of io_ctl_add_pages() boolean Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
  2020-02-13 15:58 ` [PATCH v2 5/5] btrfs: simplify error handling in __btrfs_write_out_cache() Johannes Thumshirn
  2020-02-18 16:50 ` [PATCH v2 0/5] Fix memory leak on failed cache-writes David Sterba
  5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
  To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn

free-space-cache.c has it's own set of DEBUG ifdefs which need to be
turned on instead of the global CONFIG_BTRFS_DEBUG to print debug messages
about failed block-group writes.

Switch this over to CONFIG_BTRFS_DEBUG so we always see these messages
when running a debug kernel.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 fs/btrfs/free-space-cache.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 8da592eaf6f8..41e138f2ae12 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -1196,7 +1196,7 @@ static int __btrfs_wait_cache_io(struct btrfs_root *root,
 		invalidate_inode_pages2(inode->i_mapping);
 		BTRFS_I(inode)->generation = 0;
 		if (block_group) {
-#ifdef DEBUG
+#ifdef CONFIG_BTRFS_DEBUG
 			btrfs_err(root->fs_info,
 				  "failed to write free space cache for block group %llu",
 				  block_group->start);
@@ -1422,7 +1422,7 @@ int btrfs_write_out_cache(struct btrfs_trans_handle *trans,
 	ret = __btrfs_write_out_cache(fs_info->tree_root, inode, ctl,
 				block_group, &block_group->io_ctl, trans);
 	if (ret) {
-#ifdef DEBUG
+#ifdef CONFIG_BTRFS_DEBUG
 		btrfs_err(fs_info,
 			  "failed to write free space cache for block group %llu",
 			  block_group->start);
@@ -4042,7 +4042,7 @@ int btrfs_write_out_ino_cache(struct btrfs_root *root,
 		if (release_metadata)
 			btrfs_delalloc_release_metadata(BTRFS_I(inode),
 					inode->i_size, true);
-#ifdef DEBUG
+#ifdef CONFIG_BTRFS_DEBUG
 		btrfs_err(fs_info,
 			  "failed to write free ino cache for root %llu",
 			  root->root_key.objectid);
-- 
2.24.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread
* [PATCH v2 5/5] btrfs: simplify error handling in __btrfs_write_out_cache()
  2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
                   ` (3 preceding siblings ...)
  2020-02-13 15:58 ` [PATCH v2 4/5] btrfs: use standard debug config option to enable free-space-cache debug prints Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
  2020-02-18 16:50 ` [PATCH v2 0/5] Fix memory leak on failed cache-writes David Sterba
  5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
  To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn

The error cleanup gotos in __btrfs_write_out_cache() needlessly jump back
making the code less readable then needed.

Flatten out the labels so no back-jump is necessary and the read flow is
uninterrupted.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 fs/btrfs/free-space-cache.c | 24 +++++++++++-------------
 1 file changed, 11 insertions(+), 13 deletions(-)

diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 41e138f2ae12..c7ba2b393b33 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -1372,18 +1372,6 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
 
 	return 0;
 
-out:
-	io_ctl->inode = NULL;
-	io_ctl_free(io_ctl);
-	if (ret) {
-		invalidate_inode_pages2(inode->i_mapping);
-		BTRFS_I(inode)->generation = 0;
-	}
-	btrfs_update_inode(trans, root, inode);
-	if (must_iput)
-		iput(inode);
-	return ret;
-
 out_nospc_locked:
 	cleanup_bitmap_list(&bitmap_list);
 	spin_unlock(&ctl->tree_lock);
@@ -1396,7 +1384,17 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
 	if (block_group && (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
 		up_write(&block_group->data_rwsem);
 
-	goto out;
+out:
+	io_ctl->inode = NULL;
+	io_ctl_free(io_ctl);
+	if (ret) {
+		invalidate_inode_pages2(inode->i_mapping);
+		BTRFS_I(inode)->generation = 0;
+	}
+	btrfs_update_inode(trans, root, inode);
+	if (must_iput)
+		iput(inode);
+	return ret;
 }
 
 int btrfs_write_out_cache(struct btrfs_trans_handle *trans,
-- 
2.24.1


^ permalink raw reply related	[flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
  2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
                   ` (4 preceding siblings ...)
  2020-02-13 15:58 ` [PATCH v2 5/5] btrfs: simplify error handling in __btrfs_write_out_cache() Johannes Thumshirn
@ 2020-02-18 16:50 ` David Sterba
  2020-02-18 16:54   ` Johannes Thumshirn
  5 siblings, 1 reply; 11+ messages in thread
From: David Sterba @ 2020-02-18 16:50 UTC (permalink / raw)
  To: Johannes Thumshirn; +Cc: David Sterba, linux-btrfs @ vger . kernel . org

On Fri, Feb 14, 2020 at 12:57:58AM +0900, Johannes Thumshirn wrote:
> Fstests' test case generic/475 reliably leaks the btrfs_io_ctl::pages
> allocated in __btrfs_write_out_cache().
> 
> The first patch in this series is freeing the pages when we throw away a dirty
> block group. The other patches are small things I noticed while hunting down the
> problem and are independant of fix.
> 
> Changes to v1:
> - Move fix to the first position (David)
> 
> Johannes Thumshirn (5):
>   btrfs: free allocated pages on failed cache write-out
>   btrfs: use inode from io_ctl in io_ctl_prepare_pages
>   btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
>   btrfs: use standard debug config option to enable free-space-cache
>     debug prints
>   btrfs: simplify error handling in __btrfs_write_out_cache()

I've seen some weird test failures and this patchset was in the tested
branch (either for-next or standalone). I need to retest misc-next again
to have a clean baseline so I can see the diff.

^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
  2020-02-18 16:50 ` [PATCH v2 0/5] Fix memory leak on failed cache-writes David Sterba
@ 2020-02-18 16:54   ` Johannes Thumshirn
  2020-02-24 17:49     ` David Sterba
  0 siblings, 1 reply; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-18 16:54 UTC (permalink / raw)
  To: dsterba; +Cc: linux-btrfs @ vger . kernel . org

On 18/02/2020 17:50, David Sterba wrote:
> On Fri, Feb 14, 2020 at 12:57:58AM +0900, Johannes Thumshirn wrote:
>> Fstests' test case generic/475 reliably leaks the btrfs_io_ctl::pages
>> allocated in __btrfs_write_out_cache().
>>
>> The first patch in this series is freeing the pages when we throw away a dirty
>> block group. The other patches are small things I noticed while hunting down the
>> problem and are independant of fix.
>>
>> Changes to v1:
>> - Move fix to the first position (David)
>>
>> Johannes Thumshirn (5):
>>    btrfs: free allocated pages on failed cache write-out
>>    btrfs: use inode from io_ctl in io_ctl_prepare_pages
>>    btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
>>    btrfs: use standard debug config option to enable free-space-cache
>>      debug prints
>>    btrfs: simplify error handling in __btrfs_write_out_cache()
> 
> I've seen some weird test failures and this patchset was in the tested
> branch (either for-next or standalone). I need to retest misc-next again
> to have a clean baseline so I can see the diff.
> 

Interesting, can you forward me the failures?


^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
  2020-02-18 16:54   ` Johannes Thumshirn
@ 2020-02-24 17:49     ` David Sterba
  2020-02-25  0:23       ` Johannes Thumshirn
  0 siblings, 1 reply; 11+ messages in thread
From: David Sterba @ 2020-02-24 17:49 UTC (permalink / raw)
  To: Johannes Thumshirn; +Cc: dsterba, linux-btrfs @ vger . kernel . org

On Tue, Feb 18, 2020 at 04:54:18PM +0000, Johannes Thumshirn wrote:
> On 18/02/2020 17:50, David Sterba wrote:
> > On Fri, Feb 14, 2020 at 12:57:58AM +0900, Johannes Thumshirn wrote:
> >> Fstests' test case generic/475 reliably leaks the btrfs_io_ctl::pages
> >> allocated in __btrfs_write_out_cache().
> >>
> >> The first patch in this series is freeing the pages when we throw away a dirty
> >> block group. The other patches are small things I noticed while hunting down the
> >> problem and are independant of fix.
> >>
> >> Changes to v1:
> >> - Move fix to the first position (David)
> >>
> >> Johannes Thumshirn (5):
> >>    btrfs: free allocated pages on failed cache write-out
> >>    btrfs: use inode from io_ctl in io_ctl_prepare_pages
> >>    btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
> >>    btrfs: use standard debug config option to enable free-space-cache
> >>      debug prints
> >>    btrfs: simplify error handling in __btrfs_write_out_cache()
> > 
> > I've seen some weird test failures and this patchset was in the tested
> > branch (either for-next or standalone). I need to retest misc-next again
> > to have a clean baseline so I can see the diff.
> > 
> 
> Interesting, can you forward me the failures?

So it was test btrfs/125 but I can't say for sure which patchset causes
it, there are several reports but unlikely to be caused by yours.

btrfs/125               [14:07:58][ 9038.483946] run fstests btrfs/125 at 2020-02-18 14:07:58
[ 9038.894197] BTRFS info (device vda): disk space caching is enabled
[ 9038.897644] BTRFS info (device vda): has skinny extents
[ 9039.553603] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (29881)
[ 9039.557424] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 5 /dev/vdc scanned by mkfs.btrfs (29881)
[ 9039.561587] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 3 transid 5 /dev/vdd scanned by mkfs.btrfs (29881)
[ 9039.581158] BTRFS info (device vdb): turning on sync discard
[ 9039.584130] BTRFS info (device vdb): disk space caching is enabled
[ 9039.587021] BTRFS info (device vdb): has skinny extents
[ 9039.589100] BTRFS info (device vdb): flagging fs with big metadata feature
[ 9039.595079] BTRFS info (device vdb): checking UUID tree
[ 9039.706233] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 7 /dev/vdc scanned by mount (29923)
[ 9039.711721] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 7 /dev/vdb scanned by mount (29923)
[ 9039.715946] BTRFS info (device vdb): allowing degraded mounts
[ 9039.717785] BTRFS info (device vdb): disk space caching is enabled
[ 9039.719914] BTRFS info (device vdb): has skinny extents
[ 9039.723260] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
[ 9039.726785] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
[ 9040.189984] BTRFS: device fsid 4d1f43bd-2b49-4905-a219-25dc10f1b5fe devid 1 transid 249 /dev/vda scanned by btrfs (29951)
[ 9040.211252] BTRFS info (device vdb): turning on sync discard
[ 9040.223781] BTRFS info (device vdb): disk space caching is enabled
[ 9040.225777] BTRFS info (device vdb): has skinny extents
[ 9050.252423] BTRFS info (device vdb): balance: start -d -m -s
[ 9050.256160] BTRFS info (device vdb): relocating block group 217710592 flags data|raid5
[ 9050.389701] btrfs_print_data_csum_error: 6540 callbacks suppressed
[ 9050.389706] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5357568 csum 0xb5d704296aa5f2d5 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.398381] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5361664 csum 0xf2f7aa2e723d427e expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.398899] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3424256 csum 0x15e7d27c0f4b4f20 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.403918] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5365760 csum 0x25464d4ee302ec50 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.408525] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3428352 csum 0xcd1278f31a795b3d expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.412556] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5369856 csum 0x045d00c41342cf48 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.417133] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3432448 csum 0xab3b271dc78f0131 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.425859] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3436544 csum 0xa1e70b1d4d61d53f expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.430432] repair_io_failure: 9 callbacks suppressed
[ 9050.430435] BTRFS info (device vdb): read error corrected: ino 257 off 3424256 (dev /dev/vdd sector 196512)
[ 9050.430464] BTRFS info (device vdb): read error corrected: ino 257 off 3428352 (dev /dev/vdd sector 196520)
[ 9050.430473] BTRFS info (device vdb): read error corrected: ino 257 off 5361664 (dev /dev/vdd sector 198376)
[ 9050.430681] BTRFS info (device vdb): read error corrected: ino 257 off 3432448 (dev /dev/vdd sector 196528)
[ 9050.430762] BTRFS info (device vdb): read error corrected: ino 257 off 5357568 (dev /dev/vdd sector 198368)
[ 9050.430805] BTRFS info (device vdb): read error corrected: ino 257 off 5369856 (dev /dev/vdd sector 198392)
[ 9050.430853] BTRFS info (device vdb): read error corrected: ino 257 off 3436544 (dev /dev/vdd sector 196536)
[ 9050.432557] BTRFS info (device vdb): read error corrected: ino 257 off 5365760 (dev /dev/vdd sector 198384)
[ 9050.469621] BTRFS warning (device vdb): csum failed root -9 ino 257 off 13959168 csum 0x774317b489e76165 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.476527] BTRFS info (device vdb): read error corrected: ino 257 off 13959168 (dev /dev/vdd sector 206720)
[ 9050.501930] BTRFS warning (device vdb): csum failed root -9 ino 257 off 16351232 csum 0xa735bf811aa10b77 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.512917] BTRFS info (device vdb): read error corrected: ino 257 off 17608704 (dev /dev/vdd sector 210392)
[ 9050.708681] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9050.708684] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
[ 9050.708699] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
[ 9050.708700] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
[ 9050.713287] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9050.727830] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9050.727849] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
[ 9050.727854] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
[ 9050.727876] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
[ 9050.730601] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9053.133149] BTRFS info (device vdb): balance: ended with status: -5
[failed, exit status 1][ 9053.200709] BTRFS info (device vdb): clearing incompat feature flag for RAID56 (0x80)
 [14:08:12]- output mismatch (see /tmp/fstests/results//btrfs/125.out.bad)
    --- tests/btrfs/125.out     2018-04-12 16:57:00.616225550 +0000
    +++ /tmp/fstests/results//btrfs/125.out.bad 2020-02-18 14:08:12.856000000 +0000
    @@ -3,5 +3,5 @@
     Write data with degraded mount
     
     Mount normal and balance
    -
    -Mount degraded but with other dev
    +failed: '/sbin/btrfs balance start /tmp/scratch'
    +(see /tmp/fstests/results//btrfs/125.full for details)
    ...
    (Run 'diff -u /tmp/fstests/tests/btrfs/125.out /tmp/fstests/results//btrfs/125.out.bad'  to see the entire diff)
---

^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
  2020-02-24 17:49     ` David Sterba
@ 2020-02-25  0:23       ` Johannes Thumshirn
  2020-02-25  0:56         ` Johannes Thumshirn
  0 siblings, 1 reply; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-25  0:23 UTC (permalink / raw)
  To: dsterba; +Cc: linux-btrfs @ vger . kernel . org

On 24/02/2020 09:50, David Sterba wrote:
[...]
> So it was test btrfs/125 but I can't say for sure which patchset causes
> it, there are several reports but unlikely to be caused by yours.
> 
> btrfs/125               [14:07:58][ 9038.483946] run fstests btrfs/125 at 2020-02-18 14:07:58
> [ 9038.894197] BTRFS info (device vda): disk space caching is enabled
> [ 9038.897644] BTRFS info (device vda): has skinny extents
> [ 9039.553603] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (29881)
> [ 9039.557424] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 5 /dev/vdc scanned by mkfs.btrfs (29881)
> [ 9039.561587] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 3 transid 5 /dev/vdd scanned by mkfs.btrfs (29881)
> [ 9039.581158] BTRFS info (device vdb): turning on sync discard
> [ 9039.584130] BTRFS info (device vdb): disk space caching is enabled
> [ 9039.587021] BTRFS info (device vdb): has skinny extents
> [ 9039.589100] BTRFS info (device vdb): flagging fs with big metadata feature
> [ 9039.595079] BTRFS info (device vdb): checking UUID tree
> [ 9039.706233] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 7 /dev/vdc scanned by mount (29923)
> [ 9039.711721] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 7 /dev/vdb scanned by mount (29923)
> [ 9039.715946] BTRFS info (device vdb): allowing degraded mounts
> [ 9039.717785] BTRFS info (device vdb): disk space caching is enabled
> [ 9039.719914] BTRFS info (device vdb): has skinny extents
> [ 9039.723260] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
> [ 9039.726785] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
> [ 9040.189984] BTRFS: device fsid 4d1f43bd-2b49-4905-a219-25dc10f1b5fe devid 1 transid 249 /dev/vda scanned by btrfs (29951)
> [ 9040.211252] BTRFS info (device vdb): turning on sync discard
> [ 9040.223781] BTRFS info (device vdb): disk space caching is enabled
> [ 9040.225777] BTRFS info (device vdb): has skinny extents
> [ 9050.252423] BTRFS info (device vdb): balance: start -d -m -s
> [ 9050.256160] BTRFS info (device vdb): relocating block group 217710592 flags data|raid5
> [ 9050.389701] btrfs_print_data_csum_error: 6540 callbacks suppressed
> [ 9050.389706] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5357568 csum 0xb5d704296aa5f2d5 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.398381] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5361664 csum 0xf2f7aa2e723d427e expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.398899] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3424256 csum 0x15e7d27c0f4b4f20 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.403918] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5365760 csum 0x25464d4ee302ec50 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.408525] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3428352 csum 0xcd1278f31a795b3d expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.412556] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5369856 csum 0x045d00c41342cf48 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.417133] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3432448 csum 0xab3b271dc78f0131 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.425859] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3436544 csum 0xa1e70b1d4d61d53f expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.430432] repair_io_failure: 9 callbacks suppressed
> [ 9050.430435] BTRFS info (device vdb): read error corrected: ino 257 off 3424256 (dev /dev/vdd sector 196512)
> [ 9050.430464] BTRFS info (device vdb): read error corrected: ino 257 off 3428352 (dev /dev/vdd sector 196520)
> [ 9050.430473] BTRFS info (device vdb): read error corrected: ino 257 off 5361664 (dev /dev/vdd sector 198376)
> [ 9050.430681] BTRFS info (device vdb): read error corrected: ino 257 off 3432448 (dev /dev/vdd sector 196528)
> [ 9050.430762] BTRFS info (device vdb): read error corrected: ino 257 off 5357568 (dev /dev/vdd sector 198368)
> [ 9050.430805] BTRFS info (device vdb): read error corrected: ino 257 off 5369856 (dev /dev/vdd sector 198392)
> [ 9050.430853] BTRFS info (device vdb): read error corrected: ino 257 off 3436544 (dev /dev/vdd sector 196536)
> [ 9050.432557] BTRFS info (device vdb): read error corrected: ino 257 off 5365760 (dev /dev/vdd sector 198384)
> [ 9050.469621] BTRFS warning (device vdb): csum failed root -9 ino 257 off 13959168 csum 0x774317b489e76165 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.476527] BTRFS info (device vdb): read error corrected: ino 257 off 13959168 (dev /dev/vdd sector 206720)
> [ 9050.501930] BTRFS warning (device vdb): csum failed root -9 ino 257 off 16351232 csum 0xa735bf811aa10b77 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.512917] BTRFS info (device vdb): read error corrected: ino 257 off 17608704 (dev /dev/vdd sector 210392)
> [ 9050.708681] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9050.708684] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
> [ 9050.708699] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
> [ 9050.708700] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
> [ 9050.713287] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9050.727830] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9050.727849] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
> [ 9050.727854] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
> [ 9050.727876] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
> [ 9050.730601] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9053.133149] BTRFS info (device vdb): balance: ended with status: -5
> [failed, exit status 1][ 9053.200709] BTRFS info (device vdb): clearing incompat feature flag for RAID56 (0x80)
>   [14:08:12]- output mismatch (see /tmp/fstests/results//btrfs/125.out.bad)
>      --- tests/btrfs/125.out     2018-04-12 16:57:00.616225550 +0000
>      +++ /tmp/fstests/results//btrfs/125.out.bad 2020-02-18 14:08:12.856000000 +0000
>      @@ -3,5 +3,5 @@
>       Write data with degraded mount
>       
>       Mount normal and balance
>      -
>      -Mount degraded but with other dev
>      +failed: '/sbin/btrfs balance start /tmp/scratch'
>      +(see /tmp/fstests/results//btrfs/125.full for details)
>      ...
>      (Run 'diff -u /tmp/fstests/tests/btrfs/125.out /tmp/fstests/results//btrfs/125.out.bad'  to see the entire diff)

OK it need some time (up to approx 150 runs of btrfs/125) to resproduce 
this failure but I can re-create it.

I've seen it once without the patchset applied as well though (a.k.a 
HEAD == 480be04e7fdcddfed86fd59bb668a134b6d7393e).

I have to verify that though not that I'm seeing ghosts.

^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
  2020-02-25  0:23       ` Johannes Thumshirn
@ 2020-02-25  0:56         ` Johannes Thumshirn
  0 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-25  0:56 UTC (permalink / raw)
  To: dsterba; +Cc: linux-btrfs @ vger . kernel . org

On 24/02/2020 16:23, Johannes Thumshirn wrote:
[...]
> OK it need some time (up to approx 150 runs of btrfs/125) to resproduce
> this failure but I can re-create it.
> 
> I've seen it once without the patchset applied as well though (a.k.a
> HEAD == 480be04e7fdcddfed86fd59bb668a134b6d7393e).
> 
> I have to verify that though not that I'm seeing ghosts.
> 

OK, so I've hit it again with the above commit as HEAD, so I don't think 
these patches participate in the error.

I'll try to figure out which patch does though.

Byte,
	Johannes

^ permalink raw reply	[flat|nested] 11+ messages in thread
end of thread, other threads:[~2020-02-25  0:56 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
2020-02-13 15:57 ` [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 2/5] btrfs: use inode from io_ctl in io_ctl_prepare_pages Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 3/5] btrfs: make the uptodate argument of io_ctl_add_pages() boolean Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 4/5] btrfs: use standard debug config option to enable free-space-cache debug prints Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 5/5] btrfs: simplify error handling in __btrfs_write_out_cache() Johannes Thumshirn
2020-02-18 16:50 ` [PATCH v2 0/5] Fix memory leak on failed cache-writes David Sterba
2020-02-18 16:54   ` Johannes Thumshirn
2020-02-24 17:49     ` David Sterba
2020-02-25  0:23       ` Johannes Thumshirn
2020-02-25  0:56         ` Johannes Thumshirn

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).