* [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out
2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
@ 2020-02-13 15:57 ` Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 2/5] btrfs: use inode from io_ctl in io_ctl_prepare_pages Johannes Thumshirn
` (4 subsequent siblings)
5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:57 UTC (permalink / raw)
To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn
When we fail to write out a dirty block group, we leak the pages allocated
for a block-group's io_ctl. This can be seen with generic/475 and kmemleak
turned on:
unreferenced object 0xffff8882249c9000 (size 128):
comm "fsstress", pid 1791, jiffies 4294902054 (age 32.100s)
hex dump (first 32 bytes):
80 0e 42 08 00 ea ff ff 00 0d 42 08 00 ea ff ff ..B.......B.....
00 eb 0e 08 00 ea ff ff 00 e8 0e 08 00 ea ff ff ................
backtrace:
[<00000000cd20c449>] io_ctl_init+0xa2/0x110 [btrfs]
[<00000000281944cc>] __btrfs_write_out_cache+0x71/0x410 [btrfs]
[<000000005d518c07>] btrfs_write_out_cache+0x82/0xd0 [btrfs]
[<000000002bb2675c>] btrfs_start_dirty_block_groups+0x1f6/0x440 [btrfs]
[<000000004f955ad0>] btrfs_commit_transaction+0xb7/0x970 [btrfs]
[<00000000a69c8761>] btrfs_sync_file+0x28f/0x390 [btrfs]
[<00000000fa939e06>] do_fsync+0x33/0x70
[<000000002ff0388b>] __x64_sys_fdatasync+0xe/0x20
[<00000000fdbf32d4>] do_syscall_64+0x43/0x120
[<00000000b782d265>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
When cleaning up a block group release all allocated pages. As the data in
the pages is already lost, we can at least free the memory occupied by
them.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
fs/btrfs/disk-io.c | 6 ++++++
fs/btrfs/free-space-cache.c | 6 ++++++
fs/btrfs/free-space-cache.h | 1 +
3 files changed, 13 insertions(+)
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 018681ec159b..b79c194b1126 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -4460,6 +4460,12 @@ static void btrfs_cleanup_bg_io(struct btrfs_block_group *cache)
{
struct inode *inode;
+ /*
+ * If we end up here, we want the pages to be already released
+ * otherwise we'll leak them.
+ */
+ btrfs_drop_dirty_io_ctl(&cache->io_ctl);
+
inode = cache->io_ctl.inode;
if (inode) {
invalidate_inode_pages2(inode->i_mapping);
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 0598fd3c6e3f..3c7660b04a81 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -371,6 +371,12 @@ static void io_ctl_drop_pages(struct btrfs_io_ctl *io_ctl)
}
}
+void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl)
+{
+ io_ctl_drop_pages(io_ctl);
+ io_ctl_free(io_ctl);
+}
+
static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, struct inode *inode,
int uptodate)
{
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 2e0a8077aa74..cbe25c31041d 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -147,6 +147,7 @@ int btrfs_trim_block_group_extents(struct btrfs_block_group *block_group,
int btrfs_trim_block_group_bitmaps(struct btrfs_block_group *block_group,
u64 *trimmed, u64 start, u64 end, u64 minlen,
u64 maxlen, bool async);
+void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl);
/* Support functions for running our sanity tests */
#ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
--
2.24.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v2 2/5] btrfs: use inode from io_ctl in io_ctl_prepare_pages
2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
2020-02-13 15:57 ` [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 3/5] btrfs: make the uptodate argument of io_ctl_add_pages() boolean Johannes Thumshirn
` (3 subsequent siblings)
5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn
io_ctl_prepare_pages() get's a 'struct btrfs_io_ctl' as well as a 'struct
inode', but btrfs_io_ctl::inode points to the same struct inode as this is
assgined in io_ctl_init().
Use the inode form io_ctl to reduce the arguments of
io_ctl_prepare_pages().
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
fs/btrfs/free-space-cache.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 3c7660b04a81..657a969a93ad 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -377,10 +377,10 @@ void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl)
io_ctl_free(io_ctl);
}
-static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, struct inode *inode,
- int uptodate)
+static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, int uptodate)
{
struct page *page;
+ struct inode *inode = io_ctl->inode;
gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
int i;
@@ -738,7 +738,7 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
readahead_cache(inode);
- ret = io_ctl_prepare_pages(&io_ctl, inode, 1);
+ ret = io_ctl_prepare_pages(&io_ctl, 1);
if (ret)
goto out;
@@ -1297,7 +1297,7 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
}
/* Lock all pages first so we can lock the extent safely. */
- ret = io_ctl_prepare_pages(io_ctl, inode, 0);
+ ret = io_ctl_prepare_pages(io_ctl, 0);
if (ret)
goto out_unlock;
--
2.24.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v2 3/5] btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
2020-02-13 15:57 ` [PATCH v2 1/5] btrfs: free allocated pages on failed cache write-out Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 2/5] btrfs: use inode from io_ctl in io_ctl_prepare_pages Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 4/5] btrfs: use standard debug config option to enable free-space-cache debug prints Johannes Thumshirn
` (2 subsequent siblings)
5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn
Make the uptodate argument of io_ctl_add_pages() boolean.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
fs/btrfs/free-space-cache.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 657a969a93ad..8da592eaf6f8 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -377,7 +377,7 @@ void btrfs_drop_dirty_io_ctl(struct btrfs_io_ctl *io_ctl)
io_ctl_free(io_ctl);
}
-static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, int uptodate)
+static int io_ctl_prepare_pages(struct btrfs_io_ctl *io_ctl, bool uptodate)
{
struct page *page;
struct inode *inode = io_ctl->inode;
@@ -738,7 +738,7 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
readahead_cache(inode);
- ret = io_ctl_prepare_pages(&io_ctl, 1);
+ ret = io_ctl_prepare_pages(&io_ctl, true);
if (ret)
goto out;
@@ -1297,7 +1297,7 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
}
/* Lock all pages first so we can lock the extent safely. */
- ret = io_ctl_prepare_pages(io_ctl, 0);
+ ret = io_ctl_prepare_pages(io_ctl, false);
if (ret)
goto out_unlock;
--
2.24.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v2 4/5] btrfs: use standard debug config option to enable free-space-cache debug prints
2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
` (2 preceding siblings ...)
2020-02-13 15:58 ` [PATCH v2 3/5] btrfs: make the uptodate argument of io_ctl_add_pages() boolean Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
2020-02-13 15:58 ` [PATCH v2 5/5] btrfs: simplify error handling in __btrfs_write_out_cache() Johannes Thumshirn
2020-02-18 16:50 ` [PATCH v2 0/5] Fix memory leak on failed cache-writes David Sterba
5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn
free-space-cache.c has it's own set of DEBUG ifdefs which need to be
turned on instead of the global CONFIG_BTRFS_DEBUG to print debug messages
about failed block-group writes.
Switch this over to CONFIG_BTRFS_DEBUG so we always see these messages
when running a debug kernel.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
fs/btrfs/free-space-cache.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 8da592eaf6f8..41e138f2ae12 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -1196,7 +1196,7 @@ static int __btrfs_wait_cache_io(struct btrfs_root *root,
invalidate_inode_pages2(inode->i_mapping);
BTRFS_I(inode)->generation = 0;
if (block_group) {
-#ifdef DEBUG
+#ifdef CONFIG_BTRFS_DEBUG
btrfs_err(root->fs_info,
"failed to write free space cache for block group %llu",
block_group->start);
@@ -1422,7 +1422,7 @@ int btrfs_write_out_cache(struct btrfs_trans_handle *trans,
ret = __btrfs_write_out_cache(fs_info->tree_root, inode, ctl,
block_group, &block_group->io_ctl, trans);
if (ret) {
-#ifdef DEBUG
+#ifdef CONFIG_BTRFS_DEBUG
btrfs_err(fs_info,
"failed to write free space cache for block group %llu",
block_group->start);
@@ -4042,7 +4042,7 @@ int btrfs_write_out_ino_cache(struct btrfs_root *root,
if (release_metadata)
btrfs_delalloc_release_metadata(BTRFS_I(inode),
inode->i_size, true);
-#ifdef DEBUG
+#ifdef CONFIG_BTRFS_DEBUG
btrfs_err(fs_info,
"failed to write free ino cache for root %llu",
root->root_key.objectid);
--
2.24.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v2 5/5] btrfs: simplify error handling in __btrfs_write_out_cache()
2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
` (3 preceding siblings ...)
2020-02-13 15:58 ` [PATCH v2 4/5] btrfs: use standard debug config option to enable free-space-cache debug prints Johannes Thumshirn
@ 2020-02-13 15:58 ` Johannes Thumshirn
2020-02-18 16:50 ` [PATCH v2 0/5] Fix memory leak on failed cache-writes David Sterba
5 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-13 15:58 UTC (permalink / raw)
To: David Sterba; +Cc: linux-btrfs @ vger . kernel . org, Johannes Thumshirn
The error cleanup gotos in __btrfs_write_out_cache() needlessly jump back
making the code less readable then needed.
Flatten out the labels so no back-jump is necessary and the read flow is
uninterrupted.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
fs/btrfs/free-space-cache.c | 24 +++++++++++-------------
1 file changed, 11 insertions(+), 13 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 41e138f2ae12..c7ba2b393b33 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -1372,18 +1372,6 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
return 0;
-out:
- io_ctl->inode = NULL;
- io_ctl_free(io_ctl);
- if (ret) {
- invalidate_inode_pages2(inode->i_mapping);
- BTRFS_I(inode)->generation = 0;
- }
- btrfs_update_inode(trans, root, inode);
- if (must_iput)
- iput(inode);
- return ret;
-
out_nospc_locked:
cleanup_bitmap_list(&bitmap_list);
spin_unlock(&ctl->tree_lock);
@@ -1396,7 +1384,17 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
if (block_group && (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
up_write(&block_group->data_rwsem);
- goto out;
+out:
+ io_ctl->inode = NULL;
+ io_ctl_free(io_ctl);
+ if (ret) {
+ invalidate_inode_pages2(inode->i_mapping);
+ BTRFS_I(inode)->generation = 0;
+ }
+ btrfs_update_inode(trans, root, inode);
+ if (must_iput)
+ iput(inode);
+ return ret;
}
int btrfs_write_out_cache(struct btrfs_trans_handle *trans,
--
2.24.1
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
2020-02-13 15:57 [PATCH v2 0/5] Fix memory leak on failed cache-writes Johannes Thumshirn
` (4 preceding siblings ...)
2020-02-13 15:58 ` [PATCH v2 5/5] btrfs: simplify error handling in __btrfs_write_out_cache() Johannes Thumshirn
@ 2020-02-18 16:50 ` David Sterba
2020-02-18 16:54 ` Johannes Thumshirn
5 siblings, 1 reply; 11+ messages in thread
From: David Sterba @ 2020-02-18 16:50 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: David Sterba, linux-btrfs @ vger . kernel . org
On Fri, Feb 14, 2020 at 12:57:58AM +0900, Johannes Thumshirn wrote:
> Fstests' test case generic/475 reliably leaks the btrfs_io_ctl::pages
> allocated in __btrfs_write_out_cache().
>
> The first patch in this series is freeing the pages when we throw away a dirty
> block group. The other patches are small things I noticed while hunting down the
> problem and are independant of fix.
>
> Changes to v1:
> - Move fix to the first position (David)
>
> Johannes Thumshirn (5):
> btrfs: free allocated pages on failed cache write-out
> btrfs: use inode from io_ctl in io_ctl_prepare_pages
> btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
> btrfs: use standard debug config option to enable free-space-cache
> debug prints
> btrfs: simplify error handling in __btrfs_write_out_cache()
I've seen some weird test failures and this patchset was in the tested
branch (either for-next or standalone). I need to retest misc-next again
to have a clean baseline so I can see the diff.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
2020-02-18 16:50 ` [PATCH v2 0/5] Fix memory leak on failed cache-writes David Sterba
@ 2020-02-18 16:54 ` Johannes Thumshirn
2020-02-24 17:49 ` David Sterba
0 siblings, 1 reply; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-18 16:54 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs @ vger . kernel . org
On 18/02/2020 17:50, David Sterba wrote:
> On Fri, Feb 14, 2020 at 12:57:58AM +0900, Johannes Thumshirn wrote:
>> Fstests' test case generic/475 reliably leaks the btrfs_io_ctl::pages
>> allocated in __btrfs_write_out_cache().
>>
>> The first patch in this series is freeing the pages when we throw away a dirty
>> block group. The other patches are small things I noticed while hunting down the
>> problem and are independant of fix.
>>
>> Changes to v1:
>> - Move fix to the first position (David)
>>
>> Johannes Thumshirn (5):
>> btrfs: free allocated pages on failed cache write-out
>> btrfs: use inode from io_ctl in io_ctl_prepare_pages
>> btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
>> btrfs: use standard debug config option to enable free-space-cache
>> debug prints
>> btrfs: simplify error handling in __btrfs_write_out_cache()
>
> I've seen some weird test failures and this patchset was in the tested
> branch (either for-next or standalone). I need to retest misc-next again
> to have a clean baseline so I can see the diff.
>
Interesting, can you forward me the failures?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
2020-02-18 16:54 ` Johannes Thumshirn
@ 2020-02-24 17:49 ` David Sterba
2020-02-25 0:23 ` Johannes Thumshirn
0 siblings, 1 reply; 11+ messages in thread
From: David Sterba @ 2020-02-24 17:49 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: dsterba, linux-btrfs @ vger . kernel . org
On Tue, Feb 18, 2020 at 04:54:18PM +0000, Johannes Thumshirn wrote:
> On 18/02/2020 17:50, David Sterba wrote:
> > On Fri, Feb 14, 2020 at 12:57:58AM +0900, Johannes Thumshirn wrote:
> >> Fstests' test case generic/475 reliably leaks the btrfs_io_ctl::pages
> >> allocated in __btrfs_write_out_cache().
> >>
> >> The first patch in this series is freeing the pages when we throw away a dirty
> >> block group. The other patches are small things I noticed while hunting down the
> >> problem and are independant of fix.
> >>
> >> Changes to v1:
> >> - Move fix to the first position (David)
> >>
> >> Johannes Thumshirn (5):
> >> btrfs: free allocated pages on failed cache write-out
> >> btrfs: use inode from io_ctl in io_ctl_prepare_pages
> >> btrfs: make the uptodate argument of io_ctl_add_pages() boolean.
> >> btrfs: use standard debug config option to enable free-space-cache
> >> debug prints
> >> btrfs: simplify error handling in __btrfs_write_out_cache()
> >
> > I've seen some weird test failures and this patchset was in the tested
> > branch (either for-next or standalone). I need to retest misc-next again
> > to have a clean baseline so I can see the diff.
> >
>
> Interesting, can you forward me the failures?
So it was test btrfs/125 but I can't say for sure which patchset causes
it, there are several reports but unlikely to be caused by yours.
btrfs/125 [14:07:58][ 9038.483946] run fstests btrfs/125 at 2020-02-18 14:07:58
[ 9038.894197] BTRFS info (device vda): disk space caching is enabled
[ 9038.897644] BTRFS info (device vda): has skinny extents
[ 9039.553603] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (29881)
[ 9039.557424] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 5 /dev/vdc scanned by mkfs.btrfs (29881)
[ 9039.561587] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 3 transid 5 /dev/vdd scanned by mkfs.btrfs (29881)
[ 9039.581158] BTRFS info (device vdb): turning on sync discard
[ 9039.584130] BTRFS info (device vdb): disk space caching is enabled
[ 9039.587021] BTRFS info (device vdb): has skinny extents
[ 9039.589100] BTRFS info (device vdb): flagging fs with big metadata feature
[ 9039.595079] BTRFS info (device vdb): checking UUID tree
[ 9039.706233] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 7 /dev/vdc scanned by mount (29923)
[ 9039.711721] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 7 /dev/vdb scanned by mount (29923)
[ 9039.715946] BTRFS info (device vdb): allowing degraded mounts
[ 9039.717785] BTRFS info (device vdb): disk space caching is enabled
[ 9039.719914] BTRFS info (device vdb): has skinny extents
[ 9039.723260] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
[ 9039.726785] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
[ 9040.189984] BTRFS: device fsid 4d1f43bd-2b49-4905-a219-25dc10f1b5fe devid 1 transid 249 /dev/vda scanned by btrfs (29951)
[ 9040.211252] BTRFS info (device vdb): turning on sync discard
[ 9040.223781] BTRFS info (device vdb): disk space caching is enabled
[ 9040.225777] BTRFS info (device vdb): has skinny extents
[ 9050.252423] BTRFS info (device vdb): balance: start -d -m -s
[ 9050.256160] BTRFS info (device vdb): relocating block group 217710592 flags data|raid5
[ 9050.389701] btrfs_print_data_csum_error: 6540 callbacks suppressed
[ 9050.389706] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5357568 csum 0xb5d704296aa5f2d5 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.398381] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5361664 csum 0xf2f7aa2e723d427e expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.398899] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3424256 csum 0x15e7d27c0f4b4f20 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.403918] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5365760 csum 0x25464d4ee302ec50 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.408525] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3428352 csum 0xcd1278f31a795b3d expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.412556] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5369856 csum 0x045d00c41342cf48 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.417133] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3432448 csum 0xab3b271dc78f0131 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.425859] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3436544 csum 0xa1e70b1d4d61d53f expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.430432] repair_io_failure: 9 callbacks suppressed
[ 9050.430435] BTRFS info (device vdb): read error corrected: ino 257 off 3424256 (dev /dev/vdd sector 196512)
[ 9050.430464] BTRFS info (device vdb): read error corrected: ino 257 off 3428352 (dev /dev/vdd sector 196520)
[ 9050.430473] BTRFS info (device vdb): read error corrected: ino 257 off 5361664 (dev /dev/vdd sector 198376)
[ 9050.430681] BTRFS info (device vdb): read error corrected: ino 257 off 3432448 (dev /dev/vdd sector 196528)
[ 9050.430762] BTRFS info (device vdb): read error corrected: ino 257 off 5357568 (dev /dev/vdd sector 198368)
[ 9050.430805] BTRFS info (device vdb): read error corrected: ino 257 off 5369856 (dev /dev/vdd sector 198392)
[ 9050.430853] BTRFS info (device vdb): read error corrected: ino 257 off 3436544 (dev /dev/vdd sector 196536)
[ 9050.432557] BTRFS info (device vdb): read error corrected: ino 257 off 5365760 (dev /dev/vdd sector 198384)
[ 9050.469621] BTRFS warning (device vdb): csum failed root -9 ino 257 off 13959168 csum 0x774317b489e76165 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.476527] BTRFS info (device vdb): read error corrected: ino 257 off 13959168 (dev /dev/vdd sector 206720)
[ 9050.501930] BTRFS warning (device vdb): csum failed root -9 ino 257 off 16351232 csum 0xa735bf811aa10b77 expected csum 0xdbbbd8326f9b86ac mirror 1
[ 9050.512917] BTRFS info (device vdb): read error corrected: ino 257 off 17608704 (dev /dev/vdd sector 210392)
[ 9050.708681] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9050.708684] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
[ 9050.708699] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
[ 9050.708700] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
[ 9050.713287] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9050.727830] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9050.727849] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
[ 9050.727854] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
[ 9050.727876] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
[ 9050.730601] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
[ 9053.133149] BTRFS info (device vdb): balance: ended with status: -5
[failed, exit status 1][ 9053.200709] BTRFS info (device vdb): clearing incompat feature flag for RAID56 (0x80)
[14:08:12]- output mismatch (see /tmp/fstests/results//btrfs/125.out.bad)
--- tests/btrfs/125.out 2018-04-12 16:57:00.616225550 +0000
+++ /tmp/fstests/results//btrfs/125.out.bad 2020-02-18 14:08:12.856000000 +0000
@@ -3,5 +3,5 @@
Write data with degraded mount
Mount normal and balance
-
-Mount degraded but with other dev
+failed: '/sbin/btrfs balance start /tmp/scratch'
+(see /tmp/fstests/results//btrfs/125.full for details)
...
(Run 'diff -u /tmp/fstests/tests/btrfs/125.out /tmp/fstests/results//btrfs/125.out.bad' to see the entire diff)
---
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
2020-02-24 17:49 ` David Sterba
@ 2020-02-25 0:23 ` Johannes Thumshirn
2020-02-25 0:56 ` Johannes Thumshirn
0 siblings, 1 reply; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-25 0:23 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs @ vger . kernel . org
On 24/02/2020 09:50, David Sterba wrote:
[...]
> So it was test btrfs/125 but I can't say for sure which patchset causes
> it, there are several reports but unlikely to be caused by yours.
>
> btrfs/125 [14:07:58][ 9038.483946] run fstests btrfs/125 at 2020-02-18 14:07:58
> [ 9038.894197] BTRFS info (device vda): disk space caching is enabled
> [ 9038.897644] BTRFS info (device vda): has skinny extents
> [ 9039.553603] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (29881)
> [ 9039.557424] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 5 /dev/vdc scanned by mkfs.btrfs (29881)
> [ 9039.561587] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 3 transid 5 /dev/vdd scanned by mkfs.btrfs (29881)
> [ 9039.581158] BTRFS info (device vdb): turning on sync discard
> [ 9039.584130] BTRFS info (device vdb): disk space caching is enabled
> [ 9039.587021] BTRFS info (device vdb): has skinny extents
> [ 9039.589100] BTRFS info (device vdb): flagging fs with big metadata feature
> [ 9039.595079] BTRFS info (device vdb): checking UUID tree
> [ 9039.706233] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 2 transid 7 /dev/vdc scanned by mount (29923)
> [ 9039.711721] BTRFS: device fsid 853f4928-5113-48fe-b4e5-b2d1f1af15bf devid 1 transid 7 /dev/vdb scanned by mount (29923)
> [ 9039.715946] BTRFS info (device vdb): allowing degraded mounts
> [ 9039.717785] BTRFS info (device vdb): disk space caching is enabled
> [ 9039.719914] BTRFS info (device vdb): has skinny extents
> [ 9039.723260] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
> [ 9039.726785] BTRFS warning (device vdb): devid 3 uuid a55e3334-e24b-4220-93a2-9aaec7286042 is missing
> [ 9040.189984] BTRFS: device fsid 4d1f43bd-2b49-4905-a219-25dc10f1b5fe devid 1 transid 249 /dev/vda scanned by btrfs (29951)
> [ 9040.211252] BTRFS info (device vdb): turning on sync discard
> [ 9040.223781] BTRFS info (device vdb): disk space caching is enabled
> [ 9040.225777] BTRFS info (device vdb): has skinny extents
> [ 9050.252423] BTRFS info (device vdb): balance: start -d -m -s
> [ 9050.256160] BTRFS info (device vdb): relocating block group 217710592 flags data|raid5
> [ 9050.389701] btrfs_print_data_csum_error: 6540 callbacks suppressed
> [ 9050.389706] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5357568 csum 0xb5d704296aa5f2d5 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.398381] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5361664 csum 0xf2f7aa2e723d427e expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.398899] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3424256 csum 0x15e7d27c0f4b4f20 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.403918] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5365760 csum 0x25464d4ee302ec50 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.408525] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3428352 csum 0xcd1278f31a795b3d expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.412556] BTRFS warning (device vdb): csum failed root -9 ino 257 off 5369856 csum 0x045d00c41342cf48 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.417133] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3432448 csum 0xab3b271dc78f0131 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.425859] BTRFS warning (device vdb): csum failed root -9 ino 257 off 3436544 csum 0xa1e70b1d4d61d53f expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.430432] repair_io_failure: 9 callbacks suppressed
> [ 9050.430435] BTRFS info (device vdb): read error corrected: ino 257 off 3424256 (dev /dev/vdd sector 196512)
> [ 9050.430464] BTRFS info (device vdb): read error corrected: ino 257 off 3428352 (dev /dev/vdd sector 196520)
> [ 9050.430473] BTRFS info (device vdb): read error corrected: ino 257 off 5361664 (dev /dev/vdd sector 198376)
> [ 9050.430681] BTRFS info (device vdb): read error corrected: ino 257 off 3432448 (dev /dev/vdd sector 196528)
> [ 9050.430762] BTRFS info (device vdb): read error corrected: ino 257 off 5357568 (dev /dev/vdd sector 198368)
> [ 9050.430805] BTRFS info (device vdb): read error corrected: ino 257 off 5369856 (dev /dev/vdd sector 198392)
> [ 9050.430853] BTRFS info (device vdb): read error corrected: ino 257 off 3436544 (dev /dev/vdd sector 196536)
> [ 9050.432557] BTRFS info (device vdb): read error corrected: ino 257 off 5365760 (dev /dev/vdd sector 198384)
> [ 9050.469621] BTRFS warning (device vdb): csum failed root -9 ino 257 off 13959168 csum 0x774317b489e76165 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.476527] BTRFS info (device vdb): read error corrected: ino 257 off 13959168 (dev /dev/vdd sector 206720)
> [ 9050.501930] BTRFS warning (device vdb): csum failed root -9 ino 257 off 16351232 csum 0xa735bf811aa10b77 expected csum 0xdbbbd8326f9b86ac mirror 1
> [ 9050.512917] BTRFS info (device vdb): read error corrected: ino 257 off 17608704 (dev /dev/vdd sector 210392)
> [ 9050.708681] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9050.708684] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
> [ 9050.708699] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
> [ 9050.708700] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
> [ 9050.713287] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9050.727830] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9050.727849] BTRFS error (device vdb): bad tree block start, want 39075840 have 0
> [ 9050.727854] BTRFS error (device vdb): bad tree block start, want 39092224 have 0
> [ 9050.727876] BTRFS error (device vdb): bad tree block start, want 39108608 have 0
> [ 9050.730601] BTRFS error (device vdb): bad tree block start, want 39059456 have 0
> [ 9053.133149] BTRFS info (device vdb): balance: ended with status: -5
> [failed, exit status 1][ 9053.200709] BTRFS info (device vdb): clearing incompat feature flag for RAID56 (0x80)
> [14:08:12]- output mismatch (see /tmp/fstests/results//btrfs/125.out.bad)
> --- tests/btrfs/125.out 2018-04-12 16:57:00.616225550 +0000
> +++ /tmp/fstests/results//btrfs/125.out.bad 2020-02-18 14:08:12.856000000 +0000
> @@ -3,5 +3,5 @@
> Write data with degraded mount
>
> Mount normal and balance
> -
> -Mount degraded but with other dev
> +failed: '/sbin/btrfs balance start /tmp/scratch'
> +(see /tmp/fstests/results//btrfs/125.full for details)
> ...
> (Run 'diff -u /tmp/fstests/tests/btrfs/125.out /tmp/fstests/results//btrfs/125.out.bad' to see the entire diff)
OK it need some time (up to approx 150 runs of btrfs/125) to resproduce
this failure but I can re-create it.
I've seen it once without the patchset applied as well though (a.k.a
HEAD == 480be04e7fdcddfed86fd59bb668a134b6d7393e).
I have to verify that though not that I'm seeing ghosts.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/5] Fix memory leak on failed cache-writes
2020-02-25 0:23 ` Johannes Thumshirn
@ 2020-02-25 0:56 ` Johannes Thumshirn
0 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2020-02-25 0:56 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs @ vger . kernel . org
On 24/02/2020 16:23, Johannes Thumshirn wrote:
[...]
> OK it need some time (up to approx 150 runs of btrfs/125) to resproduce
> this failure but I can re-create it.
>
> I've seen it once without the patchset applied as well though (a.k.a
> HEAD == 480be04e7fdcddfed86fd59bb668a134b6d7393e).
>
> I have to verify that though not that I'm seeing ghosts.
>
OK, so I've hit it again with the above commit as HEAD, so I don't think
these patches participate in the error.
I'll try to figure out which patch does though.
Byte,
Johannes
^ permalink raw reply [flat|nested] 11+ messages in thread