Linux-BTRFS Archive on lore.kernel.org
 help / Atom feed
* [PATCH] btrfs: don't end the transaction for delayed refs in throttle
@ 2019-01-24 14:31 Josef Bacik
  2019-02-12 16:03 ` David Sterba
  0 siblings, 1 reply; 7+ messages in thread
From: Josef Bacik @ 2019-01-24 14:31 UTC (permalink / raw)
  To: linux-btrfs, kernel-team

Previously callers to btrfs_end_transaction_throttle() would commit the
transaction if there wasn't enough delayed refs space.  This happens in
relocation, and if the fs is relatively empty we'll run out of delayed
refs space basically immediately, so we'll just be stuck in this loop of
committing the transaction over and over again.

This code existed because we didn't have a good feedback mechanism for
running delayed refs, but with the delayed refs rsv we do now.  Delete
this throttling code and let the btrfs_start_transaction() in relocation
deal with putting pressure on the delayed refs infrastructure.  With
this patch we no longer take 5 minutes to balance a metadata only fs.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 fs/btrfs/transaction.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 22b0dacae003..ef98b3cd30db 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -851,14 +851,6 @@ static int __btrfs_end_transaction(struct btrfs_trans_handle *trans,
 
 	btrfs_trans_release_chunk_metadata(trans);
 
-	if (lock && should_end_transaction(trans) &&
-	    READ_ONCE(cur_trans->state) == TRANS_STATE_RUNNING) {
-		spin_lock(&info->trans_lock);
-		if (cur_trans->state == TRANS_STATE_RUNNING)
-			cur_trans->state = TRANS_STATE_BLOCKED;
-		spin_unlock(&info->trans_lock);
-	}
-
 	if (lock && READ_ONCE(cur_trans->state) == TRANS_STATE_BLOCKED) {
 		if (throttle)
 			return btrfs_commit_transaction(trans);
-- 
2.14.3


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle
  2019-01-24 14:31 [PATCH] btrfs: don't end the transaction for delayed refs in throttle Josef Bacik
@ 2019-02-12 16:03 ` David Sterba
  2019-06-03  6:53   ` Qu Wenruo
  0 siblings, 1 reply; 7+ messages in thread
From: David Sterba @ 2019-02-12 16:03 UTC (permalink / raw)
  To: Josef Bacik; +Cc: linux-btrfs, kernel-team

On Thu, Jan 24, 2019 at 09:31:43AM -0500, Josef Bacik wrote:
> Previously callers to btrfs_end_transaction_throttle() would commit the
> transaction if there wasn't enough delayed refs space.  This happens in
> relocation, and if the fs is relatively empty we'll run out of delayed
> refs space basically immediately, so we'll just be stuck in this loop of
> committing the transaction over and over again.
> 
> This code existed because we didn't have a good feedback mechanism for
> running delayed refs, but with the delayed refs rsv we do now.  Delete
> this throttling code and let the btrfs_start_transaction() in relocation
> deal with putting pressure on the delayed refs infrastructure.  With
> this patch we no longer take 5 minutes to balance a metadata only fs.
> 
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>

For the record, this has been merged to 5.0-rc5

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle
  2019-02-12 16:03 ` David Sterba
@ 2019-06-03  6:53   ` Qu Wenruo
  2019-06-03 17:36     ` Josef Bacik
  0 siblings, 1 reply; 7+ messages in thread
From: Qu Wenruo @ 2019-06-03  6:53 UTC (permalink / raw)
  To: dsterba, Josef Bacik, linux-btrfs, kernel-team

[-- Attachment #1.1: Type: text/plain, Size: 2227 bytes --]



On 2019/2/13 上午12:03, David Sterba wrote:
> On Thu, Jan 24, 2019 at 09:31:43AM -0500, Josef Bacik wrote:
>> Previously callers to btrfs_end_transaction_throttle() would commit the
>> transaction if there wasn't enough delayed refs space.  This happens in
>> relocation, and if the fs is relatively empty we'll run out of delayed
>> refs space basically immediately, so we'll just be stuck in this loop of
>> committing the transaction over and over again.
>>
>> This code existed because we didn't have a good feedback mechanism for
>> running delayed refs, but with the delayed refs rsv we do now.  Delete
>> this throttling code and let the btrfs_start_transaction() in relocation
>> deal with putting pressure on the delayed refs infrastructure.  With
>> this patch we no longer take 5 minutes to balance a metadata only fs.
>>
>> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> 
> For the record, this has been merged to 5.0-rc5
> 

Bisecting leads me to this patch for strange balance ENOSPC.

Can be reproduced by btrfs/156, or the following small script:
------
#!/bin/bash
dev="/dev/test/test"
mnt="/mnt/btrfs"

_fail()
{
	echo "!!! FAILED: $@ !!!"
	exit 1
}

do_work()
{
	umount $dev &> /dev/null
	umount $mnt &> /dev/null

	mkfs.btrfs -b 1G -m single -d single $dev -f > /dev/null

	mount $dev $mnt

	for i in $(seq -w 0 511); do
	#	xfs_io -f -c "falloc 0 1m" $mnt/file_$i > /dev/null
		xfs_io -f -c "pwrite 0 1m" $mnt/inline_$i > /dev/null
	done
	sync

	btrfs balance start --full $mnt || return 1
	sync


	btrfs balance start --full $mnt || return 1
	umount $mnt
}

failed=0
for i in $(seq -w 0 24); do
	echo "=== run $i ==="
	do_work
	if [ $? -eq 1 ]; then
		failed=$(($failed + 1))
	fi
done
if [ $failed -ne 0 ]; then
	echo "!!! failed $failed/25 !!!"
else
	echo "=== all passes ==="
fi
------

For v4.20, it will fail at the rate around 0/25 ~ 2/25 (very rare).
But at that patch (upstream commit
302167c50b32e7fccc98994a91d40ddbbab04e52), the failure rate raise to 25/25.

Any idea for that ENOSPC problem?
As it looks really wired for the 2nd full balance to fail even we have
enough unallocated space.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle
  2019-06-03  6:53   ` Qu Wenruo
@ 2019-06-03 17:36     ` Josef Bacik
  2019-06-03 18:43       ` David Sterba
  2019-06-04  0:31       ` Qu Wenruo
  0 siblings, 2 replies; 7+ messages in thread
From: Josef Bacik @ 2019-06-03 17:36 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: dsterba, Josef Bacik, linux-btrfs, kernel-team

On Mon, Jun 03, 2019 at 02:53:00PM +0800, Qu Wenruo wrote:
> 
> 
> On 2019/2/13 上午12:03, David Sterba wrote:
> > On Thu, Jan 24, 2019 at 09:31:43AM -0500, Josef Bacik wrote:
> >> Previously callers to btrfs_end_transaction_throttle() would commit the
> >> transaction if there wasn't enough delayed refs space.  This happens in
> >> relocation, and if the fs is relatively empty we'll run out of delayed
> >> refs space basically immediately, so we'll just be stuck in this loop of
> >> committing the transaction over and over again.
> >>
> >> This code existed because we didn't have a good feedback mechanism for
> >> running delayed refs, but with the delayed refs rsv we do now.  Delete
> >> this throttling code and let the btrfs_start_transaction() in relocation
> >> deal with putting pressure on the delayed refs infrastructure.  With
> >> this patch we no longer take 5 minutes to balance a metadata only fs.
> >>
> >> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> > 
> > For the record, this has been merged to 5.0-rc5
> > 
> 
> Bisecting leads me to this patch for strange balance ENOSPC.
> 
> Can be reproduced by btrfs/156, or the following small script:
> ------
> #!/bin/bash
> dev="/dev/test/test"
> mnt="/mnt/btrfs"
> 
> _fail()
> {
> 	echo "!!! FAILED: $@ !!!"
> 	exit 1
> }
> 
> do_work()
> {
> 	umount $dev &> /dev/null
> 	umount $mnt &> /dev/null
> 
> 	mkfs.btrfs -b 1G -m single -d single $dev -f > /dev/null
> 
> 	mount $dev $mnt
> 
> 	for i in $(seq -w 0 511); do
> 	#	xfs_io -f -c "falloc 0 1m" $mnt/file_$i > /dev/null
> 		xfs_io -f -c "pwrite 0 1m" $mnt/inline_$i > /dev/null
> 	done
> 	sync
> 
> 	btrfs balance start --full $mnt || return 1
> 	sync
> 
> 
> 	btrfs balance start --full $mnt || return 1
> 	umount $mnt
> }
> 
> failed=0
> for i in $(seq -w 0 24); do
> 	echo "=== run $i ==="
> 	do_work
> 	if [ $? -eq 1 ]; then
> 		failed=$(($failed + 1))
> 	fi
> done
> if [ $failed -ne 0 ]; then
> 	echo "!!! failed $failed/25 !!!"
> else
> 	echo "=== all passes ==="
> fi
> ------
> 
> For v4.20, it will fail at the rate around 0/25 ~ 2/25 (very rare).
> But at that patch (upstream commit
> 302167c50b32e7fccc98994a91d40ddbbab04e52), the failure rate raise to 25/25.
> 
> Any idea for that ENOSPC problem?
> As it looks really wired for the 2nd full balance to fail even we have
> enough unallocated space.
> 

I've been running this all morning on kdave's misc-next and not had a single
failure.  I ran it a few times on spinning rust and a few times on my nvme
drive.  I wouldn't doubt that it's failing for you, but I can't reproduce.  It
would be helpful to know where the ENOSPC was coming from so I can think of
where the problem might be.  Thanks,

Josef


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle
  2019-06-03 17:36     ` Josef Bacik
@ 2019-06-03 18:43       ` David Sterba
  2019-06-04  0:31       ` Qu Wenruo
  1 sibling, 0 replies; 7+ messages in thread
From: David Sterba @ 2019-06-03 18:43 UTC (permalink / raw)
  To: Josef Bacik; +Cc: Qu Wenruo, dsterba, linux-btrfs, kernel-team

On Mon, Jun 03, 2019 at 01:36:11PM -0400, Josef Bacik wrote:
> > For v4.20, it will fail at the rate around 0/25 ~ 2/25 (very rare).
> > But at that patch (upstream commit
> > 302167c50b32e7fccc98994a91d40ddbbab04e52), the failure rate raise to 25/25.
> > 
> > Any idea for that ENOSPC problem?
> > As it looks really wired for the 2nd full balance to fail even we have
> > enough unallocated space.
> > 
> 
> I've been running this all morning on kdave's misc-next and not had a single
> failure.  I ran it a few times on spinning rust and a few times on my nvme
> drive.  I wouldn't doubt that it's failing for you, but I can't reproduce.  It
> would be helpful to know where the ENOSPC was coming from so I can think of
> where the problem might be.  Thanks,

That's interesting because the test btrfs/156 hasn't succesfully
finished a single run on my VM testing setup since qemu 4.0 implemented
discard on virtio devices several weeks ago. It's 4cpu/2g, file-backed
virtio-scsi devices, files are on a spinning disk.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle
  2019-06-03 17:36     ` Josef Bacik
  2019-06-03 18:43       ` David Sterba
@ 2019-06-04  0:31       ` Qu Wenruo
  2019-06-04 17:43         ` Josef Bacik
  1 sibling, 1 reply; 7+ messages in thread
From: Qu Wenruo @ 2019-06-04  0:31 UTC (permalink / raw)
  To: Josef Bacik; +Cc: dsterba, linux-btrfs, kernel-team

[-- Attachment #1.1: Type: text/plain, Size: 8838 bytes --]



On 2019/6/4 上午1:36, Josef Bacik wrote:
> On Mon, Jun 03, 2019 at 02:53:00PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2019/2/13 上午12:03, David Sterba wrote:
>>> On Thu, Jan 24, 2019 at 09:31:43AM -0500, Josef Bacik wrote:
>>>> Previously callers to btrfs_end_transaction_throttle() would commit the
>>>> transaction if there wasn't enough delayed refs space.  This happens in
>>>> relocation, and if the fs is relatively empty we'll run out of delayed
>>>> refs space basically immediately, so we'll just be stuck in this loop of
>>>> committing the transaction over and over again.
>>>>
>>>> This code existed because we didn't have a good feedback mechanism for
>>>> running delayed refs, but with the delayed refs rsv we do now.  Delete
>>>> this throttling code and let the btrfs_start_transaction() in relocation
>>>> deal with putting pressure on the delayed refs infrastructure.  With
>>>> this patch we no longer take 5 minutes to balance a metadata only fs.
>>>>
>>>> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
>>>
>>> For the record, this has been merged to 5.0-rc5
>>>
>>
>> Bisecting leads me to this patch for strange balance ENOSPC.
>>
>> Can be reproduced by btrfs/156, or the following small script:
>> ------
>> #!/bin/bash
>> dev="/dev/test/test"
>> mnt="/mnt/btrfs"
>>
>> _fail()
>> {
>> 	echo "!!! FAILED: $@ !!!"
>> 	exit 1
>> }
>>
>> do_work()
>> {
>> 	umount $dev &> /dev/null
>> 	umount $mnt &> /dev/null
>>
>> 	mkfs.btrfs -b 1G -m single -d single $dev -f > /dev/null
>>
>> 	mount $dev $mnt
>>
>> 	for i in $(seq -w 0 511); do
>> 	#	xfs_io -f -c "falloc 0 1m" $mnt/file_$i > /dev/null
>> 		xfs_io -f -c "pwrite 0 1m" $mnt/inline_$i > /dev/null
>> 	done
>> 	sync
>>
>> 	btrfs balance start --full $mnt || return 1
>> 	sync
>>
>>
>> 	btrfs balance start --full $mnt || return 1
>> 	umount $mnt
>> }
>>
>> failed=0
>> for i in $(seq -w 0 24); do
>> 	echo "=== run $i ==="
>> 	do_work
>> 	if [ $? -eq 1 ]; then
>> 		failed=$(($failed + 1))
>> 	fi
>> done
>> if [ $failed -ne 0 ]; then
>> 	echo "!!! failed $failed/25 !!!"
>> else
>> 	echo "=== all passes ==="
>> fi
>> ------
>>
>> For v4.20, it will fail at the rate around 0/25 ~ 2/25 (very rare).
>> But at that patch (upstream commit
>> 302167c50b32e7fccc98994a91d40ddbbab04e52), the failure rate raise to 25/25.
>>
>> Any idea for that ENOSPC problem?
>> As it looks really wired for the 2nd full balance to fail even we have
>> enough unallocated space.
>>
> 
> I've been running this all morning on kdave's misc-next and not had a single
> failure.  I ran it a few times on spinning rust and a few times on my nvme
> drive.  I wouldn't doubt that it's failing for you, but I can't reproduce.  It
> would be helpful to know where the ENOSPC was coming from so I can think of
> where the problem might be.  Thanks,
> 
> Josef
> 

Since v5.2-rc2 has a lot of enospc debug output merged, here is the
debug info just by enospc_debug:

BTRFS: device fsid defe70f2-d083-41f0-a4fd-28a0cc03dce7 devid 1 transid
5 /dev/test/test
BTRFS info (device dm-3): disk space caching is enabled
BTRFS info (device dm-3): has skinny extents
BTRFS info (device dm-3): flagging fs with big metadata feature
BTRFS info (device dm-3): checking UUID tree
BTRFS info (device dm-3): balance: start -d -m -s
BTRFS info (device dm-3): relocating block group 726663168 flags metadata
BTRFS info (device dm-3): relocating block group 609222656 flags data
BTRFS info (device dm-3): found 57 extents
BTRFS info (device dm-3): found 57 extents
BTRFS info (device dm-3): relocating block group 491782144 flags data
BTRFS info (device dm-3): found 112 extents
BTRFS info (device dm-3): found 112 extents
BTRFS info (device dm-3): relocating block group 374341632 flags data
BTRFS info (device dm-3): found 115 extents
BTRFS info (device dm-3): found 114 extents
BTRFS info (device dm-3): relocating block group 256901120 flags data
BTRFS info (device dm-3): found 112 extents
BTRFS info (device dm-3): found 112 extents
BTRFS info (device dm-3): relocating block group 139460608 flags data
BTRFS info (device dm-3): found 112 extents
BTRFS info (device dm-3): found 112 extents
BTRFS info (device dm-3): unable to make block group 22020096 ro
BTRFS info (device dm-3): sinfo_used=42909696 bg_num_bytes=116293632
min_allocable=1048576
BTRFS info (device dm-3): space_info 4 has 82919424 free, is not full
BTRFS info (device dm-3): space_info total=125829120, used=1015808,
pinned=0, reserved=81920, may_use=41746432, readonly=65536
BTRFS info (device dm-3): global_block_rsv: size 16777216 reserved 16744448
BTRFS info (device dm-3): trans_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): chunk_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_refs_rsv: size 61865984 reserved 25001984
BTRFS info (device dm-3): relocating block group 22020096 flags metadata
BTRFS info (device dm-3): found 54 extents
BTRFS info (device dm-3): relocating block group 13631488 flags data
BTRFS info (device dm-3): found 8 extents
BTRFS info (device dm-3): found 8 extents
BTRFS info (device dm-3): relocating block group 5242880 flags metadata
BTRFS info (device dm-3): found 56 extents
BTRFS info (device dm-3): unable to make block group 1048576 ro
BTRFS info (device dm-3): sinfo_used=32768 bg_num_bytes=4161536
min_allocable=1048576
BTRFS info (device dm-3): space_info 2 has 4161536 free, is not full
BTRFS info (device dm-3): space_info total=4194304, used=16384,
pinned=0, reserved=16384, may_use=0, readonly=0
BTRFS info (device dm-3): global_block_rsv: size 16777216 reserved 16744448
BTRFS info (device dm-3): trans_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): chunk_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_refs_rsv: size 3145728 reserved 1540096
BTRFS info (device dm-3): relocating block group 1048576 flags system
BTRFS info (device dm-3): found 1 extents
BTRFS info (device dm-3): balance: ended with status: 0
BTRFS info (device dm-3): balance: start -d -m -s
BTRFS info (device dm-3): unable to make block group 1431306240 ro
BTRFS info (device dm-3): sinfo_used=16384 bg_num_bytes=33538048
min_allocable=1048576
BTRFS info (device dm-3): space_info 2 has 33538048 free, is not full
BTRFS info (device dm-3): space_info total=33554432, used=16384,
pinned=0, reserved=0, may_use=0, readonly=0
BTRFS info (device dm-3): global_block_rsv: size 16777216 reserved 16777216
BTRFS info (device dm-3): trans_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): chunk_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_refs_rsv: size 0 reserved 0
BTRFS info (device dm-3): relocating block group 1431306240 flags system
BTRFS info (device dm-3): unable to make block group 1313865728 ro
BTRFS info (device dm-3): sinfo_used=19382272 bg_num_bytes=116342784
min_allocable=1048576
BTRFS info (device dm-3): space_info 4 has 98058240 free, is not full
BTRFS info (device dm-3): space_info total=117440512, used=1015808,
pinned=0, reserved=81920, may_use=18284544, readonly=0
BTRFS info (device dm-3): global_block_rsv: size 16777216 reserved 16744448
BTRFS info (device dm-3): trans_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): chunk_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_block_rsv: size 0 reserved 0
BTRFS info (device dm-3): delayed_refs_rsv: size 3145728 reserved 1540096
BTRFS info (device dm-3): relocating block group 1313865728 flags metadata
BTRFS info (device dm-3): found 55 extents
BTRFS info (device dm-3): relocating block group 1196425216 flags data
BTRFS info (device dm-3): found 65 extents
BTRFS info (device dm-3): found 65 extents
BTRFS warning (device dm-3): no space to allocate a new chunk for block
group 1078984704
BTRFS warning (device dm-3): no space to allocate a new chunk for block
group 961544192
BTRFS warning (device dm-3): no space to allocate a new chunk for block
group 844103680
BTRFS warning (device dm-3): no space to allocate a new chunk for block
group 726663168
BTRFS info (device dm-3): 4 enospc errors during balance
BTRFS info (device dm-3): balance: ended with status: -28

The ENOSPC is still from inc_block_group_ro().
For the first data block failure, it still looks like something in
bytes_may_use doesn't look correct.

Although for system block group, it's another story. It has no
reserved/pinned/may_use bytes, it's the min_allocable failing the check.

I can't remember if this is some bug I reported before, but looks a
little similar.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle
  2019-06-04  0:31       ` Qu Wenruo
@ 2019-06-04 17:43         ` Josef Bacik
  0 siblings, 0 replies; 7+ messages in thread
From: Josef Bacik @ 2019-06-04 17:43 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Josef Bacik, dsterba, linux-btrfs, kernel-team

On Tue, Jun 04, 2019 at 08:31:23AM +0800, Qu Wenruo wrote:
> 
> 
> On 2019/6/4 上午1:36, Josef Bacik wrote:
> > On Mon, Jun 03, 2019 at 02:53:00PM +0800, Qu Wenruo wrote:
> >>
> >>
> >> On 2019/2/13 上午12:03, David Sterba wrote:
> >>> On Thu, Jan 24, 2019 at 09:31:43AM -0500, Josef Bacik wrote:
> >>>> Previously callers to btrfs_end_transaction_throttle() would commit the
> >>>> transaction if there wasn't enough delayed refs space.  This happens in
> >>>> relocation, and if the fs is relatively empty we'll run out of delayed
> >>>> refs space basically immediately, so we'll just be stuck in this loop of
> >>>> committing the transaction over and over again.
> >>>>
> >>>> This code existed because we didn't have a good feedback mechanism for
> >>>> running delayed refs, but with the delayed refs rsv we do now.  Delete
> >>>> this throttling code and let the btrfs_start_transaction() in relocation
> >>>> deal with putting pressure on the delayed refs infrastructure.  With
> >>>> this patch we no longer take 5 minutes to balance a metadata only fs.
> >>>>
> >>>> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> >>>
> >>> For the record, this has been merged to 5.0-rc5
> >>>
> >>
> >> Bisecting leads me to this patch for strange balance ENOSPC.
> >>
> >> Can be reproduced by btrfs/156, or the following small script:
> >> ------
> >> #!/bin/bash
> >> dev="/dev/test/test"
> >> mnt="/mnt/btrfs"
> >>
> >> _fail()
> >> {
> >> 	echo "!!! FAILED: $@ !!!"
> >> 	exit 1
> >> }
> >>
> >> do_work()
> >> {
> >> 	umount $dev &> /dev/null
> >> 	umount $mnt &> /dev/null
> >>
> >> 	mkfs.btrfs -b 1G -m single -d single $dev -f > /dev/null
> >>
> >> 	mount $dev $mnt
> >>
> >> 	for i in $(seq -w 0 511); do
> >> 	#	xfs_io -f -c "falloc 0 1m" $mnt/file_$i > /dev/null
> >> 		xfs_io -f -c "pwrite 0 1m" $mnt/inline_$i > /dev/null
> >> 	done
> >> 	sync
> >>
> >> 	btrfs balance start --full $mnt || return 1
> >> 	sync
> >>
> >>
> >> 	btrfs balance start --full $mnt || return 1
> >> 	umount $mnt
> >> }
> >>
> >> failed=0
> >> for i in $(seq -w 0 24); do
> >> 	echo "=== run $i ==="
> >> 	do_work
> >> 	if [ $? -eq 1 ]; then
> >> 		failed=$(($failed + 1))
> >> 	fi
> >> done
> >> if [ $failed -ne 0 ]; then
> >> 	echo "!!! failed $failed/25 !!!"
> >> else
> >> 	echo "=== all passes ==="
> >> fi
> >> ------
> >>
> >> For v4.20, it will fail at the rate around 0/25 ~ 2/25 (very rare).
> >> But at that patch (upstream commit
> >> 302167c50b32e7fccc98994a91d40ddbbab04e52), the failure rate raise to 25/25.
> >>
> >> Any idea for that ENOSPC problem?
> >> As it looks really wired for the 2nd full balance to fail even we have
> >> enough unallocated space.
> >>
> > 
> > I've been running this all morning on kdave's misc-next and not had a single
> > failure.  I ran it a few times on spinning rust and a few times on my nvme
> > drive.  I wouldn't doubt that it's failing for you, but I can't reproduce.  It
> > would be helpful to know where the ENOSPC was coming from so I can think of
> > where the problem might be.  Thanks,
> > 
> > Josef
> > 
> 
> Since v5.2-rc2 has a lot of enospc debug output merged, here is the
> debug info just by enospc_debug:
>

Ah ok, sorry I'm travelling so I can't easily test a patch right now, but change
the btrfs_join_transaction() in btrfs_inc_block_group_ro to
btrfs_start_transaction(root, 0);  This will trigger the delayed ref flushing if
we need it and likely will fix the problem.  There's so much random cruft built
into the relocation enospc stuff that we're likely to keep finding problems like
this, we just need to rework it so it's still tripping over the normal
reservation path.  Thanks,

Josef 

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, back to index

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-24 14:31 [PATCH] btrfs: don't end the transaction for delayed refs in throttle Josef Bacik
2019-02-12 16:03 ` David Sterba
2019-06-03  6:53   ` Qu Wenruo
2019-06-03 17:36     ` Josef Bacik
2019-06-03 18:43       ` David Sterba
2019-06-04  0:31       ` Qu Wenruo
2019-06-04 17:43         ` Josef Bacik

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox