* Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
@ 2020-01-13 12:41 Craig Andrews
2020-01-13 13:37 ` David Sterba
0 siblings, 1 reply; 14+ messages in thread
From: Craig Andrews @ 2020-01-13 12:41 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]
If I perform a btrfs send receive like so:
sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230
/mnt/everything/.snapshots/root.20191231 | btrfs receive
/mnt/backup/.snapshots/
On Linux 5.4.0, the process completes successfully.
Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the
result is the OOM killer being invoked which (among other process
carnage) kills the btrfs processes stopping the backup.
I'm using the same kernel config, same hardware, etc in the two tests.
The system has 16 GB of RAM, CPU is an i5-6500, arch is amd64.
/mnt/everything is a btrfs fs on luks (/dev/mapper/sda4) on a 1 TB SATA
SSD (/dev/sda4). /mnt/backup is btfs fs on luks (/dev/mapper/backup) on
a 2 TB external spinning rust HDD (/dev/sdb).
# df -H /mnt/everything/ /mnt/backup
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/sda4 949G 698G 247G 74% /mnt/everything
/dev/mapper/backup 2.1T 1.1T 948G 53% /mnt/backup
I've attached logs and my kernel config to
https://bugzilla.kernel.org/show_bug.cgi?id=206031
Thank you,
~Craig
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-13 12:41 Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory Craig Andrews
@ 2020-01-13 13:37 ` David Sterba
2020-01-13 14:48 ` Craig Andrews
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: David Sterba @ 2020-01-13 13:37 UTC (permalink / raw)
To: Craig Andrews; +Cc: linux-btrfs
On Mon, Jan 13, 2020 at 07:41:13AM -0500, Craig Andrews wrote:
> If I perform a btrfs send receive like so:
>
> sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230
> /mnt/everything/.snapshots/root.20191231 | btrfs receive
> /mnt/backup/.snapshots/
>
> On Linux 5.4.0, the process completes successfully.
>
> Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the
> result is the OOM killer being invoked which (among other process
> carnage) kills the btrfs processes stopping the backup.
As this is on the -rc1, it's possible that changes done in the MM
subsystem could change the logic of OOM and that send now is able to
trigger the OOM.
There are 2 btrfs patches in send.c but they reduce amount of work,
namely for heavily reflinked extents so the effects should be opposite.
To find out where's the cause, is it possible that you build a kernel
from the 5.4-based branch and run the send again? It's the same set of
btrfs patches that gets merged to 5.5.
From repository: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4
If this can't be done we'll find another way to debug it.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-13 13:37 ` David Sterba
@ 2020-01-13 14:48 ` Craig Andrews
2020-01-15 18:44 ` Craig Andrews
2020-01-21 2:18 ` Craig Andrews
2020-01-25 11:08 ` Stéphane Lesimple
2 siblings, 1 reply; 14+ messages in thread
From: Craig Andrews @ 2020-01-13 14:48 UTC (permalink / raw)
To: Craig Andrews; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]
On 2020-01-13 08:37, David Sterba wrote:
> On Mon, Jan 13, 2020 at 07:41:13AM -0500, Craig Andrews wrote:
>> If I perform a btrfs send receive like so:
>>
>> sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230
>> /mnt/everything/.snapshots/root.20191231 | btrfs receive
>> /mnt/backup/.snapshots/
>>
>> On Linux 5.4.0, the process completes successfully.
>>
>> Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the
>> result is the OOM killer being invoked which (among other process
>> carnage) kills the btrfs processes stopping the backup.
>
> As this is on the -rc1, it's possible that changes done in the MM
> subsystem could change the logic of OOM and that send now is able to
> trigger the OOM.
>
> There are 2 btrfs patches in send.c but they reduce amount of work,
> namely for heavily reflinked extents so the effects should be opposite.
>
> To find out where's the cause, is it possible that you build a kernel
> from the 5.4-based branch and run the send again? It's the same set of
> btrfs patches that gets merged to 5.5.
>
> From repository:
> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
> branch: for-5.5-rc4
>
> If this can't be done we'll find another way to debug it.
I've grabbed, compiled, and installed the kernel. I'll be able to tell
you the results of running the test tomorrow at about this time.
Thank you for help,
~Craig
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-13 14:48 ` Craig Andrews
@ 2020-01-15 18:44 ` Craig Andrews
0 siblings, 0 replies; 14+ messages in thread
From: Craig Andrews @ 2020-01-15 18:44 UTC (permalink / raw)
To: Craig Andrews; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]
On 2020-01-13 09:48, Craig Andrews wrote:
> On 2020-01-13 08:37, David Sterba wrote:
>> On Mon, Jan 13, 2020 at 07:41:13AM -0500, Craig Andrews wrote:
>>> If I perform a btrfs send receive like so:
>>>
>>> sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230
>>> /mnt/everything/.snapshots/root.20191231 | btrfs receive
>>> /mnt/backup/.snapshots/
>>>
>>> On Linux 5.4.0, the process completes successfully.
>>>
>>> Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5,
>>> the
>>> result is the OOM killer being invoked which (among other process
>>> carnage) kills the btrfs processes stopping the backup.
>>
>> As this is on the -rc1, it's possible that changes done in the MM
>> subsystem could change the logic of OOM and that send now is able to
>> trigger the OOM.
>>
>> There are 2 btrfs patches in send.c but they reduce amount of work,
>> namely for heavily reflinked extents so the effects should be
>> opposite.
>>
>> To find out where's the cause, is it possible that you build a kernel
>> from the 5.4-based branch and run the send again? It's the same set of
>> btrfs patches that gets merged to 5.5.
>>
>> From repository:
>> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
>> branch: for-5.5-rc4
>>
>> If this can't be done we'll find another way to debug it.
>
> I've grabbed, compiled, and installed the kernel. I'll be able to tell
> you the results of running the test tomorrow at about this time.
>
I couldn't get the kernel to boot so I need to spend more time on that.
Is there any info I can get that would be helpful using my current
kernels (either 5.4 or 5.5.0-rc5) while I work out why I can't boot the
kernel you linked me to?
Thanks,
~Craig
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-13 13:37 ` David Sterba
2020-01-13 14:48 ` Craig Andrews
@ 2020-01-21 2:18 ` Craig Andrews
2020-01-25 11:08 ` Stéphane Lesimple
2 siblings, 0 replies; 14+ messages in thread
From: Craig Andrews @ 2020-01-21 2:18 UTC (permalink / raw)
To: Craig Andrews; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]
On 2020-01-13 08:37, David Sterba wrote:
> On Mon, Jan 13, 2020 at 07:41:13AM -0500, Craig Andrews wrote:
>> If I perform a btrfs send receive like so:
>>
>> sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230
>> /mnt/everything/.snapshots/root.20191231 | btrfs receive
>> /mnt/backup/.snapshots/
>>
>> On Linux 5.4.0, the process completes successfully.
>>
>> Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the
>> result is the OOM killer being invoked which (among other process
>> carnage) kills the btrfs processes stopping the backup.
>
> As this is on the -rc1, it's possible that changes done in the MM
> subsystem could change the logic of OOM and that send now is able to
> trigger the OOM.
>
> There are 2 btrfs patches in send.c but they reduce amount of work,
> namely for heavily reflinked extents so the effects should be opposite.
>
> To find out where's the cause, is it possible that you build a kernel
> from the 5.4-based branch and run the send again? It's the same set of
> btrfs patches that gets merged to 5.5.
>
> From repository:
> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
> branch: for-5.5-rc4
>
> If this can't be done we'll find another way to debug it.
I finally was able to test with this kernel.
With the kernel compiled from the branch "for-5.5-rc4" I do not get the
out of memory error. However, I do this error:
btrfs send -p /mnt/everything/.snapshots/root.20200119
/mnt/everything/.snapshots/root.20200120 | mbuffer -v 1 | btrfs receive
/mnt/backup/.snapshots/
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid
argument
I rebooted then immediately ran the same command with the 5.4 kernel and
it completed successfully.
So I guess it's nice that I didn't get an out of memory error...
Thanks,
~Craig
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-13 13:37 ` David Sterba
2020-01-13 14:48 ` Craig Andrews
2020-01-21 2:18 ` Craig Andrews
@ 2020-01-25 11:08 ` Stéphane Lesimple
2020-01-27 13:44 ` Filipe Manana
2020-01-27 20:08 ` Stéphane Lesimple
2 siblings, 2 replies; 14+ messages in thread
From: Stéphane Lesimple @ 2020-01-25 11:08 UTC (permalink / raw)
To: Craig Andrews; +Cc: linux-btrfs
> ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
I can test patches if needed.
--
Stéphane.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-25 11:08 ` Stéphane Lesimple
@ 2020-01-27 13:44 ` Filipe Manana
2020-01-28 14:46 ` Craig Andrews
2020-01-27 20:08 ` Stéphane Lesimple
1 sibling, 1 reply; 14+ messages in thread
From: Filipe Manana @ 2020-01-27 13:44 UTC (permalink / raw)
To: Stéphane Lesimple; +Cc: Craig Andrews, linux-btrfs
On Sat, Jan 25, 2020 at 11:18 AM Stéphane Lesimple
<stephane_btrfs2@lesimple.fr> wrote:
>
> > ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
>
> If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
>
> On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
>
> # btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
> At subvol /tank/backups/.snaps/incoming/sendme/
> At subvol sendme
> ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
> ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
This is probably the same case for which I sent a fix last week:
https://patchwork.kernel.org/patch/11350129/
Thanks.
>
> The send/receive is still going on for now, currently around the ~200 Gb mark.
>
> Bees is running on this FS (but I stopped it before doing the send/receive).
>
> I can test patches if needed.
>
> --
> Stéphane.
--
Filipe David Manana,
“Whether you think you can, or you think you can't — you're right.”
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-25 11:08 ` Stéphane Lesimple
2020-01-27 13:44 ` Filipe Manana
@ 2020-01-27 20:08 ` Stéphane Lesimple
1 sibling, 0 replies; 14+ messages in thread
From: Stéphane Lesimple @ 2020-01-27 20:08 UTC (permalink / raw)
To: fdmanana; +Cc: Craig Andrews, linux-btrfs
>> # btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
>> At subvol /tank/backups/.snaps/incoming/sendme/
>> At subvol sendme
>> ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid
>> argument
>> ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid
>> argument
>
> This is probably the same case for which I sent a fix last week:
>
> https://patchwork.kernel.org/patch/11350129
This seems very likely, as there is indeed a hole in the middle of the file:
0: ram 80000 disk(0) 25871fb42000:00000000-00080000 disk_sz 80000
80000: ram 80000 disk(0) 258720475000:00000000-00080000 disk_sz 80000
100000: ram 80000 disk(0) 2587215d4000:00000000-00080000 disk_sz 80000
180000: ram 80000 disk(0) 258721654000:00000000-00080000 disk_sz 80000
200000: ram 80000 disk(0) 25872177c000:00000000-00080000 disk_sz 80000
280000: ram 80000 disk(0) 258722383000:00000000-00080000 disk_sz 80000
300000: ram 80000 disk(0) 258722544000:00000000-00080000 disk_sz 80000
380000: ram 80000 disk(0) 25871c3a8000:00000000-00080000 disk_sz 80000
400000: ram 80000 disk(0) 25871c428000:00000000-00080000 disk_sz 80000
480000: ram 80000 disk(0) 25871c548000:00000000-00080000 disk_sz 80000
500000: ram 80000 disk(0) 25871cf02000:00000000-00080000 disk_sz 80000
580000: ram 80000 disk(0) 25871cf82000:00000000-00080000 disk_sz 80000
600000: ram 80000 disk(0) 25871d41f000:00000000-00080000 disk_sz 80000
680000: ram 80000 disk(0) 25871d833000:00000000-00080000 disk_sz 80000
700000: ram b000 disk(0) b9e86aeb000:00000000-0000b000 disk_sz b000
70b000: ram 11000 disk(0) 0:00000000-00011000 disk_sz 0 -- hole
71c000: ram 80000 disk(0) 25871eb75000:00000000-00080000 disk_sz 80000
79c000: ram 1000 disk(0) a00b6523000:00000000-00001000 disk_sz 1000
79d000: ram 1000 disk(0) a00b6cf5000:00000000-00001000 disk_sz 1000
79e000: ram 1000 disk(0) a00b6cf5000:00000000-00001000 disk_sz 1000
79f000: ram 1000 disk(0) a00b6cf5000:00000000-00001000 disk_sz 1000
7a0000: ram 1000 disk(0) a00b6d42000:00000000-00001000 disk_sz 1000
7a1000: ram 1000 disk(0) a9573bfa000:00000000-00001000 disk_sz 6000
7a2000: ram 1000 disk(0) a9573bfa000:00000000-00001000 disk_sz 6000
7a3000: ram 6000 disk(0) a9573bfa000:00000000-00006000 disk_sz 6000
7a9000: ram 1000 disk(0) af4f6240000:00000000-00001000 disk_sz 2d000
7aa000: ram 1000 disk(0) af4f6240000:00000000-00001000 disk_sz 2d000
7ab000: ram 2d000 disk(0) af4f6240000:00000000-0002d000 disk_sz 2d000
7d8000: ram 1000 disk(0) 1c8d313ba000:0001d000-0001e000 disk_sz 80000
7d9000: ram 43000 disk(0) a983253d000:00000000-00043000 disk_sz 43000
81c000: ram 80000 disk(0) 258721f18000:00000000-00080000 disk_sz 80000
89c000: ram 80000 disk(0) 258721f98000:00000000-00080000 disk_sz 80000
91c000: ram 80000 disk(0) 258722018000:00000000-00080000 disk_sz 80000
99c000: ram 80000 disk(0) 2587229a2000:00000000-00080000 disk_sz 80000
a1c000: ram d000 disk(0) bfa6df44000:00000000-0000d000 disk_sz e000
a29000: ram 1000 disk(0) 130963b99000:00a29000-00a2a000 disk_sz a2a000
file: mednafen_saturn_libretro.dll extents 29 disk size 21741568 logical size 10653836 ratio 0.49
Thanks!
--
Stéphane.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-27 13:44 ` Filipe Manana
@ 2020-01-28 14:46 ` Craig Andrews
2020-01-28 15:02 ` Filipe Manana
0 siblings, 1 reply; 14+ messages in thread
From: Craig Andrews @ 2020-01-28 14:46 UTC (permalink / raw)
To: fdmanana; +Cc: Stéphane Lesimple, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]
On 2020-01-27 08:44, Filipe Manana wrote:
> On Sat, Jan 25, 2020 at 11:18 AM Stéphane Lesimple
> <stephane_btrfs2@lesimple.fr> wrote:
>>
>> > ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
>>
>> If I may add another data point here, I'm also encountering this issue
>> on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as
>> btrfs is concerned, this is an rc7).
>>
>> On the first time, it happened after sending ~90 Gb worth of data, and
>> aborted (as I didn't specify the -E option to btrfs send). Then, I
>> retried with btrfs send -E 0, and it encountered the exact same error
>> on the same file.
>>
>> # btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
>> 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
>> At subvol /tank/backups/.snaps/incoming/sendme/
>> At subvol sendme
>> ERROR: failed to clone extents to
>> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
>> ERROR: failed to clone extents to
>> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
>
> This is probably the same case for which I sent a fix last week:
>
> https://patchwork.kernel.org/patch/11350129/
>
> Thanks.
I applied that patch to 5.5.0-rc7 and that solved the problem. I can now
do backups (which is a send/receive) successfully.
>
>>
>> The send/receive is still going on for now, currently around the ~200
>> Gb mark.
>>
>> Bees is running on this FS (but I stopped it before doing the
>> send/receive).
>>
>> I can test patches if needed.
>>
>> --
>> Stéphane.
The patch appears to have fixed my problems - thank you!
~Craig
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-28 14:46 ` Craig Andrews
@ 2020-01-28 15:02 ` Filipe Manana
2020-01-28 15:05 ` Craig Andrews
0 siblings, 1 reply; 14+ messages in thread
From: Filipe Manana @ 2020-01-28 15:02 UTC (permalink / raw)
To: Craig Andrews; +Cc: Stéphane Lesimple, linux-btrfs
On Tue, Jan 28, 2020 at 2:46 PM Craig Andrews <candrews@integralblue.com> wrote:
>
> On 2020-01-27 08:44, Filipe Manana wrote:
> > On Sat, Jan 25, 2020 at 11:18 AM Stéphane Lesimple
> > <stephane_btrfs2@lesimple.fr> wrote:
> >>
> >> > ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
> >>
> >> If I may add another data point here, I'm also encountering this issue
> >> on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as
> >> btrfs is concerned, this is an rc7).
> >>
> >> On the first time, it happened after sending ~90 Gb worth of data, and
> >> aborted (as I didn't specify the -E option to btrfs send). Then, I
> >> retried with btrfs send -E 0, and it encountered the exact same error
> >> on the same file.
> >>
> >> # btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
> >> 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
> >> At subvol /tank/backups/.snaps/incoming/sendme/
> >> At subvol sendme
> >> ERROR: failed to clone extents to
> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
> >> ERROR: failed to clone extents to
> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
> >
> > This is probably the same case for which I sent a fix last week:
> >
> > https://patchwork.kernel.org/patch/11350129/
> >
> > Thanks.
>
> I applied that patch to 5.5.0-rc7 and that solved the problem. I can now
> do backups (which is a send/receive) successfully.
>
> >
> >>
> >> The send/receive is still going on for now, currently around the ~200
> >> Gb mark.
> >>
> >> Bees is running on this FS (but I stopped it before doing the
> >> send/receive).
> >>
> >> I can test patches if needed.
> >>
> >> --
> >> Stéphane.
>
> The patch appears to have fixed my problems - thank you!
Great!
Can you reply to the patch's thread with a:
Tested-by: Your Name <email@foo.bar>
?
Or I can reply to it myself with that if you agree.
Thanks!
> ~Craig
--
Filipe David Manana,
“Whether you think you can, or you think you can't — you're right.”
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-28 15:02 ` Filipe Manana
@ 2020-01-28 15:05 ` Craig Andrews
2020-01-28 15:08 ` Filipe Manana
2020-01-28 17:31 ` David Sterba
0 siblings, 2 replies; 14+ messages in thread
From: Craig Andrews @ 2020-01-28 15:05 UTC (permalink / raw)
To: fdmanana; +Cc: Stéphane Lesimple, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2183 bytes --]
On 2020-01-28 10:02, Filipe Manana wrote:
> On Tue, Jan 28, 2020 at 2:46 PM Craig Andrews
> <candrews@integralblue.com> wrote:
>>
>> On 2020-01-27 08:44, Filipe Manana wrote:
>> > On Sat, Jan 25, 2020 at 11:18 AM Stéphane Lesimple
>> > <stephane_btrfs2@lesimple.fr> wrote:
>> >>
>> >> > ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
>> >>
>> >> If I may add another data point here, I'm also encountering this issue
>> >> on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as
>> >> btrfs is concerned, this is an rc7).
>> >>
>> >> On the first time, it happened after sending ~90 Gb worth of data, and
>> >> aborted (as I didn't specify the -E option to btrfs send). Then, I
>> >> retried with btrfs send -E 0, and it encountered the exact same error
>> >> on the same file.
>> >>
>> >> # btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
>> >> 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
>> >> At subvol /tank/backups/.snaps/incoming/sendme/
>> >> At subvol sendme
>> >> ERROR: failed to clone extents to
>> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
>> >> ERROR: failed to clone extents to
>> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
>> >
>> > This is probably the same case for which I sent a fix last week:
>> >
>> > https://patchwork.kernel.org/patch/11350129/
>> >
>> > Thanks.
>>
>> I applied that patch to 5.5.0-rc7 and that solved the problem. I can
>> now
>> do backups (which is a send/receive) successfully.
>>
>> >
>> >>
>> >> The send/receive is still going on for now, currently around the ~200
>> >> Gb mark.
>> >>
>> >> Bees is running on this FS (but I stopped it before doing the
>> >> send/receive).
>> >>
>> >> I can test patches if needed.
>> >>
>> >> --
>> >> Stéphane.
>>
>> The patch appears to have fixed my problems - thank you!
>
> Great!
>
> Can you reply to the patch's thread with a:
>
> Tested-by: Your Name <email@foo.bar>
>
> ?
>
> Or I can reply to it myself with that if you agree.
>
> Thanks!
>
>> ~Craig
Can you please send the reply?
Tested-by: Craig Andrews <candrews@integralblue.com>
And again, thank you!
~Craig
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-28 15:05 ` Craig Andrews
@ 2020-01-28 15:08 ` Filipe Manana
2020-01-28 17:33 ` Filipe Manana
2020-01-28 17:31 ` David Sterba
1 sibling, 1 reply; 14+ messages in thread
From: Filipe Manana @ 2020-01-28 15:08 UTC (permalink / raw)
To: Craig Andrews; +Cc: Stéphane Lesimple, linux-btrfs
On Tue, Jan 28, 2020 at 3:05 PM Craig Andrews <candrews@integralblue.com> wrote:
>
> On 2020-01-28 10:02, Filipe Manana wrote:
> > On Tue, Jan 28, 2020 at 2:46 PM Craig Andrews
> > <candrews@integralblue.com> wrote:
> >>
> >> On 2020-01-27 08:44, Filipe Manana wrote:
> >> > On Sat, Jan 25, 2020 at 11:18 AM Stéphane Lesimple
> >> > <stephane_btrfs2@lesimple.fr> wrote:
> >> >>
> >> >> > ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
> >> >>
> >> >> If I may add another data point here, I'm also encountering this issue
> >> >> on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as
> >> >> btrfs is concerned, this is an rc7).
> >> >>
> >> >> On the first time, it happened after sending ~90 Gb worth of data, and
> >> >> aborted (as I didn't specify the -E option to btrfs send). Then, I
> >> >> retried with btrfs send -E 0, and it encountered the exact same error
> >> >> on the same file.
> >> >>
> >> >> # btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
> >> >> 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
> >> >> At subvol /tank/backups/.snaps/incoming/sendme/
> >> >> At subvol sendme
> >> >> ERROR: failed to clone extents to
> >> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
> >> >> ERROR: failed to clone extents to
> >> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
> >> >
> >> > This is probably the same case for which I sent a fix last week:
> >> >
> >> > https://patchwork.kernel.org/patch/11350129/
> >> >
> >> > Thanks.
> >>
> >> I applied that patch to 5.5.0-rc7 and that solved the problem. I can
> >> now
> >> do backups (which is a send/receive) successfully.
> >>
> >> >
> >> >>
> >> >> The send/receive is still going on for now, currently around the ~200
> >> >> Gb mark.
> >> >>
> >> >> Bees is running on this FS (but I stopped it before doing the
> >> >> send/receive).
> >> >>
> >> >> I can test patches if needed.
> >> >>
> >> >> --
> >> >> Stéphane.
> >>
> >> The patch appears to have fixed my problems - thank you!
> >
> > Great!
> >
> > Can you reply to the patch's thread with a:
> >
> > Tested-by: Your Name <email@foo.bar>
> >
> > ?
> >
> > Or I can reply to it myself with that if you agree.
> >
> > Thanks!
> >
> >> ~Craig
>
>
> Can you please send the reply?
> Tested-by: Craig Andrews <candrews@integralblue.com>
No problem, done. Thanks.
>
> And again, thank you!
> ~Craig
--
Filipe David Manana,
“Whether you think you can, or you think you can't — you're right.”
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-28 15:05 ` Craig Andrews
2020-01-28 15:08 ` Filipe Manana
@ 2020-01-28 17:31 ` David Sterba
1 sibling, 0 replies; 14+ messages in thread
From: David Sterba @ 2020-01-28 17:31 UTC (permalink / raw)
To: Craig Andrews; +Cc: fdmanana, Stéphane Lesimple, linux-btrfs
On Tue, Jan 28, 2020 at 10:05:37AM -0500, Craig Andrews wrote:
> Can you please send the reply?
> Tested-by: Craig Andrews <candrews@integralblue.com>
Thanks for the report and testing! As the patch is a fix it will go to
mainline soon and then it'll propagate to stable trees, 5.4 and 5.5.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
2020-01-28 15:08 ` Filipe Manana
@ 2020-01-28 17:33 ` Filipe Manana
0 siblings, 0 replies; 14+ messages in thread
From: Filipe Manana @ 2020-01-28 17:33 UTC (permalink / raw)
To: Craig Andrews; +Cc: Stéphane Lesimple, linux-btrfs
On Tue, Jan 28, 2020 at 3:08 PM Filipe Manana <fdmanana@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 3:05 PM Craig Andrews <candrews@integralblue.com> wrote:
> >
> > On 2020-01-28 10:02, Filipe Manana wrote:
> > > On Tue, Jan 28, 2020 at 2:46 PM Craig Andrews
> > > <candrews@integralblue.com> wrote:
> > >>
> > >> On 2020-01-27 08:44, Filipe Manana wrote:
> > >> > On Sat, Jan 25, 2020 at 11:18 AM Stéphane Lesimple
> > >> > <stephane_btrfs2@lesimple.fr> wrote:
> > >> >>
> > >> >> > ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
> > >> >>
> > >> >> If I may add another data point here, I'm also encountering this issue
> > >> >> on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as
> > >> >> btrfs is concerned, this is an rc7).
> > >> >>
> > >> >> On the first time, it happened after sending ~90 Gb worth of data, and
> > >> >> aborted (as I didn't specify the -E option to btrfs send). Then, I
> > >> >> retried with btrfs send -E 0, and it encountered the exact same error
> > >> >> on the same file.
> > >> >>
> > >> >> # btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
> > >> >> 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
> > >> >> At subvol /tank/backups/.snaps/incoming/sendme/
> > >> >> At subvol sendme
> > >> >> ERROR: failed to clone extents to
> > >> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
> > >> >> ERROR: failed to clone extents to
> > >> >> retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
> > >> >
> > >> > This is probably the same case for which I sent a fix last week:
> > >> >
> > >> > https://patchwork.kernel.org/patch/11350129/
> > >> >
> > >> > Thanks.
> > >>
> > >> I applied that patch to 5.5.0-rc7 and that solved the problem. I can
> > >> now
> > >> do backups (which is a send/receive) successfully.
> > >>
> > >> >
> > >> >>
> > >> >> The send/receive is still going on for now, currently around the ~200
> > >> >> Gb mark.
> > >> >>
> > >> >> Bees is running on this FS (but I stopped it before doing the
> > >> >> send/receive).
> > >> >>
> > >> >> I can test patches if needed.
> > >> >>
> > >> >> --
> > >> >> Stéphane.
> > >>
> > >> The patch appears to have fixed my problems - thank you!
> > >
> > > Great!
> > >
> > > Can you reply to the patch's thread with a:
> > >
> > > Tested-by: Your Name <email@foo.bar>
> > >
> > > ?
> > >
> > > Or I can reply to it myself with that if you agree.
> > >
> > > Thanks!
> > >
> > >> ~Craig
> >
> >
> > Can you please send the reply?
> > Tested-by: Craig Andrews <candrews@integralblue.com>
>
> No problem, done. Thanks.
Actually I just realized one chunk of the patch can cause problems
with fallocated extents beyond a file's size and a hole that ends
right at the file's size (unlikely to be a common scenario anyway).
I also don't think it's needed. For my tests at least, removing it
doesn't cause any new problems and solves the problem with the invalid
clone operations.
Can you try it for your use case too?
To be clear, this is the new version of the patch:
https://pastebin.com/raw/vewZiG9J
(A shorter version of the original)
Thanks!
>
> >
> > And again, thank you!
> > ~Craig
>
>
>
> --
> Filipe David Manana,
>
> “Whether you think you can, or you think you can't — you're right.”
--
Filipe David Manana,
“Whether you think you can, or you think you can't — you're right.”
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2020-01-28 17:33 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-13 12:41 Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory Craig Andrews
2020-01-13 13:37 ` David Sterba
2020-01-13 14:48 ` Craig Andrews
2020-01-15 18:44 ` Craig Andrews
2020-01-21 2:18 ` Craig Andrews
2020-01-25 11:08 ` Stéphane Lesimple
2020-01-27 13:44 ` Filipe Manana
2020-01-28 14:46 ` Craig Andrews
2020-01-28 15:02 ` Filipe Manana
2020-01-28 15:05 ` Craig Andrews
2020-01-28 15:08 ` Filipe Manana
2020-01-28 17:33 ` Filipe Manana
2020-01-28 17:31 ` David Sterba
2020-01-27 20:08 ` Stéphane Lesimple
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.