btrfs receive deadlock and questions

btrfs receive deadlock and questions

[Eli V]

I seem to have it a deadlock trying out btrfs send & receive. Now I
haven't used btrfs send & receive much, so don't have much experience
with them. Anyways, bug report and stack traces:
https://bugzilla.kernel.org/show_bug.cgi?id=202383

Seems like the receive is hung as well as several kworkers. It's about
1.2TB into a 9TB or so transfer onto a brand new pretty empty fs. This
is just a btrfs send snapshot, not an incremental. That was supposed
to come next. If this was an rsync based backup I'd just kill the
rsync process and restart it, not sure if there's a way to restart a
btrfs send receive, or if I'd have to delete the partially created
snapshot on the destination and restart the send. I guess I could just
use rsync to finish the copy of the initial snapshot as well before
using send | receive for the incrementals. Thoughts and options would
be appreciated, thanks.

-Eli

Re: btrfs receive deadlock and questions

[Filipe Manana]

On Tue, Jan 22, 2019 at 3:26 PM Eli V <eliventer@gmail.com> wrote:
>
> I seem to have it a deadlock trying out btrfs send & receive. Now I
> haven't used btrfs send & receive much, so don't have much experience
> with them. Anyways, bug report and stack traces:
> https://bugzilla.kernel.org/show_bug.cgi?id=202383

This is the same you reported at:
https://bugzilla.kernel.org/show_bug.cgi?id=199753

It just happens through a different path, unrelated to send/receive.
You are running a 4.19.16 kernel, which doesn't have the fix [1]:

$ git tag --contains 5ce555578e0919237fa4bda92b4670e2dd176f85
v4.20
v4.20-rc1
v4.20-rc2
v4.20-rc3
v4.20-rc4
v4.20-rc5
v4.20-rc6
v4.20-rc7
v4.20.1
v4.20.2
v4.20.3
v5.0-rc1
v5.0-rc2
v5.0-rc3

All the deadlock problems you reported are fixed by [1] and [2].
The second, related to the free space tree, is very recent and only on 5.0-rcs:

$ git tag --contains a6d8654d885d7d79a3fb82da64eaa489ca332a82
v5.0-rc2
v5.0-rc3

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ce555578e0919237fa4bda92b4670e2dd176f85
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a6d8654d885d7d79a3fb82da64eaa489ca332a82


>
> Seems like the receive is hung as well as several kworkers. It's about
> 1.2TB into a 9TB or so transfer onto a brand new pretty empty fs. This
> is just a btrfs send snapshot, not an incremental. That was supposed
> to come next. If this was an rsync based backup I'd just kill the
> rsync process and restart it, not sure if there's a way to restart a
> btrfs send receive, or if I'd have to delete the partially created
> snapshot on the destination and restart the send. I guess I could just
> use rsync to finish the copy of the initial snapshot as well before
> using send | receive for the incrementals. Thoughts and options would
> be appreciated, thanks.
>
> -Eli



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: btrfs receive deadlock and questions

[Eli V]

On Tue, Jan 22, 2019 at 10:42 AM Filipe Manana <fdmanana@gmail.com> wrote:
>
> On Tue, Jan 22, 2019 at 3:26 PM Eli V <eliventer@gmail.com> wrote:
> >
> > I seem to have it a deadlock trying out btrfs send & receive. Now I
> > haven't used btrfs send & receive much, so don't have much experience
> > with them. Anyways, bug report and stack traces:
> > https://bugzilla.kernel.org/show_bug.cgi?id=202383
>
> This is the same you reported at:
> https://bugzilla.kernel.org/show_bug.cgi?id=199753
>
> It just happens through a different path, unrelated to send/receive.
> You are running a 4.19.16 kernel, which doesn't have the fix [1]:
>
> $ git tag --contains 5ce555578e0919237fa4bda92b4670e2dd176f85
> v4.20
> v4.20-rc1
> v4.20-rc2
> v4.20-rc3
> v4.20-rc4
> v4.20-rc5
> v4.20-rc6
> v4.20-rc7
> v4.20.1
> v4.20.2
> v4.20.3
> v5.0-rc1
> v5.0-rc2
> v5.0-rc3
>
> All the deadlock problems you reported are fixed by [1] and [2].
> The second, related to the free space tree, is very recent and only on 5.0-rcs:
>
> $ git tag --contains a6d8654d885d7d79a3fb82da64eaa489ca332a82
> v5.0-rc2
> v5.0-rc3
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ce555578e0919237fa4bda92b4670e2dd176f85
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a6d8654d885d7d79a3fb82da64eaa489ca332a82
>
>

Sounds good, thanks for the links. I thought the stack traces looked
different, thus the 2 different reports. I guess no further info is
needed from the hung tasks and I can start killing it and figuring out
how to resume the process.


> >
> > Seems like the receive is hung as well as several kworkers. It's about
> > 1.2TB into a 9TB or so transfer onto a brand new pretty empty fs. This
> > is just a btrfs send snapshot, not an incremental. That was supposed
> > to come next. If this was an rsync based backup I'd just kill the
> > rsync process and restart it, not sure if there's a way to restart a
> > btrfs send receive, or if I'd have to delete the partially created
> > snapshot on the destination and restart the send. I guess I could just
> > use rsync to finish the copy of the initial snapshot as well before
> > using send | receive for the incrementals. Thoughts and options would
> > be appreciated, thanks.
> >
> > -Eli
>
>
>
> --
> Filipe David Manana,
>
> “Whether you think you can, or you think you can't — you're right.”

Re: btrfs receive deadlock and questions

[Eli V]

On Tue, Jan 22, 2019 at 11:03 AM Eli V <eliventer@gmail.com> wrote:
>
> On Tue, Jan 22, 2019 at 10:42 AM Filipe Manana <fdmanana@gmail.com> wrote:
> >
> > On Tue, Jan 22, 2019 at 3:26 PM Eli V <eliventer@gmail.com> wrote:
> > >
> > > I seem to have it a deadlock trying out btrfs send & receive. Now I
> > > haven't used btrfs send & receive much, so don't have much experience
> > > with them. Anyways, bug report and stack traces:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=202383
> >
> > This is the same you reported at:
> > https://bugzilla.kernel.org/show_bug.cgi?id=199753
> >
> > It just happens through a different path, unrelated to send/receive.
> > You are running a 4.19.16 kernel, which doesn't have the fix [1]:
> >
> > $ git tag --contains 5ce555578e0919237fa4bda92b4670e2dd176f85
> > v4.20
> > v4.20-rc1
> > v4.20-rc2
> > v4.20-rc3
> > v4.20-rc4
> > v4.20-rc5
> > v4.20-rc6
> > v4.20-rc7
> > v4.20.1
> > v4.20.2
> > v4.20.3
> > v5.0-rc1
> > v5.0-rc2
> > v5.0-rc3
> >
> > All the deadlock problems you reported are fixed by [1] and [2].
> > The second, related to the free space tree, is very recent and only on 5.0-rcs:
> >
> > $ git tag --contains a6d8654d885d7d79a3fb82da64eaa489ca332a82
> > v5.0-rc2
> > v5.0-rc3
> >
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ce555578e0919237fa4bda92b4670e2dd176f85
> > [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a6d8654d885d7d79a3fb82da64eaa489ca332a82
> >

Hmm, as far as I can tell these are both already in 4.19.16, according
to the linux-stable git repo as:

d7068618ae1fbb80fc16bd7c58798e208a696483
ea9c846f54dbf03da159a1b7566aa95e9bf1674b

So I guess that would mean the btrfs receive deadlock i.e. kernel
bugzilla 202383, isn't fixed by these commits.


>
> Sounds good, thanks for the links. I thought the stack traces looked
> different, thus the 2 different reports. I guess no further info is
> needed from the hung tasks and I can start killing it and figuring out
> how to resume the process.
>
>
> > >
> > > Seems like the receive is hung as well as several kworkers. It's about
> > > 1.2TB into a 9TB or so transfer onto a brand new pretty empty fs. This
> > > is just a btrfs send snapshot, not an incremental. That was supposed
> > > to come next. If this was an rsync based backup I'd just kill the
> > > rsync process and restart it, not sure if there's a way to restart a
> > > btrfs send receive, or if I'd have to delete the partially created
> > > snapshot on the destination and restart the send. I guess I could just
> > > use rsync to finish the copy of the initial snapshot as well before
> > > using send | receive for the incrementals. Thoughts and options would
> > > be appreciated, thanks.
> > >
> > > -Eli
> >
> >
> >
> > --
> > Filipe David Manana,
> >
> > “Whether you think you can, or you think you can't — you're right.”

Re: btrfs receive deadlock and questions

[Filipe Manana]

On Tue, Jan 22, 2019 at 5:20 PM Eli V <eliventer@gmail.com> wrote:
>
> On Tue, Jan 22, 2019 at 11:03 AM Eli V <eliventer@gmail.com> wrote:
> >
> > On Tue, Jan 22, 2019 at 10:42 AM Filipe Manana <fdmanana@gmail.com> wrote:
> > >
> > > On Tue, Jan 22, 2019 at 3:26 PM Eli V <eliventer@gmail.com> wrote:
> > > >
> > > > I seem to have it a deadlock trying out btrfs send & receive. Now I
> > > > haven't used btrfs send & receive much, so don't have much experience
> > > > with them. Anyways, bug report and stack traces:
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=202383
> > >
> > > This is the same you reported at:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=199753
> > >
> > > It just happens through a different path, unrelated to send/receive.
> > > You are running a 4.19.16 kernel, which doesn't have the fix [1]:
> > >
> > > $ git tag --contains 5ce555578e0919237fa4bda92b4670e2dd176f85
> > > v4.20
> > > v4.20-rc1
> > > v4.20-rc2
> > > v4.20-rc3
> > > v4.20-rc4
> > > v4.20-rc5
> > > v4.20-rc6
> > > v4.20-rc7
> > > v4.20.1
> > > v4.20.2
> > > v4.20.3
> > > v5.0-rc1
> > > v5.0-rc2
> > > v5.0-rc3
> > >
> > > All the deadlock problems you reported are fixed by [1] and [2].
> > > The second, related to the free space tree, is very recent and only on 5.0-rcs:
> > >
> > > $ git tag --contains a6d8654d885d7d79a3fb82da64eaa489ca332a82
> > > v5.0-rc2
> > > v5.0-rc3
> > >
> > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ce555578e0919237fa4bda92b4670e2dd176f85
> > > [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a6d8654d885d7d79a3fb82da64eaa489ca332a82
> > >
>
> Hmm, as far as I can tell these are both already in 4.19.16, according
> to the linux-stable git repo as:
>
> d7068618ae1fbb80fc16bd7c58798e208a696483
> ea9c846f54dbf03da159a1b7566aa95e9bf1674b
>
> So I guess that would mean the btrfs receive deadlock i.e. kernel
> bugzilla 202383, isn't fixed by these commits.

So there are a few different paths that lead to the same deadlock and
I missed them before.
I'll got those fixed in the next few days.

Btw, it's not related to send/receive at all, it can happen from anywhere.

Thanks for the report.

>
>
> >
> > Sounds good, thanks for the links. I thought the stack traces looked
> > different, thus the 2 different reports. I guess no further info is
> > needed from the hung tasks and I can start killing it and figuring out
> > how to resume the process.
> >
> >
> > > >
> > > > Seems like the receive is hung as well as several kworkers. It's about
> > > > 1.2TB into a 9TB or so transfer onto a brand new pretty empty fs. This
> > > > is just a btrfs send snapshot, not an incremental. That was supposed
> > > > to come next. If this was an rsync based backup I'd just kill the
> > > > rsync process and restart it, not sure if there's a way to restart a
> > > > btrfs send receive, or if I'd have to delete the partially created
> > > > snapshot on the destination and restart the send. I guess I could just
> > > > use rsync to finish the copy of the initial snapshot as well before
> > > > using send | receive for the incrementals. Thoughts and options would
> > > > be appreciated, thanks.
> > > >
> > > > -Eli
> > >
> > >
> > >
> > > --
> > > Filipe David Manana,
> > >
> > > “Whether you think you can, or you think you can't — you're right.”



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”