Strange behavior (possible bugs) in btrfs

Strange behavior (possible bugs) in btrfs

[Vijay Chidambaram]

Hi,

We found two more cases where the btrfs behavior is a little strange.
In one case, an fsync-ed file goes missing after a crash. In the
other, a renamed file shows up in both directories after a crash.

Workload 1:

mkdir A
mkdir B
mkdir A/C
creat B/foo
fsync B/foo
link B/foo A/C/foo
fsync A
-- crash --

Expected state after recovery:
B B/foo A A/C exist

What we find:
Only B B/foo exist

A is lost even after explicit fsync to A.

Workload 2:

mkdir A
mkdir A/C
rename A/C B
touch B/bar
fsync B/bar
rename B/bar A/bar
rename A B (replacing B with A at this point)
fsync B/bar
-- crash --

Expected contents after recovery:
A/bar

What we find after recovery:
A/bar
B/bar

We think this breaks rename's atomicity guarantee. bar should be
present in either A or B, but now it is present in both.

Thanks,
Vijay

Re: Strange behavior (possible bugs) in btrfs

[Jeff Mahoney]

On 4/30/18 12:04 PM, Vijay Chidambaram wrote:
> Hi,
> 
> We found two more cases where the btrfs behavior is a little strange.
> In one case, an fsync-ed file goes missing after a crash. In the
> other, a renamed file shows up in both directories after a crash.

Hi Vijay -

What kernel version did you observe these with?  These seem like bugs
Filipe has already fixed.

-Jeff


> Workload 1:
> 
> mkdir A
> mkdir B
> mkdir A/C
> creat B/foo
> fsync B/foo
> link B/foo A/C/foo
> fsync A
> -- crash --
> 
> Expected state after recovery:
> B B/foo A A/C exist
> 
> What we find:
> Only B B/foo exist
> 
> A is lost even after explicit fsync to A.
> 
> Workload 2:
> 
> mkdir A
> mkdir A/C
> rename A/C B
> touch B/bar
> fsync B/bar
> rename B/bar A/bar
> rename A B (replacing B with A at this point)
> fsync B/bar
> -- crash --
> 
> Expected contents after recovery:
> A/bar
> 
> What we find after recovery:
> A/bar
> B/bar
> 
> We think this breaks rename's atomicity guarantee. bar should be
> present in either A or B, but now it is present in both.
> 
> Thanks,
> Vijay
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Jeff Mahoney
SUSE Labs

Re: Strange behavior (possible bugs) in btrfs

[Jayashree Mohan]

Hi Jeff,

On Mon, Apr 30, 2018 at 11:51 AM, Jeff Mahoney <jeffm@suse.com> wrote:
> On 4/30/18 12:04 PM, Vijay Chidambaram wrote:
>> Hi,
>>
>> We found two more cases where the btrfs behavior is a little strange.
>> In one case, an fsync-ed file goes missing after a crash. In the
>> other, a renamed file shows up in both directories after a crash.
>
> Hi Vijay -
>
> What kernel version did you observe these with?  These seem like bugs
> Filipe has already fixed.
>
> -Jeff
>

These bugs were observed on Kernel 4.16.0-041600-generic.

Thanks,
Jayashree

Re: Strange behavior (possible bugs) in btrfs

[Vijay Chidambaram]

On Mon, Apr 30, 2018 at 11:56 AM, Jayashree Mohan
<jayashree2912@gmail.com> wrote:
> Hi Jeff,
>
> On Mon, Apr 30, 2018 at 11:51 AM, Jeff Mahoney <jeffm@suse.com> wrote:
>> On 4/30/18 12:04 PM, Vijay Chidambaram wrote:
>>> Hi,
>>>
>>> We found two more cases where the btrfs behavior is a little strange.
>>> In one case, an fsync-ed file goes missing after a crash. In the
>>> other, a renamed file shows up in both directories after a crash.
>>
>> Hi Vijay -
>>
>> What kernel version did you observe these with?  These seem like bugs
>> Filipe has already fixed.
>>
>> -Jeff
>>
>
> These bugs were observed on Kernel 4.16.0-041600-generic.
>

To add on to what Jayashree said, these bugs are still present in the
latest stable release of btrfs. We aren't sure if they have been
patched on a development branch.

We should note that similar bugs have been fixed before in btrfs, but
those patches don't seem to handle these cases.

Re: Strange behavior (possible bugs) in btrfs

[Filipe Manana]

On Mon, Apr 30, 2018 at 5:04 PM, Vijay Chidambaram <vvijay03@gmail.com> wrote:
> Hi,
>
> We found two more cases where the btrfs behavior is a little strange.
> In one case, an fsync-ed file goes missing after a crash. In the
> other, a renamed file shows up in both directories after a crash.
>
> Workload 1:
>
> mkdir A
> mkdir B
> mkdir A/C
> creat B/foo
> fsync B/foo
> link B/foo A/C/foo
> fsync A
> -- crash --
>
> Expected state after recovery:
> B B/foo A A/C exist
>
> What we find:
> Only B B/foo exist
>
> A is lost even after explicit fsync to A.
>
> Workload 2:
>
> mkdir A
> mkdir A/C
> rename A/C B
> touch B/bar
> fsync B/bar
> rename B/bar A/bar
> rename A B (replacing B with A at this point)
> fsync B/bar
> -- crash --
>
> Expected contents after recovery:
> A/bar
>
> What we find after recovery:
> A/bar
> B/bar
>
> We think this breaks rename's atomicity guarantee. bar should be
> present in either A or B, but now it is present in both.

I'll take a look at these, and all the other potential issues you
reported in other threads, next week and let you know.
Thanks.

>
> Thanks,
> Vijay
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: Strange behavior (possible bugs) in btrfs

[Jayashree Mohan]

Hi Filipe,

This is to follow up the status of crash consistency bugs we reported
on btrfs. We see that there has been a patch(not in the kernel yet)
(https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg77875.html)
that resolves one of the reported bugs. However, the other bugs we
reported still exist on the latest kernel (4.19-rc1), even with the
submitted patch. Here is the list of other inconsistencies we
reported, along with the workload to reproduce them :
https://www.spinics.net/lists/linux-btrfs/msg77219.html

We just wanted to ensure that resolving these are on your to-do list.
Additionally, if there are more patches queued to address these
issues, please let us know.

Thanks,
Jayashree Mohan

Thanks,
Jayashree Mohan



On Fri, May 11, 2018 at 10:45 AM Filipe Manana <fdmanana@gmail.com> wrote:
>
> On Mon, Apr 30, 2018 at 5:04 PM, Vijay Chidambaram <vvijay03@gmail.com> wrote:
> > Hi,
> >
> > We found two more cases where the btrfs behavior is a little strange.
> > In one case, an fsync-ed file goes missing after a crash. In the
> > other, a renamed file shows up in both directories after a crash.
> >
> > Workload 1:
> >
> > mkdir A
> > mkdir B
> > mkdir A/C
> > creat B/foo
> > fsync B/foo
> > link B/foo A/C/foo
> > fsync A
> > -- crash --
> >
> > Expected state after recovery:
> > B B/foo A A/C exist
> >
> > What we find:
> > Only B B/foo exist
> >
> > A is lost even after explicit fsync to A.
> >
> > Workload 2:
> >
> > mkdir A
> > mkdir A/C
> > rename A/C B
> > touch B/bar
> > fsync B/bar
> > rename B/bar A/bar
> > rename A B (replacing B with A at this point)
> > fsync B/bar
> > -- crash --
> >
> > Expected contents after recovery:
> > A/bar
> >
> > What we find after recovery:
> > A/bar
> > B/bar
> >
> > We think this breaks rename's atomicity guarantee. bar should be
> > present in either A or B, but now it is present in both.
>
> I'll take a look at these, and all the other potential issues you
> reported in other threads, next week and let you know.
> Thanks.
>
> >
> > Thanks,
> > Vijay
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Filipe David Manana,
>
> “Whether you think you can, or you think you can't — you're right.”

Re: Strange behavior (possible bugs) in btrfs

[Filipe Manana]

On Tue, Aug 28, 2018 at 9:35 PM Jayashree Mohan <jayashree2912@gmail.com> wrote:
>
> Hi Filipe,
>
> This is to follow up the status of crash consistency bugs we reported
> on btrfs. We see that there has been a patch(not in the kernel yet)
> (https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg77875.html)
> that resolves one of the reported bugs. However, the other bugs we
> reported still exist on the latest kernel (4.19-rc1), even with the
> submitted patch. Here is the list of other inconsistencies we
> reported, along with the workload to reproduce them :
> https://www.spinics.net/lists/linux-btrfs/msg77219.html
>
> We just wanted to ensure that resolving these are on your to-do list.
> Additionally, if there are more patches queued to address these
> issues, please let us know.

Hi,

I go through the issues as time allows. Not all of these are top
priorities for me right now.
Been working on a fix for some of them but they are not yet ready to
submit (need more testing, or cause other problems or are too
complex).
If suddenly there are people hitting any of these issues frequently
and causing trouble I'll give them higher priority.

Thanks.

>
> Thanks,
> Jayashree Mohan
>
> Thanks,
> Jayashree Mohan
>
>
>
> On Fri, May 11, 2018 at 10:45 AM Filipe Manana <fdmanana@gmail.com> wrote:
> >
> > On Mon, Apr 30, 2018 at 5:04 PM, Vijay Chidambaram <vvijay03@gmail.com> wrote:
> > > Hi,
> > >
> > > We found two more cases where the btrfs behavior is a little strange.
> > > In one case, an fsync-ed file goes missing after a crash. In the
> > > other, a renamed file shows up in both directories after a crash.
> > >
> > > Workload 1:
> > >
> > > mkdir A
> > > mkdir B
> > > mkdir A/C
> > > creat B/foo
> > > fsync B/foo
> > > link B/foo A/C/foo
> > > fsync A
> > > -- crash --
> > >
> > > Expected state after recovery:
> > > B B/foo A A/C exist
> > >
> > > What we find:
> > > Only B B/foo exist
> > >
> > > A is lost even after explicit fsync to A.
> > >
> > > Workload 2:
> > >
> > > mkdir A
> > > mkdir A/C
> > > rename A/C B
> > > touch B/bar
> > > fsync B/bar
> > > rename B/bar A/bar
> > > rename A B (replacing B with A at this point)
> > > fsync B/bar
> > > -- crash --
> > >
> > > Expected contents after recovery:
> > > A/bar
> > >
> > > What we find after recovery:
> > > A/bar
> > > B/bar
> > >
> > > We think this breaks rename's atomicity guarantee. bar should be
> > > present in either A or B, but now it is present in both.
> >
> > I'll take a look at these, and all the other potential issues you
> > reported in other threads, next week and let you know.
> > Thanks.
> >
> > >
> > > Thanks,
> > > Vijay
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> >
> > --
> > Filipe David Manana,
> >
> > “Whether you think you can, or you think you can't — you're right.”



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: Strange behavior (possible bugs) in btrfs

[Filipe Manana]

On Mon, Apr 30, 2018 at 5:04 PM, Vijay Chidambaram <vvijay03@gmail.com> wrote:
> Hi,
>
> We found two more cases where the btrfs behavior is a little strange.
> In one case, an fsync-ed file goes missing after a crash. In the
> other, a renamed file shows up in both directories after a crash.
>
> Workload 1:
>
> mkdir A
> mkdir B
> mkdir A/C
> creat B/foo
> fsync B/foo
> link B/foo A/C/foo
> fsync A
> -- crash --
>
> Expected state after recovery:
> B B/foo A A/C exist

Why don't you expect A/C/foo as well? I would expect it to be persisted.
With xfs we don't get A/C/foo persisted, but it's persisted with ext4 and f2fs.

Adding xfs folks in cc to confirm the expected behaviour.

>
> What we find:
> Only B B/foo exist
>
> A is lost even after explicit fsync to A.
>
> Workload 2:
>
> mkdir A
> mkdir A/C
> rename A/C B
> touch B/bar
> fsync B/bar
> rename B/bar A/bar
> rename A B (replacing B with A at this point)
> fsync B/bar
> -- crash --
>
> Expected contents after recovery:
> A/bar
>
> What we find after recovery:
> A/bar
> B/bar
>
> We think this breaks rename's atomicity guarantee. bar should be
> present in either A or B, but now it is present in both.
>
> Thanks,
> Vijay
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”