Select DUP metadata by default on single devices.

Select DUP metadata by default on single devices.

[Forza]

Hello everyone,

I'd like to revisit the topic I opened on Github(*) a year ago, where I suggested that DUP metadata profile ought to be the default choice when doing mkfs.btrfs on single devices. 

Today we have much better write endurance on flash based media so the added writes should not matter in the grand scheme of things. Another factor is disk encryption where mkfs.btrfs cannot differentiate a plain SSD from a luks/dm-crypt device. Encryption effectively removes the possibility for the SSD to dedupe the metadata blocks. 

Ultimately, I think it is better to favour defaults that gives most users better fault tolerance, rather than using SINGLE mode for everyone because of the chance that some have deduplicating hardware (which would potentially negate the benefit of DUP metadata). 

One remark against DUP has been that both metadata copies would end up in the same erase block. However, I think that full erase block failures are in minority of the possible failure modes, at least from what I've seen on the mailing list and at #btrfs. It is more common to have single block errors, and for those we are protected with DUP metadata. 

Zygo made a very good in-depth explanation about several different failure modes in the Github issue. 

I would like voice my wish to change the defaults to DUP metadata on all single devices and I hope that the developers now can find consensus to make this change. 

* https://github.com/kdave/btrfs-progs/issues/319

Thanks. 

~ Forza 

Re: Select DUP metadata by default on single devices.

[Qu Wenruo]

On 2021/9/19 05:38, Forza wrote:
> Hello everyone,
>
> I'd like to revisit the topic I opened on Github(*) a year ago, where I suggested that DUP metadata profile ought to be the default choice when doing mkfs.btrfs on single devices.
>
> Today we have much better write endurance on flash based media so the added writes should not matter in the grand scheme of things. Another factor is disk encryption where mkfs.btrfs cannot differentiate a plain SSD from a luks/dm-crypt device. Encryption effectively removes the possibility for the SSD to dedupe the metadata blocks.
>
> Ultimately, I think it is better to favour defaults that gives most users better fault tolerance, rather than using SINGLE mode for everyone because of the chance that some have deduplicating hardware (which would potentially negate the benefit of DUP metadata).
>
> One remark against DUP has been that both metadata copies would end up in the same erase block. However, I think that full erase block failures are in minority of the possible failure modes, at least from what I've seen on the mailing list and at #btrfs. It is more common to have single block errors, and for those we are protected with DUP metadata.
>
> Zygo made a very good in-depth explanation about several different failure modes in the Github issue.
>
> I would like voice my wish to change the defaults to DUP metadata on all single devices and I hope that the developers now can find consensus to make this change.

I'm totally into the idea of DUP as default meta.

The idea that *some* SSD does dedupe internally shouldn't be a reason to
prevent us from using DUP at all.

The internal mechanism should not affect how we use the disks,
especially we didn't even have a solid statistics on the percentage of
SSDs doing that.

The initial idea of such exception is already arguable to me at least.


And such exception is already causing damage in the wild, thus I see no
real benefit from SINGLE metadata on SSD.

I hope we can get more developers agree on this.

Thanks,
Qu

>
> * https://github.com/kdave/btrfs-progs/issues/319
>
> Thanks.
>
> ~ Forza
>

Re: Select DUP metadata by default on single devices.

[Chris Murphy]

On Sat, Sep 18, 2021 at 5:58 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/19 05:38, Forza wrote:
> > Hello everyone,
> >
> > I'd like to revisit the topic I opened on Github(*) a year ago, where I suggested that DUP metadata profile ought to be the default choice when doing mkfs.btrfs on single devices.
> >
> > Today we have much better write endurance on flash based media so the added writes should not matter in the grand scheme of things. Another factor is disk encryption where mkfs.btrfs cannot differentiate a plain SSD from a luks/dm-crypt device. Encryption effectively removes the possibility for the SSD to dedupe the metadata blocks.
> >
> > Ultimately, I think it is better to favour defaults that gives most users better fault tolerance, rather than using SINGLE mode for everyone because of the chance that some have deduplicating hardware (which would potentially negate the benefit of DUP metadata).
> >
> > One remark against DUP has been that both metadata copies would end up in the same erase block. However, I think that full erase block failures are in minority of the possible failure modes, at least from what I've seen on the mailing list and at #btrfs. It is more common to have single block errors, and for those we are protected with DUP metadata.
> >
> > Zygo made a very good in-depth explanation about several different failure modes in the Github issue.
> >
> > I would like voice my wish to change the defaults to DUP metadata on all single devices and I hope that the developers now can find consensus to make this change.
>
> I'm totally into the idea of DUP as default meta.
>
> The idea that *some* SSD does dedupe internally shouldn't be a reason to
> prevent us from using DUP at all.
>
> The internal mechanism should not affect how we use the disks,
> especially we didn't even have a solid statistics on the percentage of
> SSDs doing that.

Also, if DUP is default, and both copies are corrupted, that will be
evidence of a double whammy against that SSD make/model: it's
corrupting data, and it's deduping. The dedup is in this case,
effectively, replicating corruption.


> And such exception is already causing damage in the wild, thus I see no
> real benefit from SINGLE metadata on SSD.

It'd be nice to quantify the write amplification of btrfs cow
(wandering trees) in the general case. The metadata is a teeny portion
of writes, but still useful to quantify how big of a hit it is to
double the metadata writes. It seems to be the only downside.


-- 
Chris Murphy

Re: Select DUP metadata by default on single devices.

[Su Yue]

On Sat 18 Sep 2021 at 23:38, Forza <forza@tnonline.net> wrote:

> Hello everyone,
>
> I'd like to revisit the topic I opened on Github(*) a year ago, 
> where I
> suggested that DUP metadata profile ought to be the default 
> choice when doing
> mkfs.btrfs on single devices.
>
> Today we have much better write endurance on flash based media 
> so the added
> writes should not matter in the grand scheme of things. Another 
> factor is disk
> encryption where mkfs.btrfs cannot differentiate a plain SSD 
> from a
> luks/dm-crypt device. Encryption effectively removes the 
> possibility for the SSD
> to dedupe the metadata blocks.
>
> Ultimately, I think it is better to favour defaults that gives 
> most users better
> fault tolerance, rather than using SINGLE mode for everyone 
> because of the
> chance that some have deduplicating hardware (which would 
> potentially negate the
> benefit of DUP metadata).
>
> One remark against DUP has been that both metadata copies would 
> end up in the
> same erase block. However, I think that full erase block 
> failures are in
> minority of the possible failure modes, at least from what I've 
> seen on the
> mailing list and at #btrfs. It is more common to have single 
> block errors, and
> for those we are protected with DUP metadata.
>
> Zygo made a very good in-depth explanation about several 
> different failure modes in the Github issue.
>
> I would like voice my wish to change the defaults to DUP 
> metadata on all single devices and I hope that the developers 
> now can find consensus to make this change.
>

I'd vote for the idea. It may change some users' scripts doing
mkfs.btrfs but may save others' critical data.

And hope guys running filesystem benchmarks regularly notice the 
change
rather than saying 'Btrfs became slower' :)

--
Su
> * https://github.com/kdave/btrfs-progs/issues/319
>
> Thanks.
>
> ~ Forza

Re: Select DUP metadata by default on single devices.

[David Sterba]

On Sat, Sep 18, 2021 at 11:38:54PM +0200, Forza wrote:
> Hello everyone,
> 
> I'd like to revisit the topic I opened on Github(*) a year ago, where
> I suggested that DUP metadata profile ought to be the default choice
> when doing mkfs.btrfs on single devices. 
> 
> Today we have much better write endurance on flash based media so the
> added writes should not matter in the grand scheme of things. Another
> factor is disk encryption where mkfs.btrfs cannot differentiate a
> plain SSD from a luks/dm-crypt device. Encryption effectively removes
> the possibility for the SSD to dedupe the metadata blocks. 
> 
> Ultimately, I think it is better to favour defaults that gives most
> users better fault tolerance, rather than using SINGLE mode for
> everyone because of the chance that some have deduplicating hardware
> (which would potentially negate the benefit of DUP metadata). 
> 
> One remark against DUP has been that both metadata copies would end up
> in the same erase block. However, I think that full erase block
> failures are in minority of the possible failure modes, at least from
> what I've seen on the mailing list and at #btrfs. It is more common to
> have single block errors, and for those we are protected with DUP
> metadata. 
> 
> Zygo made a very good in-depth explanation about several different
> failure modes in the Github issue. 
> 
> I would like voice my wish to change the defaults to DUP metadata on
> all single devices and I hope that the developers now can find
> consensus to make this change. 

I agree with the change, there's enough evidence for it.

The test was not done for SSDs specifically but to see if a device is
'rotational' as noted by sysfs file. This also caught NBDs or other
block devices that were emulated or hidden under some other layer.

This brings some usability issues, we may perhaps want to add a warning
that the device is detected as non-rotational, but that's all I see we
can do right now.

There was a discussion somewhere about a mode or command that would try
to guess what kind of device is it and suggest best options. That would
mean to examine the device vendor or look if it's eg. dm-crypt or
network-backed device or whatever. That should help distro partitioning
tools to suggest more appropriate options for mkfs, as there's
opposition to tweak the defaults.

Regarding the timing, 5.15 at the earliest and I think we can do a big
defaults update. Some of the features have been out for a long time and
got enabled manually.

So:
- DUP by default for metadata on single device
- single by default for data on multiple devices (now it's raid0)
- free-space-tree on
- no-holes on

I'd vote for one version doing the whole switch rather than doing the
changes by one.

Re: Select DUP metadata by default on single devices.

[Qu Wenruo]

On 2021/9/20 17:09, David Sterba wrote:
> On Sat, Sep 18, 2021 at 11:38:54PM +0200, Forza wrote:
>> Hello everyone,
>>
>> I'd like to revisit the topic I opened on Github(*) a year ago, where
>> I suggested that DUP metadata profile ought to be the default choice
>> when doing mkfs.btrfs on single devices.
>>
>> Today we have much better write endurance on flash based media so the
>> added writes should not matter in the grand scheme of things. Another
>> factor is disk encryption where mkfs.btrfs cannot differentiate a
>> plain SSD from a luks/dm-crypt device. Encryption effectively removes
>> the possibility for the SSD to dedupe the metadata blocks.
>>
>> Ultimately, I think it is better to favour defaults that gives most
>> users better fault tolerance, rather than using SINGLE mode for
>> everyone because of the chance that some have deduplicating hardware
>> (which would potentially negate the benefit of DUP metadata).
>>
>> One remark against DUP has been that both metadata copies would end up
>> in the same erase block. However, I think that full erase block
>> failures are in minority of the possible failure modes, at least from
>> what I've seen on the mailing list and at #btrfs. It is more common to
>> have single block errors, and for those we are protected with DUP
>> metadata.
>>
>> Zygo made a very good in-depth explanation about several different
>> failure modes in the Github issue.
>>
>> I would like voice my wish to change the defaults to DUP metadata on
>> all single devices and I hope that the developers now can find
>> consensus to make this change.
>
> I agree with the change, there's enough evidence for it.
>
> The test was not done for SSDs specifically but to see if a device is
> 'rotational' as noted by sysfs file. This also caught NBDs or other
> block devices that were emulated or hidden under some other layer.
>
> This brings some usability issues, we may perhaps want to add a warning
> that the device is detected as non-rotational, but that's all I see we
> can do right now.
>
> There was a discussion somewhere about a mode or command that would try
> to guess what kind of device is it and suggest best options. That would
> mean to examine the device vendor or look if it's eg. dm-crypt or
> network-backed device or whatever. That should help distro partitioning
> tools to suggest more appropriate options for mkfs, as there's
> opposition to tweak the defaults.
>
> Regarding the timing, 5.15 at the earliest and I think we can do a big
> defaults update. Some of the features have been out for a long time and
> got enabled manually.
>
> So:
> - DUP by default for metadata on single device

Awesome.

> - single by default for data on multiple devices (now it's raid0)

Is there any discussion/thread on that part?

As I'm not that aware about this.

> - free-space-tree on
> - no-holes on
>
> I'd vote for one version doing the whole switch rather than doing the
> changes by one.
>

I'm fine on DUP and FST.

On no-holes I'd prefer more feedback from Filipe as he has exposed some
no-holes related problem some time ago.

Thanks,
Qu

Re: Select DUP metadata by default on single devices.

[David Sterba]

On Mon, Sep 20, 2021 at 06:46:31PM +0800, Qu Wenruo wrote:
> > - single by default for data on multiple devices (now it's raid0)
> 
> Is there any discussion/thread on that part?
> 
> As I'm not that aware about this.

It's been discussed on IRC long time ago. The problem with raid0 is that
it's a striped profile and changing to another profile may be
problematic once all the chunk space is alloated. Unlike for single
where it's on just one device and converting to anything is easy.

I think that for multi-device fs everybody specifies the profiles
manually anyway, but the defaults should be sane.

> > - free-space-tree on
> > - no-holes on
> >
> > I'd vote for one version doing the whole switch rather than doing the
> > changes by one.
> 
> I'm fine on DUP and FST.
> 
> On no-holes I'd prefer more feedback from Filipe as he has exposed some
> no-holes related problem some time ago.

Yeah I know Filipe is not all happy about making no-holes default, it
removes some capabilities of 'check' to verify extents. Some bugs have
been fixed recently (kernel side) but that's IMO on par with regular bug
hunting & fixing.

I don't know when would be the perfect time to make no-holes dfault,
it's a feature to reduce metadata consumption which is IMO worth as FST
will increase it again. Backward compatibility is good, lots of stable
versions support it.

Re: Select DUP metadata by default on single devices.

[Qu Wenruo]

On 2021/9/20 19:23, David Sterba wrote:
> On Mon, Sep 20, 2021 at 06:46:31PM +0800, Qu Wenruo wrote:
>>> - single by default for data on multiple devices (now it's raid0)
>>
>> Is there any discussion/thread on that part?
>>
>> As I'm not that aware about this.
>
> It's been discussed on IRC long time ago. The problem with raid0 is that
> it's a striped profile and changing to another profile may be
> problematic once all the chunk space is alloated. Unlike for single
> where it's on just one device and converting to anything is easy.
>
> I think that for multi-device fs everybody specifies the profiles
> manually anyway, but the defaults should be sane.

OK, that makes completely sense.

Then I have no concern for the switch so far.

Thanks,
Qu

>
>>> - free-space-tree on
>>> - no-holes on
>>>
>>> I'd vote for one version doing the whole switch rather than doing the
>>> changes by one.
>>
>> I'm fine on DUP and FST.
>>
>> On no-holes I'd prefer more feedback from Filipe as he has exposed some
>> no-holes related problem some time ago.
>
> Yeah I know Filipe is not all happy about making no-holes default, it
> removes some capabilities of 'check' to verify extents. Some bugs have
> been fixed recently (kernel side) but that's IMO on par with regular bug
> hunting & fixing.
>
> I don't know when would be the perfect time to make no-holes dfault,
> it's a feature to reduce metadata consumption which is IMO worth as FST
> will increase it again. Backward compatibility is good, lots of stable
> versions support it.
>

Re: Select DUP metadata by default on single devices.

[David Sterba]

On Mon, Sep 20, 2021 at 07:26:38PM +0800, Qu Wenruo wrote:
> 
> 
> On 2021/9/20 19:23, David Sterba wrote:
> > On Mon, Sep 20, 2021 at 06:46:31PM +0800, Qu Wenruo wrote:
> >>> - single by default for data on multiple devices (now it's raid0)
> >>
> >> Is there any discussion/thread on that part?
> >>
> >> As I'm not that aware about this.
> >
> > It's been discussed on IRC long time ago. The problem with raid0 is that
> > it's a striped profile and changing to another profile may be
> > problematic once all the chunk space is alloated. Unlike for single
> > where it's on just one device and converting to anything is easy.
> >
> > I think that for multi-device fs everybody specifies the profiles
> > manually anyway, but the defaults should be sane.
> 
> OK, that makes completely sense.
> 
> Then I have no concern for the switch so far.

Now tracked as

https://github.com/kdave/btrfs-progs/issues/406

Re: Select DUP metadata by default on single devices.

[David Sterba]

On Mon, Sep 20, 2021 at 06:46:31PM +0800, Qu Wenruo wrote:
> > - no-holes on
> >
> > I'd vote for one version doing the whole switch rather than doing the
> > changes by one.
> 
> On no-holes I'd prefer more feedback from Filipe as he has exposed some
> no-holes related problem some time ago.

Tracked as

https://github.com/kdave/btrfs-progs/issues/405

Re: Select DUP metadata by default on single devices.

[Steven Davies]

On 2021-09-20 10:09, David Sterba wrote:

> So:
> - DUP by default for metadata on single device
> - single by default for data on multiple devices (now it's raid0)
> - free-space-tree on

Is this synonymous with space_cache=v2?

> - no-holes on
> 
> I'd vote for one version doing the whole switch rather than doing the
> changes by one.

-- 
Steven Davies

Re: Select DUP metadata by default on single devices.

[David Sterba]

On Mon, Sep 20, 2021 at 12:12:33PM +0100, Steven Davies wrote:
> On 2021-09-20 10:09, David Sterba wrote:
> 
> > So:
> > - DUP by default for metadata on single device
> > - single by default for data on multiple devices (now it's raid0)
> > - free-space-tree on
> 
> Is this synonymous with space_cache=v2?

Yes. To reduce the mount option number I wanted it to be called v2, the
free-space-tree is the implementation, we use that interchangably
though.