RE: btrfs and checksum

RE: btrfs and checksum

[Jorge Fernandez Monteagudo]

Hi Qu, thanks for answering!

>RO of the ext4, and still get corruption? Definitely looks like a
>hardware problem to me.

It's a weird error, because rebooting the machine the previous error disapears and the file could be read ok, then we know that the info in disk is ok. Then, the suspects list grows up: disk, RAM memory, kernel error (block layer?, filesystem layer?).... It's not a frequent bug so it's difficult to debug it.


>For btrfs, as long as you're using data csum (default), btrfs can detect
>such corruption.
>
>For v5.2 kernel, btrfs can even detects some easy to expose memory
>corruption.
>
>But please keep in mind that, due to the fact btrfs (at least least
>version) is very picky about corrupted on-disk data or memory, if you
>find something wrong, you need to check dmesg to see what's going wrong.
>
>Furthermore, if your ssd is not reliable, especially when it lies about
>FLUSH/FUA, btrfs can be easier to be corrupted, as btrfs completely
>relies on FLUSH/FUA and metadata COW to ensure its safety against
>powerloss, it's way easier to get corrupted if FLUSH/FUA is not
>implemented corrected.
>
>(On the other hand, btrfs is more robust against data corruption, so as
>long as your SSD is OK, you may find a better experience using btrfs)

Then, it's advised to change the ext4 to btrfs or it's better to change the ISO packages filesytem to btrfs?

Thanks!
Jorge

Re: btrfs and checksum

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1746 bytes --]



On 2019/9/4 下午2:08, Jorge Fernandez Monteagudo wrote:
> Hi Qu, thanks for answering!
> 
>> RO of the ext4, and still get corruption? Definitely looks like a
>> hardware problem to me.
> 
> It's a weird error, because rebooting the machine the previous error disapears and the file could be read ok, then we know that the info in disk is ok. Then, the suspects list grows up: disk, RAM memory, kernel error (block layer?, filesystem layer?).... It's not a frequent bug so it's difficult to debug it.

I'd say, other than trying migrating to btrfs, it's better to locate the
problem.

Without a concrete cause, migrating to btrfs may also have some weird
behavior.

Thanks,
Qu

> 
> 
>> For btrfs, as long as you're using data csum (default), btrfs can detect
>> such corruption.
>>
>> For v5.2 kernel, btrfs can even detects some easy to expose memory
>> corruption.
>>
>> But please keep in mind that, due to the fact btrfs (at least least
>> version) is very picky about corrupted on-disk data or memory, if you
>> find something wrong, you need to check dmesg to see what's going wrong.
>>
>> Furthermore, if your ssd is not reliable, especially when it lies about
>> FLUSH/FUA, btrfs can be easier to be corrupted, as btrfs completely
>> relies on FLUSH/FUA and metadata COW to ensure its safety against
>> powerloss, it's way easier to get corrupted if FLUSH/FUA is not
>> implemented corrected.
>>
>> (On the other hand, btrfs is more robust against data corruption, so as
>> long as your SSD is OK, you may find a better experience using btrfs)
> 
> Then, it's advised to change the ext4 to btrfs or it's better to change the ISO packages filesytem to btrfs?


> 
> Thanks!
> Jorge
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: btrfs and checksum

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2030 bytes --]



On 2019/9/3 下午9:50, Jorge Fernandez Monteagudo wrote:
>  Hi all,
> 
> This is my first message in this list. I'm looking for advice because I have a weird silent data corruption and maybe with btrfs and the checksum capability this could be, at least, notified to the system, and detected using some btrfs-utils tool or dmesg.
> 
> Imagine you have an SSD with an EXT4 partition in READ ONLY mode and inside several files.

RO of the ext4, and still get corruption? Definitely looks like a
hardware problem to me.

> Every file is an ISO filesystem crypted with cryptsetup. Then first, the EXT4 partition in mounted then every file is mounted using losetup and crytpsetup. We have found, sometimes reading a PNG or WAV file from any of the ISO filesystems mounted, we get an error because the data is incorrect. Flushing caches and trying again don't solve the error. Maybe we have a faulty disk controller and changing the filesystem could be useful, or a RAM with some error...

For btrfs, as long as you're using data csum (default), btrfs can detect
such corruption.

For v5.2 kernel, btrfs can even detects some easy to expose memory
corruption.

But please keep in mind that, due to the fact btrfs (at least least
version) is very picky about corrupted on-disk data or memory, if you
find something wrong, you need to check dmesg to see what's going wrong.

Furthermore, if your ssd is not reliable, especially when it lies about
FLUSH/FUA, btrfs can be easier to be corrupted, as btrfs completely
relies on FLUSH/FUA and metadata COW to ensure its safety against
powerloss, it's way easier to get corrupted if FLUSH/FUA is not
implemented corrected.

(On the other hand, btrfs is more robust against data corruption, so as
long as your SSD is OK, you may find a better experience using btrfs)

Thanks,
Qu

> 
> Changing the EXT4 to btrfs is enough to enable the checksum property or we have to change every ISO file to btrfs to enable the checksum?
> 
> Thanks!
> Jorge
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

btrfs and checksum

[Jorge Fernandez Monteagudo]

 Hi all,

This is my first message in this list. I'm looking for advice because I have a weird silent data corruption and maybe with btrfs and the checksum capability this could be, at least, notified to the system, and detected using some btrfs-utils tool or dmesg.

Imagine you have an SSD with an EXT4 partition in READ ONLY mode and inside several files. Every file is an ISO filesystem crypted with cryptsetup. Then first, the EXT4 partition in mounted then every file is mounted using losetup and crytpsetup. We have found, sometimes reading a PNG or WAV file from any of the ISO filesystems mounted, we get an error because the data is incorrect. Flushing caches and trying again don't solve the error. Maybe we have a faulty disk controller and changing the filesystem could be useful, or a RAM with some error...

Changing the EXT4 to btrfs is enough to enable the checksum property or we have to change every ISO file to btrfs to enable the checksum?

Thanks!
Jorge