Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

[Andy Smith]

Hi Cedric,

On Wed, Feb 03, 2021 at 08:33:18PM +0100,   wrote:
> it's called "dm-integrity", as mentioned in this e-mail:
> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg93037.html

If you do this it would be very interesting to see performance
figures for the following setups:

- btrfs with raid1 meta and data allocation
- mdadm raid1 on raw devices
- mdadm raid1 on dm-integrity (no encryption) on raw devices
- mdadm raid1 on dm-integrity (encryption) on raw devices

just to see what kind of performance loss dm-integrity and
encryption is going to impose.

After doing it, it would find a nice home on the Linux RAID wiki:

    https://raid.wiki.kernel.org/index.php/Dm-integrity

Cheers,
Andy

Re: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

--- Ursprüngliche Nachricht ---
Von: Andy Smith <andy@strugglers.net>
Datum: 04.02.2021 11:54:57
An: linux-btrfs@vger.kernel.org
Betreff: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs  does, what's that called?

Hi Cedric,

On Wed, Feb 03, 2021 at 08:33:18PM +0100,   wrote:
> it's called "dm-integrity", as mentioned in this e-mail:
> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg93037.html


If you do this it would be very interesting to see performance
figures for the following setups:

- btrfs with raid1 meta and data allocation
- mdadm raid1 on raw devices
- mdadm raid1 on dm-integrity (no encryption) on raw devices
- mdadm raid1 on dm-integrity (encryption) on raw devices

just to see what kind of performance loss dm-integrity and
encryption is going to impose.

After doing it, it would find a nice home on the Linux RAID wiki:

    https://raid.wiki.kernel.org/index.php/Dm-integrity

Cheers,
Andy

Hey Andy,

I would rather see performance figures for these setups:
A) btrfs with 2 (or more) hard drives and one SSD in writeback bcache configuration (unsafe against failure of the ssd):
+-----------------------------+
|      btrfs raid 1 /mnt      |
+--------------+--------------+
| /dev/Bcache0 | /dev/Bcache1 |
+--------------+--------------+
|   bcache writeback Cache    |
|           /dev/sdk1         |
+--------------+--------------+
| Data         | Data         |
| /dev/sdv1    | /dev/sdw1    |
+--------------+--------------+

B) btrfs with 2 (or more) hard drives and two SSD's in dm-raid 1 writeback bcache configuration (unsafe against corruption of any of the ssd's): 
+-----------------------------+
|      btrfs raid 1 /mnt      |
+--------------+--------------+
| /dev/Bcache0 | /dev/Bcache1 |
+--------------+--------------+
|   bcache writeback Cache    |
|           /dev/dm0          |
+--------------+--------------+
| 2x SSD in mdadm raid 1      |
| /dev/sdk1       /dev/sdl1   |
+--------------+--------------+
| Data         | Data         |
| /dev/sdv1    | /dev/sdw1    |
+--------------+--------------+

C) Full stack: btrfs with 2 (or more) hard drives and two identical SSD's in dm-raid 1 with dm-integrity writeback bcache configuration (safe against any failed drive):
+-----------------------------+
|      btrfs raid 1 /mnt      |
+--------------+--------------+
| /dev/Bcache0 | /dev/Bcache1 |
+--------------+--------------+
|   bcache writeback Cache    |
|           /dev/dm0          |
+--------------+--------------+
| 2 x dm-integrity devices    |
| in mdadm raid 1             |
+--------------+--------------+
| SSD hosting  | SSD hosting  |
| dm-integrity | dm-integrity |
| /dev/sdk1    | /dev/sdl1    |
+--------------+--------------+
| Data         | Data         |
| /dev/sdv1    | /dev/sdw1    |
+--------------+--------------+

D) Full stack: btrfs with 2 (or more) hard drives and two SSD's (one slow, and one very fast) in dm-raid 1 with dm-integrity writeback bcache configuration (safe against any failed drive):
+-----------------------------+
|      btrfs raid 1 /mnt      |
+--------------+--------------+
| /dev/Bcache0 | /dev/Bcache1 |
+--------------+--------------+
|   bcache writeback Cache    |
|           /dev/dm0          |
+--------------+--------------+
| 2 x dm-integrity devices    |
| in mdadm raid 1             |
+--------------+--------------+
| SSD hosting  | SSD hosting  |
| dm-integrity | dm-integrity |
| /dev/sdk1    | /dev/sdl1    |
+--------------+--------------+
| Data         | Data         |
| /dev/sdv1    | /dev/sdw1    |
+--------------+--------------+

In all these setups, the performance of the hard drives is irrelevant, because the speed of the setups comes from the bcache SSD.

Cheers,
Cedric

---

Take your mailboxes with you. Free, fast and secure Mail &amp; Cloud: https://www.eclipso.eu - Time to change!

Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

[Goffredo Baroncelli]

[...]
> Hey Andy,
> 
> I would rather see performance figures for these setups:
> A) btrfs with 2 (or more) hard drives and one SSD in writeback bcache configuration (unsafe against failure of the ssd):
> +-----------------------------+
> |      btrfs raid 1 /mnt      |
> +--------------+--------------+
> | /dev/Bcache0 | /dev/Bcache1 |
> +--------------+--------------+
> |   bcache writeback Cache    |
> |           /dev/sdk1         |
> +--------------+--------------+
> | Data         | Data         |
> | /dev/sdv1    | /dev/sdw1    |
> +--------------+--------------+

Doing that, you loose the protection of raid1 redundancy: now there is a single point of failure /dev/sdk1. Writeback is even more dangerous...

> 
> B) btrfs with 2 (or more) hard drives and two SSD's in dm-raid 1 writeback bcache configuration (unsafe against corruption of any of the ssd's):
> +-----------------------------+
> |      btrfs raid 1 /mnt      |
> +--------------+--------------+
> | /dev/Bcache0 | /dev/Bcache1 |
> +--------------+--------------+
> |   bcache writeback Cache    |
> |           /dev/dm0          |
> +--------------+--------------+
> | 2x SSD in mdadm raid 1      |
> | /dev/sdk1       /dev/sdl1   |
> +--------------+--------------+
> | Data         | Data         |
> | /dev/sdv1    | /dev/sdw1    |
> +--------------+--------------+
> 
> C) Full stack: btrfs with 2 (or more) hard drives and two identical SSD's in dm-raid 1 with dm-integrity writeback bcache configuration (safe against any failed drive):
> +-----------------------------+
> |      btrfs raid 1 /mnt      |
> +--------------+--------------+
> | /dev/Bcache0 | /dev/Bcache1 |
> +--------------+--------------+
> |   bcache writeback Cache    |
> |           /dev/dm0          |
> +--------------+--------------+
> | 2 x dm-integrity devices    |
> | in mdadm raid 1             |
> +--------------+--------------+
> | SSD hosting  | SSD hosting  |
> | dm-integrity | dm-integrity |
> | /dev/sdk1    | /dev/sdl1    |
> +--------------+--------------+
> | Data         | Data         |
> | /dev/sdv1    | /dev/sdw1    |
> +--------------+--------------+
> 
> D) Full stack: btrfs with 2 (or more) hard drives and two SSD's (one slow, and one very fast) in dm-raid 1 with dm-integrity writeback bcache configuration (safe against any failed drive):
> +-----------------------------+
> |      btrfs raid 1 /mnt      |
> +--------------+--------------+
> | /dev/Bcache0 | /dev/Bcache1 |
> +--------------+--------------+
> |   bcache writeback Cache    |
> |           /dev/dm0          |
> +--------------+--------------+
> | 2 x dm-integrity devices    |
> | in mdadm raid 1             |
> +--------------+--------------+
> | SSD hosting  | SSD hosting  |
> | dm-integrity | dm-integrity |
> | /dev/sdk1    | /dev/sdl1    |
> +--------------+--------------+
> | Data         | Data         |
> | /dev/sdv1    | /dev/sdw1    |
> +--------------+--------------+
> 
> In all these setups, the performance of the hard drives is irrelevant, because the speed of the setups comes from the bcache SSD.
> 
> Cheers,
> Cedric
> 
> ---
> 
> Take your mailboxes with you. Free, fast and secure Mail & Cloud: https://www.eclipso.eu - Time to change!
> 
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

--- Ursprüngliche Nachricht ---
Von: Goffredo Baroncelli <kreijack@libero.it>
Datum: 04.02.2021 19:13:50
An: Cedric.dewijs@eclipso.eu, Andy Smith <andy@strugglers.net>
Betreff: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs  does, what's that called?

[...]
> Hey Andy,
> 
> I would rather see performance figures for these setups:
> A) btrfs with 2 (or more) hard drives and one SSD in writeback bcache
configuration (unsafe against failure of the ssd):
> +-----------------------------+
> |      btrfs raid 1 /mnt      |
> +--------------+--------------+
> | /dev/Bcache0 | /dev/Bcache1 |
> +--------------+--------------+
> |   bcache writeback Cache    |
> |           /dev/sdk1         |
> +--------------+--------------+
> | Data         | Data         |
> | /dev/sdv1    | /dev/sdw1    |
> +--------------+--------------+

Doing that, you loose the protection of raid1 redundancy: now there is a
single point of failure /dev/sdk1. Writeback is even more dangerous...


Not really. if bcache is set to read cache, the SSD can die at any moment, without btrfs loosing any data. All written data has gone straight to the hard drives. I have not tried this scenario, but I would be very surprised if reading the data from /mnt is even interrupted for longer than a few seconds if the data cable from the ssd is pulled while data is written from another process.

You are correct about writeback cache, if /dev/sdk1 dies, all dirty data is lost, and even worse, both copies of the btrfs data are side by side on only the SSD. (But I already mentioned this in my previous mail: "unsafe against failure of the ssd")

Cheers,
Cedric

---

Take your mailboxes with you. Free, fast and secure Mail &amp; Cloud: https://www.eclipso.eu - Time to change!