Question about btrfs and XOR offloading

Question about btrfs and XOR offloading

[Rosen Penev]

I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
know if there is a performance advantage to using a hardware
accelerated path? I ask as I use BTRFS on a Marvelll ARM platform with
XOR offload capability.

Re: Question about btrfs and XOR offloading

[Qu Wenruo]

On 2021/1/3 上午11:50, Rosen Penev wrote:
> I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
> know if there is a performance advantage to using a hardware
> accelerated path? I ask as I use BTRFS on a Marvelll ARM platform with
> XOR offload capability.
>
AFAIK XOR is only utilized by RAID56, while RAID56 is not considered
safe due to write-hole, thus I don't believe whether btrfs supports
hardware XOR would make any difference for now.

Thanks,
Qu

Re: Question about btrfs and XOR offloading

[Rosen Penev]

On Sat, Jan 2, 2021 at 10:53 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/1/3 上午11:50, Rosen Penev wrote:
> > I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
> > know if there is a performance advantage to using a hardware
> > accelerated path? I ask as I use BTRFS on a Marvelll ARM platform with
> > XOR offload capability.
> >
> AFAIK XOR is only utilized by RAID56, while RAID56 is not considered
> safe due to write-hole, thus I don't believe whether btrfs supports
> hardware XOR would make any difference for now.
Right the question is about performance. I don't know if XOR being
async would make any difference.
>
> Thanks,
> Qu

Re: Question about btrfs and XOR offloading

[David Sterba]

On Sat, Jan 02, 2021 at 07:50:38PM -0800, Rosen Penev wrote:
> I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
> know if there is a performance advantage to using a hardware
> accelerated path? I ask as I use BTRFS on a Marvelll ARM platform with
> XOR offload capability.

Even if it's CPU, it's accelerated and best algorithm is selected at
boot time:

[   16.357703] raid6: avx2x4   gen() 30635 MB/s
[   16.425701] raid6: avx2x4   xor() 10727 MB/s
[   16.493701] raid6: avx2x2   gen() 32995 MB/s
[   16.561701] raid6: avx2x2   xor() 19596 MB/s
[   16.629701] raid6: avx2x1   gen() 26349 MB/s
[   16.697710] raid6: avx2x1   xor() 17794 MB/s
[   16.765701] raid6: sse2x4   gen() 17354 MB/s
[   16.833701] raid6: sse2x4   xor()  9653 MB/s
[   16.901706] raid6: sse2x2   gen() 18495 MB/s
[   16.969702] raid6: sse2x2   xor() 11562 MB/s
[   17.037701] raid6: sse2x1   gen() 14440 MB/s
[   17.105818] raid6: sse2x1   xor() 10387 MB/s
[   17.108300] raid6: using algorithm avx2x2 gen() 32995 MB/s
[   17.110703] raid6: .... xor() 19596 MB/s, rmw enabled
[   17.113587] raid6: using avx2x2 recovery algorithm
[   17.327666] xor: automatically using best checksumming function   avx

The xor/parity calculations are done synchronously, while the offloading
to hw usually requires asynchronous submit/wait mechanism. This brings
some overhead, so it depends. The code in btrfs would need to be adapted
to do the async way, unless it's all somehow hidden under the crypto
API.

Re: Question about btrfs and XOR offloading

[Rosen Penev]

On Mon, Jan 4, 2021 at 6:46 AM David Sterba <dsterba@suse.cz> wrote:
>
> On Sat, Jan 02, 2021 at 07:50:38PM -0800, Rosen Penev wrote:
> > I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
> > know if there is a performance advantage to using a hardware
> > accelerated path? I ask as I use BTRFS on a Marvelll ARM platform with
> > XOR offload capability.
>
> Even if it's CPU, it's accelerated and best algorithm is selected at
> boot time:
>
> [   16.357703] raid6: avx2x4   gen() 30635 MB/s
> [   16.425701] raid6: avx2x4   xor() 10727 MB/s
> [   16.493701] raid6: avx2x2   gen() 32995 MB/s
> [   16.561701] raid6: avx2x2   xor() 19596 MB/s
> [   16.629701] raid6: avx2x1   gen() 26349 MB/s
> [   16.697710] raid6: avx2x1   xor() 17794 MB/s
> [   16.765701] raid6: sse2x4   gen() 17354 MB/s
> [   16.833701] raid6: sse2x4   xor()  9653 MB/s
> [   16.901706] raid6: sse2x2   gen() 18495 MB/s
> [   16.969702] raid6: sse2x2   xor() 11562 MB/s
> [   17.037701] raid6: sse2x1   gen() 14440 MB/s
> [   17.105818] raid6: sse2x1   xor() 10387 MB/s
> [   17.108300] raid6: using algorithm avx2x2 gen() 32995 MB/s
> [   17.110703] raid6: .... xor() 19596 MB/s, rmw enabled
> [   17.113587] raid6: using avx2x2 recovery algorithm
> [   17.327666] xor: automatically using best checksumming function   avx
Yeah...

[    0.316064] raid6: neonx8   xor()  1087 MB/s
[    0.452063] raid6: neonx4   xor()  1372 MB/s
[    0.588064] raid6: neonx2   xor()  1610 MB/s
[    0.724061] raid6: neonx1   xor()  1345 MB/s
[    0.860072] raid6: int32x8  xor()   337 MB/s
[    0.996092] raid6: int32x4  xor()   373 MB/s
[    1.132087] raid6: int32x2  xor()   348 MB/s
[    1.268090] raid6: int32x1  xor()   281 MB/s
[    1.268093] raid6: .... xor() 1610 MB/s, rmw enabled

Not as fast here.
>
> The xor/parity calculations are done synchronously, while the offloading
> to hw usually requires asynchronous submit/wait mechanism. This brings
> some overhead, so it depends. The code in btrfs would need to be adapted
> to do the async way, unless it's all somehow hidden under the crypto
> API.

Re: Question about btrfs and XOR offloading

[David Sterba]

On Mon, Jan 04, 2021 at 02:56:59PM -0800, Rosen Penev wrote:
> On Mon, Jan 4, 2021 at 6:46 AM David Sterba <dsterba@suse.cz> wrote:
> >
> > On Sat, Jan 02, 2021 at 07:50:38PM -0800, Rosen Penev wrote:
> > > I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
> > > know if there is a performance advantage to using a hardware
> > > accelerated path? I ask as I use BTRFS on a Marvelll ARM platform with
> > > XOR offload capability.
> >
> > Even if it's CPU, it's accelerated and best algorithm is selected at
> > boot time:
> >
> > [   16.357703] raid6: avx2x4   gen() 30635 MB/s
> > [   16.425701] raid6: avx2x4   xor() 10727 MB/s
> > [   16.493701] raid6: avx2x2   gen() 32995 MB/s
> > [   16.561701] raid6: avx2x2   xor() 19596 MB/s
> > [   16.629701] raid6: avx2x1   gen() 26349 MB/s
> > [   16.697710] raid6: avx2x1   xor() 17794 MB/s
> > [   16.765701] raid6: sse2x4   gen() 17354 MB/s
> > [   16.833701] raid6: sse2x4   xor()  9653 MB/s
> > [   16.901706] raid6: sse2x2   gen() 18495 MB/s
> > [   16.969702] raid6: sse2x2   xor() 11562 MB/s
> > [   17.037701] raid6: sse2x1   gen() 14440 MB/s
> > [   17.105818] raid6: sse2x1   xor() 10387 MB/s
> > [   17.108300] raid6: using algorithm avx2x2 gen() 32995 MB/s
> > [   17.110703] raid6: .... xor() 19596 MB/s, rmw enabled
> > [   17.113587] raid6: using avx2x2 recovery algorithm
> > [   17.327666] xor: automatically using best checksumming function   avx
> Yeah...
> 
> [    0.316064] raid6: neonx8   xor()  1087 MB/s
> [    0.452063] raid6: neonx4   xor()  1372 MB/s
> [    0.588064] raid6: neonx2   xor()  1610 MB/s
> [    0.724061] raid6: neonx1   xor()  1345 MB/s
> [    0.860072] raid6: int32x8  xor()   337 MB/s
> [    0.996092] raid6: int32x4  xor()   373 MB/s
> [    1.132087] raid6: int32x2  xor()   348 MB/s
> [    1.268090] raid6: int32x1  xor()   281 MB/s
> [    1.268093] raid6: .... xor() 1610 MB/s, rmw enabled
> 
> Not as fast here.

What's the raw speed of the hw offload? Measured on large data so that
the overhead is negligible.

It might make sense to add the async support in case the speed is
comparable or better to the CPU, but also to reduce the CPU load.

Re: Question about btrfs and XOR offloading

[Rosen Penev]

On Tue, Jan 5, 2021 at 7:35 AM David Sterba <dsterba@suse.cz> wrote:
>
> On Mon, Jan 04, 2021 at 02:56:59PM -0800, Rosen Penev wrote:
> > On Mon, Jan 4, 2021 at 6:46 AM David Sterba <dsterba@suse.cz> wrote:
> > >
> > > On Sat, Jan 02, 2021 at 07:50:38PM -0800, Rosen Penev wrote:
> > > > I've noticed that internally, btrfs' XOR code is CPU only. Does anyone
> > > > know if there is a performance advantage to using a hardware
> > > > accelerated path? I ask as I use BTRFS on a Marvelll ARM platform with
> > > > XOR offload capability.
> > >
> > > Even if it's CPU, it's accelerated and best algorithm is selected at
> > > boot time:
> > >
> > > [   16.357703] raid6: avx2x4   gen() 30635 MB/s
> > > [   16.425701] raid6: avx2x4   xor() 10727 MB/s
> > > [   16.493701] raid6: avx2x2   gen() 32995 MB/s
> > > [   16.561701] raid6: avx2x2   xor() 19596 MB/s
> > > [   16.629701] raid6: avx2x1   gen() 26349 MB/s
> > > [   16.697710] raid6: avx2x1   xor() 17794 MB/s
> > > [   16.765701] raid6: sse2x4   gen() 17354 MB/s
> > > [   16.833701] raid6: sse2x4   xor()  9653 MB/s
> > > [   16.901706] raid6: sse2x2   gen() 18495 MB/s
> > > [   16.969702] raid6: sse2x2   xor() 11562 MB/s
> > > [   17.037701] raid6: sse2x1   gen() 14440 MB/s
> > > [   17.105818] raid6: sse2x1   xor() 10387 MB/s
> > > [   17.108300] raid6: using algorithm avx2x2 gen() 32995 MB/s
> > > [   17.110703] raid6: .... xor() 19596 MB/s, rmw enabled
> > > [   17.113587] raid6: using avx2x2 recovery algorithm
> > > [   17.327666] xor: automatically using best checksumming function   avx
> > Yeah...
> >
> > [    0.316064] raid6: neonx8   xor()  1087 MB/s
> > [    0.452063] raid6: neonx4   xor()  1372 MB/s
> > [    0.588064] raid6: neonx2   xor()  1610 MB/s
> > [    0.724061] raid6: neonx1   xor()  1345 MB/s
> > [    0.860072] raid6: int32x8  xor()   337 MB/s
> > [    0.996092] raid6: int32x4  xor()   373 MB/s
> > [    1.132087] raid6: int32x2  xor()   348 MB/s
> > [    1.268090] raid6: int32x1  xor()   281 MB/s
> > [    1.268093] raid6: .... xor() 1610 MB/s, rmw enabled
> >
> > Not as fast here.
>
> What's the raw speed of the hw offload? Measured on large data so that
> the overhead is negligible.
I have no idea how to benchmark such a thing. I assume it could be
done indirectly.
>
> It might make sense to add the async support in case the speed is
> comparable or better to the CPU, but also to reduce the CPU load.
I think the latter is the reason Marvell added hardware support for
doing parity calculations.

Re: Question about btrfs and XOR offloading

[David Sterba]

On Tue, Jan 05, 2021 at 02:40:28PM -0800, Rosen Penev wrote:
> > What's the raw speed of the hw offload? Measured on large data so that
> > the overhead is negligible.
> I have no idea how to benchmark such a thing. I assume it could be
> done indirectly.
> >
> > It might make sense to add the async support in case the speed is
> > comparable or better to the CPU, but also to reduce the CPU load.
> I think the latter is the reason Marvell added hardware support for
> doing parity calculations.

The support seems to be in NAS boxes and besides xor and raid5/6
calculations the engine can also do a memcpy offload. This could gain a
lot of performance and be cheap in terms of code. Full page copies are
wrapped under copy_page so we'd need to insert the offload code. Similar
for the raid5/6 calculations.

The MD-RAID already supports offloading so we have code to stea^Wcopy.
Overall it sounds worth to add the async support to btrfs as it would
help with the metadata updates too, there's a lot of memcpy/memmove.

Re: Question about btrfs and XOR offloading

[Rosen Penev]

On Wed, Jan 6, 2021 at 4:39 AM David Sterba <dsterba@suse.cz> wrote:
>
> On Tue, Jan 05, 2021 at 02:40:28PM -0800, Rosen Penev wrote:
> > > What's the raw speed of the hw offload? Measured on large data so that
> > > the overhead is negligible.
> > I have no idea how to benchmark such a thing. I assume it could be
> > done indirectly.
> > >
> > > It might make sense to add the async support in case the speed is
> > > comparable or better to the CPU, but also to reduce the CPU load.
> > I think the latter is the reason Marvell added hardware support for
> > doing parity calculations.
>
> The support seems to be in NAS boxes and besides xor and raid5/6
> calculations the engine can also do a memcpy offload. This could gain a
> lot of performance and be cheap in terms of code. Full page copies are
> wrapped under copy_page so we'd need to insert the offload code. Similar
> for the raid5/6 calculations.
>
> The MD-RAID already supports offloading so we have code to stea^Wcopy.
> Overall it sounds worth to add the async support to btrfs as it would
> help with the metadata updates too, there's a lot of memcpy/memmove.
That would be lovely. I assume 24 hour btrfs scrubs would become shorter.

Unfortunately I lack the expertise to implement this properly.