Using aGPU for RAID calculations (proprietary GRAID SupremeRAID)

Using aGPU for RAID calculations (proprietary GRAID SupremeRAID)

[Paul Menzel]

Dear Linux folks,


I read about GRAID SupremeRAID [1], which seems to be an Nvidia T1000 
card and software to use the card for RAID calculations.

> GRAID SupremeRAID works by installing a virtual NVMe controller onto
> the operating system and integrating a PCIe device into the system
> equipped with a high-performance AI processor to handle all RAID
> operations of the virtual NVMe controller

According to the review *GRAID SupremeRAID SR-1000 Review* [2] it 
performs quite well. I couldn’t find any driver files online.

Now I am wondering, why a graphics card seems to help so much. What 
operations are there, modern CPUs cannot keep up with?

If GPUs are that much better, are people already working on a FLOSS 
solution for the Linux kernel, so people can “just” plug in a graphics 
card to increase the speed?

Does the Linux kernel already have an API to offload calculations to 
accelerator cards, so it’s basically plug and play (with AMD graphics 
cards for example using HSA/KFD)? Entropy sources, like the ChaosKey 
[3], work like that. If not, would the implementation go under `lib/raid6`?


Kind regards,

Paul


[1]: 
https://www.graidtech.com/post/graid-reveals-the-next-generation-of-enterprise-data-protection-nvme-ssds
[2]: https://www.storagereview.com/review/graid-supremeraid-sr-1000-review
[3]: https://altusmetrum.org/ChaosKey/

Re: Using aGPU for RAID calculations (proprietary GRAID SupremeRAID)

[Wolfgang Denk]

Dear Paul,

In message <ccdbb0d7-d043-d41f-508b-4a464ffa5fe9@molgen.mpg.de> you wrote:
> 
> I read about GRAID SupremeRAID [1], which seems to be an Nvidia T1000 
> card and software to use the card for RAID calculations.

I doubt this is an efficient way.

The (IMHO) most efficient solution I have seen so far was the AMCC
PPC440SPe Power Architecture processors.  These we able to calculate
the needed parity information on the fly in the DMA controller while
writing the data to disk.

> Does the Linux kernel already have an API to offload calculations to 
> accelerator cards, so it’s basically plug and play (with AMD graphics 
> cards for example using HSA/KFD)? Entropy sources, like the ChaosKey 
> [3], work like that. If not, would the implementation go under `lib/raid6`?

We implemented Linux driver support for adma/async_tx/raid6 for this
hardware, but that was a long time ago (2008/2009), and I doubt this
has been tested recently.  But at least some code is still in
mainline, see for example arch/powerpc/platforms/44x/ppc440spe_dma_engines.c


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Another megabytes the dust.

Re: Using aGPU for RAID calculations (proprietary GRAID SupremeRAID)

[Piergiorgio Sartor]

On Tue, Nov 30, 2021 at 12:58:10PM +0100, Paul Menzel wrote:
> 
> Dear Linux folks,
> 
> 
> I read about GRAID SupremeRAID [1], which seems to be an Nvidia T1000 card
> and software to use the card for RAID calculations.
> 
> > GRAID SupremeRAID works by installing a virtual NVMe controller onto
> > the operating system and integrating a PCIe device into the system
> > equipped with a high-performance AI processor to handle all RAID
> > operations of the virtual NVMe controller
> 
> According to the review *GRAID SupremeRAID SR-1000 Review* [2] it performs
> quite well. I couldn’t find any driver files online.
> 
> Now I am wondering, why a graphics card seems to help so much. What
> operations are there, modern CPUs cannot keep up with?
> 
> If GPUs are that much better, are people already working on a FLOSS solution
> for the Linux kernel, so people can “just” plug in a graphics card to
> increase the speed?
> 
> Does the Linux kernel already have an API to offload calculations to
> accelerator cards, so it’s basically plug and play (with AMD graphics cards
> for example using HSA/KFD)? Entropy sources, like the ChaosKey [3], work
> like that. If not, would the implementation go under `lib/raid6`?

I think this was somehow discussed here
some times ago.
That is the use of "GPU" to accellerate
the parity computation.

There are a couple of things to keep in mind.

One is the data transfer to / from the video
card, which might be a bottleneck.
At any rate, there will be a write and read
streams going across the system bus(es).

An other point is that, unless an high end
video card is used, with ECC memory, the
reliability of the whole process might be
of concern.

Finally, usually video cards, while having
a lot of memory (caching could be good),
they miss the battery backup.
Power is off, data is gone...

bye,

pg

> 
> 
> Kind regards,
> 
> Paul
> 
> 
> [1]: https://www.graidtech.com/post/graid-reveals-the-next-generation-of-enterprise-data-protection-nvme-ssds
> [2]: https://www.storagereview.com/review/graid-supremeraid-sr-1000-review
> [3]: https://altusmetrum.org/ChaosKey/
> 

-- 

piergiorgio

Re: Using aGPU for RAID calculations (proprietary GRAID SupremeRAID)

[Janpieter Sollie]

Tue, 30 Nov 2021 17:20:25 +0100, Piergiorgio Sartor wrote:

 > On Tue, Nov 30, 2021 at 12:58:10PM +0100, Paul Menzel wrote:
 > > > Dear Linux folks,
 > > > > I read about GRAID SupremeRAID [1], which seems to be an Nvidia T1000 card
 > > and software to use the card for RAID calculations.
 > > > > GRAID SupremeRAID works by installing a virtual NVMe controller onto
 > > > the operating system and integrating a PCIe device into the system
 > > > equipped with a high-performance AI processor to handle all RAID
 > > > operations of the virtual NVMe controller
 > > > According to the review *GRAID SupremeRAID SR-1000 Review* [2] it performs
 > > quite well. I couldn’t find any driver files online.
 > > > Now I am wondering, why a graphics card seems to help so much. What
 > > operations are there, modern CPUs cannot keep up with?
 > > > If GPUs are that much better, are people already working on a FLOSS solution
 > > for the Linux kernel, so people can “just” plug in a graphics card to
 > > increase the speed?
 > > > Does the Linux kernel already have an API to offload calculations to
 > > accelerator cards, so it’s basically plug and play (with AMD graphics cards
 > > for example using HSA/KFD)? Entropy sources, like the ChaosKey [3], work
 > > like that. If not, would the implementation go under `lib/raid6`?
 > I think this was somehow discussed here some times ago.
 > That is the use of "GPU" to accellerate the parity computation.
 > There are a couple of things to keep in mind.
 > One is the data transfer to / from the video card, which might be a bottleneck.
 > At any rate, there will be a write and read streams going across the system bus(es).
 > An other point is that, unless an high end video card is used, with ECC memory,
 > the reliability of the whole process might be of concern.
 > Finally, usually video cards, while having a lot of memory (caching could be good),
 > they miss the battery backup. Power is off, data is gone...
 >
 > bye,
 >
 > pg

Hi pg,

I'm sorry to wake up this thread again, but I actually thought it was a very interesting idea:

I'm new here, but I'm giving up on hardware raid because I think the idea is obsolete. But I 
read your mail above:

what if the GPU could be used for verifying raid 5/6 in realtime? something like verifyread ...
the "real" parity calculation should still be done by the CPU for reliability, but at least the 
GPU can signal something may be wrong.
with nvme drives pushing 4-5GiB/s these days, a raid 6 of 10 nvme drives could easily outperform 
a single CPU core.
for raid 5, I doubt anyone will benefit from it, as it's a simple XOR calculation, but might it 
be for raid 6

For your concern about the read/write on the system bus, I agree: as with every situation, it 
will depend on the config, but it is unlikely it will never be a problem.

Your BBU remark is also interesting to me: why would data be "gone" at a power off?

kind regards,

Janpieter Sollie

Re: Using aGPU for RAID calculations (proprietary GRAID SupremeRAID)

[Keith Busch]

On Tue, Nov 30, 2021 at 12:58:10PM +0100, Paul Menzel wrote:
> If GPUs are that much better, are people already working on a FLOSS solution
> for the Linux kernel, so people can “just” plug in a graphics card to
> increase the speed?
> 
> Does the Linux kernel already have an API to offload calculations to
> accelerator cards, so it’s basically plug and play (with AMD graphics cards
> for example using HSA/KFD)? Entropy sources, like the ChaosKey [3], work
> like that. If not, would the implementation go under `lib/raid6`?

I'm not very familiar with the subsystem, but I think these types of
operation APIs are implemented under crypto/async_tx/.