Re: Regarding Raid-6 array

[Song Liu <songliubraving@fb.com>]

CC the list

Hi Vishal,

> On Jan 27, 2021, at 9:04 PM, Vishal Verma <vverma@digitalocean.com> wrote:
> 
> Hello Song,
> 
> This is Vishal Verma, Performance Engineer at DigitalOcean. 
> 
> I was recently playing with our 6x nvme drive based RAID stack assessing its performance against a RAID-10 array.
> I understand with RAID-10 there is no striping or parity work so its write performance looks really nice.
> 
> But, I was not sure about the RAID-6 piece, specifically the RMW piece. 
> I was running FIO 128K 100% sequential workload to the raid6 drive array with O_DIRECT and noticed how drives performed:
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.18    0.00    2.36    0.00    0.00   97.46
> 
> Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
> nvme1n1        546.00 8330.00  69768.00 994900.00 16896.00 240311.00  96.87  96.65    0.38    0.56   0.00   127.78   119.44   0.10  84.60
> nvme0n1        513.00 8333.50  65544.00 982314.00 15873.00 237245.00  96.87  96.61    0.35    0.32   0.00   127.77   117.88   0.10  86.80
> nvme4n1        480.00 8795.50  61440.00 1045290.00 14880.00 252535.00  96.88  96.63    0.35    0.28   0.00   128.00   118.84   0.09  87.00
> nvme3n1        513.00 8425.50  65664.00 1012554.00 15903.00 244868.50  96.88  96.67    0.37    0.70   0.01   128.00   120.18   0.09  84.20
> nvme5n1        497.00 8618.00  63496.00 1011158.00 15377.00 244201.50  96.87  96.59    0.36    0.64   0.00   127.76   117.33   0.10  88.20
> nvme2n1        529.00 8306.50  67712.00 998920.00 16399.00 241243.50  96.88  96.67    0.39    0.37   0.00   128.00   120.26   0.09  83.60
> 
> 
> I was surprised to see the amount of reads the drives were doing even though it was a full 100% write test.
> I understand for every write IO RAID-6 md array has to first read P,Q read the data and then calculate new P and Q and write the data.
> 
> However, do we expect the drives to read that much i.e ~65MB/s (6%) worth of reads for every 1GB/s (100%) of writes?

6% of read for sequential write is not surprising. In the worst case, 
for 4kB write from upper layer, RAID-6 does introduce 2x reads and 3x 
writes: read P, readQ, write data, write P, write Q. The raid layer
will optimize full stripe writes to avoid reads, but it is common for 
upper layer to do non-full-stripe writes. 

Thanks,
Song

[...]