btrfs raid multiple devices IO utilisation

btrfs raid multiple devices IO utilisation

[Martin]

Some time back, I noticed that with a two HDD btrfs raid1, some tasks
suffered ALL the IO getting choked onto just one HDD!


That turned out to be a feature of the btrfs code whereby a device is
chosen depending on the process ID. For some cases such as in a bash
loop, the PID increments by two for each iteration and so only one HDD
ever gets hit...

So... Running with a 3-disk btrfs raid1 and... I still see the same
problem for such as compile tasks where only one of the three disks is
maxed out for periods with the other two disks left nearly idle.


Perhaps?...

Is there an easy fix in the code to allocate IO according to the
following search order:

Last used disk with an IO queue < 2 items;

Any disk with an IO queue < 2 items;

Whichever disk with least queued items.


The intention there is to repeat sequential IO to the current disk, but
allow a second disk to interleave data requests for a same one file if
data is requested faster than what the one disk can support.

Other aspects are that the IO queue length checks hopefully will
proportionately balance the requests across disks that offer different
speeds. Hence a faster disk will get used to advantage if paired with a
slower device.


There looks to be two points in the code where a device gets selected.
Not having delved into filesystems before... Is there easy/safe reading
available for the IO queues for a device?

Or rather: All easily done?

(Or is the btrfs code at too high a level to be able to see/access the
device IO queues?)


Regards,
Martin

Re: btrfs raid multiple devices IO utilisation

[Duncan]

Martin posted on Thu, 12 Dec 2013 13:39:00 +0000 as excerpted:

> Some time back, I noticed that with a two HDD btrfs raid1, some tasks
> suffered ALL the IO getting choked onto just one HDD!
> 
> That turned out to be a feature of the btrfs code whereby a device is
> chosen depending on the process ID. For some cases such as in a bash
> loop, the PID increments by two for each iteration and so only one HDD
> ever gets hit...

Unfortunately, yes...

> So... Running with a 3-disk btrfs raid1 and... I still see the same
> problem for such as compile tasks where only one of the three disks is
> maxed out for periods with the other two disks left nearly idle.

It's worth noting that unfortunately, btrfs raid1 mode isn't "real" raid1 
at this point, at least not in the N-way-mirrored sense.  It's only two-
way-mirrored, regardless of the number of devices you throw at it, tho N-
way mirroring is roadmapped for introduction after raid5/6 functionality 
is fully implemented.  (The current raid5/6 implementation is missing 
some bits and is not considered production usable, as it doesn't handle 
rebuilds and scrubs well yet.)

So I'm guessing your 3-device btrfs raid1 is still stuck on the same even/
odd PID problem as before.

> Perhaps?...
> 
> Is there an easy fix in the code to allocate IO according to the
> following search order:
> 
> Last used disk with an IO queue < 2 items;
> 
> Any disk with an IO queue < 2 items;
> 
> Whichever disk with least queued items.

That does sound reasonable here.

FWIW, however, based on previous discussion here, the devs are aware of 
the alternating PID problem, and I /believe/ someone's already working on 
an alternate implementation using something else.  I think the current 
PID-based selector code was simply a first implementation to get 
something out there; not really intended to be a final working solution. 
IDR whether anything I've read discussed what algorithm they're working 
on, but given the sense your idea seems to make, at least at first glance 
at my sysadmin level of understanding, I wouldn't be surprised if the new 
solution does look something like it.  Of course that's if the queue 
length is reasonably accessible to btrfs, as you already asked in the bit 
I snipped as out of my knowledgeable reply range.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs raid multiple devices IO utilisation

[Martin]

On 13/12/13 09:13, Duncan wrote:
> Martin posted on Thu, 12 Dec 2013 13:39:00 +0000 as excerpted:
> 
>> Some time back, I noticed that with a two HDD btrfs raid1, some tasks
>> suffered ALL the IO getting choked onto just one HDD!
>>
>> That turned out to be a feature of the btrfs code whereby a device is
>> chosen depending on the process ID. For some cases such as in a bash
>> loop, the PID increments by two for each iteration and so only one HDD
>> ever gets hit...
> 
> Unfortunately, yes...


> ... Of course that's if the queue 
> length is reasonably accessible to btrfs, as you already asked in the bit 
> I snipped as out of my knowledgeable reply range.

And that's the bit that is outside of my knowledge... Hence the question.

Here's hoping someone can give a pointer before I need to delve into
other code such as for 'atop' to see how it is done there if I'm to take
a stab at it.

Or if someone is already coding something up, please let us know?


Or:

>From within btrfs code, is there an easy/fast/any way to check the IO
queue length for whatever btrfs target device?

Regards,
Martin

Re: btrfs raid multiple devices IO utilisation

[Chris Murphy]

On Dec 12, 2013, at 6:39 AM, Martin <m_btrfs@ml1.co.uk> wrote:

> Some time back, I noticed that with a two HDD btrfs raid1, some tasks
> suffered ALL the IO getting choked onto just one HDD!
> 
> 
> That turned out to be a feature of the btrfs code whereby a device is
> chosen depending on the process ID.

I wonder how much btrfs parallelism is affected by the default CFQ scheduler? On the XFS FAQ "the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS." There's also quite a bit of parallelization in Btrfs so it too may be affected, in particular multiple device scenarios.

Chris Murphy