Re: Add device while rebalancing

[Juan Alberto Cirez <jacirez@rdcsafety.com>]

WOW!
Correct me if I'm wrong but the sum total of the above seems to
suggest (at first glance) that BRTFS add several layers of complexity,
but for little real benefit (at least in the case use of btrfs at the
brick layer with a distributed filesystem on top)...

"...I've always though it'd be neat in a Btrfs + GlusterFS, if it were
possible for Btrfs to inform Gluster FS of "missing/corrupt" files,
and then for Btrfs to drop reference for those files, instead of
either rebuilding or remaining degraded. And then let GlusterFS deal
with replication of those files to maintain redundancy. i.e. the Btrfs
volumes would be single profile for data, and raid1 for metadata. When
there's n-way raid1, each drive can have a copy of the file system,
and it'd tolerate in effect n-1 drive failures and the file system
could at least still inform Gluster (or Ceph) of the missing data, the
file system still remains valid, only briefly degraded, and can still
be expanded when new drives become available..."

That in my n00b opinion would be brilliant in a real world use case.


On Wed, Apr 27, 2016 at 5:22 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2016-04-26 20:58, Chris Murphy wrote:
>>
>> On Tue, Apr 26, 2016 at 5:44 AM, Juan Alberto Cirez
>> <jacirez@rdcsafety.com> wrote:
>>>
>>>
>>> With GlusterFS as a distributed volume, the files are already spread
>>> among the servers causing file I/O to be spread fairly evenly among
>>> them as well, thus probably providing the benefit one might expect
>>> with stripe (RAID10).
>>
>>
>> Yes, the raid1 of Btrfs is just so you don't have to rebuild volumes
>> if you lose a drive. But since raid1 is not n-way copies, and only
>> means two copies, you don't really want the file systems getting that
>> big or you increase the chances of a double failure.
>>
>> I've always though it'd be neat in a Btrfs + GlusterFS, if it were
>> possible for Btrfs to inform Gluster FS of "missing/corrupt" files,
>> and then for Btrfs to drop reference for those files, instead of
>> either rebuilding or remaining degraded. And then let GlusterFS deal
>> with replication of those files to maintain redundancy. i.e. the Btrfs
>> volumes would be single profile for data, and raid1 for metadata. When
>> there's n-way raid1, each drive can have a copy of the file system,
>> and it'd tolerate in effect n-1 drive failures and the file system
>> could at least still inform Gluster (or Ceph) of the missing data, the
>> file system still remains valid, only briefly degraded, and can still
>> be expanded when new drives become available.
>
> FWIW, I _think_ this can be done with the scrubbing code in GlusterFS. It's
> designed to repair data mismatches, but I'm not sure how it handles missing
> copies of data.  However, in the current state, there's no way without
> external scripts to handle re-shaping of the storage bricks if part of them
> fails.