Add Bcache to an existing Filesystem

Add Bcache to an existing Filesystem

[FERNANDO FREDIANI]

Hello folks.

I have been using Bcache in a server in production with pretty good 
results so far.

Now I am looking to add it also to another server that is suffering from 
disk I/O. However this server in question has a fairly large storage 
running (circa 8TB of used data) and therefore stopping it for a long 
period becomes difficult.

Question is: Can I add an SSD and Bcache to this server and use it for 
the existing 8TB filesystem ? If so when I run make-bcache -B /dev/sdx1 
to create the backing device will it not overwrite what is currently on 
/dev/sdx1 ?

Thanks
Fernando

Re: Add Bcache to an existing Filesystem

[Eric Wheeler]

On Mon, 26 Jun 2017, FERNANDO FREDIANI wrote:

> Hello folks.
> 
> I have been using Bcache in a server in production with pretty good results so
> far.
> 
> Now I am looking to add it also to another server that is suffering from disk
> I/O. However this server in question has a fairly large storage running (circa
> 8TB of used data) and therefore stopping it for a long period becomes
> difficult.
> 
> Question is: Can I add an SSD and Bcache to this server and use it for the
> existing 8TB filesystem ? If so when I run make-bcache -B /dev/sdx1 to create
> the backing device will it not overwrite what is currently on /dev/sdx1 ?


Not officially.  The bcache superblock is in the first 8k of disk (by 
default, but can align the payload) so 8TB would need shifted right 
assuming the right-hand-side has room to grow.

Disclaimer: this could be dangerous, so at your own risk:

While I've not tried this, theoretically you could put dm-linear atop of 
the bcache bdev and prepend some metadata area, using the `--data-offset` 
parameter in make-bcache as the start of your data area.  Be sure to test 
on something non-critical if you go this route and make sure the sector 
math is correct or risk damaging the front of your 8TB volume.  If you go 
this route and it works, please report back with what you did!

--
Eric Wheeler



> 
> Thanks
> Fernando
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Add Bcache to an existing Filesystem

[Henk Slager]

On Mon, Jun 26, 2017 at 6:12 PM, FERNANDO FREDIANI
<fernando.frediani@upx.com> wrote:
> Hello folks.
>
> I have been using Bcache in a server in production with pretty good results
> so far.
>
> Now I am looking to add it also to another server that is suffering from
> disk I/O. However this server in question has a fairly large storage running
> (circa 8TB of used data) and therefore stopping it for a long period becomes
> difficult.
>
> Question is: Can I add an SSD and Bcache to this server and use it for the
> existing 8TB filesystem ? If so when I run make-bcache -B /dev/sdx1 to
> create the backing device will it not overwrite what is currently on
> /dev/sdx1 ?

Yes you can if you are prepared to do some re-partition tricks. Maybe
there is a tool/script that can do that, AFAIK there is some python
prog called blocks for this.

Anyhow, what make-bcache -B /dev/sdx1 does is write an 8KiB sized
header at the start of /dev/sdx1. So in case this is the only GPT
based partition on the diskdevice that currently starts at
512-byte-sized sector 2048, there is usually some free space between
GPT table and 2048. So make sure you have all backupped, then unmount
the fs and use gdisk to set the starting sector of /dev/sdx1 to 2032,
by delete first and re-create (with alignment set to 16 sectors
first).

Then make sure the kernel uses the new partition table (partprobe or
reboot) and run make-bcache -C <SSD> -B /dev/sdx1. You can now access
the fs via /dev/bcache0

Some people might consider this way too risky, but I have successfully
used it, also in reverse order for removal of bcache. You might have a
slighty different disk setup than I assume, but I hope th eprinciple
is clear. If you would script it, it takes about the time of an
unmount+mount, so seconds instead of hours or even days.

Re: Add Bcache to an existing Filesystem

[FERNANDO FREDIANI]

Thanks for reply Eric.

That sounds quiet dangerous to risk 8TB of production data. I guess the 
only safe way is re-formating the partition and therefore it would mean 
8TB copied-out then copied back in which imposes a longer downtime to 
the server.

Best regards
Fernando


On 26/06/2017 14:58, Eric Wheeler wrote:
> On Mon, 26 Jun 2017, FERNANDO FREDIANI wrote:
>
>> Hello folks.
>>
>> I have been using Bcache in a server in production with pretty good results so
>> far.
>>
>> Now I am looking to add it also to another server that is suffering from disk
>> I/O. However this server in question has a fairly large storage running (circa
>> 8TB of used data) and therefore stopping it for a long period becomes
>> difficult.
>>
>> Question is: Can I add an SSD and Bcache to this server and use it for the
>> existing 8TB filesystem ? If so when I run make-bcache -B /dev/sdx1 to create
>> the backing device will it not overwrite what is currently on /dev/sdx1 ?
>
> Not officially.  The bcache superblock is in the first 8k of disk (by
> default, but can align the payload) so 8TB would need shifted right
> assuming the right-hand-side has room to grow.
>
> Disclaimer: this could be dangerous, so at your own risk:
>
> While I've not tried this, theoretically you could put dm-linear atop of
> the bcache bdev and prepend some metadata area, using the `--data-offset`
> parameter in make-bcache as the start of your data area.  Be sure to test
> on something non-critical if you go this route and make sure the sector
> math is correct or risk damaging the front of your 8TB volume.  If you go
> this route and it works, please report back with what you did!
>
> --
> Eric Wheeler
>
>
>
>> Thanks
>> Fernando
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: Add Bcache to an existing Filesystem

[FERNANDO FREDIANI]

Well, just a bit more detail of my scenario. This 8TB is a Linux RAID 10 
on the top of 4 x 4TB disks, therefore the filesystem is currently 
mounted in a /dev/md124 which is a ext4. The physical disks in turn have 
each a single partition with type "Linux RAID" and they apparently start 
on sector 2048 (see below on of these disks):

Disk /dev/sdd: 4000.8 GB, 4000787030016 bytes, 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
#         Start          End    Size  Type            Name
  1         2048   7814037134    3.7T  Linux RAID

Fernando

On 26/06/2017 15:05, Henk Slager wrote:
> On Mon, Jun 26, 2017 at 6:12 PM, FERNANDO FREDIANI
> <fernando.frediani@upx.com> wrote:
>> Hello folks.
>>
>> I have been using Bcache in a server in production with pretty good results
>> so far.
>>
>> Now I am looking to add it also to another server that is suffering from
>> disk I/O. However this server in question has a fairly large storage running
>> (circa 8TB of used data) and therefore stopping it for a long period becomes
>> difficult.
>>
>> Question is: Can I add an SSD and Bcache to this server and use it for the
>> existing 8TB filesystem ? If so when I run make-bcache -B /dev/sdx1 to
>> create the backing device will it not overwrite what is currently on
>> /dev/sdx1 ?
> Yes you can if you are prepared to do some re-partition tricks. Maybe
> there is a tool/script that can do that, AFAIK there is some python
> prog called blocks for this.
>
> Anyhow, what make-bcache -B /dev/sdx1 does is write an 8KiB sized
> header at the start of /dev/sdx1. So in case this is the only GPT
> based partition on the diskdevice that currently starts at
> 512-byte-sized sector 2048, there is usually some free space between
> GPT table and 2048. So make sure you have all backupped, then unmount
> the fs and use gdisk to set the starting sector of /dev/sdx1 to 2032,
> by delete first and re-create (with alignment set to 16 sectors
> first).
>
> Then make sure the kernel uses the new partition table (partprobe or
> reboot) and run make-bcache -C <SSD> -B /dev/sdx1. You can now access
> the fs via /dev/bcache0
>
> Some people might consider this way too risky, but I have successfully
> used it, also in reverse order for removal of bcache. You might have a
> slighty different disk setup than I assume, but I hope th eprinciple
> is clear. If you would script it, it takes about the time of an
> unmount+mount, so seconds instead of hours or even days.

Re: Add Bcache to an existing Filesystem

[Henk Slager]

On Mon, Jun 26, 2017 at 8:14 PM, FERNANDO FREDIANI
<fernando.frediani@upx.com> wrote:
> Well, just a bit more detail of my scenario. This 8TB is a Linux RAID 10 on
> the top of 4 x 4TB disks, therefore the filesystem is currently mounted in a
> /dev/md124 which is a ext4. The physical disks in turn have each a single
> partition with type "Linux RAID" and they apparently start on sector 2048
> (see below on of these disks):
>
> Disk /dev/sdd: 4000.8 GB, 4000787030016 bytes, 7814037168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk label type: gpt
> #         Start          End    Size  Type            Name
>  1         2048   7814037134    3.7T  Linux RAID

OK, setup is clear. I am only using this stack (w.r.t. backing device):
 harddisk(gpt partitioned) - bcache - luks - btrfs. That layering for
1 HDD + 1 SSD, but also for 1 SSD + 4 HDD using btrfs raid10. I wanted
to use btrfs multi-device/raid feature, so that is one reason why I
prefer bcache directly on top of partition. So for the btrfs raid10
situation, I have 4 bcache devices.

But if you use MD, you could choose to put bcache on top of MD. There
is someone on the list who uses bcache on top of MD RAID5 AFAIR. I
think I would choose to add bcache to each of the four harddisks,
primarily because I am not familiar enough now with Linux multiple
devices tooling, only how it was 10 years back. I would definitely
need to study, trial-run, etc it first. gdisk and dd are relatively
simple and static w.r.t changes, so that is what I have been usiing to
add and remove bcache. But maybe you feel comfortable enough with
changing things just above or below the MD layer and is it the same
accaptable risk.

Re: Add Bcache to an existing Filesystem

[Jens-U. Mozdzen]

Hi *,

Zitat von Henk Slager <eye1tm@gmail.com>:
> [...] There
> is someone on the list who uses bcache on top of MD RAID5 AFAIR.

that would probably be me: backing device was an MD-RAID6 (we've  
switched to RAID1 in the meantime because the data fits on a single  
disk (netto) and we needed the HDD slots) and caching device is an  
MD-RAID1 consisting of two SSDs.

On top of that bcache device, I decided to add an LVM layer (making  
/dev/bcache* a PV), which is really helpful i.e. with the base problem  
of this thread: You can create additional bcache devices and pvmove  
your data to the new PV, without interrupting production.

> I
> think I would choose to add bcache to each of the four harddisks,

If you'd do that with a single caching device, you're in for  
contention. My gut feeling tells me that running a single bcache  
backing device/caching device combo on top of MD-RAID is less  
straining than running MD-RAID across a bunch of bcache devices with a  
common caching device: The MD advantage of spreading the load across  
multiple "disks" is countered by accessing a common SSD.

Regards
J.

Re: Add Bcache to an existing Filesystem

[Nix]

On 27 Jun 2017, Jens-U. Mozdzen verbalised:

> Hi *,
>
> Zitat von Henk Slager <eye1tm@gmail.com>:
>> [...] There
>> is someone on the list who uses bcache on top of MD RAID5 AFAIR.
>
> that would probably be me: backing device was an MD-RAID6 (we've
> switched to RAID1 in the meantime because the data fits on a single
> disk (netto) and we needed the HDD slots) and caching device is an
> MD-RAID1 consisting of two SSDs.

Me too, but the fact that my bcache cache device exploded with "missing
magic" errors after less than a week makes me perhaps not a very good
advert for this.

>> I think I would choose to add bcache to each of the four harddisks,
>
> If you'd do that with a single caching device, you're in for  contention. My gut feeling tells me that running a single bcache
> backing device/caching device combo on top of MD-RAID is less  straining than running MD-RAID across a bunch of bcache devices with
> a  common caching device: The MD advantage of spreading the load across  multiple "disks" is countered by accessing a common SSD.

It all depends what your speed is like: if the load is at all seeky, an
SSD's zero seek time will dominate. md atop bcache will certainly have
far more SSD-write overhead than bcache atop md, because every time md
does a full-stripe read or read-modify-write cycle you'll be burning
holes in the SSD for no real speed benefit whatsoever...

One thing to note about working atop md is that you do have to be
careful about alignment: you don't want every md write to incur a
read-modify-write cycle after you've gone to some lengths to tell your
filesystems what the RAID offset is. You should compute an appropriate
--data-offset for not only the array configuration now (so a multiple of
your chunk size, at least, and possibly of your stripe size) but if you
are planning to add more data spindles you should figure out a (larger)
--data-offset that will retain appropriate alignment for plausible
larger sets of spindles you might grow to in the future.

(Also note that bcache does *not* communicate the RAID geometry through
to overlying filesystems -- if you want decent write performance on
RAID-5 or -6 you'll need to do that yourself, via mke2fs -E
stripe_width/stride, mkfs.xfs sunit and swidth arguments, cryptsetup
--align-payload, etc. You can use btrace on the underlying real block
device to see if the requests look aligned after you're done, but before
populating it with data. :) )

-- 
NULL && (void)

Re: Add Bcache to an existing Filesystem

[FERNANDO FREDIANI]

Hello Nix

What exactlly you mean by "exploded" ? Has it stopped working at all and 
has that stopped the systems running on the top of it or it just stopped 
the caching device  and the backing continued to work well.

Regarding my scenario, just to clarify I have  4 x /dev/sdX forming a 
/dev/md124 in RAID 10. This /dev/md124 is the backing device of my 
bcache which in turn has /dev/nvme0n1 as the caching device making 
/dev/bcache0. I guess this would be the most coming scenario right ?

Fernando


On 27/06/2017 10:23, Nix wrote:
> On 27 Jun 2017, Jens-U. Mozdzen verbalised:
>
>> Hi *,
>>
>> Zitat von Henk Slager <eye1tm@gmail.com>:
>>> [...] There
>>> is someone on the list who uses bcache on top of MD RAID5 AFAIR.
>> that would probably be me: backing device was an MD-RAID6 (we've
>> switched to RAID1 in the meantime because the data fits on a single
>> disk (netto) and we needed the HDD slots) and caching device is an
>> MD-RAID1 consisting of two SSDs.
> Me too, but the fact that my bcache cache device exploded with "missing
> magic" errors after less than a week makes me perhaps not a very good
> advert for this.
>
>>> I think I would choose to add bcache to each of the four harddisks,
>> If you'd do that with a single caching device, you're in for  contention. My gut feeling tells me that running a single bcache
>> backing device/caching device combo on top of MD-RAID is less  straining than running MD-RAID across a bunch of bcache devices with
>> a  common caching device: The MD advantage of spreading the load across  multiple "disks" is countered by accessing a common SSD.
> It all depends what your speed is like: if the load is at all seeky, an
> SSD's zero seek time will dominate. md atop bcache will certainly have
> far more SSD-write overhead than bcache atop md, because every time md
> does a full-stripe read or read-modify-write cycle you'll be burning
> holes in the SSD for no real speed benefit whatsoever...
>
> One thing to note about working atop md is that you do have to be
> careful about alignment: you don't want every md write to incur a
> read-modify-write cycle after you've gone to some lengths to tell your
> filesystems what the RAID offset is. You should compute an appropriate
> --data-offset for not only the array configuration now (so a multiple of
> your chunk size, at least, and possibly of your stripe size) but if you
> are planning to add more data spindles you should figure out a (larger)
> --data-offset that will retain appropriate alignment for plausible
> larger sets of spindles you might grow to in the future.
>
> (Also note that bcache does *not* communicate the RAID geometry through
> to overlying filesystems -- if you want decent write performance on
> RAID-5 or -6 you'll need to do that yourself, via mke2fs -E
> stripe_width/stride, mkfs.xfs sunit and swidth arguments, cryptsetup
> --align-payload, etc. You can use btrace on the underlying real block
> device to see if the requests look aligned after you're done, but before
> populating it with data. :) )
>

Re: Add Bcache to an existing Filesystem

[Nix]

On 27 Jun 2017, FERNANDO FREDIANI said:

> Hello Nix
>
> What exactlly you mean by "exploded" ? Has it stopped working at all
> and has that stopped the systems running on the top of it or it just
> stopped the caching device and the backing continued to work well.

It stopped the caching device and refused to mount the rootfs atop it
until I explicitly dissociated it. I haven't reactivated it since (I
might do so once I've rejigged my shutdown process to fully unmount
everything via the early root filesystem, but this system is a crucial
machine in active use and it takes ages to get around to such things).

See <https://marc.info/?l=linux-bcache&m=149687387202378>.

> Regarding my scenario, just to clarify I have 4 x /dev/sdX forming a
> /dev/md124 in RAID 10. This /dev/md124 is the backing device of my
> bcache which in turn has /dev/nvme0n1 as the caching device making
> /dev/bcache0. I guess this would be the most coming scenario right ?

I think so: s/RAID 10/journalled RAID 6/ and you have my setup (well,
part of it: the start of the disk is in uncached RAID-0, and the end is
in unjournalled, uncached RAID-6 with a write bitmap, but the middle
portion is a RAID-6 with bcache -- albeit, presently, in 'none' mode.)

-- 
NULL && (void)

Re: Add Bcache to an existing Filesystem

[Henk Slager]

> On top of that bcache device, I decided to add an LVM layer (making
> /dev/bcache* a PV), which is really helpful i.e. with the base problem of
> this thread: You can create additional bcache devices and pvmove your data
> to the new PV, without interrupting production.

Yes, if you start from scratch with some disk setup, including LVM can
 give you a lot of flexibility and almost always-online.
>> I
>> think I would choose to add bcache to each of the four harddisks,
>
>
> If you'd do that with a single caching device, you're in for contention. My
> gut feeling tells me that running a single bcache backing device/caching
> device combo on top of MD-RAID is less straining than running MD-RAID across
> a bunch of bcache devices with a common caching device: The MD advantage of
> spreading the load across multiple "disks" is countered by accessing a
> common SSD.

You are right, if you use MD-RAID, after second thought, bcache on top
of MD is a better choice in most cases as far as I can see. I have
been thinking to use RAID of MD instead of btrfs raid. I tried/tested
it with 3-disk RAID5 for some time, but still decided to use only
btrfs for handling multipe devices.

W.r.t. contention for 1 SSD bcaching 4 HDDs: I can definitely notice
this for some access patterns/tasks. I once chose btrfs raid10 when
all HDD were much older/slower and I wanted high enough transfer rates
on file level. But the HDDs are now all newer/faster model and the
strict need for high transfer rates is gone. It currently makes more
sense to use less SATA ports and also less spinning disks and re-use
HDD(s) for redundancy/backup on a file-system-level.

Re: Add Bcache to an existing Filesystem

[Eric Wheeler]

On Tue, 27 Jun 2017, Henk Slager wrote:

> > On top of that bcache device, I decided to add an LVM layer (making
> > /dev/bcache* a PV), which is really helpful i.e. with the base problem of
> > this thread: You can create additional bcache devices and pvmove your data
> > to the new PV, without interrupting production.
> 
> Yes, if you start from scratch with some disk setup, including LVM can
>  give you a lot of flexibility and almost always-online.
> >> I
> >> think I would choose to add bcache to each of the four harddisks,
> >
> >
> > If you'd do that with a single caching device, you're in for contention. My
> > gut feeling tells me that running a single bcache backing device/caching
> > device combo on top of MD-RAID is less straining than running MD-RAID across
> > a bunch of bcache devices with a common caching device: The MD advantage of
> > spreading the load across multiple "disks" is countered by accessing a
> > common SSD.
> 
> You are right, if you use MD-RAID, after second thought, bcache on top
> of MD is a better choice in most cases as far as I can see. I have
> been thinking to use RAID of MD instead of btrfs raid. I tried/tested
> it with 3-disk RAID5 for some time, but still decided to use only
> btrfs for handling multipe devices.

Definitely.  bcache atop of md, not md atop of bcache.  You would confuse 
the sequential-write logic with md on bcache.  

Also, with bcache on md, align using --data-offset.  8k is most certainly 
not your md chunk size and you don't want write amplification.  You might 
wish to align to your stride width (especially if raid5/6), so 
--data-offset=chunk_size*num_data_disks in sectors.

--
Eric Wheeler