How to create bcachefs?

How to create bcachefs?

[marcin]

Hi!
I'm spamming here with simple questions because it looks that 
documentation doesn't follow current state of bcachefs.

I've upgraded bcache-tools to b0f08fc1e30daafb274181234dae3408fb2eb69b 
and linux-bcache to 9c79c00b0d02f0da5e01ad5cd1bf4ae2ce421d5d.
And now...

1. How to create simple bcachefs on one disk, without any cache?
# ./bcache format --meta_replicas=2 /dev/sdd1
Please supply a device

Help displays:
  # ./bcache format --help
Usage: bcache format [OPTION]... <list of devices>
Create a new bcache volume from one or more devices

Options:
   -C, --cache=dev             Format a cache device
   -w, --block=size            block size
   -n, --btree_node=size       Btree node size, default 256k
       --metadata_csum_type=(none|crc32c|crc64)
                               Checksum type
       --data_csum_type=(none|crc32c|crc64)
                               Checksum type
       --compression_type=(none|gzip)
                               Compression type
       --encrypted             enable encryption
       --meta_replicas=#       number of metadata replicas
       --data_replicas=#       number of data replicas
       --error_action=(continue|readonly|panic)
                               Action to take on filesystem error
   -l, --label=label           label
       --uuid=uuid             filesystem UUID
       --fs_size=size          Size of filesystem on device
   -b, --bucket=size           bucket size
   -t, --tier=#                tier of subsequent devices
       --discard               Enable discards
   -q, --quiet                 reduce output to errors only
   -v, --verbose               increase output to include informational 
messages
       --help                  display this help and exit
       --version               output version information and exit

Report bugs to <linux-bcache@vger.kernel.org>

I don't know what is correct syntax to format block device.

2. How to create and mount bcachefs witch cache device?
# ./bcache format --compression_type=lz4  --meta_replicas=2 /dev/sdd1  
-C /dev/sde1
UUID:                   2cd51dcb-6af3-4e92-8b01-64bac27300b2
Set UUID:               fd005c93-26f4-4da3-a991-18630ea37997
version:                6
nbuckets:               20480
block_size:             1
bucket_size:            1024
nr_in_set:              1
nr_this_dev:            0
first_bucket:           3

and what next?
# mount /dev/sdd1 /mnt/test/
# # ls -l  /sys/fs/bcache/33802bf3-abe4-4d85-bd44-380c1fa9c3d0/|grep 
cache
-r--r--r-- 1 root root 4096 sie 19 20:02 btree_cache_size
lrwxrwxrwx 1 root root    0 sie 19 20:02 cache0 -> 
../../../devices/pci0000:00/0000:00:1f.2/ata6/host5/target5:0:0/5:0:0:0/block/sdd/sdd1/bcache
-r--r--r-- 1 root root 4096 sie 19 20:02 cache_available_percent

It looks for me that cache device /dev/sde1 isn't used. How to use 
/dev/sde1 as a cache device?

3. Tiering. Just simple: how to?:)


Thanks in advance for tips.
Marcin

Re: How to create bcachefs?

[Kent Overstreet]

On Fri, Aug 19, 2016 at 08:07:31PM +0200, marcin@mejor.pl wrote:
> Hi!
> I'm spamming here with simple questions because it looks that documentation
> doesn't follow current state of bcachefs.
> 
> I've upgraded bcache-tools to b0f08fc1e30daafb274181234dae3408fb2eb69b and
> linux-bcache to 9c79c00b0d02f0da5e01ad5cd1bf4ae2ce421d5d.
> And now...
> 
> 1. How to create simple bcachefs on one disk, without any cache?
> # ./bcache format --meta_replicas=2 /dev/sdd1
> Please supply a device
> 
> Help displays:
>  # ./bcache format --help
> Usage: bcache format [OPTION]... <list of devices>
> Create a new bcache volume from one or more devices
> 
> Options:
>   -C, --cache=dev             Format a cache device
>   -w, --block=size            block size
>   -n, --btree_node=size       Btree node size, default 256k
>       --metadata_csum_type=(none|crc32c|crc64)
>                               Checksum type
>       --data_csum_type=(none|crc32c|crc64)
>                               Checksum type
>       --compression_type=(none|gzip)
>                               Compression type
>       --encrypted             enable encryption
>       --meta_replicas=#       number of metadata replicas
>       --data_replicas=#       number of data replicas
>       --error_action=(continue|readonly|panic)
>                               Action to take on filesystem error
>   -l, --label=label           label
>       --uuid=uuid             filesystem UUID
>       --fs_size=size          Size of filesystem on device
>   -b, --bucket=size           bucket size
>   -t, --tier=#                tier of subsequent devices
>       --discard               Enable discards
>   -q, --quiet                 reduce output to errors only
>   -v, --verbose               increase output to include informational
> messages
>       --help                  display this help and exit
>       --version               output version information and exit
> 
> Report bugs to <linux-bcache@vger.kernel.org>
> 
> I don't know what is correct syntax to format block device.
> 
> 2. How to create and mount bcachefs witch cache device?
> # ./bcache format --compression_type=lz4  --meta_replicas=2 /dev/sdd1  -C
> /dev/sde1

Hey, sorry for the long delay, been sick past several days. The way arguments
were passed to bcache format was a holdover from old style make-bcache, and
didn't make much sense for bcachefs - -C was used for all devices, and --tier
specifies fast devices and slow devices.

I finally got around to redoing the option parsing so we don't need the -C
argument today - update your bcache tools, and the command you want is now:

bcache format --tier 0 /dev/sde1 --tier 1 /dev/sdd1

(assuming sde1 is your fast device and sdd1 is your slow device).

Replication is not yet supported (it's around 90% implemented, but I haven't
worked on it in ages and the tests haven't been covering it so you'll likely hit
bugs if you try it).

Compression still isn't _quite_ done, I reverted the accounting until I fix an
issue with copygc. You can enable it and nothing will break, but it won't
actually let you store more data in your fs than if it was uncompressed.

Re: How to create bcachefs?

[marcin]

W dniu 24.08.2016 o 08:52, Kent Overstreet pisze:
[...]

Hi!
> Hey, sorry for the long delay, been sick past several days.

I was afraid that I'm asking to easy question:) And I hope everything is
fine with you now!


> The way arguments
> were passed to bcache format was a holdover from old style make-bcache, 
> and
> didn't make much sense for bcachefs - -C was used for all devices, and 
> --tier
> specifies fast devices and slow devices.
> 
> I finally got around to redoing the option parsing so we don't need the 
> -C
> argument today - update your bcache tools, and the command you want is 
> now:
> 
> bcache format --tier 0 /dev/sde1 --tier 1 /dev/sdd1
> 
> (assuming sde1 is your fast device and sdd1 is your slow device).

Does it means that cache is unavailable and only tiering will be in 
bcachefs?
And... How to mount tiered FS? When I pass one device in mount I'm 
getting:
  bcache: bch_open_as_blockdevs() register_cache_set err insufficient 
devices


> Replication is not yet supported (it's around 90% implemented, but I 
> haven't
> worked on it in ages and the tests haven't been covering it so you'll 
> likely hit
> bugs if you try it).

If you tell me how to turn it on I'll try it. In latest bcache-tools
this option disappeared. Turning on using `echo 2 > 
/sys/fs/bcache/<uid>/options/{data_replicas,metadata_replicas}` doesn't 
work.

> Compression still isn't _quite_ done, I reverted the accounting until I 
> fix an
> issue with copygc. You can enable it and nothing will break, but it 
> won't
> actually let you store more data in your fs than if it was 
> uncompressed.


What means option in /sys/fs/..../options/posix_acl? I've got zero value 
but setfacl and getfacl shows that acl works (in theory, I didn't check 
if ACL really works).

I also have feature request: it would be nice to see which file (with 
path) has
checksum error, now it looks like this:
Aug 24 17:15:23 localhost kernel: [ 4692.929060] bcache (dm-10): IO
error on dm-10 for checksum error

I can't help with writing code but I can try to help testing bcachefs. 
I'm especially interested in using:
- cache
- tiering
- compression
- erasure encoding (I know that it isn't started)



Thank you,
Marcin

Re: How to create bcachefs?

[Eric Wheeler]

On Wed, 24 Aug 2016, marcin@mejor.pl wrote:

> W dniu 24.08.2016 o 08:52, Kent Overstreet pisze:
> [...]
> 
> Hi!
> > Hey, sorry for the long delay, been sick past several days.
> 
> I was afraid that I'm asking to easy question:) And I hope everything is
> fine with you now!
> 
> 
> > The way arguments
> > were passed to bcache format was a holdover from old style make-bcache, and
> > didn't make much sense for bcachefs - -C was used for all devices, and
> > --tier
> > specifies fast devices and slow devices.
> > 
> > I finally got around to redoing the option parsing so we don't need the -C
> > argument today - update your bcache tools, and the command you want is now:
> > 
> > bcache format --tier 0 /dev/sde1 --tier 1 /dev/sdd1

Neat!  How many performance tiers are (will be) supported?  Arbitrary?

--
Eric Wheeler



> > 
> > (assuming sde1 is your fast device and sdd1 is your slow device).
> 
> Does it means that cache is unavailable and only tiering will be in bcachefs?
> And... How to mount tiered FS? When I pass one device in mount I'm getting:
>  bcache: bch_open_as_blockdevs() register_cache_set err insufficient devices
> 
> 
> > Replication is not yet supported (it's around 90% implemented, but I haven't
> > worked on it in ages and the tests haven't been covering it so you'll likely
> > hit
> > bugs if you try it).
> 
> If you tell me how to turn it on I'll try it. In latest bcache-tools
> this option disappeared. Turning on using `echo 2 >
> /sys/fs/bcache/<uid>/options/{data_replicas,metadata_replicas}` doesn't work.
> 
> > Compression still isn't _quite_ done, I reverted the accounting until I fix
> > an
> > issue with copygc. You can enable it and nothing will break, but it won't
> > actually let you store more data in your fs than if it was uncompressed.
> 
> 
> What means option in /sys/fs/..../options/posix_acl? I've got zero value but
> setfacl and getfacl shows that acl works (in theory, I didn't check if ACL
> really works).
> 
> I also have feature request: it would be nice to see which file (with path)
> has
> checksum error, now it looks like this:
> Aug 24 17:15:23 localhost kernel: [ 4692.929060] bcache (dm-10): IO
> error on dm-10 for checksum error
> 
> I can't help with writing code but I can try to help testing bcachefs. I'm
> especially interested in using:
> - cache
> - tiering
> - compression
> - erasure encoding (I know that it isn't started)
> 
> 
> 
> Thank you,
> Marcin
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: How to create bcachefs?

[Kent Overstreet]

On Wed, Aug 24, 2016 at 11:21:02PM +0200, marcin@mejor.pl wrote:
> W dniu 24.08.2016 o 08:52, Kent Overstreet pisze:
> [...]
> 
> Hi!
> > Hey, sorry for the long delay, been sick past several days.
> 
> I was afraid that I'm asking to easy question:) And I hope everything is
> fine with you now!
> 
> 
> > The way arguments
> > were passed to bcache format was a holdover from old style make-bcache,
> > and
> > didn't make much sense for bcachefs - -C was used for all devices, and
> > --tier
> > specifies fast devices and slow devices.
> > 
> > I finally got around to redoing the option parsing so we don't need the
> > -C
> > argument today - update your bcache tools, and the command you want is
> > now:
> > 
> > bcache format --tier 0 /dev/sde1 --tier 1 /dev/sdd1
> > 
> > (assuming sde1 is your fast device and sdd1 is your slow device).
> 
> Does it means that cache is unavailable and only tiering will be in
> bcachefs?
> And... How to mount tiered FS? When I pass one device in mount I'm getting:
>  bcache: bch_open_as_blockdevs() register_cache_set err insufficient devices

you should be able to do
mount -t bcache /dev/sde1:/dev/sdd1 /mnt

or, there's incremental assembly - that's going to be handled by the bcache tool
eventually, but for now you can register each device via
/sys/fs/bcache/register, and then mount one of the block devices.

> > Replication is not yet supported (it's around 90% implemented, but I
> > haven't
> > worked on it in ages and the tests haven't been covering it so you'll
> > likely hit
> > bugs if you try it).
> 
> If you tell me how to turn it on I'll try it. In latest bcache-tools
> this option disappeared. Turning on using `echo 2 >
> /sys/fs/bcache/<uid>/options/{data_replicas,metadata_replicas}` doesn't
> work.

Wouldn't be any point, you'd undoubtedly hit bugs that I don't have time to work
on yet. Current priorities are finishing encryption and compression...

> > Compression still isn't _quite_ done, I reverted the accounting until I
> > fix an
> > issue with copygc. You can enable it and nothing will break, but it
> > won't
> > actually let you store more data in your fs than if it was uncompressed.
> 
> 
> What means option in /sys/fs/..../options/posix_acl? I've got zero value but
> setfacl and getfacl shows that acl works (in theory, I didn't check if ACL
> really works).

hmm, option must be buggered. I'll have a look.

> I also have feature request: it would be nice to see which file (with path)
> has
> checksum error, now it looks like this:
> Aug 24 17:15:23 localhost kernel: [ 4692.929060] bcache (dm-10): IO
> error on dm-10 for checksum error

that's a good idea - though it'd take some plumbing to get a path, and we don't
even always have a path (may have just an inode number). I'll make a note to
have a look.

> I can't help with writing code but I can try to help testing bcachefs. I'm
> especially interested in using:
> - cache
> - tiering
> - compression
> - erasure encoding (I know that it isn't started)

I am thinking about working on replication and erasure coding after I finish
what's currently lined up, erasure coding should be a fun project.

Lots to do...

Re: How to create bcachefs?

[Kent Overstreet]

On Wed, Aug 24, 2016 at 04:12:22PM -0700, Eric Wheeler wrote:
> On Wed, 24 Aug 2016, marcin@mejor.pl wrote:
> 
> > W dniu 24.08.2016 o 08:52, Kent Overstreet pisze:
> > [...]
> > 
> > Hi!
> > > Hey, sorry for the long delay, been sick past several days.
> > 
> > I was afraid that I'm asking to easy question:) And I hope everything is
> > fine with you now!
> > 
> > 
> > > The way arguments
> > > were passed to bcache format was a holdover from old style make-bcache, and
> > > didn't make much sense for bcachefs - -C was used for all devices, and
> > > --tier
> > > specifies fast devices and slow devices.
> > > 
> > > I finally got around to redoing the option parsing so we don't need the -C
> > > argument today - update your bcache tools, and the command you want is now:
> > > 
> > > bcache format --tier 0 /dev/sde1 --tier 1 /dev/sdd1
> 
> Neat!  How many performance tiers are (will be) supported?  Arbitrary?

Currently the code just handles two, but the on disk format allows for up to 16.

Re: How to create bcachefs?

[Christopher James Halse Rogers]

On Thu, Aug 25, 2016 at 7:21 AM, marcin@mejor.pl wrote:
> W dniu 24.08.2016 o 08:52, Kent Overstreet pisze:
> [...]
> 
> Hi!
>> Hey, sorry for the long delay, been sick past several days.
> 
> I was afraid that I'm asking to easy question:) And I hope everything 
> is
> fine with you now!
> 
> 
>> The way arguments
>> were passed to bcache format was a holdover from old style 
>> make-bcache, \x7fand
>> didn't make much sense for bcachefs - -C was used for all devices, 
>> and \x7f--tier
>> specifies fast devices and slow devices.
>> 
>> I finally got around to redoing the option parsing so we don't need 
>> the \x7f-C
>> argument today - update your bcache tools, and the command you want 
>> is \x7fnow:
>> 
>> bcache format --tier 0 /dev/sde1 --tier 1 /dev/sdd1
>> 
>> (assuming sde1 is your fast device and sdd1 is your slow device).
> 
> Does it means that cache is unavailable and only tiering will be in 
> bcachefs?
> And... How to mount tiered FS? When I pass one device in mount I'm 
> getting:
>  bcache: bch_open_as_blockdevs() register_cache_set err insufficient 
> devices

Tiering gets you all the advantages of caching, plus you can (with some 
effort) have the combined filesystem size be the sum of the SSD + HDD 
capacities, rather than the capacity be determined solely by the 
capacity of the slow tier (this is not currently the case for bcachefs).

To mount a tiered FS you can either pass all the relevant devices to 
mount, like:
    mount -t bcache /dev/sda:/dev/sdb:/dev/sdc /mountpoint
(I haven't tested this recently) or ensure that all the relevant 
devices have been registered with bcache, like so:
    echo /dev/sda | sudo tee /sys/fs/bcache/register_quiet
    echo /dev/sdb | sudo tee /sys/fs/bcache/register_quiet
    echo /dev/sdc | sudo tee /sys/fs/bcache/register_quiet
    mount -t bcache /dev/sdb /mountpoint
(you can use any member of the set as the device to mount in this case).

Re: How to create bcachefs?

[Marcin Mirosław]

W dniu 25.08.2016 o 02:03, Christopher James Halse Rogers pisze:

Hi!

> On Thu, Aug 25, 2016 at 7:21 AM, marcin@mejor.pl wrote:
[...]
>> Does it means that cache is unavailable and only tiering will be in
>> bcachefs?
>> And... How to mount tiered FS? When I pass one device in mount I'm
>> getting:
>>  bcache: bch_open_as_blockdevs() register_cache_set err insufficient
>> devices
> 
> Tiering gets you all the advantages of caching, plus you can (with some
> effort) have the combined filesystem size be the sum of the SSD + HDD
> capacities, rather than the capacity be determined solely by the
> capacity of the slow tier (this is not currently the case for bcachefs).

I think that cacheing has at least such advantages over tiering:
- allow fast read and write to files compressed with slow alghoritm (gzip)
- can be optimized for using SSD drives

> To mount a tiered FS you can either pass all the relevant devices to
> mount, like:
>    mount -t bcache /dev/sda:/dev/sdb:/dev/sdc /mountpoint
> (I haven't tested this recently) or ensure that all the relevant devices
> have been registered with bcache, like so:
>    echo /dev/sda | sudo tee /sys/fs/bcache/register_quiet
>    echo /dev/sdb | sudo tee /sys/fs/bcache/register_quiet
>    echo /dev/sdc | sudo tee /sys/fs/bcache/register_quiet
>    mount -t bcache /dev/sdb /mountpoint
> (you can use any member of the set as the device to mount in this case).


Thanks for tip, good to know other way to mount.

Marcin

Re: How to create bcachefs?

[Christopher James Halse Rogers]

On Thu, Aug 25, 2016 at 7:21 PM, Marcin Mirosław <marcin@mejor.pl> 
wrote:
> W dniu 25.08.2016 o 02:03, Christopher James Halse Rogers pisze:
> 
> Hi!
> 
>>  On Thu, Aug 25, 2016 at 7:21 AM, marcin@mejor.pl wrote:
> [...]
>>>  Does it means that cache is unavailable and only tiering will be in
>>>  bcachefs?
>>>  And... How to mount tiered FS? When I pass one device in mount I'm
>>>  getting:
>>>   bcache: bch_open_as_blockdevs() register_cache_set err 
>>> insufficient
>>>  devices
>> 
>>  Tiering gets you all the advantages of caching, plus you can (with 
>> some
>>  effort) have the combined filesystem size be the sum of the SSD + 
>> HDD
>>  capacities, rather than the capacity be determined solely by the
>>  capacity of the slow tier (this is not currently the case for 
>> bcachefs).
> 
> I think that cacheing has at least such advantages over tiering:
> - allow fast read and write to files compressed with slow alghoritm 
> (gzip)

I think this is getting into ā€œwhat should we call this thingā€ 
arguments. A naive cache is just going to promote the gzipped data to 
the fast storage. On the other end, there's nothing much preventing a 
sophisticated tiering system from compressing/decompressing as a part 
of tier demotion/promotion.

> - can be optimized for using SSD drives

It's not clear to me how? Tiering and caching are doing the same sort 
of things.

Re: How to create bcachefs?

[Marcin Mirosław]

W dniu 25.08.2016 o 13:09, Christopher James Halse Rogers pisze:
> 
> 
> On Thu, Aug 25, 2016 at 7:21 PM, Marcin Mirosław <marcin@mejor.pl> wrote:
>> W dniu 25.08.2016 o 02:03, Christopher James Halse Rogers pisze:
>>
>> Hi!
>>
>>>  On Thu, Aug 25, 2016 at 7:21 AM, marcin@mejor.pl wrote:
>> [...]
>>>>  Does it means that cache is unavailable and only tiering will be in
>>>>  bcachefs?
>>>>  And... How to mount tiered FS? When I pass one device in mount I'm
>>>>  getting:
>>>>   bcache: bch_open_as_blockdevs() register_cache_set err insufficient
>>>>  devices
>>>
>>>  Tiering gets you all the advantages of caching, plus you can (with some
>>>  effort) have the combined filesystem size be the sum of the SSD + HDD
>>>  capacities, rather than the capacity be determined solely by the
>>>  capacity of the slow tier (this is not currently the case for
>>> bcachefs).
>>
>> I think that cacheing has at least such advantages over tiering:
>> - allow fast read and write to files compressed with slow alghoritm
>> (gzip)
> 
> I think this is getting into ā€œwhat should we call this thingā€
> arguments. A naive cache is just going to promote the gzipped data to
> the fast storage. On the other end, there's nothing much preventing a
> sophisticated tiering system from compressing/decompressing as a part of
> tier demotion/promotion.

When (de|re)compressing would be part of demotion or promotion then I
agree, cache device doesn't have advantages.

>> - can be optimized for using SSD drives
> 
> It's not clear to me how? Tiering and caching are doing the same sort of
> things.
I don't know how bcachefs handle SSD drives. SSD are faster in
sequential write, I'm not sure if it could be achieved using tier
device. Using cache device on disk format can be different than using in
bcachefs and can turn random writes into sequential writes.

Marcin

Re: How to create bcachefs?

[Christopher James Halse Rogers]

On Thu, Aug 25, 2016 at 10:11 PM, Marcin Mirosław <marcin@mejor.pl> 
wrote:
> W dniu 25.08.2016 o 13:09, Christopher James Halse Rogers pisze:
>> 
>> 
>>  On Thu, Aug 25, 2016 at 7:21 PM, Marcin Mirosław 
>> <marcin@mejor.pl> wrote:
>>>  W dniu 25.08.2016 o 02:03, Christopher James Halse Rogers pisze:
>>> 
>>>  Hi!
>>> 
>>>>   On Thu, Aug 25, 2016 at 7:21 AM, marcin@mejor.pl wrote:
>>>  [...]
>>>>>   Does it means that cache is unavailable and only tiering will 
>>>>> be in
>>>>>   bcachefs?
>>>>>   And... How to mount tiered FS? When I pass one device in mount 
>>>>> I'm
>>>>>   getting:
>>>>>    bcache: bch_open_as_blockdevs() register_cache_set err 
>>>>> insufficient
>>>>>   devices
>>>> 
>>>>   Tiering gets you all the advantages of caching, plus you can 
>>>> (with some
>>>>   effort) have the combined filesystem size be the sum of the SSD 
>>>> + HDD
>>>>   capacities, rather than the capacity be determined solely by the
>>>>   capacity of the slow tier (this is not currently the case for
>>>>  bcachefs).
>>> 
>>>  I think that cacheing has at least such advantages over tiering:
>>>  - allow fast read and write to files compressed with slow alghoritm
>>>  (gzip)
>> 
>>  I think this is getting into ā€œwhat should we call this 
>> thingā€
>>  arguments. A naive cache is just going to promote the gzipped data 
>> to
>>  the fast storage. On the other end, there's nothing much preventing 
>> a
>>  sophisticated tiering system from compressing/decompressing as a 
>> part of
>>  tier demotion/promotion.
> 
> When (de|re)compressing would be part of demotion or promotion then I
> agree, cache device doesn't have advantages.

You also don't get (de|re)compression for free in a caching strategy 
depending on where you put your cache. For example, a traditional 
bcache cache backing a compressed filesystem will only ever see 
compressed data.

> 
>>>  - can be optimized for using SSD drives
>> 
>>  It's not clear to me how? Tiering and caching are doing the same 
>> sort of
>>  things.
> I don't know how bcachefs handle SSD drives. SSD are faster in
> sequential write, I'm not sure if it could be achieved using tier
> device. Using cache device on disk format can be different than using 
> in
> bcachefs and can turn random writes into sequential writes.

I believe you mean *HDDs* are faster in sequential write? This is 
exactly what tiering gets you - all writes go to the fast, SSD 
storage¹, and then data are migrated off to the slower HDD in big, 
sequential chunks.

¹: Indeed, IIRC this is the only dataloss bug I've found in bcachefs - 
I wrote too much data too fast, the migration from tier 0 to tier 1 
couldn't keep up, and tier 0 filled up to the point where it could no 
longer write necessary journal entries, resulting in the filesystem 
being unmountable.

Re: How to create bcachefs?

[Eric Wheeler]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3348 bytes --]

On Fri, 26 Aug 2016, Christopher James Halse Rogers wrote:

> 
> 
> On Thu, Aug 25, 2016 at 10:11 PM, Marcin Mirosław <marcin@mejor.pl> wrote:
> > W dniu 25.08.2016 o 13:09, Christopher James Halse Rogers pisze:
> > > 
> > > 
> > > On Thu, Aug 25, 2016 at 7:21 PM, Marcin Mirosław <marcin@mejor.pl> wrote:
> > > >  W dniu 25.08.2016 o 02:03, Christopher James Halse Rogers pisze:
> > > > 
> > > >  Hi!
> > > > 
> > > > >   On Thu, Aug 25, 2016 at 7:21 AM, marcin@mejor.pl wrote:
> > > >  [...]
> > > > > >   Does it means that cache is unavailable and only tiering will be
> > > > > > in
> > > > > >   bcachefs?
> > > > > >   And... How to mount tiered FS? When I pass one device in mount 
> > > > > > I'm
> > > > > >   getting:
> > > > > >    bcache: bch_open_as_blockdevs() register_cache_set err
> > > > > > insufficient
> > > > > >   devices
> > > > > 
> > > > >   Tiering gets you all the advantages of caching, plus you can (with
> > > > > some
> > > > >   effort) have the combined filesystem size be the sum of the SSD +
> > > > > HDD
> > > > >   capacities, rather than the capacity be determined solely by the
> > > > >   capacity of the slow tier (this is not currently the case for
> > > > >  bcachefs).
> > > > 
> > > >  I think that cacheing has at least such advantages over tiering:
> > > >  - allow fast read and write to files compressed with slow alghoritm
> > > >  (gzip)
> > > 
> > > I think this is getting into ā€œwhat should we call this thingā€?
> > > arguments. A naive cache is just going to promote the gzipped data to
> > > the fast storage. On the other end, there's nothing much preventing a
> > > sophisticated tiering system from compressing/decompressing as a part of
> > >  tier demotion/promotion.
> > 
> > When (de|re)compressing would be part of demotion or promotion then I
> > agree, cache device doesn't have advantages.
> 
> You also don't get (de|re)compression for free in a caching strategy depending
> on where you put your cache. For example, a traditional bcache cache backing a
> compressed filesystem will only ever see compressed data.

Nor crypto.  Best to compress->encrypt->store.


--
Eric Wheeler


> 
> > 
> > > >  - can be optimized for using SSD drives
> > > 
> > > It's not clear to me how? Tiering and caching are doing the same sort of
> > >  things.
> > I don't know how bcachefs handle SSD drives. SSD are faster in
> > sequential write, I'm not sure if it could be achieved using tier
> > device. Using cache device on disk format can be different than using in
> > bcachefs and can turn random writes into sequential writes.
> 
> I believe you mean *HDDs* are faster in sequential write? This is exactly what
> tiering gets you - all writes go to the fast, SSD storage¹, and then data are
> migrated off to the slower HDD in big, sequential chunks.
> 
> ¹: Indeed, IIRC this is the only dataloss bug I've found in bcachefs - I wrote
> too much data too fast, the migration from tier 0 to tier 1 couldn't keep up,
> and tier 0 filled up to the point where it could no longer write necessary
> journal entries, resulting in the filesystem being unmountable.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: How to create bcachefs?

[Marcin Mirosław]

W dniu 26.08.2016 o 03:06, Christopher James Halse Rogers pisze:
Hi!


> On Thu, Aug 25, 2016 at 10:11 PM, Marcin Mirosław <marcin@mejor.pl> wrote:
>> W dniu 25.08.2016 o 13:09, Christopher James Halse Rogers pisze:
>>>
>>>
>>>  On Thu, Aug 25, 2016 at 7:21 PM, Marcin Mirosław <marcin@mejor.pl>
>>> wrote:
>>>>  W dniu 25.08.2016 o 02:03, Christopher James Halse Rogers pisze:
>>>>
>>>>  Hi!
>>>>
>>>>>   On Thu, Aug 25, 2016 at 7:21 AM, marcin@mejor.pl wrote:
>>>>  [...]
>>>>>>   Does it means that cache is unavailable and only tiering will be in
>>>>>>   bcachefs?
>>>>>>   And... How to mount tiered FS? When I pass one device in mount I'm
>>>>>>   getting:
>>>>>>    bcache: bch_open_as_blockdevs() register_cache_set err
>>>>>> insufficient
>>>>>>   devices
>>>>>
>>>>>   Tiering gets you all the advantages of caching, plus you can
>>>>> (with some
>>>>>   effort) have the combined filesystem size be the sum of the SSD +
>>>>> HDD
>>>>>   capacities, rather than the capacity be determined solely by the
>>>>>   capacity of the slow tier (this is not currently the case for
>>>>>  bcachefs).
>>>>
>>>>  I think that cacheing has at least such advantages over tiering:
>>>>  - allow fast read and write to files compressed with slow alghoritm
>>>>  (gzip)
>>>
>>>  I think this is getting into ā€œwhat should we call this thingā€
>>>  arguments. A naive cache is just going to promote the gzipped data to
>>>  the fast storage. On the other end, there's nothing much preventing a
>>>  sophisticated tiering system from compressing/decompressing as a
>>> part of
>>>  tier demotion/promotion.
>>
>> When (de|re)compressing would be part of demotion or promotion then I
>> agree, cache device doesn't have advantages.
> 
> You also don't get (de|re)compression for free in a caching strategy
> depending on where you put your cache. For example, a traditional bcache
> cache backing a compressed filesystem will only ever see compressed data.
> 
>>
>>>>  - can be optimized for using SSD drives
>>>
>>>  It's not clear to me how? Tiering and caching are doing the same
>>> sort of
>>>  things.
>> I don't know how bcachefs handle SSD drives. SSD are faster in
>> sequential write, I'm not sure if it could be achieved using tier
>> device. Using cache device on disk format can be different than using in
>> bcachefs and can turn random writes into sequential writes.
> 
> I believe you mean *HDDs* are faster in sequential write? This is
> exactly what tiering gets you - all writes go to the fast, SSD storage¹,
> and then data are migrated off to the slower HDD in big, sequential chunks.


SSD deals great with random reads but both SSD and HDD don't like random
writes.
I'm pasting a few links, you can read about problems with SSD
performance. Also SSD performance is going down after a dozens minutes
of intensive writes.

https://github.com/akiradeveloper/dm-writeboost
https://www.usenix.org/conference/fast12/sfs-random-write-considered-harmful-solid-state-drives
http://codecapsule.com/2014/02/12/coding-for-ssds-part-5-access-patterns-and-system-optimizations/
http://www.tomsitpro.com/articles/enterprise-ssd-testing,2-863.html


> ¹: Indeed, IIRC this is the only dataloss bug I've found in bcachefs - I
> wrote too much data too fast, the migration from tier 0 to tier 1
> couldn't keep up, and tier 0 filled up to the point where it could no
> longer write necessary journal entries, resulting in the filesystem
> being unmountable.