Multi-Level Caching

Multi-Level Caching

[Andrea Tomassetti]

Hi,
I know that bcache doesn't natively support multi-level caching but I
was playing with it and found this interesting "workaround":
  make-bcache -B /dev/vdb -C /dev/vdc
the above command will generate a /dev/bcache0 device that we can now
use as backing (or cached) device:
  make-bcache -B /dev/bcache0 -C /dev/vdd
This will make the kernel panic because the driver is trying to create
a duplicated "bcache" folder under /sys/block/bcache0/ .
So, simply patching the code inside register_bdev to create a folder
"bcache2" if "bcache" already exists does the trick.
Now I have:
vdb                       252:16   0    5G  0 disk
└─bcache0                 251:0    0    5G  0 disk
  └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
vdc                       252:32   0   10G  0 disk
└─bcache0                 251:0    0    5G  0 disk
  └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
vdd                       252:48   0    5G  0 disk
└─bcache1                 251:128  0    5G  0 disk /mnt/bcache1

Is anyone using this functionality? I assume not, because by default
it doesn't work.
Is there any good reason why this doesn't work by default?

I tried to understand how data will be read out of /dev/bcache1: will
the /dev/vdd cache, secondly created cache device, be interrogated
first and then will it be the turn of /dev/vdc ?
Meaning: can we consider that now the layer structure is

vdd
└─vdc
       └─bcache0
             └─bcache1
?

Re: Multi-Level Caching

[Coly Li]

> 2023年1月26日 19:30，Andrea Tomassetti <andrea.tomassetti-opensource@devo.com> 写道：
> 
> Hi,
> I know that bcache doesn't natively support multi-level caching but I
> was playing with it and found this interesting "workaround":
>  make-bcache -B /dev/vdb -C /dev/vdc
> the above command will generate a /dev/bcache0 device that we can now
> use as backing (or cached) device:
>  make-bcache -B /dev/bcache0 -C /dev/vdd
> This will make the kernel panic because the driver is trying to create
> a duplicated "bcache" folder under /sys/block/bcache0/ .
> So, simply patching the code inside register_bdev to create a folder
> "bcache2" if "bcache" already exists does the trick.
> Now I have:
> vdb                       252:16   0    5G  0 disk
> └─bcache0                 251:0    0    5G  0 disk
>  └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> vdc                       252:32   0   10G  0 disk
> └─bcache0                 251:0    0    5G  0 disk
>  └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> vdd                       252:48   0    5G  0 disk
> └─bcache1                 251:128  0    5G  0 disk /mnt/bcache1
> 
> Is anyone using this functionality? I assume not, because by default
> it doesn't work.
> Is there any good reason why this doesn't work by default?
> 
> I tried to understand how data will be read out of /dev/bcache1: will
> the /dev/vdd cache, secondly created cache device, be interrogated
> first and then will it be the turn of /dev/vdc ?
> Meaning: can we consider that now the layer structure is
> 
> vdd
> └─vdc
>       └─bcache0
>             └─bcache1
> ?

IIRC, there was a patch tried to achieve similar purpose. I was not supportive for this idea because I didn’t see really useful use case.
In general, extra layer cache means extra latency in the I/O path. What I see in practical deployments are, people try very hard to minimize the cache layer and place it close to application.

Introduce stackable bcache for itself may work, but I don’t see real usage yet, and no motivation to maintain such usage still.

Thanks.

Coly Li

Re: Multi-Level Caching

[Eric Wheeler]

[-- Attachment #1: Type: text/plain, Size: 2586 bytes --]

On Tue, 31 Jan 2023, Coly Li wrote:
> > 2023年1月26日 19:30，Andrea Tomassetti <andrea.tomassetti-opensource@devo.com> 写道：
> > 
> > Hi,
> > I know that bcache doesn't natively support multi-level caching but I
> > was playing with it and found this interesting "workaround":
> >  make-bcache -B /dev/vdb -C /dev/vdc
> > the above command will generate a /dev/bcache0 device that we can now
> > use as backing (or cached) device:
> >  make-bcache -B /dev/bcache0 -C /dev/vdd
> > This will make the kernel panic because the driver is trying to create
> > a duplicated "bcache" folder under /sys/block/bcache0/ .

panic? Ouch.

> > So, simply patching the code inside register_bdev to create a folder
> > "bcache2" if "bcache" already exists does the trick.
> > Now I have:
> > vdb                       252:16   0    5G  0 disk
> > └─bcache0                 251:0    0    5G  0 disk
> >  └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> > vdc                       252:32   0   10G  0 disk
> > └─bcache0                 251:0    0    5G  0 disk
> >  └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> > vdd                       252:48   0    5G  0 disk
> > └─bcache1                 251:128  0    5G  0 disk /mnt/bcache1
> > 
> > Is anyone using this functionality? I assume not, because by default
> > it doesn't work.
> > Is there any good reason why this doesn't work by default?
> > 
> > I tried to understand how data will be read out of /dev/bcache1: will
> > the /dev/vdd cache, secondly created cache device, be interrogated
> > first and then will it be the turn of /dev/vdc ?
> > Meaning: can we consider that now the layer structure is
> > 
> > vdd
> > └─vdc
> >       └─bcache0
> >             └─bcache1
> > ?
> 
> IIRC, there was a patch tried to achieve similar purpose. I was not 
> supportive for this idea because I didn’t see really useful use case. In 
> general, extra layer cache means extra latency in the I/O path. What I 
> see in practical deployments are, people try very hard to minimize the 
> cache layer and place it close to application.
> 
> Introduce stackable bcache for itself may work, but I don’t see real 
> usage yet, and no motivation to maintain such usage still.

Hi Coly, 

If Andrea's patch is simple and prevents a panic, would you consider 
accepting it?  Users should not be able to crash the kernel no matter what 
they do. Perhaps this should be considered a bug.

If so, then Andrea, can you post your suggested patch for review?

-Eric

> 
> Thanks.
> 
> Coly Li

Re: Multi-Level Caching

[mingzhe]

在 2023/1/31 23:51, Coly Li 写道:
> 
> 
>> 2023年1月26日 19:30，Andrea Tomassetti <andrea.tomassetti-opensource@devo.com> 写道：
>>
>> Hi,
>> I know that bcache doesn't natively support multi-level caching but I
>> was playing with it and found this interesting "workaround":
>>   make-bcache -B /dev/vdb -C /dev/vdc
>> the above command will generate a /dev/bcache0 device that we can now
>> use as backing (or cached) device:
>>   make-bcache -B /dev/bcache0 -C /dev/vdd
>> This will make the kernel panic because the driver is trying to create
>> a duplicated "bcache" folder under /sys/block/bcache0/ .
>> So, simply patching the code inside register_bdev to create a folder
>> "bcache2" if "bcache" already exists does the trick.
>> Now I have:
>> vdb                       252:16   0    5G  0 disk
>> └─bcache0                 251:0    0    5G  0 disk
>>   └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
>> vdc                       252:32   0   10G  0 disk
>> └─bcache0                 251:0    0    5G  0 disk
>>   └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
>> vdd                       252:48   0    5G  0 disk
>> └─bcache1                 251:128  0    5G  0 disk /mnt/bcache1
>>
>> Is anyone using this functionality? I assume not, because by default
>> it doesn't work.
>> Is there any good reason why this doesn't work by default?
>>
>> I tried to understand how data will be read out of /dev/bcache1: will
>> the /dev/vdd cache, secondly created cache device, be interrogated
>> first and then will it be the turn of /dev/vdc ?
>> Meaning: can we consider that now the layer structure is
>>
>> vdd
>> └─vdc
>>        └─bcache0
>>              └─bcache1
>> ?
> 
> IIRC, there was a patch tried to achieve similar purpose. I was not supportive for this idea because I didn’t see really useful use case.

Hi, Coly

Maybe we have a case like this. We are considering make-bcache a hdd as 
a cache device and create a flash device in it, and then using the flash 
device as a backing. So， completely converts writeback to sequential 
writes.

However, we found that there may be many unknown problems in the flash 
device, such as the created size, etc.

For now, we've put it due to time，but we think it might be a good thing 
to do. We also have some patches, I will post them.

mingzhe

> In general, extra layer cache means extra latency in the I/O path. What I see in practical deployments are, people try very hard to minimize the cache layer and place it close to application.
> 
> Introduce stackable bcache for itself may work, but I don’t see real usage yet, and no motivation to maintain such usage still.
> 
> Thanks.
> 
> Coly Li

Re: Multi-Level Caching

[Andrea Tomassetti]

Hi Coly,

On Wed, Feb 1, 2023 at 4:03 AM mingzhe <mingzhe.zou@easystack.cn> wrote:
>
>
>
> 在 2023/1/31 23:51, Coly Li 写道:
> >
> >
> >> 2023年1月26日 19:30，Andrea Tomassetti <andrea.tomassetti-opensource@devo.com> 写道：
> >>
> >> Hi,
> >> I know that bcache doesn't natively support multi-level caching but I
> >> was playing with it and found this interesting "workaround":
> >>   make-bcache -B /dev/vdb -C /dev/vdc
> >> the above command will generate a /dev/bcache0 device that we can now
> >> use as backing (or cached) device:
> >>   make-bcache -B /dev/bcache0 -C /dev/vdd
> >> This will make the kernel panic because the driver is trying to create
> >> a duplicated "bcache" folder under /sys/block/bcache0/ .
> >> So, simply patching the code inside register_bdev to create a folder
> >> "bcache2" if "bcache" already exists does the trick.
> >> Now I have:
> >> vdb                       252:16   0    5G  0 disk
> >> └─bcache0                 251:0    0    5G  0 disk
> >>   └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> >> vdc                       252:32   0   10G  0 disk
> >> └─bcache0                 251:0    0    5G  0 disk
> >>   └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> >> vdd                       252:48   0    5G  0 disk
> >> └─bcache1                 251:128  0    5G  0 disk /mnt/bcache1
> >>
> >> Is anyone using this functionality? I assume not, because by default
> >> it doesn't work.
> >> Is there any good reason why this doesn't work by default?
> >>
> >> I tried to understand how data will be read out of /dev/bcache1: will
> >> the /dev/vdd cache, secondly created cache device, be interrogated
> >> first and then will it be the turn of /dev/vdc ?
> >> Meaning: can we consider that now the layer structure is
> >>
> >> vdd
> >> └─vdc
> >>        └─bcache0
> >>              └─bcache1
> >> ?
> >
> > IIRC, there was a patch tried to achieve similar purpose. I was not supportive for this idea because I didn’t see really useful use case.
I didn't test it extensively but it looks like that the patch to
achieve this is just a one-line patch, it could be very beneficial. (I
just realized that mingzhe sent a relevant patch on this, thank for
your work)
Our use case will be to be able to take advantage of different
blocking devices that differ in performance and cost.
Some of these blocking devices are ephemeral and not suitable for wb
cache mode, but stacking them with non-ephemeral ones would be a very
nice and neat solution.

Cheers,
Andrea


>
> Hi, Coly
>
> Maybe we have a case like this. We are considering make-bcache a hdd as
> a cache device and create a flash device in it, and then using the flash
> device as a backing. So， completely converts writeback to sequential
> writes.
>
> However, we found that there may be many unknown problems in the flash
> device, such as the created size, etc.
>
> For now, we've put it due to time，but we think it might be a good thing
> to do. We also have some patches, I will post them.
>
> mingzhe
>
> > In general, extra layer cache means extra latency in the I/O path. What I see in practical deployments are, people try very hard to minimize the cache layer and place it close to application.
> >
> > Introduce stackable bcache for itself may work, but I don’t see real usage yet, and no motivation to maintain such usage still.
> >
> > Thanks.
> >
> > Coly Li

Re: Multi-Level Caching

[Eric Wheeler]

[-- Attachment #1: Type: text/plain, Size: 3930 bytes --]

On Wed, 1 Feb 2023, Andrea Tomassetti wrote:
> Hi Coly,
> 
> On Wed, Feb 1, 2023 at 4:03 AM mingzhe <mingzhe.zou@easystack.cn> wrote:
> > 在 2023/1/31 23:51, Coly Li 写道:
> > >> 2023年1月26日 19:30，Andrea Tomassetti <andrea.tomassetti-opensource@devo.com> 写道：
> > >>
> > >> Hi,
> > >> I know that bcache doesn't natively support multi-level caching but I
> > >> was playing with it and found this interesting "workaround":
> > >>   make-bcache -B /dev/vdb -C /dev/vdc
> > >> the above command will generate a /dev/bcache0 device that we can now
> > >> use as backing (or cached) device:
> > >>   make-bcache -B /dev/bcache0 -C /dev/vdd
> > >> This will make the kernel panic because the driver is trying to create
> > >> a duplicated "bcache" folder under /sys/block/bcache0/ .
> > >> So, simply patching the code inside register_bdev to create a folder
> > >> "bcache2" if "bcache" already exists does the trick.
> > >> Now I have:
> > >> vdb                       252:16   0    5G  0 disk
> > >> └─bcache0                 251:0    0    5G  0 disk
> > >>   └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> > >> vdc                       252:32   0   10G  0 disk
> > >> └─bcache0                 251:0    0    5G  0 disk
> > >>   └─bcache1               251:128  0    5G  0 disk /mnt/bcache1
> > >> vdd                       252:48   0    5G  0 disk
> > >> └─bcache1                 251:128  0    5G  0 disk /mnt/bcache1
> > >>
> > >> Is anyone using this functionality? I assume not, because by default
> > >> it doesn't work.
> > >> Is there any good reason why this doesn't work by default?
> > >>
> > >> I tried to understand how data will be read out of /dev/bcache1: will
> > >> the /dev/vdd cache, secondly created cache device, be interrogated
> > >> first and then will it be the turn of /dev/vdc ?
> > >> Meaning: can we consider that now the layer structure is
> > >>
> > >> vdd
> > >> └─vdc
> > >>        └─bcache0
> > >>              └─bcache1
> > >> ?
> > >
> > > IIRC, there was a patch tried to achieve similar purpose. I was not supportive for this idea because I didn’t see really useful use case.
> I didn't test it extensively but it looks like that the patch to
> achieve this is just a one-line patch, it could be very beneficial. (I
> just realized that mingzhe sent a relevant patch on this, thank for
> your work)
> Our use case will be to be able to take advantage of different
> blocking devices that differ in performance and cost.
> Some of these blocking devices are ephemeral and not suitable for wb
> cache mode, but stacking them with non-ephemeral ones would be a very
> nice and neat solution.

I like that!  

I've always wondered how a 64GB writethrough cache ram cache (/dev/ram0 or 
/dev/zram0) would perform on top of an NVMe backed by spinning disks.

I've also wondered about this kind of heirarchy:

/dev/ram0 -> /dev/zram0 -> /dev/nvme0n1 -> /dev/sda

--
Eric Wheeler



> 
> Cheers,
> Andrea
> 
> 
> >
> > Hi, Coly
> >
> > Maybe we have a case like this. We are considering make-bcache a hdd as
> > a cache device and create a flash device in it, and then using the flash
> > device as a backing. So， completely converts writeback to sequential
> > writes.
> >
> > However, we found that there may be many unknown problems in the flash
> > device, such as the created size, etc.
> >
> > For now, we've put it due to time，but we think it might be a good thing
> > to do. We also have some patches, I will post them.
> >
> > mingzhe
> >
> > > In general, extra layer cache means extra latency in the I/O path. What I see in practical deployments are, people try very hard to minimize the cache layer and place it close to application.
> > >
> > > Introduce stackable bcache for itself may work, but I don’t see real usage yet, and no motivation to maintain such usage still.
> > >
> > > Thanks.
> > >
> > > Coly Li
>