dm-cache caching network volume

dm-cache caching network volume

[Étienne BERSAC]

[-- Attachment #1.1: Type: text/plain, Size: 907 bytes --]

Hi,

I'm testing dm-cache for using a local SSD as a cache for a network
volume. My goal is to test dm-cache behaviour when network is down.

The device is shared through iSCSI. The first test is filling a huge
file with dd from /dev/zero. So, this is sequential access. While dd is
running, i remove the tun iface from the bridge on the host. iSCSI
properly detects the network failure.

I expected dm-cache to continue filling dirty blocks, even if the
network device is blocking. But it doesn't. The status shows that dirty
blocks count stays the same. dd is blocking.

I'm wondering if there is some cache desactivation due to
sequential_threshold. But increasing sequential_threshold did not help.
Where does dm-cache blocks ?

Is it possible to have dm-cache "bufferize" blocks for network
failures ?

Regards,
-- 
Étienne BERSAC <etienne.bersac@nonstop.fr>
NON STOP Systems

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache caching network volume

[Alex Elsayed]

Étienne BERSAC wrote:

> Hi,
> 
> I'm testing dm-cache for using a local SSD as a cache for a network
> volume. My goal is to test dm-cache behaviour when network is down.
> 
> The device is shared through iSCSI. The first test is filling a huge
> file with dd from /dev/zero. So, this is sequential access. While dd is
> running, i remove the tun iface from the bridge on the host. iSCSI
> properly detects the network failure.
> 
> I expected dm-cache to continue filling dirty blocks, even if the
> network device is blocking. But it doesn't. The status shows that dirty
> blocks count stays the same. dd is blocking.
> 
> I'm wondering if there is some cache desactivation due to
> sequential_threshold. But increasing sequential_threshold did not help.
> Where does dm-cache blocks ?
> 
> Is it possible to have dm-cache "bufferize" blocks for network
> failures ?
> 
> Regards,

Disclaimer: Not an expert, and not actually involved in writing bcache

Well, since you didn't mention changing it, I suspect you are operating in 
the default "writethrough" mode - this doesn't return to userspace until the 
data is on the backing (iSCSI in your case) device. For bcache over a 
network volume, this is the safe option, since the client machine dying 
won't lose data.

If you put bcache in writethrough mode, it may well do exactly as you 
describe, but only if iSCSI itself recovers from the connection loss (and 
possibly other caveats I'm not thinking of). If it doesn't, or if it does 
but doesn't manage to maintain the exact state it was in on disconnection, 
then you will lose data.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache caching network volume

[Alex Elsayed]

Alex Elsayed wrote:

> Étienne BERSAC wrote:
> 
>> Hi,
>> 
>> I'm testing dm-cache for using a local SSD as a cache for a network
>> volume. My goal is to test dm-cache behaviour when network is down.
>> 
>> The device is shared through iSCSI. The first test is filling a huge
>> file with dd from /dev/zero. So, this is sequential access. While dd is
>> running, i remove the tun iface from the bridge on the host. iSCSI
>> properly detects the network failure.
>> 
>> I expected dm-cache to continue filling dirty blocks, even if the
>> network device is blocking. But it doesn't. The status shows that dirty
>> blocks count stays the same. dd is blocking.
>> 
>> I'm wondering if there is some cache desactivation due to
>> sequential_threshold. But increasing sequential_threshold did not help.
>> Where does dm-cache blocks ?
>> 
>> Is it possible to have dm-cache "bufferize" blocks for network
>> failures ?
>> 
>> Regards,
> 
> Disclaimer: Not an expert, and not actually involved in writing bcache
> 
> Well, since you didn't mention changing it, I suspect you are operating in
> the default "writethrough" mode - this doesn't return to userspace until
> the data is on the backing (iSCSI in your case) device. For bcache over a
> network volume, this is the safe option, since the client machine dying
> won't lose data.
> 
> If you put bcache in writethrough mode, it may well do exactly as you
> describe, but only if iSCSI itself recovers from the connection loss (and
> possibly other caveats I'm not thinking of). If it doesn't, or if it does
> but doesn't manage to maintain the exact state it was in on disconnection,
> then you will lose data.

My apologies, I didn't realize until I ready my own reply on the way back 
that this was on the topic of dm-cache rather than bcache.

To properly respond, as I understand it dm-cache works by migrating chunks 
between the backing device and the faster cache. If your write would go to 
the backing device, it would block exactly as normal.

(I hope I didn't get this egregiously wrong...)


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache caching network volume

[Darrick J. Wong]

On Tue, Jun 18, 2013 at 07:07:03PM -0700, Alex Elsayed wrote:
> Alex Elsayed wrote:
> 
> > Étienne BERSAC wrote:
> > 
> >> Hi,
> >> 
> >> I'm testing dm-cache for using a local SSD as a cache for a network
> >> volume. My goal is to test dm-cache behaviour when network is down.
> >> 
> >> The device is shared through iSCSI. The first test is filling a huge
> >> file with dd from /dev/zero. So, this is sequential access. While dd is
> >> running, i remove the tun iface from the bridge on the host. iSCSI
> >> properly detects the network failure.
> >> 
> >> I expected dm-cache to continue filling dirty blocks, even if the
> >> network device is blocking. But it doesn't. The status shows that dirty
> >> blocks count stays the same. dd is blocking.
> >> 
> >> I'm wondering if there is some cache desactivation due to
> >> sequential_threshold. But increasing sequential_threshold did not help.
> >> Where does dm-cache blocks ?
> >> 
> >> Is it possible to have dm-cache "bufferize" blocks for network
> >> failures ?

I'm a user, not the author, but here's what I've experienced with dmcache:

Huge sequential writes will generally bypass the cache; what is your setting
for sequential_threshold?  (Your message hints that it might actually be high
enough.)

Second, even if it /is/ caching the blocks, I think the default mode is
writethrough mode, which means that the write request waits first for ssd
commit and then for origin device commit.

As for 'bufferizing' blocks ... in 3.9 (3.10?) writeback mode seems to write
blocks to the ssd while issuing an asynchronous write to the origin device.

(Maybe you're already using wb mode?)

> >> 
> >> Regards,
> > 
> > Disclaimer: Not an expert, and not actually involved in writing bcache
> > 
> > Well, since you didn't mention changing it, I suspect you are operating in
> > the default "writethrough" mode - this doesn't return to userspace until
> > the data is on the backing (iSCSI in your case) device. For bcache over a
> > network volume, this is the safe option, since the client machine dying
> > won't lose data.
> > 
> > If you put bcache in writethrough mode, it may well do exactly as you
> > describe, but only if iSCSI itself recovers from the connection loss (and
> > possibly other caveats I'm not thinking of). If it doesn't, or if it does
> > but doesn't manage to maintain the exact state it was in on disconnection,
> > then you will lose data.
> 
> My apologies, I didn't realize until I ready my own reply on the way back 
> that this was on the topic of dm-cache rather than bcache.
> 
> To properly respond, as I understand it dm-cache works by migrating chunks 
> between the backing device and the faster cache. If your write would go to 
> the backing device, it would block exactly as normal.

dmcache's default controller manages hot blocks...

--D
> 
> (I hope I didn't get this egregiously wrong...)
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: dm-cache caching network volume

[Joe Thornber]

On Tue, Jun 18, 2013 at 11:32:52AM +0200, Étienne BERSAC wrote:
> Is it possible to have dm-cache "bufferize" blocks for network
> failures ?

dm-cache is a slow moving cache, foor instance, it does not attempt to
pass all writes through the SSD.  Since you cannot rule out IO going
to the origin device it's not suitable for adding resilience to an
intermittent network connection.  Perhaps you should look at dm-raid?

- Joe

Re: dm-cache caching network volume

[Étienne BERSAC]

[-- Attachment #1.1: Type: text/plain, Size: 859 bytes --]

Hi,

Thanks for the reply :-)

I actually uses write-back. Here are the tables :

----8<----
cache_cacher: 0 1024 linear 254:32 2097152
cache_origin: 0 2048 linear 8:0 0
cache: 0 2048 cache 253:3 253:4 253:2 512 1 writeback default 4 sequential_threshold 2147483648 random_threshold 4
cache_metadata: 0 2097152 linear 254:32 0
---->8----

8:0 is /dev/sda, the iSCSI target.
254:32 is /dev/vdc, a virtio volume accessing a LV on a SSD.

The thing is : the cache status properlay state that new blocks in cache
are filled, they are even marked as dirty. Which suggest they are not in
the backing device. But actually, it blocks when iSCSI detects the
network failure.

Can dm-cache handle writes without querying the backing device until it
needs demotion ?

Regards,
-- 
Étienne BERSAC <etienne.bersac@nonstop.fr>
NON STOP Systems

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]