dm-cache mq policy warmup issue

dm-cache mq policy warmup issue

[Krishnasamy, Somasundaram]

[-- Attachment #1.1: Type: text/plain, Size: 6340 bytes --]

Mike,



I was doing some testing with latest dm-cache code (from 'for-next' branch). I am having difficulty in making the cache to warm-up. I used the below script to read range of blocks repeatedly to get them promoted. I assume after reading a block for mq->promote_threshold + 4 times, a block will be promoted. But it is not happening. I also tested the same with the sequential_threshold set to very high value.



Cache warming-up was relatively easier in the previous dm-cache code. Is this something you have seen in your testing?. Is there anything different should I do?



Note: /dev/mapper/mydev is the dm-cache target device. I ran the below script with "64k 100 100 0" as argument.

-----------

#!/bin/bash



if [[ $# -lt 4 ]]

then

        echo "Usage: $0 <cache_block_size> <nblocks to promote> <nread of a block> <start_block>"

        exit 1

fi



BSSIZE=$1

((TOTAL=$2+$4))

HITCNT=$3



# Read even numbered blocks

blkcnt=$4

while [[ blkcnt -lt $TOTAL ]]

do

        i=0

        echo "Reading block $blkcnt $HITCNT times"

        while [[ $i -lt $HITCNT ]]

        do

                dd if=/dev/mapper/mydev of=/dev/null bs=$BSSIZE skip=$blkcnt count=1 conv=notrunc iflag=direct 2> /dev/null

                if [[ $? -ne 0 ]]

               then

                        echo "I/O failure. exiting."

                        exit 1

                fi

                ((i=i+1))

        done



        ((blkcnt=blkcnt+2))



        PROMOTED=$(dmsetup status mydev | awk '{print $11}')

        echo "Promoted blocks so far: $PROMOTED"

done



# Read odd numbered blocks

((blkcnt=$4+1))

while [[ blkcnt -lt $TOTAL ]]

do

        i=0

        echo "Reading block $blkcnt $HITCNT times"

        while [[ $i -lt $HITCNT ]]

        do

                dd if=/dev/mapper/mydev of=/dev/null bs=$BSSIZE skip=$blkcnt count=1 conv=notrunc iflag=direct 2> /dev/null

                if [[ $? -ne 0 ]]

                then

                        echo "I/O failure. exiting."

                        exit 1

                fi

                ((i=i+1))

        done



        ((blkcnt=blkcnt+2))



        PROMOTED=$(dmsetup status mydev | awk '{print $11}')

        echo "Promoted blocks so far: $PROMOTED"

done

------------



Regards,

Somu.



-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Mike Snitzer
Sent: Thursday, October 24, 2013 1:30 PM
To: dm-devel@redhat.com
Cc: Mears, Morgan; Heinz Mauelshagen; Joe Thornber; Mike Snitzer
Subject: [dm-devel] [PATCH 00/24] dm cache: proposed changes for v3.13 merge



The following patches have been published to the 'for-next' branch of the device-mapper git repository:

git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git



The patches marked as RFC (19-24) could use the most review.  But all review would be appreciated.



Thanks,

Mike



Heinz Mauelshagen (11):

  dm cache policy mq: return NULL if mq->free list is empty in alloc_entry

  dm cache policy: variable hints support

  dm cache policy: have policy_writeback_work return -ENODATA by default

  dm cache: use is_write_io() in more places

  dm cache: use cell_defer() boolean argument consistently

  dm cache: log error message if dm_kcopyd_copy() fails

  dm cache: use a boolean when setting cache->quiescing

  dm cache: optimize commit_if_needed

  dm cache: add hints policy

  dm cache: add cache block invalidation API

  dm cache policy era: add cache block invalidation support



Joe Thornber (9):

  dm: nest targets used for testing under DM_TEST_TARGETS

  dm space map disk: optimise sm_disk_dec_block

  dm cache policy: remove return from void policy_remove_mapping

  dm cache policy mq: a few small fixes

  dm cache policy mq: implement writeback_work() and mq_{set,clear}_dirty()

  dm cache: be much more aggressive about promoting writes to discarded blocks

  dm cache metadata: return bool from __superblock_all_zeroes

  dm cache metadata: check the metadata version when reading the superblock

  dm cache: add passthrough mode



Mike Snitzer (1):

  dm table: print error on preresume failure



Morgan Mears (3):

  dm cache: support for stackable caching policies

  dm cache: add era policy shim

  dm cache: add trc policy shim



drivers/md/Kconfig                             |  68 ++-

drivers/md/Makefile                            |   9 +-

drivers/md/dm-cache-metadata.c                 | 189 +++++-

drivers/md/dm-cache-metadata.h                 |  28 +-

drivers/md/dm-cache-policy-cleaner.c           |  10 +-

drivers/md/dm-cache-policy-era.c               | 542 +++++++++++++++++

drivers/md/dm-cache-policy-hints.c             | 769 +++++++++++++++++++++++++

drivers/md/dm-cache-policy-internal.h          |  31 +-

drivers/md/dm-cache-policy-mq.c                | 343 ++++++++---

drivers/md/dm-cache-policy-trc.c               | 263 +++++++++

drivers/md/dm-cache-policy.c                   |  66 ++-

drivers/md/dm-cache-policy.h                   |  77 ++-

drivers/md/dm-cache-shim-utils.c               | 217 +++++++

drivers/md/dm-cache-shim-utils.h               |  73 +++

drivers/md/dm-cache-stack-utils.c              | 239 ++++++++

drivers/md/dm-cache-stack-utils.h              |  34 ++

drivers/md/dm-cache-target.c                   | 417 ++++++++++++--

drivers/md/dm-table.c                          |   5 +-

drivers/md/persistent-data/dm-space-map-disk.c |  18 +-

19 files changed, 3167 insertions(+), 231 deletions(-)  create mode 100644 drivers/md/dm-cache-policy-era.c  create mode 100644 drivers/md/dm-cache-policy-hints.c

create mode 100644 drivers/md/dm-cache-policy-trc.c  create mode 100644 drivers/md/dm-cache-shim-utils.c  create mode 100644 drivers/md/dm-cache-shim-utils.h  create mode 100644 drivers/md/dm-cache-stack-utils.c  create mode 100644 drivers/md/dm-cache-stack-utils.h



--

1.8.1.4



--

dm-devel mailing list

dm-devel@redhat.com<mailto:dm-devel@redhat.com>

https://www.redhat.com/mailman/listinfo/dm-devel

[-- Attachment #1.2: Type: text/html, Size: 21189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache mq policy warmup issue

[Mears, Morgan]

On Mon, Nov 04, 2013 at 12:22:26PM -0500, Krishnasamy, Somasundaram wrote:
> Mike,
> 
> I was doing some testing with latest dm-cache code (from 'for-next' branch). I
> am having difficulty in making the cache to warm-up. I used the below script
> to read range of blocks repeatedly to get them promoted. I assume after
> reading a block for mq->promote_threshold + 4 times, a block will be
> promoted. But it is not happening. I also tested the same with the
> sequential_threshold set to very high value.
> 
> Cache warming-up was relatively easier in the previous dm-cache code. Is this
> something you have seen in your testing?. Is there anything different should I
> do?

I'm also testing with the for-next branch of linux-dm.git and seeing the
same thing as Somu.  It's easy to reproduce; reading the same block on the
cached device over and over with dd does not result in that block being
promoted.

I did notice that the "read miss" counter for the cached device increments
each time I execute dd, which I believe means this isn't a buffer cache
issue (I also used dd iflag=direct).  The migration_threshold value
shouldn't be an issue for this test, but I set it to a very high value
anyway; didn't make any difference.

Are there any tests in device-mapper-test-suite that you would recommend
for testing whether promotions are occurring as expected?

--Morgan

> Note: /dev/mapper/mydev is the dm-cache target device. I ran the below
> script with "64k 100 100 0" as argument.
> -----------
> #!/bin/bash
> 
> if [[ $# -lt 4 ]]
> then
>         echo "Usage: $0 <cache_block_size> <nblocks to promote> <nread of a
> block> <start_block>"
>         exit 1
> fi
> 
> BSSIZE=$1
> ((TOTAL=$2+$4))
> HITCNT=$3
> 
> # Read even numbered blocks
> blkcnt=$4
> while [[ blkcnt -lt $TOTAL ]]
> do
>         i=0
>         echo "Reading block $blkcnt $HITCNT times"
>         while [[ $i -lt $HITCNT ]]
>         do
>                 dd if=/dev/mapper/mydev of=/dev/null bs=$BSSIZE skip=$blkcnt
> count=1 conv=notrunc iflag=direct 2> /dev/null
>                 if [[ $? -ne 0 ]]
>                then
>                         echo "I/O failure. exiting."
>                         exit 1
>                 fi
>                 ((i=i+1))
>         done
> 
>         ((blkcnt=blkcnt+2))
> 
>         PROMOTED=$(dmsetup status mydev | awk '{print $11}')
>         echo "Promoted blocks so far: $PROMOTED"
> done
> 
> # Read odd numbered blocks
> ((blkcnt=$4+1))
> while [[ blkcnt -lt $TOTAL ]]
> do
>         i=0
>         echo "Reading block $blkcnt $HITCNT times"
>         while [[ $i -lt $HITCNT ]]
>         do
>                 dd if=/dev/mapper/mydev of=/dev/null bs=$BSSIZE skip=$blkcnt
> count=1 conv=notrunc iflag=direct 2> /dev/null
>                 if [[ $? -ne 0 ]]
>                 then
>                         echo "I/O failure. exiting."
>                         exit 1
>                 fi
>                 ((i=i+1))
>         done
> 
>         ((blkcnt=blkcnt+2))
> 
>         PROMOTED=$(dmsetup status mydev | awk '{print $11}')
>         echo "Promoted blocks so far: $PROMOTED"
> done
> ------------
> 
> Regards,
> Somu.

Re: dm-cache mq policy warmup issue

[Joe Thornber]

On Wed, Nov 13, 2013 at 04:13:43PM +0000, Mears, Morgan wrote:
> On Mon, Nov 04, 2013 at 12:22:26PM -0500, Krishnasamy, Somasundaram wrote:
> > Mike,
> > 
> > I was doing some testing with latest dm-cache code (from 'for-next' branch). I
> > am having difficulty in making the cache to warm-up. I used the below script
> > to read range of blocks repeatedly to get them promoted. I assume after
> > reading a block for mq->promote_threshold + 4 times, a block will be
> > promoted. But it is not happening. I also tested the same with the
> > sequential_threshold set to very high value.
> > 
> > Cache warming-up was relatively easier in the previous dm-cache code. Is this
> > something you have seen in your testing?. Is there anything different should I
> > do?
> 
> I'm also testing with the for-next branch of linux-dm.git and seeing the
> same thing as Somu.  It's easy to reproduce; reading the same block on the
> cached device over and over with dd does not result in that block being
> promoted.
> 
> I did notice that the "read miss" counter for the cached device increments
> each time I execute dd, which I believe means this isn't a buffer cache
> issue (I also used dd iflag=direct).  The migration_threshold value
> shouldn't be an issue for this test, but I set it to a very high value
> anyway; didn't make any difference.
> 
> Are there any tests in device-mapper-test-suite that you would recommend
> for testing whether promotions are occurring as expected?
> 
> --Morgan

Hi Morgan,

I just added these tests, promotion is occuring, just not as much as I'd like:

https://github.com/jthornber/device-mapper-test-suite/blob/master/lib/dmtest/tests/cache/promotion_tests.rb

Ah, these tests all trigger promotion via a write, I'll do some that just read.

- Joe

Re: dm-cache mq policy warmup issue

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 2348 bytes --]

Okay, a little correction here. The problematic setup was with 2.3TB cache,
not 3TB. The mistake comes from the fact that at the time of the writing I
was dealing with another server which has 8x480GB ssd's in raid5, which is
little over 3TB. Obviously the number was floating in my head.





On Thu, Nov 14, 2013 at 4:37 PM, Joe Thornber <thornber@redhat.com> wrote:

> On Wed, Nov 13, 2013 at 04:13:43PM +0000, Mears, Morgan wrote:
> > On Mon, Nov 04, 2013 at 12:22:26PM -0500, Krishnasamy, Somasundaram
> wrote:
> > > Mike,
> > >
> > > I was doing some testing with latest dm-cache code (from 'for-next'
> branch). I
> > > am having difficulty in making the cache to warm-up. I used the below
> script
> > > to read range of blocks repeatedly to get them promoted. I assume after
> > > reading a block for mq->promote_threshold + 4 times, a block will be
> > > promoted. But it is not happening. I also tested the same with the
> > > sequential_threshold set to very high value.
> > >
> > > Cache warming-up was relatively easier in the previous dm-cache code.
> Is this
> > > something you have seen in your testing?. Is there anything different
> should I
> > > do?
> >
> > I'm also testing with the for-next branch of linux-dm.git and seeing the
> > same thing as Somu.  It's easy to reproduce; reading the same block on
> the
> > cached device over and over with dd does not result in that block being
> > promoted.
> >
> > I did notice that the "read miss" counter for the cached device
> increments
> > each time I execute dd, which I believe means this isn't a buffer cache
> > issue (I also used dd iflag=direct).  The migration_threshold value
> > shouldn't be an issue for this test, but I set it to a very high value
> > anyway; didn't make any difference.
> >
> > Are there any tests in device-mapper-test-suite that you would recommend
> > for testing whether promotions are occurring as expected?
> >
> > --Morgan
>
> Hi Morgan,
>
> I just added these tests, promotion is occuring, just not as much as I'd
> like:
>
>
> https://github.com/jthornber/device-mapper-test-suite/blob/master/lib/dmtest/tests/cache/promotion_tests.rb
>
> Ah, these tests all trigger promotion via a write, I'll do some that just
> read.
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 3344 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache mq policy warmup issue

[Mike Snitzer]

On Wed, Nov 13 2013 at 11:13am -0500,
Mears, Morgan <Morgan.Mears@netapp.com> wrote:

> On Mon, Nov 04, 2013 at 12:22:26PM -0500, Krishnasamy, Somasundaram wrote:
> > Mike,
> > 
> > I was doing some testing with latest dm-cache code (from 'for-next' branch). I
> > am having difficulty in making the cache to warm-up. I used the below script
> > to read range of blocks repeatedly to get them promoted. I assume after
> > reading a block for mq->promote_threshold + 4 times, a block will be
> > promoted. But it is not happening. I also tested the same with the
> > sequential_threshold set to very high value.
> > 
> > Cache warming-up was relatively easier in the previous dm-cache code. Is this
> > something you have seen in your testing?. Is there anything different should I
> > do?
> 
> I'm also testing with the for-next branch of linux-dm.git and seeing the
> same thing as Somu.  It's easy to reproduce; reading the same block on the
> cached device over and over with dd does not result in that block being
> promoted.
> 
> I did notice that the "read miss" counter for the cached device increments
> each time I execute dd, which I believe means this isn't a buffer cache
> issue (I also used dd iflag=direct).  The migration_threshold value
> shouldn't be an issue for this test, but I set it to a very high value
> anyway; didn't make any difference.

The fix for this is now staged in linux-next, see:
http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=38863832030359bfa8153bee6d7045e372780deb

> Are there any tests in device-mapper-test-suite that you would recommend
> for testing whether promotions are occurring as expected?

Joe included more tests in device-mapper-test-suite, try:

dmtest run --suite cache -t /SingleBlockPromotionTests/

(the few tests that fail are due to extra IO that occurs due readahead
because buffered IO is used to read an entire block, the test needs
improving to use direct IO)