All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sage Weil <sage@newdream.net>
To: Andrey Korolyov <andrey@xdel.ru>
Cc: ceph-devel@vger.kernel.org
Subject: Re: Mysteriously poor write performance
Date: Sun, 18 Mar 2012 11:22:51 -0700 (PDT)	[thread overview]
Message-ID: <Pine.LNX.4.64.1203181121460.580@cobra.newdream.net> (raw)
In-Reply-To: <CABYiri-C1ftxsM3dWqtcRzXBYoddRbHug=TA6j+0EY-cuMt3Mw@mail.gmail.com>

On Sat, 17 Mar 2012, Andrey Korolyov wrote:
> Hi,
> 
> I`ve did some performance tests at the following configuration:
> 
> mon0, osd0 and mon1, osd1 - two twelve-core r410 with 32G ram, mon2 -
> dom0 with three dedicated cores and 1.5G, mostly idle. First three
> disks on each r410 arranged into raid0 and holds osd data when fourth
> holds os and osd` journal partition, all ceph-related stuff mounted on
> the ext4 without barriers.
> 
> Firstly, I`ve noticed about a difference of benchmark performance and
> write speed through rbd from small kvm instance running on one of
> first two machines - when bench gave me about 110Mb/s, writing zeros
> to raw block device inside vm with dd was at top speed about 45 mb/s,
> for vm`fs (ext4 with default options) performance drops to ~23Mb/s.
> Things get worse, when I`ve started second vm at second host and tried
> to continue same dd tests simultaneously - performance fairly divided
> by half for each instance :). Enabling jumbo frames, playing with cpu
> affinity for ceph and vm instances and trying different TCP congestion
> protocols gave no effect at all - with DCTCP I have slightly smoother
> network load graph and that`s all.
> 
> Can ml please suggest anything to try to improve performance?

Can you try setting

	rbd writeback window = 8192000

or similar, and see what kind of effect that has?  I suspect it'll speed 
up dd; I'm less sure about ext3.

Thanks!
sage


> 
> ceph-0.43, libvirt-0.9.8, qemu-1.0.0, kernel 3.2
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

  reply	other threads:[~2012-03-18 18:22 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-03-17 11:35 Mysteriously poor write performance Andrey Korolyov
2012-03-18 18:22 ` Sage Weil [this message]
2012-03-19 13:46   ` Andrey Korolyov
2012-03-19 16:59     ` Greg Farnum
2012-03-19 18:13       ` Andrey Korolyov
2012-03-19 18:25         ` Greg Farnum
2012-03-19 18:40         ` Josh Durgin
2012-03-19 19:30           ` Andrey Korolyov
2012-03-20 20:37           ` Andrey Korolyov
2012-03-20 22:36             ` Samuel Just
     [not found]               ` <CABYiri9An0sYP6pP1xU_Xjz7yhXdv1eF-4q-DqtqygYH76rMHw@mail.gmail.com>
     [not found]                 ` <CACLRD_0BK0WVX6=-n3doF680efibNVjq_qB_DwxF73MXNWQ_LA@mail.gmail.com>
     [not found]                   ` <CABYiri--y7zyS_+GvNqCayQd6PXyU8R0GEiBC0BD885DK6w7Rw@mail.gmail.com>
     [not found]                     ` <CACLRD_1Ndcoz3+aOzn4O-7DLL-ePV8ujeGyaN9AeHXTxsO2AeA@mail.gmail.com>
     [not found]                       ` <CABYiri-J-+SsqXOq=ircP-v80rfiXjoFX20wLa0Pfc7KvyA0SQ@mail.gmail.com>
     [not found]                         ` <CACLRD_1PWntwwyWTWij1OuC+LaSU=jz5gittQ5vwqGSXQnyzeQ@mail.gmail.com>
     [not found]                           ` <CABYiri_bjt9Equj4JcoVsN5AEMxxX_qaNNBYZzCfi2rirsTCkA@mail.gmail.com>
     [not found]                             ` <CABYiri8TfJXC7j3L5QXXFO2nmtwiyoP=YSGCZse0VrsY+_zbLw@mail.gmail.com>
2012-03-21 21:20                               ` Samuel Just
     [not found]                                 ` <CABYiri8qHCv6=dFUc-8tFo9bxEtUTQhV1cE9K=CK8hbhW3u10A@mail.gmail.com>
2012-03-22 17:26                                   ` Samuel Just
2012-03-22 18:40                                     ` Andrey Korolyov
     [not found]                                       ` <CABYiri9SYaTFgb7GMPi_VPT1vDWV+O=Q_P-xibsBb-xjRU1E=g@mail.gmail.com>
2012-03-23 17:53                                         ` Samuel Just
2012-03-24 19:09                                           ` Andrey Korolyov
2012-03-27 16:39                                             ` Samuel Just

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Pine.LNX.4.64.1203181121460.580@cobra.newdream.net \
    --to=sage@newdream.net \
    --cc=andrey@xdel.ru \
    --cc=ceph-devel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.