Re: OSD Hardware questions

From: Alexandre DERUMIER <aderumier@odiso.com>
To: Sage Weil <sage@inktank.com>
Cc: Mark Nelson <mark.nelson@inktank.com>,
	Gregory Farnum <greg@inktank.com>,
	ceph-devel@vger.kernel.org,
	Stefan Priebe - Profihost AG <s.priebe@profihost.ag>
Subject: Re: OSD Hardware questions
Date: Thu, 28 Jun 2012 17:45:15 +0200 (CEST)	[thread overview]
Message-ID: <3069bbf7-ebcf-467f-978f-c8026befe064@mailpro> (raw)
In-Reply-To: <Pine.LNX.4.64.1206280831590.32016@cobra.newdream.net>

>>Have you tried adjusting 'osd op threads'? The default is 2, but bumping 
>>that to, say, 8, might give you better concurrency and throughput. 
Not yet,I'll try that tomorrow

>>Definitely. Seeing perf/oprofile/whatever results for the osd under that 
>>workload would be very interesting! We need to get perf going in our 
>>testing environment... 

I'm not an expert, but if you give me command line, I'll do it ;)

BTW : wip-flushmin improve a lot the performance with random write. I jump from 1500-2000 iop/s (with spikes/slowdown),to constant 5500 iop/s.
      With btrfs, I see random write flushed each x seconds sequentially. (so it's really help with seeks)
      But with xfs,I see constant random writes...(maybe it's an xfs bug...)

----- Mail original ----- 

De: "Sage Weil" <sage@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: "Mark Nelson" <mark.nelson@inktank.com>, "Gregory Farnum" <greg@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
Envoyé: Jeudi 28 Juin 2012 17:33:31 
Objet: Re: OSD Hardware questions 

On Thu, 28 Jun 2012, Alexandre DERUMIER wrote: 
> Hi, 
> maybe it can help, I'm doing same tests that stefan, 
> random write with 4K with 3 nodes with 5 osds (with 15K drives) 
> fio --filename=[disk] --direct=1 --rw=randwrite --bs=4k --size=100M --numjobs=50 --runtime=30 --group_reporting --name=file1 
> 
> I can achieve 5500io/s (disks are not at 100%) 
> 
> cpu is around 30%idle ( 8 cores E5420 @ 2.50GHz) 

Have you tried adjusting 'osd op threads'? The default is 2, but bumping 
that to, say, 8, might give you better concurrency and throughput. 

"Mark Nelson" <mark.nelson@inktank.com> wrote: 
> It would be interesting to see where all your CPU time is being spent. 
> What benchmark are you using to do the random writes? 

Definitely. Seeing perf/oprofile/whatever results for the osd under that 
workload would be very interesting! We need to get perf going in our 
testing environment... 

sage 

-- 

-- 

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 

Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 

45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html