From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dante Cinco Subject: Re: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough Date: Thu, 18 Nov 2010 17:38:24 -0800 Message-ID: References: <563ba65b-51ca-43d5-99a0-353988dad721@default> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <563ba65b-51ca-43d5-99a0-353988dad721@default> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dan Magenheimer Cc: Jeremy Fitzhardinge , Xen-devel , mathieu.desnoyers@polymtl.ca, Andrew Thomas , Konrad Wilk , "Lin, Ray" , keir.fraser@eu.citrix.com, Chris Mason List-Id: xen-devel@lists.xenproject.org On Thu, Nov 18, 2010 at 4:20 PM, Dan Magenheimer wrote: >> We did suspect it, since our old setting was HZ=3D1000 and we assigned >> more than 10 VCPUs to domU. But we don't see the performance difference >> with HZ=3D100. > > FWIW, it didn't appear that the problems were proportional to HZ. > Seemed more that somehow the pvclock became incorrect and spent > a lot of time rereading the pvclock value. We decided to enable lock stat in the kernel to track down all those lock activities in the profile report. The first thing I noticed was kmemleak was at the top of the list (/proc/lock_stat) so we disabled kmemleak. This boosted our I/O performance to 119k IOPS (from 31k). One of our developers (Bruce Edge) suggested killing ntpd so I did. This resulted in another significant bump in I/O performance to 209k IOPS. The question now is why ntpd? Is it the source of all or most of those pvclock_clocksource_read in the profile report? > >> -----Original Message----- >> From: Lin, Ray [mailto:Ray.Lin@lsi.com] >> Sent: Thursday, November 18, 2010 2:40 PM >> To: Dan Magenheimer; Dante Cinco; Konrad Wilk >> Cc: Jeremy Fitzhardinge; Xen-devel; mathieu.desnoyers@polymtl.ca; >> Andrew Thomas; keir.fraser@eu.citrix.com; Chris Mason >> Subject: RE: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.2 >> pvops domU kernel with PCI passthrough >> >> >> >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >> bounces@lists.xensource.com] On Behalf Of Dan Magenheimer >> Sent: Thursday, November 18, 2010 1:21 PM >> To: Dante Cinco; Konrad Wilk >> Cc: Jeremy Fitzhardinge; Xen-devel; mathieu.desnoyers@polymtl.ca; >> Andrew Thomas; keir.fraser@eu.citrix.com; Chris Mason >> Subject: RE: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.2 >> pvops domU kernel with PCI passthrough >> >> In case it is related: >> http://lists.xensource.com/archives/html/xen-devel/2010- >> 07/msg01247.html >> >> Although I never went further on this investigation, it appeared to me >> that pvclock_clocksource_read was getting called at least an order-of- >> magnitude more frequently than expected in some circumstances for some >> kernels. =A0And IIRC it was scaled by the number of vcpus. >> >> We did suspect it, since our old setting was HZ=3D1000 and we assigned >> more than 10 VCPUs to domU. But we don't see the performance difference >> with HZ=3D100. >> >> > -----Original Message----- >> > From: Dante Cinco [mailto:dantecinco@gmail.com] >> > Sent: Thursday, November 18, 2010 12:36 PM >> > To: Konrad Rzeszutek Wilk >> > Cc: Jeremy Fitzhardinge; Xen-devel; mathieu.desnoyers@polymtl.ca; >> > Andrew Thomas; keir.fraser@eu.citrix.com; Chris Mason >> > Subject: Re: [Xen-devel] swiotlb=3Dforce in Konrad's xen-pcifront-0.8.= 2 >> > pvops domU kernel with PCI passthrough >> > >> > I mentioned earlier in an previous post to this thread that I'm able >> > to apply Dulloor's xenoprofile patch to the dom0 kernel but not the >> > domU kernel. So I can't do active-domain profiling but I'm able to do >> > passive-domain profiling but I don't know how reliable the results >> are >> > since it shows pvclock_clocksource_read as the top consumer of CPU >> > cycles at 28%. >> > >> > CPU: Intel Architectural Perfmon, speed 2665.98 MHz (estimated) >> > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a >> > unit mask of 0x00 (No unit mask) count 100000 >> > samples =A0% =A0 =A0 =A0 =A0image name =A0 =A0 =A0 =A0 =A0 =A0 =A0 app= name >> > symbol name >> > 918089 =A0 27.9310 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 pvclock_clocksource_read >> > 217811 =A0 =A06.6265 =A0domain1-modules =A0 =A0 =A0 =A0 =A0domain1-mod= ules >> > /domain1-modules >> > 188327 =A0 =A05.7295 =A0vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco= - >> debug >> > vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug >> > mutex_spin_on_owner >> > 186684 =A0 =A05.6795 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 __xen_spin_lock >> > 149514 =A0 =A04.5487 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 __write_lock_failed >> > 123278 =A0 =A03.7505 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 __kernel_text_address >> > 122906 =A0 =A03.7392 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 xen_spin_unlock >> > 90903 =A0 =A0 2.7655 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 __spin_time_accum >> > 85880 =A0 =A0 2.6127 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 __module_address >> > 75223 =A0 =A0 2.2885 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 print_context_stack >> > 66778 =A0 =A0 2.0316 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 __module_text_address >> > 57389 =A0 =A0 1.7459 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 is_module_text_address >> > 47282 =A0 =A0 1.4385 =A0xen-syms-4.1-unstable =A0 =A0domain1-xen >> > syscall_enter >> > 47219 =A0 =A0 1.4365 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 prio_tree_insert >> > 46495 =A0 =A0 1.4145 =A0vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco= - >> debug >> > vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug >> > pvclock_clocksource_read >> > 44501 =A0 =A0 1.3539 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 prio_tree_left >> > 32482 =A0 =A0 0.9882 >> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug >> > domain1-kernel =A0 =A0 =A0 =A0 =A0 native_read_tsc >> > >> > I ran oprofile (0.9.5 with xenoprofile patch) for 20 seconds while >> the >> > I/Os were running. Here's the command I used: >> > >> > opcontrol --start --xen=3D/boot/xen-syms-4.1-unstable >> > --vmlinux=3D/boot/vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug >> > --passive-domains=3D1 >> > --passive-images=3D/boot/vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu- >> > 5.11.dcinco-debug >> > >> > I had to remove dom0_max_vcpus=3D1 (but kept dom0_vcpus_pin=3Dtrue) in >> the >> > Xen command line. Otherwise, oprofile only gives the samples from >> > CPU0. >> > >> > I'm going to try perf next. >> > >> > - Dante >> > >> > _______________________________________________ >> > Xen-devel mailing list >> > Xen-devel@lists.xensource.com >> > http://lists.xensource.com/xen-devel >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >