From mboxrd@z Thu Jan 1 00:00:00 1970 From: Scott Wood Subject: Re: [PATCH] KVM: PPC: Convert openpic lock to raw_spinlock Date: Fri, 12 Sep 2014 12:50:17 -0500 Message-ID: <1410544217.24184.397.camel@snotra.buserror.net> References: <1410463517-9824-1-git-send-email-bogdan.purcareata@freescale.com> <1410459569.24184.361.camel@snotra.buserror.net> <337352d340114a34a32f71445b496a74@BY2PR03MB189.namprd03.prod.outlook.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: "kvm-ppc@vger.kernel.org" , "kvm@vger.kernel.org" , Caraman Mihai Claudiu-B02008 , Tudor Laurentiu-B10716 To: Purcareata Bogdan-B43198 Return-path: In-Reply-To: <337352d340114a34a32f71445b496a74@BY2PR03MB189.namprd03.prod.outlook.com> Sender: kvm-ppc-owner@vger.kernel.org List-Id: kvm.vger.kernel.org On Fri, 2014-09-12 at 09:12 -0500, Purcareata Bogdan-B43198 wrote: > > -----Original Message----- > > From: Wood Scott-B07421 > > Sent: Thursday, September 11, 2014 9:19 PM > > To: Purcareata Bogdan-B43198 > > Cc: kvm-ppc@vger.kernel.org; kvm@vger.kernel.org > > Subject: Re: [PATCH] KVM: PPC: Convert openpic lock to raw_spinlock > > > > On Thu, 2014-09-11 at 15:25 -0400, Bogdan Purcareata wrote: > > > This patch enables running intensive I/O workloads, e.g. netperf, in a guest > > > deployed on a RT host. No change for !RT kernels. > > > > > > The openpic spinlock becomes a sleeping mutex on a RT system. This no longer > > > guarantees that EPR is atomic with exception delivery. The guest VCPU thread > > > fails due to a BUG_ON(preemptible()) when running netperf. > > > > > > In order to make the kvmppc_mpic_set_epr() call safe on RT from non-atomic > > > context, convert the openpic lock to a raw_spinlock. A similar approach can > > > be seen for x86 platforms in the following commit [1]. > > > > > > Here are some comparative cyclitest measurements run inside a high priority > > RT > > > guest run on a RT host. The guest has 1 VCPU and the test has been run for > > 15 > > > minutes. The guest runs ~750 hackbench processes as background stress. > > > > Does hackbench involve triggering interrupts that would go through the > > MPIC? You may want to try an I/O-heavy benchmark to stress the MPIC > > code (the more interrupt sources are active at once, the "better"). > > Before this patch, running netperf/iperf in the guest always resulted > in hitting the afore-mentioned BUG_ON, when the host was RT. This is > why I can't provide comparative cyclitest measurements before and after > the patch, with heavy I/O stress. Since I had no problem running > hackbench before, I'm assuming it doesn't involve interrupts passing > through the MPIC. The measurements were posted just to show that the > patch doesn't mess up anything somewhere else. I know you can't provide before/after, but it would be nice to see what the after numbers are with heavy MPIC activity. > > Also try a guest with many vcpus. > > AFAIK, without the MSI affinity patches [1], all vfio interrupts will > go to core 0 in the guest. In this case, I guess there won't be > contention induced latencies due to multiple VCPUs expecting to have > their interrupts delivered. Am I getting it wrong? It's not about contention, but about loops in the MPIC code that iterate over the entire set of vcpus. -Scott From mboxrd@z Thu Jan 1 00:00:00 1970 From: Scott Wood Date: Fri, 12 Sep 2014 17:50:17 +0000 Subject: Re: [PATCH] KVM: PPC: Convert openpic lock to raw_spinlock Message-Id: <1410544217.24184.397.camel@snotra.buserror.net> List-Id: References: <1410463517-9824-1-git-send-email-bogdan.purcareata@freescale.com> <1410459569.24184.361.camel@snotra.buserror.net> <337352d340114a34a32f71445b496a74@BY2PR03MB189.namprd03.prod.outlook.com> In-Reply-To: <337352d340114a34a32f71445b496a74@BY2PR03MB189.namprd03.prod.outlook.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Purcareata Bogdan-B43198 Cc: "kvm-ppc@vger.kernel.org" , "kvm@vger.kernel.org" , Caraman Mihai Claudiu-B02008 , Tudor Laurentiu-B10716 On Fri, 2014-09-12 at 09:12 -0500, Purcareata Bogdan-B43198 wrote: > > -----Original Message----- > > From: Wood Scott-B07421 > > Sent: Thursday, September 11, 2014 9:19 PM > > To: Purcareata Bogdan-B43198 > > Cc: kvm-ppc@vger.kernel.org; kvm@vger.kernel.org > > Subject: Re: [PATCH] KVM: PPC: Convert openpic lock to raw_spinlock > > > > On Thu, 2014-09-11 at 15:25 -0400, Bogdan Purcareata wrote: > > > This patch enables running intensive I/O workloads, e.g. netperf, in a guest > > > deployed on a RT host. No change for !RT kernels. > > > > > > The openpic spinlock becomes a sleeping mutex on a RT system. This no longer > > > guarantees that EPR is atomic with exception delivery. The guest VCPU thread > > > fails due to a BUG_ON(preemptible()) when running netperf. > > > > > > In order to make the kvmppc_mpic_set_epr() call safe on RT from non-atomic > > > context, convert the openpic lock to a raw_spinlock. A similar approach can > > > be seen for x86 platforms in the following commit [1]. > > > > > > Here are some comparative cyclitest measurements run inside a high priority > > RT > > > guest run on a RT host. The guest has 1 VCPU and the test has been run for > > 15 > > > minutes. The guest runs ~750 hackbench processes as background stress. > > > > Does hackbench involve triggering interrupts that would go through the > > MPIC? You may want to try an I/O-heavy benchmark to stress the MPIC > > code (the more interrupt sources are active at once, the "better"). > > Before this patch, running netperf/iperf in the guest always resulted > in hitting the afore-mentioned BUG_ON, when the host was RT. This is > why I can't provide comparative cyclitest measurements before and after > the patch, with heavy I/O stress. Since I had no problem running > hackbench before, I'm assuming it doesn't involve interrupts passing > through the MPIC. The measurements were posted just to show that the > patch doesn't mess up anything somewhere else. I know you can't provide before/after, but it would be nice to see what the after numbers are with heavy MPIC activity. > > Also try a guest with many vcpus. > > AFAIK, without the MSI affinity patches [1], all vfio interrupts will > go to core 0 in the guest. In this case, I guess there won't be > contention induced latencies due to multiple VCPUs expecting to have > their interrupts delivered. Am I getting it wrong? It's not about contention, but about loops in the MPIC code that iterate over the entire set of vcpus. -Scott