[4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Jaggi, Manish]

I am trying to run a kvm guest on a host with 4.4 rt3 patchset applied. (Cavium thunderX arm64 system)
Getting the following error: 

BUG: scheduling while atomic: qemu-system-aar/41889/0x00000002
[  341.647463] Modules linked in: ipv6 thunderx_edac_lmc thunderx_edac_ccpi i2c_octeon edac_core shpchp aes_ce_blk ablk_helper cryptd aes_ce_cipher ghash_ce sha2_ce sha1_ce uio_pdrv_genirq rtc_efi uio
[  341.647477] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
[  341.647478] 
[  341.647484] CPU: 2 PID: 41889 Comm: qemu-system-aar Not tainted 4.4.0-rt3-00120-gbb974fa #64
[  341.647486] Hardware name: www.cavium.com ThunderX CRB1S/ThunderX CRB1S, BIOS 0.3 Dec  3 2015
[  341.647488] Call trace:
[  341.647494] [<ffff800000097878>] dump_backtrace+0x0/0x160
[  341.647499] [<ffff8000000979fc>] show_stack+0x24/0x30
[  341.647503] [<ffff800000512608>] dump_stack+0x88/0xa8
[  341.647509] [<ffff8000000f25c0>] __schedule_bug+0x70/0xc0
[  341.647514] [<ffff8000008f8f38>] __schedule+0x510/0x580
[  341.647517] [<ffff8000008f90e8>] schedule+0x50/0xf0
[  341.647521] [<ffff8000008fa9a4>] rt_spin_lock_slowlock+0x124/0x2e0
[  341.647525] [<ffff8000008fc5e0>] rt_spin_lock+0x60/0x70
[  341.647530] [<ffff8000000bffe0>] kvm_vgic_flush_hwstate+0x60/0x278
[  341.647535] [<ffff8000000b3140>] kvm_arch_vcpu_ioctl_run+0x108/0x618
[  341.647547] [<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
[  341.647553] [<ffff80000024b4dc>] do_vfs_ioctl+0x364/0x628
[  341.647556] [<ffff80000024b834>] SyS_ioctl+0x94/0xa8
[  341.647560] [<ffff800000093b04>] el0_svc_naked+0x38/0x3c


The below patch enables preemption:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/arch/arm/kvm/arm.c?h=v4.4&id=1b3d546daf85ed2bc9966e12cee3e6435fb65eca

arm/arm64: KVM: Properly account for guest CPU time

Is there a way to do it without disabling preemption ? 
Or any other way to workaround

  
 Regards, 
Manish Jaggi    

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Josh Cartwright]

[-- Attachment #1: Type: text/plain, Size: 4537 bytes --]

On Wed, Feb 10, 2016 at 09:33:28AM +0000, Jaggi, Manish wrote:
> I am trying to run a kvm guest on a host with 4.4 rt3 patchset
> applied. (Cavium thunderX arm64 system) Getting the following error: 
> 
> BUG: scheduling while atomic: qemu-system-aar/41889/0x00000002
> [  341.647463] Modules linked in: ipv6 thunderx_edac_lmc thunderx_edac_ccpi i2c_octeon edac_core shpchp aes_ce_blk ablk_helper cryptd aes_ce_cipher ghash_ce sha2_ce sha1_ce uio_pdrv_genirq rtc_efi uio
> [  341.647477] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
> [  341.647478] 
> [  341.647484] CPU: 2 PID: 41889 Comm: qemu-system-aar Not tainted 4.4.0-rt3-00120-gbb974fa #64
> [  341.647486] Hardware name: www.cavium.com ThunderX CRB1S/ThunderX CRB1S, BIOS 0.3 Dec  3 2015
> [  341.647488] Call trace:
> [  341.647494] [<ffff800000097878>] dump_backtrace+0x0/0x160
> [  341.647499] [<ffff8000000979fc>] show_stack+0x24/0x30
> [  341.647503] [<ffff800000512608>] dump_stack+0x88/0xa8
> [  341.647509] [<ffff8000000f25c0>] __schedule_bug+0x70/0xc0
> [  341.647514] [<ffff8000008f8f38>] __schedule+0x510/0x580
> [  341.647517] [<ffff8000008f90e8>] schedule+0x50/0xf0
> [  341.647521] [<ffff8000008fa9a4>] rt_spin_lock_slowlock+0x124/0x2e0
> [  341.647525] [<ffff8000008fc5e0>] rt_spin_lock+0x60/0x70
> [  341.647530] [<ffff8000000bffe0>] kvm_vgic_flush_hwstate+0x60/0x278
> [  341.647535] [<ffff8000000b3140>] kvm_arch_vcpu_ioctl_run+0x108/0x618
> [  341.647547] [<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
> [  341.647553] [<ffff80000024b4dc>] do_vfs_ioctl+0x364/0x628
> [  341.647556] [<ffff80000024b834>] SyS_ioctl+0x94/0xa8
> [  341.647560] [<ffff800000093b04>] el0_svc_naked+0x38/0x3c
> 
> The below patch enables preemption:
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/arch/arm/kvm/arm.c?h=v4.4&id=1b3d546daf85ed2bc9966e12cee3e6435fb65eca

Another relevant commit is 7e16aa81f9f6a7cfe2287b788a7d62abc2880185:

  Author: Christoffer Dall <christoffer.dall@linaro.org>

  KVM: arm/arm64: Fix preemptible timer active state crazyness

  We were setting the physical active state on the GIC distributor in a
  preemptible section, which could cause us to set the active state on
  different physical CPU from the one we were actually going to run on,
  hacoc ensues.

  Since we are no longer descheduling/scheduling soft timers in the
  flush/sync timer functions, simply moving the timer flush into a
  non-preemptible section.

  Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
  Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>

> arm/arm64: KVM: Properly account for guest CPU time
>
> Is there a way to do it without disabling preemption ?

If the concern is touching the wrong per-CPU GIC distributor registers,
it should be sufficient on -rt to downgrade the preempt_disable() /
preempt_enable() to a migrate_disable() / migrate_enable(), which is
preemptible, but prevents the task from moving to another CPU.

  Josh

diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 4f5c42a..2ce9cc2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -568,7 +568,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 		 * involves poking the GIC, which must be done in a
 		 * non-preemptible context.
 		 */
-		preempt_disable();
+		migrate_disable();
 		kvm_timer_flush_hwstate(vcpu);
 		kvm_vgic_flush_hwstate(vcpu);
 
@@ -587,7 +587,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 			local_irq_enable();
 			kvm_timer_sync_hwstate(vcpu);
 			kvm_vgic_sync_hwstate(vcpu);
-			preempt_enable();
+			migrate_enable();
 			continue;
 		}
 
@@ -641,7 +641,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 
 		kvm_vgic_sync_hwstate(vcpu);
 
-		preempt_enable();
+		migrate_enable();
 
 		ret = handle_exit(vcpu, run, ret);
 	}
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index e4a0b8c..dec1156 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -2135,7 +2135,7 @@ EXPORT_SYMBOL_GPL(irq_get_irqchip_state);
  *	This call sets the internal irqchip state of an interrupt,
  *	depending on the value of @which.
  *
- *	This function should be called with preemption disabled if the
+ *	This function should be called with migration disabled if the
  *	interrupt controller has per-cpu registers.
  */
 int irq_set_irqchip_state(unsigned int irq, enum irqchip_irq_state which,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Jaggi, Manish]

Thanks Josh.
I tried both options
a) replace prempt_XXX with migrate_XXX as per the patch
b) replacing spin_lock in kvm_vgic_flush_hwstate with raw_spin_lock

Host and guest has NO_HZ_FULL=y 
The cyclictest avg latency / max latency are very high in guest.

host: Avg:   28 Max:     180
guest: Avg:  168 Max:   19532

Running guest within a cset created with -k option

Regards,
Manish Jaggi

________________________________________
From: Josh Cartwright <joshc@ni.com>
Sent: Wednesday, February 10, 2016 10:09 PM
To: Jaggi, Manish
Cc: linux-rt-users; Christoffer Dall
Subject: Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

On Wed, Feb 10, 2016 at 09:33:28AM +0000, Jaggi, Manish wrote:
> I am trying to run a kvm guest on a host with 4.4 rt3 patchset
> applied. (Cavium thunderX arm64 system) Getting the following error:
>
> BUG: scheduling while atomic: qemu-system-aar/41889/0x00000002
> [  341.647463] Modules linked in: ipv6 thunderx_edac_lmc thunderx_edac_ccpi i2c_octeon edac_core shpchp aes_ce_blk ablk_helper cryptd aes_ce_cipher ghash_ce sha2_ce sha1_ce uio_pdrv_genirq rtc_efi uio
> [  341.647477] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
> [  341.647478]
> [  341.647484] CPU: 2 PID: 41889 Comm: qemu-system-aar Not tainted 4.4.0-rt3-00120-gbb974fa #64
> [  341.647486] Hardware name: www.cavium.com ThunderX CRB1S/ThunderX CRB1S, BIOS 0.3 Dec  3 2015
> [  341.647488] Call trace:
> [  341.647494] [<ffff800000097878>] dump_backtrace+0x0/0x160
> [  341.647499] [<ffff8000000979fc>] show_stack+0x24/0x30
> [  341.647503] [<ffff800000512608>] dump_stack+0x88/0xa8
> [  341.647509] [<ffff8000000f25c0>] __schedule_bug+0x70/0xc0
> [  341.647514] [<ffff8000008f8f38>] __schedule+0x510/0x580
> [  341.647517] [<ffff8000008f90e8>] schedule+0x50/0xf0
> [  341.647521] [<ffff8000008fa9a4>] rt_spin_lock_slowlock+0x124/0x2e0
> [  341.647525] [<ffff8000008fc5e0>] rt_spin_lock+0x60/0x70
> [  341.647530] [<ffff8000000bffe0>] kvm_vgic_flush_hwstate+0x60/0x278
> [  341.647535] [<ffff8000000b3140>] kvm_arch_vcpu_ioctl_run+0x108/0x618
> [  341.647547] [<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
> [  341.647553] [<ffff80000024b4dc>] do_vfs_ioctl+0x364/0x628
> [  341.647556] [<ffff80000024b834>] SyS_ioctl+0x94/0xa8
> [  341.647560] [<ffff800000093b04>] el0_svc_naked+0x38/0x3c
>
> The below patch enables preemption:
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/arch/arm/kvm/arm.c?h=v4.4&id=1b3d546daf85ed2bc9966e12cee3e6435fb65eca

Another relevant commit is 7e16aa81f9f6a7cfe2287b788a7d62abc2880185:

  Author: Christoffer Dall <christoffer.dall@linaro.org>

  KVM: arm/arm64: Fix preemptible timer active state crazyness

  We were setting the physical active state on the GIC distributor in a
  preemptible section, which could cause us to set the active state on
  different physical CPU from the one we were actually going to run on,
  hacoc ensues.

  Since we are no longer descheduling/scheduling soft timers in the
  flush/sync timer functions, simply moving the timer flush into a
  non-preemptible section.

  Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
  Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>

> arm/arm64: KVM: Properly account for guest CPU time
>
> Is there a way to do it without disabling preemption ?

If the concern is touching the wrong per-CPU GIC distributor registers,
it should be sufficient on -rt to downgrade the preempt_disable() /
preempt_enable() to a migrate_disable() / migrate_enable(), which is
preemptible, but prevents the task from moving to another CPU.

  Josh

diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 4f5c42a..2ce9cc2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -568,7 +568,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
                 * involves poking the GIC, which must be done in a
                 * non-preemptible context.
                 */
-               preempt_disable();
+               migrate_disable();
                kvm_timer_flush_hwstate(vcpu);
                kvm_vgic_flush_hwstate(vcpu);

@@ -587,7 +587,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
                        local_irq_enable();
                        kvm_timer_sync_hwstate(vcpu);
                        kvm_vgic_sync_hwstate(vcpu);
-                       preempt_enable();
+                       migrate_enable();
                        continue;
                }

@@ -641,7 +641,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)

                kvm_vgic_sync_hwstate(vcpu);

-               preempt_enable();
+               migrate_enable();

                ret = handle_exit(vcpu, run, ret);
        }
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index e4a0b8c..dec1156 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -2135,7 +2135,7 @@ EXPORT_SYMBOL_GPL(irq_get_irqchip_state);
  *     This call sets the internal irqchip state of an interrupt,
  *     depending on the value of @which.
  *
- *     This function should be called with preemption disabled if the
+ *     This function should be called with migration disabled if the
  *     interrupt controller has per-cpu registers.
  */
 int irq_set_irqchip_state(unsigned int irq, enum irqchip_irq_state which,

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Christoffer Dall]

On Wed, Feb 10, 2016 at 5:39 PM, Josh Cartwright <joshc@ni.com> wrote:
> On Wed, Feb 10, 2016 at 09:33:28AM +0000, Jaggi, Manish wrote:
>> I am trying to run a kvm guest on a host with 4.4 rt3 patchset
>> applied. (Cavium thunderX arm64 system) Getting the following error:
>>
>> BUG: scheduling while atomic: qemu-system-aar/41889/0x00000002
>> [  341.647463] Modules linked in: ipv6 thunderx_edac_lmc thunderx_edac_ccpi i2c_octeon edac_core shpchp aes_ce_blk ablk_helper cryptd aes_ce_cipher ghash_ce sha2_ce sha1_ce uio_pdrv_genirq rtc_efi uio
>> [  341.647477] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
>> [  341.647478]
>> [  341.647484] CPU: 2 PID: 41889 Comm: qemu-system-aar Not tainted 4.4.0-rt3-00120-gbb974fa #64
>> [  341.647486] Hardware name: www.cavium.com ThunderX CRB1S/ThunderX CRB1S, BIOS 0.3 Dec  3 2015
>> [  341.647488] Call trace:
>> [  341.647494] [<ffff800000097878>] dump_backtrace+0x0/0x160
>> [  341.647499] [<ffff8000000979fc>] show_stack+0x24/0x30
>> [  341.647503] [<ffff800000512608>] dump_stack+0x88/0xa8
>> [  341.647509] [<ffff8000000f25c0>] __schedule_bug+0x70/0xc0
>> [  341.647514] [<ffff8000008f8f38>] __schedule+0x510/0x580
>> [  341.647517] [<ffff8000008f90e8>] schedule+0x50/0xf0
>> [  341.647521] [<ffff8000008fa9a4>] rt_spin_lock_slowlock+0x124/0x2e0
>> [  341.647525] [<ffff8000008fc5e0>] rt_spin_lock+0x60/0x70
>> [  341.647530] [<ffff8000000bffe0>] kvm_vgic_flush_hwstate+0x60/0x278
>> [  341.647535] [<ffff8000000b3140>] kvm_arch_vcpu_ioctl_run+0x108/0x618
>> [  341.647547] [<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750
>> [  341.647553] [<ffff80000024b4dc>] do_vfs_ioctl+0x364/0x628
>> [  341.647556] [<ffff80000024b834>] SyS_ioctl+0x94/0xa8
>> [  341.647560] [<ffff800000093b04>] el0_svc_naked+0x38/0x3c
>>
>> The below patch enables preemption:
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/arch/arm/kvm/arm.c?h=v4.4&id=1b3d546daf85ed2bc9966e12cee3e6435fb65eca
>
> Another relevant commit is 7e16aa81f9f6a7cfe2287b788a7d62abc2880185:
>
>   Author: Christoffer Dall <christoffer.dall@linaro.org>
>
>   KVM: arm/arm64: Fix preemptible timer active state crazyness
>
>   We were setting the physical active state on the GIC distributor in a
>   preemptible section, which could cause us to set the active state on
>   different physical CPU from the one we were actually going to run on,
>   hacoc ensues.
>
>   Since we are no longer descheduling/scheduling soft timers in the
>   flush/sync timer functions, simply moving the timer flush into a
>   non-preemptible section.
>
>   Reviewed-by: Marc Zyngier <marc.zyngier@arm.com>
>   Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
>
>> arm/arm64: KVM: Properly account for guest CPU time
>>
>> Is there a way to do it without disabling preemption ?
>
> If the concern is touching the wrong per-CPU GIC distributor registers,
> it should be sufficient on -rt to downgrade the preempt_disable() /
> preempt_enable() to a migrate_disable() / migrate_enable(), which is
> preemptible, but prevents the task from moving to another CPU.
>

Indeed!


>
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index 4f5c42a..2ce9cc2 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -568,7 +568,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
>                  * involves poking the GIC, which must be done in a
>                  * non-preemptible context.
>                  */
> -               preempt_disable();
> +               migrate_disable();
>                 kvm_timer_flush_hwstate(vcpu);
>                 kvm_vgic_flush_hwstate(vcpu);
>
> @@ -587,7 +587,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
>                         local_irq_enable();
>                         kvm_timer_sync_hwstate(vcpu);
>                         kvm_vgic_sync_hwstate(vcpu);
> -                       preempt_enable();
> +                       migrate_enable();
>                         continue;
>                 }
>
> @@ -641,7 +641,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
>
>                 kvm_vgic_sync_hwstate(vcpu);
>
> -               preempt_enable();
> +               migrate_enable();
>
>                 ret = handle_exit(vcpu, run, ret);
>         }
> diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
> index e4a0b8c..dec1156 100644
> --- a/kernel/irq/manage.c
> +++ b/kernel/irq/manage.c
> @@ -2135,7 +2135,7 @@ EXPORT_SYMBOL_GPL(irq_get_irqchip_state);
>   *     This call sets the internal irqchip state of an interrupt,
>   *     depending on the value of @which.
>   *
> - *     This function should be called with preemption disabled if the
> + *     This function should be called with migration disabled if the
>   *     interrupt controller has per-cpu registers.
>   */
>  int irq_set_irqchip_state(unsigned int irq, enum irqchip_irq_state which,

Without knowing the details of RT or migrate_enable/disable, this
looks fine to me.

-Christoffer

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Christoffer Dall]

On Thu, Feb 11, 2016 at 11:46 AM, Jaggi, Manish
<Manish.Jaggi@caviumnetworks.com> wrote:
> Thanks Josh.
> I tried both options
> a) replace prempt_XXX with migrate_XXX as per the patch
> b) replacing spin_lock in kvm_vgic_flush_hwstate with raw_spin_lock
>
> Host and guest has NO_HZ_FULL=y
> The cyclictest avg latency / max latency are very high in guest.
>
> host: Avg:   28 Max:     180
> guest: Avg:  168 Max:   19532
>
> Running guest within a cset created with -k option
>
Our interrupt delivery slowdown is quite high due to the expensive
world-switch path on KVM/ARM.

Marc (cc'ed) recently did some optimizations in this area which may be
worth looking at.

-Christoffer

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Sebastian Andrzej Siewior]

* Jaggi, Manish | 2016-02-11 10:46:16 [+0000]:

>Thanks Josh.
>I tried both options
>a) replace prempt_XXX with migrate_XXX as per the patch
so Josh's patch worked, correct?

>b) replacing spin_lock in kvm_vgic_flush_hwstate with raw_spin_lock
>

Sebastian

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Jaggi, Manish]

________________________________________
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Sent: Thursday, February 11, 2016 8:58 PM
To: Jaggi, Manish
Cc: Josh Cartwright; linux-rt-users; Christoffer Dall
Subject: Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

* Jaggi, Manish | 2016-02-11 10:46:16 [+0000]:

>Thanks Josh.
>I tried both options
>a) replace prempt_XXX with migrate_XXX as per the patch
so Josh's patch worked, correct?

[manish] yes.
But latency is on higher side in vm. 

>b) replacing spin_lock in kvm_vgic_flush_hwstate with raw_spin_lock
>

Sebastian

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Sebastian Andrzej Siewior]

* Josh Cartwright | 2016-02-10 10:39:30 [-0600]:

>If the concern is touching the wrong per-CPU GIC distributor registers,
>it should be sufficient on -rt to downgrade the preempt_disable() /
>preempt_enable() to a migrate_disable() / migrate_enable(), which is
>preemptible, but prevents the task from moving to another CPU.

Thanks Josh. Could you please post a patch with sign-of-by and
everything?

>
>  Josh

Sebastian

Re: [4.4rt3] Preemption disabled at:[<ffff8000000aba44>] kvm_vcpu_ioctl+0x30c/0x750

[Marc Zyngier]

On 11/02/16 14:02, Christoffer Dall wrote:
> On Thu, Feb 11, 2016 at 11:46 AM, Jaggi, Manish
> <Manish.Jaggi@caviumnetworks.com> wrote:
>> Thanks Josh.
>> I tried both options
>> a) replace prempt_XXX with migrate_XXX as per the patch
>> b) replacing spin_lock in kvm_vgic_flush_hwstate with raw_spin_lock
>>
>> Host and guest has NO_HZ_FULL=y
>> The cyclictest avg latency / max latency are very high in guest.
>>
>> host: Avg:   28 Max:     180
>> guest: Avg:  168 Max:   19532
>>
>> Running guest within a cset created with -k option
>>
> Our interrupt delivery slowdown is quite high due to the expensive
> world-switch path on KVM/ARM.
> 
> Marc (cc'ed) recently did some optimizations in this area which may be
> worth looking at.

You can have a look at the branch I have on

git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git
kvm-arm64/suck-less

though most of it is for GICv2 (and my suspicion is that we're talking
about GICv3 here, given the email addresses...).

Anyway, if there is some specific things that should be looked at to
improve RT in guests with KVM, I'm all ears.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH -rt 1/2] genirq: update irq_set_irqchip_state documentation

[Josh Cartwright]

On -rt kernels, the use of migrate_disable()/migrate_enable() is
sufficient to guarantee a task isn't moved to another CPU.  Update the
irq_set_irqchip_state() documentation to reflect this.

Signed-off-by: Josh Cartwright <joshc@ni.com>
---
 kernel/irq/manage.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index ba2a42a..8e89554 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -2060,7 +2060,7 @@ EXPORT_SYMBOL_GPL(irq_get_irqchip_state);
  *	This call sets the internal irqchip state of an interrupt,
  *	depending on the value of @which.
  *
- *	This function should be called with preemption disabled if the
+ *	This function should be called with migration disabled if the
  *	interrupt controller has per-cpu registers.
  */
 int irq_set_irqchip_state(unsigned int irq, enum irqchip_irq_state which,
-- 
2.7.0

[PATCH -rt 2/2] KVM: arm/arm64: downgrade preempt_disable()d region to migrate_disable()

[Josh Cartwright]

kvm_arch_vcpu_ioctl_run() disables the use of preemption when updating
the vgic and timer states to prevent the calling task from migrating to
another CPU.  It does so to prevent the task from writing to the
incorrect per-CPU GIC distributor registers.

On -rt kernels, it's possible to maintain the same guarantee with the
use of migrate_{disable,enable}(), with the added benefit that the
migrate-disabled region is preemptible.  Update
kvm_arch_vcpu_ioctl_run() to do so.

Cc: Christoffer Dall <christoffer.dall@linaro.org>
Reported-by: Manish Jaggi <Manish.Jaggi@caviumnetworks.com>
Signed-off-by: Josh Cartwright <joshc@ni.com>
---
 arch/arm/kvm/arm.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 4f5c42a..2ce9cc2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -568,7 +568,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 		 * involves poking the GIC, which must be done in a
 		 * non-preemptible context.
 		 */
-		preempt_disable();
+		migrate_disable();
 		kvm_timer_flush_hwstate(vcpu);
 		kvm_vgic_flush_hwstate(vcpu);
 
@@ -587,7 +587,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 			local_irq_enable();
 			kvm_timer_sync_hwstate(vcpu);
 			kvm_vgic_sync_hwstate(vcpu);
-			preempt_enable();
+			migrate_enable();
 			continue;
 		}
 
@@ -641,7 +641,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 
 		kvm_vgic_sync_hwstate(vcpu);
 
-		preempt_enable();
+		migrate_enable();
 
 		ret = handle_exit(vcpu, run, ret);
 	}
-- 
2.7.0

Re: [PATCH -rt 2/2] KVM: arm/arm64: downgrade preempt_disable()d region to migrate_disable()

[Sebastian Andrzej Siewior]

* Josh Cartwright | 2016-02-11 11:54:01 [-0600]:

>kvm_arch_vcpu_ioctl_run() disables the use of preemption when updating
>the vgic and timer states to prevent the calling task from migrating to
>another CPU.  It does so to prevent the task from writing to the
>incorrect per-CPU GIC distributor registers.
>
>On -rt kernels, it's possible to maintain the same guarantee with the
>use of migrate_{disable,enable}(), with the added benefit that the
>migrate-disabled region is preemptible.  Update
>kvm_arch_vcpu_ioctl_run() to do so.
>
>Cc: Christoffer Dall <christoffer.dall@linaro.org>
>Reported-by: Manish Jaggi <Manish.Jaggi@caviumnetworks.com>
>Signed-off-by: Josh Cartwright <joshc@ni.com>

Thanks, applied.

Sebastian